Hyperbolic Deep Reinforcement Learning
Authors: Edoardo Cetin
Benjamin Paul Chamberlain
Michael M. Bronstein
Jonathan J Hunt
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Reinforcement learning
Hyperbolic space
Representation learning
Machine learning

Abstract: In deep reinforcement learning (RL), useful information about the state is inherently tied to its possible future successors. Consequently, encoding features that capture the hierarchical relationships between states into the model's latent representations is often conducive to recovering effective policies. In this work, we study a new class of deep RL algorithms that promote encoding such relationships by using hyperbolic space to model latent representations. However, we find that a naive application of existing methodology from the hyperbolic deep learning literature leads to fatal instabilities due to the non-stationarity and variance characterizing common gradient estimators in RL. Hence, we design a new general method that directly addresses such optimization challenges and enables stable end-to-end learning with deep hyperbolic representations. We empirically validate our framework by applying it to popular on-policy and off-policy RL algorithms on the Procgen and Atari 100K benchmarks, attaining near universal performance and generalization benefits. Given its natural fit, we hope this work will inspire future RL research to consider hyperbolic representations as a standard tool.

Published as a conference paper at ICLR 2023
HYPERBOLIC DEEP REINFORCEMENT LEARNING
Edoardo Cetin"t Benjamin P Chamberlaini Michael M Bronsteint Jonathan J Hunt
King's ' College London Charm Therapeutics
University of Oxford
Twitter Inc.
ABSTRACT
In deep reinforcement learning (RL), useful information about the state is inher- ently tied its possible future successors. Consequently, encoding features that capture the hierarchical relationships between states into the model's latent rep- resentations often conducive recovering effective policies. In this work, we study new class of deep RL algorithms that promote encoding such relationships by using hyperbolic space to model latent representations. However, we find that a naive application of existing methodology from the hyperbolic deep learning literature leads fatal instabilities due the non-stationarity and variance char- acterizing common gradient estimators in RL. Hence, we design a new general method that directly addresses such optimization challenges and enables stable end-to-end learning with deep hyperbolic representations, We empirically validate our framework by applying to popular on-policy and off-policy RL algorithms the Procgen and Atari 100K benchmarks, attaining universal performance and generalization benefits. Given its natural fit, we hope this work will inspire future RL research to consider hyperbolic representations a standard tool.
INTRODUCTION
Reinforcement Learning (RL) achieved notable
milestones in several game-playing and robotics ap- plications (Mnih et al., 2013) Vinyals et al.) 2019
Kalashnikov et al. 2018: OpenAl et al.] 2019 Lee
et (a.1)) However, all these recent advances re- lied on large amounts of data and domain-speciffe practices, restricting their applicability in many im- portant real-world contexts (Dulac-Arnold et al.) 2019). We argue that these challenges are symp- tomatic of current deep RL. models lacking a proper prior to efficiently learn generalizable features for control (Kirk et al. 2021]. We propose to tackle this
Hierarchical relationship between states in breakout, visualized in hyperbolic space. issue by introducing hyperbolic geomelry to RL, as new inductive bias for representation learning.
Figure 1:
The evolution of the state in a Markov decision process can be conceptualized as a tree, with the pol- icy and dynamics determining the possible branches. Analogously, the same hierarchical evolution often applies to the most significant features required for decision-making (e.g., presence of bricks, location of paddle/ball in Fig. D. These relationships tend to hold beyond individual trajectories, making hierarchy a natural basis to encode information for RL (Flet-Berliac (2019). Consequently, hypothesize that deep RL models should prioritize encoding precisely hieranchicullysstruuuure features to facilitate learning effective and generalizable policies. In contrast, we note that non- evolving features, such the aesthetic properties of elements in the environment, are often linked with spurious correlations, hindering generalization to new states (Song et al. 2019). Similarly. hu- man cognition also appears to learn representations of actions and elements of the environment by focusing on their underlying hierarchical relationship (Barker & Wright 1955 Zhou et al. 2018).
Hyperbolic geometry (Beltrami 1868 Cannon et al. 1997) provides a natural choice to efficiently
encode hierarchically-strucei
features. A defining property of hyperbolic space is exponential volume growth, which enables the embedding tree-like hierarchical data with low distortion us- ing only a few dimensions (Sarkar, 201T). In contrast, the volume of Euclidean spaces only grows
"E-mail: edoardo. cet n@ke1 ac.uk; work done while at Twitter Inc.

---

Published as a conference paper at ICLR 2023
polynomially, requiring high dimensionality to precisely embed tree structures (Matoušek 1990),
potentially leading to higher complexity, more parameters, and overfitting. We analyze the proper- ties of learned RL representations using measure based on the 8-hyperbolicity (Gromov 1987), quantifying how close an arbitrary metric space is to a hyperbolic one. In line with our intuition, we show that performance improvements of RI algorithms correlate with the increasing hyperbolicity of the discrete space spanned by their latent representations. This result validates the importance of appropriately encoding hierarchical information, suggesting that the inductive bias provided by employing hyperbolic representations would facilitate recovering effective solutions.
Hyperbolic geometry has recently been exploited in other areas of machine learning showing sub- stantial performance and efficiency benefits for learning representations of hierarchical and graph data (Nickel Kiela, 2017 Chamberlain et al. 2017). Recent contributions further extended tools from modern deep learning to work in hyperbolic space (Ganea et al. 2018 Shimizu et al., 2020). validating their effectiveness in both supervised and unsupervised learning tasks (Khrulkov et al.)
2020 Nagano et al. 2019 Mathieu et al. 2019). However, most of these approaches showed clear improvements on smaller-seale problems that failed to hold when scaling to higher-dimensional data and representations. Many of these shortcomings are tied to the practical challenges of optimizing hyperbolic and Euclidean parameters end-to-end (Guo et al., (2022). In RL, We show the non- stationarity and high-variance characterizing common gradient estimators exacerbates these issues, making a naive incorporation of existing hyperbolic layers yield underwhelming results.
In this work, we overcome the aforementioned challenges and effectively train deep RL algorithms with latent hyperbolic representations end-to-end. In particular, we design pneettally-regulariin hy- perbolic mappings (S-RYM), simple recipe combining scaling and spectral normalization (Miyato) et al. (018)) that stabilizes the learned hyperbolic representations and enables their seamless integra- tion with deep RL. We use S-RYM to build hyperbolic versions of both on -n-oliiy (Schulman al., 2017) and off- off-policy algorithms (Hessel et al., 2018), and evaluate on both Procgen (Cobbe et al,, 2020) and Atari 100K benchmarks (Bellemare al. 2013). We show that our framework attains near universal performance and generalization improvements over established Fuclidean baselines, making even general algorithms competitive with highly-tunde SotA baselines. We hope our work will set new standard and be the first of many incorporating hyperbolic representations with RL. To this end, we share our implementation at sites. . google. wwwp///wwwweeeeeeoollleeeeeeeeee
2
PRELIMINARIES
In this section, we introduce the main definitions required for the remainder of the paper. We refer to App. A and (Cannon et al., 1997) for further details about RL and hyperbolic space, respectively.
2.1 REINFORCEMENT LEARNING
The RL problem setting is traditionally described as a Markov Decision Process (MDP), defined by the tuple (S, A, P. Po. r, At each timestep t, an agent interacts with the environment, observing some state from the state space 8 E S, executing some action from its action space a E A, and receiving some reward according to its reward function r :Sx A + R. The transition dynamics P: SxAxS + R and initial state distribution Pa : + R determine the evolution of the enviromment's state while the discount factor 7 [0, 1) quantifies the agent's 's preference for earlier rewards. Agent behavior in RL can be represented by a parameterized distribution function To. whose sequential interaction with the environment yields some trajectory (8o, ag 81, 01 ST, ar). The agent's objective is to learn a policy maximizing its expected discounted sum of rewards over trajectories:
arg max E. P
Σ )'r(s),
We differentiate two main classes of RL algorithms with very different optimization procedures based on their different usage of the collected data. On-policy algorithms collect a new set of tra- jectories with the latest policy for each training iteration, discarding old data. In contrast, off-policy algorithms maintain a large replay buffer of past experiences and use it for learning useful quantities about the environment, such as world models and value functions. Two notable instances from each class are Proximal Policy Optimization (PPO) (Schulman et al.) (017)) and Rainbow DQN (Hessel et al. 2018), upon which many recent advances have been built.
2

---

Published as a conference paper at ICLR 2023
2.2 MACHINE LEARNING IN HYPERBOLIC SPACES
A hyperbolic space H is an n-dimensional Riemannian manifold with constant negative sectional curvature -c. [eltrami (1668) showed the equiconsistency of hyperbolic and Euclidean geometry using a model named after its re-discoverer, the Poincaré ball model. This model equips an n- dimensional open ball B {x R c||x|| < 1] of radius 1/ve with a conformal metric of the form Gx = I where Ax T-clix| is the conformal factor (we will omit the dependence on the curvature -c in our definitions for notation brevity). The geodesic (shortest path) between two points in this metric is a circular are perpendicular to the boundary with the length given by:
d(x 1 cosh -1 (1 20 (1 c|x
xx yl1
22)
1P)
(1-clly111))
From these characteristics, hyperbolic spaces can be viewed as a continuous analog of trees. In particular, the volume of a ball on H" grows exponentially w.r.t. its radius. This property mirrors the exponential node growth in trees with constant branching factors. Visually, this makes geodesics between distinct points pass through some midpoint with lower magnitude, analogously to how tree geodesics be- tween nodes (defined as the shortest path in their graph) must cross their closest shared parent (Fig. 2
Key operations for learning, On Riemannian manifold, the exponential map cvp outputs unit step along geodesic starting from point x in the direction of an input velocity v. It thus allows locally treating H as Euclidean space. We use the exponential map from the origin of the Poincaré ball map Euclidean input vectors into H!"
Figure 2: Geodesics on H² and shortest
paths connecting nodes of a trec.
expo (v) tanh (vdlvil) Vellvll
Following Ganea et al. (0018), we consider the framework of gyrovector spaces (Ungar 2008 to
extend common vector operations to non- -un-Euciddenn geometries, and in particular H". The most basic such generalized operation is the Mobius addition @ of two vectors,
x y (1 + 2x(x,)) + c (1 + c||x||*)y
(4)
1 2c(x,, y) + c |||||||
Next, consider a Fuclidean affine transformation f(x) = (x, w) + b used in typical neural network layers. We can rewrite this transformation as f(x) normal and shift parameters of a hyperplane H = [y E R (y -y-pe w) 0} (Lebanon & Lafferty
(x p, w) and interpret w.p 1g as the
2004). This allows to further rewrite fxx in terms of the signed distance the hyperplane H, effectively acting as weighted 'decision boundary':
ʃ(x) sign ((x p. w)) |/w]|d(x H).
(5)
This formulation allows extend affine transformations to the Poincaré ball by considering the signed distance from a gyroplane in [dd (generalized hyperplane) H = [y E Bd (y ( w)
0}.
2/|w 2|w]]
1 sinh -1 2/cl(xx @ P. w)l
f(x) sign((x (x + -pw))
d(x, H); d(x,
(6)
VI- cp2
Ve
(1 c x + - >P
Similarly to recent hyperbolic deep learning work (Mathieu et al. 2019 Guo et al., 2022), we use these operations to parameterize hybrid neural nets: we first process the input data x with standard layers to produce Euclidean vectors XÆ = fε(x), Then, we obtain hyperbolic representations by applying the exponential map treating X as a velocity, Xи (xE) Finally, we use linear operations of the form in Eq. to output the set of policy and value scalars, fu (xx = (xxx))) }. We extend this model with new stabilization (.3) with observed benefits beyond RL (App.
3 HYPERBOLIC REPRESENTATION FOR REINFORCEMENT LEARNING
In this section, we base our empirical RL analysis on Procgen (Cobee et al. 2020). This benchmark consists of 16 visual environments, with procedurally-geeeratte practice, we train agents using exclusively the first 200 levels of each environment and evaluate on the full distribution of levels to assess agent performance and generalization.
random levels. Following common
3

---

Published as a conference paper at ICLR 2023
fun
Bapp
Figure 3: A geodesic space is 6-hyperbolic if every triangle is 6-slim, i..., each of its sides is entirely contained within a 5-sized region from the other two. We illustrate the necessary to satisfy this property for ABC in a tree triangle (Left), hyperbolic trianglc (Center) and an Euclidean triangle (Right); sharing vertex coordi- nates. In tree triangles, Stree 0 since AC always intersects both AB and BC. bigfish
dodgeball
fruitbot
starpilot
0.35
30
0.35 30
0.35
Epinodle w
0.30 4
20
0.30
0.3020
0.30
0.25
10
2
0.25
0.25 10
0.25
0.20
0,20
10000
20000
0
10000
20000
10000
20000
10000
20000
143
340
340
PPO performance (AB test levels)
--- PPO perforannce 1200 trainingg levels)
6 hyperboliity
Figure 4: Performance and relative d-hyperbolicity of the final latent representations a PPO agent.
3.1 THE INHERENT HYPERBOLICITY OF DEEP RL
Key quantities for each state, such as the value and the policy, are naturally related to its possi- ble successors. In contrast, other fixed, non-hierarchical information about the environment such its general appearance,
often be safely ignored. This divide becomes particularly relevant when considering the problem of RL generalization. For instance, |Raleaau & Fergus) (202T) found that agents' can overfit spurious correlations between the value and non-hierarchical features (e.g., background color) in the observed states. Hence, we hypothesize that effective representations should encode features directly related to the hierarchical state relationships of MDPs.
o-hyperbolicity. We analyze the representation spaces learned by RL agents, testing whether they preserve and reflect this hierarchical structure. We use the 8-hyperbolicity of metric space (X, d) (Gromov 1987, Bonk & Schramm 2011), which we formally describe in App. A.2 For our use- case, X is ò o-hyperbolic if every possible geodesic triangle Aryz X is d-slim. This means that for every point on any side of Aryz there exists some point on one of the other sides whose distance is at most In trees, every point belongs to at least two of its sides yielding ó = (Figure 3). Thus, we can interpret 6-hyperbolicity as measuring the deviation of a given metric from an exact tree metric.
The representations learned by an RL agent from encoding the collected states span some finite subset of Euclidean space Xg E X R", yielding discrete metric space Xg. To test our hypothesis, we compute the ò- o-hyperbolicity of Xg and analyze how it relates to agens performance. Similarly (Khrulkov et al. 2020), et al. 2015). To account for the scale of the representations, we normalize 5 by diamí X E), yielding a relative hyperbolicity measure dret 28/ /diam 8) (Borassi et al. 2015). which can span values between 0 (hyperbolic hierarchical tree-like structure) and I (periectly non-hyperbolic spaces).
compute using the efficient algorithm proposed by Fournier
Results. We train an agent with PPO (Schulman al. 2017) on four Procgen environments, encod- ing states from the latest rollouts using the representations before the final linear policy and value heads, Fg = fs()). Hence, we estimate 6rel from the space spanned by these latent encodings as training progresses. As shown in Figure 4 ô,at quickly drops to low values (0.22 - 0.28) in the first training iterations, reflecting the largest relative improvements agent performance. Subsequently, in the fruithot and starpilot environments, 6rei further decreases throughout training as the agent recovers high performance with a low generalization gap between the training and test distribution of levels. Instead, in bigfish and dodgeball, öret begins to increase again after the initial drop, sug- gesting that the latent representation space starts losing its hierarchical structure. Correspondingly. the agent starts overfitting test levels performance stagnates while the generalization gap with the training levels performance keeps increasing. We believe these results support our hypothesis, empirically showing the importance of encoding hierarchical features for recovering effective solu-
4

---

Published as a conference paper at ICLR 2023
All test levels bigfish
200 training levels
Final layer backpropagated gradiant
Convolutiona encoder gradient starpilot
A.1)
starpilot
B.1)
bigfish
10
Relative 1.2 norm.
Epsodde return
30
5.0
20
100
2.5
10
10-1
0.0
0
10000
20000
10000
20000
10000
20000
10000
20000
A.2)
bigfish
starpilot
10 B.2)
bigfish
starpilot
0
10
Relarive yariance
Entrogy losa
-1
10a
10
100
10
-2
0
10000 20000
10000
20000
10000
20000
0
10000
20000
PPO Hyperbolic PPO
Hyperbollic PPO + clppping
Figure 5: Analysis of key statistics for our naive implementations of hyperholic PPO agents using existing practices stabilize optimization in hyperbolic space. On the left, we display performance (A.1) and negative entropy (A.2). On the right, we display magnitudes (B.1) and variances (B.2) of the backpropagated gradients.
tions. Furthermore, they suggest that PPO's poor generalization in some environments is due to the observed tendency of Euclidean latent spaces to encode spurious features that hinder hyperbolicity.
Motivated by our findings, we propose employing hyperbolic geometry to model the latent rep- resentations of deep RL models. Representing tree-metrics in Euclidean spaces incurs non-trivial worse-case distortions, growing with the number of nodes at a rate dependent on the dimensionality (Matoušek) 1990). This property suggests that it is not possible to encode complex hierarchies in Eu- clidean space both efficiently and accurately, explaining why some solutions learned by PPO could not maintain their hyperbolicity throughout training. In contrast, mapping the latent representations to hyperbolic spaces of any dimensionality enables encoding features exhibiting a tree-structured relation over the data with arbitrarily low distortion (Sarkar 201I). Hence, hyperbolic latent repre- sentations introduce a different inductive bias for modeling the policy and value function, stemming from this inherent efficiency of specifically encoding hierarchical information (Tifrea et al. 2018
3.2 OPTIMIZATION CHALLENGES
fn m(})
Naive integration. We test a simple extension to PPO, map- ping the latent representations of states E S before the final linear policy and value heads Vg = (s) to the Poincaré ball with unitary curvature. As described in Section 2, we perform this with an exponential map to produce z = cxp] (xg), re- placing the final ReI.U. To output the value and policy logits, we then finally perform a set of affine transformations in hy- perbolic space, =(s). V(s) (x
Vis)
Gyroplane fully connected
Exp. map
Fully connected
xa C16, 32, 20
ReLU
chaunel
{f. (1=) We also consider a clipped version of this integration. following the recent stabilization practice from Guo et al. (2022), which entails clipping the magnitude of the latent representations to not exceed unit norm. We initialize the weights of the last two linear layers in both implementations to 100x smaller val- ues to start training with low magnitude latent representations, which facilitates the network first learning appropriate angular layouts (Nickel& & Kiela 2017 Ganea et al. 2018).
Impala residual block
Impala residual block
Max pooling 3x3, stride 2
Convolution 3x3
Figure 6: PPO model with hyper- bolic latent space, extending the archi- tecture from (ropholp et al. (2018).
Results. We analyze this naive hyperbolic PPO implementa- tion in F As shown in part (A.1), performance gener- ally underwhelming, lagging considerably behind the performance of standard PPO. While applying the clipping strategy yields some improvements, its results are still considerably inferior on the tasks where Euclidean embeddings appear to already recover effective representations (e.g. starpilor). In part (A.2) we visualize the negated entropy of the different PPO agents. PPO's policy optimization objective includes both a reward maximization term, which requires auxiliary estimator, and an entropy bonus term that can instead be differentiated exactly and optimized end-to-end. Its purpose is to push PPO agents to explore if they struggle to optimize performance with the current data.
5

---

Published as a conference paper at ICLR 2023
All test levels
200 training levels
Final layer backpropagated gradient
Convolutional encoder gradient
A)
bigfish
dodgeball
B)
bigfish
dodgeball
15
10a
10
Epioodic veturn
7.5
Relative 1.2 norm.
10
10a
10
2.5
100
100
10000 fruitbot
20000
10000
20000
10000
20000
0
10000
20000
starpilot
fruitbot
300
starpilot
30
40
10a
10
Egisodie roturn
Relative 12mmm.
10
10
20
109
109
10000
20000
10000 20000
10000
20000
10000
20000
PPO
Hyperbolic PPO Hyperbolic 990 S S-RMM
Euclidean PPO + 5-RM
Figure 7: Analysis of hyperbolic PPO with the proposed S- S-RYM stabilization. We visualize performance (A)
and gradient magnitudes (B) as compared to the original Euclidean and the naive hyperbolic baselines.
We note that the Hyperbolic PPO agents take significantly longer to reach higher levels of entropy in the initial training phases and are also much slower to reduce their entropy as their performance improves. These results appear to indicate the presence of optimization challenges stemming from end-to-end RL training with hyperbolic representations. Therefore, lyzing the gradients in our hyperbolic models. In part (B.1), we visualize the magnitude of the gradients both as backpropagated from the final representations and to the convolutional encoder. In part 2). we also visualize the variance of the same gradients with respect to the different in- put states in a minibatch. We find that hyperbolic PPO suffers from a severe exploding gradients problem, with both magnitudes and variances being several orders of magnitude larger than the Eu- clidean baseline. Similar instabilities have been documented by much recent literature, as described in App. B Yet, in the RI. case, common stabilization techniques such as careful initialization and clipping are visibly insufficient, resulting in ineffective learning and inferior agent performance.
turn our attention ana-
3.3 STABILIZING HYPERBOLIC REPRESENTATIONS
We hypothesize that the high variance and non-stationarity characterizing RL are the main cause of the observed optimization challenges of this naive hyperbolic PPO. implementation. Initialization and clipping have been designed for stationary ML applications with fixed dataset and targets. In these settings, regularizing the initial learning iterations enables the model to find appropriate angu- lar layouts of the representations for the underlying fixed loss landscape. Without appropriate an- gular layouts, useful representations become hard to recover due to the highly non-convex spectrum of hyperbolic neural networks, resulting in failure modes with low performance (Ganea al.) 2018 López& & Strube, 2020). We can intuitively see how this reliance is likely incompatible with the RL setting, where the trajectory data and loss landscape can change significantly throughout training, making early angular layouts inevitably suboptimal. We believe this is further exacerbated by the high variance gradients already characterizing policy gradient optimization (Sutton & Barto 2018) which facilitate entering unstable learning regimes that can lead to our observed failure modes.
Spectral norm. Another sub-field field of ML dealing with non-stationarity and brittle optimization is generative modeling with adversarial networks (GANs) [Goodfellow et al.] (2014). In GAN train- ing, the generated data and discriminator's parameters constantly evolve, making the loss landscape highly non-stationary as in the RL setting. Furthermore, the adversarial nature of the optimization makes it very brittle to exploding and vanishing gradients instabilities which lead to common failure modes (Arjovsky & Bottou, 2017: Brock et al. 2018). In this parallel literature, spectral normaliza- (0188) &) is a popular stabilization practice whose success made it ubiquitous in modern GAN implementations. Recent work (Lin et al. (021) showed that a reason for its surpris- ing effectiveness comes from regulating both the magnitude of the activations and their respective gradients very similarly to LeCun initialization (LeCun et al. 2012). Furthermore, when applied to the discriminator model, SN's effects appear to persist throughout training. while initialization strategies tend to only affect the initial iterations. In fact, they also show that ablating SN from GAN training empirically results in exploding gradients and degraded performance, closely resembling same observed instabilities. We provide details about GANs and SN in App. A.3.
tion (SN) (Miyato et |

---

Published as a conference paper at ICLR 2023
Table 1: Performance comparison for the considered versions of PPO full Procgen benchmark
Thask) (Algorithm
PPO
PPO aug.
PPO S-RYM
PPO S-RYM, dim.
Levels distribution
tradiseet
Inalofest
Irnin/test
Droineeest
bigfis
3.71 1,46&l
13,07 (2
(+797%)
(+.272)))) 12.20a2 (+737%)
20.58 (+155%) 16.57=2 (+1037%)
bonyfight
8.1811 7.0112
3.38u1 (-59%) 2.96.1 (-58%)
8.611l (+5%) 8.1 (+16%)
9.261l (+13%) 9,02a1 (+25%)
cuwellyer
7.01x1 5.86m1
6.08a1 4.88)) (-16%)
5.15a1 (-128)
6.38s] (-9%) 5.20s1 (-118)
chaser
6.58&2 5.8911
2.1400 2.1820 (-639)
6,6022 (+0%) 7.82+1 (+33%)
9.06E1 (+37%) 7.32+1 (+24%)
climber
8 3.6612 5.1121
7.6121 (-12%) 5,71 (+12%%)
8.9111 (+3%)) 6.6112 (+30%)
8.3211 (-196) 7.28 (+43%)
coiarun
9 9,50e0 8.25
8.40sl (-125)) 9.00)s1 (49%)
9.50el (-2%) 8.40as0 (4258)
9.70s0 (+2%)) 9.20gt (-12%)
dodgebull
5,0741 1.87a1
3,9421 (-22%) 3.20a1 (+71%%)
7.10 (+400%) 6.52gl (+248%)
(+53%)
(+281%)
fruilbot
30.1012 26.33u2
27,56 27,5613 (-8%) 27,98a1 (46%)
30/331 (+19) (-19) 27,9713 (+6%)
29.15L1 (-3%) 29.51a1 (+125%)
hein
7.42x1 2 2.92=1
4.20el (-43%) 3.60g0 (+23%)
5:40z1 (-275) 2.70:1 (-7%)
6.40t1 (-149) 3.60g1 (+23%)
jumper
8.8611 6.14
7,704l (-13%) 5.70 (-7%)
9,001
6,70=1 (+918)
8,50 (-1%) 6.10al (-19)
leaper
1.8612
6.5011 (-10%) 7,00g] (+61%%)
(+65%) 7.30al (+68%)
(459%)
(+61%)
mazz
9.25eD 6.50=0
8.50s1 (-85) 7.10m1 (49%)
9.50+0 6. 6:10:1 (-698
9.20+0 7.10al (+9%)
miner
12.9540 9. 1
9.812 9.3622 (+19)
12.09 (:7%) 10.08al (+9%)
12,9410 (+0%) 9.86:1 (+6%)
7.62ul 6.50s1
6.90al (-10%) 4.50al (-31%)
6.50ul (-15%) 6.10al (-6%)
7.50u1 (-2%) 5.6011 (-149)
plunder
6.36m
5.1 (-26%)
(-185)
7:26+1 (45%) 6.87s1 (41399)
7.35c1 6.68-00 (+10%)
Aurpplor
30.50u5 26.57
43.4317 (-12%) 32.4133 (+22%))
37.08a3 (+22%) 41.2243 (455%)
41,8811 (+36%) 38.27.5 (+445)
Average norm, score
0.5614 0,3476
0.6151 ( (-215) 0.3536 (+2%)
(+119) (-299))
0.6326 (+13%) 0.4730 (+36%)
Medismn
0,6085 0.3457
0,5262 (-145) 0,3312 (-19)
0,6055 (+0%) 0:4832 (-100)
0.6527 (+7%) 0,4705 (+36 %
improvements
W/16
1V/16
11/16 12/16
8/16 13/36
fruitbot
fruitbot 100 levels
fruitbot 50 levels
fruitbot 25 levels
30
30
30
30
20
20
20
20
Egloodle
10
10
10
10
0
0
0
10000
20000
0
10000
20000
0
10000
20000
10000
20000
starpilot
starpilot 100 levels
starpliot 50 levels
tet
starpilot 25 levels 1n5
40
40
40
40
Episodic
20
20
20
20
0
10000
20000
0
10000
20000
0
10000 20000
0
10000
20000
PPO PPO data sug. Hyperbolic PPO 5-RYM Hyperbolic PPO RYM, 32 All tast levals
Training levels
Figure 8: Performance comparison for the considered versions of PPO agents with Euclidean and hyperbolic latent representations, increasingly lowering the number training levels.
S-RYM. Inspired by these connections, we propose to counteract the optimization challenges in RL, and hyperbolic representations with SN. Our implementation differs from its usual application for GANs in two main ways. First, we apply SN to all layers in the Euclidean sub-network (fE), as the observed instabilities already occur in the gradients from the hyperbolic representations, but leave the final linear hyperbolic layer (f() unregularized to avoid further limiting expressivity. Second, we propose scale the latent representations to account for their dimensionality. In particular, modeling FE E IR by an independent Gaussian, the magnitude of the representations follows some scaled Chi distribution || ~ Xx. which we can reasonably approximate with E[| |] Elxm] R vn. Therefore, we propose to rescale the output of fø by 1/ Vn, such that modifying the dimensionality of the representations should not significantly affect their magnitude before mapping them H"" We call this general stabilization recipe specttaaly- hyperbolic mappings (S-RYM).
Results. As shown in Figure integrating S-RYM with our hyperbolic RL agents appears to resolve their optimization challenges and considerably improve the Euclidean baseline's 's performance (A). To validate that the performance benefits are due to the hyperbolic geometry of the latent space, we also evaluate a baseline using SN and rescaling in Euclidean space, which fails to attain consistent improvements. Furthermore. S-RYM maintains low gradient magnitudes (B), confirming its effec- tiveness to stabilize training. In App. E.I we also show that SN and rescaling are both crucial for S-RYM. Thus, in the next section, we evaluate hyperbolic deep RL on a large-scale, analyzing its efficacy and behavior across different benchmarks, RL algorithms, and training conditions.
EXTENSIONS AND EVALUATION
To test the generality of our hyperbolic deep RL framework, in addition to the on on-policy PPO we also integrate it with the off off-po Rainbow DQN algorithm (Hessel et al., 2018). Our implementations use the same parameters and models specified in prior traditional RL literature, without any addi- tional tuning. Furthermore, in addition to the full procgen benchmark (16 envs.) also evaluate on
7

---

Published as a conference paper at ICLR 2023
+266%s
2.0
+269%
+370%
Absolute nornaized perforancce diference
Relative normalzeed performance differecce
1.5
1.0
0.5
+137%
2228%
+24%
+322%
491746
+529
+33%
+31496
+30%
+264%
+39%
+288
+13
442% 14909
-128
22498
+22%
-11%
23%
0.0
dernon ate.
breakesug
jamestond
boxing
krail
battle 20.
gopter
ascault
pang
froatbite
lung fu, m.
obert
bank hwist
crazy cil
up n. down
allen
chapper. c.
ms pacmmn
asterix
senquest
privat, e.
amrittr
freeway
langaroo
hero
road
Figure 9: Absolute difference in normalized performance (Y-axis) and relative improvements (Above bars)
from integrating hyperbolic representations with S-RYM onto our Rainbow implementation.
the popular Atari 100K benchmark (Bellemare et al. 2013 Kaiser al.) 2020) (26 envs.), repeating for 5 random seeds. We provide all details about benchmarks and implementations in App. g
Generalization on Procgen. Given the documented representation efficiency of hyperbolic space, we evaluate our hyperbolic PPO implementation also reducing the dimensionality of the final repre- sentation to 32 (see App. E.2h, with relative compute and parameter efficiency benefits. We compare our regularized hyperbolic PPO with using data augmentations, a more traditional way of encoding inductive biases from inducing invariances. We consider random crop augmentations from their popularity and success in modern RL. As shown in Table [ our hyperbolic PPO implementation with S-RYM appears to yield conspicuous performance gains on most of the environments. At the same time, reducing the size of the representations provides even further benefits with significant improvements in 13/16 tasks. In contrast, applying data augmentations yields much lower and incon- sistent gains, even hurting on some tasks where hyperbolic RL provides considerable improvements (e.g. bossfight). We also find that test performance gains do not always correlate with gains on the specific 200 training levels, yielding a significantly reduced generalization gap for the hyperbolic agents. We perform the same experiment but apply our hyperbolic deep RI. framework to Rainbow DQN with similar results, also obtaining significant gains in 13/16 tasks, as reported in App. [.I]
We also evaluate the robustness of our PPO agents to encoding spurious features, only relevant for the training levels. In particular, we examine tasks where PPO tends to perform well and consider lowering the training levels from 200 to 100, 50, and 25. As shown in Figure g the performance of PPO visibly drops at each step halving the number of training levels, suggesting that the Euclidean representations overfit and lose their original efficacy. In contrast, hyperholic PPO appears much more robust, still surpassing the original PPO results with only 100 training levels in fruithot and 50 in starpilot. While also applying data augmentation attenuates the performance drops, its effects appear more limited and inconsistent, providing almost null improvements for starpilot.
Sample-efficiency on Atari 100K. We focus on the performance of our hyperbolic Rainbow DQN im- plementation, the severe data limitations of this benchmark make PPO and other on-policy algo- rithms impractical. We show the absolute and rela- tive per-environment performance changes from our hyperbolic RL framework in Figure g and provide aggregate statistics in Table 2 Also on this benchmark, the exact same hyperbolic deep RL frame- work provides consistent and significant benefits. In particular, we record improvements on 22/26 Atari environments over the Euclidean baseline, almost doubling the final human normalized score.
Table 2: Aggregate results on Atari 100K
Metric Metric\AAgoriiee AAlorithhm
Rainbow
Rainbow S-RYM
Human norn. mean
0.353
0.686 (+93%)
Human nomn. median
0.259
0.366 (+41% %)
Super human scores 2
Considerations and comparisons. Our results empirically validate that introducing hyperbolic representations to shape the prior of deep RL models is both remarkably general and effective. We record almost universal improvements on two fundamentally different RL algorithms, considering both generalizations to new levels from millions of frames (Procgen) and to new experiences from only 2hrs of total play time (Atari 100K). Furthermore, our hyperbolic RL agents outperform the scores reported in most other recent advances, coming very close to the current SotA algorithms which incorporate different expensive and domain comain--peccallzed auxiliary practices (see App. D.2. D.3). Our approach is also orthogonal to many of these advances and appears provide compatible and complementary benefits (see App. E.3). Taken together, we believe these factors show the great potential of our hyperbolic framework to become standard way of parameterizing deep RL. models.
8

---

Published as a conference paper at ICLR 2023
Agent traj aateettory
Initial state
Policy transitions
Random transitions
T- Value gyraplane
Random trajectory deviations
Figure 10: Visualization of 2-dimensional hyperbolic embedddngs in the bigfish environment as we progress through trajectory, encoding states from either policy transitions random transitions (details in App. D.4.
Representations interpretation. We train our hyperbolic PPO agent with only 2-dimensional rep- resentations, which still remarkably provide concrete generalization benefits over Euclidean PPO (App. D.4). (Ap..D.4). Then, we analyze how these representations evolve within trajectories, mapping them on the Poincaré disk and visualizing the corresponding states. We observe a recurring cyclical be- havior, where the magnitude of the representations monotonically increases within subsets of the trajectory as more obstacles/enemies appear. We show this in Fig. 10 and Fig. 12 comparing the representations of on-policy states sampled constant intervals with trajectory deviations from executing random behavior. We observe the representations form tree-like structures, with the mag- nitudes in the on-policy states growing in the direction of the Value function's 's gyroplane's 's normal. This intuitively reflects that as new elements appear the agent recognizes a larger opportunity for rewards, yet, requiring finer level of control as distances the policy gyroplanes will also grow exponentially, reducing entropy. Instead, following random deviations. magnitudes grow in direc- tions orthogonal to the Value gyroplane's normal. This still reflects the higher precision required for optimal decision-making. but also the higher uncertainty to obtain future rewards from worse states.
5 RELATED WORK
Generalization is a key open problem in RL. (Kirk et al., with RL objectives has been shown prone to overfitting from spurious features only relevant in the observed transitions (Song al. 2019 Bertran et al.] 2020). To address this, prior work consid- ered different data augmentation strategies (Laskin et al. 2020b Yarats et al. a0 Cobbe et al. 2019), and online adaption methods on top to alleviate engineering burdens (Zhang & Guo 2021 Raileanu et al.] 2020). Alternative approaches have been considering mroblem-specine properties of the environment (Zhang et al., 20200 Raileanu & Fergus (021), auxiliary losses (Laskin al., 2020a Schwarzer et al., (020). and frozen pre-trained layers Yaratt et al.) 2021b] Stooke et al,, 2021 Instead, we propose to encode a new inductive bias making use of the geometric properties of hyperbolic space, something orthogonal and likely compatible with most such prior methods.
2021). End-to-end training of deep models
While hyperbolic representations found recent popularity in machine learning, there have been notable extensions for deep RL (Peng et al., (021). Most relatedly, Tiwari & Prannoy (2018) pro- posed produce hyperbolic embeddings of the state space of tabular MDPs recover options (Sutton et al.] 1999). Yet, they did not use RL for learning, but fixed data and a supervised loss based on the co-occurrence of states, similarly to the original method by Nickel & Kiela 2017
DISCUSSION AND FUTURE WORK
In this work, we introduce hyperbolic geometry to deep RL. We analyze training agents using latent hyperbolic representations and propose regguaally-egullrieedd hyperbolic mappings, new stabiliza- tion strategy that overcomes the observed optimization instabilities. Hence, we apply our framework to obtain hyperbolic versions of established on-policy and off-policy RL algorithms, which we show substantially outperform their Euclidean counterparts in two popular benchmarks. We provide nu- merous results validating that hyperbolic representations provide deep models with more suitable prior for control, with considerable benefits for generalization and sample-efficiency We share our implementation to facilitate future RI advances considering hyperbolic space new, general tool.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
We proposed to provide deep RL models with a more suitable prior for learning policies, using hyperbolic geometry. In terms of carbon footprint, our implementation does not introduce additional compute costs for training, and even appears to perform best with more compact representation sizes. Consequently, given the nature of our contribution, its ethical implications are bound to the implications of advancing the RL field. In this regard, autonomous agents become more applicable, poor regulation and misuse may cause harm. Yet, we believe these concerns are currently out-weighted by the field's significant positive potential to advance human flourishing.
REPRODUCIBILITY STATEMENT
We provide detailed descriptions of our integration of hyperbolic space, experimental setups, and network architectures in Section 3 and also Appendix B We provide all details, including full list of pyper-parameters, in Appendix C We currently shared an anonymized version of our code to reproduce the main experiments in the supplementary material. We shared our open-source imple- mentation to facilitate future extensions.
REFERENCES
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 17X1:17001.0875. 2017.
Wasserstein gans. arXiv preprint
Roger G Barker and Herbert F Wright. Midwest and its children: The psychological ecology of an american town. 1955.
Gary Bécigneul and Oetavian-Fugen Ganea. Riemannian adaptive optimization methods. In Inter- national Conference on Learning Representations, 2019.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective reinforcement learning. In International Conference on Machine Learning, 449-458. PMLR, 2017.
Richard Bellman. markovian decision process. Journal of mathematics and mechanics, pp. 679 684, 1957.
Eugenio Beltrami. Teoria fondamentale degli spazii di curvatura costante. Annali di Matematica Pura ed Applicata (18))-25 1868.
Martin Bertran, Natalia Martinez, Mariano Phielipp. and Guillermo Sapiro. Instance-based gen- eralization reinforcement learning. Advances in Neural Information Processing Systems, 33: 11333-1 2020.
Johan Bjorck, Carla P Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learn- ing. In Advances in neural information processing systems, 2021.
Mario Bonk and Oded Schramm. Embeddings gromov hyperbolic spaces. In Selected Works of Oded Schramm, pp. 243-284. Springer, 2011.
Silvere Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 5809):2217-2299 2013.
Michele Borassi, Alessandro Chessa, and Guido Caldarelli. Hyperbolicity measures democracy in real-world networks. Physical Review E, 9233)2332 2015.
10

---

Published as a conference paper at ICLR 2023
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2018.
James W Cannon, William J Floyd, Richard Kenyon, Walter R Parry, et al. Hyperbolic geometry.
Flavors of geomelry, (1)5--133)2, 1997.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare.
Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http:
Edoardo Cetin, Philip J Ball, Stephen Roberts, and Oya Celiktutan. Stabilizing off off-policy deep reinforcement learning from pixels. In International Conference on Machine Learning, pp. 2784 2810. PMLR, 2022.
Benjamin P Chamberlain, James Clough, and Marc P Deisenroth. Neural embeddings of graphs in hyperbolic space. arXiv preprint arXiv: 10359, 2017.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. In International Conference on Machine Learning, pp. 1282-1288 PMLR, 2019.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. 2048- 2056. PMLR, 2020.
Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In Interna- tional Conference on Machine Learning, pp. 2020-2022 PMLR, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Gabriel Dulac-Amold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. Machine Learning, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan. Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference machine learning, pp. 1407-1416. PMLR, 2018.
Yannis Flet-Berliac The promise of hierarchical reinforcement learning. The Gradient, 9, 2019.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis. Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Learn- ing Representations, 2018. /tts httpeeewwwwwweeeeeeeeeeeeeeeeeee wetfooomiddee (hbrruunmnddryyHCCcaa
Hervé Fournier, Anas Ismail, and Antoine Vigneron. Computing the gromov hyperbolicity of
discrete metric space. Information Processing Letters, 115(6-8):566-79,, 2015.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018
Florin Gogianu. Tudor Berariu, Mihaela C Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference Machine Learning, pp. 3734-3744, PMLR, 2021.
Gene H Golub and Henk A Van der Vorst. Eigenvalue computation in the 20th century. Journal of Computational and Applied Mathematics, 123(1-2):35-63 2000.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 267222680, 2014.

---

Published as a conference paper at ICLR 2023
Mikhael Gromov. Hyperbolic groups. In Essays in group theory, pp. 75 15-263. Springer, 1987.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, pp- 5767-5777, 2017a.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, 30, 2017b.
Yunhui Guo, Xudong Wang. Yubei Chen, and Stella X Yu, Clipped hyperbolic classifiers are super- hyperbolic classifiers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11-20, 2022.
Anupam Gupta. Embedding tree metrics into low dimensional euclidean spaces. In Proceedings of the thirty-first annual ACM symposium Theory of computing, pp. 694-700, 1999.
Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23:2613-
2621, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint 1Tt.:1700.04333 2017.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp. 4940-4950. PMLR, 2021.
Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for Atari. In International Conference Learning Representations, 2020.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for rision-based robotic manipulation. arXiv preprint arXiv:1806. 1006, 2018.
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6418 6428, 2020.
Kacper Piotr Kielak. Do recent advancements in model-based deep reinforcement learning really improve data efficiency? 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. survey of generalisation in deep reinforcement learning. arXiv preprint arXiv:21 2021.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
12

---

Published as a conference paper at ICLR 2023
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. pp. 5639-5650, 2020a.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in neural information processing systems, 33: 19884-19895, 2020b.
Guy Lebanon and John Lafferty. Hyperplane margin classifiers the multinomial manifold. In Proceedings of the twenty-first international conference on Machine learning, pp. 66, 2004.
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Mobbrrt Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
Alex Lee, Coline Manon Devin, Yuxiang Zhou, Thomas Lampe, Konstantinos Bousmalis, Jost To- bias Springenberg, Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, et al. Beyond pick-and-place: Tackling robotic stacking diverse shapes. In 5th Annual Confer- on Robot Learning, 2021.
Zinan Lin, Vyas Sekar, and Giulia Fanti, Why spectral normalization stabilizes gans: Analysis and improvements. Advances in neural information processing systems, 34:9625-9638, 2021.
Federico López and Michael Strube. A fully hyperbolic neural model for hierarchical multi-class classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp- 460-475 2020.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In In-
ternational Conference on Learning Representations, 2017. URL https::/openrevieex
net/ fort/Eorum?a wtt.rrmmmmmiissgsmeemc 9Scxx
Marlos C Machado, Marc G Bellemare, Erik Talvitic, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Shie Mannor, Dunean Simester, Peng Sun, and John N Tsitsiklis, Bias and variance approximation in value function estimates. Management Science, 53(2):308 322, 2007.
Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh. Contin- uous hierarchical representations with poincaré variational auto-encoders. In Advances in neural information processing systems, volume 32, 2019.
Jiñ Matoušek. Bi-lipschitz embeddings into diw-eimensional euclidean spaces. Commentationes Mathematicae Universitatis Carolinae, 31(3):589-66 1990.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05953 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: /3/2.562,, 2013.
Sharada Mohanty, Jyotish Poonganam, Adrien Gaidon, Andrey Kolobov, Blake Wulfe, Dipam Chakraborty, Grazvydas Semetulskis, Joao Schapke, Jonas Kubilius, Jurgis Pasukonis, et al. Mea- suring sample efficiency and generalization in reinforcement learning benchmarks: Neurips 2020 procgen benchmark. arXiv preprint arXiv:2103.153: 2021.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped nor- mal distribution on hyperbolic space for gradient- based learning. International Conference on Muchine Learning, pp. 4693-4702. PMLR, 2019.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hicrarchical representa- tions. Advances in neural information processing systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2023
OpenAl, Ilge Akkaya, Marcin Andrychowiez, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand. CoRR, abs/1910.071 13, 2019.
Mark Pendrith, Malcolm RK Ryan, et al. Estimator variance reinforcement learning: Theo- retical problems and practical solutions. University of New South Wales, School of Computer Science and Engineering, 1997.
Wei Peng, Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyperbolic deep neural networks: A survey. IEEE Transactions Pattern Analysis and Machine Intelli- gence, 2021.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint
2015.
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. In International Conference on Machine Learning, pp. 8787-8798. PMLR, 2021.
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Auto- matic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006. 12862, 2020.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Rik Sarkar. Low distortion delaunay embedding of trees in hyperholic plane. In International Symposium on Graph Drawing, pp. 355-366. Springer, 2011.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations, 2016.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint
2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, Nes1707.006447 2017. URL http::/ wwwexiiivorgga... abs/
1707.0634
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations. 2020.
Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic neural networks++. arXiv preprint arXiv:2006.082 10, 2020.
Ondrej Skopek, Octavian-Eugen Ganea, and Gary Bécigneul. Mixed-curvature variational autoen- coders. In International Conference on Learning Representations, 2019.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfit- ting in reinforcement learning. In International Conference on Learning Representations, 2019.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference Machine Learning, pp. 9870-9879. PMLR, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
Alexandru Tifrea, Gary Becigneul, and Octavian-Eun -ugann Ganea. Poincare glove: Hyperbolic word embeddings. In International Conference Learning Representations, 2018.
14

---

Published as a conference paper at ICLR 2023
Saket Tiwari and M Prannoy. Hyperbolic embeddings for learning options in hierarchical reinforce- ment learning. arXiv preprint arxiv:1112.01487, 2018.
Abraham Albert Ungar. A gyrovector space approach to hyperbolic geometry. Synthesis Lectures on Mathematics and Statistics, 1(1):1-194, 2008.
Hado Van Hasselt, Matteo Hessel, and John Aslanides. When use parametric models in rein- forcement learning? Advances in Neural Information Processing Systems, 32, 2019.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David Н Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-333 2019.
Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. Advances in Neural Information Processing Systems, 33: 7968-7978, 2020.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003. PMLR, 2016.
Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre Bayen. Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. International Conference on Learning Representations, 2018.
Omry Yadan. Hydra a framework for elegantly configuring complex applications. Github, 2019.
URL https: /ttp:::///tttttttt fawa//ooooeeeeeerr/////y.tr
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference Learning Representa- tions, 2021a. htttt:ttt::///ao https:::/ppeneeieew net/ www.foodooooooooooooooooo 6sTvGaf
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improv- ing sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10674-10681, 2021b.
Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. International Conference on Machine Learning. pp. 11214-11224 PMLR, 2020.
Hanping Zhang and Yuhong Guo. Generalization of reinforcement learning with policy-aware ad- versarial data augmentation. arXiv preprint arXiv:2106.1558% 2021.
Yuansheng Zhou, Brian H Smith, and Tatyana O Sharpee. Hyperbolic geometry of the olfactory space. Science advances, 4(8), 2018
15

---

Published as a conference paper at ICLR 2023
APPENDIX
A EXTENDED BACKGROUND
A.1 I RL ALGORITHMS DESCRIPTIONS
Continuing from section 2.1 we provide an overview of standard RL definitions and the deep RL algorithms we use in this work.
Two important functions in RL are the value function and the action-value function (also called the Q function). These quantify, for policy æ. the expected sum of discounted future rewards given any initial fixed state or state-action pair, respectively:
(F(5s) = Eo 1.
Σ P==0
r r (e4+,1,0444)
(" = r(1,, at) + - [V (8++1)).
Relatedly, the advantage fuunction (s,,,, 0z) (st, - (sx)s) quantifies the expected im- provement from executing any given action a, from 8; rather than following the policy. These func- tions summarize the future evolution of an MDP and are often parameterized and learned auxiliary to or even in in-place of the policy model.
On-policy methods. Modern on-policy RL algorithms collect a new set of trajectories at each it- eration with the current policy, discarding old data. They use these trajectories to learn the current policy's 's value function and recover a corresponding advantage function from the observed Monte- Carlo returns, using techniques such the popular Generalized Advantage Estimator (GAE) (Schul-) man et al., 2015). The estimated advantages A AGAE are then used to compute the policy gradient and update the policy, maximizing the probability of performing the best-observed actions (Sutton) & Barto 2018). Since the values of AGAR are based a limited set of trajectories. on-policy methods generally suffer from high-variance targets and gradients (Pendrith et al.] 1997] Mannor al., 2007; Wu al. 2018). Proximal Policy Optimization (PPO) (Schulman et al. (017) is one
of the most established on-policy algorithms that attenuates these issues by taking conservative up- dates, restricting the policy update from making larger than changes to the probability of executing any individual action. PPO considers the ratio between the updated and old policy probabilities Telay(ss)
R2 (aels)) Fordla, (s.)
to optimize pessimistic clipped objective of the form:
GAE AGAE (st, at), clip (alls:))) 1 c, 1 C)AGEE (st, at)
(8)
As mentioned in the main text, PPO also includes small entropy bonus incentivize exploration and improve data diversity. This term can be differentiated and optimized without any estimator since we have full access to the policy model and its output logits, independently of the collected data.
Off-policy methods. In contrast, ool-py algorithms generally follow a significantly different op- timization approach. They store many different trajectories collected with mixture of old policies in large replay buffer, B. They use this data to directly learn the Q function for the optimal greedy policy with squared loss based the Bellman backup (Bellman 1957):
E EE Q(sr, at) - -(:: max Q(sL41
(Bellman 1957). Agent behavior is then implicitly defined by the epsilon-greedi policy based on the actions with the highest estimated Q values. We refer to the deep Q-networks paper (Mnih et al.) 2013) for a detailed description of the seminal DQN algorithm. Rainbow DQN (Hessel et al. 2018 modern popular extension that introduces several auxiliary practices from proposed orthogonal improvements, which they show provide compatible benefits. In particular, they use n-step returns (Sutton & Barto 2018). prioritized experience replay (Schaul et al.) 2016), double Q-learning (Has- selt 2010). distributional RL (Bellemare et al. (017). 7). noisy layers (Fortunato et al., 2018), and dueling network architecture (Wang et 2016).
16

---

Published as a conference paper at ICLR 2023
A.2 H-HPPEBBOLICITY
5-hyperbolicity was introduced by /rrmmo (1987) as a criterion to quantify how hyperbolic a metric space (X,d) d) is. We can express d-hyperbolicity in terms of the Gromov product, defined for x; y X at some base point r E X as measuring the defect from the triangle inequality:
1 2 [d((z,)) r) + d(r, v) d(z,y)) v)).
(xly),
(10)
Then, X is 5-hyperbolic if for all base points E X and for any three points x, U, E X the Gromov product between x and y is lower than the minimum Gromov product of the other pairs by at most some slack variable d:
(xly), 2 min((xly)). (x(l))))
(II)
In our case (a complete finite-dimensional path-connected Riemannian manifold, which is geodesic metric space), d-hyperbolicity means that for every point on one of the sides of a geodesic triangle Azyz, there exists some other point on one of the other sides whose distance is at most 6, or
in other words, geodesic triangles are 5-slim. In trees, the three sides of a triangle must all intersect at some midpoint (Figure 33. Thus, every point belongs to least two of its sides yielding O. Thus the d-hyperbolicity can be interpreted as measuring the deviation of a given metric from an exact tree metric.
A.3 GENERATIVE ADVERSARIAL NETWORKS AND SPECTRAL NORMALIZATION
GANs. In GAN training, the goal is to obtain generator network to output samples resembling some 'true' target distribution. To achieve this, GGoodfellw et al.] (2014) proposed to alternate train- ing of the generator with training an additional discriminator network, tasked to distinguish between the generated and true samples. The generator's s objective is then to maximize the probability of its own samples according to the current discriminator, backpropagating directly through the discrim- inator's ' network. Since both the generated data and discriminator's 's network parameters constantly change from this alternating optimization, the loss landscape of GANs also highly non non-stationary, resembling, to some degree, the RL. setting. As analyzed by several works, the adversarial nature of the optimization makes very brittle to exploding and vanishing gradients instabilities (Arjovsky) & Bottou 2017: Brock et al. 2018) which often result in common failure modes from severe diver- gence or stalled learning (Lin et al. 2021). Consequently, numerous practices in the GAN litcrature have been proposed to stabilize training (Radford et al. 2015 Arjovsky et al. [0017] Gulrajani al. 20176). Inspired by recent work, in this work, we focus specifically on spectral normalization (Miyaoo et al., 2018), one such practice whose recent success made in ubiquitous in modern GAN implementations.
Spectral normalization. In the adversarial interplay characterizing GAN training, instabilities com- monly derive from the gradients of the discriminator network, I' (Salimans et al. 2016). Hence, Miyato al. (2018) proposed regularize the spectral norm of discriminator's ' layers, Iy E fD. i.c., the largest singular values of the weight matrices ||W I| - o (WSS)), to be approximately one. Consequently, spectral normalization effectively bounds the Lipschitz constant of the whole discrim- inator network since, I S 11 S
Il.. = 1. In practice, the proposed implementation approximates the largest singular value of some original unconstrained weight ma- trices by running power iteration (Golub & Van der Vorst, (000).. Thus, it recovers the spectrally- normalized weights with a simple re-parameterization dividing the unconstrained weights by their W, relative singular values WSN a(W,). As mentioned in the main text, recent work (Lin et al.
202T) showed that one of the main reasons for the surprising effectiveness of spectral normalization in GAN training comes from effectively regulating both the magnitude of the activations and their respective gradients, very similarly to LeCun initialization (LeCun et al. 2012). Furthermore, when applied to the discriminator, spectral normalization's 's effects appear persist throughout training, while initialization strategies tend to only affect the initial iterations. In fact, in Figure 2 of their paper, they also show that ablating spectral normalization empirically results in exploding gradients and degraded performance, closely resembling our same observed instabilities in Figure (B).
S-RYM entails applying spectral normalization to all the layers in the Euclidean encoder sub- network, as already after backpropagation through the final latent hyperbolic representation we ob- serve exploding high-variance gradients. We leave the final linear transformation in hyperbolic space
17

---

Published as a conference paper at ICLR 2023
unregularized do not want unnecessarily restrict the expressivity of the model. Furthermore, there is also no direct way of performing power iteration with our final layer parameterization (see Section 2.2). We note that by applying spectral normalization to all layers, models of the value and policy to be 1-Liisshitzz, a property that is likely not reflective of the true op- timal policy and value functions. We validate this hypothesis in Appendix[H.4] continuity with gradient penalties (Gulrajani et al. (2017) on top of our regularized Hyperbolic PPO implementation. For Euclidean PPO, preliminary experiments confirmed that enforcing the model to be 1-Lipschitz by applying either spectral normalization to all layers or gradient penalties leads to worse results than applying S-RYM. This intuition is also consistent with other recent works that studied the application of spectral normalization for reinforcement learning (Bjorck et al., [202]] Gogianu al. 2.2Tb. Yet, these works also observed performance benefits when applying spectral normalization exclusively to particular layers of the model. These empirical insights could inform future improvements for S-RYM to retain the stability benefits of spectral normalization with even less restrictive regularization.
would enforce
enforcing Lipschitz
B STABILIZATION OF HYPERBOLIC REPRESENTATIONS
One of the main challenges of incorporating hyperbolic geometry with neural networks comes from end-to-end optimization of latent representations and parameters located in hyperbolic space. For instance, numerical issues and vanishing gradients occur as representations get too close to either the origin or the boundary of the Poincaré ball (Ganea et al. 2018). Moreover, training dynamics can tend to push representations towards the boundary, slowing down learning and make optimization problems of earlier layers ineffective (Guo et al., 2022). A number of methods have been used to help stabilize learning of hyperbolic representations including constraining the representations to have a low magnitude early in training. applying clipping and perturbations (Ganca et al., 2018 Khrulkov et al., 2020), actively masking invalid gradients (Mathieu et al.] 2019). and designing initial "burn-in" periods of training with lower learning rates (Nickel & Kiela| [017] Bécigneul & Ganea 2019). More recently Guo et al. (2022) also showed that very significant magnitude clipping of the latent representations can effectively attenuate these numerical and learning instabilities when training hyperbolic classifiers for popular image classification benchmarks.
B. MAGNITUDE CLIPPING
Guo et al. (2022) recently proposed to apply significant clipping of the magnitude of the latent representations when using hyperbolic representations within deep neural networks. in our work, they also consider hybrid architecture where they apply an exponential map before the final layer to obtain latent representations in hyperbolic space. They apply the proposed clipping to constrain the input vector of the exponential map to not exceed unit norm, producing hyperbolic representations via:
TH expo min
x FE
(12)
|Eg
The main motivation for this practice is to constrain representation magnitudes, which the authors linked to a vanishing gradient phenomenon when training on standard image classification datasets (Krizhevsky al. 2009 Deng et al. 2009). However, side effect of this formulation is that the learning signal from the representations exceeding a magnitude of will solely convey information about the representation' 's direction and not its magnitude. Since the authors do not share their im- plementation, we tested applying their technique as described in the paper. We found some benefits in additionally initializing the parameters of the last two linear layers (in Euclidean and hyperbolic space) to 100x smaller values to facilitate learning initial angular layouts.
B.2 IMAGE CLASSIFICATION EXPERIMENTS
To empirically validate and analyze our clipping implementation we consider evaluating deep hy- perbolic representations datasets from Guo et al. (2022). In particular, we utilize a standard ResNet18 architecture (He et al. 2016) and test our network CIFAR10 and CIFAR100 (Krizhevsky et al.) 2009). We optimize the Euclidean parameters of the classifier using stochastic gradient descent with momentum and the hy- perbolic parameters using its Riemmanian analogue (Bonnabel) 2013). We train for 100 epochs with
image classification tasks, following the same training practices and
18

---

Published as a conference paper at ICLR 2023
True label:
Truck
True label:
Bird Bird
True label:
Ship Ship 0.5024
True label:
Dog
Predicted: Truck
Predicted:
Predicted:
Predictad:
Dog
Magnitude:
0.5468
Magnitude:
0.4948
Magnitude:
Magnitude:
0.4986
True label:
Dog
True label:
Car Airplane 0.3501
True label:
Frog
True label:
Deer
Predicted:
Dog 0.3547
Predicted:
Predicted:
Frog 0.3670
Predicted:
Bird
Magnitude:
Magnitude:
Magnitude:
Magnitude:
0.3523
Figure 11: Visualization of test images from CIFARIO, with the corresponding final latent represen- tations magnitudes from our hyperbolic R classifier implemented with S-RYM. We sample datapoints with the 5% highest magnitudes (Top) and the 5% lowest magnitudes (Bottom).
an initial learning rate of 0.1 and a cosine schedule (Loshchilov & Hutter 2017). using a standard batch size of 128. We repeat each experiment 3 times, recording the final top-1 classification accu- racy together with the latent representations in Euclidean space right before applying the exponential map at the final layer.
Table 3: Performance results on standard image classification benchmarks
CIFARIO with ResNet18
Metrie M Architecture
Fuclidean
Hyperbolic Clipping Hyperbolic + S-RYM
Top-1 accuracy
94.92 + 0.19
94.81 + 0.17
95.12 1. 0.09
L2 representations magnitudes
5.846
1.00
0.481
Magnitudes standard deviation
0,747
0.00
0.039
CIFAR100 with ResNet18
Metric Architecrrr
Euclidean ResNet18 Hyperbolic Clipping Hyperbolic S-RYM
Top-1 accuracy
76.86 + 0.23
76.75
0.23
77.49 0.35
L.2 representations magnitudes
11.30
1.00
0.852
Magnitudes standard deviation
1.571
0.00
0.076
In Table 3 we report the different classifiers' performance together with the mean and standard de- viation of the representations' magnitudes from the images in the test set. The performance of the clipped hyperbolic classifier is very close to the performance of the Euclidean classifier, matching Guo et al. (2022)'s results and validating our implementation. However, the learned representations' magnitudes soon overshoot the clipping threshold and get mapped to constant-magninee throughout training. Therefore, the model will effectively stop optimizing for the representations' magnitudes and only focus on their unit direction. As volume and distances on the Poincaré ball grow expoentially with radius, the magnitude component of the hyperbolic representations is pre-
vectors
19

---

Published as a conference paper at ICLR 2023
cisely what facilitates encoding hierarchical information, providing its intuitive connection with tree structures. Hence, the resulting clipped 'hyperbolic' space spanned by the clipped latent represen- tations will lose its defining degree of freedom and approximately resemble Euclidean space with a rescaled metric, potentially explaining its performance similarity with stan- dard Euclidean classifiers. Even though the focus of our work is not image classification, find S-RYM's performance remarkably recovers and even marginally exceeds the performance of both the Euclidean and the clipped hyperbolic classifiers on these saturated benchmarks. Furthermore, its representations do not explode and maintain magnitude diversity, enabling to more efficiently capture the relative hierarchical nature of maage-clasiification benchmarks (Khrulkov et al., 2020). Overall, these results suggest that clipping simply treats the symptoms of the instabilities caused by end-to-end large scale training by essentially resorting back to Euclidean representations for image classification.
1-dimensional
Analyzing the magnitude component of the latent representations for our hyperbolic classifier with S-RYM, we find it correlates with classification performance. For instance, on CIFARIO 10 the test performance on the images with representations's with the top 5% magnitudes is 97.17%, while for the bottom 5% is 79.64%. Furthermore, display some samples from these two distinct groups in Figure II From these results and visualizations, it appears that the hyperbolic hierarchical structure serves encode the degree of uncertainty disambiguate between multiple image labels due to the blurriness and varying difficulty of the CIFARIO datapoints. Hence, we believe the observed accuracy improvements of our hyperbolic classifier might be specifically due to more efficiently capturing this specific hierarchical property of the considered datasets.
C IMPLEMENTATION AND EXPERIMENT DETAILS
We provide details of the experimental settings and implementations with the corresponding hyper- parameters for both our Proximal Policy Optimization (PPO) (Schulman et al., 2017) and Rainbow DQN experiments (Hessel et al. 2018 We consider these two main baselines since they two of the most studied algorithms in the recent RL literature onto which many other recent advances also build upon (e.g.. (Cobbe et al. 2021 Laskin et al., 2020b, Mohanty et al. 2021 Raileanu jet al. 2020; Raileanu & Fergus 2021 Yarats
Yarats et al. [202 la Van Hasselt et al.] 2019: Laskin et al. 2020a Schwarzer al. (2020). Furthermore. PPO and Rainbow DQN are based on the main families of model-free RL algorithms, with very distinct properties as described in Appendix A. Hence, unlike most prior advances, we do not constrain our analysis
single class of methods, empirically showing the generality of hyperbolic deep RL. Our implementations closely follow the reported details from recent research, and were not tuned to facilitate our integration of hyperbolic representations. The main reason for this choice is that we wanted to avoid introducing additional confounding factors from our evaluation of hyperbolic representations, as ad-hoe tuning frequently plays significant role in RL performance (Islam et al. 2017).
We implemented the proposed dimensionality-base rescaling, followed by the exponential map and the linear layer in hyperbolic space as a single Pytorch module (the PoincarePlaneDistance class in the shared code). Our implementation allows us to easily integrate our framework with existing neural network models by simply swapping the final layer with new module. We would like to acknowledge the Geoopt optimization library (Kochurov et al. (020). which we used to efficiently train and store the network parameters located in hyperbolic space. We also make use of the Hydra and quickly specifying the different
library (Yadan) 2019) to facilitate storing algorithm variations for the experiments in our work.
C.I BENCHMARKS
Procgen. The Procgen generalization benchmark (Cobbe et al.) 2020) consists of 16 game environ- ments with procedurally-generaae
random levels. The state spaces of these environments consist of the RGB values from the 64x64 rescaled visual renderings. Following common practice and the recommended settings, we consider training agents using exclusively the first 200 levels of each environment and evaluate on the full distribution of levels to assess agent performance and gen- cralization. Furthermore, we train for 25M total environment steps and record final training/test performance collected across the last 100K steps averaged over 100 evaluation rollouts.
20

---

Published as a conference paper at ICLR 2023
Atari 100K. The Atari 100K benchmark (Kissrr al. 2020 is based on the seminal problems from the Atari Learning Environment (Bellemare et al. 2013). particular, this benchmark consists of 26 different environments and only 100K total environment steps for learning each, corresponding roughly to 2hrs of play time. The environments are modified with the specifications from (Machado) al. 2018), making the state spaces of these environments 84x84 rescaled visual renderings and introducing randomness through sticky actions. We note that this is significantly different setting than Procgen, testing the bounds for the sample efficiency of RL agents.
C.2 PPO IMPLEMENTATION
Our PPO implementation follows the original Procgen paper (Cobbe et al., 2020), which entails residual convolutional network (Espeholt et al. 2018) and produces a final 256-dimensional latent representation with shared backbone for both the policy and value function. Many prior im- provements over PPO for on-policy learning have been characterized by either introducing auxiliary domain-specific practices, increasing the total number of parameters, performing additional op- timization phases leading to significant computational overheads (Cobbe et al. 2021] Raileanu & Fergus 2021 Mohanty al. (002T) Instead,
approach strives for orthogonal direction by simply utilizing hyperbolic geometry to facilitate encoding hierarchically-strue
features into
the final latent representations. Thus, it can be interpreted new way to modify the inductive bias of deep learning models for reinforcement learning.
Table PPO hyper-parameters used for the Procgen generalization benchmark
PPO hyperparameters
Parallel environments
64
Stacked input frames
Steps per rollout
16384
Training epochs per rollout
Batch size
2048
Normalize rewards
True
Discount
0.999
GAE X (Schulman et al. 2015
0.95
PPO clipping
0.2
Entropy coefficient
0.01
Value coefficient
0.5
Shared network
True
Impala stack filter sizes
16, 32, 32
Default latent representation size
256
Optimizer
Adam (Kingma Ba 2015
Optimizer learning rate
5x10
Optimizer stabilization constant (E)
1x10
Maximum gradient norm.
0.5
In Table we provide further details of our PPO hyper- ppper-parameters, as also described by the origi- nal Procgen paper (Cobbe et al.) 2020). When using hyperbolic latent representations, we optimize the hyperbolic weights of the final Gyroplane linear layer with the Riemannian Adam optimizer (Bécigneul & Ganea 2019), keeping the same learning rate and other default parameters. As per common practice in on-policy methods, we initialize the parameters of the final layer with 100x times lower magnitude. We implemented the naive hyperbolic reinforcement learning implementa- tions introduced in Subsection by2 initializing also the weights of the preceding layer with 100x lower magnitudes to facilitate learning appropriate angular layouts in the early training iterations. We found our S-RYM stabilization procedure also enable to safely remove this practice with no effects performance.
21

---

Published as a conference paper at ICLR 2023
C.3 RAINBOW IMPLEMENTATIONS
Our implementation of Rainbow DQN uses the same residual network architecture as our PPO implementation (Espeholt et al. 2018) but employs a final latent dimensionality of 512, as again specified by Cobbe al. (202)) Since Cobbe et al. (2020) do not open-source their Rainbow im- plementation and do not provide many of the relative details, we strive for simple implementation removing unnecessary complexity and boosting overall efficiency. Following Castro et al. (018)) only consider Rainbow DQN's three most significant advances over vanilla DQN (Mnih et al. 2013): distributional critics (Bellemare et al. 2017). prioritized experience replay (Schaul et al, 2016 and n-step returns (Sutton & Barto 2018). While the methodology underlying offlcy algorithms is fundamentally different from their on-policy counterparts, we apply the same exact recipe of integrating hyperbolic representations in the final layer, and compare against the same variations and baselines.
Table 5: Rainbow DQN hhper-paramenn parameters used for the Proegen generalization benchmark
Rainbow DQN Procgen hyperparameters
Parallel environments
64
Stacked input frames
1
Replay buffer size
1.28M
Batch size
512
Minimum data before training
32K steps
Update network every
256 env. steps
Update target network every e-greedy exploration schedule Discount
12800 env. steps
1->0.01 in 512K steps
0.99
N-step
3
Use dueling [Wang et al. 2016)
False
Use noisy layers (Fortunato et al. 2018)
False
Use prioritized replay (Schaul et al. 2016
True
Use distributional value (Bellemare et al. Distributional bins
2017
True
51
Maximum distributional value
10
Minimum distributional value Impala stack filter sizes
-10
16, 32, 32
Default latent representation size Optimizer
512
Adam (Kingma & Ba 2015
Optimizer learning rate
5x10-4
Optimizer stabilization constant (E)
1x10-5
Maximum gradient norm.
0.5
In Table 5 we provide details of our Rainbow DQN hyper-parameters. We note that sampling of off- off--policy transitions with n-step returns requires retrieving the future n rewards and observations. To perform this efficiently while gathering multiple transitions from the parallelized environment, we implemented parellalized version of a segment tree. In particular, this extends the original im- plementation proposed by /Schaul et al. (0016), through updating a set of segment trees implemented as unique data-structure with a single parallelized operation, allowing for computational efficiency without requiring any storage redundancy. We refer to the shared code for further details. As with hyperbolic PPO extensions, we also optimize the final layer's 's hyperbolic weights with Rieman- nian Adam, keeping the same parameters as for the Adam optimizer used the other Euclidean layers.
22

---

Published as a conference paper at ICLR 2023
Table 6: Rainbow DQN hyper-parameters changes for the Atari 100K benchmark
Rainbow DQN Atari 100K training hyper-parameters
Stacked input frames
4
Batch size
32
Minimum data before training
1600 steps
Network updates per step
2
Update target network every
1 env. steps
e-greedy exploration schedule
1 1~0.01 in 20K steps
The characteristics of the Atari 100K benchmark are severely different from Procgen, given by the lack of parallelized environments and the x50 reduction total training data. Hence, we make minimal set of changes to the training loop hyper-parameters of our Rainbow DQN implementation to ensure effective learning, as detailed in Table 6 These are based on standard practices employed by oof-pc algorithm evaluating the Atari T00K benchmark (Van Hasselt et al. 2019 Laskin et al. 2020a Yarats et al. (2221a) and were not tuned for our specific implementation.
C.4
EXPERIMENTAL RECORDINGS
By default, repeat each experiment with five random seeds. To visualize the evolution of relevant values using performance curves, we collect measurements from each algorithm in five evaluation rollouts every 64 1024 frames (64 is the number of parallel environments used in Procgen). Hence, the reported values correspond to the mean results over all the collected seeds and runs, while the shaded regions correspond the standard deviation between the mean results from different random seeds. For the tabular data illustrating final performance, we perform ten additional evaluation rollouts at the end of the specified training allowance. The reported uncertainty again represents the standard deviation between the mean results from different seeds.
C.5 CURRENT LIMITATIONS
We identify three main current limitations the proposed implementation. First, stabilizing hy- perbolic representations with S-RYM inherently constrains the expressivity of the Euclidean sub- network encoder model (fr) to be 1-Lipschitz. This loss expressivity might hinder the network's ability to learn complex representations, preventing our hyperbolic framework to achieve its full potential. Second, the training and evaluation time of our hyperbolic agents are consistently higher than their Euclidean counterparts. Our hyperbolic PPO implementation takes average 4.27 sec- onds to collect rollouts and train for three epochs, and takes 0.961 seconds to collect a full episode of experience the end of training. In contrast, our Euclidean baseline takes 3.69 seconds for training (16% speedup) and 0.854 seconds for evaluation (13% speedup). We find this slowdown mainly due the power iteration procedure performed apply spectral normalization in S-RYM and, to a lesser extent, an overhead when computing and backpropagating through hyperbolic operations. Third, our algorithm utilizes model of hyperbolic space with fixed negative curvature to build rep- resentations of the whole state space. However, as different RL problems might have considerably different structures, we believe that any fixed curvature might not always yield the most appropriate inductive bias. To this end, recent work showed potential benefits in using mixed curvature latent spaces and even learning the curvature parameter for unsupervised tasks (Skopek et al. (019).. We hope these limitations will be addressed in future work, further studying how differential geometry can be used empower RL.
D EXTENDED RESULTS AND COMPARISONS
In this Section, provide detailed per-environment Rainbow DQN Procgen results that were omit- ted from the main text due to space constraints. For both Rainbow DQN and PPO. We also compare the performance improvements from the integration of our deep hyperbolic representations with the reported improvements from recent state-of-the-aa (SotA) algorithms, employing one or several or- thogonal domain-specifie practices. In Appendix
provide examples empirically validating
23

---

Published as a conference paper at ICLR 2023
Table 7: Detailed performance comparison for the Rainbow DQN algorithm on the full Procgen benchmark. We train for a total of 25M steps on 200 training levels and test on the full distribution of levels. We report the mean returns, the standard deviation, and relative improvements from the original Rainbow DQN baseline over 5 random seeds.
Task's T (Algrrihhm
Rainbow DQN
Rainbow DQN + data aug.
Rainbow DQN S-RYM
Rainbow DQN + S-RYM, 32 dim.
Levels distribation
tradnfiesr
tnsin/eest
mainchest
tralioftet
highoh
23.1744
19.61
(-15%) 17.39 (+125)
27.61+0 (+19%) 23,0=2 (+49%) 30.8522 (+335) 22.3762 (+45%)
bongfight
7.17+1
7.29
6.22+1
(-13%) 6.97+1 (-4%)
9.4++1 (+31%) 7,75+1 (46%)
8.21#1 (+15%) 8.71=1 (+20%)
cavellyer
7,00+1 3.92a1
7,5910 (+8%) 5.3641 (4379)
6,3911 (-9%) 3.13+1 (-21%)
6,4521 5,46:1 (+39%)
claser
3,09+1 2.31=0
2,8900 (-6%) 2.61
4,0341 (+30%%) 3.68&1 (+58%)
3,78:0 (+23%) 3:29=0 (+43%)
climber
3.6821 L.73gl
2.57al (-30%) 2.3641 (+36%)
3.9120 2.39 (+38%)
4,8022 (+31%) 3,00z0 (+73%)
5,56+1 4.33el
3.22+1
(-42%) 3.00=1 (-31%)
5,20 (-6%) 5.07&1 (+179)
6.00gl (+8%) 6.33=1 (+46%)
dodgeball
7.42+1 4.67a1
8.91a1
(+20%) 6,9621 (+49
6.07A1 (-18%) 3.60 (-23%)
6.89a1
(-14%)
fruithor
21,51+3 16.94+2
22.29+2 (145) 20.5343 (+21%)
20.31=1 (-6%) 20.30=1 (+20%)
22.81=l (16%) 2187=2
heist
0.67+0 0.11=D
1.67=0
(+150%) 0.67#0 (+500%)
1,27+0 (+90%) 0,40+0 (+260%)
0.93e1 (+40%) 0:47:0 (-320%)
jumper
5.33+1 3.11=0
4.22+0 (-21%) 2.78+1 (-11%)
4,78+1 (-10%) 2,44#1 (-21%)
5.5tel (145) 3.47=1 (+11%)
leaper
1,78+1 2,56
6.11=1
(-2449)) 5.11=I (+100%)
2,00+1
(+13%) 1,00 (-61%)
0,80=0 (-55%) 0.53z0 (-79%)
miner
2.2241 2.33
L89aD (-15%)
(-43%)
2.40 (+8%) 1.40.0 (-40%)
2.73al (+23%) 2.00g0 (-14%)
mage
2.01#0 0.6720
2,07 (+3%) 1.58t1 (+137%)
1,91 (-5%) 0.95 (+40%)
1.9 (-29) 0:92 (-38%)
ninja
3,3340 2.33gl
3.44al (+3%) 2.56al (+10%)
3.35al (+0%) 2.1 (-108)
3,73el (=12%) 3.33&1 (+43%)
plunder
8.69+0 6.28=1
6.06+1 (.30%)
(-16%)
7.33+1 (-16%)
(-5%)
7.11=1 (-185) 5.71=1 (-99)
starpioot
47,82266 42,42=1
51.79
(+8%) 46.2345 (+9%)
57.64a2 (+21%) 55.86a3 (+32%)
59.94=1 (-255) 54.77a3 (+29%)
Average score
0.2679 1605
0,2698
(+1%) 0.2106 (+319)
0,2774
0,1959 (422%)
0.3097 (+16%) 0.2432 (+51%)
Median norm. score
0.1856
0.0328
0.1830 (-1%) 0.1010 (+208%)
0.2171 (+17%) 0.0250 (-24%)
0.2634 (+42%% 0.1559 (+376%)
# Eow. improvements
0/16 0/16
8/16 11/16
8/16 9/16
11/16 13/16
that hyperbolic representations provide mostly complementary benefits and compatible with dif- ferent domain-specific practices, potentially yielding even further performance gains. Finally, we provide additional qualitative 2-dimensional visualizations of learned trajectory representation and 5-hyperbolicity recordings also for our regularized hyperbolic PPO algorithm.
D.1 RAINBOW DQN PROCGEN RESULTS
As shown in Table 7 our hyperbolic Rainbow DQN with S-RYM appears yield conspicuous performance gains on the majority of the environments. Once again, we find that reducing the di- mensionality of the representations to 32 provides even further benefits, outperforming the Euclidean baseline in 13 out of 16 environments. This result not only highlights the efficiency of hyperbolic geometry to encode hierarchical features, but also appears to validate our intuition about the useful- ness of regularizing the encoding of non-hierarchical and potentially spurious information. While still inferior to our best hyperbolic implementation, data augmentations seem to have a greater over- all beneficial effect when applied to Rainbow DQN rather than PPO. We believe this result is linked with recent literature (Cetin et al.] 2022) showing that data- aata--uggentation also provides off- off-policy RL with auxiliary regularization effect that stabilizes temporal-diffcee differrence learning.
D.2 SOTA COMPARISON ON PROCGEN
In Table R we compare our best hyperbolic PPO agent with the reported results for the current SotA Procgen algorithms from Raileanu & Fergus (2021). All these works propose domain-specific prac- tices on top of PPO (Schulman et al. |20 designed and tuned for the Procgen benchmark: Mixture Regularization (MixReg) (Wang al. 2020), Prioritized Level Replay (PLR) (Jiang et al., 1021),
Data-regularized Actor-Cntic (DraC) (Raileanu et al. 2020). Phasic Policy Gradient (PPG) (Cobbe ct al., 202I). p. and Invariant Decoupled Advantage Actor Critic (Raileanu & Fergus 2021) Validating our implementation, we see that our Euclidean PPO results closely match the previously reported ones, lagging severely behind all other methods. In contrast, we see that introducing our deep hy- perbolic representations framework makes PPO outperform all considered baselines but IDAAC, attaining overall similar scores to this algorithm employing several domain-specific practices. In particular, IDAAC not only makes use of very specialized architecture, but also introduces aux- iliary objective to minimize the correlation between the policy representations and the number of steps until completin... Raileanu & Fergus (22T)) found this measure to be effective heuris- tic correlating with the occurrence of overfitting in many Procgen environments. Moreover, we see that hyperbolic PPO attains the best performance on 7 different environments, more than any
24

---

Published as a conference paper at ICLR 2023
Table Performance comparison on the distribution of levels for our Euclidean and Hyperbolic PPO agents with the reported results of recent RL algorithms designed specifically for the Proegen benchmark.
Task'Alloriine
PPO (Reported) Mixieg
PLR
UCB-DrAC
PPG
IDAAC
PPO (Ours) Hyperbolic PPO S-RYM (Ours)
bigfish
3.7
7.1
10.9
9.2
11.2
18.5
1.4611
(617222++103795
bougfight
7.4
8.2
8.9
7,8
10.3
9.8
7.04=2
9.02=1 (+289))
caveflyer
6.1
6.3
5.2021 (-11%)
chaser
3.5
5.8
6.9
6.3
9.8
6.8
5.8921
1.32=1 (+24%)
climber
5.6
6.9
6.3
6.3
2.8
8.3
5.11=1
7,28=1 (+43%)
coinran
8.6
8.6
8.8
8.6
8.9
9.4
8.25=0
9.20=0 (+12%)
dodggehall
1.6
1.7
18
4.2
2.3
3.2
1,87=1
1.14=1 (+281%)
fruitbor
28
29.51
heist
2.5
2.6
29
3.5
2.8
3.5
2.92=1
3.60+1 (+23%)
jumper
5.9
5.8
6.2
5.9
6.3
6.14
6. (11S)
leaper
4.9
5.3
6.8
4,8
8.5
7.7
4.3622
7.00= (+615)
maze
5.5
5.2
5.5
6,3
5.1
5.6
6.50=0
7.10:1 (+9%)
miner
9.4
(+6%)
ninja
5.9
6.8
7.2
6.6
6.6
6.8
6,50=1
5.6021 (-14%)
plunder
5.9
8.7
8.3
14.3
23.3
6.0623
6.6800 (+10%)
starpilot
24.9
32.4
27.9
30
47.2
37
26.5745
38.2745 (+44%)
Average score
0.3078
0,3712
0.4139
0.3931
0,4458
0,5048
0.3476
0,4730 (+36%)
Median norm. score
0.3055
0.4263
0.4093
0.4264
0.4456
0.5343
0.3457
0,4705 (436%))
Table Performance comparison for our Euclidean and Hyperbolic Rainbow DQN agents with the reported results of recent RL algorithms designed specifically for the Atari 100K benchmark.
Taskk)AAgortte AAggorithn
OTRsinbow
DxQ
DQN (Ours) Rainhow DQN (Ours)
Alien
227.80
7127.70
739.9
824.7
558.2
771.2
801.5
548. 13
679.20 (141%)
Amidar
5,80
1719.50
188.6
82.8
142.1
102.8
176.3
132.55
118.62 (-11%))
222.40
706.26 (+52%)
Asterix
210.00
8503.30
470.8
628.5
734.5
603.5
977.8
448.33
535,00 (+36%))
Bank Heist
14.20
753.10
51
182.1
131.6
168.9
380,9
187.5
255.00 (+39%)
Rontl Zime
256000
37187.50
10124.6
4060.6
14870
12954
25800.00 (+132%)
Boxing
12,10
9.28 (1226))
Breakout
1,70
30,50
1.9
9.8
4,9
16.1
17.1
13.72
58.18 (+3700%)
Chopper Command
811.00
7387,80
861.8
1033,3
1058.5
780.3
974.8
791.67
858 00 (+11859)
Crazy Climber
10780.
Demon Attakk
152.10
1971.00
508
7118
817.6
1113.4
545.2
1204.75
4031.60 (+269%)
Freeway
0.00
29.60
27.9
25
26.7
9.8
24.4
30.5
29, (-3%)
Fronthie
331.1
Gopher
257.60
2412.50
669.3
63663
715.2
343.67
1132.80 (1977%)
Hero
1027.00
30826. 10
6857
6458.8
6279.3
3736.3
7019.2
9453.25
7654 4D (-21%%)
Jamesbond
29.00
302,80
301.6
112.3
236
365.4
190.83
380.00 (177%)
Kangomoo
779.3
94066
3276.4
1290
1020,90
Kru!l
1598.00
2665.50
2851.5
3277.9
4229.6
4018.1
3688.9
3445.02
3885.02 (+24))
Kung Fix Muster
258.50
22736.
143466.1
5722.2
14307.8
9111
13192.7
7145
1000 DD (+509E)
960.5
(-1
Pong
-20.70
14.60
-19.3
1.3
-16.5
-8.5
-5.9
3.85
11,98 (+33%)
Private Eye
24.90
69571.30
97.8
100
218.4
-13.6
124
72.28
106.06 (+719))
Qbert
163.90
13455.00
1152.9
509.3
1042.4
854.4
669.1
$60.83
2702.00 264%)
Rood Runner
11.50
7845 DO
9600)
2696.7
5661
8895.1
14220.5
6090
22256.00 (+266%)
Seaquest
68.40
42154.71
70
354.1
286.9
384.5
301.2
583.1
259.33
476 (+ 114%)
Up N Dewn
533.40
11693.20
2877.4
2847.6
2955.2
3180.8
28138.5
2935.67
3255.00 0055500000 (-1398)
(+94%)
Human Norm. Median
0.000
1,000
0.161
0.204
0.175
0.268
0.415
0.259
0.366 (+41%) 5
# Super
N/A
N/A
2
2
other method. Furthermore, in these environment the other Fuclidean algorithms specifically strug- gle, again indicating the orthogonal effects of our approach as compared to traditional RL advances.
D.3 SOTA COMPARISON ON ATARI 100K
In Table we provide detailed raw results for our hyperbolic Rainbow DQN agent, comparing with the results for recent off-policy algorithms for the Atari 100K benchmark, as reported by Schwarzer et al. (2020). All the considered algorithms build on top of the original Rainbow al- gorithm (Hessel et al. 2018). We consider Data Efficient Rainbow (DER) (an Hasselt et al., 2019) and Overtrained Rainbow (OTRainbow) (Kielak 2019) which simply improve the model architec- tures and other training-looy byper-parameters for instance, increasing the number of update steps per collected environment step. We also compare with other more recent baselines that incorpo-
25

---

Published as a conference paper at ICLR 2023
rate several additional auxiliary practices and ftta-augmentation such as Data-regularized Q (DrQ)
Yaratt et al. 2021a), Contrastive Unsupervised Representations (CURL) Laskin et al. 2020a). and Self-Predictive Representations (SPR) (Schwarzer et al.) (020). While our Euclidean Rainbow im- plementation attains only mediocre scores, once again we see that introducing our deep hyperbolic representations makes our approach competitive with the state-of-the-art and highly-tuned SPR al- gorithm. In particular, SPR makes use of several architectural advancements, data-augmentation strategies from prior work, and model-based contrastive auxiliary learning phase. Also on this benchmark, our hyperbolic agent attains the best performance on 8 different environments, more than any other considered algorithm.
D.4 2-DIMENSIONAL REPRESENTATIONS PERFORMANCE AND INTERPRETATION
To visualize and allow us to interpret the struc- ture of the learned representations, we analyze our hyperbolic PPO agents using only two di- mensions to model the final latent representa- tions. As mentioned in Section 4 and shown in Table IO we find even this extreme imple- mentation to provide performance benefits on the test levels over Euclidean PPO. Further- more, the generalization gap with the train- ing performance is almost null in three out of the four considered environments. As the 2-dimensional representation size greatly con-
Table 10: Performance of 2-dimensional hyper-
bolic PPO as compared to the original PPO algo- rithm.
Task Algorithm
PPO + S-RYM, dim.
Levels distribution
train/test
bigfish dodgeball fruitbot
5.65 (+52%) 2.34+3 (+60%)
2. 2.62+0 (-48%) 2.36#1 (+26%)
27. 18:4 (-10%)
25.75&1 (-2%)
starpilot
30,27+3 (-1%) 29.7246 (+12%)
strains the amount of encoded information, these results provide further validation for the affin- ity of hyperbolic geometry to effectively prioritize features useful for RL. We then observe how these 2-dimensional hyperbolic latent representations evolve within trajectories, mapping them the Poincaré disk and visualizing the corresponding input states. As summarized in Section 4 we observe recurring cyclical behavior, where the magnitude of the representations monotonically increases within subsets of the trajectory as more obstacles and/or enemies appear. Together with Figure [0 (on the bigfish enviromment), we provide additional qualitative visualizations of this phe- nomenon in Figure 12 (on the starpilot (A), dodgebull (B), and fruitbol (C) environments). These plots compare the representations of on-policy states sampled at constant intervals within a tra- jectory, every 15 timesteps, and deviations from executing 15 timesteps of random behavior after resetting the environment to the previous on-policy state. We observe the state representations form tree-like branching structures, somewhat reflecting the tree-like nature of MDPs. Within the sub- trajectories in starpilot and fruitbot, we find that the magnitudes in the on-policy trajectory tend to grow in the direction of the Value function's 's gyroplane's 's normal. Intuitively, this indicates that as new elements appear (e.g., new enemies in starpilor). the agent recognizes a larger opportunity for rewards (e.g., from defeating them) and also that it now requires a much finer level of control for optimality. This is because as magnitudes increase, the signed distances with the policy gyroplanes will also grow exponentially, and so will the value of the different action logits, decreasing the policy's 's entropy. In contrast, the magnitudes of the state representations following the random devi- ations grow in directions with considerably larger orthogonal components to the Value gyroplane's normal. This still reflects the higher precision required for optimal decision-making. as magnitudes still increase, but also the higher uncertainty to obtain future rewards from these less optimal states. As opposed to the rest of the environments, in dodgeball all enemies and other elements are al- ready present from the first time-step of a trajectory. Furthermore, our 2-dimensional hyperbolic agent appears to lack the representation power to recover good performance. These two properties make the observed branching phenomenon less accentuated, with the magnitude of both random and on-policy transitions changing in mostly orthogonal directions to the gyroplane's 's normal.
26

---

Published as a conference paper at ICLR 2023
Agent trajectory
A)
Initial state
Random trajectory deviations Agent trajectory
B)
Initial
ya
Random trajectory deviations Agent trajectory
C)
Initial
VT
Policy transitions
Random transitions
Volue gyroplane
Random trajectory deviations
Figure 12: Visualization of 2-dimensional hyperbolic embeddings in the starpilor (A), dodgeball (B), and fruitbor (C) Procgen environment. We sub-sample states from recorded agent trajectories every 15 timesteps. We show the evolution of the hyperbolic latent representations following the recorded policy transitions compared random transitions collected by resetting the environ- ments from each state and executing a random policy for the same 15 timesteps.
D.5 8 HYPERBOLICITY
We repeat the experiment in Figure 4 collecting the relative 5-hyperbolicity of the latent represen- tations space produced by our regularized Hyperbolic PPO agent throughout training. Our analy- sis assumes that as different RL tasks likely require encoding different amounts of hierarchically- structured information, relative changes in o-hyperbolicity should be more informative than its over- all scale. Furthermore, we would like to emphasize that we tractably estimate 8ret using the efficient but approximate algorithm from Fournier et al. (2013) and that, in practice, there are many sources of noise that affect RI. optimization. Both these factors inevitably add uninformative noise to the latent representations and our hyperbolicity recordings, which likely affects many of the small local observed changes in drel.
We visualize the evolution of the d-hyperbolicity our hyperbolic PPO agent regularized with S- RYM in Figure 13 As we would expect, using hyperbolic latent space yields latent representations with significantly lower values 8ret, as compared to standard PPO, implying they possess an in- creased hierarchical tree-like structure. We observe this consistently for all considered tasks and during all stages of the RL training process. This difference is particularly evident at initialization where 8 R 0.2 for our hyperbolic PPO while e s 0.4 for standard PPO agent, reflecting
27

---

Published as a conference paper at ICLR 2023
bigfish
dodgeball
fruitbot
starpilot
Dpisedic return
0.227.5
30
0.24
40
0.205.0
0.22
0.20
10
20
0.18
2.5
0.20 10
0.20
0
0.16
10000
20000
10000 20000
10000
20000
Hyperbolic PPO performance (All text leves)
hyperbolic PPO performence (200 tranning leves)
diypeebolicity
Figure 13: Performance and relative d-hyperbolicity of the final latent representations of our hyper- bolic PPO agent regularized with S-RYM.
how the properties of hyperbolie space facilitate capturing hierarchical relationships training. Interestingly, in the dodgeball environment, the recorded value of ôrel for our Hyperbolic PPO experiences some considerable fluctuations in some of the very initial and later training stages. Analogously to our results in Figure it appears that during the iterations where 8ret increases or attains its higher values, test performance grows significantly slower. Overall, dodgeball also the environment with the largest generalization gap relative to the hyperbolic agent's training perfor- mance.
with no
E FURTHER EXPERIMENTS AND ABLATION STUDIES
In this section, we further analyze the properties of our hyperbolic RL framework and its implemen- tation, through additional experiments and ablations. We focus on our hyperbolic PPO algorithm and four representative tasks from the Procgen benchmark.
E. S-RYM'S COMPONENTS CONTRIBUTION
bigfish
dodgeball
fruitbot
starpilot
Episodic vetum
15
30
40
10
20
20
5
10
0
10000
20000
10000
20000
0
10000
20000
10000
20000
S-RYM
360
rescaling
163
Figure 14: Performance ablating either spectral normalization rescaling from our Hyperbolic PPO agent stabilized with S-RYM.
Our proposed spectrally hyperbolic mappings (S-RYM) relies on two main distinct com- ponents: spectral normalization and rescaling. As described Section 3 we design our deep RL models to produce representation by applying traditional neural network layers in Euclidean space Jg (s)... Before the final linear layer fH. we then use an exponential map from the origin of the Poincaré yield a final representation in hyperbolic space zи exp](=g) As shown by Lin et al. (2021) ) applying spectral normalization to the layers of fÆ regulates both the values and gradients similarly LeCun initialization (LeCun al. 2012). Hence, make the regularization approximately dimensionality-io invvariant by rescaling TE E IR,", simply dividing its value by vñ. In Figure 14, we show the results from ablating either component from S-RYM. From our results, both components seem crucial for performance. As removing spectral normalization simply recov- ers the unregularized hyperbolic PPO implementation with some extra rescaling in the activations, its performance is expectedly close to the underwhelming performance of our naive implementa- tions in Figure 5 Removing our dimensionality- effect, with almost no agent improvements in 3 out of 4 environments. The necessity of appropriate scaling comes from the influence the representations magnitudes have on optimization. When ap- plying spectral normalization, the dimensionality of the representations directly affects its expected magnitude. Thus, high-ding -igh--imensiooal latents will result in high -aggiiude representations, making challenging optimize for appropriate angular layouts in hyperbolie space (Nickel & Kiela 2017
rescaling appears to have an even larger
28

---

Published as a conference paper at ICLR 2023
Ganea et al. 2018 and making the gradients of the Euclidean network parameters stagnate (Guo et al. 2022). These issues cannot even be alleviate with appropriate network initialization, since the magnitudes of all weights will be rescaled by the intruduced spectral normalization.
E.2
REPRESENTATION SIZE
bigfish
dodgeball
fruitbot
starpilot
20
30
40
Episocic
10
25
30
20
2
2
2
2
2
2
2
2
PPO
Hyperbolic PPO + S-RYM
Latent dimonsionality
All test levels
---
200 training levels
Figure 15: Final performance comparison between PPO agents with Euclidean and hyperbolic rep- resentations with different dimensionalities.
In Figure 15
we show the final train and test performance attained by our Euclidean and hyperbolic PPO agents with different dimensionalities for their final latent representations. We collect results on log scale 2" with E {3, 4, 5, 6, 7,8), i.e., ranging from 2 = 8 to 2 256 latent dimen- sions. Integrating our hyperbolic representations framework with PPO boosts performance across all dimensionalities. Moreover, in 3/4 environments we see both train and test performance of the Fuclidean PPO agent considerably dropping as we decrease the latent dimensions. In contrast, the performance of hyperbolic PPO is much more robust, even attaining some test performance gains from more compact representations. As described Section 2 Euclidean representations require high dimensionalities
encode hierarchical features with low distortion (Matoušek [990 Gupta 1999). which might explain their diminishing performance. Instead, as hyperbolic representations do not have such limitation, lowering the dimensionality should mostly affect their ability of encod- ing non-hierarchical information, which we believe to counteract the agent's tendency of overfitting to the limited distribution of training levels and observed states.
E.3 COMPATIBILITY WITH ORTHOGONAL PRACTICES
Introducing hyperbolic geometry to model the representations of RL agents is fundamentally or- thogonal to most recent prior advances. Thus, we validate the compatibility of our approach with different methods also aimed at improving the performance and generalization of PPO.
bigfish
dodgeball
fruitbot
starpilot
30
10
30
40
20
20
Epitodie
10
S
10
20
0
10000
20000
0
10000
20000
0
10000
20000
0
10000 20000
PPO
PPG
Hyperbolic PPO SRRM
Hyperbalic PPG
best
200 train
Figure 16: Performance comparison from integrating the advances from the PPG algorithm our hyperbolic reinforcement learning framework.
Phasic Policy Gradient (PPG). We re-implement this recent PPO extension designed by Cobbe al. (202T) specifically for the Procgen benchmark. PPG adds non-trivial algorithmic and compu- tational complexity, by performing two separate optimization phases. In the first phase, it optimizes the same policy and value optimization objective as in PPO, utilizing the latest on-policy data. In the second phase, it utilizes much larger buffer of past experience to learn better representations in its policy model via an auxiliary objective, while avoiding forgetting with an additional behavior cloning weighted term. The phases alternated infrequently after several training epochs. Once again, we incorporate our hyperbolic representation framework on top of PPG without any additional tuning. In Figure 16 show the results from adding our deep hyperbolic representation
29

---

Published as a conference paper at ICLR 2023
framework PPG. Even though PPG's performance already far exceeds PPO, hyperbolic represen- tations appear to have similar effects on the two algorithms, with performance on the 200 training levels largely invaried, and especially notable test performance gains the bigfish and dodgeball environments. Hence, in both PPO and PPG, the new prior induced by the hyperbolic representa- tions appears largely reduce overfitting the observed data and achieve better generalization to unseen conditions. Our approach affects RL in an orthogonal direction to most other algorithmic advances, and our results appear to confirm the general compatibility of its benefits.
bigfish
dodgeball
fruitbot
starpilot
Epizodic return
20
30
7.5
40
20
10
5.0
10
20
0
0
10000
20000
20000
10000
PPO
PPO data aug.
Hyperbolic PPD + S RY
Hyperbalic PPO + RMM data aug.
All test levels --- 200 train levels
Figure 17: Performance comparison from integrating data augmentation with the Euclidean and hyperbolic PPO agents.
Data augmentation. Finally, we also test introducing data augmentation to our Hyperbolic PPO implementation. We consider the same popular random shifts from Yarats al. (2021a) evaluated in Section 4 We note that the problem diversity characterizing procgen makes it challenging for individual hand-designed augmentations to have generally beneficial effect, with different strate- gies working best in different environments (Raileanu et al., (020). In fact, applying random shifts PPO appears to even hurt performance on considerable subset of environments (see Table [ likely due to the agents losing information about the exact position and presence of key objects at the borders of the environment scene. This inconsistency is reflected onto the hyperbolic PPO agent. In particular, while the addition of random shifts further provides benefits the bigfish environment, appears to hurt performance on dodgeball. Overall, integrating our hyperbolic framework still appears considerably beneficial even for the test performance of the data-augmented agent, further showing the generality of our method.
E.4 ENFORCING LIPSCHITZ CONTINUITY
bigfish
dodgeball
fruitbot
starpilot
15
30
40
7.5
10
20
Eqinndic
5.0
20
5
2.5
10
0.0
0
10000
20000
10000
20000
10000
20000
10000
20000
tot
PPO
PPO
GP
Hygerbolic PPO + S-RYM
Hyperbolic PPO S-RYM & test levels
200 train levels
Figure 18: Performance of a standard PPO agent and the hyperbolic PPO agent stabilized with S-
RYM after integrating gradient penalties (GP) (Gulrajani et al. 2017a).
S-RYM leaves the final layer of the network unregularized since there is direct way of perform- ing power iteration with its parameterization and we also do not want to constrain our models of the value and policy be 1-Lipschitz. We validate that this property not reflective of the true optimal policy and value functions, by enforcing Lipschitz continuity with gradient penalties (GP) (Gulrajani al. 20) We apply GP on top of both our Hyperbolic PPO with S-RYM and standard PPO. Our results in Figure E.4 appper to validate our hypothesis by showing that enforcing either the hyperbolic or Euclidean PPO models to be 1-Lipschitz makes performance collapse across all environments.
30


Reviews:
[{"review_id": "lYPiI5By6jz", "reviewer": "Reviewer_7pBh", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes spectrally-regularized hyperbolic mappings (S-RYM) to learn a hyperbolic representation for deep RL, by applying a spectral normalization for learning the hyperbolic representation. ", "summary_of_the_review": "In summary, since the empirical results of this work beat the SOTA, the reviewer believes it definitely reaches the bar for acceptance. However, the reviewer believes there is a large room for the writing part (for clarifying the methods and addressing how the proposed method is different from SN + Hydra) so that other readers to better appreciate the merits of this work.", "strength_and_weaknesses": "Strength:\n1. As an empirical paper, it demonstrates superior performance on many benchmarks (Atari and Procgen).\n2. It illustrates the connection between the hierarchical structure in hyperbolic representation and reinforcement learning\n\nWeakness/Questions:\n1. To the best of the reviewer’s understanding, the implementation details of this paper are not clearly stated:\n(a) If the reviewer understands the implementation clearly, it seems that the implementation adopts the spectral normalization to the hydra package (https://github.com/facebookresearch/hydra)? The reviewer would appreciate it if the author can clarify the novelty in the implementation for better readability! (b) Regardless of whether the novelty in implementation is purely “spectral normalize+hydra”, the author should at least cite the Hydra Package if they are using it as an implementation backbone. (c) The reviewer guesses the last paragraph of section 2.2 “In line with recent use of hyperbolic geometry in supervised …” characterizes the main contents of the main implementation details of the hyperbolic embedding, but perhaps the authors can elaborate more on this part so that the main contribution of this work is much better than  “spectral normalization + some hyperbolic representation learning method other people proposed”.\n2. The motivation from $\\delta$-hyperbolicity to the pursuit of hyperbolic representation is very inspiring. The reviewer is wondering whether the authors can reproduce Figure 4 using PPO + S-RYM. If the authors can demonstrate the $\\delta$-hyperbolicity decreases with PPO + S-RYM, it would also improve the results of this work.\n3. How does the implementation of the hyperbolic embeddings different from other online packages (e.g., https://github.com/nalexai/hyperlib)? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UFzunj_b9b7", "reviewer": "Reviewer_56hh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to model the latent state representation in RL with hyperbolic space. They find that naively applying existing hyperbolic deep learning methods is not helpful, and introduce two techniques (spectral normalization and feature rescaling) to address the issues. The integrated method (S-RYM) is evaluated on Procgen and Atari-100K benchmarks, effectively boosting the performance compared to baselines.", "summary_of_the_review": "In summary, this paper identifies and resolves the issues when applying hyperbolic deep learning methods to RL. Modelling latent representations with hyperbolic space naturally fits the hierarchical relationship between states in RL. Extensive experiments on Procgen and Atari-100K demonstrate the effectiveness of hyperbolic RL.", "strength_and_weaknesses": "Strength:\n* The paper is clearly written and easy to follow.\n* The idea of learning latent representations using hyperbolic space for RL is novel, and successfully adapting hyperbolic deep learning methods to RL is non-trivial.\n* The empirical performance improvements on Procgen and Atari are significant.\n* Visualizations of 2-dimensional hyperbolic embeddings (Figure 10 and 12) are very intriguing.\n\nQuestions:\n* In general, the state relationship is more like a directed graph than a tree. I am curious how this would fit into the hyperbolic latent learning framework in this paper?\n* Can the authors provide more details on how \"Euclidean PPO + S-RYM\" in Figure 7 is implemented?\n* It would be better if the authors can make more visualization (like Figure 10) for other games in Procgen.\n\nMinor issues:\n* Typo: below Table 4, \"common practice win on-policy methods\", win -> in?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-YsgQyMN4nW", "reviewer": "Reviewer_4ZkJ", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces hyperbolic representations to deep reinforcement learning, with the intuition that hyperbolic space should be better for representing the sorts of hierarchical relationships between subsequent states found in deep RL. However, a straightforward implementation built on top of PPO does not seem to improve performance, and in fact results in worse performance on some environments. The paper hypothesizes that this is due to optimization difficulties introduced by the hyperbolic layer, and proposes using spectral normalization and rescaling of the hyperbolic layer's outputs to alleviate the optimization difficulties. The resulting module improves performance when combined with both PPO and Rainbow DQN on a broad set of benchmarks. Extensive empirical investigations suggest that hyperbolic representations could be a good basis for building future deep RL algorithms.", "summary_of_the_review": "I recommend accepting the paper for publication; it is very clearly written, proposes a new approach, shows a general way to get it to work with existing algorithms, and demonstrates the benefits on multiple benchmarks. It would be nice if my concerns were addressed, but even without doing so the paper should probably be published anyways.\n\n**Update:** The authors have addressed all of my concerns satisfactorily, and I can now strongly recommend accepting the paper.", "strength_and_weaknesses": "**Strengths:**\n+ Very well-written and clear; the analogy between hyperbolic spaces and tree structures was very helpful for gaining intuition, as were the examples provided throughout (Figures 2 and 3 in particular).\n+ The introduction does a good job of arguing for hyperbolic representations for a variety of reasons.\n+ An ablation study performed on the two components (spectral normalization of the euclidean sub-network and rescaling of its output) of the proposed method S-RYM showing they are both necessary.\n+ Broad empirical demonstrations that result in fairly consistent performance improvements.\n\n**Potential weaknesses/questions:**\n1. In Figure 4, the relationship between $\\delta$-hyperbolicity and the train-test gap seems very questionable. In all of the plots, the train-test gap grows while $\\delta$-hyperbolicity is shrinking. Additionally, in the fruitbot plot $\\delta$-hyperbolicity increases several times with no apparent change in the train-test gap. Furthermore, the scale of the $\\delta$-hyperbolicity is the same between representations that learned to successfully solve the task (fruitbot and starpilot) and those that failed to generalize (bigfish and dodgeball), suggesting all of the representations are of a similar level of hyperbolicity. Doesn't that suggest that something else is the cause of the failure to generalize? It definitely does not validate the hypothesis.\n1. Much of the reasoning in the first part of Section 3.3 seems very speculative. It would be good to adjust the language to better convey that this explanation is a hypothesis.\n1. How do standard neural net layers yield Euclidean velocity vectors? This detail seemed to be glossed over.\n1. Some of the figures and most of the plots are too small to read easily. Consider removing the depictions of the environments, and moving the legends to be horizontally laid out underneath the plots. That should allow for larger plots without too much vertical space being used.\n1. The colour scheme is not accessible to people with colourblindness, and the size of the lines in the legends are too small to easily see their colour.\n1. What are the shaded regions in the plots? Confidence intervals? I wasn't able to find this information.\n1. \"our instabilities appear to occur in the gradients from the hyperbolic representations\". This was not clear to me from Figure 6. Is there some reason to not use spectral normalization on the hyperbolic representations? Did it hurt performance?\n1. Do the representations learned using S-RYM actually exhibit increased $\\delta$-hyperbolicity over the base algorithms? I'm very curious about this, but couldn't find this information in the appendices.\n1. What are some of the shortcomings of the proposed algorithm? I could not find any discussion of limitations.\n\n**Minor comments/questions:**\n- Bottom of page 2: duplicate \"upon\".\n- Bottom of page 3: \"of the form in Equation 6\" would be clearer.\n- Figure 6 appears before Figure 5.\n- What does the dashed line represent in Figure 5?\n- Start of related work section: \"objectives appears has been shown prone to overfitting\"\n- The legend in Figure 8 seems not quite right. The legend indicates dashed lines represent performance on the 200 training levels, but the plots involve a varying number of training levels.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%