Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Authors: Hojoon Lee
Youngdo Lee
Takuma Seno
Donghu Kim
Peter Stone
Jaegul Choo
Venue: ICML 2025 spotlightposter (2025)
Keywords: reinforcement learning
normalization
Primary area: reinforcement_learning->deep_rl
TL;DR: Normalizing weights and features on unit norm hypersphere allows scaling up parameters and computations in RL

Abstract: Scaling up the model size and computation has brought consistent performance improvements in supervised learning. 
However, this lesson often fails to apply to reinforcement learning (RL) because training the model on non-stationary data easily leads to overfitting and unstable optimization.
In response, we introduce SimbaV2, a novel RL architecture designed to stabilize optimization by (i) constraining the growth of weight and feature norm by hyperspherical normalization; and (ii) using a distributional value estimation with reward scaling to maintain stable gradients under varying reward magnitudes. Using the soft actor-critic as a base algorithm, SimbaV2 scales up effectively with larger models and greater compute, achieving state-of-the-art performance on 57 continuous control tasks across 4 domains.

Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Hojoon Lee 1* Youngdo Lee"
Takuma Seno
Donghu Kim
Peter Stone
Jaegul Choo
Abstract
Success-normalized score
0.92
SimbaV2
Scaling up the model size and computation has brought consistent performance improvements in supervised learning. However, this lesson often fails apply to reinforcement learning (RL.) be- cause training the model non-stationary data easily leads to overfitting and unstable optimiza- tion. In response, we introduce Simba V2, novel RI. architecture designed to stabilize optimization by (i) constraining the growth weight and fea- ture norm by hyperspherical normalization; and (ii) using a distributional value estimation with reward scaling maintain stable gradients der varying reward magnitudes. Using the soft actor-critic as base algorithm, SimbaV2 scales effectively with larger models and greater com- pute, achieving state-of-the-an performance on 57 continuous control tasks across domains. The code is available at dojeon-ai. wit.aaaiigittbbiii/iiiiiii io/Simba2.
0.84
Simba
BRO
10
0.76
TD-MPC2
0.68
TD7
0.60
0.0
1.5
3.0
4.5
6.0
7.5
Training Time (hr)
Figure Compute RL Performance. Performance scales with increased compute when using Soft Actor- Actor-Critic with Si architecture, outperforming other statc-of-the-art RI algorithms, SimbaV2 achieves 0.848 normalized return with an update-to-data (UTD) ratio surpassing TD-MPC2 (0.749 at UTD=1), Simba (0.818 at UTD=8), and BRO (0.807 at UTD=8). Grey numbers below each point indicate UTD ratio. Results are averaged over 57 continuous control tasks from MuJoCo, DMC, MyoSuite, and HumanoidBench, each trained on 1 million samples.
1. Introduction
sources frequently leads to overfitting to earlier experiences and reduced adaptability to new tasks (Lyle al., 2022; Dohare al., 2023).
Over the past decade. a scaling law has emerged as the cornerstone of supervised learning (SL), suggesting that increasing model size, compute, and data consistently im- prove performance (Kaplan et al., 2020; Dehghani et al., 2023). This paradigm has driven significant breakthroughs, from large language models (Gemini et al., 2023; Achiam et al., 2023) to diffusion models (Ramesh al., 2021; Rom- bach al., 2022), where bigger models reliably translate to better performance.
One of the root causes of RL's scaling challenges lies in un- controlled norm growth during training, which destabilizes optimization in multiple ways. Feature norms grow uncon- trollably due to the implicit bias of TD loss (Kumar et al., 2022), where dominant dimensions cause overfitting and loss of plasticity (Lyle al., 2022; Ma al., 2023). Parame- norms grow unbounded, reducing effective learning rates (gradient-to- parameter retrrmnttt--parameeer ratio) and making weight updates increasingly difficult (Dohare et al., 2023; Lyle et al., 2024). Gradient norms fluctuate due to varying reward scales and outliers, further disrupting optimization. These instabilities are compounded with an increased model size or update frequency, making RL harder to scale than SL.
In contrast, scaling laws often fail to apply in reinforcement learning (RL) (Song et al., 2019; Li et al., 2023). Unlike SL's static data distributions, RL agents must contend with continuously evolving data distributions and shifting ob- jectives throughout their training process (Sutton & Barto, 2018). This fundamental non-stationarity creates scaling paradox: increasing model capacity or computational re-
Previous work has addressed these norm instabilities through separate, isolated approaches. Normalization lay- ers such as Ez-nornalization (Bjorck et al., 2021; Hussing et al., 2024), layer normalization (Lei Ba et al., 2016; Lyle et al., 2023), and RL-specific variants (Bhatt et al., 2024; Lee et al., 2024c) control the growth of feature norm. Weight de- cay (Farebrother al., 2018) manages the growth of param-
'quual contribution !KAIST SSony AI 'UT Austin. Correspon- dence to: Hojoon Lee <joonlecsky kr?>
Proceedings of the 4 International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SimbaV22 V2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Gym Mujoco
DMC Suite
MyoSuite
Humanoid Bench
Locomotion, tasks
Locomotion, tasks
Manipulation, tasks
Locomotion, tasks
1.6
0.8
Total Reward (1k)
TD3-Normalizer
1.2
Success rate
0.6
Success-Normal
0.6
0.8
1D3+0M
Simba22
0.4
TD MPCP
SimbaV2
0.4
TD-MPC2
SimbaV2
0.4
TD-MPC2
SimbaV2
0.4
MR.Q
CrossQ
TD7
0.2
1D7
Simba
MR.Q
0.2
SAC
BRO
Simba
0.2
1DJ
BRO
Simba
0.0
0.0
0.0
0.0
Online RL (All)
Offline
Training FPS
Evaluation FPS
Total 57 tasks
D4RL Mujoco, 9 tasks
DMC -Cartplee
DMCC-Catpplle
Mean Performance
0.8
Mean Performance
0.8
Frames second
80
Frames per second
1.2
0.6
0.6
SimbaV2 4BC
&
0.9
Dreamerv3
Dreamerv3
Dreamerv3
TD MP-M
SimbaV2
TD7+BC
40
TD MPC2
SimbaV2
0.6
TD-MPC2
SimbaV2
0.2
TD7
Simba
0.2
IDQL
DQL
20
BRO
Simba
0.3
BRO
Simba
DC
0.0
Figure 2. Benchmark Summary. (a) S2,baV2, with an aydate-to-daaie (UTD) ratio of 2, outperforms state-of-the-an RL algorithms
across diverse continuous control benchmarks using fixed hyperparameters
all domains. (b) Simba delivers competitive
performance in both online and offline RL while requiring significantly less training computation and offering faster inference times.
eter norm. Reward scaling and cross-entropy loss (Schaul et al., 2021; Farebrother et al., 2024) were adopted to control gradient norm fluctuations. However, these techniques are applied individually without a unified framework, making coordination and scaling difficult. Periodic weight reinitial- ization (Nikishin al., 2022; D'Oro al., 2023; Schwarzer al., 2023) offers alternative by completely retraining networks periodically. While effective, this approach re- quires additional training time and causes sharp performance drops, making it impractical for sufety-critical applications.
Using Soft Actor-Critic (Haarnoja et al., 2018) our base algorithm, Simba V2 effectively stabilizes all three types of norms while maintaining consistent effective learning rates throughout training (Section 5.2 and Figure 4). We eval- uated SimbaV2 on four standard online RL benchmarks: MuJoCo (Todorov al., 2012), DMC Suite (Tassa al., 2018 8), MyoSuite (Caggiano et al., 2022), and Humanoid- Bench (Sferrazza al., 2024); as well the D4RL MuJoCo benchmark (Fu et al., 2020) for offline RL. As shown in Fig- ures and 2, Simba V2 achieves state-of-the- the-rhr performance without requiring algorithmic modifications or hyperparam- eter tuning, and scales effectively with increased model size and computation without using periodic reinitialization.
In response, we present SimbaV2, a novel RL architecture that addresses these challenges by simultaneously stabiliz- ing weight, feature, and gradient norms within a unified framework. Building on the Simba architecture (Lee et al., 2024c), which uses pre-layernorm residual blocks (Xiong et al., 2020) and weight decay (Krogh& & Hertz, 1991), Sim- baV2 introduces three key modifications:
2. Related Work
2.1. Regularization in Deep Reinforcement Learning
Deep RL is particularly susceptible to overfitting due to its inherently non-stationary optimization process (Song et al., 2019). To address overfitting, researchers have adapted regularization techniques from SI., including weight decay (Farebrother al., 2018), dropout (Hiraoka et al., 2021). various normalization layers (Gogianu et al., 2021; Bjorck et al., 2021; Lyle al., 2023; Gallici et al., 2024; Bhatt et al., 2024; Lee et al., 2024c; Elsayed et al., 2024; Palenicek et al., 2025), and mixture of expert (Obando-Ceron et al., 2024; Willi et al., 2024). However, these methods often prove in- sufficient when scaling RI. models, as larger computational resources and increased model sizes can easily exacerbate overfitting (Li et al., 2023; Nauman al., 2024a).
Hyperspherical Feature Normalization: We replace all layer normalization with hyperspherical normalization (2-normalization).
Hyperspherical Weight Normalization: We remove weight decay and instead project weights onto the unit- norm hypersphere after each gradient update (Loshchilov al., 2024). Combined with hyperspherical feature nor- malization, this ensures consistent effective learning rates across layers and eliminates the need for weight regular- ization tuning.
Distributional Value Estimation with Reward Scaling: To address unstable gradient norms caused by varying reward scales and outliers, we integrate a distributional critic (Bellemare et al., 2017) and apply reward scaling to maintain unit variance of the target throughout training.
To further scale computations and model sizes in RL, recent studies have explored periodic weight reinitialization strate- gies to rejuvenate learning and escape local minima (D' Or'Oro
2

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
al., 2023; Nauman al., 2024b). These strategies include reinitializing weights to their initial distributions (Nikishin et al., 2022), interpolating between random and current weights (Xu et al., 2023; Schwarzer et al., 2023), utilizing momentum networks (Lee et al., 2024b), and selectively reinitializing dormant weights (Sokar et al., 2023). While promising, reinitialization has notable limitation: can lead to the loss of useful information and incur significant computational overhead model size increases.
The Q-function Qo(o, a) is trained to minimize the Bellman residual loss:
L - (Qslo,a) (0Q((o0,,a) aloog To (a )
where d' TO(-lo)), [0, is the discount factor, and Q3 represents the target Q-network updated via expo- nential moving average of ø.
3.2. Simba Architecture
To address these limitations, we introduce SimbaV2, an architecture that explicitly constrains parameter, feature, and gradient norms throughout training. By constraining norms through hyperspherical normalization, SimbaV2 stabilizes an optimization process and eliminates the need for weight decay or periodic weight reinitialization.
Simba (Lee et al., 2024c) is an RI. architecture with nor- malization layers composed of the following stages:
Input Embedding. Given an input observation o, E IR ol Simba applies Running Statistics Normalization (RSNorm) to normalize each dimension to zero mean and unit variance.
2.2. Hyperspherical Representations in Deep Learning
At each timestep I, the running mean μ4, E | and vari- ance O² E [ are updated recursively as:
Hyperspherical representations are widely used in deep learning across image classification (Salimans & Kingma, 2016; Liu al., 2017b), face recognition (Wang al., 2017: Liu et al., 2017a), variational autoencoders (Xu & Durrett, 2018), and contrastive learning (Chen et al., 2020). Using spherical embeddings is known to enhance feature sepa- rability (Wang& & Isola, 2020), improving performance in tasks requiring precise discrimination. Recently, researchers have applied the hyperspherical normalization to intermedi- ate features and weights to stabilize training in large-scall scale models such as diffusion models (Karras et al., 2024) and transformers (Loshchilov et al., 2024).
Nr = M1- + 1 oo o;
o² + 1 - -0;11 (3)
where ò; 0: 11-1-
Given running statistics, the observation is normalized as:
ã. RSNorm(o.) O. Mr
(4)
Voi +
Then, the normalized observation, oG, is embedded with a linear layer W E R|O|xds defined as:
In this work, we apply hyperspherical normalization to RL. Unlike previous studies that focus on training the network on stationary data distributions with discrete inputs and outputs, demonstrate their effectiveness on non-stationary data distributions with continuous inputs and outputs.
h = W
(5)
Latent Encoding. Next, the embedding h. is encoded by stack of L residual blocks with pre-layer normalization. For E {l....,,b L}. each of the (-th block is defined as:
3. Preliminaries
h h MLP(LayerNNom(((
(6)
As background, we briefly explain the Soft Actor-Critic (SAC) algorithm (Haarnoja et al., 2018) and the Simba architecture (Lee al., 2024c).
After the final block, the output is normalized again obtain the latent feature:
2, LayerNormmhh))
3.1. Soft Actor Critic
Output Prediction:
Finally, to predict the policy or Q-
SAC is a prominent off-policy algorithm for continuous control. aims maximize both expected cumulative re- ward and policy entropy, where (o, a, r, o') represents transition tuple. SAC comprises stochastic policy To (alo). Q-function @o a), and entropy coefficient a that balances reward and entropy.
value, a linear layer W. E Rdd xd. maps 2, to:
Pr W
(8)
4. Simba V2
SimbaV2 builds on Simba by adding constraints on weights, features, and gradients to enhance training stability, particu- larly when scaling to larger models and more computation. The modifications include:
The policy network optimized to maximize the expected return while encouraging entropy, which formalized as:
L = Eaaty (a log (a)o)) Qo(o, a))
(1)
3

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Input
Shift & Norm
MSE Loss KL--divergence Loss: MSE-based Bell-
Add a New Axis
loss is replaced with KI.-divergence loss, using categorical critic (Bellemare et al., 2017).
RSNorm
New
Shift
No Reward Scaling + Reward Scaling: Rewards are normalized with running statistics to stabilize the scale of both actor loss (Fquation.) 1) and critic loss (Equation.2).
L2 Norm
bota Awes
Linear
Shift Upwards
In the following subsections, we describe these modifica- tions in detail.
Scaler
L2 Norm
4.1. Input Embedding
L2 Normalization
Linear
Following Simba, S2 first standardize the raw obser- vations O, E [Ol using RSNorm, yielding o, To further stabilize training, we map o, onto the unit hypersphere be- fore applying a linear layer.
Scaler
ReLU
Linear
L2 Norm
LERP & Norm
Shift + 22-Norm. Direct f--normalization can discard magnitude information (e.g., O, 0] and [2, 0] both map [1, 0)). To retain magnitude information, we embed o into an (|O) + 1) d-dimensional vector by concatenating positive constant Cahift > 0, then apply éz-normalization:
LERP
LERP
L2 Norm
Snuthed
XN
or lg No-Norm( ([oe: Cahift]
Linear
Normalization
As illustrated in Figure 3, this additional coordinate encodes the original norm of o,, preserving magnitude information.
Scaler
Linear
Linear Scaler. We then embed o using linear layer W (1(0)+1)xd, and scaling vector sh. E IRd as:
Output
h = 62-Norm $ (W Norm(ō.))- (10)
Figure 3. SimbaV2 architecture. The input observation is first normalized using running statistics, then shifted along axis with a constant Gain to preserve magnitude information before be- ing projected onto unit hypersphere. The projected observation is passed through a linear layer, followed by series of non-lincar blocks and refined with LERP, serving as residual connection. A final linear layer predicts the policy value function.
where the Ey-normalization projects back to the hyper- sphere.
4.2. Feature Encoding
Starting from the initial hyperspherical embedding hü, apply I. consecutive blocks of non-linear transformations. Each (-th block transforms h into h|+1 as follows:
LayerNorm + fo-Norm: Layer normalization is re- placed with fz-normaiization constraining intermediate features to have unit norm.
MLP + 22-Norm. Each block uses an inverted bottle- neck MLP (Vaswani, 2017) followed by lg norommallzation to project the output back onto the unit hypersphere.
Linear > Linear + Scaler: Standard linear layer is decoupled into a linear layer with weights constrained to a unit norm hypersphere, without a bias, and learnable scaling vector that performs element-wise scaling.
h 42-Norr ReLU((W) h() O sk)). (1I)
where W 1 E R44d xdn and W E Rdd. Add, are weight matrices, and s E R44d is learnable scaling vector.
Residual Connection LERP: Residual connection is replaced with learnable linear interpolation (LERP), which combines raw and transformed features via a learn- able interpolation vector.
LERP 22-Norm. We then linearly interpolate between the original input hú and its non-linearly transformed output R1 followed by another fz-normalization:
I+1 Lg-Norm a') h a' @ h/). (12)
Weight Decay + Weight Projection: Weight decay is replaced with direct weight projection onto the unit hypersphere after each gradient update.
where 1 E dh and a' E dh are one vector and learnable interpolation vector, respectively.
4

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforeement Learning
LERP acts analogous a learnable residual connection but can also be viewed as a first-order approximation of Riemannian retraction on the hypersphere (Absil et al., 2008). Please refer to Appendix A.1 for further discussion.
Then, we track the running variance of G2, denoted oi and maintain running maximum:
Gr.max + max(Gr, Gt).
(18)
4.3. Output Prediction
We then scale the reward as follows:
We use linear layer to parameterize both the policy dis- tribution and Q-value. Because Simba's single Q-value estimate with an MSE-based Bellman loss is susceptible to outliers, we adopt a categorical critic with KL-divergence loss (Bellemare et al., 2017), which provides smoother gra- dients and more stable optimization (Imani & White, 2018).
Fz
(19)
max ( tc, Gt, G
This formula stabilizes gradients for both high-variance and low-variance returns, while thresholding with Gt,max/ Gmax ensures target returns remain within [Gmin Gmax]]
Distributional Critic. We represent the Q-value cate- gorical distribution over a discrete set of returns:
4.4. Initialization and Update
{o: Gmin + (i- Giax Gmia
In this subsection, we outline how weight matrix W, scaler 8, and interpolation vector a are initialized and updated.
Natom J:
Natom
(13)
Weight. All weight vectors are initialized orthogonally and then projected onto the unit hypersphere which forms an orthonormal basis. At each gradient step, we re-project them onto the unit sphere to maintain unit norm.
where Gunin and mmax denote the minimum and maximum possible returns, and Paaoom is the number of discrete atoms.
Given the encoded representation h we compute unnor- malized logits 2, E Al for all actions as follows:
Formally, let W be the weight matrix before the update, and let L denote the loss function. The update rule is defined as:
zt Wo,2(WW.W.... h so),
(14)
ac W + E2-Norm W - aaw
where WA, E Rdaxds, W., E RRAAXmammddd Xdn and 8, E dd are trainable parameters.
(20)
where 7) > O is a learning rate and fg-Norm is the b-- normalization operator along the embedding axis.
For each action a E A. the categorical probability is repre- sented by applying the softmax function to 2t.a E "
Scaler. Following Loshchilov et al. (2024), we decouple the initialization scale of s from its learning dynamics by using two scalars, Siit and 8scale Although 8 is initialized to Secale, it behaves if it was initialized to Sinit during the forward pass by:
Pt. = softmax(Z1.d)
(15)
The resulting Q-value the expected return under Pr.a:
Plaion
Q(o,,a)= = Σe S Pt.
(16)
S - Sacale 2 (Sinit Ø S
(21)
where O and Ø are element-wise product and division, respectively. This formulation lets Sscale control the learning rate of s independently from the global learning rate v).
Reward Bounding and Scaling. To use a categorical critic, we first bound the target returns within [Gmin: Gmax] and then scale the reward to maintain unit variance, ensuring stable gradients for both the actor and the critic. Unlike previous work (Schaul al., 2021), which scaled the critic loss, we scale the reward itself, affecting both components simultaneously. Moreover, unlike observation normaliza- tion, we do not center the reward, as shifting the reward can alter the optimal policy in episodic tasks (Naik et al., 2024).
When both the feature vector h E Rd and the randomly orthonormal initialized weight matrix W E Rd, xd, lie on the unit hypersphere, each component of Wh E RR4 can be approximated by cos(f) with Eg ccos" (0) = 1/2. Therefore, we set 8init 8scale = (v 22ddn) 1 to maintain unit norm after scaling at initialization. detailed derivation is in Appendix A.2.
Given a reward rt at time and a discounted factor y, we track running discounted return:
Interpolation vector: Analogous to the scaler, the inter- polation vector, a. also has Qivit and Oscaae- Following Loshchilov et al. (2024), we initialize ainit = 1/(L + 1) and @scale = 1/ to preserve residual feature and grad- ually integrate non-linear features.
Gt GG +
(17)
where G re-initialized to 0 at the start of each episode.
5

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
(a) Average Return
(b) Feature Norm
(c) Parameter Norm
(d) Gradient Norm
[e]
Effective LR
k10
x10-2
1.00
240
80
2.0
1.00
60.00
DMC-Hard
180
0.75
45.00
0.50
120
40
1.0
0.50
30,00
0.25
60
20
0.5
0.25
15.00
0.00
10
0.01
0.01
0.2
0.2
0.6
1.0
K10-1
x1001
HBench-Hard
1.00
100
120
1.00
10,00
0.75
75
90
0.75
7.50
50
60
0.50
5.00
0.25
25
30
0.25
2,50
0.00
0.01
0.01
0.2 0.4 0.5 0.8 1.0
0
0,2 0.4 0.6 0.8 1.0
0
0.2 0.4 0.6 0.8 1.0
0
0.2 0.4 0.6
1.0
0.2 0.4 0.6 0.8 1.0
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
SimbaV2
Simba
Si Encoder
SimbaV2 Predictor
Simba Encoder
Simbe Predictor
Figure 4. SimbaV2 vs. Simba Training Dynamics. We track 4 metrics during training to understand the learning dynamics of Simba (a) Average normalized return across tasks. (b) Weighted sum of f2-norms of intermediate features critic. (c) Weighted sum no-mor of all critic parameters (d) Weighted sum of 22 f2-norms of all gradients in critic (e) Effective learning rate (ELR) of the critic. On both environments, SimbaV2 V2 maintains stable norms and ELR, while Simba exhibits divergent fluctuations.
5. Experiments
and after scaling, we evaluate BRO, Simba, and Simba V2 under both low UTD ratios (< 2) and high UTD ratios (< 8). Additional details are described in Appendix E.
We now present a series of experiments designed to evaluate V2. Our investigation centers on four main setups:
Metrics. To aggregate performance across diverse domains, each environment's return is normalized to a near [0, 1) range. Specifically, MuJoCo performance is normalized by TD3 (Fujimoto et al., 2018); DMC returns are divided by 1000; MyoSuite scores use success rates; and Humanoid- Bench scores are normalized by their success score.
Optinmization Analysis (Section 5.2). Investigate whether Simba V2 stabilizes the optimization process.
Scaling Analysis (Section 5.3). Investigate whether Sim- baV2 allows scaling model capacity and computation.
Comparisons (Sections 5.4). Compare Simba V2 against state-of-ther thrt--r RL algorithms.
Training. If possible, we tried to closely follow Simba's training configuration aiming to provide an apples-to-apples comparison. Unless otherwise specified, the actor and critic have hidden dimensions of 128 and 512, respectively (ap- proximately 5M parameters). The model is trained for IM environment steps, using a UTD ratio 2. We used an Adam (Kingma & Ba, 2014) optimizer without weight de- cay and set the batch size to 256. The learning rate is linearly decayed from x 10-4 to 3 10-5, Full hyperparameter configurations are provided in Appendix C.
Design Study (Section 5.5.) Conducts ablation studies individual architectural components of V2.
5.1. Experimental Setup
Environment. A total of 57 continuous-control tasks are considered across 4 domains: MuloCo (Todorov et al., 2012), DMC Suite (Tassa et al., 2018), MyoSuite (Caggiano et al., 2022), and HumanoidBench (Sferrazza et al., 2024). Also, two challenging subsets are defined for an empir- ical analysis: DMC-Hard (7 tasks involving dog and humanoid embodiments) and HBench-Hard (5 tasks: run, bwwabcalaaaa.......aa.aa..aaa..a simple,, sir-hard, stair, walk),
5.2. Optimization Analysis
To understand the optimizatiom dynamics of SimbaV2, we measure the feature norm, weight norm, gradient norm, and the effective learning rate (ELR), defined as the ratio of the gradient norm to the weight norm (Kodryan et al., 2022; Lyle et al., 2024) (See Appendix G for details). We weighted average each metric across layers where weights correspond to each layer's fraction of total parameters. Additionally, we divide the layers into encoder layers (all layers before the output prediction) and predictor layers (those after) to analyze their respective dynamics.
Baselines. Comparisons include a broad range of deep RL algorithms, PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018), TD3 (Fujimoto et al., 2018) TD3+OFE (Ota et al., 2020), TQC (Kuznetsov al., 2020), DreamerV3 (Hafner et al., 2023), TD7 (Fujimoto et al., 2023), TD- MPC2 (Hansen et al., 2023), Cross-Q (Bhatt et al., 2024), BRO (Nauman et al., 2024b), MAD-TD (Voelcker et al., 2024), MR.Q (Fujimoto et al., 2025), and Simba (Lee et al., 2024c). Whenever available, we report the results from the original paper; otherwise, we run the authors' official code. In addition, to further compare performance before
Figure 4 compares Simba and Simba on DMC-Hard and HBBenchHHard As shown in Figure (.(b) Simba exhibits
6

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
DMC-Hard
HBench-Hard
DMC-Hard
HBedch-Hard
Total Total Reward (K)
0.75
Sucess Norm Score
0.90
Total Reward (K)
0.80
Sucess-Nome Norm Score
1.00
0.62
0.75
0.73
0.85
0.50
0.60
0.65
0.70
0.38
0.45
0.57
0.55
0.25
0.50
0.40
128
256
512
1024
128
256
512
1024
Hidden Dim Critic (dh)
Hidden Dim of Critic (dg)
Number of Critic Blocks (L)
Number of Critic Blocks (L)
SimbaV2
Simba
SimbaV2
Simba
Figure 5. Width Scaling. ters by increasing the width of the critic network. On DMC-Hard, both Simba and S2 benefit from increased model size. On HBBnnch-Hard however, Simba plateaus at larger model sizes, whereas V2 continues to improve.
We scale the number of model parame-
Figure 6. Depth Scaling, We scale the number of model parame- ters by increasing the depth of the critic network. On both DMC- Hard and HBench-Hard, SimbaV2 benefits from increased depth, while Simba's 'ss performance degrades beyond a shallow configura- tion 2 2.
large, often divergent fluctuations in feature, weight, and gradient norms between the encoder and predictor. Conse- quently, Figure 4.(e) shows that the encoder's ELR trending upward while the predictor's s ELR declines.
However, on the more challenging HBench-Hard bench- mark, while both methods achieve comparable performance at the smallest scale (dn, 128), their scaling behavior di- verges significantly. Simba plateaus at larger scales with peak performance at d, 1024, while SimbaV2 continues to improve with increased width. This demonstrates that V2'' stabilized training dynamics effectively lever- ages larger model capacity.
In contrast, S2 enforces tighter constraints, stabiliz- ing norms and ELRs throughout training. Although certain parameters (c.g., scalers or interpolation vectors) can ex- ceed the unit norm, the majority of parameters remain on the hypersphere, resulting in more robust optimization. A standalone visualization of S2 is in Appendix G.
Figure 6 presents depth scaling results. In HBench-Hard. Simba shows consistent performance improvements as the depth of the critic L increases, successfully solving the five complex tasks in I, = 8. On DMC-Hard, SimbaV23 performance also improves with depth but begins saturating around L = 4, likely due to task complexity limitations rather than architectural constraints. In contrast, Simba's 'ss performance either plateaus around L = 2 or slightly de- creases after initial improvement. This clear difference demonstrates SimbaV2's superior depth scalability, which we attribute to its effective regularization mechanisms that enable stable training of deeper networks.
5.3. Scaling Analysis
For this experiment, we investigate whether SimbaV2's stable training dynamics enable better scaling performance as model parameters or computational resources increase, while reducing overfitting compared to existing methods.
Experimental Setup. We conduct two types of scaling ex- periments. For parameter scaling, we focus scaling the critic network, as prior studies indicate that scaling the ac- tor provides limited benefits (Nauman et al., 2024b; Lee et al., 2024c). We test two scaling approaches: width scaling by varying the critic's hidden dimension across {128, 256, 512, 1024], increasing parameters from 0.3M to 17.8M; and depth scaling by varying the number of critic blocks L across 21, 2, 4, 8), growing parameters from 2.2M to 17.8M.
Compute Scaling. We next explore compute scaling through increased UTD ratios, a key factor in improving sample efficiency in deep RL (Li et al., 2023). While higher UTD ratios can enhance sample efficiency, they also in- crease the risk of overfitting. Previous approaches address this through ensembling (Chen et al., 2021b), periodic reini- tialization (Lee et al., 2024a; D'Oro et al., 2023; Nauman et al., 2024b), or both (Kim et al., 2023). We investigate whether Simba V2's stable dynamics enable effective scaling without these additional mechanisms.
For compute scaling experiments, we vary the update-to- data (UTD) ratio across {1, 2, 8). We compare results both with and without periodic weight reinitialization, since prior work suggests that compute scaling requires periodic reinitialization to avoid overfitting (D' Oro et al., 2022). Following Nauman et al. (2024b), we apply reinitialization every 500,000 update steps when used.
Figure 7 shows the effect of varying the UTD ratio on DMC- Hard (left) and HBench-Haa (right), comparing Simba and V2 with and without reinitialization (solid lines: no reinitialization; dashed lines: reinitialization). In Simba, performance plateaus at a UTD ratio of on DMC-Hard and I on HBench- Hard. When combined with reinitialization, but further improves with reinitialization, consistent with
Parameter Scaling, Figure shows width scaling results on Dar-Hard (left) and HBench- Hard (right). Both Simba and Simba V2 benefit from larger models on DMC-Hard.
7

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforeement Learning
Table 1. Online RL. Average final performance after 1M environment steps, where + captures a 95% confidence interval (CI) computed over all raw benchmark samples. For algorithms with only average scores for each task available, we approximate the CI using these averages (t). Note that this estimation may be inaccurate. The highest performance is highlighted. Any performance that not statistically worse than the highest performance (according to Welch's t-test with significance level 0.05) is highlighted.
Mujoco (5)
DMC-Easy (21)
DMC-Hard (7)
MyoSuite (10)
HBench (14) Success.Nore
All (57)
Method
TD3.Norm
Return (Ik)
Return (IR)
Success Rate
(a) Low UTD 2)
PPO (Schulman et al. 2017)
0.447 1: 0.270
0.327 1: 0,1281
0.033 +: 0.030!
SAC (Huamoja al., 2018)
1.092 0.081
0.762 1 0.094
0.136 1 0.04
0.607 0.088
0.279 1 0.050
0.554 0.007
TD3 (Fujimoto al., 2018)
1.000 0.000"
TD3-OFE (Ota et al., 2020)
1.322 0.263'
TQC (Kuznetsov et al., 2020)
1.137 0.125"
'T17 (Fujimoto et al., 2023)
1.570 + 0.030
0.689
+ 0.144
0.182 + 0.197
0.356 = 126
0.289 +: 0.083
0.617 = 0 358
TD-MPC2 (Hansen et al., 2023)
1.040 0.115
0.889 0.064
0.465 +: 0.139
0.650 + 0.34M
0.710 + 0.149
0.749 =: 0.108"
CrossQ (Bhatt
2024)
1.475 0.141
MR.Q (Fujimoto et al., 2025)
1,448 0.56
0.868 0.006
0.723 0.061
BRO (Nauman et al., 2024b)
1.101 0.82
0,861
:: 0.006
0.693
1: 0.066
0.714
0.076
0,468 :: 0.107
0.731 = 0.039
Simba (Lee et al., 2024c)
1.147 1 0.077
0.864
1 0.024
0.706
1. 0.05
0.743 1 0.079
0.606 1 0.073
0.780 0.028
SimbaV2 (ours)
1.617 0.003
0.874
1 0.025
0.729
0.065
0.847
00066
0.776 1 0.064
0.892 0.052
(b) High UTD C
REDQ (Chen et al., 2021h)
1,160 0.01
DroQ (Hiraoka al., 2021)
1.134 0.70
V3 (Hafieer et al. 2023)
0.760 0.095
0.714 + 0.324*
0.009 o.mns?
0.482
0.366
0.022 + 0.113
0.397
0.289
MAD-TD (Voeleker et al., 2024)
0.708
+ 0.065
BRO (Nauman et al,, 2024b)
1.150 2.00000 0.202
0.871
# 0.084
0.767
:: 0.050
0.814
0.066
0.619 =: 0.117
0.807 = 0.037
Simba (Lee et al., 2024c)
1.175 0.136
0.866 1: 0.006
0.720 0.087
0.834
0.098
0.657 1: 0.099
0.818 0.043
SimbaV2 (ours)
1.598 0.176
0.876 0.005
0.769
0,866
00080
0.822 0.099
0.911
DMC-Hard
HBench-Hard
Table 1.(a) presents results at a UTD ratio below Sim- V2 with UTD=2 attains average normalized score of 0.892, exceeding the previous best of 0.780. Only except for DMC-Easy suite, Simba outperforms leading model-free (CrossQ (Bhatt et al., 2024), BRO (Nauman et al., 2024b), Simba (Lee et al., 2024c))) and model-based (TD-MPC2 (Hansen et al., 2023), MR.Q (Fujimoto et al., 2025)) base- lines, demonstrating superior sample efficiency.
0.80
1.00
Total Total Reward (K)
0.75
Noress-Norrsee
0.85
0.70
0.70
0.65
0.55
0.60
0.40
1
4 Ratio
2
4
8
UTD Ratio
SimbaV2
SimbaV2+Reset
Simba
Simba
Table 1.(b) evaluates higher UTD settings. Increasing Sim- ba's's UTD from 2 to 8 further elevates its average score from 0.892 to 0.911. SimbaV2 also surpasses BRO with UTD=10, which utilizes periodic reinitialization to avoid overfitting at high update rates. These consistent gains at larger UTD ratios underscore the efficacy of hyperspherical normalization in stabilizing training.
Figure 7. Compute Scaling. We scale compute by increasing the UTD ratio. We compare Simba and S2mbaV2, both with and with- out periodic reset. Simba saturates at lower ratios without reset, but improves with reset. contrast, Simha V2 scales smoothly even without reset, where using reset slightly degrades its performance.
D'Oro et al. (2023). In contrast, SimbaV2 scales consis- tently as the UTD ratio increases, even without reinitializa- tion. Notably, reinitialization slightly degrades SimbaV2's performance, as it disrupts training and adds time to recover.
For offline RL, we simply add behavioral cloning loss during training with using identical configurations to the online RL. Despite minimal changes. Simba V2 performs competitively with existing baselines (Appendix D).
To verify the importance of hyperspherical weight and fea- ture normalization for UTD scaling, we test variant in Appendix H.1 that includes distributional critics and re- ward scaling on Simba. This variant fails to scale at higher UTD ratios, confirming the critical role of hyperspherical normalization for effective scaling.
5.5. Design Study
Table 2 presents the results from ablation studies isolating the contributions of various architectural choices.
Input Projection. Projecting observations onto a hyper- sphere before passing them through the linear layer is crucial for performance (Table 2.(a)), where omitting this step leads to significant performance drop. Equally important de- sign is preserving the original magnitude during projection
5.4. Online RL
Having observed V2's scalability, we now compare it against standard model-free and model-based RI..
8

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Table Design Study. report the final performance for each design choice in the online RL benchmarks, averaged over 3 random seeds. Performance changes relative to the default SimbaV2 are highlighted according to their percentile difference: mild positive
changes [0.02, 0.05) and mild negative changes (-0.05, 0.02] are highlighted lightly, damaging changes (-0.1, 0.05] are highlighted
moderately, and catastrophic changes (-0.0- 0.1] are highlighted boldly.
Design
Mujoco (5)
Easy (21)
DMC-Hard (7)
MyoSuite (10) Success Rate
HBench (14)
All (57)
(idx) Original Changed
TD3.Nom
Return (1k)
Return (Ik)
Success.Norm
Input Projection
(a) L2 Normalize - No L2 Normalize
1,370 0.230
0.779 1 0,085
0,700 1 0.04
0.815
0.100
0.711 1 0.325
0.809
1: 0.059
(b) Shifting -> Shifting
1,406 0.0
0.771 1: 0.066
0.724
0.088
0.800
0.16
0.700
0.12
0.810
0.061
(0) Caieft 3 1
1,558 1.167
0.862 1 0.009
0.718 0.08
0.870 0.085
0.791 0.130
0.888
0.058
(d) Shift Projection -> Resize Projection
1.623 0.176
0.842 #: 0,043
0.720 0.053
0.852
0.083
0.779
+ 0.322
0.884
0.058
Output Prediction
(e) Categorical Loss MSE Loss
1.343 0.00
0.868 1: 0.004
0.708 0.07
0.757
0.767 :: 0.129
0.846
0.0062
(1) Reward Scaling - No Scaling
1.395 0.151
0.852 + 0.034
0,712 +: 0.092
0.840 0.077
0.735 # 0.085
0,852
#: 0.042
(g) Reward Bounding -> No Bounding
1.620 142
0.824
1: 0.013
0.733 0.072
0.805 0.13
0.787
:: 0.188
0.868 0.051
(h) Soft Target - Hard Tageet
1,589
0.175
0.878
0,087
0.746 0.081
0.848
0.096
0.770 0.094
0.890
0.051
Initialization & Update
(i) LR Decay -> No LR Decay
1,562 0.162
0.858 1. 0,042
0,719 1. 0.065
0.810 0.111
0.754 1 0.111
0,863 1.0.064
(i) Soast: : v2/v
1,571 1.10
0.873 +: 0.002
0.718 0.052
0.855 0.060
0.781
+: 0.004
0.890 0.083
(0) Sanate? //2//dh ++3
1.594 0:100
0.870
:: 0.055
0.706 0.055
0.836 0.05
0.789
:: 0.002
0.887 0.003
Oiiiit 1/(L + + 0.5
1,583 0.172
0.866 0,088
0.728 0.084
0.843 0.068
0.745 0.302
0.877 0.057
(m) 1/vdd ++1
1,520 0.1777
0.856
:: 0.094
0,714 0.089
0.875 0.079
0.792
#: 0.125
0.885 0.099
V2
1.617 0.133
0.874 + 0,025
0.729 0.064
0.847 0.066
0.776 +: 0.071
0.892 0.032
(Table 2.(b)). We also explore an alternative "resize" pro- jection, where inputs are first divided by Cshift dd, before being projected onto an (n + 1) --dimennsional hypersphere. The resize projection yields comparable performance as it can also retain magnitude information (Table 2.(d).
where sample efficiency is crucial, and extending it to model-based (Hansen et al., 2023) or visual RL (Kostrikov et al., 2020). Furthermore, with increasing interest in RL for training large language models (Ouyang et al., 2022; Guo et al., 2025), the potential benefits of using stricter normalization for large models remain an exciting open question.
Output Projection. Incorporating a distributional critic and reward scaling improves performance, especially in environments with high reward variance like MuJoCo (Ta- ble (f). Bounding target returns proves essential for easier tasks (Table 2.(g)), such as cartpole in the DMC- Easy suite (Table 25). Without bounding, consistent high returns can diminish return variance, and scaling returns push target values beyond the range of the distributional critic, leading to collapse in the TD loss.
Impact Statement
This paper presents work aimed at advancing the field of Machine Learning. There are many potential societal con- sequences of our work, none of which we feel must be specifically highlighted here.
Initialization & Update. Gradually decaying the learning rate is critical. Without decay, the model may struggle to re- fine its predictions during later training stages, as SimbaV2 maintains an effective constant learning rate throughout training (Table 2.))). Tuning initial scaler values has mini- mal impact on performance where the architecture remains stable by these changes (Table (m)).
A Acknowledgements
We would like to express our gratitude to Dongyoon Hwang and Hawon Jeong for their valuable feedback on this paper.
This work was supported by Institute for Information & communications Technology Planning & Evaluation (UIP) grant funded by the Korea government (MSIT) (RS-2019- 11190075, Artificial Intelligence Graduate School Program (KAIST)). This work was supported by the National Re- search Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2025-00555665 work was mainly supported by Institute of Information & communications Technology Planning & Evaluation (ITP) grant funded by the Korea government (MSIT) (No.RS- 2021 2021-1121206,, Artificial Intelligence Innovation Hub).
6. Lessons and Opportunities
Lessons. Historically, RL research has relied on complex regularizations to address overfitting and scalability issues (Klein al., 2024). Our findings suggest that suitably cho- constraints, exemplified by SimbaV2, simplify these design complexities while retaining strong performance.
This
Opportunities. Future opportunities include deploying SimbaV2 V2 in real-world robotics (Hwangbo et al., 2019),
9

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
References
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15844-15097, 2021a. (Cited on page 21)
Absil, P.-., Mahony, R., and Sepulchre, R. Optimization algoritiums on matrix manifolds. Princeton University Press. 2008. (Cited on page 5, 15)
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023. (Cited on page 1)
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. simple framework for contrastive learning of visual rep- resentations. In International conference machine learning, pp. 1597-1607. PMLR, 2020. (Cited on page 3)
Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. Ef- ficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023. (Cited on page 21)
Chen, X., Wang, C., Zhou, Z., and Ross, K. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021b. (Cited page 7, 8, 22)
Bellemare, M. G., Dabney, W., and Munos, R. A distribu- tional perspective on reinforcement learning. In Inter- national conference on machine learning, pp. 449-458. PMLR, 2017. (Cited on page 2. 4, 5)
Dehghani, M.. Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023. (Cited on page
Bhatt, A., Palenicek, D., Belousov, B., Argus, M., Ami- ranashvili, A., Brox, T., and Peters, Crossq: Batch normalization deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth In- ternational Conference Learning Representations, net / forum??
Do Carmo, M. P. and Flaherty Francis, J. Riemannian geometry, volume 2. Springer, 1992. (Cited on page 15)
2024. URL httttp:://ppereeeee id-Pc2OtIsTIX (Cited on page 1, 2, 6, 8, 22)
Dohare, S., Hernandez- Garcia, J. F., Rahman, P., Sutton, S., and Mahmood, A. R. Maintaining plasticity deep continual learning. arXiv preprint arXiv:2306.1380
Bjorck, N., Gomes, C. P., and Weinberger, K. Q Towards deeper deep reinforcement learning with spectral nor- malization. Advances in neural information processing systems, 34:8242-8255, 2021. (Cited on page 1, 2)
2023. (Cited on page 1)
D' D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022. (Cited on page
Bonnabel, S. Stochastic gradient descent riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217-2229, September 2013. ISSN 1558-2523,
doi: 10.1109/tac220133255460 URL http: / /dxx doi gttg/10..109/TAC.... 2013, 2254619. (Cited on page 16)
D' D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L.,, Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. The Eleventh International Conference on Learning Representations, 2023. (Cited on page 2, 7, 8)
Boumal, N. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023. (Cited on page 15)
Elsayed, M., Vasan, G., and Mahmood, A. R. Streaming deep reinforcement learning finally works. arXiv preprint 241:::2410.146066 2024. (Cited on page 2)
Brockman, G. Openai gym. arXiv preprint arXiv:1606.00 5400, 2016. (Cited on page 24)
Farebrother, J., Machado, M. C., and Bowling, M. Gen- eralization and regularization in dqn. arXiv preprint arXiv: 181000033,, 2018. (Cited on page 1,
Caggiano, V., Wang, H., Durandau, G., Sartori, M., and Kumar, V. Myosuite-a a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205. 200.2..5..00......... 13600, 2022. (Cited on page 2, 6, 24)
Farebrother, J., Orbay, J., Vuong, Q. Taiga, A. A., Cheb- otar, Y., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., et al. Stop regressing: Training value func- tions via classification for scalable deep rl. arXiv preprint N0V::240330395500 2024. (Cited on page 2)
Cai, T. T., Fan, J., and Jiang, T. Distributions of angles in random packing on spheres. Journal of Machine Learning Research, 14(136): (00):1877-1660 2013. (Cited on page 17)
10

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Feller, W. An introduction probability theory and its applications, Volume 2, volume 81. John Wiley & Sons, 1991. (Cited page 17)
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor- critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018. (Cited on page 2, 3, 6, 8, 22)
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. (Cited on page 2, 21)
Hafner, D., Pasukonis. J., Ba, J., and Lillicrap. T. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04 2023, (Cited on page 6, 8, 22, 38)
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021. (Cited on page 21)
Hansen, N., Su, H.,, and Wang, X. Td-mpc2: Scalable, bust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023. (Cited on page 6, 8, 9, 20, 22, 38)
Fujimoto, S., Hoof, H.,, and Meger, Addressing function approximation error in actor-critic methods. In Interna- tional conference on machine learning. pp. 1587-1596. PMLR. 2018. (Cited on page 6, 8, 22)
Hatsenn P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idgl: Implicit q q-learning as an actor- critic method with diffusion policies. arXiv preprint arXiv:2304.1057 2023. (Cited on page 21)
Fujimoto, S., Chang, W.-D., Smith, E. J., Gu, S., Precup, D., and Meger, D. For sale: State-action representation learning for deep reinforcement learning, arXiv preprint (rXiv:2306.0245/ 2023. (Cited on page 6, 8, 21, 22, 24, 38)
Hiraoka, T., Imagawa, T., Hashimoto, T., Onishi, T., and Tsuruoka, Y. Dropout q-functions for doubly efficient re- inforcement learning. arXiv preprint arXiv:2 10.02034, 2021. (Cited on page 2, 8)
Hussing, M., Voelcker, C. A., Gilitschenski, I., Farahmand, A.-.,, and Eaton, E. Dissecting deep rl with high update ratios: Combatting value divergence. In Reinforcement Learning Conference, 2024. (Cited on page 1)
Fujimoto, S., D'Oro, P., Zhang, A., Tian, Y., and Rabbat, M. Towards general-purpose model-free reinforcement learning. arXiv preprint 2xXv:2551116612. 2025. (Cited on page 6, 8, 22, 23, 38)
Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsou- nis, V., Koltun, V., and Hutter, M. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26))eaa5872. 2019. (Cited on page 9)
Gallici, M.. Fellows, M., Ellis, B., Pou, B., Masmitja, I., Foerster, J. N., and Martin, M. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.000 2024. (Cited on page 2)
Imani, E. and White, M. Improving regression performance with distributional losses. In International conference on machine learning, pp. on page 5)
Garg, D., Hejna, J., Geist, M., and Ermon, S. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02325 2023. (Cited on page 21)
PMLR, 2018. (Cited
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.088661 2020. (Cited on page 1)
Gemini, Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint urXiv:2312.1 1805, 2023. (Cited on page 1)
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamics of diffussin models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024, (Cited on page 3)
Gogianu, F., Berariu, T., Rosca, M. C., Clopath, C., Bu- soniu, L., and Pascanu, R. Spectral normalisation for deep reinforcement learning: an optimisation perspec- tive. In International Conference on Machine Learning, pp. 3734-373777 3744. PMLR, 2021. (Cited on page 2)
Kim, W., Shin, Y., Park, J., and Sung, Y. Sample- efficient and safe deep reinforcement learning via re- set deep ensemble agents. In Thirty-seventh Con- ference on Neural Information Processing Systems, 2023. URL https: openreview nnt/ forum? (d=bTidcHTKon (Cited on page 7)
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma. S., Wang, P., Bi, X., et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint S.ti::2501.1944 2025. (Cited on page 9)

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.699% 2014. (Cited on page 6)
Lee, H., Hwang, D., Kim, D., Kim, H., Tai, J. J., Sub- ramanian, K., Wurman, P. R., Choo, J., Stone, P., and Seno, T. Simba: Simplicity bias for scaling up param- eters in deep reinforcement learning. arXiv preprint arXiv:2410.09754. 10.09554, 2024c. (Cited on page 1, 2, 3, 6, 7, 8, 22, 23, 30, 38)
Klein, T., Miklautz, L., Sidak, K., Plant, C., and Tschi- atschek, S. Plasticity loss in deep reinforcement learning: A survey. arXiv preprint arXiv:2411.048.. 2024. (Cited on page 9)
Lee, J. M. Riemannian manifolds: introduction to cur- vature, volume 176. Springer Science & Business Media, 2006. (Cited on page 15)
Kodryan, M., Lobacheva, E., Nakhodnov, M., and Vetrov, D. P. Training scale-invariant neural networks on the sphere can happen in three regimes. Advances in Neural Information Processing Systems, 35:14058- 14070, 2022. (Cited on page 6, 28)
Lei Ba, J., Kiros, R., and Hinton, G. E. Layer normal- ization. ArXiv e-prints, pp. uIXiv-1607, (Cited on page 1)
Li, Q. Kumar, A., Kostrikov, I., and Levine, S. Efficient deep reinforcement learning requires regulating overfit- ting. arXiv preprint arXiv:2304.10466, 2023. (Cited on page 2,
Kostrikov, I., Yarats, D., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649 2020. (Cited on page 9)
Li, Concise formulas for the area and volume of a hyper- spherical cap. Asian Journal of Mathematics & &Satisticss, 4(1 4(1):66-77 :066-70,2206 2010. (Cited on page 17)
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint arXiv:2 10,06666,, 2021. (Cited on page 21)
Lillicrap, T. Continuous control with deep reinforcement learning. arXiv preprint on page 22, 30)
Krogh, A. and Hertz, J. A simple weight decay can im- prove generalization. Advances in neural information processing systems, 4, 1991. (Cited on page 2)
02971 2015. (Cited
Liu, W., Wen, Y., Yu, Z., Li, M., Raj. B., and Song, L. Sphereface: Deep hypersphere embedding for face recog- nition. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 212-2202202222212222 2017a. (Cited page 3)
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020. (Cited page 21)
Kumar, A., Agarwal, R., Ma, T., Courville, A., Tucker, G., and Levine, S. DR3: Value-based deep reinforce- ment learning requires explicit regularization. In In- ternational Conference on Learning Representations, 2022. URL ptttps://openrevee pnteeneviewwnee///eee tite/fffrm? id=POvMvLi91E (Cited on page 1)
Liu, W. Zhang, Y.-M., Li, X., Yu, Z., Dai, B., Zhao, T., and Song. L. Deep hyperspherical learning. Advances in neural information processing systems, 30, 2017b. (Cited on page 3)
Loshchilov, I., Hsieh, C.-P,, Sun, S., and Ginsburg, B. ngpt:
Normalized transformer with representation learning on the hypersphere. arXiv preprint 13X1v:2411.001111,, 2024. (Cited on page 2, 3, 16)
Kuznetsov, A., Shvechikov, P., Grishin, A., and Vetrov, D. Controlling overestimation bias with truncated mixture of continuous distributional quantile crities. In Interna- tional Conference on Machine Learning, pp. 5556-5566. PMLR, 2020. (Cited on page 6, 8, 22)
Lyle, C., Rowland, M., and Dabney, W. Understanding and preventing capacity loss in reinforcement learning. Proc. the International Conference on Learning Representa- tions (ICLR), 2022. (Cited on page
Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, Yun, S.-Y., and Yun, C. Plastic: Improving input and la- bel plasticity for sample efficient reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024a. (Cited on page 7)
Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural networks. Proc. the International Conference on Machine Learning (ICML), 2023. (Cited on page 1, 2)
Lee, H., Cho, H., Kim, H., Kim, D., Min, D., Choo, J., and Lyle, C. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. arXiv preprint arXiv:2406.00 02596, 2024b. (Cited on page 3)
Lyle, C., Zheng, Z., Khetarpal, K., Martens, J., van Hasselt, H., Pascanu, R., and Dabney, W. Normalization and effective learning rates in reinforcement learning. arXiv preprint arXiv:2407.01800 2024. (Cited on page 1, 6)
12

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Ma, G., Li, L., Zhang, S., Liu, Z., Wang, Z., Chen, Y., Shen, L.., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages, arXiv preprint arXiv:2310.07411 2023, (Cited on page 1)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022. (Cited on page 1)
Naik, A., Wan, Y., Tomar, M., and Sutton, R. S. Reward centering. arXiv preprint arXiv:2405. 0.w.0, 2024. (Cited on page 5)
Salimans, T. and Kingma, D. P. Weight normalization: simple reparameterization to accelerate training of deep neural networks. Advances in neural information pro- cessing systems, 29, 2016. (Cited on page 3)
Nauman, M., Bortkiewicz, M., Ostaszewski, M., Miloš, P., Trzciński,, T., and Cygan, M. Overestimation, overfitting. and plasticity in actor-critic: the bitter lesson of reinforce- 2024a.
Scannell, A., Kujanpää, K., Zhao, Y., Nakhaei, M., Solin, A., and Pajarinen, J. iqrl-implicitly quantized represen- tations for sample-efficient reinforcement learning. arXiv preprint arXiv:2406.02662 2266666.6 2024. (Cited on page 23)
ment learning. arXiv preprint (Cited on page 2)
Schaul, T., Ostrovski, G., Kemaev, L., and Borsa, D. Return- based scaling: Yet another normalisation trick for deep rl. arXiv preprint arXiv:2105.05347 2021. (Cited on page 5)
Nauman, M., Ostaszewski, M., Jankowski, K., Miłoš, P., and Cygan, M. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint 1rXiv:2405.16155 2024b. (Cited on page 3, 6, 7, 23)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017. (Cited on page 6, 8)
Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-..,, and Courville, A. The primacy bias in deep reinforcement learning. Proc. the International Conference Machine Learning (ICML), 2022. (Cited page 2)
Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency. International Conference on Machine Learning, pp- 30365-30380. PMLR, 2023, (Cited on page 2. 3)
Obando-Ceron. J.. Sokar, G., Willi, T. Lyle, C., Farebrother, J., Foerster, J., Dziugaite, G. K., Precup, D., and Castro, P. S. Mixtures of experts unlock parameter scaling for deep rl. arXiv preprint arXiv:2402.08609 2024. (Cited on page 2)
Sferrazza, C., Huang, D D.-M., Lin, X., Lee, Y., and Abbeel, P. Humanoidhench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403. 10506, 2024. (Cited on page 6, 25)
Ota, K., Oiki, T., Jha, D., Mariyama, T., and Nikovski, D. Can increasing input dimensionality improve deep reinforcement learning? In International conference on machine learning, pp. 7424-7433 PMLR, 2020. (Cited on page 6, 8, 22)
Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dor- mant neuron phenomenon in deep reinforcement learning. arXiv preprint arXiv:2302. 2023. (Cited on page 3)
Ouyang, L., WU, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2777 2022. (Cited on page 9)
Song. X., Jiang, Y., Tu, S., Du, Y., and Neyshabur, B. Observational overfitting in reinforcement learning, arXiv preprint arXiv:1912.02975 2019. (Cited on page 1, 2)
Spivak, M. D. A comprehensive introduction to differential geometry. (No Title), 1970. (Cited on page 15)
Palenicek, D., Vogt, F., and Peters, J. Scaling off- pol-policy reinforcement learning with batch and weight normaliza- tion. arXiv preprint arXiv:2502.07523, 2025. (Cited page 2)
Sutti, M. and Yuch, M.-H. Riemannian gradient descent for spherical area-preserving mappings. arXiv preprint M.XXv:2403.11220 2024. (Cited on page 16)
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-8831. Pmlr, 2021. (Cited on page 1)
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018. (Cited on page 1)
Tassa, Y., Doron, Y.. Muldal, A., Erez, T., Li, Y., Casas, D. d. L.,, Budden, D., Abdolmaleki, A., Merel, J., Lefrancq.
13

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
A., al. Deepmind control suite. arXiv preprint arXiv: 11011100100000 2018. (Cited page 2, 6, 24)
minimization. arXiv preprint arXiv:2310. 19668, 2023. (Cited page 3)
Todorov, E., Frez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. 2012 IEEE/RSJ in- ternational conference on intelligent robots and systems, pp. 0023-5033. IEEE, 2012. (Cited on page 24)
Xu, J. and Durrett, G. Spherical latent spaces for stable vari- ational autoencoders. arXiv preprint arXiv: 10805, 2018. (Cited on page
Zhou, Z., Peng, A., Li, Q., Levine, S., and Kumar, A. Effi- cient online reinforcement learning fine-tuning need not retain offline data. arXiv preprint arXiv:2412.0776,, 2024. (Cited on page 21)
Towers. M. Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krim- mel, M., KG, A., et Gymnasium: standard interface reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024. (Cited on page 24)
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017. (Cited on page 4)
Voelcker, C. A., Hussing, M., Eaton, E., Farahmand, A.- m., and Gilitschenski, I. Mad-td: Model-augmented data stabilizes high update ratio rl. arXiv preprint arXiv:2410.08896, 2024. (Cited on page 6, 8, 23)
Wang, F., Xiang, X., Cheng, J., and Yuille, A. L.. Norm- face: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1041-1049, 2017. (Cited on page 3)
Wang, T. and Isola, P. Understanding contrastive repre- sentation learning through alignment and uniformity on the hypersphere. In International conference on machine learning, pp. 0929-99399999 PMLR, 2020. (Cited page 3)
Wang, Z., Hunt, J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprini M2Xi::208.006933 2022. (Cited page 21)
Weisstein, E. W. Hypersphere. https://mawwwrll. wolfram. com/, 2002. (Cited on page 17)
Weisstein, E. W. Solid angle. https:/mattwwrlld wolfram. com/, 2005. (Cited page 17)
Willi, T., Obando-Ceron, J., Foerster, J., Dziugaite, K., and Castro, Mixture of experts in mixture of rl settings. arXiv preprint 24X.:24066.14200 2024. (Cited on page 2)
Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. On layer normalization in the transformer architecture. In Inter- national Conference on Machine Learning, pp. 10524- 10533. PMLR, 2020. (Cited page 2)
Xu, G., Zheng, R., Liang, Y., Wang, X., Yuan, Z., Ji, T., Luo. Y., Liu, X., Yuan, J., Hua, P., et al. Drm: Master- ing visual reinforcement learning through dormant ratio
14

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Appendix
A. Architectural Details
A.1. LERP: A Retraction-based Approximation on Riemannian Manifolds.
During the feature encoding stage in Simba V2, the input h and its non-linearly transformed output h are linearly interpolated using a learnable interpolation vector a E IRdh:
- E2-Norm((11 - @ h
(22)
followed by fg-normalization.
Intuitively, this can be interpreted as first-order (retraction-basee approximation of the Riemannian update formula on the hypersphere. This section provides brief introduction to the differential geometry concepts that underpin the Riemannian optimization perspective of a. For brevity, we omit the mathematical definitions, derivations, and proofs here. The comprehensive introduction to differential geometry and Riemannian optimization can be found in Spivak (1970), Do Carmo & Flaherty Francis (1992), and Boumal (2023).
Let - denote the n-dimensional hypersphere embedded in R2, i.c., S = {h E R 1h12211
Manifold. A manifold M of dimension is space that can locally be approximated by a Euclidean space R The simplest examples of manifold include the open ball U {az E R" ||ح|| r for Ryo, and the hypersphere -n is also a manifold in I
Tangent Spaces. At each point æ E M. the tangent space M is an n-dimensional vector space that locally approximates M near æ. Tangent vectors generalize the concept of directional derivatives. For the hypersphere Sn-1, the tangent space at a point p consists of all vectors orthogonal to p:
T--n-1 [h E IR" (p, h) = 0}
(23)
where (5-) denotes the Euclidean inner product.
Riemannian Metrics and Manifolds. The tangent space T&M is not inherently equipped with an inner product. A Riemannian metric P provides a collection of inner products P=(::-)- TaM x T.MM - IR on the tangent spaces, P == ( which locally define the geometry of M. Riemannian manifold (M,p) is a smooth manifold M equipped with such a metric. This enables us to define geometric notions such as distance, angle, length, volume, and curvature of manifold. For a detailed explanation of geometrics on Riemannian manifolds, refer to Lee (2006).
Exponential Mapping and Retraction. Under some conditions (Do Carmo & Flaherty Francis, 1992), the exponential map expx TaM + M can be defined at point E .M. exp=(v) maps tangent vector E M to a point on the manifold along the geodesic from æ in the direction of v. Therefore, for small E R, exp, (tv) represents the shortest path on M starting at æ with initial direction v. In Euclidean space (R"" I,), the exponential map exp, (v) = x + o is simply defined as a straight path. In practice, for computational efficiency (e.g., the mappings do not have closed-form), we often approximate the exponential map exp,, by retraction (Absil al., 2008) Ry:
Definition A.1 (Retraction). A retraction R on manifold M is a smooth map:
R:
U T&M + #EM
vx) + Rw(v)
with the following properties:
(0)
and (dR2)) = id
where Rs denotes the restriction of R M, (dRx) )o denotes the differential of Rg 0. and id is the identity map.
Intuitively, retraction (v) provides a first-order approximation of the exponential map exp (Boumal, 2023). Figure 8 illustrates the difference between the exponential map and retraction on Sy. For the hypersphere S, the retraction of a tangent vector & E Sn onto Sn is given by (Absil et al., 2008):
h+6 E
(x(E) Nn-Nomm + +E)
(24)
h + E
15

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
T S2
( expa (
S
Figure 8. Exponential Map Retraction 3-dimensional sphere. Comparison of the exponential map exp, and the retraction R. on the 3-dimensional sphere S2. The exponential mapping sends a tangent vector g E T2S2 exactly along the geodesic from æ to point the manifold, while the retraction locally approximates this mapping first order. Figure adapted from Sutti & Yueh (2024).
Riemannian Optimization. On Riemannian manifolds, gradient updates ideally follow the curved geodesics, rather than straight lines in Euclidean space. To this end, Bonnabel (2013) introduce Riemannian SGD that generalizes SGD to Riemannian manifolds using exponential map: + expr
(25)
where a > 0 is the global learning rate and g E M denotes the Riemannian gradient.
In our case, aase.-(h- h)) can be viewed as the gradient g in the Euclidean space. Then, we project the gradient onto the tangent space -=
Oproj 9 (g, hi2
(26)
-- -h h)) - --h h hi/h
(27)
-h (h, h
(28)
Applying the retraction exp, (-agrr)) Rn R=(-ooppo)):
+ (2-Norm
hh h a(h - (h, hith1) (1- (1-a(h,) (h, h)) + ah
(29)
=
Eg-Norm
(30)
Thus, the LERP operation in S2 can be interpreted rule on the hypersphere, where the learning rate is replaced by a learnable vector and the inner product (h, h) term is neglected. Also, Loshchilov al. (2024) empirically show that neglecting the inner product term has significant impact on performance.
iirraction-based approximation of the Riemannian update
16

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
A.2. Scaler Initialization
In our algorithm, the scaler RR a learnable vector that element-wise scales the output z of the linear layer:
Wh R
(31)
where W E Rd is the weight matrix of the linear layer, and h R" is the input vector. To ensure that (approximately) maintains unit norm at initialization, we initialize initialization.
2 V 1. The following section provides the derivation for this
We assume that each normalized embedding wt E R" W, and a random n-dimensional normalized vector h RR,, are uniformly distributed on the n-dimensional hypersphere Sn Furthermore, we assume that the vectors wi and mutually independent (Feller, 1991). We denote the angle between wy and h by 0,, such that cos 0: wy h since ||rr |||||2 1.
Distribution of the Cosine of the Angle. For simplicity, assume that w; is fixed. Since h is uniformly distributed on the hypersphere, the distribution of the angle 02 depends on the solid angle (Weisstein, 2005) subtended by h with respect to wy. The surface area An-2 an (n - 1)-dimensional hyperspherical cap (Li, 2010) leads to probability density function
f(0)) (Cai et al., 2013):
2x (-
f(0:)
An-1
T(
sin (0:))
(TT("=1) sin" (0))
I(
(32)
Sn-1
2x T(T)
where 0: E T), I' is the gamma function and S is the surface area of Sn-1
2/2 T1
(Weisstein, 2002).
Norm of Output Vector. Let @ Wh E d be the output of the linear layer. Each element of and s, denoted by z and 8), respectively, corresponds to the scaled cosine of the angle 02 between wy and h:
81 (wi h)) $2 W22 h)
81 cos 01
$2 cos 82
Wh
(33)
8d,, (wdr 12)
Sd,. cos Odn
The expected squared norm of z is then given by:
=|
Σ
s B/oos 0r]
(34)
Using the trigonometric identity c (0) l+cos(28) and the following integrals:
2
sin n - d0 I(())))(()
(
20)
(35)
r(
((
|0 cos(28) sin" n sin) dB =0
(36)
where the second integral vanishes due to the symmetry of cos(20) about v.
2. we compute the expectation:
E(cos?(BI)] = [-" cos²((1)
r(9)
sin" n sin) da,
(37)
f(Or)
cos (0t) 2-2 2 (0:
(38)
V p-1
r(
2 (
TI I
(39)
2 2 n-1
Thus, by setting / for all f E (1.-.., d we expect that the expected norm ||]|2||]] is 1 at initialization.
17

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
B. Implementation Details
Listings I, 2 and provide Google JAX implementation of scaling vector (Section 4.4), input embedding (Section 4. and MLP block (Section 4.2), respectively.
import flax.linen as nn
class Scaler (nn. ;
dim: int
init: float
scale: float
def setup ( = selt.param(
nn. mn.initializers consiiiane.constatt constant (1.0 self. scale), self.dim,
13
self. worward_scalee self.init / self.scal
14
def call casll_ssef,, x: inp inarray)) -> inp. :darray: return self.scaler folf.forward_scaler
Listing I. A JAX implementation of Scaler (Section 4.4)
1. import as jnp
import linen as on
class InputAmbeddi ng (onn.
5
observaticn_dim int
6
hidden_dim: int
Chiff_ccnstt float
input_ caaut_scaler_iiiie float input_scaler_scall float
9
10
def sctup (self): self.ob cbs_rms RunningMeanStd laap==seff.observattee
15
self.wo wo = nn. Densel
features-se1f h....iioee wtw.edffhhddddiiiiii
bie_bias=aalse
19
self. imput scaler Scaler(
20
din-self. obbrsslf..bbeerraiionn.....
inir-input_scaler_int, scale=inpur_eaale_eeall
23
24
def __all__see,,, observation: jop.ndarray) -> jnp. ndarray:
RSNorm
27
(observations
rms.rmmmmma.. / jap.sqrt
28
sclf.obs_rms rms. var + self.epsilon
29
Shift + 12-Norm
new_axis jnp.ones (o. shape[:-1] (1,11) self.shift_comm
32
o jnp. concatenate (lo, new new_axis], axis=-1)
33
o = 12normalize (o, axis=-1)
# Linear + Scaler
h self.w0
self. inattslert_sclerrr (h)
37
h (hh axis=-1)
38
return h
Listing JAX implementation of Input Embedding (Section 4.1).
18

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
import flax.linen as nn
class Simbav281 ock (nn. (nn.Module)
hidden_dim: int
5
ffn_scaler_ini:: float ffn sccasr_sssaaeeeeee float
6
Alpha_scaler_iie float alpha_scaler__calee float
10
def sctup (sclf):
solf.wl = nn. [
use_bias-False
(x)
15
self.mip self.mup_scaler = (caler(
sim==+seeffhhidee dim,
init, scale=ffn_ccalerssclle
19
20
self.w2 - nn (ense
features=self. hi hidden dim, bias=False
23
24
self.alpha Scaler
25
fin=sclffhidden_.e Init=alpha_scaler__te
gaale-alpha_scalerr.
28
29
def calllll (se___ x: iny ndarray) -> inp. ndarray:
residual MLP 12-Norm
32
35
x self.wl(x)
34
X self.min self.mlp_scaler (x)
rn.reluxx)
self.w2 (x)
37
x 12nermalize axis=-1)
38
# LERP 12-Norm
= 12nermalize (residual + sllf. (x residual), axis=-1)
return
Listing 3. A JAX implementation of MLP block (Section 4.2).
19

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
C. Hyperparameters
For all experiments, we use consistent hyperparameters across benchmarks. The default settings are listed Table 3.
Table 3. Hyperparameters Table. The hyperparameters listed below are used consistently across all tasks using Simba 22m unless stated otherwise. For the discount factor 7, we set automatically using heuristies used by TD-MPC2 (Hansen al., 2023).
Hyperparameter
Notation
Value
Input
Shift constant
Caift
3.0
Number of return bins
Patons
101
Output
Support return
Guin; Gmax]
[-5. 5]
Reward scaler epsilon
1e-8
Input scaler
(ffrinis sestes
(v2/ Vdd /2)
MLP scaler
(s scate
(2/v4d,, v2/v/da)
Training
Outpur scaler
(o,inits So,scale)
(v2/vdn, v2/v
LERP vector
Olias Okscate)
(1/(L + 1), 1/1/d
Behavior cloning weight
X
Online: 0.0
Offline: 0.1
Discount factor
7
Heuristic (Hansen al., 2023)
Replay buffer capacity
1M
Buffer sampling
Uniform
Common
Batch size
256
Update-to-data (UTD) ratio TD steps
&
Number of blocks
L
Hidden dimension
da
128
Actor
Initial lemperature
010
le-2
Target entropy
[A]/2
Number of blocks
L
2
Hidden dimension
da
512
Critic
Number of atoms
101
Target critic momentum
5e-3
Clipped double
Has Failure Termination (Mujoco, HBench): True
No Failure Termination (DMC, MyoSuite): False
Optimizer
Adam
Optimizer momentum
(21,82)
(0.9, 0.999)
Optimizer
Weight Decay
0,0
Learning rate init
"
1e-4
Learning rate final
3e-1
20

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
D. Offline RL
In this section, we assess whether the Simba V2 architecture also provide benefits offline RL, training from a stationary distribution. We adopt the minimalist offline RL method from (Fujimoto & Gu, 2021), where the behavioral cloning loss integrated into the reinforcement learning objective. The objective is defined as:
π R arg max ((s,a> [Q(s, (s)) | (())]] (a(s) - a)
(40)
where we used 0.1, as in (Fujimoto et al., 2023), and no parameter tuning is performed.
D.I. Experimental Setup
Environment. We use MuloCo tasks from the D4RL (Fu et al., 2020) benchmark, covering 3 environments (HalfCheetah, Hopper. Walker2d) and 3 difficulty levels (Medium, Medium-Repe -awuummm-e...-.aeee Medium-Expert),
Baselines. We compare SimbaV2 against standard offline RL methods: Percentile BC, Decision Transformer (DT, (Chen et al., 2021a)). Diffusion Q-Learning (DQL, (Wang et al., 2022)), Implicit Diffusion Q-Learning (IDQL, (Hansen-Esstuuch et al., 2023)), Conservative Q-Learning (CQL,, (Chen et al., 2021a)), TD3+BC (Fujimoto & Gu, 2021), Implicit Q-Learning (IQL, (Kostrikov et al., 2021)), Extreme Q-Learning (X'QL, (Garg et al., 2023)), and TD7+BC (Fujimoto et al., 2023).
The results for Percentile BC, DT, DQL, and IDQL is from "Hansen-Estruce et al., 2023), while CQL, TD3+BC, IQL, X-QQ,, and TD7 results come from (Fujimoto et al., 2023).
Metries. Following the standard offline RL protocol (Fu et al., 2020), normalize the score of each environment based on the expert trajectory in the dataset.
Training. We use the same training configuration as in online RL (Appendix C), with learning rate decaying linearly from 1 x 10-Ato to x 10-5 over 100 epochs, and include an additional behavioral cloning loss.
D.2. Results
Table 4 reports the performance of Simba V2 + BC, averaged over 10 random seeds.
Table 4. Offline RL. Average final performance on the D4RL mujoco benchmark, averaged over 10 trials. Methods are listed in chronological order. For fair comparison, we used a unified hyperparameter configuration for each method. The highest performance highlighted. Any performance that is not statistically worse than the highest performance (according to Welch's f-test with significance level 0.05) is highlighted. The environment's 's average variance was used for the statistical test methods without reported variance.
HalfCheetah
Hopper
Walker2d
Average
Method
m
m-f
m=2
CQL (Kumar et al., 2020)
46.7=003
45.5
76.8=7.4
59.3
78.8 18.8
79.9=1988
$1.4=117
79.9
108.5=122
73.0=2.7
Percent BC (Chen et al., 2021a)
48.4
40.6
92.9
56.9
75.9
110.9
75.0
62.5
109.0
74.0
DT [Chen et al., 2a)
42.6
36.6
86.8
67.6
82.7
110.9
74.0
66.6
108.1
74.7
TD3+BC (Fujimoto & Gu, 2021)
48.1=003
44.6=04
93.7=099
59.1=330
52.0=106
1107
84.3-00
81.0#34
110.5
74.6
IQL (Kostrikov et al., 2021)
47.4=00
43,9=1.3
89.6-1.5
63,9:49
93.4=7.
64.2-32.
84.2=1.6
71.2+3.
108,9:144
74.1=33
(Wang et al., 2022)
50.6
45.8
75.2
94.5
102.1
83.4
86.7
109.6
82.4
X-QL (Garg al., 2023)
47.4=003
44,2
90.2=1.7
67.7=36
82.0=149
92.0=1000
79.2=440
61.877
110.3=02
75.0=23
IDQL (Hamsen Estruch el al., 2023)
49.7
45.1
94.4
63.1
82.4
105.3
80.2
79.8
111.6
79.1
TD7+BC (Fujimoto ot al.,, 2023)
58.0
53.8 L0.5
104.6 11
76.1
91.1180
108.2 108.2
91.1 1778
89.7
111.8 10.
87.2
SimbaV2-BC (ours)
54,8
48.6
92.2-1.4
98.1 224
99.9 00.5
106.2
12.7=103
87.7
110.6.00
86.7
With minimal changes, SimbaV2 performs highly competitively with existing offline RL algorithms, with statistically significantly better performance on Hopper. Again, this experimental results reinforces the importance of architectural design over complex algorithmic modifications. We believe our architectural approach offers exciting future potential for bridging offline and online RL (Ball et al., 2023; Zhou et al., 2024).
21

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
E. Baselines
PPO (Lillicrap, 2015). Proximal Policy Optimization (PPO) is an on-policy policy gradient method that constrains updated policies to remain proximal to the old policies to circumvent performance collapse. Results for Gym MMJoCo and DMC were obtained from Fujimoto et al. (2025), which are averaged over 10 seeds.
SAC (Haarnoja et al., 2018). Soft Actor-Critic (SAC) is off-policy actor-critic algorithm in which the actor simultaneously maximizes expected return and entropy, encouraging both stability and exploration. For the MuloCo tasks, results averaged over 10 random seeds were obtained directly from the Bhatt et al. (2024) authors, with the update-to-data (UTD) ratio set to 1. For DMC, MyoSuite, and HBench tasks, we use the results from Lee et al. (2024c) which were obtained by running the official repository for 10 random seeds, with the update-to-data (UTD) ratio set to 2.
TD3 (Fujimoto al., 2018). Twin Delayed DDPG (TD3) is an off-policy actor-critic algorithm that mitigates Q- overestimation bias via three key techniques: (i) clipped double Q-learning, (ii) delayed policy updates, (iii) target policy smoothing. Results for Gym-MuJoCo were obtained from Table 1 of Fujimoto et al. (2023). These scores are averaged over 10 random seeds.
TD3+OFE (Ota et al., 2020). By replacing the encoder with an Online Feature Extractor (OFE) via dynamics prediction task to produce high-dimensional representations of observation-aatioon pairs- TDS3-FFE outperforms the original TD3 without requiring any hyperparameter adjustments. Results for Gym-MuJoCo were obtained from Table 1 of Fujimoto al. (2023). These scores are averaged over 10 random seeds. We attach these results into Table 1 by TD3-normalizing the scores outlined in Appendix F.1.
TQC (Kuznetsov et al., 2020). Truncated Quantile Critic (TQC) proposes to truncate the return distribution of the distributional crities to flexibly balance between under- and overestimation bias of Q-value, Results for Gym-MuJoCo were taken directly from Table 1 of Fujimoto et al. (2023). We attach these results into Table 1 by TD3-normalizing the scores as described in Appendix F.I.
REDQ (Chen et al., 2021b). Randomized Ensembled Double Q-Learning (REDQ) expands clipped double Q-learning from two Q-networks an ensemble of ten control estimation bias and variance, and enhance training stability. For the MuJoCo tasks, results averaged over 10 random seeds were obtained directly from the Bhatt et al. (2024) authors, with the update-to-dee -o-tt- (UTD) ratio set to 20.
DroQ (Chen et al., 2021b). Dropout Q-Function (DroQ) reduces the computational burden of REDQ by using a smaller ensemble of Q functions while employing Dropout and Layer Normalization to stabilize training against Dropout-induced noise. For the MuJoCo tasks, results averaged over 10 random seeds were obtained directly from the Bhatt et al. (2024) authors, with the update-to-data (UTD) ratio set to 20.
DreamerV3 V3 (Hafner et al., 2023). DreamerV3 encodes sensory inputs into categorical representations to build learned world model, enabling long-horizon behavior learning in its compact latent space. Results for Gym-MuJoCo and DMC were obtained from Fujimoto et al. (2025), which are averaged over 10 seeds. For MyoSuite, and HBench tasks, we use the results from Lee et al. (2024c) which were obtained by running the official repository (https: / /gthtb..bbb/.gg////////..g.thh.t com/ SonyResearch/ s imba) over random seeds.
TD7 (Fujimoto al., 2023). TD7 improves TD3 by combining TD3 with four key improvements: (i) state-action representation learning (SALE), (ii) prioritized experience replay, (iii) policy checkpoints, and (iv) additional behavior cloning loss for offline RL. Results for Gym-MuJoCe and DMC were obtained from Fujimoto et al. (2025). which are averaged over 10 seeds. For MyoSuite, and HBench tasks, we use the results from Lee et al. (2024c) which were obtained by running the official repository (https: / / h//ggittuuu thub. com/ SonyRssearch/ s simba) over 5 random seeds.
TD-MPC2 (Hansen et al., 2023), TD-MPC2 a model-based algorithm that learns an implicit (decoder-free) world model through multiple dynamics prediction tasks and performs local trajectory optimization within the learned latent space. Results for Gym-MuJoCo and DMC were obtained from Fujimoto et al. (2025), which are averaged over 10 seeds. For the results from Lee et al. (2024c) which were obtained by running the official repository (https:/ / htggttttt....ogoggg SonyReseaachh simba) over 3 random seeds.
MyoSuite, and HBench tasks,
CrossQ (Bhatt et al.. 2024). CrossQ achieves superior performance and sample efficiency with low replay ratio, by removing target networks and employing careful batch normalization. Results for Gym-MuloCo were obtained by running the official repository (ht (https:/ / /github. com/ httm///dtytaa//rrrreee for 10 random seeds,
22

---

update-to-datte SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
iQRL (Scannell al., 2024). Implicitly Quantized Reinforcement Learning (iQRL) a representation learning technique of model-free RL that prevents representation collapse and improve sample-effiiency via latent quantization. For the DMC hard tasks, results averaged over 3 random seeds were obtained directly from the authors.
BRO (Nauman et al., 2024b). Bigger, Regularized, Optimistic (BRO) scales the critic network of SAC by integrating distributional Q-learning, optimistic exploration, and periodic resets. Results for Gym-MuloCo and DMC Easy were obtained by running the official repository (https: / gii hub. com /ttpammmee/iiieeeguluuuuiieieee random seeds. For DMC hard, MyoSuite, and HBench tasks, we use the results from Lee al. (2024c) which were obtained by running the official repository (https: / /ggtthb.....///// com/ /ww///eeareee///eeeeeee simba over 5 random seeds for HBench tasks and 10 random seeds for DMC hard and MyoSuite tasks. Unless stated otherwise, (UTD) ratio to be 2.
Amistic) for
MAD-TD (Voelcker et al., 2024). Model-Augmented Data for Temporal Difference learning (MAD-TD) aims to stabilize high UTD training by mixing a small fraction a of model-generated on-policy data with real off-policy replay data. For the DMC hard tasks, results averaged over 10 random seeds were obtained directly from the authors using the best algorithm setting (UTD- 8, a 0.05).
MR.Q (Fujimoto et al., 2025). Model-based Representations for Q-learnngg (MR.Q) is model-free algorithm that uses model-based objectives, such as dynamics and reward prediction, to obtain rich representation for actor-critic agent. We use the results for Gym-MuJoCo and DMC from Fujimoto et al. (2025) which were obtained by running the official repository over 10 random seeds.
fottps://ge / http:/hc...cc//ccooooooeeeeeeeee http://acboooerrrrccccccaceeee
Simba (Lee et al., 2024c). SimBa is an architecture designed to scale up parameters in deep reinforcement lcarning by injecting simplicity bias with observation normalizer, residual blocks, and layer normalizations. For Gym-MuloCo, DMC, MyoSuite, and HBench tasks, we use the results from Lee et al. (2024c) which were obtained by running the official repository (https:/ bettps://githbb //tpttut.b...//////// com/ Seem//onnneeeeecbb//.. simba) over 15 random seeds for DMC hard tasks and 10 random seeds otherwise. Unless stated otherwise, we set update-to-date (UTD) ratio to be 2.
23

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Table Environment details. We the episode length, action repeat each domain, total environment steps, performance metrics used for benchmarking V2.
Gym
DMC
MyoSuite
HumanoidBench
Episode length
1,000
1, 1,000
100
500 1,0
Action repeat
1
2
2
Effective length
1, 1,000
500
50
250 - 500
Total env. steps
1M
1M
1M
1M
Performance metric
Average Return
Average Return
Average Success
Average Return
F. Environment Details
This section outlines the benchmark environments used our evaluation. complete list of all tasks from each benchmark, including their observation and action dimensions, is provided the end of this section. Additionally, Table 5 outlines the episode length, action repeat, total number of environment steps, and performance metrics for each task domain.
F.1. Gym MuJoCo
Gym (Brockman, 2016; Towers al., 2024) is suite of benchmark environments spanning finite MDPs Multi-Joint dynamics with Contact (Todorov et al., 2012, MuJoCo) simulations. It offers diverse range of tasks, including classic Atari games, small-scale tasks such as Toy Text and classic controls, well as physics- continuous robot control. For our experiments, we focus on 5 locomotion tasks within MuJoCo environments, which simulate complex physical interactions involving multi-body dynamics and contact forces. A complete list of these tasks is provided Table Note that we use the v4 version.
For comparison across different score scales of each task, all MuJoCo scores are normalized using TD3 and the random score for each task, provided in TD> (Fujimoto al., 2023).
random score
TD3-Normalized((a)
TD3 score random score
Task
Random
TD3
Ant-v4
-70.288
3942
Hal Fhheehhet
-289.415
10574
Hopper-v4
18.791
3226
Humanoid-v4
120.423
5165
Malker2d-v4 2.791
3946
F.2. DeepMind Control Suite
DeepMind Control Suite (Tassa al., 2018, DMC) is standard continuous control benchmarks, encompassing a variety of locomotion and manipulation tasks with varying levels of complexity. These tasks range from simple low- dimennioonl settings (O E R A E I to highly complex scenarios (O E 223 A R-39). Our evaluation includes 27 DMC tasks, divided into two categories: DMC-Easy&Mediinn and DMC-Hard. All Humanoid and Dog tasks are grouped as DMC-Hard, while the rest are fall under DMC-Easy&Mediae Medium. Comprehensive lists of DMC-Easy&Medium and DMC-Hard are available in Tables 7 and 8, respectively.
F.3. MyoSuite
MyoSuite (Caggiano et al., 2022) models human motor control using musculoskeletal simulations of the human elbow, wrist, and hand, focusing on physiologically accurate movements. provides benchmarks for intricate real-world object manipulation, ranging from simple posing tasks the simultaneous manipulation of two Baoding balls. Our evaluation
24

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
focuses 10 MyoSuite tasks involving the hand. As defined by the authors, each task is categorized as hard when the goal is randomized; otherwise the goal is fixed. The full list of MyoSuite tasks is presented in Table 9.
F.4. HumanoidBench
HumanoidBench (Sferrazza al., 2024) serves high-dimensional simulated robot learning benchmark, leveraging the Unitree HI humanoid robot equipped with dexterous hands. It encompasses diverse set of whole-body control tasks, spanning from fundamental locomotion complex human-like activities that require refined manipulation. In our experiments, we concentrate on 14 locomotion tasks. A comprehensive list of tasks is provided in Table 10.
Note that the locomotion tasks do not necessitate hand dexterity. Therefore, to reduce the complexity arising from high degrees of freedom (DoF) and complex dynamics, we streamline the environments setup by excluding the hands of humanoid. For example, in case of 1k, this drastically declines the dimension of the observation and action spaces by approximately 66%.
walk
Without hand
With 2 hand
Observation dim |0|
151
Action dim |AA| DoF (body) DoF (two hands)
19
25
50
For comparison across different score scales of each task, all HumanoidBench scores are normalized using each task's 's target success score provided by the authors and random score. Random scores are measured by the average undiscounted returns over 10 episodes of random agent. Each measurement is repeated over 10 seeds.
random score
Success-Normalize((a)
Target success score random score
Task
Random
Target Success
h1 si-balance-siiple
9.391
800
h1 h1-ballanc--hard
9.044
800
hl-crawl
272.658
700
h1 h1-hurdle
2.214
700
h1 -maze
106.441
1200
h1 h1-pole
20.09
700
h1-res aelreacch
260.302
12000
hl-run
2.02
700
h1-sit h1-sit-simple
9.393
750
h1 h1-sit-hard
2.448
750
h1-slide
3.191
700
hl-stair
3.112
700
h1-stand
10.545
800
h1-walk
2.377
700
25

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Table Gym-MuJoCo. We evaluate total continuous control tasks from the Gym-M MuloCo benchmark. Below, provide a list of all the tasks considered. The baseline performance for each task is reported at 1M environment steps.
Task
Observation dim |
Action dim |A|
Ant-v4
27
halfChhetah-v4
17
Hopper-v4.
11
Humanoid-v4
376
Walker2d-v4
17
Table 7. DMC-Fasy Complete List. We evaluate a total of 21 continuous control tasks from the DMC-Fasy benchmark. Below, we provide all the tasks considered. The baseline performance for each task reported at 1M environment steps.
Task
Observation dim |O|
Action dim [AA]
acrobot-swingu
6
ball-in-cup-catchh
6
cartpole-ballacce
5
cattpole-ballnne-spaaee
5
cartpole-swingup
5
cartpolee-sinngppp-ppaae
5
cheetah-run
17
finger-spin
9
2
finger-turn-easy
12
2
finger-turn-hard
12
2
fish-swim
24
5
hopper-hop
15
4
hopper-stand
15
pendulun-swwingup
3
1
quadruped-run
78
12
quadruped-wal
78
12
reacher-easy
6
2
reacher-hard
6
2
walker-zun
24
6
walker-stand
24
6
walker-wall
24
6
Table 8. DMC-Hard Complete List. We evaluate total of 7 continuous control tasks from the DMC-Hard benchmark. Below, we provide a list of the tasks considered. The baseline performance for each task reported at 1M environment steps.
Task
Observation dim |0|
Action dim |A]
dog-rum run
223
38
dog-trot
223
38
dog-stand
223
38
dog-walk
223
38
humaroid-run
67
24
humanoid-stand
67
24
humanoid-walk
67
24
26

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Table 9. MyoSuite Complete List. evaluate total of 10 continuous control tasks from the MyoSuite benchmark including both fixed-goal and randomized- goal (harc) settings. Below, we provide a list of all the tasks considered. The baseline performance for each task reported at 1M environment steps.
Task
Observation dim
|
Action dim |A|
myo-key-turn
93
39
myo-key-turn-bard
93
39
myo-obj-hold
39
myo-obj-hold-haard
39
nyo-pen-twirl
83
39
myo-pen-twirllhard
83
39
myo-pose
108
39
myo-pose-hard
108
39
nyo-reach
115
39
myo-reach-hard
115
39
Table 10. HumanoidBench Complete List. evaluate total of continuous control locomotion tasks from the HumanoidRench benchmark that simulates the UniTree HI humanoid rohot. Below, provide list
of the tasks considered. The baseline performance for each task is reported at 1M environment steps.
Task
Observation dim
Action dim
|A|
h1 m1-balance-hare
77
19
h1 al-balance-simee
64
19
h1-crawl
51
19
h1 h1-hurdle
19
hl-maze
51
19
h1 -l-pole
51
19
h1 hl-reach
57
19
h1 h1-run
51
19
h1 bl-sit-simpe
51
19
h1-sit-hard
64
19
h1-slide
51
19
h1-stair
19
h1 stast
51
19
h1-waik
51
19
27

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
G. Training Stability
In Section 5.2, we investigated the training dynamics of Simba V2 on DMC-Hard and HBBncch-Hrr via four metrics: feature norm, parameter norm, gradient norm, and effective learning rate (ELR) of neural networks. This section presents these standalone metries for S2mbaV2 to highlight its stable behavior throughout training.
Effective Learning Rate. We base our notion of ELR on the effective step size of Kodryan al. (2022), omitting the
global learning rate 17 and using dimension-bare based weighting wy 9.
instead of gquaree-parameter-nne weighting 02)
|0.1 W/ 35 10
Definition G.1 (Effective Learning Rate), Let 0
{0. be the parameter set of neural network, and gi be the back-propagated gradient associated with 0,. The (total) effective learning rate ELR of the network is defined as:
N
g 1?
ELR
A
Wi
0.11
(41)
where w, = Σ | Intuitively, our ELR measures the "effective" gradient step per parameter dimension-bettee scaled by the global learning rate.
10.
Metries. To reflect dimensional contributions across layers, we also apply the same weighting wy when computing the feature norm, parameter norm, and gradient norm. For instance, our gradient norm defined as:
N
||
w. w
(42)
where || -- ||2 is the standard f--norm (Frobenius norm || ||| in case of matrices). Analogous expressions are applied for feature and parameter norms. We separate encoder layers (all layers preceding the output) from predictor layers (all layers after) to capture their distinct roles in the network. Average returns are normalized by maximum score 1000 for DMC-Hard, by success and random scores for HBench-Hard (Appendix F.4).
Results. Figure 9 shows the tracked metrics over 1 million training steps. Certain features (e.g., logits) and parameters (e.g.. scalers and interpolation vectors) may occasionally exceed unit norm, the overall parameter norms are tightly controlled (Figure 9.(b)-c)), and gradient magnitudes are consistently balanced across modules (Figure 9.(d). This leads to the consistent trend and scales of their ELRs over time. We hypothesize that this stable behavior contributes to SIMBAV2's
improved performance and scalability.
(a) Average Return
(b) Feature Norm
(c) Parameter Norm
(d) Gradient Norm
Effective LR
x10
x10
DMC-Hard
1.00
40
40
1.00
30
35
0.75
3
0.50
20
30
0.50
2
0.25
10
25
1
0.00
20
0.01
0.2
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
0.4
0.6
0.8
0.2 0.4
0.6
0.8 1.0
0.2 0.4 0.6 1.0
x10
#Bench-Hard
1.00
40
60
10.00
0.75
30
50
7.50
3
0.50
20
40
5.00
2
0.25
10
30
2.50
1
0.00
20
0.01
0.2 0.4 0.6 0.2 0.4 0.6 0.8 1.0
0.8
1.0
0.2 0.4 0.6 0.8
1.0
0.2 0.4 0.6 1.0
0 0.2 0.4 0.6 0.8 1.0
Env steps (M)
Env steps (M)
Env steps (M)
Env steps
Env steps (M)
SimbaV2
Simbav2 Encoder
SimbaVZ Predictor
Figure 9. Si Training Dynamies. We track 4 metrics during training to understand the learning dynamics of SimbaV2: (a) Average normalized return across tasks. (b) Weighted sum forms of all intermediate features in critic. (c) Weighted sum of fo-nss of all critic parameters (d) Weighted sum of fz-norms of all gradients critic (e) Effective learning rate (ELR) of the critic. On both environments, V2 maintains feature and parameter norms aligned, producing consistent gradient norms and ELRs.
28

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
H. Additional Experiments
This section complements Section 5.2 by presenting further experiments probing the properties and robustness Simba V2:
Scalability Effect of Hyperspherical Normalization (Section H.1). Investigate the necessity of hyperspherical normal-
ization for achieving Simba V2's scalability.
Effectiveness beyond SAC (Section H.2.) Assess Simba V2's broader applicability by substituting SAC with DDPG.
H.I. Scalability Effect of Hyperspherical Normalization
In Section 5.3, we observe that Simba V2 consistently scales with an increasing update-to-data (UTD) ratio, even with- out reinitialization, while Simba saturates at a ratio of 2. However, this raises the question of whether hyperspherical normalization critical for UTD scaling. This section investigates the effectiveness hyperspherical normalization in scalability.
Experimental Setup. In this experiment, examine "Simba-like" variant, named Simba+, which incorporates distribu- tional critic and reward scaling but only excludes the hyperspherical normalization. In other words, Simba+ is identical to V2 except that it excludes hyperspherical normalization. On DMC-Hard tasks, we compare SimbaV2, Simba, and Simba+ under varying model sizes and UTD ratios to determine the role of hyperspherical normalization in scaling performance.
Result. Figure shows the scaling results. In Figure 10 (left), all three methods benefit from increased model capacity, but Simba+ slightly underperforms at larger parameter counts. More critically, Simba and Simba+ both plateau when the UTD ratio surpasses 2 in Figure 10 (right), while V2 continues to improve. These results confirm that hyperspherical normalization is truly indispensable for UTD scaling.
0.80
Total Total Reward (K)
0.75
0.62
0.75
0.50
0.38
0.65
0.25
0.60
0.3
1.1
4.5
17.8
1
2
8
Parameters (M)
UTD Ratio
Simbav2
Simba+
Simba
Figure 10. Performance Scaling under DMC-Hard. We compare Vi, Simba+, and Simba as scaling the number of model
parameters by increasing the critic network width and UTD ratio. Simba+ fails to scale effectively at higher UTD ratios, highlighting the essential role of the hyperspherical normalization for scalability.
29

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
H.2. Effectiveness beyond SAC
In Section 5.2, we observe that replacing the neural network of SAC with V2 consistently improves the performance of wide range of domains. To assess the broader applicability and robustness of SimbaV2's architectural advantages beyond single algorithm, conducted additional experiments using Deep Deterministic Policy Gradient (Lillicrap, 2015, DDPG), another widely adopted off-polis, algorithm for continuous control.
Experimental Setup. We evaluated SimbaV2 against the original Simba (Lee et al., 2024c) and a standard MLP baseline on two challenging continuous control benchmark suites: DMC-Hard and HBench-Haad Hard. All methods utilized the DDPG algorithm as their underlying learning framework. The MLP architecture we adopted consists of sequence of linear layers followed by ReLU non-linearities.
Result. Comparative results are presented in Figure 11. On the DMC-Hard benchmark, Simba V2 achieved performance competitive with the original Simba, with both significantly outperforming the MLP baseline. More notably, on the more complex HBench-Haad benchmark, Vi demonstrated a clear improvement over Simba. These results indicate that S2 not only generalizes to the DDPG algorithm but also exhibits enhanced stability and generalization capabilities in more demanding environments, likely attributable to its refined architecture and regularization mechanisms.
DMC-Hard
HBench-Hard
Ducess-Norm Score
Total Reward (K)
0.54
0.54
0.36
0.36
0.00
0.00
0.2
1.0
Env steps (M)
Env steps (M)
Simbav2 + DDPG
Simba + DDPG
MLP + DDPG
Figure 11. DDPG with V2. Learning curves of S2mbaV2, Simba (Lee et al., 2024c), and the MLP baseline on DMC-Hard and HBench- Hard benchmarks using DDPG (Lillicrap, 2015). SimbaV2 performs competitively with Simba DMC-Hard, both significantly outperforming the MI.P baseline. In the more challenging HBenchh-ard, Simba V2 shows clear improvements over Simba, indicating enhanced stability and generalization beyond SAC.
30

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
I. Complete UTD Scaling Results
L.I Gym- MuJoCo
Table H. Gym MuJoCo UTD Scaling Results. Final average performance 1M environment steps for each of the locomotion tasks in the Gym- - MuloCo benchmark. The number of evaluated random seeds for each update-to-data (UTD) ratio is 5. The values in [brackets] represent 95% hootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the TD3- normalized score as described in Appendix F.I.
Task
UTD =1 1
UTD
UTD
UTD 8
Ant-v4
7405 (7315, 7496]
7429 [7209, 7649]
7230 [6968, 7492]
6940 [6431. 7449]
HalEChestah-v4
11425 [10798. 12052]
12022 [116400, 12404]
12007 [11458, 12557]
11592 [9956, 13229]
Hopper-v4
3579 3847]
4053 [3928, 4178]
4003 [3647, 4359]
4151 42093,4269]
Humanoid-v4
7696 (4385, 10088]
10545 [10195, 10896]
11133 [10908, 1 11358]
11703 [11282, 12125]
Walker2d-v4
6069 [5724, 6414]
6938 [189)
6804 [6459, 7148]
6163 [4522, 7804]
IQM
1.433 [1.225, 1.648]
1.637 [1.471, 1.788]
1.617 [1,402, 1.83]
1.581 [1.358, 1.82]
Median
1.468 [1.269, 1.625]
1.616 [1.491, 1.743]
1.615 [1,438, 1.809]
1.602 [1.377, 1.821]
Mean
1.418 [1.264, 1,5691
1.617 [1,513, 1.7191
1.62 [1.47, 1.7731
1.598 [1.419, 1.78]
HalfCheetah-v4
Hoppervv4
Walker2d-va
Ant-v4
Average Return (1K)
18K
4K
10K
13K
2K
5K
6K
8K
1K
3K
0K
-2K
-1K
-1K
-2K
0.2
0.4
0.6 0.8
1.0
0
0.2
0.4
0.6
0.8
0.2
0.4 0.6
1.0
0.2
0.4
0.6
0.8
1.0
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
Humanoid ./4
Average Return (1K)
12K
8K
5K
-2K
0.2
0.4 0.6 0.8 1.0 Env steps (M)
UTD
UTD =
UTD
UTD 8
Figure 12. Gym-MuJoCo UTD Scaling Learning Curves. Average episode return (1k) for the Gym-MuJoCo environment. Results are averaged over 5 random seeds, and the shaded areas indicate 95% bootstrap confidence intervals.
31

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
1.2. Deepmind Control Suite Easy
Table 12. DMC-Fasy UTD Scaling Results. Final average performance at 1M environment steps for cach of the 21 tasks of the DMC-Easy henchmark. The number of evaluated random seeds for each update-to-data (UTD) ratio is provided The values in [brackets] represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are reported in units of Ik.
Task
UTD
UTD
UTD
UTD=
acrobot-swingup
[376, 450]
436 [391, 482]
458 [359, 558]
477 (438, 516]
ball-in-cup-catehh
[977, 985]
982 [980, 984]
982 (979, 986]
982 [979, 985]
castpoie-ballncee
[999, 999]
[999, 9991
[999, 999]
[999, 999]
ca:tppole-bbalaacee--aaa
1000 [1000, 1000]
[904, 1030]
1000 [1000, 1000]
1000 [1000, 1000]
cartpo:e--wwnggup
881 1881, 881]
880 1876, 883]
880 1879, 881]
881 [880, 882]
845 [843, 848]
848 [848, 849]
848 1848, 849]
841 [824, 858]
cheetah-run
917 [913, 920]
920 [918, 922]
902 1868, 937]
916 [912, 920]
finger-spir
940 1895, 985]
891 [810, 972]
762 [608, 915]
910 [790. 1030]
finger-turn-easy
951 (916, 987]
953 [925, 980]
954 [917. 992]
936 [857. 1014]
finger-turn-hard
928 [885, 972]
951 [925, 9771
902 1866, 9391
950 [910, 990]
fish-swim
818 [779, 856]
826 [806, 846]
815 [780, 850]
807 (778, 836]
hopper-top
379 [224, 535]
290 [233, 348]
326 [243, 410]
317 (230 404]
hopper-stand
845 [704, 986]
944 [926, 962]
781 [449, 1112]
932 1898, 967]
pendulummswingee
817 [776, 858]
827 [805, 849]
820 [781, 8591
821 [784, 859]
quadruped run
931 [922, 940]
935 [928, 943]
943 (936, 949]
935 [930, 940]
quadruped-walk
962 [955, 970]
962 [955, 969]
964 (958, 971]
965 (958, 972]
reacher-easy
963 [927, 1000]
983 [979, 986]
975 (958, 992]
983 [981, 985]
reacher-bard
975 1971, 980]
967 [946. 987]
976 1972. 980]
974 1970. 978]
walker-run
813 [806, 819]
817 [812, 821]
821 [819, 823]
802 1774, 831]
walker-stard
986 1980, 992]
987 [984, 990]
988 1984, 992]
987 1984, 991]
kerker-walk
977 1976, 979]
976 [974, 978]
976 (973, 979]
976 1972, 981]
IQM
0.928 [0.906. 0,948]
0.933 [0.918, 0.948]
0.925 [0.9. 0.946]
0.935 [0.91. 0.956]
Median
0.876 (0.85 834, 0.912]
0.875 846, 0,904]
0.866 [0.8 018818 0,905]
0.878 (0.835, 0.917]
Mean
0.873 (0.838, 0.905]
0.874 [0.848, 0.897]
0.861 [0.823, 0.896]
0.876 [0.841, 0.908]
32

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
acrebot:swinguup
ball-n-cup-catch
cartpole-balance
cartpoll-ballncce-pprree
1.000
1,000
1,000
1,000
Average Return
750
750
750
500
500
500
500
250
0.6 EnV steps (M)
0.8
0.4
1.0
0.2
0.6 Env steps (M)
Eny steps (M)
Env steps (M)
cartpole-swingup
cartpole-swin wwwpooll--wingupsppprras
cheetah-run
fing finger-spin
1,000
1,000
1,000
1,000
Average Return
750
750
750
750
500
500
250
250
250
250
0.2
0.4
0.6
0.6
1.0
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.6
1.0
0.2 0.4 0.6 0.8 1.0
Env steps (M)
Env steps
Env (M)
Env steps (M)
finger-turneassy
finger-turnharrd
fish-swim
hopper-hop
1,000
1,000
1,000
1,000
Average Return
750
500
500
500
500
250
0
0
En steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
hopper-stand
pendulum-swingup
quadruped-walk
quadruped-run
1.000
1,000
1,000
Average Return
750
750
750
750
250
250
250
250
0.2 0.4 0.6
0.8 1.0
0.2 0.4 0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
0.2 0.4
0.6 0.8
1.0
steps (M)
Env steps
steps (M)
steps (M)
reacher-easy
reacher-hard
walker-stand
walker-walk
1.000
1.000
1,000
1.000
Average Return
750
500
500
500
500
250
0.2
0.6 Env steps (M)
1.0
0.2
0.6 Env steps (M)
Env steps (M)
Env steps (M)
walker-run
Average Return
750
250
0.2 0.4 0.6 0.8 1.0 steps (M)
UTD =1
UTD =2
UTD =4
UTD =8
Figure 13. DMC-Easy UTD Scaling Learning Curves. Average episode return for the DMC-Easy environment. Results are averaged over 5 random seeds, and the shaded areas indicate 95% bootstrap confidence intervals.
33

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
I.3. Deepmind Control Suite Hard
Table 133. DMC-Hard UTD Scaling Results. Final average performance at 1M environment steps for each of the 7 tasks of the DMC-Hard henchmark. The number of evaluated random seeds for each update-to-data (UTD) ratio is provided 5. The values in [brackets] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are reported in units of Ik.
Task
UTD 1
UTD
UTD 4
UTD 8
dog-run
477 [429, 525]
562 [516, 608]
655 [620, 691]
555 [523, 587]
dog-stand
967 [959, 974]
981 (977, 985]
967 [960, 974]
972 [967, 976]
dog-trot
850 [810, 890]
861 [772, 950]
846 [782, 910]
898 [888, 909]
dog-walk
921 [912, 930]
935 [927, 944]
923 [905, 941]
949 [945, 953]
humancid-run
183 [164, 203]
194 [182, 207]
272 [230, 313]
253 [228, 278]
humancid-stare
660 [585, 734]
916 [886, 945]
928 [926, 930]
933 [924, 941]
humanci humancidwwalk d-walk
568 [533, 603]
651 [590, 713]
818 [751, 885]
819 [762, 877]
IQM
0.713 [0.598, 0.809]
0.808 [0.725, 0.88)
0.851 [0.755, 0.916]
0.849 [0.727. 0.924]
Median
0.666 [0.563, 0.774]
0.729 [0.655, 0.81]
0.771 [0.678, 0.868]
0.767 [0.652, 0.861]
Mean
0.669 [0.581, 0.753]
0.729 [0.663, 0.791]
0.769 [0.687, 0.845]
0.759 [0.67. 0.84]
humanoid-stand
numanoid-wall
humanoid-run
dog-stand
1,000
1,000
1,000
1,000
Average Retum
500
500
500
500
250
250
250
250
0.2
0.8
1.0
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
dog-walk
dog-run
dog-trot
1,000
1,000
Average Return
750
750
750
500
500
500
250
0
0
0
0.2 0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
Env steps (M)
Env steps (M)
Env steps (M)
UTD =1
UTD = 2
UTD = 4
UTD
Figure 14. DMC-Hard UTD Scaling Learning Curves. Average episode return for the DMC-Hard environment. Results are averaged over 5 random seeds, and the shaded areas indicate 95% bootstrap confidence intervals.
34

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
1.4. MyoSuite
Table 14. MyoSuite UTD Scaling Results. Final average performance at 1M environment steps across cach of the 10 continuous control tasks in the MyoSuite benchmark, including hoth fixed-goal and randomized-goal (hard) settings. The number of evaluated random seeds for each update- to- to-data (UTD) ratio is 5. The values in [brackets] represent a 95% bootstrap confidence interval. Performance is measured by the average success rate of each task.
Task
UTD
UTD =
UTD =4
myo pen-twirl-hard
76.0 [57.8, 94.2]
93.0 [88.8, 97.2]
92.0 [84.7, 99.3]
98.0 [94.1, 101.9]
nyo-pen-twirl
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
myo myo-key-turn-bard
46.0 [8.5, 83.5]
62.0 [42.7, 81.3]
70.0 [34.9, 105.1]
80.0 [49.6, 110.4]
nyo-key-turn
80.0 [40.8, 119.2]
100.0 [1000, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
myo-obj-hold-bare
100.0 [100.0, 100.0]
98.0 195.4, 100.6]
98.0 [94.1, 101.91
92.0 184.7, 99.31
myo-obj-hold
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
nyo-pose-hard
[00,,. 0.0]
[0.0, 0.01
0.0 [0.0,
0.0 [0.0, 0.01
myo-pose
100.0 [100.0. 100.0]
100.0 [100,0, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
myo-reach-hard
92.0 [84.7, 99.31
94.0 187.3, 100.7]
98.0 [94.1, 101.91
96.0 [91.2, 100.8]
myo-reach
100.0 [100.0. 100.0]
100.0 [100.0. 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
IQM
96.9 185.8, 100.01
99.0 196.8, 100.01
99.2 196.2, 100.01
99.6 196.9, 100.0]
Median
79.0 [67.0, 91.0]
84.5 [78.0, 93.0]
87.0 [76.0, 98.0]
88.0 [77.0, 98.0]
Mean
79.4 168.6, 88.8]
84.7 [78.3, 90.6]
85.8 176.4, 94.01
86.6 177.4, 94.61
myo-reach
myo-reach-hard
myo-pose
myo-pose-haie
100
100
100
100
Success Rate (%6
75
75
50
50
50
50
25
25
25
25
0.2
0.4
0.6
0.8
1.0
0.2 0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Env steps (M)
Env steps (M)
Env steps (M)
steps (M)
nyy-obj-holdd
myo hyo--ob--hlldhard
myo myo-key-turn
myo-key key-tumm-had
100
100
100
100
Success Rate (%)
75
75
75
75
50
25
25
25
25
0
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
0.2
0.4 0.6 0.8
1.0
0.2
0.4 0.6 0.8 1.0
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
myo-pen-twirl
myo-pen pen-twirr-hard
100
100
Success Rate (%6)
75
50
50
25
25
0.2 0.4
0.6
0.8
1.0
0
0.2 0.4 0.6 0.8 1.0
Env steps (M)
Env steps (M)
UTD
UTD=
UTD
UTD
Figure 15. MyoSuite UTD Scaling Learning Curves. Average episode success rate (%) for the MyoSuite environment. Results are averaged over 5 random seeds, and the shaded areas indicate 95% bootstrap confidence intervals.
35

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
I.5. Humanoid Bench
Table 15. HumanoidBench UTD Scaling Results. Final average performance 1M enviromment steps for each of the 14 locomotion tasks in the HumanoidBench benchmark. The number of evaluated random seeds for each update-to-data (UTD) ratio is 5. The values in [brackets] represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) computed over the success normalized score as deserihed in Appendix F.4
Task
UTD =1
UTD =2
UTD =
UTD 8
aa-sit-hard-vi
681 [506, 857]
679 [548, 811]
719 [664, 773]
810 [784, 836]
bl-walk-vo
732 [522, 941]
845 [840, 850]
846 [841, 851]
844 [840, 847]
salstair-vv v0
473 [444, 503]
493 [467, 518]
546 [541, 550]
532 [512, 552]
h1 run v0
247 [152, 342]
415 [307, 524]
318 [176, 461]
425 [293, 558]
806 [773, 839]
723 [651, 795]
775 1719, 831]
813 (797, 828]
h1-pole-v0
769 [758, 780]
791 1785, 797]
799 [780, 817]
827 [787, 868]
b1-slide-v0
412 [279, 544]
487 [404, 571]
544 [500, 588]
534 [505, 563
bl-bbelaance-bardd-v0
135 [111, 160]
143 1128, 157]
128 [118, 139]
167 [157, 178]
bi-sit-simpple-v0
873 [868, 879]
875 1870, 880]
908 [861, 955]
867 [839, 894]
bl-maze-v0
350 [332, 368]
313 [287. 340]
343 [327, 359]
338 [325, 351]
hl-crawl-v0
923 [884, 962]
946 (933, 959]
939 (927, 951]
954 [923, 984]
b1-hardle-vo
193 [171, 215]
202 [167. 236]
244 [230, 259]
246 [206. 287]
all-raah--on
4166 [3706, 4627]
3850 [3272. 4427]
4003 [36144 4392]
3449 [2541. 4358)
h1 stand v0
771 [669, 873]
814 [770, 857]
765 [695, 835]
855 [828, 882]
IQM
0.734 [0.574, 0.887]
0.799 [0.685, 0.905]
0.813 [0.657, 0.958]
0.873 [0.706, 1.002]
Median
0.71 [0.612, 0.859]
0.781 (0.691. 0.863]
0.782 (0.665, 0,914]
0.824 (0.699, 0.944]
Mean
0.737 (0.637, 0.836]
0.776 (0.704, 0.846]
0.791 [0.687. 0.893]
0.822 [0.72, 0,92]
36

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
h1-walk-v0
h1-stand-v0
h1-run-vO
h1 h1-reach-ve
1000
15000
Average Retum
12500
Success
Success:
Success
10000
Success
500
500
500
7500
5000
250
250
250
0
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4 0.6
0.8
1.0
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
h1-hurdle-v0
h1-crawi-vo
hl-maze-v0
h1-sit-simple-vO
Average Retum
1250
750
750
Surcess
750
Success
500
500
750
500
250
250
250
250
0.2
0.4
0.6
0.8
1.0
0.2
0.4 Env steps (M)
0.2
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Env steps (M)
Env steps (M)
Env steps (M)
h1-sit-hard-v0
h1-balance-simple-vo
h1-balance-hard-vo
h1-stair-v0
1000
1000
1000
1000
Average Retum
750
750
750
Success
750
500
500
500
500
250
250
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
h1-slide-vO
h1-pole-v0
1000
1000
Average Retur
750
Success
Succos
250
0.2
0.4
0.6
0.8
1.0
0.2 0.4
0.6
0.8 1.0
Env steps (M)
steps (M)
UTD =1
UTD =
UTD =4
UTD =
Figure 16. Humanoidbench UTD Scaling Learning Curves. Average episode return for the HumanoidBench environment. Results are averaged over random seeds, and the shaded areas indicate 95% bootstrap confidence intervals. The black dotted line indicates the success score of each tasks (Appendix F.4)
37

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
J. Complete Main Results
This section provides learning curves and final performance for each online RL task across the evaluated algorithms.
Learning Curve. For visibility learning curve, we focus on DreamerV3 (Hafner et al., 2023). TD7 (Fujimoto 2023), TD-MPC2 (Hansen al., 2023), MR.Q (Fujimoto et al., 2025), and Simba (Lee et al., 2024c) as main baselines, selected for their strong performance and community adoption. We omit curves for algorithms with unavailable raw samples at each task.
Confidence Interval. The light-colored area in the figures and the gray-shaded, bracketed terms in the tables represent 95% bootstrap confidence intervals. For each task evaluated over n random seeds, the 95% bootstrap confidence interval CI computed as:
CI
# 1.96 Vn. + 1.96 X
Vn
where a and are the sample mean and standard deviation (with Bessel's correction) of the evaluation, respectively, For aggregated scores (mean, median, and interquartile mean), confidence intervals are computed over all n x raw samples, where n and T are the number of evaluated random seeds and tasks in the benchmark, respectively. For algorithms with only average scores for each task available, we approximate the CI of aggregated scores using these averages (denoted with gray-colored t). We caution that this estimation may be inaccurate.
38

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
J.I. Gym- MuJoCo
Table 16. Gym- -MuJCoo Final average performance at 1M enviromment steps for each of the 5 locomotion tasks in the Gym - MuJoCo henchmark. The numher of evaluated random seeds for each algorithm is provided in Appendix F. The values in [brackets] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the TD3-normalized score as described in Appendix F.I.
Task
D3eamerV3
TD7
TD-MPC2
MR.Q
Simba
Simba
Ant-vd
1947 [1076, 2813]
8509 [8168, 8844]
4751 (2988, 6145]
6989 [6203, 7617]
5882 [5354, 6411]
7429 [7209. 7649]
Hall ICheeLLeaa
5502 [3717. 7125]
17433 [17301. 17559]
L5078 [14065, 15932]
13305 [11841, 14140]
9422 [8745, 10100]
12022 [11640, 12404]
Hoppec-vd
2666 (2106, 3210]
3511 [3236. 3736]
2081 [1197, 2921]
2684 [2154, 3269]
3231 [3004, 3458]
4054 [3929, 4179]
Bumenold-vv
4217 5523)
7428 [7304, 7553]
6071 [5770, 6353]
7259 [5080, 9336]
6513 [5634, 7392]
10546 [10195, 10897]
Na kerrk4dd
4519 [3692, 5244]
6096 [5621, 6547]
3008 [1706, 4321]
6629 (5816, 7493]
4290 [3864,
6938 [1891,7185]
IQM
(0.620, 0.850]
1,540
[1.500, 1,580]
1,050 (0.890, 1.190]
1.450 [1.270, 1.580]
1.114 [1.043, 1,200]
1.637 [1.470, 1,791]
Median
0.810 [0.580, 0.930]
1,550 [1 [1.450. 1.630]
1.180 [0.830, 1.220]
1.420 [1.190, 1.710]
1.143 [1.063, 1.227]
1.616 [1 49, 1.744]
Mean
0.760 (0.670, 0.860]
1.570
(1.540, 1,600]
(0.920, 1.150]
1.390 [1.270, 1.490]
1.147 [1 [1.075. 1.223]
1.617 (1511, 1.718)
PPQ
SAC
TDC
REIN
Creef
HRO
(1760) 1815)
4754 01911
40171
7340 17780 19100
1907
1188 (675)
JESTA (0RR)
MAN' (1000 65120)
2009 (9O1 71453
Hoppez-
3622 (2535, 5539)
2787
(2349 38251
5226
(2011 3543)
3121
12515, 3627)
1536
1950, 37500
3299 (373AL 39691
2799
(2580) 3208
2407 (1SS) 30795
2122
(1S5S, 25363
477 (435, S18)
4825 (S75A, 5955
5165
(500) 53.00
0032
(SS5, 5260)
5029
1495, 60901)
S276 15127) 54301
5500 (SSSX, 54071
19480 (10007 (0055)
4757
(11.9) 63766
Malkwadd 144
2467 3022]
4556
4843)
42261
5195
(4593, STOT))
5663)
50201
4781 (4559), 50141
6237
(527)) 72370
(2054, 4501)
TOM
6,410 (0. 110, 0.834"
1,07711 (90. (.155
1.000
0000
11:035, T,5ROP
3.14
(0.911 1.3901
1941
1100
8.7001
1.3331
Mevilar
(0.071, 065667
1.03311
0000)
1.36
2411
8.83%
1999
Mexs
CA47 (0.186 0.725'
1,892
[1 11003, L.60)
1,000
(1000)
2
1.32 (1/09) 1.U55)
3.13P
(1,022 12617
1,160 (1.0%) 224)
1.334
[1.00) 12305
1.455
[1.300, 1,008]
1.101
(0927) 1.278)
HalfCheetah-v4
Hopper-v4
Walker2d-v4
Ant-v4
Average Return (1K)
18K
4K
8K
10K
13K
5K
8K
1K
3K
2K
3K
0K
1K
-2K
-1K
-1K
-2K
0.2
0.4
0.6 0.8
1.0
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
Humanoid-v4
Average Return (1K)
12K
8K
5K
1K
-2K
0
0.2
0.4
0.6 0.8 1.0
Env steps (M)
MR.Q
Simba
SimbaV2
Figure 17. Gym-MuJoCo Learning Curves. Average episode return (1k) for the Gym-MuloCo environment. Results are averaged over random seeds of each algorithm, and the shaded areas indicate 95% bootstrap confidence intervals.
39

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
J.2. Deepmind Control Suite- Easy
Table 17. DMC Easy. Final average performance at 1M environment steps for each of the 21 tasks of the DMC Easy benchmark. The number of evaluated random seeds for each algorithm is provided in Appendix F. The values in [brackets] represent 95% hootstrap
confidence interval. The aggregate mean, median and interquartile mean (IQM) are reported in units of Ik.
DreamerV3
TD7
ID-MPC2
MK.Q
Simba
Simbaa22
acroboc-ox1 sooo---xxnnggg nguu
230 [193, 266]
[18. 75)
[S51, 615]
567 [523, 616]
[504. 555]
[379. 482]
[19), 482)
ball-in- hal1-ia-cup-catch
968 1965, 973]
984 (982. 996]
983 [981. 9851
981 1979. 984]
982 [551. 954]
981 1978. 99.3
982 [980) 984]
cartpole be Lallane
998 1997. 1003]
999 1958. 1000)
996 (9055 908]
999 (999. 1000)
999 1958. 999]
998 1998. 999)
992 (759) 909)
cortpo.e-ac..onc--ppe
1000 [1000. 1000)
999 1000. 1000]
1000 [1000, 1000]
1000 (1000. 1000)
852 553. 11413
991 [973. 1008]
967 (904. 1030]
736 [591. 838)
869 [856, 873]
875 [870. 630)
666 [565. 865]
879 [877. 652]
676 [S71.551]
880 [876, 58.3
702 [560) 7921
573 [333, 895]
845 (839, 849]
798 (780. 818)
840 1827. 85.21
525 [755. 355
848 [548, 549
cheeteh run
917 [915. 920)
699 [655. 744]
914 [911] 9177
877 (840) 905)
863 [822. 0034
930 [918. 922)
821 [642. 91.3
ringer-spin
666 [577. 763)
335 (99. 595]
986 (936. 988)
937 [917. 955)
988 1957.
649 [758, 9.9)
89] [510. 972
Ginger-tuseeeayy
953
Finger-turn--on
rish-svin
[808. 819)
(773. 8101
hoppee: hop
165]
125,
1195,
hepper-stand
747 [669. 306]
670 (456, 825)
(944. 958]
951 [9455 955)
(94). 957]
[900) 957]
(926. 966
pendolur--wwinee
774 [7400. 302]
300 [251. 7433
846 1830, 562]
748 (597. 8.29]
829 1795. 8664]
737 [575. 8991
827 [925, 5.49
quedruped run
130 (22. 169)
645 (567. 71.5
042 (058) 9447
947 (140)) (254]
859 1834. 8955
928 (016. 9399
955 (928. 948]
qussruped-vaik
190 [1.77 243]
9:19 [939. 957]
963 (959. 965]
963 [559. 9677
958 (919 967]
957 [951. 96.3
962 (955. 969)
reacher-ease eeay
966 (9644 9701
970 1151] 1982
983 (960) 9966
985 (985. 1985
9935 1983. 19644
983 [981. 9966
983 1979) 966
reacher-hard
919 [864, 955)
808 1861] (336)
(906 079]
077 (075. (880)
074 (070) 077)
066 (947. 98-)
967 (046. 967)
510 [430. 598)
(783.
854 [851, 839]
199 (765. 815]
790 (766,
[792. 801)
[812. 821
walkwe aabanad
988 9900
(164.
walkoz-walk
890 [875,
97%
9788
1QM
0.813 (0.621. 0.899)
0.771 (0,570, 9077)
0.941 (0.890, 9.377)
0.927
0.966)
0,928 (0.899, 0,953)
0.922 (0 0 0.938
0.913 (0.9188 0 0.948)
Median
0.813 (0.702, 0.917)"
0.804 (0.573, 0.949)7
0.952 [0,875, 0.988 1
0.950 (0.306, 0.977)
0.872 (0.819, 0.900)
0,870 (0.541, 0.896]
0,875 (0.847, 0.905]
0,714 (0.584, 0.832)
(0.548, 0.816]"
(0.78%, 0.935)"
0,861
01.996
0.864 (0.54, 0.887)
0.874
40

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
acrnbot-swinguu
ball-n-cup-catch
cartpole-balance
cartpoll-balanceessarree
1.000
1,000
1,000
1,000
Average Return
750
750
750
500
500
500
500
250
0.6 EnV steps (M)
0.8
0.4
1.0
0.2
0.6 Env steps (M)
Eny steps (M)
Env steps (M)
cartpole-swingup
cartpole-swin http:lll-swiinuusspppprse
cheetah-run
fing finger-spin
1,000
1,000
1,000
1,000
Average Return
750
750
750
750
500
500
250
250
250
250
0.2
0.4
0.6
0.6
1.0
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.6
1.0
0.2 0.4 0.6 0.8
1.0
Env steps (M)
Env steps
Env (M)
Env steps (M)
finger-turneassy
finger-tur-harrd
fish-swim
hopper-hop
1,000
1,000
1,000
1,000
Average Return
750
500
500
500
500
0
0.6 Env steps (M)
0.2 Env steps (M)
Env steps (M)
Env steps (M)
hopper-stand
pendulum-swingup
quadruped-walk
quadruped-run
1.000
1,000
1,000
Average Return
750
750
750
750
250
250
250
250
0.2 0.4 0.6
0.8
1.0
0.2 0.4 0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
0.2 0.4
0.6
0.8
1.0
steps (M)
Env steps
steps (M)
steps (M)
reacher-easy
reacher-hard
walker-stand
walker-walk
1.000
1.000
1,000
1.000
Average Return
750
500
500
500
500
250
0.2
0.6 Env steps (M)
1.0
0.2
0.6 Env steps (M)
Env steps (M)
Env steps (M)
walker-run
Average Return
750
250
0.2 0.4 0.6 0.8 1.0 steps (M)
MR.Q
Simba
SimbaV2
Figure 18. DMC-Easy Learning Curves. Average episode return for the DMC-Easy environment, Results are averaged over random seeds of each algorithm, and the shaded areas indicate 95% bootstrap confidence intervals.

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
J.3. Deepmind Control Suite- Hard
Table 18. DMC Hard. Final average performance at 1M environment steps for cach of the 7 tasks of the DMC Hard benchmark. The number of evaluated random seeds for each algorithm is provided in Appendix E. The values in [brackets] represent a 95% hootstrap
confidence interval. The aggregate mean, median and interquartile mean (IQM) are reported in units of Ik.
Task
DreamerV3
TD7
TD-MPC2
MR.Q
Simba
Simba V2
dog-run
[4]
69 [36, 101]
265 [166, 342]
569 [547, 595]
544 (525, 564]
562 [516, 608]
dog-stand
22 220
582 [432, 741] 21 [13, 30]
506 [266, 715
967 [960. 975]
960 [951. 9691
981 1977. 985]
dog-trot
10 [6. 17]
407 [265. 530]
877 [845, 898]
824 [773, 876]
861 [772. 950]
dog-walle
17 [15. 21]
[19. 166]
486 [240, 704]
916 [908, 924]
916 [905, 928]
935 [927. 944]
humanoid d-run
[ 1]
57 [23, 92]
181 [121, 231]
200 236)
181 [171, 191]
194 [182, 207]
humanof d-stland
[ 6]
317 [117, 516]
658 [506, 745]
868 [822, 903]
846 [801, 890]
916 [886, 945]
humanold walk
[1.2)
176 (42, 320]
754 [725, 791]
662 [610, 724]
668 [608, 728]
651 [590, 713]
1QM
0.008 [0.002, U.016)!
0.134 10.047, 0,343||
0.464 [0,305, 0.632
0.778 [0,500, 0.911/!
0.773 [0.713, 0.831
0.808 [0.726. 0.8791
Median
0.005 [0.001, 0.0188*
0.069 (0.052, 0.317)
0.486 [0.265, 0.658)
0.868 [0.569, 0.916]*
0.706 [0.647, 0.772]
0.729 (0.655, 0.808]
Mean
0.009 (0.003, 0.015]*
0.182 (0.062, 0.336]
0.465 [0.329, 0.606]*
0.723 (0.516, 0.886]*
0.706 [0.656, 0.755]
0.729 [0.664, 0.791]
Task
PPO
SAC
iQRL
BRO
MAD-TD
dog-run
26 [26, 28]
36 [3. 69)
380 [336, 424]
374 [338, 411]
437 [396. 478)
dog-stand
129 [122, 139]
102 [39, 164]
926 [897. 955]
966 [956, 977]
967 [952. 982]
dog trot
31 [30, 34]
29 52]
713 [516, 909]
783 [717, 848]
867 [805, 929]
dog-walk
40 [37, 43]
38 [11, 65]
866 [827, 905]
931 [920, 942]
924 [906, 943]
1 1] humanoid-stand
humanol durun
116 189, 144]
188 [167, 210]
204 [186, 223]
200 [180, 220]
5 [5. 6]
352 [225, 479]
727 [655, 799]
920 [909, 931]
870 [840. 901]
humanoid walk
1 [1, 2]
273 [128, 418]
688 [642, 735]
672 [619, 725]
684 [609, 759]
IQM
0.021 [0.003, 0.069]*
0.069 [0.042, 0.114]
0.694 [0.528, 0.805)
0.772 [0.662, 0.854]
0.787 [0.691, 0.865)
Median
0.026 [0.001. 0.040]*
0.159 [0.08, 0.183]
0.64 [0.516, 0.766]
0.694 [0.615, 0.774]
0.707 [0.634, 0.786]
Mean
0.033 (0.009, 0.068]
0.136 (0.098, 0.175]
0.642 [0.531, 0.747]
0.693 (0.625, 0.757]
0.708 (0.642, 0.771]
humanoid-stand
humanoid-wall
humanoid-run
dog-stand
1,000
1,000
1,000
1,000
Average Return
750
750
750
500
500
500
500
250
250
250
250
0.2
0.4
0.6
0.2
0.4
0.6
0.8
1.0
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
dog-walk
dog-run
dog-trot
1,000
1,000
1,000
Average Return
750
750
750
500
500
500
250
0
0
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
Env steps (M)
Env stops (M)
Env steps (M)
MR.
Simba
SimbaV2
Figure 19. DMC-Hard Learning Curves. Average episode return for the DMC-Hard environment. Results are averaged over random seeds of each algorithm, and the shaded areas indicate 95% bootstrap confidence intervals.
42

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
J.4. MyoSuite
Table 19. MyoSuite. Final average performance at IM environment steps across each of the 10 continuous control tasks in the MyoSuite henchmark, including both fixed-goal and randomized-goal (hard) settings. The number of evaluated random seeds for each algorithm provided in Appendix E. The values in [brackets] represent 95% bootstrap confidence interval. Performance is measured by the average success rate of each task.
Task
D3eamerV3
TD7
TD-MPC2
Simba
Simba V2
myo pe--penttwil--hee
53.3 [29.8, 76.9]
12.0 [2.4, 21.6]
40.0 [40.0, 40.0]
77.0 [66.4, 87.6]
93.0 [88.8, 97.2]
myo myo-pen-twi:1
96.7 [90.1, 103.2]
100.0 [100.0, 100.0]
70.0 [11.2. 128.8]
80.0 [53.9, 106.1]
100.0 [100.0, 100.0]
myo-key-turn hard
0.0 [0.0, 0.0]
0.0 [0.0, 0.0]
0.0 [0.0, 0.0]
7.0 [-3.1, 17.1]
62.0 [42.7. 81.3]
myo-key-turn
88.9 [67.1, 110.7]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
myo obi-hold hard
9.4 [8.4, 10.5]
10.0 [-0.7, 20.7]
56.7 [39.4, 74.0]
96.0 [92.8, 99.2]
98.0 [95.4, 100.6]
myo-obi ooo-obj-oold
33.3 [-32.0, 98.7]
20.0 [-19.2, 59.2]
100.0 [100.0, 100.0]
90.0 [70.4, 109.6]
100.0 [100.0, 100.0]
myo-pose-hard
0.0 [0.0, 0.0]
0.0 [0.0, 0.0]
0.0 [0.0, 0.0]
0.0 [0.0, 0.0]
0.0 [0.0, 0.0]
myo-pose
100.0 [100.0, 100.0]
0.0 [0.0, 0.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
myo-reach-hard
0.0 [0.0, 0.0]
14.0 [0.7, 27.3]
83.3 [66.0, 100.6]
93.0 [86.4, 99.6]
94.0 [87.3, 100.7]
myo-reach
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
100.0 [100.0, 100.0]
IQM
46.6 [16.7, 76.9]
22.3 [4.2, 46.2]
77.5 [50.6, 94.4]
95.2 [82.6, 98.8]
99.0 [96.8, 100.0]
Median
47.0 [29.0, 67.0]
34.0 [21.0, 51.0]
65.0 [47.0, 82.0]
77.0 [66.5, 85.5]
84,5 [78.0, 93.0]
Mean
48.2 [31.8, 64.6]
35.6 [23.8, 48.0]
65.0 [50.3, 78.7]
74.3 [66.3, 81.7]
84.7 [78.2, 90.6]
myo-reach
myo-reach-hard
myo-pose
myo-pose-hard
100
100
100
100
Success Rate (%6
75
75
75
50
50
50
50
25
25
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Env steps (M)
Env steps (M)
Env steps (M)
steps (M)
myo-obj-hold
myo-oojjhold-hard
myo-key-turn
myo-key-tumn-han
100
100
100
100
Success Rate (%)
75
75
75
75
25
25
25
25
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6 0.8
0.2
0.4 0.6 0.8 1.0
0.2
0.4
0.6 0.8 1.0
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
myo-pen-twirl
myo-pen- syyo-pen-twirlhard
100
Success Rate (96)
75
75
50
50
25
0.2 0.4 0.6
0.8
1.0
0
0.2 0.4 0.6 0.8 1.0
Env steps (M)
Env steps (M)
DreamerV3
TD7
TD-MPC2
Simba
Simbav2
Figure 20. MyoSuite Learning Curves. Average episode success rate (%) for the MyoSuite environment. Results are averaged over random seeds of each algorithm, and the shaded arcas indicate 95% bootstrap confidence intervals.
43

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
J.S. Humanoid Bench
Table HumanoidBench. Final average performance at 1M environment steps for each of locomotion tasks in the Humanoid- Bench benchmark. The number of evaluated random seeds for each algorithm is provided in Appendix E. The values in [brackets] represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over success normalized score described in Appendix F.4.
Task
DreamerV3
TD7
TD-MPC2
Simba
SimbaV2
h1-pole-v0
41 [28, 54]
441 [320, 563]
744 [609, 879]
716 [667, 765]
791 [785. 797]
h1-slide-v0
11 15]
[26,
334 [304, 364]
277 [252, 303]
487 5711
hl-stair-v0
[2, 12]
[31. 74]
378 [108, 648]
269 [153, 385]
493 5187
ba--as.ance-hard--00
11 [7, 15]
79 [51. 107]
31 [6]
75 [71, 80]
143 [128. 157]
11-=ba_.rrce--smmple-000
[6, 12]
69 [58, 80)
42 [14, 70]
337 [193, 482]
723 [651, 795]
hl-sit-tard-vo
15 [-4,
235 [154, 315]
723 [660, 786]
512 [354, 670)
679 [548, 811]
-1-sit-simp1e--00
[9, 28]
874 (869, 879]
790 [772, 809]
833 [814, 853]
875 [870, 880]
h1-maze-v0
113 [107, 118]
[137, 156]
[106,
354 [342,
313 [287, 340]
b1-crawl-v0
248 [176, 319]
582 600]
1959, 965]
923 [904,
946 959]
h1-hurdle-vo
4
60 118, 102]
387 [254, 519]
175 [150. 201]
202 [167. 236]
h1-reach-v0
3203 [2824, 3581]
1409 [998, 1821]
2654 [1951, 3357]
3874 45277
3850 [3272, 4427]
b1-run-vo
[2, 6]
91 [54, 128]
778 [763, 793]
232 [185, 279]
415 [307, 524]
b1-st.and-v0
15 [7, 22]
433 [138, 727]
798 [779, 817]
772 [701, 843)
814 [770, 857]
b1-waik-vD
[1, 16]
[22, 45J
(813, 815]
550 [391, 709)
845 [840, 850]
1QM
0.007 [0.004, 0.012]
0.134 (0.088, 0.245]
0.734 [0.510, 0.936]
0.521 [0.413, 0.633)
0.799 [0.686, 0.908)
Median
0.021 [0.000. 0.047]
0.284 [0. 183. 0.392]
0.696 (0.536, 0.881]
0.598 [0.514, 0.692]
0.781 [0.693, 0.865]
Mean
0.022 [0.000, 0.046]
0.289 [0.207, 0.375]
0.710 (0.562, 0.858]
0.606 (0.536, 0.678)
0.776 [0.705, 0.849]
44

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
h1-walk-v0
h1-stand-v0
h1-run-vO
h1 h1-reach-ve
1000
15000
Average Retum
12500
Success
Success
Success
10000
Success
500
500
500
7500
5000
250
250
250
0
0.2
0.4
0.6
0.8
1.0
0
0.2
0.4
0.6
0.8
1.0
0.2 0.4 0.6 0.8
1.0
0.2
0.4
0.6
0.8
1.0
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
h1-hurdle-v0
h1-crawi-vo
hl-maze-v0
h1 h1-sit-simple-vO
Average Retum
1250
750
750
Surcess
750
500
500
750
500
250
250
250
250
0.2
0.4 0.6
0.2 0.4
0.2
0.6 0.8
1.0
0.2
0.4
0.6
0.8
1.0
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
h1-sit-hard-v0
h1 h1-balance-simpleevo
h1-balance-hard-vo
h1-stair-v0
1000
1000
1000
1000
Average Retum
750
750
750
Success
750
500
500
500
500
250
250
Env steps (M)
Env steps (M)
Env steps (M)
Env steps (M)
h1-slide-vO
h1-pole-v0
1000
1000
Average Retur
750
Success
Success
250
0.2 0.4 0.6
0.8
1.0
0.2 0.4
0.6
0.8
1.0
Env steps (M)
steps (M)
DreamerV3
TD7
TD-MPC2
Simba
Simbav2
Figure 21. Humanoidbench Learning Curves. Average episode return for the HumanoidBench environment, Results are averaged over random seeds of cach algorithm, and the shaded areas indicate 95% bootstrap confidence intervals. The black dotted line indicates the success score of each tasks (Appendix F.4)
45

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
K. Complete Ablation Results
This section presents a per-environment analysis of the design variations discussed in Section 5.5. Each table includes raw scores for individual environments, with [bracketed values] indicating 95% bootstrap confidence intervals. The aggregate mean, median, and interquartile mean (IQM) are calculated based on the differences in normalized scores, To illustrate the magnitude of these differences, use the following highlight scale:
( 0.1)
[0.05. 0.1)
[0.02, 0.05)
0..02 0.05)
0.05, -0.1)
<< -0.05)
K.1. 1om--Mon MuJoCo
Table 21. Mujoco (Input Design). Final average performance at IM environment steps averaged over seeds. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the nor-normalized score as described in Appendix F.I.
Task
Si
No L. Normalize
No Shifting
Canaye :1
Resize Projection
Ant-v4
7429 [7209, 7649]
7267 [7065, 7469]
7203 [6765. 7641]
7367 [7302, 7433]
7834 [7374, 8294]
felftheeeth-yd
12022 [11640, 12404]
5386 [4901, 5870
5464 [5178, 5750]
11913 (11241, 12584]
12047 [11315, 12778]
Hopper-v4
4053 [3928, 4178)
3764 [3550, 3978]
3676 [3619, 3734]
3703 [3217, 4189]
3788 [3602, 3974]
Humanoid-vé
10545 [10195, 10896]
9820 18982, 10658]
10655 [10050, 11259]
10680 [10586, 10775]
10530 [10308, 10753]
Ralker2d-vd
6938 (6691, 7185]
5560 (4757, 6364]
5750 (5368, 6131]
6192 (5986, 6399]
6985 [6378, 7591]
IQM
1.637 [1.471, 1.792]
1.448 11.102, 1.715]
1.463 11,115, 1.746]
1.552 [1.304, 1,809]
1.645 [1.34, 1.919]
Median
1616 [1.495, 1.746]
1.364 [1.122, 1.618]
1.411 [1.14, 1.671
1.573 [1.372, 1.749]
1.623 [1.434, 18813]
Mean
1.617 [1.514, 1.72]
1.37 [1.137, 1,591]
1.406 (1.164, 1.644]
1.558 11339, 1.728)
1.623 [1.45, 1.796]
Table 22. Mujoco (Output Design). Final performance at IM environment steps averaged over 3 seeds. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the TD3-normalized score described in Appendix F.I.
Task
SimbaV2
MSE Loss
No Reward Scaling
No Return Bounding
Hard Target
Ant-v4
7429 [7209, 7649]
6195 [5459, 6931]
7087 [6935, 7239]
7622 [7486, 7757]
7373 [7342, 7405]
Hal ah-vetah-v4
12022 (1 1640, 12404)
12222 [11753, 1 12691]
12775 [12608, 12941]
12724 [12133, 13315]
11986 [1 1434,, 12538]
Hopper-v4
4053 [3928, 4178]
3507 [3333, 3682]
2932 [2007, 3858]
4113 [3999, 4228]
3623 [3445, 3800]
Hunanoid-v4
10545 [10195, 10896]
7764 [7227, 8302]
8265 [5867. 10664]
10583 [10506, 10660]
9973 (9763, 10184]
Ra ker2d-vd
6938 [6691, 7185]
5267 [4667, 5866]
5786 (5116, 6456]
6442 [6112, 6772]
7428 [6463, 8393)
IQM
1.637 [1.474, 1,792]
1.334 [1,162, 1.5]
1.405 [1,236, 1,597]
1.612 [1.367 1.864]
1.624 [1.32, 1.874]
Median
1.616 [1.495, 1.745]
1.341 [1.202, 1.483)
1.426 [1.201. 1 1.609]
1.616 [1.453, 1.79]
1.593 [1.399, 1.771]
Mean
1.617 [1.516, 1.719]
1.343 [1 11.225, 1.462]
1.395 [1.247, 1,544]
1.62 (1.471, 1.773]
1.589 [1.414, 1.757]
Table 23. Mujoco (Training Design). Final performance at IM environment steps averaged over 3 seeds. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the TD3-normalized score described in Appendix F.I.
Task
V2
No LR Decay
Ronet
Escate 11
Ctuis 0.5
Oueate :
Ant-v4
7429 /7209, 7649]
7553 16914, 8192]
7429 (7237. 7621]
7296 (7146, 7447]
7552 (7323, 77811
7258 (6859, 7556]
ll Seiffheetehh-m
12022 [11640 12404)
12227 111760, 12694]
12090 [11758, 12422]
11548 [10470, 12626]
12538 [12472. 12604]
11982 [11585, 12378)
Hopper-v4
4053 [3928, 4178]
3635 13000, 4269]
4046 [3944, 4148]
4008 [3869, 4146]
3568 [3279, 3857]
4023 [3830, 4216]
Humanoid-v4
10545 [10195, 10896]
9907 18412, 11401]
18615, [1023]
10636 [10465, 10807]
108228 [10426, 11229]
[61188 10831]
Welker2d-we
6938 (6691, 7185]
6661 [6010, 2311]
6583 (5893, 7274]
6770 [6541, 6998)
6328 (5726, 6929)
6744 (6476, 2013]
IQM
1.637 [1.478, 1,789)
1,556 [1.31, 1.81]
1,588 [1,418, 1,749]
1.613 [1,456, 1,767]
1,563 (1.29, 1.848)
(1,296, 1,746]
Median
1.616 [1.494, 1.743]
1.541 [1.374, 1.745]
1.605 15,451, 1.709)
1.606 [1 [1.477 1.721]
1.584 [1.388, 1.78)
1.546 (1,362, 1.681)
Meun
1.617 [1.518. 1.718]
1.562 [1.393, 1.73]
1.571 (1.467, L6675]
1.594 [L488, 1.699]
1,583 [14055 1,757]
1.52 [1.377, L.699]
46

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
K.2. Deepmind Control Suite- Easy
Table 24. DMC-Easy (Input Design). Final performance 1M environment steps averaged seeds. The [bracketed values]
represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean are computed over the default reward.
Task
SimbaV2
No Normalize
No Shifting
Cahuft :
Resize Projection
aczobat- swingut
436 [391, 482]
385 [364, 406]
399 (276. 522]
466 (402, 5300
293 [184, 4011
tall-in-cup-cated.
982 [980, 984
564 (92. 1036]
586 [145. 1026]
983 (979.
983 (979, 986]
cartpole-balancee
999 (999, 999]
999 (999, 999]
999 1999, 999]
999 (999. 999]
999 (999, 999]
co:ape.le-ballanse--arae
967 [904, 1030]
1000 [1000, 1000)
992 (978, 1007]
1000 [1000, 1000]
1000 [1000. 1000)
caripsieesswinguu
860 (876, 883]
881 [881. 882]
881 [880. 882]
882 [881, 882]
799 [646, 951]
848 [848, 849]
838 1829, 846]
829 7796, 862]
846 [843, 848]
803 [719, 888]
chect-ah-run
920 (918, 922]
499 (426, 572]
519 [477, 560]
919 [914, 924]
917 [914, 920]
finger-spin
891 [810, 972]
4485,
620 [404, 836
855 [674. 1036]
883 [737. 1030]
finger-turn-easy
953 [925, 980]
922 [858, 986]
924 [831, 1017]
922 [874, 970]
925 [866, 985]
Fingar-turn-hard
951 [925, 977]
888 1796, 980]
937 1895, 979]
870 [861, 879]
917 [864, 970]
fish-swim
826 [806, 846]
(316,
442 [326, 558]
806 [783,
758 [706, 811]
hopper-hop
290 [233, 348]
347 1270, 424]
208 [132, 284]
212 779. 345]
350 [191. 5091
hoaper-stand
944 (926, 962]
804 [578, 1031]
680 [444, 915]
925 [883, 967]
753 [440) 1066]
pandulum-swinngup
827 [805, 849)
1011]
620 (213. 1026]
[762,
678 [453, 902]
quadruped-ron
935 [928, 943]
937 [923, 950]
917 (902, 932]
900 [850, 949]
935 [927, 943]
qusdraped-welk
962 (955, 969]
(949, 975]
963 (958, 967]
960 (945, 974]
957 [947. 967]
reacher-easy
983 [979, 986]
970 [952, 989]
984 [982. 985]
982 [980, 985]
reacher-hard
967 [946, 987]
881 [707, 1054]
935 [890, 979]
973 [966, 979]
975 1972, 978]
walker-run
817 [812, 821]
762 (672, 853]
793 [778, 809]
816 [81 820)
813 [809, 816]
walker-stand
987 [984, 990)
(984,
987 [983, 991]
986 (978,
979 [967. 991]
walker-walk
976 1974, 978]
974 1968, 981]
978 (975, 981]
976 (973, 980]
969 [963, 975]
IQM
0.933 [0.918. 0.948]
0.874 [0.795. 0,922]
0.871
[0.789, 0.921]
0.919 [0.891. 0.942]
0.92 (0.888, 0.946]
Median
0.875 [0.847, 0,905]
0.787 (0.717, 0.839]
0.781 [0.712, 0.837]
0.867 [0.818, 0 0.905]
0.844 [0.792, 0.892]
Mean
0.874 0.849, 0.898]
0.779 70.722, 0.832]
0.771 10.713. 0.826]
0.862 [0.819. 0.9]
0.842 (0.794. 0.885]
47

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforeement Learning
Table 25. DMC-Easy (Output Design). Final performance at 1M environment steps averaged over sceds. The [bracketed values]
represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean are computed over the default reward.
Task
SimbaV2
MSE Loss
No Reward Scaling
No Return Bounding
Hard Target
aczobat-awingup
436 [391, 482]
384 [273,
383 [329] 438)
439 [386, 492]
423 [350, 496)
982 [980. 984]
982 (979, 985]
981 (978. 984]
983 (979. 986]
983 [979. 986]
cartpole-balance
999 [999, 999]
999 (998, 1000]
999 [999, 999]
694 [659, 730]
999 [999, 999]
967 [904, 1030]
1000 [1000, 1000)
970 [913, 1027]
998 (994, 1001]
1000 [1000, 1000]
caripole-swicguup
860 [876, 883]
881 [880, 882]
881 [880, 881]
758 [737. 779]
881 [880, 882]
car7pole-swinggg--aae
848 [848, 849]
845 [843, 847]
715 [493, 937]
844 [839, 849]
846 [844, 848]
cheetssh-run
920 [918, 922]
796 (563, T030]
869 [813, 925]
[829, 946]
905 1873, 937]
finger-spin
891 [810, 972]
959 (937. 980]
824 [684, 963]
774 [632. 916]
954 [919. 989]
finger-turn-eag
953 [925, 980]
973 [965, 981]
916 [873, 959]
970 [964. 976]
970 [964, 977]
finger-turn-hard
951 [925, 977]
1793, 978]
853 [761, 945
[861, 972]
966 [958, 973]
fish-swim
826 [806, 846]
838 (827, 849)
844 (834, 853]
819 [790. 847]
809 [792. 826]
hopper-hop
290 [233, 348]
380 1988, 561]
294 [237. 350]
211 [114, 309]
316 [293. 339]
hooper-stand
944 [926, 962]
920 [858, 982]
928 [897, 959]
710 [464, 956]
918 [862, 973]
pendujum-swinguu
827 [805, 849]
773 [754, 792]
814 [776, 852]
809 [763, 854)
809 [760. 858]
quadruped-run
935 [928, 943]
938 [914, 962]
929 [901, 956]
923 [903, 944]
943 [930, 956]
quadruped-walk
962 [955, 969]
(943, 967]
968 (960, 976]
(949, 968]
964 (953, 974]
zeacher-easy
983 [979. 986]
968 [944, 992]
966 [938, 994]
983 [981. 985]
983 [982. 9851
reacher-hard
967 [946, 987]
978 (977. 979]
976 [972, 981]
976 1972, 981]
969 [949, 989]
walker-run
817 [812, 821]
[793, 797]
818 [815, 820]
732 [687.
817 [812, 821]
walker-stand
987 [984, 990]
992 (990, 993]
987 [983, 991]
945 [912. 978
986 (979. 994]
malker-walk
976 (974, 978]
977 (974, 980]
974 [970, 978]
972 [966. 979]
978 (976. 981]
1QM
0.933 [0.918. 0.948]
0,928 [0.894] 0,954]
0.914 [0.89. 0.937)
0.887 [0.843, 0, 0.922]
0.938 [0.911, 0.96]
Median
0.875 (0.846. 0,905]
(0.813, 0.918]
0.853 [0.808, 0.896
0.823 [0:777, 0.871]
0.878 [0.836, 0.918]
Mean
0.874 [0.848. 0.898]
0.868 [0.821. 0.909]
0.852 [0.814. 0.888]
0.824 [0.78, 0.865]
0.878 [0.838, 0.913]
Table 26. DMC-Easy (Training Design). Final performance at IM environment steps averaged over 3 seeds. The [bracketed values]
represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean are computed over the default reward.
Tisk
SimbaV2
No L.R. Decay
Suste :1
Secate 1
Olesie : 0.5
Oecode :1
screbot-swingg
136 [391, (82)
393 [269, 517]
490 [153, 527]
448 [100, 188]
452 [359, 5:15
449 [391,
ball--n-cup--ae
982 1980,
982 8978, 986
982 (980, 984
982 (980, 984]
(979, 987]
983 (979, 987]
ca cartpole- 1bal ance
999 (999. 989)
999 (199), 909]
999 (959. 999]
999 (999), 999]
999 (999, 999]
999 (999) 999]
cactpole-bast anncelnneeeee anca- sparse
967 [90+. 1030]
1000 [1000, 1000]
999 1997. 1000)
1000 [1000, 1000)
1000 [1000. 1000)
1000 [1000, 1000)
castpols-soiingus
[876,
882]
[881,
[881, 882]
[881,
881 [88
:ca cartpole-sse inniinngpppp rse
848 (548, 849)
699 (409) 9009
847 (846, 848]
787 (697, 878)
848 [847, 849]
847 [845, 849)
cheetssh-run
920 [918, 92.]
917 1913, 922]
917 1913, 921]
914 1903, 925]
918 [914. 921]
866 [758, 974]
=inges-spin
[8100,
871 1716, 102.5
[941,
[813, 9771
(921, 9791
928 1004]
Finger-tusn-eass
953 (925, 980)
895 [846, 943]
945 (9044 986
955 [931. 979]
864 (294, 934]
878 [800) 956]
Finges-turn-hssdd
951 [925, 977]
869 [755, 982]
941 [910, 971]
959 (937. 981]
939 (895. 983]
943 [898, 988]
fish-swim
(506,
1766, 844]
(804,
821 [792, 850]
[759,
825 8351
hoppaz-hop
290 (233, 348)
321 [305, 338
282 (217. 347]
304 (265, 343]
171 [59 2831
313 (293. 333)
hopper-st
944 [926, 9662]
922 (882, 9633
840 2705, 9776)
879 (730, 1027]
929 1882, 976]
622 [240) 10044
pendulum-eviae uugnggu
[505,
810 (763, 858]
827 [806,
1797, 8131
[766,
[767. 857]
quadrupad-zun
935 (928, 9431
940 1923, 958]
929 19211 9377
929 (910, 948]
931 [911, 952]
930 [922. 938]
quadruped-wal1 1
962 1955, 9W9)
963 (1622) 9733
966 (9622 971]
946 (139) 969]
953 (946, 961]
954 [9488 9593
zescheg-eazy
(979,
1979, 9851
(000,
683 1981,,986 986]
(980,
981 1981
zeachez-hazd
967 (966, 9871
977 1971, 9821
949 (920, 9771
858 (933, 9831
973 1968, 9791
970 [958, 982]
wa Iaaa-r-run
817 [812, 821)
816 (810, 822]
814 (810, 818]
819 (817. 821]
819 [817, 822]
819 [817. 821]
walkes-stand
987 1984, 990)
987 1986, 989]
990 (988, 991]
988 (987. 990]
989 (985,
990 (988. 993]
wa kel
976 [974, 978]
976 (9600 984]
975 (972, 979]
977 (973, 981]
975 (974, 976]
977 (974, 981]
IQM
0.933 [0.918, O 0.948]
0.923
[0. 8994 0.947]
0,932 (0.916, O 0.947]
0.934 (0.918, 0.949)
0,927 [0 (0.901, 0.5499
0.922 (0.892, 0.947)
Median
[0.8477 0.9041
0,858 10 0.902]
[0.8 (0.847.00 0.906]
0,871 [0.812 0.902]
0.863 0.911]
0.839 [0.809, 0.902]
Mean
0.874 [0.849. 0.897]
0.858 [0.813, 0.896]
0.873 (0.848, 0.897]
0.87 [0.843, 0.894)
0.866 [0.819, 0.905)
0.856 [0.812, 0.8951
48

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
K.3. Deepmind Control Suite- Hard
Table 27. DMC-Hard (Input Design). Final performance at 1M environment steps averaged over 3 seeds. The [bracketed values]
represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean are computed over the default reward.
Task
SimbaV2
No L2 Normalize
No Shifting
Cabeye 1
Resize Projection
dog-run
562 (516, 608]
573 [515, 632]
530 (415, 644]
610 [575, 645]
541 [418, 664]
dog-atand
981 (977. 985]
953 [931, 975]
963 [955, 971]
964 [939, 989]
977 [970, 985]
dog-LcoL
861 (772, 950]
813 [747. 878]
823 [753, 892]
772 [640, 901]
875 [842. 907]
dog-wa
935 [927, 944]
895 [868, 922]
[894, 909]
938 (933, 943]
[899. 929]
humanoid-rop
194 [182, 207]
189 [173, 205]
236 [213, 259]
182 [169, 196]
204 [177, 231]
humanoid-staad
916 [886, 945]
716 [341, T092]
913 [895, 932]
876 [794, 958]
904 [886, 921]
humanoid-walk
651 [590, 713]
756 [695,811
700 [553, 846]
683 [541, 824)
621 [585, 658)
IQM
0.808 (0.726, 0.88]
0.789 [0.65, 0.868]
0.805 [0.667, 0.893]
0.783 [0.663, 0,882]
0.795 [0.659, 0.894]
Median
0.729 (0.655, 0,808]
0.732 [0.595, 0,816]
0.73 (0.619, 0.826]
0.717 [0. (0.606, 0.823]
0.724 [0.61, 0.825]
Mean
0.729 (0.665, 0.79]
0.7 (0.6, 0.795]
0.724 [0.629, 0.814]
0,718 [0.619, 0.809)
0.72 [0.619, 0.811]
Table 28. DMC-Hard (Output Design). Final performance at IM enviromment steps averaged over 3 seeds. The [bracketed values]
represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean are computed over the default reward.
Task
Sinbu
MSE
No Reward Scaling
Return Bounding
Hard Target
dog-run
562 [516, 608)
545 [450, 639]
478 [402, 554]
617 [537, 696)
676 [654, 699
doq-stand
1977, 985]
976 [959, 993]
[956, 978]
969 (958, 981]
[971, 989]
dog-trot
861 [772, 950]
841 [796, 886]
737 [614, 859]
884 [803, 964]
848 1763, 934]
dog-walk
935 [927, 944]
905 1883, 928]
925 [915, 935]
922 [899, 945]
928 [891. 964]
bumanoid-zun
194 [182, 207]
173 [146, 200]
237 [181, 293]
182 [154, 209]
209 [159, 260)
humano commannod-sLanne
916 [886, 945]
786 (612, 960]
879 [821, 936]
851 [744, 958]
928 [920, 937]
humanoi d-walk
651 [590, 713]
729 (577, 880]
[643, 865)
706 [563,
645 [602, 689]
IQM
0.808 (0.728, 0.881]
0.78 [0.62, 0.877]
0.778 [0.655, 0.871]
0.812 [0.69, 0,901]
0.814 [0.706, 0,902]
Median Mean
0.729 [0.655, 0.809]
0.705 [0.57, 0.836]
0.715 [0.61, 0.817]
0.731 [0.621, 0.84]
0.746 [0.641, 0.847]
0.729 [0.665, 0.79]
0.708 [0.586, 0.813]
0.712 [0.626, 0.792]
0.733 [0.632, 0.824]
0.746 [0.648, 0.833]
Table 29. DMC-Hard (Training Design). Final performance at 1M environment steps averaged over 3 seeds. The [bracketed values]
represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean are computed over the default reward.
Tisk
SimhaV2
No L.R Decay
Siaie :1
Recate
Ociect 0.5
Orccae
dog- cun
[S16,
516 [407, 625
586 621]
543 598]
556 [420, 691]
[493, 599]
dog-stand dog-tcot
981 [977, 985]
957 (933, 980)
976 (970, 982)
972 (964, 9979
951 [912] 989]
979 (969. 989)
861 [772, 950]
770 [663, 878
836 [756. 916]
850 [798, 901]
870 [842, 898]
814 [692. 937]
dog-walk
935 [927, 944]
927 [919, 935]
938 [921, 955]
949 (937. 960]
924 [903, 945]
931 [921, 942]
humanoid-ris
194 [182, 207]
221 [165, 273]
187 [178, 196]
194 [177, 211]
193 [170, 217]
182 [172, 192]
hunamooddsste
[886,
932 (916,
874 [812, 936]
823 [749, 897]
900 (877.924) 924]
918 (892,
hunanof d-walk
651 [590, 713]
706 (590, 802]
624 [592, 656)
610 (589, 630)
697 [583, 812]
622 [6111, 633)
IQM
0.808 (0.725. 0.879)
0.798 [0.658, 0.895]
0.783 (0,708, 0.851]
0.766 (0.687, 0.838]
[0.685, 0.892]
0.781 [0.642, 0:891]
Median
0.729 (0.655, 0.808)
0.716 (0.616, 0. 822)
0.719 (0.646, 0.794]
0.711 [0.634, 0.782]
0.724 (0.62, 0.833]
0.715 (0.605, 0.823)
Mean
0.729 [0.664. 0.791]
0.719 [0.623, 0.809]
0.718 [0.656, 0.777]
0,706 [0.644, 0.767]
0,728 [0.627. 0.819]
0.714 0.81
49

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforeement Learning
K.4. Myosuite
Table 30. Myosuite (Input Design), Final performance at IM environment steps averaged over 3 seeds. The [bracketed values] represent
a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean are computed over the default reward.
Task
SimbuV2
No L2 Normalize
No Shifting
Cstiitt 1
Resize Projection
myo-key turr.
1.0 [1.0, 1.0]
1.0 (1.0, 1.0]
[1.00 1.0]
[1.00 1.0]
[1.0, 1.0]
syo-key-tuse Beo-keey-tu-baaae
0.62 [0:427.0.8) 0.813]
0.325 50:009. 0.659]
0.25 [-0.444 0.544]
[0.661. 0,939]
0.85 (0.752, 0.948)
myo-ob1-hold
1.0 [1.0 1.0]
[.00.0 1.0]
[1.0 1.0]
[.00 1.01
[1.00 1.0]
myo-obj-hold-hare
0.98 (0.954, 1.006]
0.975 (0.926, L.024]
0.975 (0.926, 1.024]
[... 1.0]
0.975 [0.926, 1.024]
myo-pen-twirrl
[1.0 1.0]
[1.0 1.0]
[.0 1.0]
1...1.0 1.01
1.0 [1.0, 1.0]
myo-pen-twirr-hare
0.93 (0.888, 0.972)
0.9 [0.82, 0.98)
0.875 (0.689, 1.61]]
0.975 [0.926, 1.04]]
0.8 [0.604, 0.996]
myo-posa
1.0 [1.0, 1.0]
1.0 [1.0, 1.0]
[1.00 1.0]
1.0 1.0J
11.0, 1.0]
nyo-posa-hardd
0.0 (0.0.
0.0 10.0. 0.0]
0.0 0:0)
0.010.00 0.0]
0.0 [0:0. 0.0]
myo-reach
1.0 (1.0. 1.0]
[1.0. 1.0]
[1.0 1.0]
[1.00 1.0]
1.0 [1.0. 1.0]
myo-reach-hard
0.94 (0.873, 1.007]
0.95 [0.893, 1.007]
[0.82, 0.98)
0.925 (0.831, 1.019]
(0.82, 0.98]
IQM
0.99 (0.968, 1.]
0.98 [0.885, 1.0]
0.98 [0.86, 1.0]
10 (0.955, 1.0]
0,975 [0.925, 1.0]
Median
0,845 [0.78. 0.925]
0.815 [0.695, 0.935]
0,805 [0.68, 0.92]
0,875 [0.765, 0.975]
0,86 [0.75, 0.955]
Mean
0.847 [0.782. 0.906]
0.815 (0.702. 0.915]
0.8 [0.682, 0,905]
0.87 (0.77, 0.953]
0.852 [0.75, 0.938]
Table 31. Myosuite (Output Design). Final performance 1M environment steps averaged over 3 seeds. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean are computed over the default reward.
Task
SimbuV2
MSE Loss
No Reward Scaling
No Return Bounding
Hard Target
myo key turn.
1.0 [1.0, 1.0]
1.0 [1.0, L.)
1.0 [1.0, 1.0]
[1.0 1.0]
1.0 [1.0, 1.0]
myo-key-turr-hard
0.62 [0.427, 0.813]
0.2 LO 0.550
0.76 [0:66, 0186)
0,225 LO 153, 0.603]
0.675 [0.417. 0.933
myo-ob) hold
[.0 [1.0, 1.0]
1.0 [1.0, 1.0]
1.0 [1.0, 1.0]
1..0 1.0]
11.0, 1.0]
myo-obj-hold-hag d-hard
0.98 [0.954, 1.006]
0.933 [0.868, 0.999]
0.98 [0.941, 1.019]
L.0, 1.0]
0,975 [0.926, 1.024]
myo-pen-twirrl
1.0 (1.0, 1.0]
0.667 (0.013, 1.32]
[1.0, 1.0]
1.0 (1.0. 1.0]
[1.0. 1.0]
myo-pen-twirl-haree
0.93 (0.888, 0.972)
0.867 (0.605, 1.128)
0.98 [0:941. L.019]
[0.82, [0.82.0.98]
[0.82,
myo-pose
1,0 [1.0. 1.0)
1.0 (1.0. 1.0)
0.8 [0.408. 1.192)
[1.0, 1.0]
1.0 [1.0. 1.0]
myo-posa-hard
0.0 (0.0. 0.00
0.0 [0.0. 0.0]
0.0 [0.0. 0.0)
0.0 [0.0. 0.0)
0.0 [0.0. 0.0
myo-reach
[10.,
(1.0.
1.0 [1.0. 1.0)
1.0 [1.0, 1.0]
10 (1.0, 1.0]
myo-reach-
0,94 [0.873, 1.007]
0.9 [0.787, 1.013]
0.88 [0.766, 0.994]
0.925 [0.831, 1.019]
0,925 [0.831. 1.019]
IQM
0.99 [0.968. 1.0]
0.944 10.712, 1.0]
0.985 [0.931, 1.0]
0.985 [0.87, 1.0]
0,98 [0.93, 1.0]
Median
0.845 [0.78. 0.93]
0.77 [0.58, 0.94]
0.85 [0.74, 0,96]
0.79 [0.68, 0,93]
0.845 [0.74, 0,955]
Mean
0.847 [0.783, 0.906]
0.757 [0.613, 0.887]
0.84 [0.746, 0.922]
0,805 [0.685, 0,912]
0.848 [0.745, 0.938]
Table 32. Myosuite (Training Design). Final performance at 1M environment steps averaged over 3 seeds. The [bracketed values] represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean are computed over the default reward.
Task
SimbuV2
No LR Decay
Sun =1
Escate
Clinet 0.5
Carsle
myo-key-ture
[..0 1.0]
11.0 L.0]
[... 1.0]
U.Y (0,704, 18966]
L.0 1.0g
[.0 1.0)
myo-key-torn-har
0.62 (0.427, 0.813
0.65 (0:345, 09555
0.74 (0:585 (3895)
(0,502,
0.675 0.760)
0.85
[10662, 1.08)
myo-ob?-han
1.0 [1.0. 1.0]
[1.0 L.]
1.0 (1.0, Loj
1.0 (L.0, 1.0]
[.0 1.0)
1.0
[100. 1.0)
myo-ob/-hold-an
0.98 [0.954, 1.06)
1.0 [1.0, L.0
0.95 10.897, 1.0031
0.98 [0.954, 1.006]
0.95 [0:893, 1.0071
1.0
[1.0,
1.01
mys-pen-twiel
[1,00 10]
0.75 10.26,
1.0 [1.0, L.]
1.0 [1.0. 1.0
[L., 1.0]
[1.0.
1.0)
mye-pan-twiel--ae
0.93 (0.888, 0.972)
0.725 (0.451.
0.97)
[0:816,
[0.831, L019)
1.0)
nyo-pose
1.0 (1.0) 1.0]
1.0 (1.0. 1.0]
1.0 [1.0, 1.0]
1.01 [L., 1.0
1.0 1.0j
1.0
1.0, 1.0)
nyo-pose-hard
0.0 (000 000
0.0 (0:0, D.0
0.0 (ORD) 0:0
BJD [000 0.0
D.D (100000)
D.0 1000 (10
myo-resch
L.0
[1.0, L.0
1.0
1.0]
avo-reach-haaee
0.94 (0,873, (007)
0.925 10,811, 1.019]
(0,928, 1/012J
0.91 [0:848, 0.923
0.875 707752 0.908)
[0.82, 0.98)
IQM
0.99 (0.966 1.0]
0 (0,855 1.0)
0.99 (0: (0.964, 1.4J
08982 (0.948, 1.0)
0.975 10.905, 1.0)
1.0 (0.97, 1.0)
Mediun
(0.78. 0.925]
0.84 [0.7, 0.94]
[0.785,, 0.9355
10.77. 0.921
(0.735, 0,9451
00.77.0.88)
Mean
0.847 (0,782, 0.906)
0.81 [0.698, 0.908)
0.855 (0,792, 0.9913
0.336 [0.77, 0.896]
0.842 (0.742, 0,928)
0.755 (0.772, 0.966
50

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
K.S. Humanoid Bench
Table 33. HumanoidBench (Input Design), Final performance at IM environment steps averaged over 3 seeds. The [bracketed values]
represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile
computed over the default reward.
Task
S2
No 1.2 Normalize
No Shifting
1
Resize Projection
h1 hh h1-balanco-han lnccl-tar----
143 [128, 157]
113 [62, 164]
94 [80, T09]
89 174, 104]
150 [105, 194]
a1--eelance--iiple--00
723 [651, 795]
760 (666, 854)
512 (236, 789]
799 [744, 854]
760 [658, 861]
h1 b1-crawl-v0
946 [933, 959]
901 [870, 952]
838 (813, 864]
948 [927, 968
884 [723, 1044]
al-hurdle-vG
202 [167. 236]
177 [168, 187]
172 [162, 183)
191 [160, 222]
218 [152. 285]
b1-maae-vv
313 [287, 340]
337 [313, 361]
302 [230, 374]
[337, 3451
[311,
hl-pole-vo
791 797]
746 (722, 769]
759 [754. 764]
784 [781, 786]
805 (795, 816]
h1-reach-v0
3850 [3272, 4427]
4564 [3107, 6020)
3263 [2743, 3783]
5415 [4354, 6475]
3722 [2105, 5339]
bi-ruz-vo
415 [307. 524]
212 1208. 217]
218 240]
365
585
h1-sit-hard-v0
679 [548, 81
493 [227, 758]
721 [499, 944]
757 [667, 847]
667 [402, 933]
hl-oit-simple-vl
875 [870, 880]
829 [783, 874]
803 1745, 860)
869 (858, 880)
890 (872, 909)
h1-g1-da-so
487 [404, 571]
518 (499, 537]
497 [483, 511]
520 [467, 573]
435 [318,
hl-stair-v0
493 [467. 518]
477 (460, 494]
475 (433, 517]
540 (480, 600)
315 [504, 525
h1-
814 [770, 857]
821 [786, 856]
835 (833, 837]
848 [845, 851)
764 [661,
hl-welk-v0
845 [840, 850)
675 (560, (89)
832 (814, 850)
842 [830, 853)
845 (835, 855]
IQM
0.799 [0.684. 0.907]
0,709 [0:525, 0.881]
0.708 (0:509, 0.908]
0.833 [0.635, 0.997]
0.808 0,977]
Median
0.781 [0.69, 0.862]
0.698 (0.566, 0 8585]
0.698 (0 (0,552, 0 8.85]
0.801 [0.639, 0.944]
0.801 [0.631, 0.934]
Mcan
0.776 (0.704. 0.847]
005589, 0.833]
[0.578. 0.825)
0.791 0.921]
(0.653, 0.905]
51

---

SimbaV2: Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Table 34. HumanoidBench (Output Design), Final performance at IM environment steps averaged over seeds. The [bracketed values]
represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean are computed over the default reward.
Task
SimbaV2
MSE Loss
No Reward Sealing
No Return Bounding
Hard Target
143 [128, 157]
89 [73. 106]
92 [82. 101]
136 [117. 155]
173 [133, 2131
Daalance-ssmgge
723 [651, 795]
765 [664, 865]
779 (680, 877]
832 [825, 839)
739 [553, 924]
hl-crawi-vo
946 [933. 959]
945 (921, 969]
953 (934, 971]
942 (896, 989]
938 [913. 964]
202 [167,, 236]
243 [241, 245]
140 [105, 174]
192 [132,
207 [186, 228]
bi-meze-vo
313 [287. 340]
324 [282, 367]
273 (220. 327]
320 [291, 350]
339 [309. 370
h1 h1-pole-vo
791 [785, 797]
774 [773, 776]
724 [625, 823]
796 [791, 800]
787 [786, 788]
h1-reach-v0
[3272, 4427)
[2733, 7355]
[2583, 5480]
4396 [2059, 6732]
3984 [2231. 5738]
h1-run-vo
415 [307, 524]
485 [199, 771]
359 [150, 568]
360 [234, 487]
341 [175, 508]
hl-sit-hard-v0
679 [548, 811]
611 [350, 873]
729 (648, 811)
727 [681, 773
684 [362, 1005]
h1-sit-sinn b1-ait-simmle-vv
875 [870, 880]
869 [864, 875]
871 [859, 883]
868 [863, 872]
871 [870, 871]
bl-slide-v0
487 [404, 571]
352 (260, 445]
410 [313, 506
534 [484, 583]
581 [550, 613]
h1-stair-v0
493 [467. 518]
512 [470, 554)
450 3322, 517]
510 (495. 526]
515 [495, 536]
bl-stand-v0
814 [770. 857]
742 [621, 862]
775 (684, 865)
751 [570, 933]
645 [25]. 1038]
h1 ww.kaakk-. v0
845 [840, 850]
851 [840, 863)
742 [556, 928]
845 [834, 855]
838 [835, 842]
IQM
0.799 [0.684, 0 0.907]
0.778 (0.594, 0.95]
0.745 (0.484, 0. 096]
0.819 (0.623, 0.977]
0.778 [0.572, 0.965]
Median
0.781 00.69..00 0.863]
0.762 (0.614, 0.917)
0.722 (0.605, 8646
0.776 (0.636, 0.933]
0.772 [0.614, 0.923]
Mean
0,776 (0.704, 0.848]
0.767 [0.637, 0.894]
0.735 (0.629, 0.842]
0.787 [0.658, 0,915]
0.77 [0.64, 0.897]
Table 35. HumanoidBench (Training Design). Final performance 1M env steps averaged over 3 seeds. The [bracketed values] represent 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean are computed over the default reward.
Tisk
SimhaV2
No LR Decay
Rouit
Recale
0.5
Qurate
a1 aal-a-ance--hacee
143 [128, 157]
118 186, 150]
139 [119, 159]
139 [116, 162)
182 [440, 164]
139 [119, 159]
bll-ba-ance-sinn 1e-vie-vo
723 [651, 795]
812 [258, 867
815 [794, 836)
813 [793, 8331
763 [620, 907)
732 [645, 820]
b1-erawi-VC
946 [933,
956 (94),
946
933
962 (954,
a1-hurdle-v0
202 [167, 236]
150 [64, 235]
228 (218. 238]
188 (159, 218)
203 (187.199 219]
200 [204. 214]
>11-mane-vD
513 [287. 340)
283 [150, 417]
347 [399, 355]
337 [318. 336)
167 1552, 582]
333 1324 342]
h1-poie-v0
791 [785, 797]
736 [625, 847]
774 (760, 788]
767 [736. 798)
791 (777.801) 80+1
792 [785, 798]
a1-reach-vt
4427)
8430)
5187]
(42511
7207]
a1-run-v0
415 [307, 524]
424 [172, 675]
357 [277. 136)
351 (216, 457)
234 209. 2591
337 [171, 503]
a1l-ait-haxd-vl
679 [118.811]
678 [382, 975]
686 (575. 796)
734 (604, 8633
507 (257, 757)
702 (493, 911]
a31-sit-simpee-v
875 (870, 880]
845 (793, 897]
875 [869, 88.1
877 [871, 882)
868 8852,
870 (868, 872)
h1--1:deevt
487 [404,
(268: 5007
(420, 553)
447
[512,
h1-ssair-v6
493 [467. 518]
$12 [499, 525]
507 [491. 523]
504 [483, 525)
513 [505, 521]
503 [468, 538]
h1-stand-vC
814 [770, 857]
603 (434, 275
751 [697) 804
789 (700), 879)
793 (707, 878]
799 [720) 877]
a1-va_k-v0
[8400
[832, 855]
(49)
850)
834 (829,839)
[802,
1QM
0.799 (0.6833 0.908)
0.764 (0:578, 0.93)
0.796 (0.688, 0.901)
0.823 (0.709, 0.927]
0.725 (0.532, 0.917]
0.817 (0.615, 0.978)
Mediun
0.781 (0.69, 0.86)
0.769 (0.599, 0.912]
0.776 [0.7, 0.866]
0,792 10.705, 0.876]
0,746 (0.595, 0.893
0,802 (0.64, 0.M38)
00,704.0.847
(0.625, 0.883]
0.78L [0.711, 0.852]
0.789 (0.719, 0.861]
[0.619, 0.872]
(0.659, 0.916]
52


Reviews:
[{"review_id": "ScnFGfEsVE", "reviewer": "Reviewer_R7VW", "summary": "The paper introduces SimbaV2, an RL architecture that stabilizes training and improves scalability through hyperspherical normalization and distributional value estimation with reward scaling. Built on Soft Actor-Critic (SAC), it achieves state-of-the-art performance across 57 continuous control tasks and scales effectively with increased model size and compute. Experiments confirm its stability, outperforming existing RL methods without requiring periodic weight reinitialization.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer R7VW,  \n\nThank you for your thoughtful and constructive feedback. We address your concerns in detail below and would be happy to clarify any remaining questions.\n\n\n> **Question 1.1**\nIt would be helpful if the authors could provide more theoretical justification or intuitive explanations for why previous methods struggle to scale while SimbaV2 does.\n\n> Why does hyperspherical normalization stabilize training and eliminate the need for weight reinitialization?\n\nWe appreciate this insightful question. Our use of hyperspherical normalization is motivated by empirical observations in SimbaV1, where we identified unstable growth in key quantities—including feature norm, parameter norm, gradient norm, and effective learning rate per layer (Fig. 4). These instabilities are known to impair generalization and reduce plasticity in RL.\n\nSpecifically:\n1. **Feature Norm:** Prior work [1] shows that TD loss induces an implicit bias toward growing feature norms, which can cause overfitting and reduced feature rank. This is often driven by a few dominant dimensions, leading to loss of plasticity [2]. Techniques such as feature norm regularization [1] and hyperspherical normalization [3,4] were proposed to mitigate this.\n2. **Parameter Norm and Effective Learning Rate:**  As parameter norms grow, the effective learning rate (gradient norm divided by parameter norm) declines, which hampers gradient flow and learning dynamics [5,6]. In SimbaV1, we observed this specifically in the encoder, where the effective learning rate collapses over time (Fig. 4e). Simply increasing the global learning rate is not viable, as other layers (e.g., the predictor) may already be operating at high effective rates.\n\nSimbaV2 addresses these challenges through the following design choices:\n1. **Feature Norm:** Hyperspherical normalization to control feature norm growth.\n2. **Parameter Norm:** Weight projection onto a hypersphere to control parameter norm.\n3. **Gradient Norm:** Distributional critic with reward scaling to regulate gradient norm \n\nTogether, these mechanisms ensure stable learning dynamics (i.e., stable effective learning rate across layers) and sustained plasticity, thereby eliminating the need for weight reinitialization. We recognize that our original draft did not clearly explain these intuitions and will revise the introduction accordingly to highlight these design motivations.\n\n- [1] DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization, Kumar et al, ICLR'22.\n- [2] Understanding and Preventing Capacity Loss in Reinforcement Learning, Lyle et al, ICLR'22.\n- [3] Is high variance unavoidable in rl? a case study in continuous control, Bjorck et al, ICLR'22.\n- [4] Dissecting Deep RL with High Update Ratios: Combatting Value Divergence, Hussing et al, RLC'24.\n- [5] Loss of plasticity in deep continual learning, Dohare et al, Nature'24.\n- [6] Normalization and effective learning rates in reinforcement learning, Lyle et al, NeurIPS'24.\n\n\n> **Question 1.2**\nCould the authors disclose the computational resources used for this project, including hardware specifications and training time?\n\nAll experiments were conducted using NVIDIA RTX 3090 GPUs and an AMD EPYC 7402 24-Core Processor.\n\nFor wall-clock time, a single run of SimbaV2 with UTD=2 typically takes 1.6 hours. This varies by environment: around 1.0 hour for simpler tasks like Cartpole (no collisions, minimal joints) and up to 2.5 hours for more complex tasks like Dog (many joints and intricate interactions).", "claims_and_evidence": "The paper provides strong empirical evidence to support its claims. The experiments are well-structured, covering scalability, stability, and performance comparisons across 57 continuous control tasks. The ablation studies confirm the importance of hyperspherical normalization and reward scaling, reinforcing the paper’s core contributions.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for the problem. Hyperspherical normalization and reward scaling directly address RL's instability and overfitting issues, making them relevant for scaling RL effectively. The evaluation is thorough, using continuous control tasks across four standard benchmarks (MuJoCo, DMC, MyoSuite, HumanoidBench), ensuring broad applicability. Comparisons with strong baselines and ablation studies further validate the approach.", "theoretical_claims": "This paper does not contain theoretical proofs.", "experimental_designs_or_analyses": "I checked all the experimental designs and analyses, and there are no issues.", "relation_to_broader_scientific_literature": "The paper builds on prior work in RL scalability, regularization techniques, and normalization methods. It extends ideas from weight decay, dropout, and layer normalization, commonly used in supervised learning, by introducing hyperspherical normalization to stabilize RL training. The distributional value estimation approach aligns with prior work on distributional RL, enhancing gradient stability. Additionally, the paper addresses challenges seen in periodic weight reinitialization methods by providing an alternative that scales without overfitting. It contributes to the broader discussion on scaling laws in RL, challenging the notion that increasing model size and computation necessarily leads to instability.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper's experimental design is a strong point, with thorough evaluations across multiple benchmarks and well-structured ablation studies that clearly isolate the contributions of each component. The writing is also clear and well-organized, making it easy to follow the ideas and their significance. I enjoyed reading this paper.", "other_comments_or_suggestions": "It would be helpful if the authors could provide more theoretical justification or intuitive explanations for why previous methods struggle to scale while SimbaV2 does. Specifically, a deeper discussion on why hyperspherical normalization stabilizes training and why it eliminates the need for weight reinitialization would strengthen the paper’s contributions.\n\nCould the authors disclose the computational resources used for this project, including hardware specifications and training time? This information would be valuable for the community to better understand the practicality and scalability of the approach.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i4rYKIhdsZ", "reviewer": "Reviewer_ZJka", "summary": "This paper proposes SimbaV2, an improved version of Simba, by replacing several key components of Simba with a scale-preserved l2-normalization (i.e., hyperspherical normalization), distributional value function approximation, reward scaling, etc. The authors presents a comprehensive experimental study with 57 continuous control tasks across 4 domains, against a wide range of existing online RL methods. The experimental results demonstrate the superiority of SimbaV2 in scaling with larger networks and higher UTDs effectively, along with careful analysis on learning dynamics metrics and design choices.\n\n\n## update after rebuttal\n\nI've read all the other reviewers' comments, and the authors' rebuttal provided additional experimental evidence that addressed my questions well. Therefore, I will keep the rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ZJka,\n\nThank you for your constructive feedback and positive support! We respond to each of your points below and would be happy to clarify further if needed.\n\n\n> **Q2.1:**\n> The necessity or effects of the design changes “Linear → Linear + Scaler” and “Residual Connection → LERP” are not supported with direct evidence.  \n> Are there experiments for evaluating the effects of the design choices “the inverted bottleneck MLP”, “Residual Connection → LERP” and “Linear → Linear + Scaler” in this work?\n\nThank you for highlighting this. Below we provide both the rationale and supporting evidence for these design choices.\n\n1. **Linear → Linear + Scaler:**  \nThis modification enables two key benefits:\n(i) weight projection ensures control over parameter norm and effective learning rate, and\n(ii) the learnable scalar allows selective emphasis of important features.\n\nA standard linear layer without projection leads to unbounded parameter growth. On the other hand, projection without a scalar severely limits representational capacity. The Linear + Scaler combination is therefore necessary to balance norm control with expressivity.\n\n2. **Residual Connection → LERP:**  \n   Since all features in SimbaV2 are normalized to lie on a hypersphere, standard residual connections are no longer applicable. The closest analog would be fixed-$\\alpha$ interpolation (e.g., $\\alpha$=0.5). We conducted an ablation to compare **LERP** vs. fixed-$\\alpha$ residual:\n\n| Method      | DMC-Hard | HBench-Hard |\n|-------------|----------|-------------|\n| LERP (learnable, $\\alpha_{init}$=1/(L+1)) | 0.729 $\\pm$ 0.065 | 0.946 $\\pm$ 0.089 |\n| Residual (fixed, $\\alpha_{init}$=0.5) | 0.687 $\\pm$ 0.092 | 0.843 $\\pm$ 0.123 |\n\nLERP achieves higher performance in both benchmarks. We attribute this to two factors: its learnable mixing coefficient, which enables adaptive interpolation, and its initialization ($\\alpha$=1/(L+1)), which biases early training toward identity mapping, enhancing stability.\n\n\n> **Q2.2:**  \n> There are some other related papers not included:\n> - Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers. arXiv 2411.15370  \n> - Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps. arXiv 2412.17113  \n> - Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn. arXiv 2409.04792  \n\nThank you for pointing out these relevant works. We appreciate the suggestions and will include them in the related work section. We have also added the following relevant works:\n\n- Is High Variance Unavoidable in RL?., Bjorck et al., ICLR 2022\n- Understanding, Predicting, and Better Resolving Q-Value Divergence in Offline RL., Yue et al., NeurIPS 2023\n- Mixtures of Experts Unlock Parameter Scaling for Deep RL., Ceron et al., ICML 2024\n- Dissecting Deep RL with High Update Ratios., Hussing et al., RLC 2024 \n- Don’t Flatten, Tokenize!., Sokar et al., ICLR 2025\n\n\n> **Q2.3:**  \n> The performance of SimbaV2 is based on SAC, leaving its effects on PPO and DQN-variants unknown.\n\nThank you for highlighting this. While extending to PPO and DQN is a valuable direction, these algorithms present challenges: PPO is inherently on-policy and difficult to scale to high update-to-data (UTD) ratios, while DQN is restricted to discrete action spaces.\n\nTo assess the generality of SimbaV2 beyond SAC, we conducted additional experiments using DDPG, a widely adopted off-policy algorithm for continuous control:\n\n| Method   | DMC-Hard | HBench-Hard |\n|----------|----------|-------------|\n| SimbaV2  | 0.636 $\\pm$ 0.087 | 0.693 $\\pm$ 0.119 |\n| Simba    | 0.649 $\\pm$ 0.089 | 0.445 $\\pm$ 0.101 |\n| MLP      | 0.149 $\\pm$ 0.034 | 0.115 $\\pm$ 0.047 |\n\nIn DMC-Hard, SimbaV2 performs competitively with Simba, both significantly outperforming the MLP baseline. In the more challenging HBench-Hard benchmark, SimbaV2 shows clear improvements over Simba, indicating enhanced stability and generalization beyond SAC. \n\n\n> **Q2.4:**  \n> In Simba, the observation is centered and rescaled by running mean and std, and in SimbaV2, the observation is rescaled by the L2 norm. If there is a variant of SimbaV2 that only replaces the L2-norm rescaling with running std rescaling, how would it perform?\n\nWe appreciate the opportunity to clarify. There appears to be a misunderstanding: both Simba and SimbaV2 use running mean and standard deviation (RSNorm) for input normalization.\n\nThe key difference is architectural. SimbaV2 replaces LayerNorm with L2 normalization for internal features, not for input observations. This change targets internal stability and effective learning rates, while the input normalization strategy rem", "claims_and_evidence": "Most claims made in this paper are well supported with experimental results. The necessity or effects of the design changes “Linear → Linear + Scaler” and “Residual Connection → LERP” are not supported with direct evidence (Please correct me if I missed them).", "methods_and_evaluation_criteria": "The proposed methods are mainly for better addressing non-stationarity in observation, intermediate features and network output (i.e., target values).\n\nThe evaluation criteria are diverse in this work. The performance metrics are well normalized. The analysis metrics (including weight norm, feature norm, ELR) also make sense in the context.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The experiments are well organized with comprehensive content, from the analysis of learning dynamics, to performance comparison and then the ablation study of design choices. Training and inference cost are also included.", "relation_to_broader_scientific_literature": "The proposed architecture is of potential to be a norm for DRL, beyond the SAC base algorithm considered in this paper. The proposed methods and the experimental study can provide a useful reference to the related studies on learning under non-stationarity, e.g., continual RL, streaming/incremental RL.", "essential_references_not_discussed": "Most essential related works are included. There are some other related papers not included:\n\n- Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers. arXiv 2411.15370\n- Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps. arXiv 2412.17113\n- Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn. arXiv 2409.04792", "other_strengths_and_weaknesses": "#### Strengths\n\n- SimbaV2 significantly improves the stability of learning dynamics in terms of metrics like ELR, especially for more challenging tasks in HBench-Hard.\n- SimbaV2 is free of adding various learning-based/optimization-based regularization to address the non-stationarity issue, which offers better generality and feasibility.\n- The comprehensive experiments can provide useful references for related studies.\n\n#### Weaknesses\n\n- The performance of SimbaV2 is based on SAC, leaving its effects on PPO and DQN-variants unknown (the authors also mentioned this).\n- The environments used in the experiments include only proprioceptive observations (correct me if I misunderstand it)", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Are there experiments for evaluating the effects of the design choices “the inverted bottleneck MLP”, “Residual Connection → LERP” and “Linear → Linear + Scaler” in this work? It seems that I did not find direct empirical evidence for these ones.\n2. In Simba, the observation is centered and rescaled by running mean and std, and in SimbaV2, the observation is rescaled by the L2 norm. If there is a variant of SimbaV2, that only replaces the L2-norm rescaling by the running std rescaling, how would it performs differently with SimbaV2?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rLJhHgqyMI", "reviewer": "Reviewer_4564", "summary": "This paper introduces SimbaV2, an RL architecture that improves scalability and stability in deep RL. The authors use hyperspherical normalization to control weight and feature norm growth, alongside distributional value estimation with reward scaling to maintain stable gradients. Using SAC as the base algorithm, SimbaV2 outperforms existing RL methods across a wide range of continuous control tasks.\n\n## Update after Rebuttal\n\nAfter the rebuttal, I am keeping my positive score for this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 4564,\n\nThank you for suggesting future research direction! We respond to each of your comments below and are happy to clarify further if needed.\n\n\n> **Question 3.1**\nThe claim that this leads to 'scalable' RL remains incorrect, as scaling limits are reached very quickly (See Fig. 5).\n\nWe appreciate your concern. While it is true that performance on DMC-Hard saturates earlier, this is largely due to task-specific ceilings rather than architectural constraints. Most tasks in DMC-Hard (except humanoid-run) reach near-optimal scores around 4.5 million parameters. In contrast, HBench-Hard continues to benefit from increased capacity up to 17.8 million parameters, demonstrating substantial headroom for scaling.\n\nThis level of parameter scalability is atypical in RL. Standard SAC or DDPG architectures often use around 2 million parameters, and larger models tend to degrade performance. In contrast, SimbaV2 scales consistently and robustly, without the need for reinitialization or tuning.\n\nSimilarly, in Fig. 6, compute scaling on DMC-Hard saturates around UTD = 4 due to task limitations, but HBench-Hard continues improving up to UTD = 8. This behavior highlights SimbaV2’s stable training dynamics under both model and compute scaling, which we believe justifies its claim to scalability within the current RL landscape.\n\n\n> **Question 3.2**\nThis paper only feels like a minor step up of SimbaV1. I would have liked to see experiments in a different set of environments, such as pixel-based control. \n\n> Why have the authors not attempted to do a performance analysis in at least a few pixel-based Mujoco tasks? How would the authors implement their techniques on CNN's ?\n\nThank you for this suggestion. We agree that applying SimbaV2 to pixel-based control is a promising direction. However, extending our techniques to convolutional architectures presents unique challenges. In CNNs, a shared kernel operates over overlapping spatial regions, making it nontrivial to enforce hyperspherical constraints on both features and weights, as we do with MLPs.\n\nOne possible approach is to project each C-dimensional fiber of the feature map and kernel onto a hypersphere, preserving the underlying normalization principle. However, this would require considerable architectural tuning, which we consider an important direction for future work.\n\nThat said, we believe the core ideas behind SimbaV2, *controlling feature and parameter norms, and maintaining stable effective learning rates*, can extend to vision-based RL. We look forward to exploring this as a future work.", "claims_and_evidence": "Most claims made in the submission are supported by clear and convincing evidence. However, the claim that this leads to 'scalable' RL remains incorrect, as scaling limits are reached very quickly (See Fig. 5).", "methods_and_evaluation_criteria": "The methods and evaluation criteria are good, although I would have liked to see experiments also in another domain or algorithm, as it feels somewhat repetitive to Simba V1.", "theoretical_claims": "-", "experimental_designs_or_analyses": "Experimental design and analysis is sound.", "relation_to_broader_scientific_literature": "This paper is at the forefront of performance-based RL in continuous control tasks.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "Strengths: Strong performance on the state-based continuous control benchmarks. Well written paper.\n\nWeaknesses: This paper only feels like a minor step up of SimbaV1. I would have liked to see experiments in a different set of environments, such as pixel-based control. However, at this level of performance a step up is also hard to accomplish, so there is still a valid contribution (Especially if we compare it to Supervised Learning papers which make minuscule step ups.)", "other_comments_or_suggestions": "-", "questions_for_authors": "Why have the authors not attempted to do a performance analysis in at least a few pixel-based Mujoco tasks? How would the authors implement their techniques on CNN's ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2rJWGitNSv", "reviewer": "Reviewer_fmAi", "summary": "The main claim of the paper is that a novel architecture (SimbaV2) can improve the scaling of RL algorithms. The benefits of using SAC and SimbaV2 are demonstrated across a variety of domains.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer fmAi,\n\nThank you for your thoughtful and constructive feedback. We address your concerns below and would be happy to clarify further.\n\n> **Question 4.1**\nThe scalability claims are not fully justified:\n(1) Only SAC is tested, (2) only width scaling is shown, (3) UTD scaling shows limited benefit.\n\n(1) Algorithm: To evaluate generality beyond SAC, we ran SimbaV2 with DDPG on DMC-Hard and HBench-Hard (UTD = 2):\n\n| Method   | DMC-Hard | HBench-Hard |\n|-|-|-|\n| SimbaV2  | 0.636  | 0.693 |\n| Simba    | 0.649  | 0.445 |\n| MLP      | 0.149  | 0.115 |\n\nSimbaV2 performs comparably to Simba on DMC-Hard and significantly better on HBench-Hard, confirming its effectiveness beyond SAC.\n\n(2) Depth scaling: We conducted additional experiments on DMC-Hard by varying the critic depth (1, 2, 4, 8 layers) using 5 random seeds. \n\n| Method  | 1 | 2 | 4 | 8 |\n|-|-|-|-|-|\n| SimbaV2  | 0.525 | 0.729 | 0.740 | **0.743** |\n| Simba    | 0.512 | 0.706 | 0.675 | -  |\n\nUnlike SimbaV1, which degraded with depth, SimbaV2 improves consistently. This supports our view that effective regularization enables stable scaling in depth. Due to a limited time, HBench-Hard results are underway and will be included in the final manuscript.\n\n(3) UTD scaling: While simple tasks (e.g., Cartpole-Balance) saturate quickly, more complex tasks such as HumanoidBench-Hard continue improving with higher UTD (see Fig. 6). Notably, prior work [1, 2] shows that high UTD often degrades performance unless combined with weight reinitialization. SimbaV2 maintains stable learning at high UTD without reinitialization, which we believe is a key contribution.\n\n- [1] Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier, ICLR'23.\n- [2] Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control, NeurIPS'24.\n\n> **Question 4.2**\nIt's unclear whether the reported metrics are necessary for stability and scaling.\nMany architectural choices in Section 4 are not motivated beyond empirical results.\n\nThe architectural decisions in SimbaV2 are based on training instabilities observed in SimbaV1 and supported by prior literature.\n\n1. Feature Norm: TD learning introduces an implicit bias toward increasing feature norm [1], which can lead to overfitting and a reduction in feature rank. Both effects are closely linked to loss of plasticity during training [2, 3].\n2. Parameter Norm, Gradient Norm, and ELR: As parameter norms grow, the ELR decreases, impeding gradient flow and leading to stagnation in learning [4]. This dynamic was observed in SimbaV1, particularly in the encoder.\n\nThese insights motivated the following designs in SimbaV2:\n1. Hyperspherical normalization to control feature norm\n2. Weight projection to constrain parameter norm.\n3. Distributional critic with reward scaling to stabilize gradient norm.\n\nTogether, these components maintain a stable ELR across layers, preserve plasticity, and eliminate the need for weight reinitialization. We agree that this motivation was not clearly described and will revise the introduction to clarify these points.\n\n- [1] DR3: Value-Based Deep RL Requires Explicit Regularization, ICLR'22.\n- [2] Understanding and Preventing Capacity Loss in RL, ICLR'22.\n- [3] Dissecting Deep RL with High Update Ratios: Combatting Value Divergence, RLC'24.\n- [4] Normalization and effective learning rates in RL, NeurIPS'24.\n\n\n> **Question 4.3**\n> The use of a distributional critic is not defended.\n\nAs shown in [this ablation](https://drive.google.com/file/d/15nCDgxhbPL20LnrPCa6k_gNYfoR5smzO/view?usp=sharing), removing the distributional critic and reward scaling degrades performance and leads to a sharp decline in gradient norm, which in turn reduces the ELR. This supports its role in stabilizing training and preserving plasticity.\n\n> **Question 4.4**\nWhat was the motivation behind Linear + Scaler?\n\nLinear + Scaler serves two purposes:\n(i) weight projection controls parameter norm and ELR, and\n(ii) The learnable scalar amplifies important features.\n\nWithout the scalar, projection limits expressivity; without projection, training becomes unstable. Their combination is essential for balancing stability and flexibility.\n\n> **Question 4.5**\nI learned very little from this paper, other than the fact that it works well.\n\nThank you for the candid feedback. Beyond empirical gains, our central insight is that stabilizing norm dynamics enables scalable, stable, and sample-efficient RL without relying on resets—a common workaround in the RL community. SimbaV2 offers a principled, architecture-level solution to this challenge.\n\nWe also believe these insights may transfer to other domains. For example, recent [NLP work](https://arxiv.org/abs/2503.19206) suggests long pretraining reduces model plasticity. SimbaV2’s techniques may inform the design of more robust architectures for large-scale training in other domains.", "claims_and_evidence": "I find that the authors do an excellent job at demonstrating the empirical performance benefits of using SimbaV2 and SAC. The empirical evidence is overwhelming, across a large number of domains.\n\nHowever, the authors also make some more general claims about scaling (SimbaV2 enables better scaling), that are not as clearly justified. \n1. SimbaV2 improves scaling of **RL algorithms** $\\rightarrow$ The authors only demonstrate this claim for SAC.\n\n2. SimbaV2 improves **scaling** of RL algorithms $\\rightarrow$ The authors only demonstrate that SimbaV2 scales better in width than the original Simba, and not SAC. The authors also don’t demonstrate this claim for depth (increasing the number of blocks). In the UTD dimension, the scaling results are less convincing. Looking at Appendix I, we see that often an increase in UTD offers no performance gain (and sometimes slightly harms). From many previous works, we know that this is still an improvement (naively increasing UTD often harms performance), but this isn’t exactly a strong defense of UTD scaling. As a baseline, the authors only compare against one setting of resets and the original Simba. \n\n3. X design change is beneficial $\\rightarrow$  The authors make a number of design choices to improve over the original Simba (listed in Section 4). Many of these design choices are not defended or motivated outside of empirical performance (Table 2). Including the use of a distributional critic, which is specifically outlined as a key contribution in the abstract of the paper.", "methods_and_evaluation_criteria": "The SOTA claims are well-defended: the authors cover a lot of popular benchmarks and compare against many SOTA algorithms. \n\nAs mentioned above, some of the evaluation of the broader claims made by the authors fall short (only a single base algorithm, limited baselines for analytical claims).", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "I find the analysis limited. In Figure 4, the authors demonstrate that SimbaV2 improves over the original Simba in a number of dimensions (feature norm, parameter norm, gradient norm, effective LR). However, it’s unclear to me whether any of these dimensions are necessary for stability, scaling, or better performance. Furthermore, it’s unclear which design choices contribute to these changes.", "relation_to_broader_scientific_literature": "SOTA algorithm is obviously beneficial. UTD and scaling laws are very popular topics in the community right now. I think the authors could do a better job at comparing against existing work in this dimension.", "essential_references_not_discussed": "Hussing, Marcel, et al. \"Dissecting deep rl with high update ratios: Combatting value divergence.\" arXiv preprint arXiv:2403.05996 (2024).\n\nThis paper analyses high UTD in more detail and also suggests l2 normalization as a solution.", "other_strengths_and_weaknesses": "**Strengths:**\n\nSo far, the construction of this review process has mostly forced me to list weaknesses, but I think the paper is a valuable contribution to the community. As a scientific paper, the authors make some overclaims, but as an empirical contribution, the authors are introducing a powerful, widely applicable algorithm that has been thoroughly tested. \n\nGiven the impact we have seen from the same types of papers in the model-based space (DreamerV3, TD-MPC2), I see no reason why a model-free algorithm should be any less useful. Furthermore, many of the design choices are likely widely applicable to other algorithms, although I believe the authors could do a better job defending that claim.  \n\n**Weaknesses:**\n\nI learned very little from this paper, other than the fact that it works well.", "other_comments_or_suggestions": "Typos\n- Output Preidiction Line395\n- ”resize” (inverse quotes) Line434", "questions_for_authors": "Empirical:\n- Does SimbaV2 scale in depth? (i.e., adding more blocks).\n- Does SimbaV2 work with other algorithms (besides SAC).\n\nAnalytical:\n- Why does SimbaV2 (and each of its components) improve performance?\n- What was the motivation behind the design choices, especially the more subtle ones like Linear + Scalar? \n\nOther reviewers may disagree, but I would also be satisfied if some of the broader claims were reduced, and the contribution of the paper was reduced to just the specific SimbaV2+SAC algorithm.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: There is a broad consensus that the paper deserves acceptance due to the strong empirical evidence to support its claims and SOTA performance on a range of tasks. The paper is seen as a useful step towards scaling existing deep RL approaches. A minor weakness is that this paper might be related on many aspects to SimbaV1 (reviewer 4564).
Decision: Accept (spotlight poster)