SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning
Authors: Hojoon Lee
Dongyoon Hwang
Donghu Kim
Hyunseung Kim
Jun Jet Tai
Kaushik Subramanian
Peter R. Wurman
Jaegul Choo
Peter Stone
Takuma Seno
Venue: ICLR 2025 Spotlight (2025)
Keywords: reinforcement learning
Primary area: reinforcement learning
TL;DR: In NLP and CV, scaling up model parameters improves performance, but in RL, scaling up often degrades it. This paper proposes a network architecture that allows RL algorithms to improve performance as model size increases.

Abstract: Recent advances in CV and NLP have been largely driven by scaling up the number of network parameters, despite traditional theories suggesting that larger networks are prone to overfitting.
These large networks avoid overfitting by integrating components that induce a simplicity bias, guiding models toward simple and generalizable solutions. 
However, in deep RL, designing and scaling up networks have been less explored.
Motivated by this opportunity, we present SimBa, an architecture designed to scale up parameters in deep RL by injecting a simplicity bias. SimBa consists of three components: (i) an observation normalization layer that standardizes inputs with running statistics, (ii) a residual feedforward block to provide a linear pathway from the input to output, and (iii) a layer normalization to control feature magnitudes. 
By scaling up parameters with SimBa, the sample efficiency of various deep RL algorithms—including off-policy, on-policy, and unsupervised methods—is consistently improved.
Moreover, solely by integrating SimBa architecture into SAC, it matches or surpasses state-of-the-art deep RL methods with high computational efficiency across DMC, MyoSuite, and HumanoidBench.
These results demonstrate SimBa's broad applicability and effectiveness across diverse RL algorithms and environments.

Published as a conference paper at ICLR 2025
SIMBA: SIMPLICITY BIAS FOR SCALING UP PARAMETERS IN DEEP REINFORCEMENT LEARNING
Hojoon Lee Dongyoon Hwang 1,3+ Donghu Kim Hyunseung Kim Jun Jet Tai Kaushik Subramanian' Peter R. R.Wurman2 Jaegul Choo' Peter Stone Takuma Seno? KKIIST 2Sony AI *KRAFTON *Coventry University
UT Austin
(joonleesky, godupeterJekaisstacc.k
ABSTRACT
Recent advances CV and NLP have been largely driven by scaling up the num- ber of network parameters, despite traditional theories suggesting that larger net- works are prone to overfitting. These large networks avoid overfitting by integrat- ing components that induce simplicity bias, guiding models toward simple and generalizable solutions. However, deep RL, designing and scaling up networks have been less explored. Motivated by this opportunity, we present SimBa, an architecture designed to scale up parameters in deep RL by injecting simplic- ity bias. SimBa consists of three components: (i) an observation normalization layer that standardizes inputs with running statistics, (ii) a residual feedforward block provide linear pathway from the input to output, and (iii) layer nor- malization to control feature magnitudes. By scaling up parameters with SimBa, the sample efficiency of various deep RL algorithns-iincuding off-policy, on- policy, and unsupervised methods- consistently improved. Moreover, solely by integrating SimBa architecture into SAC, it matches or surpasses state-of-the art deep RL methods with high computational efficiency across DMC, MyoSuite, and HumanoidBench. These results demonstrate SimBa's broad applicability and effectiveness across diverse RI. algorithms and environments.
Explore codes and videos at http::/ssonyrsearchh github.io/simbe
Off Palicy
Off Policy MB UMC Harci (1M, tasks)
On Policy
Unsupervised
DMC Hard (IM, tasks)
Craflax 110, tack)
UMC Humanord (10M, task)
SAC Smblle
METRA SimDa
SAC
METRA
600
300
(3)
420
16
230
200
200
7D SPC2 Simbo
PnD Simsa
130
TD SPC2
190
0
0.2
0.5
0.0
1.0
0.2
0.5
0.6
1.0
0
0.7
D.5
0.0
1.0
0
2.5
5.0
7.5
10.0
Ere Step (M)
Eny Step
Env Step
Ene Step (M)
DMC E&M (500K, 20 tesks)
DMC Herd (1M, tasks)
MycSute (3M, 10 lesksl
HumeroidBerch (2N, 14 tesks)
3,000
1,000
750
(b)
500
500
50
500
250
250
25
250
0
0.7
1.4
2,3
2.5
1.4
2.3
4.3
5.5
1.5
3.0
4.4
5.0
2.9
3.7
5.6
11.4
Compute Time (14)
Compute Time
Compute Time (hr)
Compute (h1)
SimBa
TD-MPC2
DrcamerVS
Figure 1: Benchmark Summary. (a) Sample Efficiency: SimBa improves sample efficiency across various RL algorithms, including off-policy (SAC, TD-MPC2), on on-policy (PPO), and unsupervised RL (METRA). (b) Compute Efficiency: When applying SimBa with SAC, it matches or surpasses state-of-the-art off- oof-cy RL methods across 51 continuous control tasks, by only modifying the network architecture and scaling up the number of network parameters.
"Equal Contribution
Work done during internship Sony AT

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Scaling up neural network sizes has been key driver of recent advancements in computer vision (CV) (Dehghani et al., 2023) and natural language processing (NLP) (Google et al., 2023; Achiam al., 2023). By increasing the number of parameters, neural networks gain enhanced expressivity, enabling them to cover diverse functions and discover effective solutions that smaller networks might miss. However, this increased capacity also heightens the risk of overfitting, as larger networks can fit intricate patterns in the training data that do not generalize well to unseen data.
Despite this risk of overfitting, empirical evidence shows that neural networks tend to converge toward simpler functions that generalize effectively (Kaplan et al., 2020; Nakkiran et al., 2021), This phenomenon attributed to a simplicity bias inherent in neural networks, where standard opti- mization algorithms and architectural components guide highly expressive models toward solutions representing simple, generalizable functions (Shah et al., 2020; Berchenko, 2024). For instance, gra- dient noise in stochastic gradient descent prevents models from converging on sharp local minima, helping them avoid overfitting (Chizat & Bach, 2020; Gunasekar et al., 2018; Pesme et al., 2021). Architectural components such ReLU activations (Hermann al., 2024), layer normalization (Ba et al., 2016), and residual connections (He et al., 2020) are also known to amplify simplicity bias. These components influence the types of functions that neural networks represent at initialization, where networks that represent simpler function at initialization are more likely to converge to simple functions Valle-Perez et al., 2018; Mingard et al., 2019; Teney al., 2024).
While scaling up network parameters and leveraging simplicity bias have been successfully applied in CV and NLP, these principles have been underexplored in deep reinforcement learning (RL), where the focus has primarily been on algorithmic advancements (Hessel et al., 2018; Hafner et al., 2023; Fujimoto et al., 2023; Hansen et al., 2023). Motivated by this opportunity, we introduce the SimBa network, novel architecture that explicitly embeds simplicity bias to effectively scale up parameters deep RL. SimBa comprises three key components:
(i) an observation normalization layer that standardizes inputs by tracking the mean and variance of each dimension, reducing over- fitting to high-varianee features (Andrychowicz et al., 2020); (ii) a pre-layer normalization residual feedforward block (Xiong al., 2020), which maintains direct linear information pathway from input to output and applies non-linearity only when necessary; and (iii) a post-layer normalization before the output layer to stabilize activations, ensuring more reliable policy and value predictions.
To verify whether SimBa amplifies simplicity bias, we compared it against the standard MLP ar- chitecture often employed in deep RL. Following Teney et al. (2024), measured simplicity bias by (i) sampling random inputs from a uniform distribution; (ii) generating network outputs; and (ii) performing Fourier decomposition on these outputs. A smaller sum of the Fourier coefficients indicates that the neural network represents a fow-frequency function, signifying greater simplicity. We define the simplicity score as the inverse of this sum, meaning higher score corresponds to a stronger simplicity bias. As illustrated in Figure 2.(a), our analysis revealed that SimBa has a higher simplicity score than the MLP (Further details are provided in Section 2).
To evaluate the benefit of leveraging simplicity bias on network scaling, we compared the performance of Soft Actor-Critie (Haarnoja et al., 2018, SAC) using both MI.P and our SimBa architecture across 3 humanoid tasks from DMC benchmark (Tassa et al., 2018). We increased the width of both ac- tor and critic networks, scaling from 0.1 to 17 mil-
DMC-Humanonid w/ SAC
SimBB ML.
Fal Simplicity Score(1) (1)
8.7
Ib' Average net am [1]
600
8.6
850
6.1
300
150
lion parameters. As shown in Figure 2.(b), SAC with MLP experiences performance degradation as the number of parameters increases. In con- trast, SAC with the SimBa network consistently improves its performance the number of param- eter increases, highlighting the value of embedding simplicity bias when scaling deep RL networks.
SimBa MLF
0.1
0.3
3.5
3.0
10.0
Archizecture
Params [M]
Figure 2: (a) SimBa exhibits higher simplic- ity bias than MI.. (b) SAC with SimBa im- proves its performance with increased param- eters, whereas SAC with MLP degrades it. Each standard deviation is 95% CI.
To further evaluate SimBa's versatility, we applied it to various RL algorithms by only changing the network architecture and scaling up parameters. The algorithms included off policy (SAC (Haarnoja al., 2018), TD-MPC2 (Hansen al., 2023)), on-policy model-free (PPO (Schulman al., 2017)).
2

---

Published as a conference paper at ICLR 2025
and unsupervised RL (METRA (Park et al., 2023)). As illustrated in Figure I.(a), SimBa consis- tently enhances the sample efficiency these algorithms. Furthermore, as shown in Figure 1.(b), when SimBa is integrated into SAC, it matches or surpasses state-of-the-art off-policy methods across 51 tasks in DMC, MyoSuite (Caggiano et al., 2022), and HumanoidBench (Sferrazza et al., 2024). Despite the increased number of parameters, SAC with SimBa remains computationally effi cient because it does not employ any computationally intensive components such as self-supervised objectives (Fujimoto al., 2023), planning (Hansen al., 2023), or replay ratio scaling (Nauman et al., 2024), which state-of- ofate-of-the-ane the--rt art methods rely on to achieve high performance.
PRELIMINARY
Simplicity bias refers to the tendency neural networks to prioritize learning simpler patterns over capturing intricate details (Shah et al., 2020; Berchenko, 2024). In this section, introduce metrics to quantify simplicity bias; in-depth definitions are provided in Appendix A.
2.1
MEASURING FUNCTION COMPLEXITY
We begin by defining complexity measure c: F - [0, oa) to quantify the complexity of functions in = X + y where C R" denote the input space and y C IR' the output space.
Traditional complexity measures, such the Cheni- dimension (Blumer et al., 1989) and Rademacher complexity (Bartlett & Mendelson, 2002), are well-established but of- ten intractable for deep neural networks. Therefore, we follow Teney et al. (2024) and adopt Fourier analysis our primary complexity measure. Given fax (2m)d/2 f(k) ed dk where = f= fxx) eik-- is the Fourier transform, we perform a discrete Fourier transform by uni- formly discretizing the frequency domain, k .o. KK. The value of K is chosen by the Nyquist-Shand Shannon limit (Shannon, 1949) to ensure accurate function representation. Our complexity measure e(f) is then computed as the Trequency-weigheee average of the Fourier coefficients:
c(f) >
ΣK fkk).
Intuitively, larger c(f) indicates higher complexity due dominance of high-frequency compo- nents such as rapid amplitude changes or intricate details. Conversely, lower c(f) implies a larger contribution from low-frequency components, indicating a lower complexity function.
2.2 MEASURING SIMPLICITY BIAS
In theory, simplicity bias can be measured by evaluating the complexity of the function to which the network converges after training. However, directly comparing simplicity bias across different archi- tectures after convergence is challenging due to the randomness of the non-stationary optimization process, especially RL, where the data distribution changes continuously.
Empirical studies suggest that the initial complexity of network strongly correlates with the com- plexity of the functions it converges to during training (Valle-Perez al., 2018; De Palma et al., 2019; Mingard et al., 2019; Teney et al., 2024). Therefore, for given network architecture f with an initial parameter distribution Og, we define the simplicity bias score s(f) ; FF + (0, oo) as:
s(f) Rd Eg-eo
c(So)
where fo denotes the network architecture parameterized by
This measure indicates that networks with lower complexity at initialization are more likely to ex- hibit a higher simplicity bias, thereby converging to simpler functions during training.
RELATED WORK
3.1
SIMPLICITY BIAS
Initially, simplicity bias was mainly attributed to the implicit regularization effects of the stochastic gradient descent (SGD) optimizer (Soudry al., 2018; Gunasekar al., 2018; Chizat & Bach, 2020;
3

---

Published as a conference paper at ICLR 2025
RSNorm
Linear
LayerNorm
Linear
ReLU
Linear
LayerNorm
XN
Figure 3: SimBa architecture. The network integrates Running Statistics Normalization (RSNorm),
Residual Feedforward Blocks, and Post-Layer Normalization embed simplicity bias into deep RL.
Pesme et al., 2021). During training, SGD introduces noise which prevents the model from converg- ing to sharp minima, guiding it toward flatter regions of the loss landscape (Wu al., 2022). Such flatter minima are associated with functions of lower complexity, thereby improving generalization.
However, recent studies suggest that simplicity bias is also inherent in the network architecture it- self (Valle-Perez et al., 2018: Mingard et al., 2019). Architectural components such as normalization layers, ReLU activations (Hermann et al., 2024), and residual connections (He et al., 2020) promote simplicity bias by encouraging smoother, less complex functions. Fourier analysis has shown that these components help models prioritize learning low-frequency patterns, guiding optimization ward flatter regions that generalize better (Teney et al., 2024). Consequently, architectural design plays a crucial role in favoring simpler solutions, enabling the use of overparameterized networks.
3.2 DEEP REINFORCEMENT LEARNING
For years, deep RL has largely focused on algorithmic improvements to enhance sample efficiency and generalization. Techniques like Double Q-learning (Van Hasselt et al., 2016; Fujimoto et al., 2018), and Distributional RL. (Dabney et al., 2018) have improved the stability of value estimation by reducing overestimation bias. Regularization grrateicss-inlluding periodic reinitialization (Nik- ishin et al., 2022; Lee et al., 2024), layer normalization (Lee et al., 2023; Gallici al., 2024), batch normalization (Bhatt et al.. 2024), and spectral normalization (Gogianu et al., 2021) have been employed prevent overfitting and enhance generalization. Incorporating self-supervised objec- tives with model-based learning (Fujimoto et al., 2023; Hafner et al., 2023; Hansen et al., 2023) has further improved representation learning and sample efficiency.
Despite these advances, scaling up network architectures in deep RL. remains underexplored, espe- cially by leveraging the simplicity bias principle. Several recent studies have attempted scale up network throuht ensembling (Chen et al., 2021; Obando-Ceron et al., 2024), widen- ing (Schwarzer al., 2023), residual connections (Espeholt et al., 2018) and deepening networks (Bjorck et al., 2021; Nauman et al., 2024). However, they often rely on computationally intensive layers such as spectral normalization (Bjorck et al., 2021) or require sophisticated training protocols (Nauman et al., 2024), limiting their applicability.
In this work, we aim to design an architecture that amplifies simplicity bias, enabling us to effectively scale up parameters in RL, independent of using any other sophisticated training protocol.
4
SIMBA
This section introduces the SimBa network, an architecture designed to embed simplicity bias into deep RL. The architecture is composed of Running Statistics Normalization, Residual Feedforward Blocks, and Post-Layer Normalization. By amplifying the simplicity bias, SimBa allows the model to avoid overfitting for highly overparameterized configurations.
Running Statistics Normalization (RSNorm). First, RSNorm standardizes input observations by tracking the running and variance of each input dimension during training, preventing features with disproportionately large values from dominating the learning process.
Given input observation o, IR at timestep t, update the running observation mean /4 ddd and variance of E R as follows:
-
Pt = 1-1 7 of
(3)
where 6, 0, Pt-1 and d, denotes the dimension of the observation.
4

---

Published as a conference paper at ICLR 2025
Once μi and of computed, each imput observation o, is normalized as:
0, 14e
o RSNorm(o;)
(4)
Vo +
where E RR is the normalized output, and is a small constant for numerical stability.
While alternative observation normalization methods exist, RSNorm consistently demonstrates su- perior performance. A comprehensive comparison is provided in Section 7.1.
Residual Feedforward Block. The normalized observation o, is first embedded into d,- dimensional vector using linear layer:
Linear(ō,).
(5)
At each block, / E (1...., the input x passes through pre-layer normalization residual feed- forward block, introducing simplicity bias by allowing direct linear pathway from the input the output. This direct linear pathway enables the network to pass on the input unchanged throughout the entire network unless non-linear transformations necessary. Each block is defined
x
x + (LP(LayyNNorm(x))
Following Vaswani (2017), the MLP is structured with an inverted bottleneck, where the hidden dimension is expanded to d,, and ReLU activation is applied between the two linear layers.
Post-Layer Normalization. To ensure that activations remain consistent scale before predicting the policy or value function, we apply layer normalization after the final residual block:
24 LayerNorm(x7")
The normalized output 2, is then processed through a linear layer, generating predictions for the actor's policy or the critic's 's value function.
ANALYZING SIMBA
In this section, analyze whether each component of SimBa amplifies simplicity bias and allows scaling up parameters in deep RL. We conducted experiments on challenging environments in DMC involving Humanoid and Dog, collectively referred to DMC-Hard. Throughout this section, used Soft Actor Critic (Haarnoj et al., 2018, SAC) as the base algorithm.
5.1 ARCHITECTURAL COMPONENTS
To quantify simplicity bias in SimBa, network f. we estimate the simplicity bias score s(f) by averaging over 100 random initializations 0 Og. Following Teney et al. (2024), the input space X' 100, 100J2 I divided into grid of 90,000 points, and the network outputs a scalar value for each input which are represented as grayscale image. By applying the discrete Fourier transform to these outputs, we compute the simplicity score s(f) defined in Equation 2 (see Appendix B for further details).
Fourier analysis described in Section 2. For each
Figure 4.(a) shows that SimBa's key compo- nents-such as residual connections and layer nor- malization--iccraasee the simplicity bias score s()), biasing the architecture toward simpler functional representations initialization. When combined, these components induce a stronger preference for low-complexity functions (i.e., high simplicity score) than when used individually.
(e) Simplicity Score (1)
[b] Awrrrage Return (1)
MLP
MLP + RSNomm
Residual
MLP PreLN
MLP PostLN
Simba
5.8
6.2
6.4
200
4.0
Figure 4: Component Analysis. (a) Sim- plicity bias scores estimated via Fourier anal- ysis. Mean and 95% CI are computed over 100 random initializations. (b) Average turn in DMC-Hard for IM steps. Mean and 95% CI over 10 seeds, using SAC. Stronger
Figure 4.(b) reveals clear relationship between simplicity bias and performance: architectures with higher simplicity bias scores lead enhanced per- formance. Specifically, compared to the MIP, adding residual connections increases the average return by 50 points, adding layer normalization adds 150 points, and combining all components results substantial improvement of 550 points.
simplicity bias correlates with higher returns for overparameterized networks.
5

---

Published as a conference paper at ICLR 2025
5.2 COMPARISON WITH OTHER ARCHITECTURES
To assess SimBa's Ba's scalability, we compared it to BroNet (Nauman et al., 2024), SpectralNet (Bjorck al., 2021), and MLP. Detailed descriptions of each architecture are provided in Appendix F.
The key differences between SimBa and other architectures lie in the placement of components that promote simplicity bias. SimBa maintains a direct linear residual pathway from the input the out- put, applying non-linearity exclusively through residual connections. In contrast, other architectures introduce non-linearities within the input-to-output pathway, increasing functional complexity. Ad- ditionally, SimBa employs post-layer normalization ensure consistent activation scales across all hidden dimensions, reducing variance in policy and value predictions. Conversely, other architec- tures omit normalization before the output layer, which can lead to high variance in their predictions.
To ensure that performance differences were attributable to architectural design choices rather than varying input normalization, uniformly applied RSNorm the input normalization across all models. Our investigation focused on scaling up the critic network's 's parameters by increasing the hidden dimension, scaling up the actor network showed limited benefits (see Section 7.2).
As illustrated in Figure 5.(a), SimBa achieves the highest simplicity bias score compared to the experimented architectures, demonstrating strong preference for simpler solutions. In Fig-
Iel Simplicity Score (1)
(b) Average (1)
100
8.1
800
6.0
800
ure 5.(b), while the MLP failed to scale with in-
200
creasing network size, BroNet, SpectralNet, and
SimBa all showed performance improvements.
Architacture
Crttic Params IMI
0.3
1.3
37.5
Importantly, the scalability of each architecture correlated with its simplicity bias score, with higher simplicity bias leading better scalabil- ity. SamBa demonstrated the best scalability, sup- porting the hypothesis that simplicity bias plays a key role in scaling deep RI. Further analysis on this correlation is provided in Appendix D.
Simbo
BroNet
SpectralNet
Figure 5: Architecture Comparison.
SimBa consistently exhibits a higher simplicity bias score. (b) SimBa demonstrates strong scal- ing performance in terms of average return for DMC-Hate compared to the other architectures. The results are from 5 random seeds.
6 EXPERIMENTS
This section evaluates SimBa's applicability across various deep RL algorithms and environments. For each baseline, we either use the authors' reported results or run experiments using their rec- ommended hyperparameters. Detailed descriptions of the environments used in our evaluations are provided in Appendix H. In addition, we also include learning curves and final performance for each task in Appendix J and K, along with hyperparameters used our experiments in Appendix I.
6.1 OFF-POLICY RL
Experimental Setup. We evaluate the algorithms on 51 tasks across three benchmarks: DMC (Tassa al., 2018), MyoSuite (Caggiano et al., 2022), and HumanoidBench (Sferrazza et al., 2024). The
(a) DMC
(b) MyoSuite
(c) HumanoidBench
(d) Craftax
Figure 6: Environment Visualizations. SimBa is evaluated across four diverse benchmark envi- ronments: DMC, MyoSuite, and HumanoidBench. which feature complex locomotion and manipu- lation tasks, and Craftax, which introduces open-ended tasks with varying complexity.

---

Published as a conference paper at ICLR 2025
DMC-Ear Easy & Medium 20 tasks, 500k steps
DMC Hard
MyoSulte
HumanaldBench
7 tasks, IM steps
10 tasks, 1M steps
24 tasks, 2M staps
Neerge Recurn
800
5imBa
BRO
Avosage Return
600
SimBa
"RR
Nverge Success
0.6
SimBa
"ano
MPVV2
600
Bno
SiniBa
TD-MPGV2
Averoge
700
SSAC
6DD7
Dreamerva
600
STD7
0.4
TTD?
Dreamerva
600
SAC
D7
600
200
0.2
200
SACC
DreamerV
Dreamerv
0.0
0.0
0.5
1.5
2.0
3,0
10
12
Figure 8: Off-policy RL Benchmark. Average episode return for DMC and HumanoidBench and average success rate for MyoSuite across 51 continuous control tasks. SimBa (with SAC) achieves high computational efficiency by only changing the network architecture.
DMC tasks are further categorized into MIIC-Eas&&Medeee and DMC-Hard based on complexity.
We vary the number of training steps per benchmark according to the complexity of the environment: 500K for MMCC-aay&MMddiun) 1M for DMC-Hard and MyoSuite, and 2M for HumanoidBench.
Baselines. We compare SimBa against sate-of-theean off-policy RL algorithms. (i) SAC (Haarnoja et al., 2018), an actor-critic algorithm based on maximum entropy RL; (ii) DDPG (Lillicrap, 2015), a deterministic policy gradient algorithm with deep neural networks: (iii) TD7 (Fujimoto al., 2023), an enhanced version of TD3 incorporating state-action representation learning; (iv) BRO (Nauman et al., 2024), which scales the critic network of SAC while integrating distributional Q-learning, op- timistic exploration, and periodic resets. For a fair comparison, we use BRO-Fast, which is the most computationally efficient version; (v) TD-MPC2 (Hansen al., 2023), which combines trajectory planning with long-return estimates using a learned world model; and (vi) DreamerV3 (Hafner et al., 2023), which learns generative world model and optimizes a policy via simulated rollouts.
Integrating SimBa into Off-Policy RI. The SimBa ar- chitecture is algorith and can be applied to any deep RL method. To demonstrate its applicability, we in- tegrate SimBa into two model-free (SAC, DDPG) and one model-based algorithm (TD-MPC2), evaluating their per- formance on the DMC-Hard benchmark. For SAC and DDPG, we replace the standard MLP-based actor-critic networks with SimBa. For TD-MPC2, we substitute the shared encoder from MIP to SimBa while matching the number of parameters as the original implementation.
Average Return (f)
SAC
SimBa
DDPG
DDPG SimBa
TD-MPC2
TD-MPC2 + SimBa
200
400
600
800
Figure 7: Off-Policy RL with SimBa. Replacing MLP with SimBa leads to substantial performance improvements across various off- policy RL methods. Mean and 95% CI are averaged over 10 seeds for SAC and DDPG, and 3 seeds for MD-MPC2 in DMC-Hard.
Figure 7 shows consistent benefits across all algorithms: integrating SimBa increased the average return by 570, 480, and, 170 points for SAC, DDPG, and TD-MPC2 respectively. These results demonstrate that scaling up parameters with a strong simplicity bias significantly hances performance in deep RL.
Comparisons with thate-of-the-Art Methods. Here, we compare SAC + SimBa against state-of- the-art off-policy deep RL algorithms, which demonstrated the most promising results in previous experiments. Throughout this section, we refer to SAC + SimBa as SimBa.
Figure 8 presents the results, where the x-axis represents computation time using an RTX 3070 GPU, and the y-axis denotes performance. Points in the upper-left corner (%) indicate higher compute efficiency. Here, SimBa consistently outperforms most baselines across various environments while requiring less computational time. The only exception is TD-MPC2 on HumanoidBench;; however, TD-MPC2 requires 2.5 times the computational resources of SimBa achieve better performance, making SimBa the most efficient choice overall.
The key takeaway is that SimBa achieves these remarkable results without the bells and whis- tles often found in state-of-the-ar deep RL algorithms. It is easy to implement, which only re- quires modifying the network architecture into SimBa without additional changes to the loss func- tions (Schwarzer et al., 2020; Hafner et al., 2023) using domain speciffc regularizers (Fujimoto et al., 2023; Nauman et al., 2024). Its effectiveness stems from an architectural design that leverages simplicity bias and scaling up the number of parameters, thereby maximizing performance.
7

---

Published as a conference paper at ICLR 2025
Caripo 14mm: 3, Sada::44 41
1,000
Valce (dim: 24, tatts: 30
1030
Humenood (dim: 67, tasts:
3,000
Quadruped 1dm: 78, takk:r
1,000
Dog rdim: 223, basks:
500
300
530
500
500
230
254
230
21V a02 19)
Dry slop DN
Cie Stop (N]
5ne aopp (1)
Enr Stap 180
Figure 9: Impact of Input Dimension. Average episode return for DMC tasks plotted against increasing state dimensions. Results show that the benefits of using SimBa increase with higher input dimensions, effectively alleviating the curse of dimensionality.
Average Return
Make Iron Sword (9))
Make Iron Pickaxe (56)
Defest Orc Mage (%)
60
45
45
30
8
15
15
0.5
1.0
0
0.2
0.5
0.8
0
0.7
0.8
1.0
%
1.0
Step (B)
Env Step
Eny Step
Enw Step
PPO + SimBa
PPO
Figure 10: On-policy RI with SimBa. Average return of achievements and task success rate for three different tasks comparing PPO + SimBa and PPO on Craftax. Integrating SimBa enables effective learning of complex behaviors.
Impact of Input Dimension. To further identify in which cases SimBa offers significant benefits, we analyze its performance across DMC environments with varying imput dimensions. As shown in Figure 9, the advantage of SimBa becomes more pronounced as input dimensionality increases. We hypothesize that higher-dimensional inputs exacerbate the curse of dimensionality, and the simplic- ity bias introduced by SimBa effectively mitigates overfitting in these high-dimensional settings.
6.2 ON-POLICA RL
Experimental Setup. We conduct on-policy RL experiments in Craftax (Matthews al., 2024), an open-ended environment inspired by Crafter (Hafner, 2022) and NetHack (Küttler et al., 2020). Craftax poses unique challenge with its open-ended structure and compositional tasks. Following Matthews et al. (2024), we use Proximal Policy Optimization (Schulman et al., 2017, PPO) as the baseline algorithm. When integrating SimBa with PPO, we replace the standard MLP-based actor- critic networks with SimBa and train both PPO and PPO + SimBa on 1024 parallel environments for total of 1 billion euvironment steps.
As illustrated in Figure 10, integrating SimBa into PPO significantly enhances performance across multiple complex tasks. With SimBa, the agent learns to craft iron swords and pickaxes more rapidly. enabling the early acquisition of advanced tools. Notably, by using these advanced tools, the SimBa-based agent successfully defeats challenging adversaries like the Orc Mage using significantly fewer time steps than an MLP-based agent. These improvements arise solely from the architectural change, demonstrating the effectiveness of the SimBa approach for on-policy RL.
Results.
6.3 UNSUPERVISED RI.
Experimental Setup. In our unsupervised RL. study, we incorporate SimBa for online skill dis- covery, aiming to identify diverse behaviors without relying on task-specific rewards. We focus our experiments primarily on METRA (Park et al., 2023), which serves as the state-of-the-an algorithm in this domain. We evaluate METRA and METRA with SimBa on the Humanoid task from DMC, running 10M environment steps. For a quantitative comparison, we adopt state coverage as our main metric. Coverage is measured by discretizing the and y axes into grid and counting the number of grid cells covered by the learned behaviors at each evaluation epoch, following prior literature (Park al., 2023; Kim et al., 2024a;b).
8

---

Published as a conference paper at ICLR 2025
Results. As illustrated in Figure I.(a) and 11, inte- grating SimBa into METRA significantly enhances state coverage the Humanoid task. The high- dimensional input space makes challenging for METRA to learn diverse skills. By injecting simplie- ity bias to manage high input dimensions and scal- ing up parameters to facilitate effective diverse skill acquisition, SimBa effectively leads to improved ex- ploration and broader state coverage,
METRA SimBa
METRA
Figure 11: URL with SimBa. Integrating SimBa to METRA enhances state coverage.
ABLATIONS
We conducted ablations on DMC-Hard with SAC + SimBa, running seeds for each experiment.
7.1 OBSERVATION NORMALIZATION
A key factor in SimBa's success is using RSNorm for observation normalization. To validate its effectiveness, we compare alternative methods: (i) LayerNorm (Ba et al., 2016); (ii) RMSNorm (Zhang & Sennrich, 2019); (iii) BatchNorm (loffe & Szegedy, 2015); (iv) Env Wrapper RSNorm, which tracks running statistics for each dimension, normalizes observations upon receiving from the environment, and stores them in the replay buffer; (v) Initial N Steps, which fixes the statistics derived from the initially collected transition samples, where we used N = 5, 000; and (vi) Oracle Statistics, which rely pre-computed statistics from previously collected expert data.
As illustrated in Figure 12, layer normalization and batch normalization offer little to no performance gains. While the env-wrapper RSNorm is somewhat effective, it falls short of RSNorm's 's performance. Although widely used in deep RL frameworks (Dhariwal et al., 2017; Hoffman al., 2020; Raffin al., 2021), the env-wrapper introduces inconsisten- cies in off-policy settings by normalizing samples with dif- ferent statistics based their collection time. This causes identical observations to be stored with varying values in the replay buffer, reducing learning consistency. Fixed ini- tial statistics also show slightly worse performance than RSNorm, potentially due their inability to adapt to the evolving dynamics during training. Overall, only RSNorm matched the performance of Oracle statistics, making the most practical observation normalization choice for deep RI.
Average Return (1)
None
LayerNorm
RMSNorm
BatchNormm
Env Wrapper
Initial N Steps
Oracle Stats
R5Norm
300
450
600
750
Figure 12: Obs Normalization.
RSNorm consistently outperforms alternative normalization methods. Mean and 95% CI over 5 seeds.
7.2 SCALING THE NUMBER OF PARAMETERS
Here, investigate scaling parameters in the actor-critic architecture with SimBa by focusing on two aspects: (i) scaling the actor versus the critic network, and (ii) scaling network width versus depth. For width scaling, the actor and critic depths are set to I and blocks, respectively. For depth scaling, the actor and critic widths are fixed at 128 and 512, respectively, following our default setup.
As illustrated in Figure 13, scaling up the width or depth of the critic (+)) generally improves per- formance, while scaling up the actor's width or depth (1) tends to reduce performance. This contrast suggests that the target complexity of the actor may be lower than that of the critic,
Scaling
Scaling Depth
1006
103
205
382
548
055
110
328
505
624
617
Weeth
254 512
Desth
254
298
513
706
713
95
382
051
642
645
where scaling up the actor's 's parameters might be ineffective. These findings align with previous study (Nauman al., 2024). which highlights the benefits of scaling up the critic while showing limited advantages in scaling up the actor.
3-
347
430
630
500
674
0-
113
446
440
450
483
125
Crttic
Crtic Dapth
Figure 13: Scaling Actor and Critic Network. Performance of SAC with SimBa by varying width and depth for the actor and critic network.

---

Published as a conference paper at ICLR 2025
Furthermore, for the critic network, scaling up the width is generally more effective than scaling up the depth. While both approaches can enhance the network's 's expressivity, scaling up the depth can decrease the simplicity bias as it adds more non-linear components within the network. Based on our findings, we recommend width scaling as the default strategy.
7.3 SCALING REPLAY RATIO
According to scaling laws (Kaplan et al., 2020), performance can be enhanced not only by scaling up the number of parameters but also by scaling up the computation time, which can be done by scaling up the number of gradient updates per collected sample (i.e., replay ratio) in deep RI..
However, in deep RL., scaling up the replay ratio has been shown to decrease performance due to the risk of overfitting the network to the initially collected samples (Nikishin et al., 2022; D'Oro et al., 2022; Lee et al., 2023). To address this issue, recent research has proposed periodically reinitializing the network to prevent overfitting, demonstrating that performance can be scaled with respect to the replay ratio. In this section, aim to assess whether the simplicity bias induced by SimBa can mitigate overfitting under increased computation time (i.e., higher replay ratios).
To evaluate this, we trained SAC with SimBa using 2, 4, 8, and 16 replay ratios, both with and without peri- odic resets. For SAC with resets, we reinitialized the en- tire network and optimizer every 500,000 gradient steps. We also compared our results with BRO (Nauman al., 2024), which incorporates resets.
Hard (1M steps, tasks)
RR16
785
Average Return
RR10 AR16
750
RRB
Surprisingly, as illustrated in Figure 14, SimBa's ' per- formance consistently improves the replay ratio in- creases, even without periodic resets. We have ex- cluded results for BRO without resets, as fails learn meaningful behavior for all replay ratios (achieves lower than 300 points). Notably, when resets are included for SimBa, the performance gains become even more pro- nounced, with a replay ratio of outperforming the most computationally intensive BRO algorithm (RR10).
715
RR2
RR2
RR2
680
Compute Time (hr)
Simba
Simba+
BRD (+Reset)
Figure 14: Scaling Replay Ratio. Per- formance of SimBa with and without periodic resets for various replay ratios.
8 LESSONS AND OPPORTUNITIES
Lessons. Deep reinforcement learning has historically struggled with overfitting, requiring complex training protocols and tricks to mitigate these issues (Hessel et al., 2018; Fujimoto et al., 2023). These complexities can hinder practitioners with limited resources. In this paper. we improved performance solely by modifying the network architecture while keeping the underlying algorithms unchanged, simplifying SimBa's implementation and making it casy to adopt. By incorporating simplicity bias through architectural design and scaling up parameters, our network converges to simpler, generalizable functions, matching or surpassing sfate-of-the-art methods. This aligns with Richard Sutton's Bitter Lesson (Sutton, 2019): while task-specific designs may offer immediate gains, scalable approaches provide more sustainable long-term benefits.
Opportunities. While our exploration has focused network architecture, such as dropout (Hiraoka et al., 2021), data augmentation (Kostrikov et al., 2020), and advanced op- timization algorithms (Foret et al., 2020) are also crucial for promoting convergence to simpler functions. Extending our insights to vision-based RI. presents a promising direction; specifically, designing convolutional architectures guided by simplicity bias principles could substantially en- hance learning efficiency in visual tasks. Furthermore, integrating SimBa with multi-task RL. may improve performance across diverse tasks by facilitating the learning of shared, simple, and gen- eralizable representations across tasks. Although our current work centers model-free RL al- gorithms, model- based approaches have also demonstrated considerable success (Schwarzer et al., 2020: Hansen al., 2023: Hafner et al., 2023). Our preliminary investigations applying SimBa to model-based RI, specifically TD-MPC2 (Hansen et al., 2023), have yielded promising results that warrant further exploration. We encourage the research community to pursue these directions to advance deep RL architectures and facilitate their successful application in real-world scenarios.
cetpteccture.ootmiizttiio optimization techniques
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY
To ensure the reproducibility of our experiments, we provide the complete source code to run SAC with SimBa. Moreover, have included Dockerfile to simplify the testing and replication of our results. The raw scores each experiment including all baselines are also included. The code is available at http://gtthbbbooom//Snnsssseerrt
ACKNOWLEDGEMENT
This work was supported by the Institute for Information & communications Technology Planning & Evaluation(IITT) grant funded by the Korea (SV)) (RS-2019-[1190075, Artificial Intel- ligence Graduate School Program(KAIST) grant funded by the Korea government(MSIT) (No. RS-2025-00555621) We would also like to express our gratitude to Craig Sherstan, Bram Grooten, Hanseul Cho, Kyungmin Lee, and Hawon Jeong for their invaluable discussions and insightful feedback, which have significantly contributed to this work.
and the National Research Foundation of Korea(NRF)
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744 2023. (Cited on page 2)
Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? large-scale empirical study. arXiv preprint arXiv:2006. 059900 2020. (Cited page 2)
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. (Cited on page 2)
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002. (Cited on page 3)
Yakir Berchenko. Simplicity bias in overparameterized machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11052-11660, 2024. (Cited on page 2)
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. Crossq: Batch normalization deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Representationss 2024. URL https: /ttp///opp/eeeeeeeeeeeeeeeeeeeee
net httee//ooounnndppPPPPeeee
(Cited on page 4)
Nils Bjorek, Carla Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning with spectral normalization. Advances in neural information processing systems, 04:2242-8255 2021. (Cited on page 4, 6, 20)
Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Learnability and the vapnik-chervonenkis dimension. Journal of the ACM (JACM), 36(4):929-965, 1989. (Cited
on page 3)
Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. Myosuite-a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.136600 2022. (Cited on page 3, 6, 24)
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double qelearning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021. (Cited on page 4)
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on learning theory, pp. 1305-1333 PMLR, 2020. (Cited on page 2, 3)
11

---

Published as a conference paper at ICLR 2025
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. (Cited on page 4)
Giacomo De Palma, Bobak Kiani, and Seth Lloyd. Random deep neural networks are biased towards simple functions. Advances in Neural Information Processing Systems, 32, 2019. (Cited on page 3, 17)
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023. (Cited on page 2)
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: lines, 2017. (Cited on page 9)
//github. com/
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022. (Cited on page 10)
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dumning, et Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018. (Cited on page 4)
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020. (Cited on
page 10)
Scott Fujimoto, Herke Hoof, and David Meger. Addressing funetion approximation error in actor- critic methods. In International conference machine learning. pp. 1587-1596. PMLR, 2018.
(Cited on page 4)
Scott Fujimoto, Wei-Di Chang, Edward J Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. arXiv preprint arXiv:2306.0245/1 2023. (Cited on page 2, 3, 4, 7, 10)
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811, 2024. (Cited on page 4)
Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, and Razvan
Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, pp. 3734-3744. PMLR, 2021. (Cited on page 4)
Gemini Google, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023. (Cited on page 2)
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent linear convolutional networks. Advances in neural information processing systems, 31, 2018. (Cited on page 2, 3)
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018. (Cited on page 2, 2,
Danijar Hafner. Benchmarking the spectrum of agent capabilities. In International Confer- ence Learning Representations, 2022. URL https: http///pppeeeiieeeeeeeeeeeeeeeee http:itw..weeeeeeeeeeeeeeeeeeeee net/ 1w0z 6MF 9WC296MFE... EOH. (Cited on page 8, 24)
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04 1044, 2023. (Cited on page 2, 7, 10)
12

---

Published as a conference paper at ICLR 2025
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contin- wous control. arXiv preprint arXiv:2310. 2023. (Cited on page 2, 3, 4, 7, 10, 19, 24, 27, 28)
Fengxiang He, Tongliang Liu, and Dacheng Tao. Why resnet works? residuals generalize. IEEE transactions on neural networks and learning systems, 31(12):534-5362, 2020. (Cited on page 2, 4)
Katherine L. Hermann, Hossein Mobahi, Thomas Fel, and Michael C Mozer. On the foundations of shortcut learning. Proc. the International Conference on Learning Representations (ICLR), 2024. (Cited on page 2, 4)
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018. (Cited on page 2, 10)
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsu- ruoka. Dropout g-functions for doubly efficient reinforcement learning. arXiv preprint 10.iv:2110.02034, 2021. (Cited on page 10)
Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momehev, Danila Sinopalnikov, Piotr Stańczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulac-Amold, Manu Orsini, Alexis Jacq. Johan Ferret, Nino Vieillard, Seyed Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979, 2020, URL https: //tp:i////aiiiia/ org abs 9)
/ 20000/095500. (Cited on page
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015. (Cited on page 9)
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020. (Cited on page 2, 10)
Hyunseung Kim, Byung Kun Lee, Hojoon Lee, Dongyoon Hwang, Sejik Park, Kyushik Min, and Jaegul Choo. Learning to discover skills through guidance. Advances Neural Information Processing Systems, 36, 2024a. (Cited on page 8)
Hyunseung Kim, Byungkun Lee, Hojoon Lee, Dongyoon Hwang, Donghu Kim, and Jaegul Choo. Do's and don't ts: Learning desirable skills with instruction videos. arXiv preprint arXiv:2406.003324 00324, 2024b. (Cited on page 8)
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020. (Cited on page 10)
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR3: Value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Representations, 2022. URL https: //tpp/ppeeeeweeeeeeeeeeee www.:neetforuume
id=PovMwLi91s 91f. (Cited on page 21)
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. In Proceedings of the Conference Newral Information Processing Systems (NeurIPS), 2020. (Cited page 8, 24)
13

---

Published as a conference paper at ICLR 2025
Hojoon Lee, Hanseul Cho, Hyunseung Kim, Daehoon Gwak, Joonkee Kim, Jaegul Choo, Se- Young Yun, and Chulhee Yun. Plastic: Improving input and label plasticity for sample efficient reinforce ment learning. In shirty-sevntth Conference on Neural Information Processing Systems, 2023. (Cited on page 4, 10)
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. arXiv preprint 224.29000.02596, 2024. (Cited on page 4, 21)
TP Lillicrap. Continuous control with deep reinforcement learning. arXiv:1509.02971, 2015, (Cited page 7)
arXiv preprint
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. Proc. the International Conference on Machine Learning (ICML), 2023. (Cited page 21)
Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Thomas Jack- son, Samuel Coward, and Jakob Nicolaus Foerster. Craftax: A lightning-fart benchmark for open-ended reinforcement learning. In Forty-first International Conference Machine Learn- ing, 2024. URL https::///// httpss://ppppeeeeeeeeee .nt/ 8, 24, 28)
(Cited on page
Chris Mingard, Joar Skalse, Guillermo Valle-Pérez, David Martínez-Rubio, Vladimir Mikulik, and Ard A Louis. Neural networks are a priori biased towards boolean functions with low entropy. arXiv preprint arXhv:1909.11522 2019. (Cited on page 2, 3, 4, 17)
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz. Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021, 2021. (Cited page 2)
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mitoš, and Marek Cygan. Big- ger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint arXiv:2405. 2024. (Cited on page 3, 4, 6, 7, 9, 10, 20)
Evgenii Nikishin, Max Schwarzer, Pierluca D' D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. Proc. the International Conference on Machine Learning (ICMI.,, 2022. (Cited on page 4, 10, 21)
Johan Obando-Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures experts unlock parameter scaling for deep rl. arXiv preprint arXiv:2402.08609 2024. (Cited on page 4)
Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric-aware abstraction. arXiv preprint Xiiv22310008887, 2023. (Cited on page 3, 8, 24, 28)
Scott Pesme, Loucas Billaud-Vivien. and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: provable benefit of stochasticity. Advances in Neural Information Processing Systems, 01-22218-22234 2021. (Cited on page 2,
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-haselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http::/tt/// /ttt:://mmma..// wwwill..rgg/pppee pttppapprs/v22///
10-13664.1 html. (Cited on page 9)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017. (Cited on page 2, 8, 28)
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self preficredi representations. arXiv preprint arXiv:2007.059929 2020. (Cited page 7, 10)
14

---

Published as a conference paper at ICLR 2025
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agar- wal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level ef- ficiency. In International Conference Machine Learning, pp. 30365-30380. PMLR, 2023.
(Cited on page 4)
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403. 10506, 2024. (Cited on page 3, 6, 24)
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli, The pitfalls simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573 2020. (Cited page 2,
Claude Elwood Shannon. Communication in the presence of noise. Proceedings of the IRE, 37(1): 10-21, 1949. (Cited on page 3, 17)
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon deep reinforcement learning. arXiv preprint arXiv:2302. /2902, 2023. (Cited on page 21)
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias gradient descent on separable data. Journal of Machine Learning Research, 19(70): 1-57, 2018. (Cited page 3)
Richard Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1):38, 2019. (Cited on page 10)
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 10000000 2018. (Cited on page 2, 24)
Damien Teney, Armand Mihai Nicolicioiu, Valentin Hartmann, and Ehsan Abbasnejad. Neural red- shift: Random networks not random functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4786 4786-4796 4796, 2024. (Cited on page 2, 3, 5, 17, 18)
Guillermo Valle-Perez, Chico Camargo, and Ard A Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions. arXiv preprint arXiv: 11XX::8805.0852,,, 2018. (Cited on page 2, 3, 4, 17)
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016. (Cited on page 4)
Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
(Cited on page 5)
Lei Wu, Mingze Wang, and Weijie Su. The alignment property of sgd noise and how it helps select flat minima: A stability analysis. Advances in Neural Information Processing Systems, 35:4680- 4693, 2022. (Cited on page 4)
Ruibin Xiong, Yunchang Yang, Di Hc, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533. PMLR, 2020. (Cited on page 2)
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 32, 2019, (Cited on page 9)
15

---

Published as a conference paper at ICLR 2025
APPENDIX
Definition of Simplicity Bias
17
A.I Complexity Measure
17
A.2 Simplicity Bias Score
17
B Measuring Simplicity Bias
18
Additional Ablations
19
C.1 Architectural Ablation
19
C,2 Extended Training...
19
Correlation between Simplicity Bias and Performance
20
E Plasticity Analysis
21
E.1 Metrics
E,2 Results
21
Comparison to Existing Architectures
23
Computational Resources
23
H Environment Details
24
H.I DeepMind Control Suite
24
H.2
MyoSuite
24
H.3
HumanoidBench
24
H.4 Craftax
24
Hyperparameters
27
L1
Off-Policy
27
1.2 On-Policy
28
L.3 Unsupervised RL
28
J Learning Curve
29
K Full Result
32
16

---

Published as a conference paper at ICLR 2025
A DEFINITION OF SIMPLICITY BIAS
This section provides more in-depth definition of simplicity bias for clarity.
A.1 COMPLEXITY MEASURE
Let & C R" denote the input space and y C IR the output space. Consider the function space F= [flf: X y). Given training dataset D {(x,, (i) and tolerance > 0, define the
set of functions I C achieving a training loss below E:
{
I= | Lirain (J)
N Σ (((x))) < <ej.
(8)
where L { x 10, oo) is loss function such
squared cross-entropy.
To quantify the complexity functions within F, we adopt a Fourier-based complexity measure as described in Section Specifically, for function with Fourier series representation:
f(x) (2m) / j(k) j(k))th th dk,
(9)
where f()) dr is the Fourier transform of f.
Given a Fourier series representation J, we perform a discrete Fourier transform by uniformly dis- cretizing the frequency domain. Specifically, consider frequencies k E {0, 1,..., K), where K is selected based on the Nyquist-Shannon sampling theorem (Shannon, 1949) to ensure an accurate representation of This discretization transforms the continuous frequency domain into finite of frequencies suitable for computational purposes.
Then, the complexity measure coefficients from discretization:
[0, 00) defined the weighted average of the Fourier
c Ek o f(k) k
(10)
Σ f(k)
Equation 10 captures the intuition that higher complexity arises from the dominance of high- frequency components, which correspond to rapid amplitude changes intricate details in the func- tion f. Conversely, lower complexity measure indicates a greater contribution from low- wwwffreuency components, reflecting simpler, smoother functions.
A.2 SIMPLICITY BIAS SCORE
Let be neural network, and let alg E A denote the optimization algorithm used for training, such as stochastic gradient descent. quantify the simplicity bias score s
x A + (0, oo) as:
sff, alg) = Ey- [((f))
(11)
where Pr.alg denotes the distribution over I induced by training f with alg. A higher simplicity bias score indicates stronger tendency converge toward functions with lower complexity.
Then, from equation 11, we can compute the simplicity bias score with respect to the architecture by marginalizing over the distribution of the algorithm as:
s(f) Entg~A
E
(12)
((f)))
Directly measuring this bias by analyzing the complexity of the converged function is challenging due to the non-stationary nature of training dynamics, especially reinforcement learning where data distributions evolve 2019; Mingard et al., 2019; Teney et al., 2024) suggest that the initial complexity of a network is strongly correlated with the complexity of the functions it learns after training.
time. Empirical studies (Valle-Perez et al., 2018; De Palma et al.,
17

---

Published as a conference paper at ICLR 2025
Therefore, as practical proxy for simplicity bias, we define the approximate of the simplicity bias score sff) for a network architecture f with an initial parameter distribution O as:
s(f) = Eg
(13)
c(fo)
where fo denotes the network f parameterized by
A higher simplicity bias score indicates that, on average, the network initialized from On represents simpler functions prior to training, suggesting a predisposition to converge to simpler solutions during optimization. This measure aligns with the notion that networks with lower initial complexity are more likely to exhibit higher simplicity bias
B MEASURING SIMPLICITY BIAS
To quantify the simplicity bias defined in Equation 13, we adopt methodology inspired by the Neural Redshift framework (Teney et al., 2024). We utilize the original codebase provided by the authors' to assess the complexity of random neural networks. This approach evaluates the inherent complexity of the architecture before any optimization, thereby isolating the architectural effects from those introduced by training algorithms such as stochastic gradient descent.
Following the methodology outlined in Teney al. (2024), perform the following steps to mea- sure the simplicity bias score of a neural network (-(:):
Sampling Initialization. For each network architecture
generate N random initializations 0 ~ Og. This ensemble of initial parameters captures the variability in complexity introduced by different random seeds. In this work, we used N 100.
Sampling Input. We define the input space X' = [-100, 100)² c IR2 and uniformly divide it into grid of 90,000 points, achieving 300 divisions along each dimension. This dense sampling ensures comprchensive coverage of the input domain.
Evaluating Funetion. For each sampled input point æ E X, compute the corresponding output fo(r). The collection of these output values forms 2-D grid of scalars, effectively representing the network's function as grayscale image.
Discrete Fourier Transform. We apply discrete Fourier transform (DFT) to the grayscale image obtained from the function evaluations. This transformation decomposes s-( into sum of basis functions of varying frequencies.
Measuring Function Complexity. Utilizing the Fourier coefficients obtained from the DFT, we compute the complexity measure c(fe) defined in Equation 10. Specifically, we calculate the frequency-weightte average of the Fourier coefficients:
Ek= fo(k)
c(fo)
(14)
Ex (k())
where (k) denotes the Fourier coefficient at frequency k.
Estimating Simplicity Bias Score. For each network f. we estimate the simplicity bias score s(f) by averaging the inverse of the complexity measures over all N random initializations:
sf)
(15)
N 1=1 c(fo.)
where 6; represents the i-th random initialization.
redshi
18

---

Published as a conference paper at ICLR 2025
C ADDITIONAL ABLATIONS
C.I ARCHITECTURAL ABLATION
Experimental Setup. Following the architectural component analysis Section 5.1, we further conducted ablation studies by removing each key component from SimBa to assess its impact on the simplicity score and average return. We performed these experiments on DMC-Hard using the same training setup in Section 5.1, training for IM environment steps.
Results. As shown in Figure (a)(a), we observe that, except for RSNorm, removing key components of SimBa-sunn such as residual connections, pre-layer normalization, or post-layer normalization leadd to reduction in the simplicity bias score. This in- dicates that the functional representation at initial- ization becomes biased toward more complex func- tions when these components are absent. This find- ing further emphasizes the critical role of each com- ponent in promoting simplicity bias within SimBa's design. Additionally, Figure 15.(b) demonstrates that removing any key component significantly im- pacts SimBa's performance, underscoring the ne- cessity of each architectural component.
(a) Simplcity
Average
MLP
Simba RSNom
simba Residual
simba
Simba PostLN
Simba
5.75 6.00
6.25
6.50
200
400
600
800
Figure 15: Additional Component Analy- sis. (a) Simplicity bias scores estimated via Fourier analysis. Mean and 95% CI are com- puted over 100 random initializations. (b) Average return in DMC-Hard for 1M steps. Mean and 95% CI over 10 seeds, using SAC.
Regarding RSNorm, although absence significantly impacts the agent's performance negatively, the overall simplicity bias score of the architecture increases rather than decreases. We speculate that this discrepancy arises because our Fourier Transform based complexity measure relies on evaluat- ing the network's 's outputs over uniformly distributed input space. RSNorm is designed to normal- ize activations based on the statistics of non-uniform or dynamically changing input distributions, excelling with inputs that exhibit variance and complexity not captured by uniform distribution. Therefore, since our simplicity analysis measure based on a uniform input distribution, it may not fully capture RSNorm's practical contribution promoting simplicity bias.
C.2 EXTENDED TRAINING
Experimental Setup. We assess SimBa's robustness by evaluating its performance on the Deep- Mind Control Suite (DMC) Hard tasks over an extended training period of 5 million timesteps. We compared three methods: SAC (Haamojj et al., 2018), TD-MPC2 (Hansen et al., 2023), and SAC with SimBa. All models are trained under identical hyperparameters and configurations as in our main experiments to ensure a fair comparison.
SimBa
TD MPC2
SAC
dog trot
dog run
dog walk
dog stand
avg, return
750
500
250
emv. Step (M)
env. step (M)
env. jrv_step (M)
env. step (M)
humanoid, run
humanoid walk
humanoid stand
avg return
750
500
0
5.0
0.0
5.0
0.0
5.0
emv. stop (M)
env_ Step (M)
env. env_step (M)
Figure 16: Per task learning curve for DMC Hard. Mean and 95% CI are computed over 5 seeds for SimBa, 3 seeds for TD-MPC2 and SAC.
19

---

Published as a conference paper at ICLR 2025
Results. Figure 16 presents the learning curve of each method for million timesteps. The learning curves indicate that SimBa + SAC not only learns faster but also maintains more stable and consistent performance improvements compared to the baselines.
time, demonstrating better sample efficiency and reduced variance
D CORRELATION BETWEEN SIMPLICITY BIAS AND PERFORMANCE
To investigate the relationship between simplicity bias score and performance deep reinforcement learning, computed the correlation between simplicity bias scores and average returns for DMC- Hard across various architectures evaluated in our study. Our analysis includes 12 architectures: four baselines architectures-MLP SimBa, BroNet (Nauman et al., 2024), and SpectralNet (Bjorck al., 2021) 202lng with eight SimBa variants. These variants consists of four variants with component additions (Section 5.1) and four variants with component reductions (Appendix C.1), representing gradual transition from MLP to SimBa.
As discussed in Appendix the current simplicity bias measure may not accurately capture the influence of RSNorm. To address this limitation, we separately evaluated the correlation for archi- tectures incorporating RSNorm. All twelve architectures have nearly identical parameter counts, approximately 4.5 million parameters each, with maximum variation of 1%. Performance was assessed on the DMC-Hard benchmark, trained for one million steps.
Architectures RSNorm
Architectures
Results. As illustrated in Figure 17(a), ar- chitectures employing RSNorm exhibit Pear- son correlation coefficient of 0.79, indicating a strong positive correlation between simplicity bias score and average return scaled architec- tures. In contrast, Figure 17(b) shows Pear- son correlation coefficient of 0.54, representing moderate correlation across all architectures. We attribute the lower correlation in the broader set the current limitations of the simplicity bias measure. We believe these findings sup- port our hypothesis that higher simplicity bias scores are associated with better performance in deep reinforcement learning with scaled-up ar- chitectures.
1000
Spearman's 's 1-79
Spearan's rw0.54
600
400
200
6.0
5.8
6.4
Simplictty
Simplicity
Correlation Analysis. Pearson cor- relation between simplicity bias score and aver- age return on DMC-Hard. corporating RSNorm show strong correlation, = 0.79. (b) All evaluated architectures exhibit moderate correlation, p = 0.54.
Figure 17:
(a) Architectures in-
20

---

Published as a conference paper at ICLR 2025
E PLASTICITY ANALYSIS
Recent studies have identified the loss plasticity as a significant challenge in non-stationary train- ing scenarios, where neural networks gradually lose their ability adapt new data over time (Nikishin et al., 2022; Lyle et al., 2023; Lee et al., 2024). This phenomenon can severely impact the performance and adaptability of models in dynamic learning environments such RL. To quantify and understand this issue, several metrics have been proposed. including stable-rank (Kumar et al., 2022), dormant ratio (Sokar al., 2023), and L2-feature norm (Kumar al., 2022).
E. METRICS
The stable-rank assesses the rank of the feature matrix, reflecting the diversity and richness of the representations learned by the network. This is achieved by performing an eigen decomposition on the covariance matrix of the feature matrix and counting the number of singular values oy that exceed a predefined threshold
m I1Oo, >T)
s-Rank
(16)
j=1
where F E Rdxm is the feature matrix with d samples and m features, oj the singular values, and I(.) is the indicator function.
The dormant ratio measures the proportion of neurons with negligible activation. It calculated as the number of neurons with activation norms below small threshold divided by the total number of neurons D:
{
|a.| E
Dormant Ratio
(17)
D
where a; represents the activation of neuron i.
The L2-feature norm represents the average L2 norm of the feature vectors across all samples:
Feature Norm
Σ
Fll2
(18)
N
1=1
where F; is the feature vector for the i-th sample. Large feature norms can signal overactive neurons, potentially leading numerical instability.
E.2 RESULTS
To evaluate the impact of different architectures network plasticity, we compared the Soft Actor- Critic (SAC) algorithm implemented with a standard Multi-Layer Perceptron (MLP) against SAC with the SimBa architecture we conducted an ablation study to analyze how each component of SimBa contributes to plasticity measures. Each configuration was evaluated across five random seeds.
the DMC-Easy & Medium and DMC-Hard tasks. Additionally,
21

---

Published as a conference paper at ICLR 2025
Average Return
Srank (1)
(4)
Feature Norm
155
(a)
250
62
0.3
0.2
0.4
0.5
0
0.1
0.9
0.4
0.5
0
0.3
0.2
0.4
0.5
0
0.1
0.7
0.4
0.5
Eriv Step (M)
Crv Step (M)
Enw Step (M)
Erv Step (M)
Average Retum
Srank (1)
Dormant (4)
Feature Norm (4)
600
187
60
15
(b)
200
62
20
8
0.2
0.5
0.8
1.0
9
0.2
0.5
0.5
1.0
%
0.2
0.5
0.8
1.0
0
0.2
0.5
0.8
1.0
Env Step (M)
Enw Step (M)
Env Step (M)
Env Step (M)
SimBa
Figure 18: Plasticity Analysis of SimBa versus MLP. Metries include dormant ratio, s-rank, and feature norm. Figures (a) and (b) represent DMC-Easy Medium and DMC-Hard environments, respectively. A higher dormant ratio and feature norm, along with a lower s-rank, indicate a greater loss of plasticity.
As illustrated in Figures 18.(a) and (b), SAC with the MLP architecture exhibits high dormant ratios and large feature norms across both DMC-Easy Medium and DMC-Hard tasks, indicating sig- nificant loss of plasticity. While MLP achieves higher s-rank values than SimBa in the DMC-Easy Medium tasks, SimBa outperforms MLP s-rank for the DMC-Hard tasks. Importantly, SimBa consistently maintains lower dormant ratios, balanced feature norms, and overall higher s-rank val- ues in more complex cnvironments.
These findings indicate that SimBa effectively preserves plasticity, avoiding the degenerative trends observed with the MLP-based SAC.
Average Retur (11
Srank (1)
Dormant Ratio [1)
Feature Norm (4)
600
187
60
15
0
1.0
1.0
Step
Step
Step
SimBa
SimBa
PreLN
Residual
SimBa
PostLN
SimBa
Figure 19: Plasticity Analysis of SimBa Components. Metrics include dormant ratio, s-rank, and feature norm. Figures (a) and (b) represent DMC-Fasy & Medium and DMC-Hard environments, respectively. A higher dormant ratio and feature norm, along with a lower s-rank, indicate a greater loss of plasticity.
Furthermore, for the component ablation study (Figure 19), removing each component of SimBa individually shows diminished average returns. However, ablating each component does not signif- icantly reduce plasticity measures, with only post-layer normalization showing an improved feature norm. This suggests that while individual components may not significantly impact plasticity mea- sures with their sole use, their integrated use within the SimBa architecture is crucial for effectively mitigating plasticity loss.
22

---

Published as a conference paper at ICLR 2025
F COMPARISON TO EXISTING ARCHITECTURES
Here we provide more in-depth discussion related to the architectural difference between SimBa, BroNet, and SpectralNet. As shown in Figure 20, SimBa differs from BroNet in three key aspects: (i) the inclusion of RSNorm, (ii) the implementation of pre-layer normalization, (iii) the utilization of a linear residual pathway. (iv) and the inclusion of post-layer normalization layer. Similarly, compared to SpectralNet, SimBa incorporates RSNorm, employs a linear residual pathway, and leverages spectral normalization differently.
LayerNorm
LayerNorm
Spectral
Linear
Linear
Linear
ReLU
ReLU
LayerNorm
SpectralNorm
Linear
Linear
xN
LayerNorm
N
LayerNorm
ReLU
xN
Linear
LayerNorm
ReLU
RSNorm
Linear
Linear
Observation
Observation
Observation
(a) SimBa
(b) BroNet
(c) SpectralNet
Figure 20: Architecture Comparison. Illustration of SimBa, BroNet, and SpectralNet.
G COMPUTATIONAL RESOURCES
The training was performed using NVIDIA RTX 3070, A100, or H100 GPUs for neural network computations and either 16-core Intel i7-1 1800H 32-core AMD EPYC 7502 CPU for running simulators. Our software environment included Python 3.10, CUDA 12.2, and Jax 4.26.
When benchmarking computation time, experiments conducted same hardware equipped with an NVIDIA RTX 3070 GPU and 16-core Intel i7-1 1800H CPU.
23

---

Published as a conference paper at ICLR 2025
Table 1: Environment details. We list the episode length, action repeat for each domain, total environment steps, and performance metrics used for benchmarking SimBa.
DMC
MyoSuite
HumanoidBench
Craftax
Episode length
1, 000
100
500 1,000
Action repeat
2
Effective length
500
50
250 500
Total env. steps
500K-IM
1M
2M
1B
Performance metric
Average Return
Average Success
Average Return
Average Score
H ENVIRONMENT DETAILS
This section details the benchmark environments used in evaluation. We list all tasks from each benchmark along with their observation and action dimensions. Visualizations each environment provided in Figure 6, and detailed environment information is available in Table 1.
H.1 DREPMIND CONTROL SUITE
DeepMind Control Suite (Tassa al., 2018, DMC) is standard benchmark for continuous trol, featuring range of locomotion and manipulation tasks with varying complexities, from simple lowddimessional tasks (s R to highly complex ones (s IR 223). We evaluate 27 DMC tasks, cat- egorized into two groups: IMCC-asy&MMrn
and DMC-Hard All Humanoid and Dog tasks are classified DMC-Hard, while the remaining tasks grouped under DMC-Easy& Medium. The complete lists of DMC-Easy& Medium and DMC-Hard are provided in Tables 2 and 3, respectively.
For our URL experiments, adhere the protocol in Park al. (2023), using an episode length and 2 coordinates. For the
of 400 steps and augmenting the agent's observations with its representation function (i.c., reward network) METRA, we only used æ, y, and positions as input. This approach effectively replaces the colored floors used in the pixel-based humanoid setting of METRA with based inputs.
H.2 MYOSUITE
MyoSuite (Caggiano et al., 2022) simulates musculoskeletal movements with high-dimensional state and action spaces, focusing on physiologically accurate motor control. It includes benchmarks for complex object manipulation using dexterous hand. We evaluate 10 MyoSuite tasks, categorized as "easy" when the goal is fixed and "hard" when the goal is randomized, following Hansen et al. (2023). The full list of MyoSuite tasks is presented in Table
H.3 HUMANOIDBENCH
HumanoidBench (Sferrazza al., 2024) high-dimensional simulation benchmark designed to advance humanoid robotics research. It features the Unitree HI humanoid robot with dexterous hands, performing a variety of challenging locomotion and manipulation tasks. These tasks range from basic locomotion to complex activities requiring precise manipulation. The benchmark fa- cilitates algorithm development and testing without the need for costly hardware by providing a simulated platform. In experiments, we focus on 14 locomotion tasks, simplifying the setup by excluding the dexterous hands. This reduces the complexity associated with high degrees of free- dom and intricate dynamics. All HumanoidBench scores normalized based on each task's 'ss target success score as provided by the authors. complete list of tasks available in Table 5.
H.4 CRAFTAX
Craftax (Matthews et al., 2024) an open-ended RL environment that combines elements from Crafter (Hafner, 2022) and NetHack (Küttler et al., 2020). It presents challenging scenario requir- ing the sequential completion of numerous tasks. A key feature of Craftax is its support for vec- torized parallel environments and full end-to-end GPU learning pipeline, enabling large number
24

---

Published as a conference paper at ICLR 2025
of environment steps at low computational cost. The original baseline performance in Craftax is reported as percentage of the maximum score (226). our experiments, we report agents' raw average scores instead.
Table 2: DMC Easy & Medium. We consider total of 20 continuous control tasks for the DMC Easy & Medium benchmark. We list all considered tasks below and baseline performance for each task reported at 500K environment steps.
Task
Observation dim
Action dim
Acrobot Swingup
6
Cartpole Balance
5
Cartpole Balance Sparse
5
Cartpole Swingup
5
1
Cartpole Swingup Sparse
5
Cheetah Run
17
6
Finger Spin
9
2
Finger Turn Easy
12
2
Finger Turn Hard
12
2
Fish Swim
24
5
Hopper Hop
15
4
Hopper Stand
15
4
Pendulum Swingup
3
1
Quadruped Run
78
12
Quadruped Walk
78
12
Reacher Easy
6
2
Reacher Hard
6
2
Walker Run
24
6
Walker Stand
24
6
Walker Walk
24
6
Table 3: DMC Hard. We consider a total continuous control tasks for the DMC Hard bench- mark. We list all considered tasks below and baseline performance for each task is reported at 1M environment steps.
Task
Observation dim
Action dim
Dog Run
223
Dog Trot
223
38
Dog Stand
223
38
Dog Walk
223
38
Humanoid Run
67
24
Humanoid Stand
67
24
Humanoid Walk
67
24
25

---

Published as a conference paper at ICLR 2025
Table MyoSuite. We consider total of 10 continuous control tasks for the MyoSuite benchmark, which encompasses both fixed-goal ("Easy") and randomized ("Hard') settings. We list all considered tasks below and baseline performance for each task reported at 1M environment steps.
Task
Observation dim
Action dim
Key Turn Easy
93
39
Key Turn Hard
93
39
Object Hold Easy
91
39
Object Hold Hard
91
39
Pen Twirl Easy
83
39
Pen Twirl Hard
83
39
Pose Easy
108
39
Pose Hard
108
39
Reach Easy
115
39
Reach Hard
115
39
Table 5: HumanoidRench. We consider total of 14 continuous control locomotion tasks for the UniTree HI humanoid robot from the HumanoidBench domain. We list all considered tasks below and baseline performance for each task is reported at 2M environment steps.
Task
Observation dim
Action dim
Balance Hard
77
19
Balance Simple
64
19
Crawl
51
19
Hurdle
51
19
Maze
51
19
Pole
51
19
Reach
57
19
Run
51
19
Sit Simple
51
19
Sit Hard
64
19
Slide
51
19
Stair
51
19
Stand
51
19
Walk
51
19
Table 6: Craftax. We consider the symbolic version of Craftax. Baseline performance is reported
1B environment steps.
Task
Observation dim
Action dim
Craftax- Loittax-Symbolic-1
8268
43
26

---

Published as a conference paper at ICLR 2025
1 HYPERPARAMETERS
I.1 OFF-POLICY
Table 7: SAC+SimBa hyperparameters. The hyperparameters listed below used consistently across all tasks when integrating SimBa with SAC. For the discount factor, we set it automatically using heuristies used by TD-MPC2 (Hansen al., 2023).
Hyperparameter
Value
Critic block type
SimBa Residual
Critic num blocks
2
Critic hidden dim
512
Critic learning rate
lc-4
Target critic momentum (T)
5e-3
Actor block type
SimBa Residual
Actor num blocks
Actor hidden dim
128
Actor learning rate
le-4
Initial temperature (og)
1e-2
Temperature learning rate
le-4
Target entropy (H-)
|A//2
Batch size
256
Optimizer
AdamW
Optimizer momentum (81, 82)
(0.9, 0.999)
Weight decay (A)
le-2
Discount (?)
Heuristic
Replay ratio
2
Clipped Double Q
HumanoidBench: True
Other Envs: False
Table 8: DDPG+SimBa hyperparameters. The hyperparameters listed below are used consistently across all tasks when integrating SimBa with DDPG. For the discount factor, set it automatically using heuristics used by TD-MPC2 (Hansen et al., 2023).
Hyperparameter
Value
Critic block type
SimBa Residual
Critic num blocks
Critic hidden dim
512
Critic learning rate
le-4
Target critic momentum (r)
3e-3
Actor block type
SimBa Residual
Actor num blocks
Actor hidden dim
128
Actor learning rate
le44
Exploration noise
(0 0.1")
Batch size
256
Optimizer
Adam
Optimizer momentum (81, (2)
(0.9, 0.999)
Weight decay (A)
lc-2
Discount (7)
Heuristic
Replay ratio
2
27

---

Published as a conference paper at ICLR 2025
Table 9: TDMPC2+SimBa hyperparameters. We provide detailed list of the hyperparameters used for the shared encoder module in TD-MPC2 (Hansen et al., 2023). Aside from these, we follow the hyperparameters specified in the original TD-MPC2 paper. The listed hyperparameters are applied uniformly across all tasks when integrating SimBa with TD-MPC2.
Hyperparameter
Value
Encoder block type Encoder num blocks
SimBa Residual
2
Encoder hidden dim
256
Encoder learning rate
le-4
Encoder weight decay
le-1
1.2 ON-POLICY
Table 10: PPO+SimBa hyperparameters. We provide detailed list of the hyperparameters when integrating SimBa with PPO (Schulman et al., 2017). Aside from these, follow the hyperparam- eters specified in the Craftax paper (Matthews et al., 2024).
Hyperparameter
Value
Critic block type
SimBa Residual
Critic num blocks
Critic hidden dim
512
Critic learning rate
le-4
Actor block type
SimBa Residual
Actor num blocks
Actor hidden dim
256
Actor learning rate
le-4
Optimizer
AdamW
Optimizer momentum (21,
(0.9, 0.999)
Optimizer eps
1e-8
Weight decay (A)
le-2
UNSUPERVISED RL 1.3
Table 11: SimBA+ hyperparameters. We provide detailed list of the hyperparameters when integrating SimBa with METRA (Park et al., 2023). Aside from these, we follow the hyper- parameters specified in the original METRA paper.
Hyperparameter
Value
Critic block type
SimBa Residual
Critic num blocks
Critic hidden dim
512
Actor block type
SimBa Residual
Actor num blocks
Actor hidden dim
128
28

---

Published as a conference paper at ICLR 2025
J LEARNING CURVE
Swingup
Cartpole
Sparse
Compoocc Swingup
Cortpoic SW wwingup Sparso
1,200
750
250
0
Cheetah Hun
Finger Spin
Finger turn tasy
tinger turn Hard
Fish swinn
750
300
250
Hopper Hop
Hopper
Pendulum Swingup
Quadruped
1.000
750
500
Reacher Easy
Reacher Hard
Welker Run
Walker Stand
Walker Walk
1.000
250
Step
(M)
SAC + SimBa
BRO
TD-MPC2
TD7
Dreomerv3
SAC
Figure 21: Per task learning curve for DMC Easy Medium. Mean and 95% CI over 10 seeds for SimBa and BRO, 5 seeds for TD7 and SAC, seeds for TD-MPC2 and DreamerV3,
Dec Stand
Dog That
Walk
Humanote
1.000
750
250
0
Humanoid Stard
Humanoid Walk
750
500
0.2
0.5
0.8
1.0
0.3
0.5
0.5
1.0
Enw Step (M)
Ery Step (M)
Figure 22: Per task learning curve for DMC Hard. Mean and 95% CI over 10 seeds for SimBa and BRO, 5 seeds for TD7 and SAC, seeds for TD-MPC2 and D3eamerV3.
29

---

Published as a conference paper at ICLR 2025
Key lurn basy
Key lumn Herd
Ooj Hold Easy
Obj Hold riard
Men [wirl Lasy
www
50
25
Pose
Hard
Reach
Rearhh
130
25
Env Step
ErV
Erv Step (M)
Enw Step (M)
SAC
SimBa
BRD
TD:MFC2
TD7
Dreamerva
SAC
Figure 23: Per task learning curve for MyoSuite. Mean and 95% Cl over 10 seeds for SimBa and BRO, 5 seeds for TD7 and SAC, 3 seeds for TD-MPC2 and DreamerV3.
Balance Hard
Balance Simple
Crawi
Hurdle
1,200
800
Pole
Reach
Run
Sit Hard
Sit Simple
1.200
800
400
Stair
Stand
Walk
1.500
830
800
Step
Emv Step (M)
Step
SimBo
Dreamorva
Figure 24: Per task learning curve for HumanoidBench. Mean and 95% CI over 10 seeds for SimBa, seeds for BRO, TD7, and SAC, 3 seeds for TD-MPC2 and Dreamer V3. We no
30

---

Published as a conference paper at ICLR 2025
Cas: loboll
Collect Cool
Collect Diamond
Collect Drink
Collect iron
Collect Ruby
180
25
Collect Sapling
Collct Sagohire
Collect Stone
Collect Wood
Do raae Nezromanncer
Deteat Archer
100
Defeat
Semental
Defeat Thail
Defeat
Deleat Wartio
Defezt Elemental
100
0
Necromancer
Nage
100
Defcat Pigmar
Deleat Skeleton
Deteat Tra
Defeat Zomble
ni Potton
Eat Bat
100
25
Eat Cow
Eat Plant
Eat Shall
Enchant Armour
Enchant Sword
Enter Dungeon
Enter
Enter Gnomish
100
0
100
Make Glamond Armour
Mase Diamond Fickave
Malte Diamond Bword
Make Iron Armour
Make Iron Pickasee
Make Iron Sword
100
25
Male Stone Pickare
Nake Store Sword
Make Türch
Make Wlod Pickaxe
Mare Wood Sword
Open Chest
Ploce
Place Part
Pboc
Place
Place
100
50
0
Step (8)
Stea (8)
Step
Step
Step (R)
Frw Step (R)
PPO
SimBa
PPO
Figure 25: Per task learning curve for Craftax. We visualize the success learning curve for 66 tasks in Craftax. Mean and 95% CI over 5 seeds for PPO + SimBa and PPO.
31

---

Published as a conference paper at ICLR 2025
K FULL RESULT
Table 12: Per task results for DMC Easy Medium. Results for SimBa and BRO are averaged
10 seeds, for TD7 and SAC over seeds, and for TD-MPC2 and DreamerV3 over 3 seeds.
Method
SimBa
BRO
TD7
SAC
TD-MPC2
DreamerV3 V3
Acrobot Swingup
331.57
390.78
39.83
57.56
361.07
360.46
Cartpole Balance
999.05
998.66
998.79
998.79
993.93
994.95
Cartpole Balance Sparse
940.53
954.21
988.58
1000.00
1000.00
800.25
Cartpole Swingup
866.53
878.69
878.09
863.24
876.07
863.90
Cartpole Swingup Sparse
823.97
833.18
480.74
779.99
844.77
262.69
Cheetah Run
814.97
739.67
897.76
716.43
757.60
686.25
Finger Spin
778.83
987.45
592.74
814.69
984.63
434.06
Finger Turn Easy
881.33
905.85
485.66
903.07
854.67
851.11
Finger Turn Hard
860.22
905.30
596.32
775.21
876.27
769.86
Fish Swim
786.73
680.34
108.84
462.67
610.23
603.34
Hopper Hop
326.69
315.04
110.27
159.43
303.27
192.70
Hopper Stand
811.75
910.88
445.50
845.89
936.47
722.42
Pendulum Swingup
824.53
816.20
461.40
476.58
841.70
825.17
Quadruped Run
883.68
818.62
515.13
116.91
939.63
471.25
Quadruped Walk
952.96
936.06
910.55
147.83
957.17
472.31
Reacher Easy
972.23
933.77
920.38
951.80
919.43
888.36
Reacher Hard
965.96
956.52
549.50
959.59
913.73
935.25
Walker Run
687.16
754.43
782.32
629.44
820.40
620.13
Walker Stand
983.02
986.58
984.63
972.59
957.17
963.28
Walker Walk
970.73
973.41
976.58
956.67
978.70
925.46
IQM
885.70
888.02
740.19
799.75
895.47
748.58
Median
816.78
836.62
623.18
676.59
836.93
684.18
Mean
823.12
833.78
636.18
679.42
836.34
682.16
OG
0.1769
0.1662
0.3638
0.3206
0.1637
0.3178
Table 13: Per task results for DMC Hard. Results for SimBa and BRO averaged over 10 seeds, for TD7 and SAC over 5 seeds, and for TD-MPC2 and Dreamer V3 over 3 seeds.
Method
SimBa
BRO
TD7
SAC
TD-MPC2
DreamerV3
Dog Run
544.86
374.63
127.48
36.86
169.87
15.72
Dog Stand
960.38
966.97
753.23
102.04
798.93
55.87
Dog Trot
824.69
783.12
126.00
29.36
500.03
10.19
Dog Walk
916.80
931.46
280.87
38.14
493.93
23.36
Humanoid Run
181.57
204.96
79.32
116.97
184.57
0.91
Humanoid Stand
846.11
920.11
389.80
352.72
663.73
5.12
Humanoid Walk
668.48
672.55
252.72
273.67
628.23
1.33
IQM
773.28
771.50
216.04
69.03
527.11
9.63
Median
706.39
694.20
272.62
159.36
528.26
17.13
Mean
706.13
693.40
287.06
135.68
491.33
16.07
OG
0.2939
0.3066
0.7129
0.8643
0.5087
0.9839
32

---

Published as a conference paper at ICLR 2025
Table 14: Per task results for MyoSuite. Results for SimBa and BRO are averaged over 10 seeds, for TD7 and SAC over seeds, and for TD-MPC2 and DreamerV3 over 3 seeds.
Method
SimBa
BRO
TD7
SAC
TD-MPC2
Dreamer V3
Key Turn Easy
100.00
100.00
100.00
100.00
100.00
88.89
Key Turn Hard
7.00
42.00
0.00
10.00
0.00
0.00
Object Hold Easy
90.00
90.00
20.00
90.00
100.00
33.33
Object Hold Hard
96.00
42.00
10.00
96.00
56.67
9.44
Pen Twirl Easy
80.00
90.00
100.00
50.00
70.00
96.67
Pen Twirl Hard
77.00
76.00
12.00
55.00
40.00
53,33
Pose Easy
100.00
100.00
0,00
90.00
100.00
100.00
Pose Hard
0.00
0.00
0.00
0.00
0.00
0.00
Reach Easy
100.00
100.00
100.00
100.00
100.00
100.00
Reach Hard
93.00
74.00
14.00
16.00
83.33
0.00
IQM
95.20
87.60
22.31
71.40
77.50
46.56
Median
77.00
72.00
34.00
62.50
65.00
47.00
Mean
74.30
71.40
35.60
60.70
65.00
48.17
OG
99.93
99.93
99.96
99.94
99.94
99.95
Table 15: Per task results for HumanoidBench. Results for SimBa averaged 10 seeds, for BRO, TD7 and SAC over seeds, and for TD-MPC2 and DreamerV3 V3 over 3 seeds.
Method
SimBa
BRO
TD7
SAC
TD-MPC2
Dreamer
Balance Hard
137.20
145.95
79.90
69.02
64.56
16.07
Balance Simple
816.38
246.57
132.82
113.38
50.69
14.09
Crawl
1370.51
1373.83
868.63
830.56
1384.40
906.66
Hurdle
340.60
128.60
200.28
31.89
1000.12
18.78
Maze
283.58
259.03
179.23
254.38
198.65
114.90
Pole
1036.70
915.89
830.72
760.78
1269.57
289.18
Reach
523.10
317.99
159.37
347.92
505.61
341.62
Run
741.16
429.74
196.85
168.25
1130.94
85,93
Sit Hard
783.95
989.07
293.96
345.65
1027.47
9.95
Sit Simple
1059.73
1151.84
1183.26
994.58
1074.96
40.53
Slide
577.87
653.17
197.07
208.46
780.82
24.43
Stair
527.49
249.86
77.19
65.53
398.21
49.04
Stand
906.94
780.52
1005.54
1029.78
1020.59
280.99
Walk
1202.91
1080.63
143.86
412.21
1165.42
125.59
IQM
747.43
558.03
256.77
311.46
885.67
72.30
Median
733.09
632.93
385.41
399.93
796.
160.49
Mean
736.30
623,05
396.33
402.31
787.64
165.55
OG
0.3197
0.4345
0.6196
0.6016
0.2904
0.8352
33


Reviews:
[{"review_id": "eQWBJ2mc6A", "reviewer": "Reviewer_9P9G", "summary": "The paper presents SimBa, an architecture specifically designed for deep reinforcement learning that scales up network parameters by integrating a simplicity bias. The SimBa architecture employs observation normalization, residual feedforward blocks, and layer normalization, fostering simplicity in function representation. By adopting these components, SimBa enhances the sample efficiency and compute efficiency of various RL algorithms. It performs comparably or better than state-of-the-art methods in diverse RL benchmarks, demonstrating effectiveness across off-policy, on-policy, and unsupervised settings.", "questions": "1. I am curious about the scaling results on multi-task RL policies, since the proposed architecture is added to the representation part. By the way, maybe the author should also compare the results with baseline methods under scaled-up backbones.\n\n2. The author chose a new metric named simplicity bias score for comparison and analysis. The results also show some effects of the initial parameter distributions. Why not compare with the baseline methods with initialization periodically?\n\nNikishin, E., Schwarzer, M., D’Oro, P., Bacon, P. L., & Courville, A. (2022, June). The primacy bias in deep reinforcement learning. In International conference on machine learning (pp. 16828-16847). PMLR.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Innovative Application of Simplicity Bias**: The use of simplicity bias in SimBa to manage overparameterization effectively is a novel contribution to deep RL.\n\n2. **Comprehensive Empirical Validation**: SimBa's performance is rigorously tested across multiple RL tasks, including DMC, MyoSuite, and HumanoidBench, showing consistent improvements in efficiency and scalability.\n\n3. **Adaptability**: SimBa’s architecture is algorithm-agnostic and can integrate seamlessly with various RL algorithms, allowing its broad applicability.\n\n4. **Effective Design Choices**: The architectural components (e.g., RSNorm and residual connections) seem thoughtfully selected to reduce complexity while enabling parameter scaling.", "weaknesses": "1. Missing relation works about normalization. Researchers show that the RMS Norm works well in training foundation models. The proposed RSNorm is too similar to the RMS Norm, while the author has no discussion about the RMS Norm (even not in related work). Since this work investigates designing and scaling up networks in deep RL, the same tricks in designing and scaling up LLMs should be considered.\n\n2. Unfair comparison between SAC+SimBa v.s. others. Since SAC+SimBa introduces some other layers, it might include more parameters for the neural networks. The author can make some ablations with different sizes of SAC to exclude the capacity issue.\n\n3. Missing details of the model size. I didn't find the detailed sizes of each model. If the sizes are small (such as 10M), the conclusions about the so-called \"scaling up\" are doubtful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BF23Kd7BJY", "reviewer": "Reviewer_F5aT", "summary": "The paper introduces SimBa (Simplicity Bias), a novel architecture designed to scale up parameters in deep reinforcement learning (RL) by incorporating simplicity bias. The key components of SimBa are:\n- Observation normalization layer\n- Residual feedforward block\n- Layer normalization\n\nThe authors demonstrate that SimBa exhibits higher simplicity bias compared to standard MLPs and shows consistent performance improvements as the number of parameters increases. When integrated with various RL algorithms, including SAC, TD-MPC2, PPO, and METRA, SimBa improves sample efficiency. Moreover, SimBa matches or surpasses state-of-the-art off-policy RL methods across 51 continuous control tasks while maintaining computational efficiency", "questions": "- Applicability: Are there any specific types of RL problems or environments where SimBa might not be as effective?\n- Why are the standard errors in Figure 10 so high?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Novel approach: SimBa addresses an important gap in deep RL research by exploring how to scale up network parameters while leveraging simplicity bias effectively.\n- Versatility: The architecture improves sample efficiency across various RL algorithms, including off-policy, on-policy, and unsupervised methods.\n- Performance: When applied to SAC, SimBa matches or surpasses state-of-the-art off-policy RL methods across a wide range of tasks.\n- Computational efficiency: SimBa achieves high performance without relying on computationally intensive components or complex training protocols.\n- Theoretical foundation: The authors provide a clear explanation of simplicity bias and how it's measured, grounding their work in existing theory.\n- Significance: This work addresses an important gap in deep RL research by exploring how to effectively scale up network parameters while leveraging simplicity bias. The SimBa architecture offers a promising approach to improve performance across various RL algorithms and tasks without relying on computationally intensive components or complex training protocols", "weaknesses": "- While the evaluation covers 51 continuous control tasks, it might be beneficial to see SimBa's performance on a wider range of RL domains, particularly with images.\n- The paper doesn't discuss potential limitations or scenarios where SimBa might not be as effective", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FDyJcQmWaZ", "reviewer": "Reviewer_2Hpz", "summary": "The authors propose the SimBa architecture, which, by inducing simplicity bias, allows scaling the size of neural networks in deep reinforcement learning problems. The proposed architecture is a neural network with residual connections, with post-layer normalization and observation normalization at the beginning. The authors claim that SimBa enables networks to scale up without sacrificing generalizability, avoiding the overfitting issues often associated with high-parameter models in RL.", "questions": "1. Out of curiosity, SimBa is usually the most time-efficient (except SAC), but on HumanoidBench, it is slightly worse than TD7 and BRO. Do you know why?\n2. The simplicity bias hypothesis looks pretty general; therefore, how will this architecture work on image-based problems?\n3.  How do you expect SimBa to be applied in the discrete control tasks? Will it be enough to beat BBF on Atari as BRO on continuous control? -- Of course, new experiments on Atari would be too much for the rebuttal period, but maybe the authors have some thoughts about this.\n4. How do your results about simplicity bias measured through Fourier features relate to this paper [1]? As I understand, authors of [1] claim that neural networks are naturally biased to learn low frequencies faster, and it is hard for them to learn high-frequency signals, which is present in reinforcement learning. SimBa promotes architectures that are biased towards low frequencies. Could you share your perspective on how these findings interact or complement each other?\n\n[1] Yang, G., Ajay, A., & Agrawal, P. (2022). Overcoming the spectral bias of neural value approximation. arXiv preprint arXiv:2206.04672.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Clear and Well-Motivated Objective: The paper identifies simplicity bias as an underexplored factor in RL network scaling.\n2. Reproducibility Efforts: A public codebase and descriptions of evaluation setups are provided.\n3. Broad experimental setup.", "weaknesses": "1. Statistical significance is not apparent from the presented results. The standard error overlaps in some plots, e.g., Figures 1b, BRO, and SimBa on Figure 5b, and Figure 8 has no rages, but I guess it would be more challenging to add them since the analysis is in two dimensions, and also from Figure 14. The tables in the Appendix do not have standard deviations. Moreover, authors could use some t-test or Mann–Whitney U test to demonstrate statistical significance.\n2. It would be valuable to see reversed to Fig. 4 ablation, i.e., simplicity and Return when you turn off every single component. Clearly, all components analyzed in section 5.1 have some synergic effect -- they do not help much alone. However, it is not apparent how all of them are important in this synergy, i.e., from section 7.1, we know that RSNorm is very important -- If I understand correctly, SimBa without observation normalization works very poorly. But how will the lack of RSNorm affect the simplicity of the architecture? -- I would suggest visualizing this similarly to Figure 4, but on the Y-axis, it would be, e.g., SimBa - RSNorm or SimBa - Residual.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "09uyjLFSHV", "reviewer": "Reviewer_8rZo", "summary": "This paper builds upon the insights of the paper by [Teney et al, 2024], which shows that certain architectural implementations provide a simplicity bias, which could be one of the more causal explanations to their practical usefulness.\n\nIn order to transfer this notion of simplicity bias to RL, the authors run tests with SAC on proprietary state-based RL, ranging from easy (Cartpole etc.) to hard (Humanoid, Dog, etc.) environments.\n\nTo maximize the simplicity bias of vanilla SAC, the authors augment the original architecture with RSnorm, Residual Feedforward blocks and Layer Normalization. Additionally, the authors scale up the architecture.\n\nThe results are promising, showing that keeping a simplicity bias in the network allows for scaling up the network, elevating vanilla SAC to the level where it can perform very well on DMC-Hard tasks, without using algorithmic advancements. \n\nAlthough results are only done on state-based architectures, the harder environments like Humanoid & Dog still allow for sufficient difficulty in the evaluation. Additionally, pixel-based environments have already shown correlation between simplicity bias in the CNN and performance (Residual connections in CNN’s -> Impala architecture), although to the best of my knowledge, the reason being a simplicity bias was never concluded by these researchers.", "questions": "See Weaknesses section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Easy to implement, which is promising for future work.\n\n- Good comparison with a SOTA algorithm TD-MPC2, and even an ablation using SimBa on TD-MPC2.\n\n- A strong step forward in the domain of non-algorithmic improvements in RL.", "weaknesses": "- The plasticity analysis in Appendix C. could be confusing to the average reader. It shows a comparison to a basically ‘collapsed’ baseline MLP, as the effective rank is negligible. This might give a reader the impression that these are the representative network properties of training with a vanilla MLP, and paints a too strong picture of SimBa’s added network properties. I believe it would therefore be much more informative to the reader to add an additional 2 rows of figures to Fig. 15:\n \t\t- Row 1: Plasticity analysis on DMC - medium\n                - Row 2: Plasticity analysis with all the ablations of Fig. 4, to show the unique effects of every module.\n\n- As important as it is to see that SimBa is computationally cheap, it would be also interesting to add some figures showing longer training (e.g. 5 million timesteps) on DMC hard and comparing to TD-MPC2 and the baseline SAC.\n\n- I believe a mention of the improvements that the Impala network gave to RL should be in the paper, and maybe a short sentence about the correlation (Residuals connections -> Simplicity Bias -> better Convergence).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)