SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning

## Abstract

Abstract Recent advances in CV and NLP have been largely driven by scaling up the number of network parameters, despite traditional theories suggesting that larger networks are prone to overfitting.
These large networks avoid overfitting by integrating components that induce a simplicity bias , guiding models toward simple and generalizable solutions.
However, in deep RL, designing and scaling up networks have been less explored.
Motivated by this opportunity, we present SimBa , an architecture designed to scale up parameters in deep RL by injecting a simplicity bias. SimBa consists of three components: (i) an observation normalization layer that standardizes inputs with running statistics, (ii) a residual feedforward block to provide a linear pathway from the input to output, and (iii) a layer normalization to control feature magnitudes.
By scaling up parameters with SimBa, the sample efficiency of various deep RL algorithms—including off-policy, on-policy, and unsupervised methods—is consistently improved.
Moreover, solely by integrating SimBa architecture into SAC, it matches or surpasses state-of-the-art deep RL methods with high computational efficiency across DMC, MyoSuite, and HumanoidBench.
These results demonstrate SimBa’s broad applicability and effectiveness across diverse RL algorithms and environments.

## 1 Introduction

Scaling up neural network sizes has been a key driver of recent advancements in computer vision (CV) (Dehghani et al., ) and natural language processing (NLP) (Google et al., ; Achiam et al., ). By increasing the number of parameters, neural networks gain enhanced expressivity, enabling them to cover diverse functions and discover effective solutions that smaller networks might miss. However, this increased capacity also heightens the risk of overfitting, as larger networks can fit intricate patterns in the training data that do not generalize well to unseen data.

Despite this risk of overfitting, empirical evidence shows that neural networks tend to converge toward simpler functions that generalize effectively (Kaplan et al., ; Nakkiran et al., ). This phenomenon is attributed to a *simplicity bias* inherent in neural networks, where standard optimization algorithms and architectural components guide highly expressive models toward solutions representing simple, generalizable functions (Shah et al., ; Berchenko, ).
For instance, gradient noise in stochastic gradient descent prevents models from converging on sharp local minima, helping them avoid overfitting (Chizat & Bach, ; Gunasekar et al., ; Pesme et al., ).
Architectural components such as ReLU activations (Hermann et al., ), layer normalization (Ba et al., ), and residual connections (He et al., ) are also known to amplify simplicity bias. These components influence the types of functions that neural networks represent at initialization, where networks that represent simpler function at initialization are more likely to converge to simple functions (Valle-Perez et al., ; Mingard et al., ; Teney et al., ).

While scaling up network parameters and leveraging simplicity bias have been successfully applied in CV and NLP, these principles have been underexplored in deep reinforcement learning (RL), where the focus has primarily been on algorithmic advancements (Hessel et al., ; Hafner et al., ; Fujimoto et al., ; Hansen et al., ). Motivated by this opportunity, we introduce the SimBa network, a novel architecture that explicitly embeds simplicity bias to effectively scale up parameters in deep RL. SimBa comprises of three key components: (i) an observation normalization layer that standardizes inputs by tracking the mean and variance of each dimension, reducing overfitting to high-variance features (Andrychowicz et al., ); (ii) a pre-layer normalization residual feedforward block (Xiong et al., ), which maintains a direct linear information pathway from input to output and applies non-linearity only when necessary; and (iii) a post-layer normalization before the output layer to stabilize activations, ensuring more reliable policy and value predictions.

To verify whether SimBa amplifies simplicity bias, we compared it against the standard MLP architecture often employed in deep RL. Following Teney et al. (), we measured simplicity bias by (i) sampling random inputs from a uniform distribution; (ii) generating network outputs; and (iii) performing Fourier decomposition on these outputs. A smaller sum of the Fourier coefficients indicates that the neural network represents a low-frequency function, signifying greater simplicity. We define the simplicity score as the inverse of this sum, meaning a higher score corresponds to a stronger simplicity bias. As illustrated in Figure 2.(a), our analysis revealed that SimBa has a higher simplicity score than the MLP (Further details are provided in Section 2).

To evaluate the benefit of leveraging simplicity bias on network scaling, we compared the performance of Soft Actor-Critic (Haarnoja et al., , SAC) using both MLP and our SimBa architecture across 3 humanoid tasks from DMC benchmark (Tassa et al., ). We increased the width of both actor and critic networks, scaling from 0.1 to 17 million parameters. As shown in Figure 2.(b), SAC with MLP experiences performance degradation as the number of parameters increases. In contrast, SAC with the SimBa network consistently improves its performance as the number of parameter increases, highlighting the value of embedding simplicity bias when scaling deep RL networks.

Figure: Figure 2: (a) SimBa exhibits higher simplicity bias than MLP. (b) SAC with SimBa improves its performance with increased parameters, whereas SAC with MLP degrades it. Each standard deviation is 95% CI.

To further evaluate SimBa’s versatility, we applied it to various RL algorithms by only changing the network architecture and scaling up parameters. The algorithms included off-policy (SAC (Haarnoja et al., ), TD-MPC2 (Hansen et al., )), on-policy model-free (PPO (Schulman et al., )), and unsupervised RL (METRA (Park et al., )).
As illustrated in Figure 1.(a), SimBa consistently enhances the sample efficiency of these algorithms. Furthermore, as shown in Figure 1.(b), when SimBa is integrated into SAC, it matches or surpasses state-of-the-art off-policy methods across 51 tasks in DMC, MyoSuite (Caggiano et al., ), and HumanoidBench (Sferrazza et al., ).
Despite the increased number of parameters, SAC with SimBa remains computationally efficient because it does not employ any computationally intensive components such as self-supervised objectives (Fujimoto et al., ), planning (Hansen et al., ), or replay ratio scaling (Nauman et al., ), which state-of-the-art methods rely on to achieve high performance.

## 2 Preliminary

Simplicity bias refers to the tendency of neural networks to prioritize learning simpler patterns over capturing intricate details (Shah et al., ; Berchenko, ). In this section, we introduce metrics to quantify simplicity bias; in-depth definitions are provided in Appendix A.

### 2.1 Measuring Function Complexity

We begin by defining a complexity measure $c:\mathcal{F}\to[0,\infty)$ to quantify the complexity of functions in $\mathcal{F}=\{f|f:\mathcal{X}\rightarrow\mathcal{Y}\}$ where $\mathcal{X}\subseteq\mathbb{R}^{n}$ denote the input space and $\mathcal{Y}\subseteq\mathbb{R}^{m}$ the output space.

Traditional complexity measures, such as the Vapnik–Chervonenkis dimension (Blumer et al., ) and Rademacher complexity (Bartlett & Mendelson, ), are well-established but often intractable for deep neural networks.
Therefore, we follow Teney et al. () and adopt Fourier analysis as our primary complexity measure.
Given $f(x):=(2\pi)^{{d}/{2}}\int\tilde{f}(k)\,e^{ik\cdot x}dk$ where $\tilde{f}(k):=\int f(x)\,e^{-ik\cdot x}dx$ is the Fourier transform, we perform a discrete Fourier transform by uniformly discretizing the frequency domain, $k\in\{0,1,\dots,K\}$. The value of $K$ is chosen by the Nyquist-Shannon limit (Shannon, ) to ensure accurate function representation.
Our complexity measure $c(f)$ is then computed as the frequency-weighted average of the Fourier coefficients:

$$ $c(f)~{}=~{}\Sigma_{k=0}^{K}\tilde{f}(k)\cdot k~{}~{}/~{}~{}\Sigma_{k=0}^{K} \tilde{f}(k).$ (1) $$

Intuitively, larger $c(f)$ indicates higher complexity due to a dominance of high-frequency components such as rapid amplitude changes or intricate details. Conversely, lower $c(f)$ implies a larger contribution from low-frequency components, indicating a lower complexity function.

### 2.2 Measuring Simplicity Bias

In theory, simplicity bias can be measured by evaluating the complexity of the function to which the network converges after training. However, directly comparing simplicity bias across different architectures after convergence is challenging due to the randomness of the non-stationary optimization process, especially in RL, where the data distribution changes continuously.

Empirical studies suggest that the initial complexity of a network strongly correlates with the complexity of the functions it converges to during training (Valle-Perez et al., ; De Palma et al., ; Mingard et al., ; Teney et al., ).
Therefore, for a given network architecture $f$ with an initial parameter distribution $\Theta_{0}$, we define the simplicity bias score $s(f):\mathcal{F}\rightarrow(0,\infty)$ as:

$$ $s(f)\approx\mathbb{E}_{\theta\sim\Theta_{0}}\left[\frac{1}{c(f_{\theta})}\right]$ (2) $$

where $f_{\theta}$ denotes the network architecture $f$ parameterized by $\theta$.

This measure indicates that networks with lower complexity at initialization are more likely to exhibit a higher simplicity bias, thereby converging to simpler functions during training.

## 3 Related Work

### 3.1 Simplicity Bias

Initially, simplicity bias was mainly attributed to the implicit regularization effects of the stochastic gradient descent (SGD) optimizer (Soudry et al., ; Gunasekar et al., ; Chizat & Bach, ; Pesme et al., ).
During training, SGD introduces noise which prevents the model from converging to sharp minima, guiding it toward flatter regions of the loss landscape (Wu et al., ). Such flatter minima are associated with functions of lower complexity, thereby improving generalization.

However, recent studies suggest that simplicity bias is also inherent in the network architecture itself (Valle-Perez et al., ; Mingard et al., ). Architectural components such as normalization layers, ReLU activations (Hermann et al., ), and residual connections (He et al., ) promote simplicity bias by encouraging smoother, less complex functions.
Fourier analysis has shown that these components help models prioritize learning low-frequency patterns, guiding optimization toward flatter regions that generalize better (Teney et al., ). Consequently, architectural design plays a crucial role in favoring simpler solutions, enabling the use of overparameterized networks.

### 3.2 Deep Reinforcement Learning

For years, deep RL has largely focused on algorithmic improvements to enhance sample efficiency and generalization. Techniques like Double Q-learning (Van Hasselt et al., ; Fujimoto et al., ), and Distributional RL (Dabney et al., ) have improved the stability of value estimation by reducing overestimation bias.
Regularization strategies—including periodic reinitialization (Nikishin et al., ; Lee et al., ), Layer Normalization (Lee et al., ; Gallici et al., ), Batch Normalization (Bhatt et al., ), and Spectral Normalization (Gogianu et al., )—have been employed to prevent overfitting and enhance generalization. Incorporating self-supervised objectives with model-based learning (Fujimoto et al., ; Hafner et al., ; Hansen et al., ) has further improved representation learning and sample efficiency.

Despite these advances, scaling up network architectures in deep RL remains underexplored. While larger models improved performance in supervised learning by leveraging simplicity bias, this principle has not been fully explored in RL. Several recent studies have attempted to scale up network sizes—through ensembling (Chen et al., ; Obando-Ceron et al., ), widening (Schwarzer et al., ), and deepening networks (Bjorck et al., ; Nauman et al., ). However, they often rely on computationally intensive layers like spectral normalization (Bjorck et al., ) or require sophisticated training protocols (Nauman et al., ), limiting their applicability.

In this work, we aim to design an architecture that amplifies simplicity bias, enabling us to effectively scale up parameters in RL, independent of using any other sophisticated training protocol.

Figure: Figure 3: SimBa architecture. The network integrates Running Statistics Normalization (RSNorm), residual feedforward blocks, and post-layer normalization to embed simplicity bias into deep RL.
Refer to caption: x3.png

## 4 SimBa

This section introduces the SimBa network, an architecture designed to embed simplicity bias into deep RL. The architecture comprises Running Statistics Normalization, Residual Feedforward Blocks, and Post-Layer Normalization.
By amplifying the simplicity bias, SimBa allows the model to avoid overfitting for highly overparameterized configurations.

Running Statistics Normalization (RSNorm). First, RSNorm standardizes input observations by tracking the running mean and variance of each input dimension during training, preventing features with disproportionately large values from dominating the learning process.

Given an input observation $\mathbf{o}_{t}\in\mathbb{R}^{d_{o}}$ at timestep $t$, we update the running observation mean $\mathbf{\mu}_{t}\in\mathbb{R}^{d_{o}}$ and variance $\mathbf{\sigma}^{2}_{t}\in\mathbb{R}^{d_{o}}$ as follows:

$$ $\mathbf{\mu}_{t}=\mathbf{\mu}_{t-1}+\frac{1}{t}\delta_{t},\quad\mathbf{\sigma} _{t}^{2}=\frac{t-1}{t}(\mathbf{\sigma}_{t-1}^{2}+\frac{1}{t}\delta_{t}^{2})$ (3) $$

where $\delta_{t}=\mathbf{o}_{t}-\mathbf{\mu}_{t-1}$ and $d_{o}$ denotes the dimension of the observation.

Once $\mu_{t}$ and $\sigma^{2}_{t}$ are computed, each input observation $\mathbf{o}_{t}$ is normalized as:

$$ $\mathbf{\bar{o}}_{t}=\text{RSNorm}(\mathbf{o}_{t})=\frac{\mathbf{o}_{t}- \mathbf{\mu}_{t}}{\sqrt{\mathbf{\sigma}_{t}^{2}+\epsilon}}$ (4) $$

where $\mathbf{\bar{o}}_{t}\in\mathbb{R}^{d_{o}}$ is the normalized output, and $\epsilon$ is a small constant for numerical stability.

While alternative observation normalization methods exist, RSNorm consistently demonstrates superior performance. A comprehensive comparison is provided in Section 7.1.

Residual Feedforward Block. The normalized observation $\mathbf{\bar{o}}_{i}$ is first embedded into a $d_{h}$-dimensional vector using a linear layer:

$$ $\mathbf{x}^{l}_{t}=\text{Linear}(\mathbf{\bar{o}}_{t}).$ (5) $$

At each block, $l\in\{1,...,L\}$, the input $\mathbf{x}_{t}^{l}$ passes through a pre-layer normalization residual feedforward block, introducing simplicity bias by allowing a direct linear pathway from input to output. This direct linear pathway enables the network to pass the input unchanged unless non-linear transformations are necessary. Each block is defined as:

$$ $\mathbf{x}_{t}^{l+1}=\mathbf{x}_{t}^{l}+\text{MLP}(\text{LayerNorm}(\mathbf{x} _{t}^{l})).$ (6) $$

Following Vaswani (), the MLP is structured with an inverted bottleneck, where the hidden dimension is expanded to $4\cdot d_{h}$ and a ReLU activation is applied between the two linear layers.

Post-Layer Normalization. To ensure that activations remain on a consistent scale before predicting the policy or value function, we apply layer normalization after the final residual block:

$$ $\mathbf{z}_{t}=\text{LayerNorm}(\mathbf{x}_{t}^{L}).$ (7) $$

The normalized output $\mathbf{z}_{t}$ is then processed through a linear layer, generating predictions for the actor’s policy or the critic’s value function.

## 5 Analyzing SimBa

In this section, we analyze whether each component of SimBa amplifies simplicity bias and allows scaling up parameters in deep RL.
We conducted experiments on challenging environments in DMC involving Humanoid and Dog, collectively referred to as DMC-Hard. Throughout this section, we used Soft Actor Critic (Haarnoja et al., , SAC) as the base algorithm.

### 5.1 Architectural Components

To quantify simplicity bias in SimBa, we use Fourier analysis as described in Section 2. For each network $f$, we estimate the simplicity bias score $s(f)$ by averaging over 100 random initializations $\theta\sim\Theta_{0}$. Following Teney et al. (), the input space $\mathcal{X}=[-100,100]^{2}\subset\mathbb{R}^{2}$ is divided into a grid of 90,000 points, and the network outputs a scalar value for each input which are represented as a grayscale image.
By applying the discrete Fourier transform to these outputs, we compute the simplicity score $s(f)$ defined in Equation 2 (see Appendix B for further details).

Figure 4.(a) shows that SimBa’s key components—such as residual connections and layer normalization—increase the simplicity bias score $s(f)$, biasing the architecture toward simpler functional representations at initialization. When combined, these components induce a stronger preference for low-complexity functions (i.e., high simplicity score) than when used individually.

Figure 4.(b) reveals a clear relationship between simplicity bias and performance: architectures with higher simplicity bias scores lead to enhanced performance.
Specifically, compared to the MLP, adding residual connections increases the average return by 50 points, adding layer normalization adds 150 points, and combining all components results in a substantial improvement of 550 points.

Figure: Figure 4: Component Analysis. (a) Simplicity bias scores estimated via Fourier analysis. Mean and 95% CI are computed over 100 random initializations. (b) Average return in DMC-Hard at 1M steps. Mean and 95% CI over 10 seeds, using the SAC algorithm. Stronger simplicity bias correlates with higher returns for overparameterized networks.

### 5.2 Comparison with Other Architectures

To assess SimBa’s scalability, we compared it to BroNet (Nauman et al., ), SpectralNet (Bjorck et al., ), and MLP.
Detailed descriptions of each architecture are provided in Appendix B.

The key differences between SimBa and other architectures lie in the placement of components that promote simplicity bias.
SimBa maintains a direct linear residual pathway from input to output, applying non-linearity exclusively through residual connections. In contrast, other architectures introduce non-linearities within the input-to-output pathway, increasing functional complexity. Additionally, SimBa employs post-layer normalization to ensure consistent activation scales across all hidden dimensions, reducing variance in policy and value predictions. Conversely, other architectures omit normalization before the output layer, which can lead to high variance in their predictions.

To ensure that performance differences were attributable to architectural design choices rather than varying input normalization strategies, we uniformly applied RSNorm as the input normalization method across all models. Additionally, our investigation primarily focused on scaling the critic network’s hidden dimension, as scaling the actor network showed limited benefits (see Section7.2).

As illustrated in Figure 5.(a), SimBa achieves the highest simplicity bias score compared to the experimented architectures, demonstrating its strong preference for simpler solutions.
In Figure 5.(b), while the MLP failed to scale with increasing network size, BroNet, SpectralNet, and SimBa all showed performance improvements as their networks scaled up.

Importantly, the scalability of each architecture was strongly correlated with its simplicity bias score, where a higher simplicity bias led to better scalability. SimBa demonstrated the best scalability, supporting the hypothesis that simplicity bias is a key factor in scaling deep RL models.

Figure: Figure 5: Architecture Comparison. (a) SimBa consistently exhibits a higher simplicity bias score. (b) SimBa demonstrates superior scaling performance in terms of average return for DMC-Hard compared to the other architectures. The results are from 5 random seeds.

## 6 Experiments

This section evaluates SimBa’s applicability across various deep RL algorithms and environments. For each baseline, we either use the authors’ reported results or run experiments using their recommended hyperparameters.
Detailed descriptions of the environments used in our evaluations are provided in Appendix F. In addition, we also include learning curves and final performance for each task in Appendix H and I, along with hyperparameters used in our experiments in Appendix G.

Figure: Figure 6: Environment Visualizations. SimBa is evaluated across four diverse benchmark environments: DMC, MyoSuite, and HumanoidBench, which feature complex locomotion and manipulation tasks, and Craftax, which introduces open-ended tasks with varying complexity.
Refer to caption: x6.png

### 6.1 Off-policy RL

Experimental Setup.
We evaluate the algorithms on 51 tasks across three benchmarks: DMC (Tassa et al., ), MyoSuite (Caggiano et al., ), and HumanoidBench (Sferrazza et al., ). The DMC tasks are further categorized into DMC-Easy&Medium and DMC-Hard based on complexity. We vary the number of training steps per benchmark according to the complexity of the environment: 500K for DMC-Easy&Medium, 1M for DMC-Hard and MyoSuite, and 2M for HumanoidBench.

Baselines. We compare SimBa against state-of-the-art off-policy RL algorithms. (i) SAC (Haarnoja et al., ), an actor-critic algorithm based on maximum entropy RL; (ii) DDPG (Lillicrap, ), a deterministic policy gradient algorithm with deep neural networks; (iii) TD7 (Fujimoto et al., ), an enhanced version of TD3 incorporating state-action representation learning; (iv) BRO (Nauman et al., ), which scales the critic network of SAC while integrating distributional Q-learning, optimistic exploration, and periodic resets.
For a fair comparison, we use BRO-Fast, which is the most computationally efficient version;
(v) TD-MPC2 (Hansen et al., ), which combines trajectory planning with long-return estimates using a learned world model; and (vi) DreamerV3 (Hafner et al., ), which learns a generative world model and optimizes a policy via simulated rollouts.

Integrating SimBa into Off-Policy RL. The SimBa architecture is algorithm-agnostic and can be applied to any deep RL method. To demonstrate its applicability, we integrate SimBa into two model-free (SAC, DDPG) and one model-based algorithm (TD-MPC2), evaluating their performance on the DMC-Hard benchmark. For SAC and DDPG, we replace the standard MLP-based actor-critic networks with SimBa. For TD-MPC2, we substitute the shared encoder from MLP to SimBa while matching the number of parameters as the original implementation.

Figure 7 shows consistent benefits across all algorithms: integrating SimBa increased the average return by 570, 480, and, 170 points for SAC, DDPG, and TD-MPC2 respectively. These results demonstrate that scaling up parameters with a strong simplicity bias significantly enhances performance in deep RL.

Figure: Figure 7: Off-Policy RL with SimBa. Replacing MLP with SimBa leads to substantial performance improvements across various off-policy RL methods. Mean and 95% CI are averaged over 10 seeds for SAC and DDPG, and 3 seeds for TD-MPC2 in DMC-Hard.

Figure: Figure 8: Off-policy RL Benchmark. Average episode return for DMC and HumanoidBench and average success rate for MyoSuite across 51 continuous control tasks. SimBa (with SAC) achieves high computational efficiency by only changing the network architecture.
Refer to caption: x8.png

Comparisons with State-of-the-Art Methods.
Here, we compare SAC + SimBa against state-of-the-art off-policy deep RL algorithms, which demonstrated the most promising results in previous experiments. Throughout this section, we refer to SAC + SimBa as SimBa.

Figure 8 presents the results, where the x-axis represents computation time using an RTX 3070 GPU, and the y-axis denotes performance. Points in the upper-left corner ( ) indicate higher compute efficiency.
Here, SimBa consistently outperforms most baselines across various environments while requiring less computational time. The only exception is TD-MPC2 on HumanoidBench; however, TD-MPC2 requires 2.5 times the computational resources of SimBa to achieve better performance, making SimBa the most efficient choice overall.

The key takeaway is that SimBa achieves these remarkable results without the bells and whistles often found in state-of-the-art deep RL algorithms. It is easy to implement, which only requires modifying the network architecture into SimBa without additional changes to the loss functions (Schwarzer et al., ; Hafner et al., ) or using domain-specific regularizers (Fujimoto et al., ; Nauman et al., ).
Its effectiveness stems from an architectural design that leverages simplicity bias and scaling up the number of parameters, thereby maximizing performance.

Impact of Input Dimension. To further identify in which cases SimBa offers significant benefits, we analyze its performance across DMC environments with varying input dimensions.
As shown in Figure 9, the advantage of SimBa becomes more pronounced as input dimensionality increases. We hypothesize that higher-dimensional inputs exacerbate the curse of dimensionality, and the simplicity bias introduced by SimBa effectively mitigates overfitting in these high-dimensional settings.

### 6.2 On-policy RL

Experimental Setup.
We conduct our on-policy RL experiments in Craftax (Matthews et al., ), an open-ended environment inspired by Crafter (Hafner, ) and NetHack (Küttler et al., ). Craftax poses a unique challenge with its open-ended structure and compositional tasks. Following Matthews et al. (), we use Proximal Policy Optimization (Schulman et al., , PPO) as the baseline algorithm. When integrating SimBa with PPO, we replace the standard MLP-based actor-critic networks with SimBa and train both PPO and PPO + SimBa on 1024 parallel environments for a total of 1 billion environment steps.

Results. As illustrated in Figure 10, integrating SimBa into PPO significantly enhances performance across multiple complex tasks. With SimBa, the agent learns to craft iron swords and pickaxes more rapidly, enabling the early acquisition of advanced tools. Notably, by using these advanced tools, the SimBa-based agent successfully defeats challenging adversaries like the Orc Mage using significantly fewer time steps than an MLP-based agent. These improvements arise solely from the architectural change, demonstrating the effectiveness of the SimBa approach for on-policy RL.

Figure: Figure 9: Impact of Input Dimension. Average episode return for DMC tasks plotted against increasing state dimensions. Results show that the benefits of using SimBa increase with higher input dimensions, effectively alleviating the curse of dimensionality.
Refer to caption: x9.png

Figure: Figure 10: On-policy RL with SimBa. Average return of achievements and task success rate for three different tasks comparing PPO + SimBa and PPO on Craftax. Integrating SimBa enables effective learning of complex behaviors.
Refer to caption: extracted/5922698/figures/sec5_ppo_sample_curve.png

### 6.3 Unsupervised RL

Experimental Setup.
In our unsupervised RL study, we incorporate SimBa for online skill discovery, aiming to identify diverse behaviors without relying on task-specific rewards. We focus our experiments primarily on METRA (Park et al., ), which serves as the state-of-the-art algorithm in this domain. We evaluate METRA and METRA with SimBa on the Humanoid task from DMC, running 10M environment steps.
For a quantitative comparison, we adopt state coverage as our main metric. Coverage is measured by discretizing the $x$ and $y$ axes into a grid and counting the number of grid cells covered by the learned behaviors at each evaluation epoch, following prior literature (Park et al., ; Kim et al., ; ).

Results. As illustrated in Figure 1.(a) and  11, integrating SimBa into METRA significantly enhances state coverage on the Humanoid task. The high-dimensional input space makes it challenging for METRA to learn diverse skills. By injecting simplicity bias to manage high input dimensions and scaling up parameters to facilitate effective diverse skill acquisition, SimBa effectively leads to improved exploration and broader state coverage.

Figure: Figure 11: URL with Simba. Integrating SimBa to METRA enhances state coverage.

## 7 Ablations

We conducted ablations on DMC-Hard with SAC + SimBa, running 5 seeds for each experiment.

### 7.1 Observation Normalization

A key factor in SimBa’s success is using RSNorm for observation normalization.
To validate its effectiveness, we compare 5 alternative methods: (i) Layer Normalization (Ba et al., ); (ii) Batch Normalization (Ioffe & Szegedy, ); (iii) Env Wrapper RSNorm, which tracks running statistics for each dimension, normalizes observations upon receiving from the environment, and stores them in the replay buffer;
(iv) Initial N Steps: fixed statistics derived from initially collected transition samples, where we used $N=5,000$; and (v) Oracle Statistics, which rely on pre-computed statistics from previously collected expert data.

As illustrated in Figure 12, layer normalization and batch normalization offer little to no performance gains.
While the env-wrapper RSNorm is somewhat effective, it falls short of RSNorm’s performance. Although widely used in deep RL frameworks (Dhariwal et al., ; Hoffman et al., ; Raffin et al., ), the env-wrapper introduces inconsistencies in off-policy settings by normalizing samples with different statistics based on their collection time. This causes identical observations to be stored with varying values in the replay buffer, reducing learning consistency.
Fixed initial statistics also show slightly worse performance than RSNorm, potentially due to their inability to adapt to the evolving dynamics during training.
Overall, only RSNorm matched the performance of oracle statistics, making it the most practical observation normalization choice for deep RL.

Figure: Figure 12: Obs Normalization. RSNorm consistently outperforms alternative normalization methods. Mean and 95% CI over 5 seeds.

### 7.2 Scaling the Number of Parameters

Here, we investigate scaling parameters in the actor-critic architecture with SimBa by focusing on two aspects: (i) scaling the actor versus the critic network, and (ii) scaling network width versus depth. For width scaling, the actor and critic depths are set to 1 and 2 blocks, respectively. For depth scaling, the actor and critic widths are fixed at 128 and 512, respectively, following our default setup.

As illustrated in Figure 13, scaling up the width or depth of the critic ( ) generally improves performance, while scaling up the actor’s width or depth ( ) tends to reduce performance. This contrast suggests that the target complexity of the actor may be lower than that of the critic, where scaling up the actor’s parameters might be ineffective. These findings align with previous study (Nauman et al., ), which highlights the benefits of scaling up the critic while showing limited advantages in scaling up the actor.

Figure: Figure 13: Scaling Actor and Critic Network. Performance of SAC with SimBa by varying width and depth for the actor and critic network.

Furthermore, for the critic network, scaling up the width is generally more effective than scaling up the depth. While both approaches can enhance the network’s expressivity, scaling up the depth can decrease the simplicity bias as it adds more non-linear components within the network.
Based on our findings, we recommend width scaling as the default strategy.

### 7.3 Scaling Replay Ratio

According to scaling laws (Kaplan et al., ), performance can be enhanced not only by scaling up the number of parameters but also by scaling up the computation time, which can be done by scaling up the number of gradient updates per collected sample (i.e., replay ratio) in deep RL.

However, in deep RL, scaling up the replay ratio has been shown to decrease performance due to the risk of overfitting the network to the initially collected samples (Nikishin et al., ; D’Oro et al., ; Lee et al., ). To address this issue, recent research has proposed periodically reinitializing the network to prevent overfitting, demonstrating that performance can be scaled with respect to the replay ratio. In this section, we aim to assess whether the simplicity bias induced by SimBa can mitigate overfitting under increased computation time (i.e., higher replay ratios).

To evaluate this, we trained SAC with SimBA using 2, 4, 8, and 16 replay ratios, both with and without periodic resets. For SAC with resets, we reinitialized the entire network and optimizer every 500,000 gradient steps. We also compared our results with BRO (Nauman et al., ), which incorporates resets.

Surprisingly, as illustrated in Figure 14, SimBa’s performance consistently improves as the replay ratio increases, even without periodic resets. We have excluded results for BRO without resets, as it fails to learn meaningful behavior for all replay ratios (achieves lower than 300 points). Notably, when resets are included for SimBa, the performance gains become even more pronounced, with a replay ratio of 8 outperforming the most computationally intensive BRO algorithm (RR10).

Figure: Figure 14: Scaling Replay Ratio. Performance of SimBa with and without periodic resets for various replay ratios.

## 8 Lessons and Opportunities

Lessons.
Historically, deep RL has suffered from overfitting, necessitating sophisticated training protocols and implementation tricks to mitigate these issues (Hessel et al., ; Fujimoto et al., ).
These complexities serve as significant barriers for practitioners with limited resources, hindering the effective usage of deep RL methods.
In this paper, we improved performance by solely modifying the network architecture while keeping the underlying algorithms unchanged.
This approach simplifies the implementation of SimBa, making it easy to adopt.
By incorporating simplicity bias through architectural design and scaling up the number of parameters, the network converges to simpler, generalizable functions, matching or surpassing state-of-the-art deep RL methods.
We also believe that SimBa follows the insights from Richard Sutton’s Bitter Lesson (Sutton, ): although task-specific designs can yield immediate improvements, an approach that scales effectively with increased computation may provide more sustainable benefits over the long term.

Opportunities.
Our exploration of simplicity bias has primarily concentrated on the network architecture; however, optimization techniques are equally vital.
Strategies such as dropout (Hiraoka et al., ), data augmentation (Kostrikov et al., ), and diverse optimization algorithms (Foret et al., ) can further enhance convergence to simpler functions during training. Integrating these techniques with continued parameter scaling presents a promising avenue for future research in deep RL. Furthermore, while our focus has been on the basic model-free algorithm SAC, there exists considerable empirical success with model-based algorithms (Schwarzer et al., ; Hansen et al., ; Hafner et al., ). Preliminary investigations into applying SimBa to model-based RL, specifically TD-MPC2 (Hansen et al., ), have shown promise.
We encourage the research community to collaboratively pursue these avenues to advance deep RL architectures and facilitate their successful application in real-world scenarios.

## Reproducibility

To ensure the reproducibility of our experiments, we provide the complete source code to run SAC with SimBa. Moreover, we have included a Dockerfile to simplify the testing and replication of our results.
The raw scores of each experiment including all baselines are also included.
The code is available at [https://github.com/SonyResearch/simba](https://github.com/SonyResearch/simba).

## Acknowledgement

We would like to express our gratitude to Craig Sherstan, Bram Grooten, Hanseul Cho, Kyungmin Lee, and Hawon Jeong for their helpful discussions and insightful feedback on this paper.

## Appendix

## Appendix A Definition of Simplicity Bias

This section provides more in-depth definition of simplicity bias for clarity.

### A.1 Complexity Measure

Let $\mathcal{X}\subseteq\mathbb{R}^{n}$ denote the input space and $\mathcal{Y}\subseteq\mathbb{R}^{m}$ the output space. Consider the function space $\mathcal{F}=\{f|f:\mathcal{X}\rightarrow\mathcal{Y}\}$. Given a training dataset $D=\{(x_{i},y_{i})\}_{i=1}^{N}$ and a tolerance $\epsilon>0$, define the set of functions $\mathcal{I}\subseteq\mathcal{F}$ achieving a training loss below $\epsilon$:

$$ $\mathcal{I}=\left\{f\in\mathcal{F}\,\middle|\,\mathcal{L}_{\text{train}}(f)= \frac{1}{N}\sum_{i=1}^{N}\ell(f(x_{i}),y_{i})<\epsilon\right\},$ (8) $$

where $\ell:\mathcal{Y}\times\mathcal{Y}\to[0,\infty)$ is a loss function such as mean squared error or cross-entropy.

To quantify the complexity of functions within $\mathcal{F}$, we adopt a Fourier-based complexity measure as described in Section 2. Specifically, for a function $f$ with a Fourier series representation:

$$ $f(x)=(2\pi)^{d/2}\int\tilde{f}(k)\,e^{ik\cdot x}\,dk,$ (9) $$

where $\tilde{f}(k)=\int f(x)\,e^{-ik\cdot x}\,dx$ is the Fourier transform of $f$.

Given a Fourier series representation $\tilde{f}$, we perform a discrete Fourier transform by uniformly discretizing the frequency domain. Specifically, we consider frequencies $k\in\{0,1,\dots,K\}$, where $K$ is selected based on the Nyquist-Shannon sampling theorem (Shannon, ) to ensure an accurate representation of $f$. This discretization transforms the continuous frequency domain into a finite set of frequencies suitable for computational purposes.

Then, the complexity measure $c:\mathcal{F}\to[0,\infty)$ is defined as the weighted average of the Fourier coefficients from discretization:

$$ $c(f)=\frac{\sum_{k=0}^{K}\tilde{f}(k)\cdot k}{\sum_{k=0}^{K}\tilde{f}(k)}.$ (10) $$

Equation 10 captures the intuition that higher complexity arises from the dominance of high-frequency components, which correspond to rapid amplitude changes or intricate details in the function $f$. Conversely, a lower complexity measure indicates a greater contribution from low-frequency components, reflecting simpler, smoother functions.

### A.2 Simplicity Bias Score

Let $f$ be a neural network, and let $\text{alg}\in\mathcal{A}$ denote the optimization algorithm used for training, such as stochastic gradient descent. We quantify the simplicity bias score $s:\mathcal{F}\times\mathcal{A}\rightarrow(0,\infty)$ as:

$$ $s(f,\text{alg})=\mathbb{E}_{f^{*}\sim P_{f,\text{alg}}}\left[\frac{1}{c(f^{*}) }\right]$ (11) $$

where $P_{f,\text{alg}}$ denotes the distribution over $\mathcal{I}$ induced by training $f$ with alg. A higher simplicity bias score indicates a stronger tendency to converge toward functions with lower complexity.

Then, from equation 11, we can compute the simplicity bias score with respect to the architecture by marginalizing over the distribution of the algorithm as:

$$ $s(f)=\mathbb{E}_{\text{alg}\sim\mathbf{A}}\left[\mathbb{E}_{f^{*}\sim P_{f, \text{alg}}}\left[\frac{1}{c(f^{*})}\right]\right].$ (12) $$

Directly measuring this bias by analyzing the complexity of the converged function is challenging due to the non-stationary nature of training dynamics, especially in reinforcement learning where data distributions evolve over time. Empirical studies (Valle-Perez et al., ; De Palma et al., ; Mingard et al., ; Teney et al., ) suggest that the initial complexity of a network is strongly correlated with the complexity of the functions it learns after training.

Therefore, as a practical proxy for simplicity bias, we define the approximate of the simplicity bias score $s(f)$ for a network architecture $f$ with an initial parameter distribution $\Theta_{0}$ as:

$$ $s(f)\approx\mathbb{E}_{\theta\sim\Theta_{0}}\left[\frac{1}{c(f_{\theta})} \right].$ (13) $$

where $f_{\theta}$ denotes the network $f$ parameterized by $\theta$.

A higher simplicity bias score indicates that, on average, the network initialized from $\Theta_{0}$ represents simpler functions prior to training, suggesting a predisposition to converge to simpler solutions during optimization. This measure aligns with the notion that networks with lower initial complexity are more likely to exhibit a higher simplicity bias.

## Appendix B Measuring Simplicity Bias

To quantify the simplicity bias defined in Equation 13, we adopt a methodology inspired by the Neural Redshift framework (Teney et al., ). We utilize the original codebase provided by the authors(^1^11[https://github.com/ArmandNM/neural-redshift](https://github.com/ArmandNM/neural-redshift)) to assess the complexity of random neural networks. This approach evaluates the inherent complexity of the architecture before any optimization, thereby isolating the architectural effects from those introduced by training algorithms such as stochastic gradient descent.

Following the methodology outlined in Teney et al. (), we perform the following steps to measure the simplicity bias score of a neural network $f_{\theta}(\cdot)$:

Sampling Initialization. For each network architecture $f$, we generate $N$ random initializations $\theta\sim\Theta_{0}$. This ensemble of initial parameters captures the variability in complexity introduced by different random seeds. In this work, we used $N=100$.

Sampling Input. We define the input space $\mathcal{X}=[-100,100]^{2}\subset\mathbb{R}^{2}$ and uniformly divide it into a grid of 90,000 points, achieving 300 divisions along each dimension. This dense sampling ensures comprehensive coverage of the input domain.

Evaluating Function. For each sampled input point $x\in\mathcal{X}$, we compute the corresponding output $f_{\theta}(x)$. The collection of these output values forms a 2-D grid of scalars, effectively representing the network’s function as a grayscale image.

Discrete Fourier Transform. We apply a discrete Fourier transform (DFT) to the grayscale image obtained from the function evaluations. This transformation decomposes $f_{\theta}(\cdot)$ into a sum of basis functions of varying frequencies.

Measuring Function Complexity. Utilizing the Fourier coefficients obtained from the DFT, we compute the complexity measure $c(f_{\theta})$ as defined in Equation 10. Specifically, we calculate the frequency-weighted average of the Fourier coefficients:

$$ $c(f_{\theta})=\frac{\sum_{k=0}^{K}\tilde{f}_{\theta}(k)\cdot k}{\sum_{k=0}^{K} \tilde{f}_{\theta}(k)},$ (14) $$

where $\tilde{f}_{\theta}(k)$ denotes the Fourier coefficient at frequency $k$.

Estimating Simplicity Bias Score. For each network $f$, we estimate the simplicity bias score $s(f)$ by averaging the inverse of the complexity measures over all $N$ random initializations:

$$ $s(f)\approx\frac{1}{N}\sum_{i=1}^{N}\frac{1}{c(f_{\theta_{i}})},$ (15) $$

where $\theta_{i}$ represents the $i$-th random initialization.

## Appendix C Plasticity Analysis

Recent studies have identified the loss of plasticity as a significant challenge in non-stationary training scenarios, where neural networks gradually lose their ability to adapt to new data over time (Nikishin et al., ; Lyle et al., ; Lee et al., ). This phenomenon can severely impact the performance and adaptability of models in dynamic learning environments such as RL. To quantify and understand this issue, several metrics have been proposed, including stable-rank (Kumar et al., ), dormant ratio (Sokar et al., ), and L2-feature norm (Kumar et al., ).

The stable-rank assesses the rank of the feature matrix, reflecting the diversity and richness of the representations learned by the network. This is achieved by performing an eigen decomposition on the covariance matrix of the feature matrix and counting the number of singular values $\sigma_{j}$ that exceed a predefined threshold $\tau$:

$$ $\text{s-Rank}=\sum_{j=1}^{m}\mathbb{I}(\sigma_{j}>\tau)$ (16) $$

where $F\in\mathbb{R}^{d\times m}$ is the feature matrix with $d$ samples and $m$ features, $\sigma_{j}$ are the singular values, and $\mathbb{I}(\cdot)$ is the indicator function.

The dormant ratio measures the proportion of neurons with negligible activation. It is calculated as the number of neurons with activation norms below a small threshold $\epsilon$ divided by the total number of neurons $D$:

$$ $\text{Dormant Ratio}=\frac{|\{i\mid\|a_{i}\|<\epsilon\}|}{D}$ (17) $$

where $a_{i}$ represents the activation of neuron $i$.

The L2-feature norm represents the average L2 norm of the feature vectors across all samples:

$$ $\text{Feature Norm}=\frac{1}{N}\sum_{i=1}^{N}\|F_{i}\|_{2}$ (18) $$

where $F_{i}$ is the feature vector for the $i$-th sample. Large feature norms can signal overactive neurons, potentially leading to numerical instability.

To assess how different architectures impact plasticity, we conducted experiments in DMC-Hard. We compared Soft Actor-Critic (SAC) implemented with a standard MLP architecture against SAC with the SimBa architecture. Each configuration was evaluated across 5 random seeds.

Figure: Figure 15: Plasticity Metrics Comparison. Average episode return for the DMC-Hard environment. Metrics compared include dormant ratio, s-rank, and feature norm. Higher dormant ratio and feature norm, along with lower s-rank, indicate a greater loss of plasticity.
Refer to caption: extracted/5922698/figures/appendix_aggregate_dmchard_various_metrics.png

Figure 15 shows that SAC with the MLP architecture exhibits a high dormant ratio, low s-rank, and large feature norms. These metrics indicate a significant loss of plasticity.
In contrast, the SimBa architecture maintains lower dormant ratios, higher s-rank values, and balanced feature norms. This ensures more active neurons, diverse and rich feature representations, and stable activations.

These results demonstrate that SimBa effectively preserves plasticity, avoiding the degenerative trends seen with the MLP-based SAC. It will be interesting for future work to understand how simplicity bias and architectural choices influence network plasticity.

## Appendix D Comparison to Existing Architectures

Here we provide a more in-depth discussion related to the architectural difference between SimBa, BroNet, and SpectralNet. As shown in Figure 16, SimBa differs from BroNet in three key aspects: (i) the inclusion of RSNorm, (ii) the implementation of pre layer-normalization, (iii) the utilization of a linear residual pathway, (iv) and the inclusion of a post-layer normalization layer. Similarly, compared to SpectralNet, SimBa incorporates RSNorm, employs a linear residual pathway, and leverages spectral normalization differently.

Figure: Figure 16: Architecture Comparison. Illustration of SimBa, BroNet, and SpectralNet.
Refer to caption: x14.png

## Appendix E Computational Resources

The training was performed using NVIDIA RTX 3070, A100, or H100 GPUs for neural network computations and either a 16-core Intel i7-11800H or a 32-core AMD EPYC 7502 CPU for running simulators. Our software environment included Python 3.10, CUDA 12.2, and Jax 4.26.

When benchmarking computation time, experiments were conducted on a same hardware equipped with an NVIDIA RTX 3070 GPU and 16-core Intel i7-11800H CPU.

**Table 1: Environment details. We list the episode length, action repeat for each domain, total environment steps, and performance metrics used for benchmarking SimBa.**
|  | DMC | MyoSuite | HumanoidBench | Craftax |
| --- | --- | --- | --- | --- |
| Episode length | $1,000$ | $100$ | $500$ - $1,000$ | $-$ |
| Action repeat | $2$ | $2$ | $2$ | $1$ |
| Effective length | $500$ | $50$ | $250$ - $500$ | $-$ |
| Total env. steps | $500$K-$1$M | $1$M | $2$M | $1$B |
| Performance metric | Average Return | Average Success | Average Return | Average Score |

## Appendix F Environment Details

This section details the benchmark environments used in our evaluation. We list all tasks from each benchmark along with their observation and action dimensions. Visualizations of each environment are provided in Figure 6, and detailed environment information is available in Table 1.

### F.1 DeepMind Control Suite

DeepMind Control Suite (Tassa et al., , DMC) is a standard benchmark for continuous control, featuring a range of locomotion and manipulation tasks with varying complexities, from simple low-dimensional tasks ($s\in\mathbb{R}^{3}$) to highly complex ones ($s\in\mathbb{R}^{223}$). We evaluate 27 DMC tasks, categorized into two groups: DMC-Easy&Medium and DMC-Hard. All Humanoid and Dog tasks are classified as DMC-Hard, while the remaining tasks are grouped under DMC-Easy&Medium. The complete lists of DMC-Easy&Medium and DMC-Hard are provided in Tables 2 and 3, respectively.

For our URL experiments, we adhere to the protocol in Park et al. (), using an episode length of 400 steps and augmenting the agent’s observations with its $x$, $y$, and $z$ coordinates.
For the representation function $\phi$ (i.e., reward network) in METRA, we only used $x$, $y$, and $z$ positions as input. This approach effectively replaces the colored floors used in the pixel-based humanoid setting of METRA with state-based inputs.

### F.2 MyoSuite

MyoSuite (Caggiano et al., ) simulates musculoskeletal movements with high-dimensional state and action spaces, focusing on physiologically accurate motor control. It includes benchmarks for complex object manipulation using a dexterous hand. We evaluate 10 MyoSuite tasks, categorized as ”easy” when the goal is fixed and ”hard” when the goal is randomized, following Hansen et al. (). The full list of MyoSuite tasks is presented in Table 4.

### F.3 HumanoidBench

HumanoidBench (Sferrazza et al., ) is a high-dimensional simulation benchmark designed to advance humanoid robotics research. It features the Unitree H1 humanoid robot with dexterous hands, performing a variety of challenging locomotion and manipulation tasks. These tasks range from basic locomotion to complex activities requiring precise manipulation. The benchmark facilitates algorithm development and testing without the need for costly hardware by providing a simulated platform. In our experiments, we focus on 14 locomotion tasks, simplifying the setup by excluding the dexterous hands. This reduces the complexity associated with high degrees of freedom and intricate dynamics. All HumanoidBench scores are normalized based on each task’s target success score as provided by the authors. A complete list of tasks is available in Table 5.

### F.4 Craftax

Craftax (Matthews et al., ) is an open-ended RL environment that combines elements from Crafter (Hafner, ) and NetHack (Küttler et al., ). It presents a challenging scenario requiring the sequential completion of numerous tasks. A key feature of Craftax is its support for vectorized parallel environments and a full end-to-end GPU learning pipeline, enabling a large number of environment steps at low computational cost. The original baseline performance in Craftax is reported as a percentage of the maximum score (226). In our experiments, we report agents’ raw average scores instead.

**Table 2: DMC Easy & Medium. We consider a total of $20$ continuous control tasks for the DMC Easy & Medium benchmark. We list all considered tasks below and baseline performance for each task is reported at $500$K environment steps.**
| Task | Observation dim | Action dim |
| --- | --- | --- |
| Acrobot Swingup | $6$ | $1$ |
| Cartpole Balance | $5$ | $1$ |
| Cartpole Balance Sparse | $5$ | $1$ |
| Cartpole Swingup | $5$ | $1$ |
| Cartpole Swingup Sparse | $5$ | $1$ |
| Cheetah Run | $17$ | $6$ |
| Finger Spin | $9$ | $2$ |
| Finger Turn Easy | $12$ | $2$ |
| Finger Turn Hard | $12$ | $2$ |
| Fish Swim | $24$ | $5$ |
| Hopper Hop | $15$ | $4$ |
| Hopper Stand | $15$ | $4$ |
| Pendulum Swingup | $3$ | $1$ |
| Quadruped Run | $78$ | $12$ |
| Quadruped Walk | $78$ | $12$ |
| Reacher Easy | $6$ | $2$ |
| Reacher Hard | $6$ | $2$ |
| Walker Run | $24$ | $6$ |
| Walker Stand | $24$ | $6$ |
| Walker Walk | $24$ | $6$ |

**Table 3: DMC Hard. We consider a total of $7$ continuous control tasks for the DMC Hard benchmark.We list all considered tasks below and baseline performance for each task is reported at $1$M environment steps.**
| Task | Observation dim | Action dim |
| --- | --- | --- |
| Dog Run | $223$ | $38$ |
| Dog Trot | $223$ | $38$ |
| Dog Stand | $223$ | $38$ |
| Dog Walk | $223$ | $38$ |
| Humanoid Run | $67$ | $24$ |
| Humanoid Stand | $67$ | $24$ |
| Humanoid Walk | $67$ | $24$ |

**Table 4: MyoSuite. We consider a total of $10$ continuous control tasks for the MyoSuite benchmark, which encompasses both fixed-goal (’Easy’) and randomized-goal (’Hard’) settings. We list all considered tasks below and baseline performance for each task is reported at $1$M environment steps.**
| Task | Observation dim | Action dim |
| --- | --- | --- |
| Key Turn Easy | $93$ | $39$ |
| Key Turn Hard | $93$ | $39$ |
| Object Hold Easy | $91$ | $39$ |
| Object Hold Hard | $91$ | $39$ |
| Pen Twirl Easy | $83$ | $39$ |
| Pen Twirl Hard | $83$ | $39$ |
| Pose Easy | $108$ | $39$ |
| Pose Hard | $108$ | $39$ |
| Reach Easy | $115$ | $39$ |
| Reach Hard | $115$ | $39$ |

**Table 5: HumanoidBench. We consider a total of $14$ continuous control locomotion tasks for the UniTree H1 humanoid robot from the HumanoidBench domain. We list all considered tasks below and baseline performance for each task is reported at $2$M environment steps.**
| Task | Observation dim | Action dim |
| --- | --- | --- |
| Balance Hard | $77$ | $19$ |
| Balance Simple | $64$ | $19$ |
| Crawl | $51$ | $19$ |
| Hurdle | $51$ | $19$ |
| Maze | $51$ | $19$ |
| Pole | $51$ | $19$ |
| Reach | $57$ | $19$ |
| Run | $51$ | $19$ |
| Sit Simple | $51$ | $19$ |
| Sit Hard | $64$ | $19$ |
| Slide | $51$ | $19$ |
| Stair | $51$ | $19$ |
| Stand | $51$ | $19$ |
| Walk | $51$ | $19$ |

**Table 6: Craftax. We consider the symbolic version of Craftax. Baseline performance is reported at $1$B environment steps.**
| Task | Observation dim | Action dim |
| --- | --- | --- |
| Craftax-Symbolic-v1 | $8268$ | $43$ |

## Appendix G Hyperparameters

### G.1 Off-Policy

**Table 7: SAC+SimBa hyperparameters. The hyperparameters listed below are used consistently across all tasks when integrating SimBa with SAC. For the discount factor, we set it automatically using heuristics used by TD-MPC2 (Hansen et al., ).**
| Hyperparameter | Value |
| --- | --- |
| Critic block type | SimBa Residual |
| Critic num blocks | 2 |
| Critic hidden dim | 512 |
| Critic learning rate | 1e-4 |
| Target critic momentum ($\tau$) | 5e-3 |
| Actor block type | SimBa Residual |
| Actor num blocks | 1 |
| Actor hidden dim | 128 |
| Actor learning rate | 1e-4 |
| Initial temperature ($\alpha_{0}$) | 1e-2 |
| Temperature learning rate | 1e-4 |
| Target entropy ($\mathcal{H}^{*}$) | $|\mathcal{A}|/2$ |
| Batch size | 256 |
| Optimizer | AdamW |
| Optimizer momentum ($\beta_{1}$, $\beta_{2}$) | (0.9, 0.999) |
| Weight decay ($\lambda$) | 1e-2 |
| Discount ($\gamma$) | Heuristic |
| Replay ratio | 2 |
| Clipped Double Q | HumanoidBench: True |
|  | Other Envs: False |

**Table 8: DDPG+SimBa hyperparameters. The hyperparameters listed below are used consistently across all tasks when integrating SimBa with DDPG. For the discount factor, we set it automatically using heuristics used by TD-MPC2 (Hansen et al., ).**
| Hyperparameter | Value |
| --- | --- |
| Critic block type | SimBa Residual |
| Critic num blocks | 2 |
| Critic hidden dim | 512 |
| Critic learning rate | 1e-4 |
| Target critic momentum ($\tau$) | 5e-3 |
| Actor block type | SimBa Residual |
| Actor num blocks | 1 |
| Actor hidden dim | 128 |
| Actor learning rate | 1e-4 |
| Exploration noise | $\mathcal{N}(0,{0.1}^{2})$ |
| Batch size | 256 |
| Optimizer | AdamW |
| Optimizer momentum ($\beta_{1}$, $\beta_{2}$) | (0.9, 0.999) |
| Weight decay ($\lambda$) | 1e-2 |
| Discount ($\gamma$) | Heuristic |
| Replay ratio | 2 |

**Table 9: TDMPC2+SimBa hyperparameters. We provide a detailed list of the hyperparameters used for the shared encoder module in TD-MPC2 (Hansen et al., ). Aside from these, we follow the hyperparameters specified in the original TD-MPC2 paper. The listed hyperparameters are applied uniformly across all tasks when integrating SimBa with TD-MPC2.**
| Hyperparameter | Value |
| --- | --- |
| Encoder block type | SimBa Residual |
| Encoder num blocks | 2 |
| Encoder hidden dim | 256 |
| Encoder learning rate | 1e-4 |
| Encoder weight decay | 1e-1 |

### G.2 On-Policy

**Table 10: PPO+SimBa hyperparameters. We provide a detailed list of the hyperparameters when integrating SimBa with PPO (Schulman et al., ). Aside from these, we follow the hyperparameters specified in the Craftax paper (Matthews et al., ).**
| Hyperparameter | Value |
| --- | --- |
| Critic block type | SimBa Residual |
| Critic num blocks | 2 |
| Critic hidden dim | 512 |
| Critic learning rate | 1e-4 |
| Actor block type | SimBa Residual |
| Actor num blocks | 1 |
| Actor hidden dim | 256 |
| Actor learning rate | 1e-4 |
| Optimizer | AdamW |
| Optimizer momentum ($\beta_{1},\beta_{2}$) | (0.9, 0.999) |
| Optimizer eps | 1e-8 |
| Weight decay ($\lambda$) | 1e-2 |

### G.3 Unsupervised RL

**Table 11: METRA+SimBa hyperparameters. We provide a detailed list of the hyperparameters when integrating SimBa with METRA (Park et al., ). Aside from these, we follow the hyperparameters specified in the original METRA paper.**
| Hyperparameter | Value |
| --- | --- |
| Critic block type | SimBa Residual |
| Critic num blocks | 2 |
| Critic hidden dim | 512 |
| Actor block type | SimBa Residual |
| Actor num blocks | 1 |
| Actor hidden dim | 128 |

## Appendix H Learning Curve

Figure: Figure 17: Per task learning curve for DMC Easy&Medium. Mean and 95% CI over 10 seeds for SimBa and BRO, 5 seeds for TD7 and SAC, 3 seeds for TD-MPC2 and DreamerV3.
Refer to caption: extracted/5922698/figures/appendix_dmc_easy_medium_per_env_sample_curve.png

Figure: Figure 18: Per task learning curve for DMC Hard. Mean and 95% CI over 10 seeds for SimBa and BRO, 5 seeds for TD7 and SAC, 3 seeds for TD-MPC2 and DreamerV3.
Refer to caption: extracted/5922698/figures/appendix_dmc_hard_per_env_sample_curve.png

Figure: Figure 19: Per task learning curve for MyoSuite. Mean and 95% CI over 10 seeds for SimBa and BRO, 5 seeds for TD7 and SAC, 3 seeds for TD-MPC2 and DreamerV3.
Refer to caption: extracted/5922698/figures/appendix_myo_per_env_sample_curve.png

Figure: Figure 20: Per task learning curve for HumanoidBench. Mean and 95% CI over 10 seeds for SimBa, 5 seeds for BRO, TD7, and SAC, 3 seeds for TD-MPC2 and DreamerV3. We no
Refer to caption: extracted/5922698/figures/appendix_hb_per_env_sample_curve.png

Figure: Figure 21: Per task learning curve for Craftax. We visualize the success rate learning curve for 66 tasks in Craftax. Mean and 95% CI over 5 seeds for PPO + SimBa and PPO.
Refer to caption: extracted/5922698/figures/appendix_Craftax_per_env_sample_curve.png

## Appendix I Full Result

**Table 12: Per task results for DMC Easy&Medium. Results for SimBa and BRO are averaged over 10 seeds, for TD7 and SAC over 5 seeds, and for TD-MPC2 and DreamerV3 over 3 seeds.**
| Method | SimBa | BRO | TD7 | SAC | TD-MPC2 | DreamerV3 |
| --- | --- | --- | --- | --- | --- | --- |
| Acrobot Swingup | 331.57 | 390.78 | 39.83 | 57.56 | 361.07 | 360.46 |
| Cartpole Balance | 999.05 | 998.66 | 998.79 | 998.79 | 993.93 | 994.95 |
| Cartpole Balance Sparse | 940.53 | 954.21 | 988.58 | 1000.00 | 1000.00 | 800.25 |
| Cartpole Swingup | 866.53 | 878.69 | 878.09 | 863.24 | 876.07 | 863.90 |
| Cartpole Swingup Sparse | 823.97 | 833.18 | 480.74 | 779.99 | 844.77 | 262.69 |
| Cheetah Run | 814.97 | 739.67 | 897.76 | 716.43 | 757.60 | 686.25 |
| Finger Spin | 778.83 | 987.45 | 592.74 | 814.69 | 984.63 | 434.06 |
| Finger Turn Easy | 881.33 | 905.85 | 485.66 | 903.07 | 854.67 | 851.11 |
| Finger Turn Hard | 860.22 | 905.30 | 596.32 | 775.21 | 876.27 | 769.86 |
| Fish Swim | 786.73 | 680.34 | 108.84 | 462.67 | 610.23 | 603.34 |
| Hopper Hop | 326.69 | 315.04 | 110.27 | 159.43 | 303.27 | 192.70 |
| Hopper Stand | 811.75 | 910.88 | 445.50 | 845.89 | 936.47 | 722.42 |
| Pendulum Swingup | 824.53 | 816.20 | 461.40 | 476.58 | 841.70 | 825.17 |
| Quadruped Run | 883.68 | 818.62 | 515.13 | 116.91 | 939.63 | 471.25 |
| Quadruped Walk | 952.96 | 936.06 | 910.55 | 147.83 | 957.17 | 472.31 |
| Reacher Easy | 972.23 | 933.77 | 920.38 | 951.80 | 919.43 | 888.36 |
| Reacher Hard | 965.96 | 956.52 | 549.50 | 959.59 | 913.73 | 935.25 |
| Walker Run | 687.16 | 754.43 | 782.32 | 629.44 | 820.40 | 620.13 |
| Walker Stand | 983.02 | 986.58 | 984.63 | 972.59 | 957.17 | 963.28 |
| Walker Walk | 970.73 | 973.41 | 976.58 | 956.67 | 978.70 | 925.46 |
| IQM | 885.70 | 888.02 | 740.19 | 799.75 | 895.47 | 748.58 |
| Median | 816.78 | 836.62 | 623.18 | 676.59 | 836.93 | 684.18 |
| Mean | 823.12 | 833.78 | 636.18 | 679.42 | 836.34 | 682.16 |
| OG | 0.1769 | 0.1662 | 0.3638 | 0.3206 | 0.1637 | 0.3178 |

**Table 13: Per task results for DMC Hard. Results for SimBa and BRO are averaged over 10 seeds, for TD7 and SAC over 5 seeds, and for TD-MPC2 and DreamerV3 over 3 seeds.**
| Method | SimBa | BRO | TD7 | SAC | TD-MPC2 | DreamerV3 |
| --- | --- | --- | --- | --- | --- | --- |
| Dog Run | 544.86 | 374.63 | 127.48 | 36.86 | 169.87 | 15.72 |
| Dog Stand | 960.38 | 966.97 | 753.23 | 102.04 | 798.93 | 55.87 |
| Dog Trot | 824.69 | 783.12 | 126.00 | 29.36 | 500.03 | 10.19 |
| Dog Walk | 916.80 | 931.46 | 280.87 | 38.14 | 493.93 | 23.36 |
| Humanoid Run | 181.57 | 204.96 | 79.32 | 116.97 | 184.57 | 0.91 |
| Humanoid Stand | 846.11 | 920.11 | 389.80 | 352.72 | 663.73 | 5.12 |
| Humanoid Walk | 668.48 | 672.55 | 252.72 | 273.67 | 628.23 | 1.33 |
| IQM | 773.28 | 771.50 | 216.04 | 69.03 | 527.11 | 9.63 |
| Median | 706.39 | 694.20 | 272.62 | 159.36 | 528.26 | 17.13 |
| Mean | 706.13 | 693.40 | 287.06 | 135.68 | 491.33 | 16.07 |
| OG | 0.2939 | 0.3066 | 0.7129 | 0.8643 | 0.5087 | 0.9839 |

**Table 14: Per task results for MyoSuite. Results for SimBa and BRO are averaged over 10 seeds, for TD7 and SAC over 5 seeds, and for TD-MPC2 and DreamerV3 over 3 seeds.**
| Method | SimBa | BRO | TD7 | SAC | TD-MPC2 | DreamerV3 |
| --- | --- | --- | --- | --- | --- | --- |
| Key Turn Easy | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 88.89 |
| Key Turn Hard | 7.00 | 42.00 | 0.00 | 10.00 | 0.00 | 0.00 |
| Object Hold Easy | 90.00 | 90.00 | 20.00 | 90.00 | 100.00 | 33.33 |
| Object Hold Hard | 96.00 | 42.00 | 10.00 | 96.00 | 56.67 | 9.44 |
| Pen Twirl Easy | 80.00 | 90.00 | 100.00 | 50.00 | 70.00 | 96.67 |
| Pen Twirl Hard | 77.00 | 76.00 | 12.00 | 55.00 | 40.00 | 53.33 |
| Pose Easy | 100.00 | 100.00 | 0.00 | 90.00 | 100.00 | 100.00 |
| Pose Hard | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Reach Easy | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| Reach Hard | 93.00 | 74.00 | 14.00 | 16.00 | 83.33 | 0.00 |
| IQM | 95.20 | 87.60 | 22.31 | 71.40 | 77.50 | 46.56 |
| Median | 77.00 | 72.00 | 34.00 | 62.50 | 65.00 | 47.00 |
| Mean | 74.30 | 71.40 | 35.60 | 60.70 | 65.00 | 48.17 |
| OG | 99.93 | 99.93 | 99.96 | 99.94 | 99.94 | 99.95 |

**Table 15: Per task results for HumanoidBench. Results for SimBa are averaged over 10 seeds, for BRO, TD7 and SAC over 5 seeds, and for TD-MPC2 and DreamerV3 over 3 seeds.**
| Method | SimBa | BRO | TD7 | SAC | TD-MPC2 | DreamerV3 |
| --- | --- | --- | --- | --- | --- | --- |
| Balance Hard | 137.20 | 145.95 | 79.90 | 69.02 | 64.56 | 16.07 |
| Balance Simple | 816.38 | 246.57 | 132.82 | 113.38 | 50.69 | 14.09 |
| Crawl | 1370.51 | 1373.83 | 868.63 | 830.56 | 1384.40 | 906.66 |
| Hurdle | 340.60 | 128.60 | 200.28 | 31.89 | 1000.12 | 18.78 |
| Maze | 283.58 | 259.03 | 179.23 | 254.38 | 198.65 | 114.90 |
| Pole | 1036.70 | 915.89 | 830.72 | 760.78 | 1269.57 | 289.18 |
| Reach | 523.10 | 317.99 | 159.37 | 347.92 | 505.61 | 341.62 |
| Run | 741.16 | 429.74 | 196.85 | 168.25 | 1130.94 | 85.93 |
| Sit Hard | 783.95 | 989.07 | 293.96 | 345.65 | 1027.47 | 9.95 |
| Sit Simple | 1059.73 | 1151.84 | 1183.26 | 994.58 | 1074.96 | 40.53 |
| Slide | 577.87 | 653.17 | 197.07 | 208.46 | 780.82 | 24.43 |
| Stair | 527.49 | 249.86 | 77.19 | 65.53 | 398.21 | 49.04 |
| Stand | 906.94 | 780.52 | 1005.54 | 1029.78 | 1020.59 | 280.99 |
| Walk | 1202.91 | 1080.63 | 143.86 | 412.21 | 1165.42 | 125.59 |
| IQM | 747.43 | 558.03 | 256.77 | 311.46 | 885.67 | 72.30 |
| Median | 733.09 | 632.93 | 385.41 | 399.93 | 796.18 | 160.49 |
| Mean | 736.30 | 623.05 | 396.33 | 402.31 | 787.64 | 165.55 |
| OG | 0.3197 | 0.4345 | 0.6196 | 0.6016 | 0.2904 | 0.8352 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. I am curious about the scaling results on multi-task RL policies, since the proposed architecture is added to the representation part. By the way, maybe the author should also compare the results with baseline methods under scaled-up backbones.

2. The author chose a new metric named simplicity bias score for comparison and analysis. The results also show some effects of the initial parameter distributions. Why not compare with the baseline methods with initialization periodically?

Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P. L., & Courville, A. (2022, June). The primacy bias in deep reinforcement learning. In International conference on machine learning (pp. 16828-16847). PMLR.
soundness: 3
strengths: 1. **Innovative Application of Simplicity Bias**: The use of simplicity bias in SimBa to manage overparameterization effectively is a novel contribution to deep RL.

2. **Comprehensive Empirical Validation**: SimBa's performance is rigorously tested across multiple RL tasks, including DMC, MyoSuite, and HumanoidBench, showing consistent improvements in efficiency and scalability.

3. **Adaptability**: SimBa’s architecture is algorithm-agnostic and can integrate seamlessly with various RL algorithms, allowing its broad applicability.

4. **Effective Design Choices**: The architectural components (e.g., RSNorm and residual connections) seem thoughtfully selected to reduce complexity while enabling parameter scaling.
summary: The paper presents SimBa, an architecture specifically designed for deep reinforcement learning that scales up network parameters by integrating a simplicity bias. The SimBa architecture employs observation normalization, residual feedforward blocks, and layer normalization, fostering simplicity in function representation. By adopting these components, SimBa enhances the sample efficiency and compute efficiency of various RL algorithms. It performs comparably or better than state-of-the-art methods in diverse RL benchmarks, demonstrating effectiveness across off-policy, on-policy, and unsupervised settings.
weaknesses: 1. Missing relation works about normalization. Researchers show that the RMS Norm works well in training foundation models. The proposed RSNorm is too similar to the RMS Norm, while the author has no discussion about the RMS Norm (even not in related work). Since this work investigates designing and scaling up networks in deep RL, the same tricks in designing and scaling up LLMs should be considered.

2. Unfair comparison between SAC+SimBa v.s. others. Since SAC+SimBa introduces some other layers, it might include more parameters for the neural networks. The author can make some ablations with different sizes of SAC to exclude the capacity issue.

3. Missing details of the model size. I didn't find the detailed sizes of each model. If the sizes are small (such as 10M), the conclusions about the so-called "scaling up" are doubtful.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: - Applicability: Are there any specific types of RL problems or environments where SimBa might not be as effective?
- Why are the standard errors in Figure 10 so high?
soundness: 4
strengths: - Novel approach: SimBa addresses an important gap in deep RL research by exploring how to scale up network parameters while leveraging simplicity bias effectively.
- Versatility: The architecture improves sample efficiency across various RL algorithms, including off-policy, on-policy, and unsupervised methods.
- Performance: When applied to SAC, SimBa matches or surpasses state-of-the-art off-policy RL methods across a wide range of tasks.
- Computational efficiency: SimBa achieves high performance without relying on computationally intensive components or complex training protocols.
- Theoretical foundation: The authors provide a clear explanation of simplicity bias and how it's measured, grounding their work in existing theory.
- Significance: This work addresses an important gap in deep RL research by exploring how to effectively scale up network parameters while leveraging simplicity bias. The SimBa architecture offers a promising approach to improve performance across various RL algorithms and tasks without relying on computationally intensive components or complex training protocols
summary: The paper introduces SimBa (Simplicity Bias), a novel architecture designed to scale up parameters in deep reinforcement learning (RL) by incorporating simplicity bias. The key components of SimBa are:
- Observation normalization layer
- Residual feedforward block
- Layer normalization

The authors demonstrate that SimBa exhibits higher simplicity bias compared to standard MLPs and shows consistent performance improvements as the number of parameters increases. When integrated with various RL algorithms, including SAC, TD-MPC2, PPO, and METRA, SimBa improves sample efficiency. Moreover, SimBa matches or surpasses state-of-the-art off-policy RL methods across 51 continuous control tasks while maintaining computational efficiency
weaknesses: - While the evaluation covers 51 continuous control tasks, it might be beneficial to see SimBa's performance on a wider range of RL domains, particularly with images.
- The paper doesn't discuss potential limitations or scenarios where SimBa might not be as effective

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Out of curiosity, SimBa is usually the most time-efficient (except SAC), but on HumanoidBench, it is slightly worse than TD7 and BRO. Do you know why?
2. The simplicity bias hypothesis looks pretty general; therefore, how will this architecture work on image-based problems?
3.  How do you expect SimBa to be applied in the discrete control tasks? Will it be enough to beat BBF on Atari as BRO on continuous control? -- Of course, new experiments on Atari would be too much for the rebuttal period, but maybe the authors have some thoughts about this.
4. How do your results about simplicity bias measured through Fourier features relate to this paper [1]? As I understand, authors of [1] claim that neural networks are naturally biased to learn low frequencies faster, and it is hard for them to learn high-frequency signals, which is present in reinforcement learning. SimBa promotes architectures that are biased towards low frequencies. Could you share your perspective on how these findings interact or complement each other?

[1] Yang, G., Ajay, A., & Agrawal, P. (2022). Overcoming the spectral bias of neural value approximation. arXiv preprint arXiv:2206.04672.
soundness: 3
strengths: 1. Clear and Well-Motivated Objective: The paper identifies simplicity bias as an underexplored factor in RL network scaling.
2. Reproducibility Efforts: A public codebase and descriptions of evaluation setups are provided.
3. Broad experimental setup.
summary: The authors propose the SimBa architecture, which, by inducing simplicity bias, allows scaling the size of neural networks in deep reinforcement learning problems. The proposed architecture is a neural network with residual connections, with post-layer normalization and observation normalization at the beginning. The authors claim that SimBa enables networks to scale up without sacrificing generalizability, avoiding the overfitting issues often associated with high-parameter models in RL.
weaknesses: 1. Statistical significance is not apparent from the presented results. The standard error overlaps in some plots, e.g., Figures 1b, BRO, and SimBa on Figure 5b, and Figure 8 has no rages, but I guess it would be more challenging to add them since the analysis is in two dimensions, and also from Figure 14. The tables in the Appendix do not have standard deviations. Moreover, authors could use some t-test or Mann–Whitney U test to demonstrate statistical significance.
2. It would be valuable to see reversed to Fig. 4 ablation, i.e., simplicity and Return when you turn off every single component. Clearly, all components analyzed in section 5.1 have some synergic effect -- they do not help much alone. However, it is not apparent how all of them are important in this synergy, i.e., from section 7.1, we know that RSNorm is very important -- If I understand correctly, SimBa without observation normalization works very poorly. But how will the lack of RSNorm affect the simplicity of the architecture? -- I would suggest visualizing this similarly to Figure 4, but on the Y-axis, it would be, e.g., SimBa - RSNorm or SimBa - Residual.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 3
questions: See Weaknesses section.
soundness: 3
strengths: - Easy to implement, which is promising for future work.

- Good comparison with a SOTA algorithm TD-MPC2, and even an ablation using SimBa on TD-MPC2.

- A strong step forward in the domain of non-algorithmic improvements in RL.
summary: This paper builds upon the insights of the paper by [Teney et al, 2024], which shows that certain architectural implementations provide a simplicity bias, which could be one of the more causal explanations to their practical usefulness.

In order to transfer this notion of simplicity bias to RL, the authors run tests with SAC on proprietary state-based RL, ranging from easy (Cartpole etc.) to hard (Humanoid, Dog, etc.) environments.

To maximize the simplicity bias of vanilla SAC, the authors augment the original architecture with RSnorm, Residual Feedforward blocks and Layer Normalization. Additionally, the authors scale up the architecture.

The results are promising, showing that keeping a simplicity bias in the network allows for scaling up the network, elevating vanilla SAC to the level where it can perform very well on DMC-Hard tasks, without using algorithmic advancements. 

Although results are only done on state-based architectures, the harder environments like Humanoid & Dog still allow for sufficient difficulty in the evaluation. Additionally, pixel-based environments have already shown correlation between simplicity bias in the CNN and performance (Residual connections in CNN’s -> Impala architecture), although to the best of my knowledge, the reason being a simplicity bias was never concluded by these researchers.
weaknesses: - The plasticity analysis in Appendix C. could be confusing to the average reader. It shows a comparison to a basically ‘collapsed’ baseline MLP, as the effective rank is negligible. This might give a reader the impression that these are the representative network properties of training with a vanilla MLP, and paints a too strong picture of SimBa’s added network properties. I believe it would therefore be much more informative to the reader to add an additional 2 rows of figures to Fig. 15:
 		- Row 1: Plasticity analysis on DMC - medium
                - Row 2: Plasticity analysis with all the ablations of Fig. 4, to show the unique effects of every module.

- As important as it is to see that SimBa is computationally cheap, it would be also interesting to add some figures showing longer training (e.g. 5 million timesteps) on DMC hard and comparing to TD-MPC2 and the baseline SAC.

- I believe a mention of the improvements that the Impala network gave to RL should be in the paper, and maybe a short sentence about the correlation (Residuals connections -> Simplicity Bias -> better Convergence).

Meta-review: This paper introduces a simple and effective architecture(SimBa) to scale up network parameters in a deep RL setting, by defining and leveraging a simplicity bias score. SimBa consists of three key components that are drop-in replacement to existing RL methods: RSNorm (running statistics norm), residual feedforward blocks and layer norm. The key idea is that by promoting simplicity bias SimBa enhances performance in over-parametrized regimes. SimBa is shown to improve sample & computational efficiency, and overall performance across a variety of standard RL environments and algorithms. The key finding then is that by using a simplicity bias, we can effectively scale parameters without algorithmic modifications, achieving competitive or superior results compared to SOTA methods. 

I think this paper proposes a method that is simple and effective, with clear strengths. Leveraging simplicity bias is a relatively under explored topic in RL. It likely has relationship to algorithmic kolmogorov complexity and should be studied further to discover simple architectural changes that could yield better generalization in neural networks. The experimental results are rigorous, spanning more than 50 tasks. The simplicity bias score is theoretically motivated and makes sense to me. 

The biggest limitation of SimBa is that its only evaluated on state space RL tasks - there is no vision or large observation spaces. This will limit its applicability but I do not think this is a severe limitation. I think it is important and interesting to study these problems in isolation. We will learn something new when the authors or other researchers do further studies in that direction and compare it back to this paper. Another problem with the simplicity score framework is to discuss and highlights failure modes - what happens in under parametrized regimes or what are other failure modes? 

Despite these limitations, I think the paper is strong enough and there is clear consensus amongst the authors regarding its strength. I would recommend accepting this paper and it will be fruitful for the ICLR community to explore more in this direction.

Decision: Accept (Spotlight)