SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement
  Learning
Authors: name: Hojoon Lee; name: Dongyoon Hwang; name: Donghu Kim; name: Hyunseung Kim; name: Jun Jet Tai; name: Kaushik Subramanian; name: Peter R. Wurman; name: Jaegul Choo; name: Peter Stone; name: Takuma Seno
Published: 2024-10-13 07:20:53

Abstract: Recent advances in CV and NLP have been largely driven by scaling up the
number of network parameters, despite traditional theories suggesting that
larger networks are prone to overfitting. These large networks avoid
overfitting by integrating components that induce a simplicity bias, guiding
models toward simple and generalizable solutions. However, in deep RL,
designing and scaling up networks have been less explored. Motivated by this
opportunity, we present SimBa, an architecture designed to scale up parameters
in deep RL by injecting a simplicity bias. SimBa consists of three components:
(i) an observation normalization layer that standardizes inputs with running
statistics, (ii) a residual feedforward block to provide a linear pathway from
the input to output, and (iii) a layer normalization to control feature
magnitudes. By scaling up parameters with SimBa, the sample efficiency of
various deep RL algorithms-including off-policy, on-policy, and unsupervised
methods-is consistently improved. Moreover, solely by integrating SimBa
architecture into SAC, it matches or surpasses state-of-the-art deep RL methods
with high computational efficiency across DMC, MyoSuite, and HumanoidBench.
These results demonstrate SimBa's broad applicability and effectiveness across
diverse RL algorithms and environments.