Mixtures of Experts Unlock Parameter Scaling for Deep RL
Authors: name: Johan Obando-Ceron; name: Ghada Sokar; name: Timon Willi; name: Clare Lyle; name: Jesse Farebrother; name: Jakob Foerster; name: Gintare Karolina Dziugaite; name: Doina Precup; name: Pablo Samuel Castro
Published: 2024-02-13 17:18:56

Abstract: The recent rapid progress in (self) supervised learning models is in large
part predicted by empirical scaling laws: a model's performance scales
proportionally to its size. Analogous scaling laws remain elusive for
reinforcement learning domains, however, where increasing the parameter count
of a model often hurts its final performance. In this paper, we demonstrate
that incorporating Mixture-of-Expert (MoE) modules, and in particular Soft MoEs
(Puigcerver et al., 2023), into value-based networks results in more
parameter-scalable models, evidenced by substantial performance increases
across a variety of training regimes and model sizes. This work thus provides
strong empirical evidence towards developing scaling laws for reinforcement
learning.