Value-Based Deep RL Scales Predictably
Authors: name: Oleh Rybkin; name: Michal Nauman; name: Preston Fu; name: Charlie Snell; name: Pieter Abbeel; name: Sergey Levine; name: Aviral Kumar
Published: 2025-02-06 18:59:47

Abstract: Scaling data and compute is critical to the success of machine learning.
However, scaling demands predictability: we want methods to not only perform
well with more compute or data, but also have their performance be predictable
from small-scale runs, without running the large-scale experiment. In this
paper, we show that value-based off-policy RL methods are predictable despite
community lore regarding their pathological behavior. First, we show that data
and compute requirements to attain a given performance level lie on a Pareto
frontier, controlled by the updates-to-data (UTD) ratio. By estimating this
frontier, we can predict this data requirement when given more compute, and
this compute requirement when given more data. Second, we determine the optimal
allocation of a total resource budget across data and compute for a given
performance and use it to determine hyperparameters that maximize performance
for a given budget. Third, this scaling behavior is enabled by first estimating
predictable relationships between hyperparameters, which is used to manage
effects of overfitting and plasticity loss unique to RL. We validate our
approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI
gym, and IsaacGym, when extrapolating to higher levels of data, compute,
budget, or performance.