ICML 2025

Value-Based Deep RL Scales Predictably
Oleh Rybkin 1 Michal Nauman 1 2 Preston Fu 1 Charlie Snell 1 Pieter Abbeel 1 Sergey Levine 1 Aviral Kumar 3
Abstract
Abstract: Scaling data and compute is critical
to the success of modern ML. However, scaling
demands predictability: we want methods to not
only perform well with more compute or data, but
also have their performance be predictable from
small-scale runs, without running the large-scale
experiment. In this paper, we show that value-
based off-policy RL methods are predictable de-
spite community lore regarding their pathological
behavior. First, we show that data and compute
requirements to attain a given performance level
lie on a Pareto frontier, controlled by the updates-
to-data (UTD) ratio. By estimating this frontier,
we can predict this data requirement when given
more compute, and this compute requirement
when given more data. Second, we determine
the optimal allocation of a total resource budget
across data and compute for a given performance
and use it to determine hyperparameters that max-
imize performance for a given budget. Third, this
scaling is enabled by first estimating predictable
relationships between hyperparameters, which is
used to manage effects of overfitting and plastic-
ity loss unique to RL. We validate our approach
using three algorithms: SAC, BRO, and PQL on
DeepMind Control, OpenAI gym, and IsaacGym,
when extrapolating to higher levels of data, com-
pute, budget, or performance.
1. Introduction
Many latest advances in various areas of machine learning
have emerged from training big models on large datasets.
In this scaling guided research landscape, successfully ex-
ecuting even one single training run often requires a large
amount of data, computational resources, and wall-clock
time, such as weeks or months (Achiam et al., 2023; Team
et al., 2023; Ramesh et al., 2022; Brooks et al., 2024). To
1UC Berkeley 2University of Warsaw 3CMU. Correspondence
to: Oleh Rybkin <oleh.rybkin@gmail...