Value-Based Deep RL Scales Predictably
Authors: Oleh Rybkin
Michal Nauman
Preston Fu
Charlie Victor Snell
Pieter Abbeel
Sergey Levine
Aviral Kumar
Venue: ICML 2025 poster (2025)
Keywords: scaling laws
online reinforcement learning
q-learning
Primary area: reinforcement_learning->deep_rl
TL;DR: We establish that value-based online RL can be scaled predictably to larger data, larger compute, or generally larger budget

Abstract: Scaling data and compute is critical in modern machine learning. However, scaling also demands _predictability_: we want methods to not only perform well with more compute or data, but also have their performance be predictable from low compute or low data runs, without ever running the large-scale experiment. In this paper, we show predictability of value-based off-policy deep RL. First, we show that data and compute requirements to reach a given performance level lie on a _Pareto frontier_, controlled by the updates-to-data (UTD) ratio. By estimating this frontier, we can extrapolate data requirements into a higher compute regime, and  compute requirements into a higher data regime. Second, we determine the optimal allocation of total _budget_ across data and compute to obtain given performance and use it to determine hyperparameters that maximize performance for a given budget. Third, this scaling behavior is enabled by first estimating predictable relationships between different _hyperparameters_, which is used to counteract effects of overfitting and plasticity loss unique to RL. We validate our approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI gym, and IsaacGym, when extrapolating to higher levels of data, compute, budget, or performance.

Value-Based Deep RL Scales Predictably
Oleh Rybkin'
Michal Nauman 12 Preston Fu!
Charlie Snell Pieter Abbeel Sergey Levine Aviral Kumar
Abstract
maximize the success of these large-scale runs, the trend in the machine learning (ML) community has shifted toward not just performant, but also more predictable algorithms that scale reliably with more computation and training data size, such that downstream performance can be predicted from small-scale experiments, without actually running the large-scale experiment (McCandlish et al., 2018; Kaplan et al., 2020; Hoffmann et al., 2023; Dubey et al., 2024).
Abstract: Scaling data and compute is critical to the success of modern ML. However, scaling demands predictability:
want methods to not only perform well with more compute or data, but also have their performance be predictable from small-scale runs, without running the large-scale experiment. In this paper, we show that value- based off-policy RI. methods are predictable de- spite community lore regarding their pathological behavior. First, we show that data and compute requirements to attain given performance level lie on Pareto frontier, controlled by the updates- to-data (UTD) ratio. By estimating this frontier, we can predict this data requirement when given more compute, and this compute requirement when given more data. Second, we determine the optimal allocation of total resource budget across data and compute for given performance and use it to determine hyperparameters that max- imize performance for a given budget. Third, this scaling is enabled by first estimating predictable relationships between hyperparameters, which is used to manage effects of overfitting and plastic- ity loss unique to RL. We validate our approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAl gym, and IsaacGym, when extrapolating to higher levels of data, com- pute, budget, or performance.
In this paper, we study if deep reinforcement learning (RL) is also amenable to such scaling and predictability benefits. We focus on value-based methods that train value func- tions using temporal difference (TD) learning, which are known to be performant at small scales (Mnih et al., 2015; Lillicrap et al., 2016; Haarnoja et al., 2018). Compared to policy gradient (Mnih et al., 2016; Schulman et al., 2017) and search methods (Silver et al., 2016), value-based RL can learn from arbitrary data and require less sampling or search, which can be inefficient or infeasible for open-world problems where environment interaction is costly.
We study scaling properties by predicting relationships be- tween different resources required for training. Data requirement D is the amount of data needed to attain cer- tain level of performance. Likewise, compute requirement C refers to the amount of FLOPS or gradient steps needed to attain a certain level of performance. In RL uniquely, performance can be improved by increasing either available data or compute (c.g., training multiple times on the same data), which we capture via budget requirement that com- bines data and compute J C 8. D, where 8 is some constant. An additive budget function is useful when the cost of data and compute can be expressed in similar units, such as wall-clock time or required finances.
1. Introduction
Many latest advances in various areas of machine learning have emerged from training big models on large datasets. In this scaling guided research landscape, successfully ex- ecuting even one single training run often requires a large amount of data, computational resources, and wall-clock time, such as weeks or months (Achiam et al., 2023; Team et al., 2023; Ramesh et al., 2022; Brooks et al., 2024). To
To establish scaling relationships, we first require a way to predict the best hyperparameter settings at each scale. We find that learning rate n, batch size B, and the updates- to-data (UTD) ratio a are the most erucial hyperparame- ters for value-based RL. While supervised learning benefits from abundant theory to establish optimal hyperparame- ters (Krizhevsky, 2014; McCandlish et al., 2018; Yang et al., 2021), value-based RL often does not satisfy assumptions typical of supervised learning. For example, vasue RL needs to account for the non-i.i.d. nature of training data. Distribution shift due periodic changes in the data col-
'UC Berkeley Uniierssity of Warsaw *CMU. Correspondence to: Oleh Rybkin <oleh.rybkin@ wwwrrrkkinnngii.ooss..... gomi, Aviral Kumar <avi- ralku @ andrew.cmu.cdu>>
Proceedings of the 42nd International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

high performance. Right: Pareto frontiers for Value-Based Deep RL Scales Predictably
x1e3
2.00
x1c3
800
120
Emplrical value
(P)
256
Parorr Informan anne level
Dy Data until
95
Asymptate prm
:oppiiall UTD
1.41
Extrepo abion
Dj Data until
128
600
DMC
1.00
500
400
60
0.71
300
Optimal budget
0.50
200
10
28 Cj: Compute until /
14
20
40x1c14
10
16
25
407.14
32
64
x1c14
Py: Budget until
Cj
Compute until
Xled
Empincal vo ue
8.00
x164
Optimal budget
800
Portos level
OpenAl Gym
Fit PyGe)
aopkimall UTD
64
700
D2 Data unbil
Asymptate pan
4.00
B3c Data until
64
38
-600
23
500
2.00
32
13
100
Ixtrapolation
8
16
1.00
300
25
33
45
60x1e13
20
34
5B
100
27
Xlel3
Gj Compute until
Fj: Budget until
C/ Compute J
0.60
x1e7
800
Empirical value Fit DAC))
(P)
128
Optima Budget
Performann level
Isaac Gym
1): Data untilf
64
D': optimal UTD
0.46
Dj. Data untily
600
Asymptote gron
x
Extrapolation
32
0.35
32
400
16
0.26
200
8
0.20
16
32
64 Xle4
12
24 xle5
80
x1e4
Gj Gradieen Steps until/
fy: Budgget until
Gj Gradient steps until/
(1) Compute- Data Pareto frontier
(II) Budget extrapolation
(III) Fits for multiple
Figure 1: Scaling properties when increasing compute C, data D, budget F, or performance J. Left: Compute versus data requirements Pareto frontier controlled by the UTD ratio a. We observe that we can trade off data for compute and vice versa, and this relationship predictable. Middle: Extrapolation from low high performance. We observe that the optimal resource allocation controlled by evolves predictably with increasing budget, and can be used to extrapolate from several performance levels
lection policy (Levine et al., 2020) contributes to a form of overfitting where minimizing training TD error may not re- sult in low TD error under the data distribution induced by the new policy. In addition, objective shift due to changing target values (Dabney et al., 2021) contributes to "plasticity loss" (D'Oro et al., 2023; Kumar et al., 2021). We show that it is possible to account for the training dynamics unique to value-based RL, and are able to find the best hyperparam- eters by setting the batch size and learning rate inversely proportional to the UTD ratio. We estimate this dependency using power law (Kaplan et al., 2020), and observe that this model makes effective predictions.
from low to high performance levels. Instead of extrap- olating as a function of return, which can be arbitrary and extrapolate
function of the allowed budget F. We can define optimal tradeoff between data and compute, and we observe that such optimal tradeoff value evolves predictably to higher budgets, which also at- tains a higher performance level (Figure 1, middle). Thus we are able to predict optimal hyperparameters, as well as data and compute allocation, for high-budget runs using only data from low-budget runs.
non-smooth,
Our contribution is showing that the behavior of value- based deep RL methods based on TD-learnng is predictable in larger data and compute regimes. Specifically, we:
Using the best predicted hyperparameters, we are now able to establish that data and compute requirements evolve as a predictable function of the UTD ratio a. Furthermore, a defines the tradeoff between data and compute, which can be visualized as a Pareto frontier (Figure 1, left). Using this model, we are able extrapolate the resource requirements from low-compute to high-compute setting, as well as from low-data to high-data setting shown in the figure.
establish predictable rules for dependencies between hyperparameters batch size (B), learning rate (1), and UTD ratio (a) value-based RL, and show that these rules enable more effective scaling.
2. show that data and compute required to attain given performance level lie on Pareto frontier, and are re- spectively predictable in the higher-compute or higher- data regimes.
Using the Pareto frontiers, we are now able to extrapolate
2

---

Value-Based Deep RL Scales Predictably
3. show the optimal allocation of budget between data and compute, and predict how such allocation evolves with higher budgets for best performance.
Iraining run? We will answer such questions by fitting em- pirical laws from low data and compute runs to determine relationships between hyperparameters. Doing so, in turn, enables us to determine how to set hyperparameters and allocate resources to maximize performance when provided with a larger data and compute budget. Note that we wish to make these hyperparameter predictions without running the large data and compute budget experiment. While questions of this form have been studied in supervised learning, the answers are different for online RL, because online RL con- tinuously collects its own data, which ties data and compute in a complex manner and breaks i.i.d. nature of datapoints.
Our findings apply to algorithms such as SAC, BRO, and PQL., and domains such as the DeepMind Control Suite (DMC), OpenAl Gym, and IsaacGym. The generality of our conclusions challenges conventional wisdom that value- based deep RL does not scale predictably.
2. RL Preliminaries and Notation
We study standard off-policy online RL, which maximizes the agent's return by training on a replay buffer and pe- riodically collecting new data (Sutton and Barto, 2018). Value- deep RL methods train a Q-network, Qo. to minimize the temporal difference (TD) error:
Concretely, we study three resource optimization questions: (1) maximizing sample efficiency (i.e., minimize the amount of data D to attain a target performance under a given com- pute budget), (2) conversely, minimizing compute C (e.g.. FLOPs or gradient steps, whichever is more appropriate) to attain a given performance given an upper bound on data that can be collected, and (3) maximizing performance given a total bound on data and compute.
L(0) = Ep (r(s,a) + ((('ss()))) - 2)) (2.1)
where P is the replay buffer, Q is the target Q-network, denotes a state, and a' is an action drawn from a policy (-Is) that aims to maximize Qo a). We implement this operation by sampling a batch of size B from the buffer and taking a gradient step along the gradient of this loss with a learning rate 17. In theory, off off-policy algorithms can be made very sample efficient by minimizing the TD er- ror fully over any data batch, which in practice translates to making more update steps to the Q-network per environment step, or higher "updates-to-data" ratio (UTD) (Chen et al., 2020). However, increasing the UTD ratio naïvely can lead to worse performance (Nikishin et al., 2022; Janner et al., 2019). To this end, unlike the standard supervised learning or LLM literature that considers B and 17 as two main hyper- parameters affecting training (Kaplan et al., 2020; Hoffmann et al., 2023), our setting presents another hyperparameter, the UTD ratio o, that we also study in our paper.
Problem 3.1 (Resource optimization problems). Find the best configuration (B, 17, σ) for algorithm Alg that minimizes either the data 'D or compute C consumed to obtain performance Jo:
1. Maximal sample efficiency:
(B (B"nn'o') == arg min
D
(B.n.)
s.t. J (B. n. Jo CoC
2. Maximal compute efficiency:
(B' (B",n",'") == arg min C
(B.n.)) s.t. J (TAAI( (B, 2 Jo
Notation. In this paper, we focus on the following key hyperparameters: the UTD ratio a, learning rate 7), and the batch size B. We will answer questions pertaining to performance of policy denoted by JTT,, the total data utilized by algorithm to reach a given target level of performance J (denoted by D,), and the total compute budget utilized by the algorithm to reach performance J (denoted by C,), which is measured in terms of FLOPs or wall-clock time taken by the algorithm.
D Do-
We solve these problems by fitting empirical models of the minimum data and compute needed to attain a target performance for different values of J/. Doing so allows us to then solve the third setting (3) for maximizing performance given total budget on data and compute as shown below.
3. Problem Statement and Formulation
Problem 3.2 (Maximize performance at large data and compute budget). Find the best configuration (B, 17, o) and resource allocations for data 'D and compute C that enable Alg to maximize performance at budget Fo
To demonstrate that the behavior of value-based RL can be predicted reliably at scale, we first post multiple resource optimization questions that guide our scaling study. Viewing data and compute as two resources, we answer questions of the form: what is the minimum value of [rescurce] needed to attain a given target performance? And what should the hyperparameters (e.g., B, 11, σ) be in such this
(B" .n"."". ,²))"") == arg max (B,).)
J A n o))
s.t. C + 8. D S Fo-
3

---

Value-Based Deep RL Scales Predictably
120
C: Computte unt 1
50
Empiicaal balue <lo?
Scaling Observation 1: Data Requirements
By Dota undij
n/o1
13
95
Acympeste oere
The amount of data Dj needed to reach a given return target Jo decreases as a predictable function of the UTD 0, and is a power law (Eq. (4.1)).
22
76
60
10
48
0.50
1.00
2,00 4.00
8.00
0.25
0.30
1:00 2.00 4.00 8.30 UTD Ratio
To answer the optimization questions in Problem 3.1, we also need an expression for required compute until the target return Cj. As determines the number of gradient steps per data point, C. is function of a. In particular, total compute is equal to the number of gradient steps taken multiplied by the parameter count of the model. Our study does not optimize over the model size and treats it as a constant. Thus, we write the compute C, as a function of a as:
Figure 2: The data-compule tradeoff DMC. Left: The minimum required data D, scales with the UTD as power law. Right: The minimum required compute Cy increases with the UTD a as a sum of two power laws.
4. Scaling Results For Value-Based Deep RL
We will now present our main results addressing Problem 3.1 under the two settings discussed above. We will then use these results to present results for Problem 3.2. In order to do so, we run several experiments and estimate scaling trends from the results. Although this procedure might appear stan- dard from scaling studies in language modeling, we found that instantiating for value-based RL requires understand- ing the interaction of the various hyperparameters appearing in TD updates, and the data and compute efficiency of the algorithm. We will formalize these relationships via empiri- cally estimated laws and show that these laws extrapolate reliably to new settings not used to obtain these empirical laws. Therefore, in this section, we present empirical and conceptual arguments to build functional forms of relation- ships between different hyperparameters. Before doing so, we provide our answers to Problems 3.1 and 3.2.
(4.2) where N denotes the model size, B(r) denotes the "best choice" batch size for a given UTD value a, and other vari- ables follow definitions from before. Note the additional factor of 10 in Eq. (4.2) emerges from the use of multiple forward passes to compute the loss function for value-based RL and the backward pass, through the Q-network (to con- trast with language modeling, the typical multiplier is 6; the gap in our setting comes from the use of multiple forward passes). We plot C)( for different values of a and J = Jo in Figure 2. Since Dy(a) is not a constant and depends itself on a, we note that this particular relationship between C)( and is not a simple power law unlike Eq. (4.1). Instead, our derivation in Eq. (A.4) shows that C)(a) is given by a sum of two different power laws in α.. Similarly to DJ, we also observe that the compute utilized is prediciable func- tion of a: we are able to accurately estimate the compute at larger values of a using the relationship in Eq. (4.2).
Cy(a) R 10 N. B(o)- a. -D((o)
4.1. Main Scaling Results
We begin by answering Problem 3.1 where we maximize sample efficiency. We wish to estimate the minimal amount of data D., needed to attain a given target performance, given an upper bound on compute C S Co. To do so, we fit D, needed to attain the target performance J = Jo pa- rameterized by the UTD ratio (Eq. (4.))). Intuitively, the minimum amount of data needed to attain a given perfor- mance is lower as more updates are made per datapoint (i.e., when o is high), as more "valuc" could be derived from the same datapoint. In addition, we would expect that even for the best value of o, there is a minimum number of dat- apoints Dinn that are needed to learn given the "intrinsic" difficulty of the task at hand. Based on these intuitions, we hypothesize a power law relationship between (a) and a, with an offset Duin and constants a.j and By.
Scaling Observation 2: Compute Requirements
The compute C, to attain a given return target Jo increases as predictable function of the UTD ratio o, and is sum of two power laws (Eq. (4.2)).
We observe that both required compute and data are con- trolled by the UTD ratio a, which allows tradeoff between compute and data controlled by a. We plot this tradeoff as a curve with compute C)(rr as æ-axis and Dj (a) as y-axis in Figure 1 (left). Further, as D)(a) is a monotonically decreasing function of a, this curve defines a Pareto frontier: we can move left on the curve to increase data efficiency as the expense of compute and move right to increase compute efficiency at the expense of data. Also interestingly, due to the compute law being a sum of two power laws, in many environments there is a minimum after which compute efficiency no longer improves as seen on OAI Gym in Figure 1.
define
B,
Dy(0) R Dygin +
(4.1)
Empirical fits of D., and a on the DMC suite are in Figure 2 and they validate the efficacy of this fit. We also emphasize that the existence of this power law makes D, predictable, in that we can predict D, for larger values of a that fall outside the range of a values used to get the fit (Figure 6).
Solving for maximal data efficiency (Problem 3.1, (1)) We can now solve Problem 3.1 in setting (1). our strategy

---

Value-Based Deep RL Scales Predictably
address setting (1) is to find the largest (say max that satisfies the compute constraint (r) S Co, and then plug this max into ) (r) to obtain the data estimate. This approach enables us to express Dj directly the available compute Co. as we calculate in Eq. (4.2). This can be visualized as finding the value D, corresponding to some value Co on the Pareto frontier (Figure left)
x1e3
800
256
Performant ce level
700
function of
Dj: Data until
128
10000
600
64
500
J. J: /
400
32
Solving for maximal compute efficiency (Problem 3.1, (2)). Likewise, the solution in (2) can be obtained by finding the smallest value of in the range that satisfies the data constraint Do( < Do. and computing the corresponding value of C)(o). This can similarly be visualized on the Pareto frontier (Figure 1, left). We summarize our observa- tions in terms of the following takeaway.
Optimal budget
300
Iso-cost lines
16
200
8
16
32
64 x1e14
Gj Compute until
Figure 3: Visualization of the solution to Problem 3.2. Several Pareto frontiers (Figure left) shown, together with lines of iso-budget F. which define optimal budget points (D", C*). Corresponding optimal UTD ratios o² are a predictable function of the budgets Fo. trend line shown dashed.
Solving 3.1: The Compute Compute-Data Data Pareto frontier
The UTD ratio defines a Pareto frontier between data and compute requirements, and estimating this frontier yields predictable solutions to resource op- timization problems in settings (1) and (2). Theo- retically, the optimal D for an available compute budget Co is:
this point with a given Ø value. Hence, we can estimate the best value of o² (Foo for a given budget threshold Fo- Concretely, we observe a power law between a (Fo) and Fu. with constants 3, and Or..
ar
(Fo)
(4.5)
(C) Co- (10 N B(o"))o")) o-) (4.3)
The optimal C,y for given data budget Do is:
Solving 3.2: Maximize Return Given Budget
(Po) & 10 N. B(o") 0² Do- (4.4)
Above, o² denotes the minimizing UTD value. Cal- culation details are in Appendix A.
The best UTD value a that leads to maximal J is a predictable function of the budget Fo over data and compute, this relationship follows a power law, and also extrapolates to large budgets.
Maximize return within a budget (Problem 3.2). Finally, we tackle Problem 3.2 in order to extrapolate from low to high return. Here, we do not want to minimize resources, but rather want to maximize performance within a given total "budget" tion 3, we consider budget functions linear in both data and compute, i.e., F = C+ C 5 D, for a given constant 8. Our estimated Pareto frontier in Eq. (4.4) will enable answering this question. To do so, we turn to directly predicting a good UTD value σ² This UTD value is one that not only leads maximal performance, but also stays within the total re- source budget Fo- Once the UTD value has been identified, it prescribes a concrete way to partition the total resource budget into good data and compute requirements using the solutions to Problem 3.1.
This relationship produces the optimal o, and as a result, the optimal data and compute allocations to reliably attain maximum performance. As shown in Figure I, estimating this law from low- bow-budget experiments is sufficient for pre- dicting good values for large budget runs. These predicted a (Fo) values extrapolate reliably to budgets outside the range used to fit this law (as shown by x in Figure 1). This concludes an exposition of our main results.
data and compute. As discussed in Sec-
4.2. Fitting Relationships Between (B,
To arrive at these scaling law fits above, we had to set hy- perparameters B and 17, which we empirically observed to be important. We fit these hyperparameters as a function of a, the only variable appearing in many of the scaling relationships discussed above. In this section, we will now describe how estimate good values of B and 7 in terms of a. Our analysis here relies crucially on the behavior of TD-learning that is distinct from supervised learning, where the UTD ratio does not exist.
We plot the data-compute Pareto frontiers for multiple val- ues of Jo in Figure 3 and in Figure 1 (right), and find that these curves move diagonally to the top-right for larger Jo- Intersecting these curves with iso-budget frontiers over D and C. prescribed by the budget function, gives us the largest possible Jo for which there is still a (D,C) pair that just falls just within the budget Fo but attains performance Jo (see Figure 3 for a worked out version of this procedure).
To understand relationships between batch size B, learning rate y7, and the UTD ratio a, we ran an extensive grid search. We first attempted to explain the relationship between the B
Since both D and are explained by a, we can associate
5

---

Value-Based Deep RL Scales Predictably
Empirical value
Empirical value Trend
Validation
Trend
Overfiting
Training
Overfitting
Overfitting
Loss
Best batch size
Critical batch Size
4
4
2
0
2
128
256
512
Batch size
a: UTD Ratio
B: Batch Size
Learning
Empirical value
Empirical value
Parameter Norm
4
Trend
Parameter Norm
4
Trend
Supervised Learning
Plasticity
17 oc o/a
Loss
1y VB
2
0
1
2
4
1.5e-4
3e-4
6e-4
1.2e-3
Learning rate
a: UTD Ratio
7:: Learning Rate
()) Hparam choice for SL RL
(II) Effect of UTD ratio 0
(III) Effect of B and n
Figure 4: Hyperparameter effects in supervised learning and TD learning on DMC. Top: Overfitting increases with UTD while batch size be used counteract it. Bottom: Higher UTD leads to poor training dynamics and plasticity loss (D'Oro Oro et al., 2023). Lower learning rates can be used to counteract it. While these relationships are not perfectly predictable, we use them to inform our design choices.
and 77 values that attain the highest data efficiency (denoted B" ,n") using the standard heuristic in supervised learning: when the batch size is smaller than the critical butch size, B and n) are inversely correlated with each other (McCandlish et al., 2018). However, as shown in Figure 5 (right), we find that without including the UTD ratio a, best B" and n7" exhibit very weak correlation. Further, the critical batch size (McCandlish al., 2018) does not correlate with em- pirically best batch size
et al., 2023a; Nauman et al., 2024a). One way to counteract this form of "overfitting" from high UTD value is to instead reduce the batch size in the run so that the training process sees a given sample fewer times. In fact, for fixed UTD value a, we empirically validate this hypothesis that a lower B leads to substantially reduced overfitting on several tasks in Figure between the best batch size B and the UTD value a. We show in Figure 5 that indeed this inverse relationship be estimated well by power law, given formally as:
Hence, we post an inverse relationship
show in Appendix F. Instead, surprisingly, observe strong correlation between 8* and a, as well as 17 and respectively. Since B" and ry" exhibit near zero correlation among themselves, we can simply omit their dependency and opt for modeling them independently as a function of the UTD ratio, o. We con- ceptually explain relationships between B* and ar, and 17" and below and show that models developed from this un- derstanding enable us to reliably predict good values of B and 17. allowing us to fully answer Problem 3.1.
BB
&
(*(
(4.6)
Predicting best choice of learning rate 1, as a function of a. Next we turn to understanding the relationship between r) and a. We start from a simple observation: a very large a typically leads to worse performance not only due to overfitting but also due to plasticity loss (Kumar et al., 2021; D' Oro et al., 2023; Lyle et al., 2023), defined broadly the inability of the value network fit TD targets appearing later in training. Prior work states that plasticity loss is inherently related to the number of gradient steps performed and claims that larger norms of parameters of the Q-network are indicative of plasticity loss (D'Oro et al., 2023; Lyle et al., 2023). We would expect larger learning rate to make higher magnitude updates against the same TD target, and hence move parameters
Predicting best choice of B in terms of a. Our proposed functional form for the best batch size 3* takes the form of power law in a, which we also empirically validate in Figure (left). We posit this form because, intuitively, large batch sizes increase the risk of overfitting because they lead to repetitive training small training loss on the distribution of data in the buffer does not necessarily reflect the behavior policy distribution of a learning agent (Levine et al., 2020). This means that minimizing the training loss to large extent can result in poor test performance (T)) as also seen by prior work (Li
fixed set of data. Furthermore,
state that suffers from difficulty in fitting subsequent targets (Dabney et al., 2021; Lee et al., 2024). As shown in Figure 4, the parameter norm indeed increases with high learning rate. Therefore, given UTD
6

---

Value-Based Deep RL Scales Predictably
x1e-4
8 :Best batch size
BBstt learning rate
x1e4
8.0
(d)
n" Best learning rate
12.0
Constant fit
512
7.1
5.8
4.2
256
4.2
2.5
B ['(o)
Constant fit
3.0
1.5
Trend
128
1
2
4
8
1
2
4
8
32
64
128
256 512
a: UTD ratio
a: UTD ratio
8: Best batch size
Figure 5: Left, middle: Fitting the best learning rate 1/" and batch size B* given UTD a on DMC. Modeling the dependency on a crucial to obtain good hyperparameters, whereas using constant B, 7 as is commonly done leads too poor extrapolation. Right: the best learning rate and batch size are not significantly correlated, major difference from supervised learning.
value a. we hypothesize that the best choice of learning rate, "/" (ar) for a given performance should scale inversely in a. Empirically we observe that this is indeed the case (Figure 5 (middle)), and we model this relationship:
workflow can serve as a useful skeletion for scaling law studies with other value-based algorithms as well.
4.4. Evaluating Extrapolation
B,g
Evaluating budget extrapolation. Results on all environ- ments are shown Figure (middle). We estimate several Pareto frontiers corresponding to points with equal changes in budget. We perform the o (Fo) fit, while holding out two largest budgets. The quality of our fit for these two extrapolated budgets be seen in the figure.
1/" (0)
(4.7)
Scaling Observation 3: Hyperparameter Selection
The best choices for the batch size and learning rate predictable functions of the UTD a, and both of these relationships follow a power law.
Evaluating Pareto frontier extrapolation. Results Ope- nAI Gym are shown in Figure 6. We fit the data efficiency equation (o Eq. (4.1) while holding out either two UTD values with largest data requirement (left) or two a values with largest compute requirement (right). The quality of our fit for these extrapolated values can be seen in the figure.
4.3. Empirical Workflow
Fitting Empirical Relationships
1. Run sweep for batch size B and learning rate 7) for several values of UTD Since the batch size and learning rate are independent for the best a, we can run these sweeps independently.
Hyperparameter fit extrapolation. Results on OpenAl Gym are shown in Figure 6 (right). We plot the data ef- ficiency fit when using hyperparameters according to our found dependency B" (a(a), (7),))(() (shown in red). These fits are estimated from o 1,....8 8 and extrapolated to = 0.5. We compare the typical approach of tuning hyper- parameters in online RL, where hyperparameters for one setting of a and this setting is used for all UTD values (shown in blue). We see that our proposed hyper- parameter fits improve results for values other than a 2. Further, this improvement larger for larger values of o, showing that accounting for hyperparameter dependency critical.
2. Estimate empirically the best of batch size B and learning rate r). with statistical bootstrapping.
3. Fit (c) and 17" (a) on B, according to Equa- tions (4.6) and (4.7).
tuned
4. Using the found fits (*)a), (o), run different values of a that cover range spanning an order of magnitude; we use 16x, i.c., / > 16.
5. Fit (a) according to Eq. (4.1).
6. Using fits of (a) for different values of Jo. fit a² (Fo) according to Eq. (4.5).
5. Experimental Details
7. Optimal hyperpurameters can now be extrapo- lated to larger data, larger compute, or larger bud- get settings according to Problem 3.1.
Experimental Setup We focus 12 tasks from 3 do- mains in our study. On OpenAl Gym (Brockman et al., 2016), we use Soft Actor Critic, a commonly used TD- learning algorithm (Haarnoja al., 2018). We first run a sweep on 5 values of 7, then grid of runs with values of and 3 values of B, and then use hyperparameter fits
Having presented solutions to Problems 3.1 and 3.2, we now present the workflow we utilize to estimate these empirical fits. Further details are in Section and Appendix D. This

---

Value-Based Deep RL Scales Predictably
x1e3
X1e3
X1c4
128
Empirica iniirical value
128
Emairical value
100
Dy. Data untily
Held out Fit Dylo)
Dy: Data Jntil)
Ext Extrapalation
Dj: Data until J
Fit 2yl0)
62
64
64
39
Empirical value
24
Ours 2y(0)
Constant fit 2)(0)
32
32
15
0.50
1.00 2.00 4.00 8.00 σ: UTD Ratio
0.50
1.00
2.00 4.00 8.00
0.50
1.00
2.00
4,00
8.00
a: UTD Ratio
a: UTD Ratio
Figure 6: Extrapolation towards unseen values of OpenAI Gym. Left: We show Pareto frontier extrapolation towards higher data regime. Middle: Wc show Pareto frontier extrapolation towards higher compute regime. Right: Wc compare the best-performing hyperparameters (red) for a = 2 to hyperparameters predicted via our proposed workflow (bluc).
to run 2 more value of with seeds per task. To test our approach with larger models, we use DMC (Tassa et al., 2018), where, we utilize the ofate-of-the-ar Bigger, Regu- larized, Optimistic (BRO) algorithm (Nauman et al., 2024b) that uses larger and more modern architecture. We first run 5 values of B, 4 values of 17, and 4 σ; and then use hyperparameters fits to run more values of o, with 10 seeds per task. Finally, we test our approach with more data on IsaacGym (Makoviychuk et al., 2021), where we use the Parallel Q-Learning (PQL) algorithm (Li et al., 2023b), which was designed to leverage massively parallel simu- lation like Isaac Gym that can quickly produce billions of environment samples. Because of computational expense, we only run one IsaacGym task. We first run 4 values of o, 3 values of 1j, as well as values of B, with seeds per task, after which we run a second round of grid search with 7 values of a. Further details are in Appendices B and D and Table 3.
the effect of model size and training data on validation loss, while marginalizing out hyperparameters like batch size (McCandlish et al., 2018) and learning rate (Kaplan et al., 2020). There are several extensions of such scal- ing laws for language models, such as laws for settings with data repetition (Muennighoff et al., 2023) or mixture- of-experts (Ludziejewski et al., 2024), but most focus on cross-entropy loss, with an exception of Gadre et al. (2024), which focuses downstream metrics. While scaling laws have guided supervised learning experiments, little work explores this for RL. The closest works are: Hilton et al. (2023) which fits power laws for on-policy RL methods using model size and the number of environment steps; Springenberg et al. (2024) who study model size scaling for offline RL; Jones (2021) which studies the scaling of AlphaZero on board games of increasing complexity; and Gao et al. (2023) which studies reward model overoptimiza- tion in RLHE In contrast, we are the first ones to study predictability off-policy value-based RI. methods that are trained via TD-learning, Not only do off-policy methods exhibit training dynamics distinct from supervised learn- ing and on-policy methods (Kumar et al., 2022; Lyle et al., 2023), but we show that this distinction also results in a different functional form for scaling law altogether. We also note that while Hilton et al. (2023) pute, i.e., C, in our notation as a metric of performance, our analysis goes further in several respects: (1) we also study the tradcoff between data and compute (Figure 1), (2) we can predict the algorithm configuration for best performance (Problem 3.1); (3) we study many budget functions (C+6-D can be any affine function).
Fitting Functional Forms for Scaling Laws We approxi- mate Eq. (4.1) via brute-force force search followed by LBFG-S with log-MSE loss following (Hoffmann et al., 2023). For Equations (4.6) and (4.7), we fit a line in log space using least squares regression following Kaplan et al. (2020). In our experiments, we run a single fit that is shared across different tasks in given benchmark. Specifically, we share the slope B On, and use task-specific intercepts o o7 (as defined in Equations (4.6) and (4.7)) to be different for separate tasks. This technique is standard in ordinary least squares modeling and is referred to as fixed effect re- gression (Bishop and Nasrabadi, 2006). Sharing this slope serves the goal of variance reduction, which can be impor- tant if the granularity of the grid search over various hyper- parameters run is coarse. More details are in Appendices B and D.
minimal com-
Methods for large-scale deep RL. Recent work has scaled deep RL across three axes: model size (Kumar et al., 2023; Schwarzer et al., 2023; Nauman et al., 2024b), data (Kumar et al., 2023; Gallici et al., 2024; Singla et al., 2024), and UTD (Chen et al., 2020; D' Oro et al., 2023). Naäve scaling of model size or UTD often degrades performance or causes divergence (Nikishin al., 2022; Schwarzer al., 2023), mitigated by classification losses (Kumar et al., 2023), layer normalization (Nauman et al., 2024a), or feature normaliza-
6. Related Work
Scaling laws and predictability. Prior work has studied scaling laws in the context of supervised learning (Kaplan et al., 2020;
Hoffmann et al., 2023), primarily to predict
8

---

Value-Based Deep RL Scales Predictably
tion (Kumar al., 2022). In our work, we use scaled net- work architectures from Nauman et al. (2024b) (Section 5). In on-policy RL, prior works focus on effective learning from parallelized data streams in simulator or world model (Mnih et al., 2016; Silver et al., 2016; Schrittwieser et al., 2020). Follow-up works like IMPALA (Espeholt et al., 2018) and SAPG (Singla et al., 2024) use a central- ized learner that collects experience from distributed work- with importance sampling updates. These works differ substantially from our study as we focus exclusively value-based off-policy RI. algorithms that use TDDHeerning and not on-policy methods. In value-based RL, prior work on data scaling focuses on offline (Yu et al., 2022; Kumar et al., 2023; Park et al., 2024) and multi-task RL (Hafner et al., 2023). In contrast, we study online RL and fit scaling laws answer resource optimization questions.
Our work is only one step in studying scaling laws for value-based RL methods. Further research has the potential to improve our understanding of value-based RL at scale, provide researchers with tools to focus innovation on more important components, and eventually provide guidelines towards scaling value-based RL similarly to scaling enjoyed by other modern deep learning approaches.
A Acknowledgements
We would like to thank Zhang- Ching-Wei  Hong, Amrith Setlur, Rishabh Agarwal, Seohong Park, and Max Simchowitz for feedback on an earlier version of this paper. We would like to thank Andrea Zanette, Seohong Park, Kyle Stachow- icz, and Qiyang Li for informative discussions. This re- search was supported by ONR under N00014-24-12266. N00014-22- 1-2773,, and ONR DURIP grant, with compute support from the Berkeley Research Compute, Polish high- performance computing infrastructure, PLGrid (HPC Cen- ter: ACK Cyfronet AGH), that provided computational re- sources and support under grant no. PLG/2024/017817. Pieter Abbeel holds concurrent appointments as a Professor at UC Berkeley and an Amazon Scholar. This work was done at UC Berkeley and CMU, and is not associated with Amazon.
7. Discussion, Limitations, and Future Work
In this paper, we show that basee deep RI. algorithms scale predictably. We establish relationships between good values of hyperparameters of value-based RL. We then es- tablish a relationship between required data and required compute for a certain performance. Finally, this allows us to determine an optimal allocation of resources either data and compute. Although only estimated from small-scale runs, empirical models reliably extrapolate to large com- pute, data, budget, or performance regimes. Despite folk wisdom to the contrary, we show it is possible to predict behavior of value-based off-policy RL algorithms at larger scale using small-scale experiments.
Impact Statement
This paper aims to contribute to the advancement of re- inforcement learning. While our work may have various societal implications, none warrant specific emphasis here.
At the same time, this first study also presents number of open questions and challenges:
References
1. While simple power law models work well, an open question remains as to whether such laws are theoreti- cally grounded, and whether there are better and more refined functional forms.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint, 2023.
2. Our study only focused on three hyperparameters (B,
n, and g). We do not focus optimal tradeoff between model size and UTD, which is important for compute scaling. For data efficient RI., it is important to ana- lyze the dependency of weight decay and weight reset frequency on UTD, which are typical tricks employed by many of the most performant methods in literature.
Richard E. Barlow and Hugh D) Brunk. The isotonic re- gression problem and its dual. Journal of the American Statistical Association, 1972
Christopher M Bishop and Nasser M Nasrabadi. Pattern Recognition and Machine Learning, Springer, 2006.
3. While we focus on online RI,, it is important to study scaling of offline-to-onlin and offline RL, which will allow direct applications of scaling law findings to large model training.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym, 2016.
201.
4. Finally, while we study relatively small models, future work will focus on verifying our results with larger model scales, larger scale tasks, study the effect of modern architectures, and cover a larger range of com- pute scales spanning multiple orders of magnitude.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
9

---

Value-Based Deep RL Scales Predictably
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross. Randomized ensembled double Q-learning: Learning fast without model. In International Conference Learning Representations, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large lan- guage models. Advances in Neural Information Process- ing Systems, 2023.
Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G Bellemare, and David Silver. The value-improvement path: Towards better representa- tions for reinforcement learning. In AAAI Conference on Artificial Intelligence, 2021.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based pol- icy optimization. In Advances in Neural Information Processing Systems, 2019.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre- Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In International Conference on Learn- ing Representations, 2023.
Andy L. Jones. Scaling scaling laws with board games, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint, 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint, 2024.
Alex Krizhevsky. One weird trick for parallelizing convolu- tional neural networks. arXiv preprint, 2014.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameetriizatoon inhibits data- efficient deep reinforcement learning. In International
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Si- monyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. International Conference Machine Learning, 2018
Conference on Learning Representations, 2021.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron
Courville, George Tucker, and Sergey Levine. DR3:
Value-based deep reinforcement learning requires explicit regularization. International Conference on Learning Representations, 2022.
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint, 2024.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline Q-learnin on diverse multi-task data both scales and generalizes. In Interna- tional Conference on Learning Representations, 2023.
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint, 2024.
Hojoon Lee, Hanseul Cho, Hyunseung Kim, Daehoon Gwak, Joonkee Kim, Jaegul Choo, Young Yun, and Chulhee Yun. Plastic: Improving input and label plastic- ity for sample efficient reinforcement learning. Advances Neural Information Processing Systems, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Con- ference Machine Learning, 2023.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprins, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018.
Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting, In International Conference Learning Representations, 2023a
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timo- thy Lillicrap. Mastering diverse domains through world models. arXiv preprint, 2023.
Zechu Li, Tao Chen, Zhang-Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel Q-learning: Scaling off- off-policy reinforcement learning under massively parallel simula- tion. In International Conference on Machine Learning, 2023b.
Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint, 2023.
10

---

Value-Based Deep RL Scales Predictably
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforce- ment learning. International Conference on Learning Representations, 2016.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre- Lue Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning. 2022.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Ku- mar. Is value learning really the main bottleneck in offline RL? Advances in Neural Information Processing Systems, 2024.
Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michat Krutul. Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóždž, Piotr Sankowski, et al. Scaling laws for fine- ane-ggrined mix- ture of experts. In International Conference on Machine Learning, 2024.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint, 2022.
Clare Lyle. Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, 2023.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering Atari, Go, chess and Shogi by planning with learned model. Nature, 2020.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac Gym: High performance GPU-based physics simulation for robot learning. Advances in Neural Information Pro- cessing Systems Datasets and Benchmarks Track, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, 2017.
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level Atari with human-level efficiency. In International Con- ference on Machine Learning, 2023.
Sam McCandlish, Jared Kaplan, Dario Amodei, and Ope-
nAI Dota Team. An empirical model of large-batch train- ing. arXiv preprint, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, An- drei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep rein- forcement learning. fauure20155
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrit- twieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of Go with deep neural networks and tree search. Nature, 2016
Volodymyr Mnih, Adria Puigdomenech Badia, Mchdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, 2016.
Jayesh Singla, Ananye Agarwal, and Deepak Pathak. SAPG: Split and aggregate policy gradients. International Con- ference on Machine Learning, 2024.
Jost Tobias Springenberg, Abbas Abdolmalcki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Bechtle, Steven Kapturowski, Roland Hafner, al. Offline actor-critic reinforcement learning scales to large models. International Conference on Machine Learning 2024.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 2023.
Michal Nauman, Michal Bortkiewicz, Piotr Miloš, Tomasz Trzeinski, Mateusz Ostaszewski, and Marek Cygan. Over- estimation, overfitting, and plasticity in actor-critie: The bitter lesson of reinforcement learning. In International Conference on Machine Learning, 2024a.
Richard Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefraneq, et al. Deep- Mind control suite. arXiv preprint, 2018.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mitos, and Marek Cygan. Bigger, regularized, optimistic: Scaling for compute and sample-efficient continuous control. Advances in Neural Information Processing Systems, 2024b.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalk- wyk, Andrew M Dai, Anja Hauth, Katie Millican, et al.

---

Value-Based Deep RL Scales Predictably
Gemini: A family of highly capable multimodal models. arXiv preprini, 2023.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Im- pacts, 2020.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 2020.
Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: Tuning large neural networks via zero-shot hyperparame- ter transfer. Advances in Neural Information Processing Systems, 2021.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Haus- man, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. In Inter- national Conference on Machine Learning, 2022.
12

---

Here, solve the compute optimization problem presented in Section We write the Value-Based Deep RL Scales Predictably
Appendices
A. Additional details on derivations
FLOPs calculation. Recall that FLOPs per forward and backward passes are equal to Cyjrward x 2. B(o) a D)(a) and Chackwaad( (o) R 4. Bo) a DDD(()) with o denoting the number of gradient steps per environment steps. Q-learning methods used in our study use MLP and ResNet architectures, which are well modeled with this approximation. Assuming same size for actor and critic an approximation, a training iteration of the critic requires three forward passes and one backward pass, totaling Cj C (ar) 10- N B(o). D)(a). A training iteration of the actor requires backward passes, totaling Cj (a) = 12 N B(o) a Pr(a). Here follow the standard practice of updating the actor every time new data point collected, while the critic is updated according to the UTD ratio σ. Since we expect the critic to be updated more then the actor. As such, in this study we assume
forward and two
C)(a) Crilic (o) 10 N. o
(A.1)
Compute and sample efficiency. Following Eq. (4.1), the number of data points required to achieve performance is
equal to:
Dj(a) ~ Dyii
(A.2)
Given the expressions for required data points, practical batch size, and FLOPs Equations (4.1), (4.6) and (A.1), can now derive the expression for compute required to reach particular performance expressed in terms of a. First, note that the number of parameter updates is
33
DD(() a Dyin aos-l
(A.3)
Combining above, Eq. (4.6) with Eq. (A.1) yields:
C)(0) = 10 N. B(o).
σ D
gaJ-1
BB
3
2 10 N.
σ D
(A.4)
ras-l
Dinin
9 A
~ N N.
008-1
aatta-l
We observe that the resulting expression is a sum of two power laws. In practice, one of the power laws will dominate the expression and simple mental model is that compute increases with UTD as power law with coefficient < (see Figure 2).
Maximal compute efficiency. problem:
(B" (B",n", σ²) arg min C (Ba,o)
J (TAA Jo D S Do-
(A.5)
Firstly, we formulate the Lagrangian L:
13

---

Value-Based Deep RL Scales Predictably
((o, = Cr(α) + +.. (D,(o) - Do) = 1 N B(o). (D. + )
19 A. (Dyin + (B/) ( -Du
(A.6)
Here, the constrained with respect to performance Jo is upheld through the use of (a) and ()(a) which are defined such that J = Jo- We proceed with calculating the derivative with respect to to find the minimal a that able to achieve the desired sample efficiency Dj. We denote such optimal UTD as o²:
2C 0X
[8,
aj
-B,
Dyin
Do =0
0'
(A.7)
(Diinn Doo) 1/45
Then, we substitute the o² into the expression defining compute, as well as use Eq. (4.6);
30 g -1
$9
C)(a") R0 10.1. N.
Dmin
(Dyin Do)
2 10 N
Dyin
(A.8)
(0*))o--
$3
10 N 5g (o (o") 1001 Du
Maximal sample efficiency. Firstly, we note that we treat (or) constant and do not optimize with respect to it. We start with the problem definition:
(B", n", o²) arg min D s.t. J (TAIg 2 Jo A C (B,goo)
(A.9)
Similarly the maximal compute efficiency problem, we formulate the Lagrangian
-(,A))=;((o)) + (C)(a)- - -C)
By aj +Λ
(A.10)
Dijin
+
B(o) a. (Dyin +
-Co)
gas
Again, we uphold the constraint with respect to the performance through the use of DJ(o) and C)( We calculate the derivative with respect to A:
0c 0x
10- - - (Dgin
894
Co =0 ga.
Dysin
39
Co
D,
(A.11)
gas
10 N B(o)
Since D, is monotonic in a and does model impact of B on the sample efficiency, the optimization problem can be solved via Weierstrass extreme value theorem. As such, find the biggest and that fulfills the compute constraint, and find the data requirement for such o.
B. Experimental details
For our experiments, we use a total of 12 tasks from 3 benchmarks (DeepMind Control (Tunyasuvunakool et al., 2020), Isaac Gym (Makoviychuk et al., 2021), and OpenAl Gym (Brockman et al., 2016)). We list all considered tasks in Table
14

---

Value-Based Deep RL Scales Predictably
Table 1: Tasks used in presented experiments.
Domain
Task
Optimal π Returns
DeepMind Control
Cartpole-Swingup
1000
Cheetah-Run
1000
Dog-Stand
1000
Finger-Spin
1000
Humanoid-Stane
1000
Quadruped-We -Wak
1000
Walker- Walk
1000
Isaac Gym
Franka-Puus Push
0.05
OpenAl Gym
HalfCheetab-v4
8500
Walker2d-v4
4500
Ant-v4
6625
Humanoid-v4
6125
Figure We use available UTD values for the fits, which is for DMC, for OAI Gym, and 7 for Isaac Gym. Given the dependency of compute and data on UTD, we plot the resulting curve. We average the data efficiencies across all tasks in each domain, as described in Appendix D. For plots on the left, we use J 800.
We calculate compute given the model sizes of N = 4.92e6 for DMC, N 1.5c5 for OAI Gym, and N 2e6 following standard implementations of the respective algorithms.
For budget extrapolation, we use tradeoff values 7 to mimic the wall-clock time of the algorithm. We use 5 = 1e10 for DMC, 5 = 5e9 for OAI Gym, and d = le4 for Isaac Gym. We exclude runs affected by resets (o = 8) for DMC since the returns right after the reset are lower, which adds noise to the results.
Figure We use the same data as for DMC in Figure I (left).
Figure 3. We use the same data for DMC in Figure (right).
Figure 4. Left: show an illustration that reflects observed empirical results about the dependencies between hyperparameters.
Right, middle: we investigate the correlations between overfitting, parameter norm of the critic network, and a., We observed the same relationships on all tasks. Here, to avoid clutter, we plot tasks from DMC benchmark: cheetah-run, dog-stand, and quadruped-wan walk. To measure overfitting, we compare the TD loss calculated on samples randomly sampled from the buffer (corresponding to training data) TD loss calculated 16 newest transitions (corresponding validation data) according to:
Overfitting = Ttraiiiig - D validaiion
(B.1)
We fit the linear curves using ordinary least squares with mean absolute error loss.
Figure 5. In the left and central Figures, we evaluate the B and 17' models. For each DMC task, we find the best hyperparameters according our workflow and procedure described in Section 5 and Appendix D. While the intercepts vary across environments, for simplicity we plot data points and fits from all environments in the same figure by shifting them with the corresponding intercept. In the right Figure, we marginalize over and visualize best performing pairs of B and n,.
Figure 6. Here, we investigate 4 tasks from OpenAl Gym, listed in Table 1, and compare the extrapolation performance of two hyperparameter sets: the best performing hyperparameters for a = 1, found by testing different hyperparameter values listed in Table 3 (we refer to this configuration as baseline); and hyperparameters predicted by our proposed models of 8" and r7" We fit our models using a E (1, 4, 8), and extrapolate to a (0.5, 16). The graph shows the data efficiency with threshold as 700, normalized according to the procedure in Appendix D.
15

---

Value-Based Deep RL Scales Predictably
The goal of the left Figure is visualize the effects of isotropic regression fit noisy data. We the SciPy package (Virtanen al., 2020) to run the isotropic model. In the right Figure visualize the process of best hyperparameter selection using bootstrapped confidence intervals. We describe the bootstrapping strategy in Appendix D.
Figure 7.
C. Resulting Fits
DMC Refer Table for environment-speciinn values.
n Bn 0.26
B 3g 0.47
D, Dmin 1
-0.74
(C.I)
0..4
o² 1.4c8 Fo
OpenAl Gym
Refer to Table 5 for environmeent-specifie values.
"¹ = 3, 0.30
8 3Bо -0.33
D, = Dmin
1+ O
-0.09
(C.2)
4.02
0' 1.4c8 Fo 0.53
Isaac Gym
-022
17" = 8.77
1 +
2.57e-3
-0.68
B' 38.6
1.42e-2
(C.3)
D = 6.8e7
+
0
-0.87
1.88
o¹ 11.3 Fo
0.57
Table Coefficients for DMC and OpenAl Gym fits.
Domain
Task
BB
Dmin
DMC
cartpole-swingup
7.55e-4
538.2
2.4e4
cheetah-run
6.25e-4
564.9
3.5e5
finger-spin
8.77c-4
608.2
2.9e4
humanoid-stand
3.86e-4
451.8
3.8e5
quadruped-walt
8.46e-4
526.4
6.2e4
walker-walk
9.38e-4
313.3
3.3e4
OpenAI Gym
Ant-v4
1.35e-4
447.0
2.7e5
HalfCheetah-v4
1.86e-3
415.4
7.8e4
Humanoid-v4
1.65e-4
351.6
1.8e5
Walker2d-v4
7.85e-4
399.1
1.7e5
D. Additional details the fitting procedure
Preprocessing return values. In order to estimate the fits from our laws, we need to track the data and compute needed by run to hit a target performance level. Due stochasticity both in training and and evaluation, naive measurements of this
16

---

Value-Based Deep RL Scales Predictably
Table 3: Tested configurations.
Hyperparameters
DeepMind Control
Isaac Gym
OpenAl Gym
Updates-to-data
1, 2, 4, 32, 64, 128, 256, 512 15e-5, 3e-4, 6e-4, 12e-3
1, 4, 8 128, 256, 512
1021 2015: 4096 8192 16381 32768 65536
Batch size B
512, 1024, 2048, 4096, 8192
Learnngg rate ')
le-4, 2e-4, 3e-4
le-4, 2e-4, 5e-4, 1e-3, 2e-3
point can exhibit high variance. This turn would result low-quality fits for D, and C., Thus, we preprocess the return values before estimating the fits by running isotonic regression (Barlow and Brunk, 1972). Isotonic regression transforms return values to the most aligned monotonic sequence of values that then be used to estimate D,, While in general return values can decrease with more training after reaching target value, and this will result in large deviation between the isotonic fit and true return values, the proposed isotonic transformation still suffices for us as our goal is to simply fit the minimum number of samples or compute needed to attain target return. As we can still make reliable predictions that extrapolate to larger scales, the downstream impact of this error is clearly not substantial. We also average across random seeds before running isotonic regression to further reduce noise. We normalize the returns for all environments to be between 0 and 1000 (Table lists pre-normalized returns), and reserve the points of 700 and 800 for budget extrapolation Figure 1,
Uncertainty-adjjuted optimal hyperparameters. While averaging across seeds and applying isotonic regression reduces noise, observe that the granularity of our grid search learning rate and batch size limits the precision of the resulting hyperparameter fits B. rj. Noise due to random seed generation makes hyperparameter selection harder as some hyperparameters that appear empirically optimal might simply be so due to noise. We observe that we can correct for this precision loss by constructing a more precise estimate B, fj adjusted for this uncertainty. Specifically, we run K = 100 bootstrap estimates by sampling n random seeds with replacement out of the original random seeds, applying isotonic regression, and selecting the optimal hyperparameters BR, în. We then use the mean of this bootstrapped estimate to improve the precision:
1 Σ
Buootstrap
BR
K
k
(D.I)
Σ
Thootstrp
K M
k
We have also experimented with more precise laws for learning and batchsize by adding additive offset. In this case, we follow Hoffmann et al. (2023) and fit the data using brute-force search followed by LBFG-S. We use MSE in log space as the error: MSEngg (a, b) = (log log b)².
B" (o) R Bmin +
(D.2)
0n
17" (o) R Mmin
(D.3)
However, we found that this more complex fit did not validate the decrease of degrees of freedom given limited sweep range, resulting in accuracy of extrapolation.
Independence of and 17. Whereas the optimal choice of B and v is often intertwined as UTD changes, we observe in our experiments that the correlation between them is relatively low (Figure 5). we ran a cross-produce grid search with hyperparameter space [B,..., Brg) {.),..., "m,,,, ). we can use this fact to further improve the results by averaging the estimate B over different values of 77. That is, we produce the estimate Bla-n) (respectively BilB-Bii by only looking at the runs where 7 = M) and averaging such estimates.
Brsean
Σ
nh
(D.4)
1
Σ
Amean
nB
17

---

Value-Based Deep RL Scales Predictably
512
f. Return fraction
800
B 3* Best batch size
256
600
128
400
Point estimate
10 seeds
64
Bootstrap
200
Isotonic regression
Mean across f]
Dj: data /ntil
32
Bootstrap mean across n
0
1
12
23
44 D: Data
33
55 x1e4
2
4
8
a: UTD ratio
Figure 7: Left: Determining performance via isotonic regression on DMC. Right: improving hyperparameter selection with uncertainty adjustment DMC. Further details are Appendix D.
Data efficiency. We fit data efficiency of the runs with our found practical hyperparameters B" ny" according to Eq. (4.1). We follow Hoffmann et al. (2023) and fit the data using brute-force search followed by LBFG-S, We use MSE in log space as the error: MSFaeg (a, b) = (log log b)².
In DeepMind Control Suite, we would like to share the data efficiency across differeent enviromments env. We normalize the data efficiency D by the entra-enviromment median data efficiency medians Demd median(Derv (D 1..m,). For interpretability, we further re-normalize D with the overall median Dmed: D = D Dmee/ DDnyy We will need to express data efficiency law alternatively as:
)
(
B3 as
Du(a) = Dipin 1
(D.5)
This is equivalent to Eq. (4.1) because the coefficient 3, absorbs Dysn, However, this expression makes explicit an overall multiplicative offset! Dyrin. Our median normalization is then equivalent fitting per-environment coefficients Di following our procedure for enviromment-shared hyperparameter fits. However, further improve robustness by fixing the per-environment coefficients be the median data efficiency and do not require fitting them.
E. Additional experimental results
Table Correlation coefficients empirically optimal DMC hyperparameters.
R
learning rate and batch size
0.04
batch size and UTD learning rate and UTD
-0.40
-0.46
Table 5: Error of Pareto frontier extrapolation.
R
toward larger compute
7.8%
toward larger data
10.6%
18

---

Value-Based Deep RL Scales Predictably
Cartpole-swingup, utd 8
Cheetah-run, utd 0.25
800
800
600
600
J: Return
J: Return
400
400
Original data
Original data
200
200
Gaussian smoothing
Gaussian smoothing
Isotonic smoothing
Isotonic smoothing
0
0
0
1
2
3
xle4
0
2
4
6
8
xle5
D: Data
D: Data
Figure 8: Another example of isotonic regression. Using gaussian smoothing with variance undersmoothing (left).
3 leads both oversmoothing (right) and
120
300
128
400
128
Empirical value Dyjo
Empirical value Dyle)
rical value
Dy. Data Jntil
64
Dr: Data until
G4
Dj: Data J
64
zyjo)
32
32
16
16
16
0.50
1.00 2.00 4.00 8.00
0.50
1.00 2.00 4.00 8.00 UTD Ratio
0.50
1.00 2.00 4.00 8.00 UTD Raliu
a: UTD Ralio
X1e3
- 600
x1e3
-= 700
x1e3
J 800
128
128
Empirical value
Empirical value
128
Emp rical value
Dj: Data untily
Fit Dyoo
Dj: Data /
Fit Dylo)
Dc Data u
Fit D/oo
64
64
64
32
32
32
16
16
16
0.50
1.00 2.00 4,00 8.00
0.50
1.00 2.00 4.00 8.00
0.50 1.00 2.00 4.00 8.00 ar: UTD Ratio
a: UTD Ratio
a: UTD Ratio
Figure 9: Additional fit results OpenAl gym for different values of J.
19

---

Value-Based Deep RL Scales Predictably
HalfCheetah-v4
Walker2d-v4
Humanoid-ve
4x19
10
10*
10
OpenAl Gym
3X10
4x
10a
102
6x100
x1e6
xled
Xled
Xled
Environment Steps
Environment Steps
Enviroommnnt Steps
Environment Steps
cart partpole-swingee
cheetah-run
dog-stand
finger-spin
10.
101
DeepMind Control
109
X385
0.0
0.2 Environment Steps
X3e5
xtes
0.0
0.2
xted
Environment Steps
Environment Steps
Environment Steps
humanoid-stee -tandd
quadruped walk
walker-walk
Bot
utd=1.0
10a
utd=2.0
10a
10a
utd=4.0
utd-8.0
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.4 Environment Steps
0.2
Environment
Environment Steps
Figure 10: An approximation of the critical batch size over training. Further details are in Appendix F.
F. Critical batch size analysis
Previous work has argued that there is a critical batch size Bert for neural network training in image classification, generative modeling, and reinforcement learning with policy gradient algorithms (McCandlish al., 2018) - transition point at which increasing the batch size begins yield diminishing returns. We follow this work and compute estimate of the gradient noise scale Buoise Ben according to the following procedure: throughout training, compute the gradient norm |G of the critic network for batches of size B Bamall == 64 and B Beig 1024. Then, we evaluate
1 = Buig B
BBig G Bus
2 B G B (?)
S 1/ B/B
(IG Benti
12 |G Bayg 12)
1/ Buig
and take Bun S/19/3. In practice, to account for the noisiness of |G|2, we first take rolling averages of |G Bannl and P.. over training, and tune the window size so that the estimates for |g|2 and S are stable.
We show the values of Bun over training in Figure 10. Unlike policy gradient methods, we find that the critical batch size (averaged over training) has little correlation with the optimal batch size, as shown in Figure 11.
"This form enforces Diii positive.
20

---

Value-Based Deep RL Scales Predictably
hullhhetth Joyni
500
Walkerdded [Cyml
Humanold (aytt)
spppe--senggpp (DWO)
Biiaal
cheeton run (DMC)
sland (DMCI
firgerspin
100
lumandid stand (IMG)
wadruseshhal (DFVC)
woltewwak (DMK)
UTD=1
UTD-2
30
100
y-
Brit
Figure 11: Bana Ban grouped by task and UTD.
Table 6: Batch size values predicted by the proposed model on DMC.
Task
0.25
0.5
0
0 4
8
cartpole-swingup
1040
752
544
384
288
208
cheetah-run
1088
784
560
400
288
208
dog-stand
240
176
128
96
64
48
finger-spin
1168
848
608
432
320
224
humanoid-stane
864
624
448
320
240
176
quadruped-ve walk
1008
736
528
384
272
192
walker-wall
608
432
320
224
160
112
Table 7: Learning rate values predicted by the proposed model on DMC.
Task
o 0.25
σ 0.5
a= 1
a = 2
4
= 8
cartpole-swingup
,00108
.000902
.000755
.000631
.000528
.000442
cheetah-run
.000893
.000747
.000625
.000523
.000438
.000366
dog-stand
.000664
.000555
.000465
.000389
.000225
.000272
finger-spin
.00125
.00105
.000877
.000734
.000614
0000514
humanoid-stand
.000551
.000461
.000386
.000323
.00027
.000226
quadruped-wad
.00121
.00101
.000846
.000708
.000592
.000996
walker-walk
.00134
,00112
.000938
.000785
_000657
.000549
Table 8: Batch size values predicted by the proposed model on OpenAl Gym.
Task
= 0.25
=
0.5
0
1
2
a
8
16
Ant-v4
704
560
448
352
288
224
176
HalfCheetah-v4
672
528
416
336
256
208
160
Humanoid-v4
560
432
352
272
224
176
144
Walker2d-v4
640
496
400
320
256
192
160
Table 9: Learning rate values predicted by the proposed model on OpenAl Gym.
Task
a 0.25
a 0.5
2
a 16
Ant-v4
.000206
.000167
.000138
.000109
.000087
.000070
.000060
HalfCheetah-v4
.002820
.002280
.001900
.001510
.001210
.000972
.000827
Humanoid-v v4
.000251
.000203
.000169
.000134
.000107
.000886
.000073
Walker2d-v4
.001180
.000958
.000806
.000640
.000512
.000412
.000347
21

---

Value-Based Deep RL Scales Predictably
Table 10: Batch size values predicted by the proposed model on IsaacGym.
Task
65536
32768
16384
8192
4096
2048
1024
Franka-Push
7927
5105
3234
2030
1269
791
493
Table 11: Learning rate values predicted by the proposed model on IsaacGym.
Task
65536
32765
16384
8192
1096
2048
1024
Franka-Push 0.000317
0.000265
0.000221
0.000185
0.000154
0.000129
0.000107
22


Reviews:
[{"review_id": "qN1t2Dk4sg", "reviewer": "Reviewer_VoCp", "summary": "This paper investigates the scalability and predictability of value-based RL using TD learning. It establishes predictable, hypothetic equations between three key hyperparameters (batch size, learning rate, and UTD ratio) and shows that data and compute requirements for a given performance lie on a Pareto frontier. By modeling the tradeoff between data and compute, the authors predict resource needs and optimal hyperparameter settings for large-scale experiments based on low-budget data. Finally, they empirically demonstrate that such findings can be extended to algorithms like SAC, BRO, and PQL and domains such as DeepMind Control, OpenAI Gym, and IsaacGym.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for valuable feedback regarding our work. In accordance with your suggestions, we have made a number of changes to our manuscript. These include expanded discussion of limitations and experimental design. We discuss these in detail below:\n\n> (other parameters such as model size, weight decay, and target network update rates are also known to influence training stability and performance in deep RL\n\nWhile studying a number of these factors is definitely along our vision of better understanding the behavior of scaling value-based RL algorithms along various axes, we were faced with a decision of only studying certain axes for the submission due to a limited compute budget. Therefore, we chose to initiate our study into this line of work with the most fundamental hyperparameters such as batch size, learning rate, and UTD values. We are now working to extend this line of work to study the impact of model size and target network update rates as the next piece of work along this research vision. We will also add the following discussion to limitations of this paper:\n\n_Whereas this study investigated the relationship between learning rates, batch sizes, and UTDs, there are a variety of other design choices and hyperparameters that could potentially impact the scaling profile of a given algorithm. In particular, previous works have shown that model size, weight decay, and target network update strategy can significantly affect the training dynamics between different UTD values._ \n\n> It was surprising that the compute-data pareto frontiers (Figure 1, left) for OpenAI Gym seems to be much flatter than other environments. Do the authors have any intuition about this phenomenon?\n\nIndeed, it seems that the effectiveness of UTD is benchmark-specific, where increased UTD leads to pronounced sample efficiency improvements in some environments, and substantially less pronounced in others. We believe that studying the underlying phenomena that lead to such differences is an exciting avenue for future research. \n\n>  (...) The rationale for choosing SAC, BRO, and PQL is weak and not fully justified. SimBa [1] offers a more practical and performant alternative…\n\nWhile in general, one can study scaling laws for any value-based RL algorithm, we were faced with the tough choice of only studying a few algorithms due to limited compute. We chose SAC, BRO, and PQL because prior work has shown them to be effective in the respective benchmarks (Gym, DMC, Isaac Gym) under different UTD configurations. We will add this justification to the paper.\n\nOur framework is independent and can be applied to both BRO and SimBa, which are both methods focused on scaling the number of parameters. We observed consistent findings across a wide class of SAC-derived methods and therefore believe the results would also extend to SimBa. Due to compute constraints, we were unable to run SimBa experiments for rebuttal.\n\nWe also note that our goal is not to argue that our scaling laws give rise to the best possible value-based RL algorithm or that we are able to attain the best possible results in our experiments, but rather our point is to simply show that we can predict the behavior of value-based RL in larger-scale settings using small-scale experiments. We believe that showing this for a certain class of algorithms is still valuable as a starting step in this direction of research. We believe future state of the art methods will use our framework to improve performance, closing the loop.\n\n> Additionally, the paper does not compare against other well-established state-of-the-art methods such as MR.Q\n\nThank you for pointing us towards that work, which we now cite. We agree that studying the scaling capabilities of this general algorithm would be particularly interesting, however we were not able to include MrQ in our submission as it was arXived after the ICML deadline.\n\n> However, more works on scaling model size [1] and UTD ratio [2, 3] in RL could be included:\n\nThank you for pointing these missing references, we added all of them to our manuscript.\n\nBeyond the changes described above, we add a number of improvements suggested by other reviewers (e.g., additional quantitative analysis of proposed scaling curves, estimation of scaling laws for different levels of J), which we invite you to inspect under the following link: https://sites.google.com/view/value-based-rl-scales/ and have released the code here: https://colab.research.google.com/drive/1BaqvAMb6svGojAuiOV8qFAUrZQwfPlDg?usp=sharing. We hope that these changes increase the reviewers' confidence in our work. If so, we kindly ask to consider updating the score of our work.", "claims_and_evidence": "The authors make several key claims regarding the predictability of scaling in value-based RL, which are well-supported, as the authors provide empirical fits, validate their model across different datasets, and show that extends to well-estabilished algorithms like SAC, BRO, and PQL.", "methods_and_evaluation_criteria": "The method for proposing their claim and empirical relationships are well-constructed and appropriate for the problem. Their study clearly formulates and supports the predictability of value-based RL scaling.", "theoretical_claims": "No theoretical claims were made.", "experimental_designs_or_analyses": "The experimental setup is generally well-structured, but a key concern is the limited selection of baseline algorithms. The rationale for choosing SAC, BRO, and PQL is weak and not fully justified. For example, I don't think BRO is the state-of-the-art algorithms in Gym; SimBa [1] offers a more practical and performant alternative by simply modifying SAC’s architecture, which could help assess the impact of network design on the proposed predictability framework. Additionally, the paper does not compare against other well-established state-of-the-art methods such as MR.Q [2]. I believe resolving my concern would enhance the evaluation and strengthen the study’s claims.\n\n[1] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning., ICLR'25.\n\n[2] Towards General-Purpose Model-Free Reinforcement Learning., ICLR'25.", "relation_to_broader_scientific_literature": "This paper extends scaling law research from supervised learning to value-based reinforcement learning (RL), providing a contribution towards a large-scale RL foundation model.", "essential_references_not_discussed": "I think most references in mind were present. However, more works on scaling model size [1] and UTD ratio [2, 3] in RL could be included:\n\n1. SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning., ICLR'25.\n2. DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization., ICLR'24.\n3. The Dormant Neuron Phenomenon in Deep Reinforcement Learning., ICML'23", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "n/a", "questions_for_authors": "1. The study establishes predictable scaling laws for batch size, learning rate, and the UTD ratio, which are well-supported by empirical evidence. However, other parameters such as model size, weight decay, and target network update rates are also known to influence training stability and performance in deep RL. I wonder if the authors have considered these configurations too. \n2. It was surprising that the compute-data pareto frontiers (Figure 1, left) for OpenAI Gym seems to be much flatter than other environments. Do the authors have any intuition about this phenomenon?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n/a"}}, {"review_id": "YDTDyqt4oi", "reviewer": "Reviewer_beKg", "summary": "The authors demonstrate that value-based deep RL scales predictably, showing a Pareto frontier controlled by the updates-to-data (UTD) ratio. This paper shows how the optimal hyperparameters can be predicted from low-cost experiments, enabling an extrapolation to higher data or compute experiments. Validation is done using algorithms like SAC, BRO, and PQL across multiple benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and the positive feedback. Please let us know if the response below addresses your concerns, and if there are any concerns remaining. \n\n> Although the evaluation criteria is fitting for the problem at hand, I would have liked to see some experiments on harder, pixel-based environments such as Atari. It would be interesting to see if these scaling laws hold with increased environment or network complexity\n\nOur research vision is to understand the impact of various design choices on scaling of value-based RL algorithms. This includes a number of choices such as the choice of environments, networks, replay buffers, algorithms, etc, but unfortunately due to the limited compute budget we were left with the choice of only being able to study a few design choices only that could fit in the scope of the first paper along this direction. Therefore, we decided to first start from environments that are cheap to run (state-based) and considered different network architectures including small vanilla MLPs (used by SAC and PQL) and relatively big resnet-based models (BRO) in their specific domains. \n\nWe agree that expanding to pixel-based environments would further strengthen the generalizability claim. While this is costly to do, we are happy to add preliminary results with a subset of Atari environments in the camera-ready version. \n\nIn addition, in the rebuttal we have added a number of improvements suggested by other reviewers (e.g., additional quantitative analysis of proposed scaling curves, estimation of scaling laws for different levels of J), which we invite you to inspect under the following link: https://sites.google.com/view/value-based-rl-scales/ and have released the code here: https://colab.research.google.com/drive/1BaqvAMb6svGojAuiOV8qFAUrZQwfPlDg?usp=sharing. We hope that these changes increase the reviewers' confidence in our work.", "claims_and_evidence": "The claims are supported by a large amount of empirical evidence. The scale of the experiments lead to convincing results.", "methods_and_evaluation_criteria": "Although the evaluation criteria is fitting for the problem at hand, I would have liked to see some experiments on harder, pixel-based environments such as Atari. It would be interesting to see if these scaling laws hold with increased environment or network complexity.", "theoretical_claims": "The claims are empirically derived. However, the derivations seem sound.", "experimental_designs_or_analyses": "-", "relation_to_broader_scientific_literature": "The biggest improvement over broader scientific literature is that the authors also study the tradeoff between available data and compute.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "Strengths: Clear, practically helpful paper with a lot of empirical evidence.\n\nWeaknesses: I will not claim that the experiments are limited, but, as said before, I would have liked to see some harder, pixel-based environments.", "other_comments_or_suggestions": "I would like to see some experiments on a more complex environment such as Atari. Although I know this might seem compute intensive, the recent PQN [1] algorithm seems to be very fast.\n\n\n[1] Gallici et al, 2024. Simplifying Deep Temporal Difference Learning.", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jK14Cu7FT2", "reviewer": "Reviewer_sgc1", "summary": "The main claim of the paper is that it is possible to predict optimal hyperparameters, data quantity and compute allocation for high budget from low-budget experiments. This is broken down into the following sub-claims: \n\n1. The amount of data needed for a given performance is predictable as a function of the UTD according to power law defined in Equation 4.1.\n2. The amount of data needed for a given performance is predictable as a function of the UTD according to power law defined in Equation 4.2.\n3. For a given desired performance and budget, there exists a predictable “best” UTD that follows a power law in Equation 4.5. This relationship extrapolates to larger budgets. \n4. The optimal choice of batch size and learning rate are predictable functions of the UTD.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and feedback! We have added several new results & clarifications to address the concerns. Specifically, for the main points:\n- We have performed the requested additional analysis to improve clarity here: https://sites.google.com/view/value-based-rl-scales/\n- We eliminated inconsistencies in evaluation protocol and report the additional results on the same link\n- To ensure transparency and reproducibility we have released the code: https://colab.research.google.com/drive/1BaqvAMb6svGojAuiOV8qFAUrZQwfPlDg?usp=sharing. \n\nWe think that these changes substantially strengthen the paper and are glad that we concur on the vision of understanding how to predictably scale value-based RL. However, we also think some of the comments might stem from a misunderstanding of certain parts of the paper, and we attempt to clarify these below. We will update the paper to clarify these. Please let us know if you find your concerns addressed and if so, we would be glad if you are willing to raise your score.\n\n> what J  is \n\nAs specified on Line 139, J denotes the return of the policy normalized between 0 and 1000 (with normalization detailed in Table 1).\n\n> unclear whether the result holds for all values of J\n\nIt does: https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.mlu2zpb9cilb\n\n> hand-picked values of J, inconsistent number of observations  \nWe standardized the number of observations in this plot to be 10 and standardized the values of J to be equally spaced in log space: https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.2u2k4w2tndmu.  \n\n> It is unclear how accurate the result is.\n\nWe provide an additional extrapolation result where we use our model to predict the data/compute required to reach J (https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.erwyxjer0f42).\nThe error is only 7.8% and 10.6% for extrapolating toward larger data and larger compute, respectively. \n\n>  y-axis follows an unknown exponential scheme\n\nFollowing prior work (Kaplan’20), y-axis follows a logarithmic scale. To improve readability, we also created a version that uses logarithm with base 2 (https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.8e53cf8zcw66). \n\n> unclear how the result was averaged (...), unclear whether the result holds over individual environments\n\nWe shared fits across several environments to increase the generality and robustness of the results (paragraph in line 391 and Appendices B & D). On IsaacGym, we also show that it is possible to obtain a good fit on a single task.\n\n>  challenges associated with offline RL and high UTDs \n\nFollowing DroQ, we use LayerNorm in our SAC implementation and BRO uses LayerNorm and resets by default. This allows higher utd. We further add a new experiment with high UTD of 64: https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.mo01yp5qtkz0. Here, we use the existing hyperparameter fits based on utd 0.25...8, extrapolating hyperparameters far beyond that, and we find that the data requirement fit is still accurate.\n\n> It is unclear how accurate the relationship is between UTD and best batch size/learning rate.\n\nWe provide quantitative correlations here https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.9t39ptle0tkt. We also provide quantitative evaluation of a baseline approach that uses hyperparameters optimal for UTD=2 to extrapolate to larger UTD. \n\n> It is unclear how the compute is affected by the unknown variable quantity B(σ). It is unclear how the best batch size or learning rate is determined over a collection of environments.\n\nBest choice batch size is provided in Eq (4.6). The workflow used to obtain the value is provided in section 4.3. We use different hyperparameters per environment.\n\n> It is unclear how these results [for Eq 4.5] are generated.\n\nThe description the reviewer provides is correct. Please also refer to Section 4.3 for exact workflow.\n\n> How are empirical values [for Eq 4.5] averaged over a collection of environments?\n\nThis is described in l864. We normalize the data requirements by per-environment median.\n\n> Figures 3 and 1 (right) do not directly show UTD.\n\nPlease refer to the corresponding points in Figure 2.\n\n> The line of best fit is based on points sampled on another line of best fit (each pareto frontier), rather than empirical samples.\n\nIt is not possible to use empirical samples since the grid search includes only a small number of UTDs. Estimating scaling laws, however, enables us to optimize hyperparameters with higher precision than the grid search, similar to LLM literature (Kaplan’20, Dubey’24). \n\n> There are an inconsistent number of seeds used.\n\nWe used the maximum number of seeds that was feasible given our compute limitations. Different algorithms allow different number of seeds to be run in parallel.", "claims_and_evidence": "**1. The amount of data needed for a given performance is predictable as a function of the UTD according to power law defined in Equation 4.2.**\n\nIntuitively, this first claim is mostly believable. The literature contains many examples that show there are sample efficiency benefits to higher UTD ratios, with diminishing returns when properly regularized. To defend this claim, the authors present Figure 2 (left). I do not find this figure sufficient to defend the claim for the following reasons:\n1. It is unclear what $J$ is. \n2. It is unclear whether the result holds for all values of $J$.\n3. It is unclear how accurate the result is outside of visual inspection.\n4. The scale of the y-axis follows an unknown exponential scheme that makes visual inspection difficult. \n5. It is unclear how the result was averaged over the collection of DMC environments.\n6. It is unclear whether the result holds over individual environments, or only the average. \n\nFurthermore, I’m not sure I buy that the result necessarily extends to some asymptote. For example, the challenges associated with offline RL and high UTDs have been well-documented [Kumar 2019, Levine 2020, Li 2023], requiring specific algorithmic modifications. If we were to push the UTD to some extreme values, we should expect to see either (1) collapse/divergence, where an algorithm never achieves a desired performance or (2) increasing regularization/algorithmic modifications (e.g., frequent resets, policy regularization) that would also impact the quantity of data required. \n \nThere also appears to be some missing experimental details. Knowing that most methods do not scale naively to higher UTD values [Chen 2021, Li 2023, D’Oro 2023], what modifications were used with the base algorithms to obtain these results? \n\n---\n\n**2. The amount of compute needed for a given performance is predictable as a function of the UTD according to a sum of power laws in Equation 4.2.**\n\nSimilarly, this claim is believable based on prior results in the literature. To defend this claim, the authors present Figure 2 (right). I do not find this claim to be adequately defended. On top of my concerns mentioned above for Claim 1, \n- It is unclear how the compute is affected by the unknown variable quantity $B(\\sigma)$ (“best choice” batch size for each UTD).\n\nAs an aside, it is unclear to me where the “sum” of power laws appears in Equation 4.2, as it consistently solely terms multiplied together. \n\n---\n\n**3. For a given desired performance and budget, there exists a predictable “best” UTD that follows a power law in Equation 4.5. This relationship extrapolates to larger budgets.**\n\nTo defend this claim, the authors reference Figure 3 and Figure 1 (right). Again, I do not find that these figures provide sufficient evidence to defend the authors’ claims. \n- It is unclear how these results are generated. What is the process for determining data points? For example, the way I would expect a data point to be produced would be the following (this is my best guess, because the process is not documented). \n    1. Pick an environment and a UTD value. \n    2. Determine the best hyperparameters for this UTD value. \n    3. Run the algorithm until some performance J is obtained.\n    4. Report the (compute, data).\n\nHowever, in this process, neither compute, nor data is a fixed quantity. How are empirical values averaged over a collection of environments? \n\nThis is highly exacerbated by the fact that different environments learn at very different rates. For example, for the OpenAI gym environments, the authors use SAC but only include 4 out of the 5 environments used by the original SAC paper (Table 1). Furthermore, the values of J seem arbitrarily selected. Based on the original SAC paper [Haarnoja, 2018], we can see that a score of 8500 on HalfCheetah is obtainable in fewer than 1M timesteps, but a score of 6625 on Ant would take more than 3M timesteps. Looking at more modern results, the authors of Cross-Q [Bhatt, 2024] report SAC results, where a score > 6000 is not obtained before 3M timesteps on Ant, but a score of 8500 on HalfCheetah is obtained in fewer than 1M timesteps. \n\n- Figures 3 and 1 (right) do not directly show UTD. While each of these points corresponding to optimal budget do correspond to some value of UTD, we do not know what that UTD value is, and obtaining this UTD value has been obfuscated by an unknown “best choice” batch size that varies between UTD values. We cannot use this information to verify Equation 4.5, nor the claim that this power law extrapolates. \n\n---\n\nContinued in \"other strengths and weaknesses\" for Claim 4, due to a lack of space.", "methods_and_evaluation_criteria": "The evaluation criteria appears to be visual inspection. I do not find this sufficient.\n\nI find the scope of benchmarks (3 domains) to be sufficient. I do think more than one algorithm per benchmark is necessary when attempting to prescribe sweeping statements.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "It is unclear whether the experimental design is valid because they are not adequately described. See my issues with the claims made by the paper, and the lack of scientific rigor in the presentation of the results.", "relation_to_broader_scientific_literature": "The contributions, if correct, provide an interesting use case for hyperparameter selection for large-scale experiments. I am not aware of any work in the RL space with similar observations, which would make this a valuable contribution as the community scales RL algorithms and applications.", "essential_references_not_discussed": "No issues.", "other_strengths_and_weaknesses": "Missing comments on Claim 4:\n\n**4. The optimal choice of batch size and learning rate are predictable functions of the UTD.**\n\nThe authors defend this claim with Figure 4. Figure 4 suffers from the same lack of scientific rigour as many of the previous figures.\n\n1. It is unclear how the best batch size or learning rate is determined over a collection of environments.\n2. It is unclear if the results hold over different domains and algorithms.\n3. It is unclear if the results hold over individual environments.\n4. It is unclear how accurate the relationship is between UTD and best batch size/learning rate outside of visual inspection. \n5. It is unclear if the results hold over different target performance levels or budgets. \n\nWhile I do find the results are more plainly presented than previous figures, and the trends appear to be more clear, I am not confident that there is sufficient evidence to claim the existence of a widely-applicable power law.\n\n---\n\nI have additional concerns about the scientific rigor in Figure 1.\n1. For the middle row (Budget extrapolation), there are an inconsistent number of points across environments. \n2. Given this inconsistency, it’s unclear how the target performance J was determined for each point. \n3. The exponential scale, and use of linear extrapolation makes the relationship appear linear at first glance, but the relationships in the top and bottom row are clearly non-linear.\n4. For the right column, the line of best fit is based on points sampled on another line of best fit (each pareto frontier), rather than empirical samples. \n5. There are an inconsistent number of seeds used. Standard error is not reported. \n\n---\n\nReferences:\n\n- Kumar, Aviral, et al. \"Stabilizing off-policy q-learning via bootstrapping error reduction.\" Advances in neural information processing systems 32, 2019. \n- Levine, Sergey, et al. \"Offline reinforcement learning: Tutorial, review, and perspectives on open problems.\" arXiv preprint arXiv:2005.01643, 2020.\n- Li, Qiyang, et al. \"Efficient Deep Reinforcement Learning Requires Regulating Overfitting.\" The Eleventh International Conference on Learning Representations, 2023.\n- Chen, Xinyue, et al. \"Randomized Ensembled Double Q-Learning: Learning Fast Without a Model.\" International Conference on Learning Representation, 2021. \n- D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P. L., Bellemare, M. G., & Courville, A. Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier. In The Eleventh International Conference on Learning Representations, 2023.\n- Haarnoja, Tuomas, et al. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\" International conference on machine learning. PMLR, 2018.\n- Bhatt, A., et al. \"CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity.\" International Conference on Learning Representations (ICLR). 2024.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "I do not think the scope of an ICML rebuttal would convince me to adjust my evaluation given the current critical lack of scientific rigor in the paper.\n\nFor future revisions, the authors should aim to provide verifiable evidence of their claims. \n- Transparent experimental details to make the paper reproducible. \n- More complete and clearer presentation of the results, such as numerical analysis (including measures of uncertainty or standard error) and results on individual environments, and improving transparency on unknown quantities (e.g., variable \"best batch size\"). \n- Eliminate inconsistencies in evaluation protocol, removing elements such as: hand-picked values of J, inconsistent number of observations, inconsistent exponential axis scales.\n\n----\n\n## update after rebuttal\n\nWhile I appreciate the authors attempts to improve the clarity and transparency of the results, the number of concerns I had in the original draft, on top of a number of unaddressed concerns, means I remain firm with my original score.\n\n**Quantitative analysis:** The paper makes significant claims based only on visual inspection. The authors have not addressed this issue during the rebuttal. \n\n**Arbitrary normalizing:** The authors use an unknown system for selecting environment scores for normalizing that may influence final results. Quote from original review below:\n\n> This is highly exacerbated by the fact that different environments learn at very different rates. For example, for the OpenAI gym environments, the authors use SAC but only include 4 out of the 5 environments used by the original SAC paper (Table 1). Furthermore, the values of J seem arbitrarily selected. Based on the original SAC paper [Haarnoja, 2018], we can see that a score of 8500 on HalfCheetah is obtainable in fewer than 1M timesteps, but a score of 6625 on Ant would take more than 3M timesteps. Looking at more modern results, the authors of Cross-Q [Bhatt, 2024] report SAC results, where a score > 6000 is not obtained before 3M timesteps on Ant, but a score of 8500 on HalfCheetah is obtained in fewer than 1M timesteps.\n\n**Poor clarity or missing details:** Numerous issues were detailed in the original review. For example, in Figure 2, the value of J is not described. In response, the authors mention that this value is described by a color chart in Figure 1. I do not think this is adequate organization or clarity (the reader must infer the value of J in Figure 2, by looking at Figure 1 and making a guess based on a color).\n\n**Missing experimental analysis:** The authors show that their power law provide good fits on the single environment in Isaac gym, but do not include results for individual environments in the other domains. If the results depend on aggregation across environments, and the aggregation is based on an arbitrary normalization, then I have concerns that these results are cherry-picked. I hope to see results that show otherwise. \n\nI hope these concerns are addressed in the next draft.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J0xk1nDFsf", "reviewer": "Reviewer_vVTn", "summary": "This paper studies the scaling law in value-based RL methods. In particular, it provides a thorough analysis on how different components, such as batch size and number of gradient steps, affect the performance and computation budget.\n\n***\n## update after rebuttal\n\nI have read the rebuttal as well as other responses. I will keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback and a positive review of the paper. We are glad that you find our evidence clear and convincing. We answer your questions below. Please let us know if you find your concerns addressed, and if so we would be grateful if you would be willing to raise your score. We are happy to address any remaining concerns.\n\n> What is the wall-time efficiency of value-based algorithms in terms of scalability? For example, How long does the SAC algorithm take to complete a 5M environment-step training in DMC?\n\nThe wall-clock efficiency of these algorithms is highly dependent on the implementation of both the agent and simulator, as well as the computing infrastructure. For an NVIDIA A100 GPU, UTD=1 configuration, and DeepMind Control Suite of tasks (which present a CPU-based simulator), training for 5M environment steps should take around 12 hours. In our study, we model compute by making it proportional to the product of the model size, training length, UTD, and batch size (equation 4.2 of our manuscript) akin to how we measure FLOPs for language model training. As such, our compute metric is highly correlated to actual wall clock requirements. To avoid confusion, we will add an explanation to our compute metric definition where we discuss its relation to wall clock requirements, as well as add wall clock graphs of our tested algorithms to the appendix. Thank you for the suggestion! \n\n> Does it improve the memory usage when using the optimal hyperparameter\n\nOur study uses a fixed memory size between all variants of algorithms (i.e. we use a fixed model and replay buffer sizes). As such, we left the study of memory/scaling interaction for future work. We hypothesize that the optimal or “good enough” hyperparameters indeed improve the memory usage of the algorithm, in a sense that it allows for training a successful agent with either less data or less parameters. \n\nBeyond the changes described above, we add a number of improvements suggested by other reviewers (e.g., additional quantitative analysis of proposed scaling curves, estimation of scaling laws for different levels of J), which we invite you to inspect under the following link: https://sites.google.com/view/value-based-rl-scales/ and have released the code here: https://colab.research.google.com/drive/1BaqvAMb6svGojAuiOV8qFAUrZQwfPlDg?usp=sharing. We hope that these changes increase the reviewers' confidence in our work. If so, we kindly ask to consider updating the score of our work.", "claims_and_evidence": "Claims made in the paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "This paper has made extensive evaluation in both theoretical and empirical aspects.", "theoretical_claims": "There is no proof in the paper.", "experimental_designs_or_analyses": "Extensive empirical results are presented.", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "Essential references are adequately discussed.", "other_strengths_and_weaknesses": "+ I believe this work tries to handle a very important problem. Scalability is perhaps the biggest challenge in off-policy algorithms, and this study attempts to demonstrate that they can still be scaled effectively. The methodology used to identify the optimal hyperparameter settings is also interesting.\n\n+ In practice, off-policy algorithms are considered to have poor scalability, not only due to the large batch sizes and excessive gradient steps but also because they require a buffer to store all the data collected so far, which leads to high memory usage. Additionally, the wall-time efficiency of off-policy methods is poor, as they often take much longer to train an agent compared to on-policy algorithms. Therefore, it would be more convincing if it could be demonstrated that value-based methods can scale in terms of both memory usage and wall-time efficiency.", "other_comments_or_suggestions": "Please see other parts of the review.", "questions_for_authors": "+ What is the wall-time efficiency of value-based algorithms in terms of scalability? For example, How long does the SAC algorithm take to complete a 5M environment-step training in DMC?\n\n+ Does it improve the memory usage when using the optimal hyperparameter?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper examines the scalability and predictability of value-based RL through extensive empirical experiments.

During the review stage, the paper received conflicting comments. While reviewers concur that the problem is significant and the observations and results presented in the paper are valuable, the primary concern lies in transparency and reproducibility. Some of the experiments appear to be cherry - picked. I suggest that the authors include all experiments in the revised version and conduct a failure case study. In my opinion, a failure case study does not diminish the value of a research paper; rather, it enhances it. Overall, I would recommend accepting the paper if the authors can take these suggestions into account.
Decision: Accept (poster)