Compute-Optimal Scaling for Value-Based Deep RL
Authors: Preston Fu
Oleh Rybkin
Zhiyuan Zhou
Michal Nauman
Pieter Abbeel
Sergey Levine
Aviral Kumar
Venue: NeurIPS 2025 poster ()
Keywords: scaling laws
online reinforcement learning
q-learning
Primary area: reinforcement_learning

Abstract: As models grow larger and training them becomes expensive, it becomes increasingly important to scale training recipes not just to larger models and more data, but to do so in a compute-optimal manner that extracts maximal performance per unit of compute. While such scaling has been well studied for language modeling, reinforcement learning (RL) has received less attention in this regard. In this paper, we investigate compute scaling for online, value-based deep RL. These methods present two primary axes for compute allocation: model capacity and the update-to-data (UTD) ratio. Given a fixed compute budget, we ask: how should resources be partitioned across these axes to maximize data efficiency? Our analysis reveals a nuanced interplay between model size, batch size, and UTD. In particular, we identify a phenomenon we call TD-overfitting: increasing the batch quickly harms Q-function accuracy for small models, but this effect is absent in large models, enabling effective use of large batch size at scale. We provide a mental model for understanding this phenomenon and build  guidelines for choosing batch size and UTD to optimize compute usage. Our findings provide a grounded starting point for compute-optimal scaling in deep RL, mirroring studies in supervised learning but adapted to TD learning. Project page: https://value-scaling.github.io/.

Compute-Optimal Scaling for Value-Based Deep RL
Preston Fu'"
Oleh Rybkin'
Zhiyman Zhou'
Michal Nauman 1.2
Pieter Abbeel
Sergey Levine'
Aviral Kumar
UC Berkeley University of Warsaw CCrnegie Mellon University
Abstract
As models grow larger and training them becomes expensive, it becomes increas- ingly important scale training recipes not just to larger models and more data, but to do so in a compute-optinne unit of compute. While such scaling has been well studied for language modeling, reinforcement learning (RL) has received less attention in this regard. In this paper, we investigate compute scaling for online, value-based deep RL. These methods present two primary axes for compute allocation: model capacity and the update- to-data (UTD) ratio. Given a fixed compute budget, ask: how should resources be partitioned across these axes to maximize data efficiency? Our analysis reveals nuanced interplay between model size, batch size, and UTD. In particular, we identify phenomenon we call TD-overfittin; increasing the batch quickly harms Q-function accuracy for small models, but this effect absent in large models, enabling effective use of large batch size at scale. We provide a mental model for understanding this phenomenon and build guidelines for choosing batch size and UTD to optimize compute usage. Our findings provide a grounded starting point for compute-optimal scaling deep RI,, mirroring studies supervised learning but adapted to TD learning. Project page: value-scall
manner that extracts maximal performance per
github.io.
Introduction
Scaling compute plays a crucial role in the success of modern machine learning (ML). In natural language and computer vision, compute scaling takes a number of different forms: model size [19], the number of experts in a mixture-of-experts model [20], or test-time compute [46]. Since these approaches exhibit different opportunities and tradeoffs, natural line of study has been identify strategies for commutt-optimall" scaling [19], that prescribe how to allocate a given fixed amount of compute attain the best downstream performance.
In this paper, we are interested in understanding tradeoffs between different ways to scale compute for value-based deep reinforcement learning (RL.) methods based on temporal-difference (TD) learning to realize a similar promise of transforming more compute to better data efficiency. Value-based TD-learning methods typically provide two mechanisms scale compute: first, increasing the capacity of the network representing the Q-function, and second, increasing the number of updates made per data point (i.e., the updates-to-data, UTD ratio) collected by acting in the environment. Scaling along these two sources present different benefits, challenges, and desiderata [34]. Therefore, in this paper, we ask: What the best strategy to scale model size and UTD to translate a given fixed compute budget into maximal performance?
Analogous to prior scaling studies in language models [19] and deep RL [39], addressing this question requires us to understand how scaling compute in different ways affects the behavior of the underlying TD-learning algorithm. Concretely, we will need mental model of how scaling model size interacts with various other hyperparameters of the TD-learning algorithm, notably the UTD ratio. Most
'qqual contribution. Corresponding authors: prestonffu@erreeley. edu, oleh. oleh. www.rrrkknnnmal.ccommm com, aviralku@andrev. cmu, edu. Code; gwthub.comm//ace...e htt..////meeeetttttttneeeeeaaaal
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

prior work focuses on presenting a single performant set of hyperparameters, instead of providing an analysis to help obtain such a set [23, 34]; we start with a number of controlled analysis experiments.
Our analysis reveals several insights into the distinct, and perhaps even opposing, behavior of TD- learning when using small versus large model sizes. In contrast to supervised learning, where the largest useful batch size primarily depends on gradient noise and is otherwise independent of model size [30], we find that in TD-learning, smaller models perform best with small batch sizes, while larger models benefit from larger batch sizes. At the same time, corroborating prior work [39], we find that for any fixed model size, increasing the UTD ratio reduces the maximally admissible batch size. To convert these observations into actionable guidance, we develop a mechanistic understanding of the interplay between batch size, model capacity, and UTD ratio, discussed in Section 5.
We observe that for any fixed UTD ratio, increasing batch size reduces training TD-error across all model sizes. However, generalization, as measured by the validation TD-error a held-out set of transitions, is highly dependent on the model size. For small models, attempting to reduce the training TD-error with larger batch sizes leads to worse validation TD-error a phenomenon we term TD-overfitting. In contrast, for large models, reducing training TD-error by increasing the batch size up to a threshold enables a lower validation TD-error. We trace the source of TD-overfitinng to poor-quality TD-targets produced by smaller networks: updating to fit these targets can harm generalization on unseen state-action pairs. Empirically, we find that for each model size, there exists a maximal udmissible batch size: further increasing the batch size to reduce the variance in the TD gradient amplifies overfitting. Equipped with this finding and the observation that high UTDs reduce the maximal admissible batch size, we prescribe a rule to identify optimal batch sizes for scaling up RL training under large compute budgets.
We then identify the best way to allocate compute between model size and the UTD ratio, given an upper bound on either compute or on a combination of compute and data budget. We obtain scaling rules that extrapolate to new budgets/compute for practitioners. Our contributions are:
We analyze the behavior of TD-learning with larger models and observe that larger models mitigate a phenomenon we call -veerfiii,, where value generalization suffers due to poor TD-targets.
Based on this analysis, we establish an empirical model of batch size given a UTD ratio and model size, and observe that larger models admit larger batch sizes.
We provide an empirical model of jointly scaling UTD ratio and model size, and the laws for the optimal tradeoff between them.
2 Related Work
Model scaling in deep RL. While large models have been essential to many of the successes in ML [6. 7, 48, 4, 53]. typical models used for standard state-based deep RL tasks remain small, usually limited to a few feedforward MIP layers [38, 17, 18). This is partly because naive model size scaling often causes divergence [2, 41, 34]. Previous works have shown that RL can be scaled to bigger models [52, 34, 25, 41, 24, 47] by using layer normalization [34], feature normalization [23, 26), or using classification losses [24, 8J. While these works focus on techniques that stabilize RL training, they do not investigate the relationship between model capacity and UTD. We leverage our proposed understanding of this relationship achieve compute-optimat RL training. Furthermore, prior work considered the aspect of predictability in scaling model capacity in RL, but in the context of online policy gradients [15] or for RLHF reward model overparameterizzatoon [12]. In contrast, we study model scaling in value-based RL where gradients come from backpropagating the TD loss.
Data and compute scaling in deep RL. A considerable amount of research in RL focused on improving data efficiency through scaling the UTD ratio [3, 5] and find that one key challenge is overfitting [27, 3, 33]. Previous work reported mixed results with evaluating overfitting in online RL [222, 9], but we find validation TD-error to be predictive of TD-overfitting in our experiments, akin to Li et al. [27]. Our TD-overfitting analysis additionally contextualizes prior work showing that large batch sizes can degrade performance with small models [36] Prior works also considered scaling up data in parallelized simulations world models for on-policy RL [31, 44, 40, 11, 45]. Instead, we focus on data-efficient off- oollco learning algorithms and study resource allocation problems pertaining compute allocations instead.
Scaling laws in value-based RI. Most extensions of scaling laws from supervised learning focus on language models and cross-entropy loss [19, 30, 32, 28], with few exceptions targeting downstream
2

---

metrics [10]. In contrast, off -ooicyy RL involves distinct dynamics due to bootstrapping [9, 23, 15], making direct transfer of supervised scaling laws unreliable. Prior work shows that scaling UTD in off-policy RI. yields peculiar law [39]. but leaves model capacity unexplored. We extend this line of work by showing that off- off-policy RI. scales predictably with both UTD and model size and in the process, uncover interesting insights about the interplay between batch sizes, overfitting, and UTD.
3 RL Preliminaries and Notation
In this paper, we study off-policy online RL. where the goal is to maximize an agent's return by training on a replay buffer and periodically collecting new data [49]. Value-based deep RL methods train a Q-network, Qa by minimizing the temporal difference (TD) error:
(3.1) where P is the replay buffer, Q is the target Q-network, s denotes a state, and a' is an action drawn from policy <--ss) that aims to maximize (s a). The ratio of the number of gradient steps per unit amount of data is called the UTD ratio (i.e., the updates-to-data ratio) and we will denote it as o.
L(0) ("s....""""Pa'"ataa)
(r(ss + rQ(s', Qo(s,
4 A Formal Definition of Compute-Optimal Scaling
Our goal in this paper is to develop a prescription for allocating a fixed compute budget or a fixed compute and data budget, between scaling the model size and the aodate-to-datte (UTD) ratio for value-based RI. algorithm. As mentioned earlier, scaling model size and increasing the UTD ratio involve different trade-offs in terms of computational cost and practical feasibility. For example, scaling the UTD ratio results in more "sequential" computation for training the value function, which in turn implies higher wall-clock time but does not substantially increase GPU memory needed. On the other hand, increasing model size largely results in more parallel computation (unless the model architecture itself requires sequential computation, case that we do not study in this work). Answering how to best partition compute between the UTD ratio and model size enables us to also build an understanding of sequential vs parallel compute for training value functions. In this section, we formalize this resource allocation problem, building upon the framework of Rybkin et al. [39].
To introduce this resource allocation problem, we need relationship between the compute C, and the total data D, needed to attain a given target return value J. the model size N, and the UTD ratio a. Formally, we can represent the total compute in FLOPs follows [39]:
Cy(a, N) x a N DJ(a,
(4.1)
where (a, .N) denotes the total amount of samples needed to attain performance J, and Cs N) denotes the corresponding compute. Since batch size is typically parallelizable, does not significantly affect wall-clock time, and is typically much smaller than the replay buffer, we drop the dependency of compute the batch size and aim to optimize compute per unit datapoint. Finally, denote the performance of a value-based RL algorithm Alg as J [(TAgg). With these definitions, we formalize:
Problem 4.1 (Compute allocation problem). Find the best configuration for the UTD ratio and the model size N, such that algorithm Alg attains:
1. Maximal compute efficiency in attaining performance Jo given data budget Do:
No*, = arg min C s.t. (TAII((,NN) 2 Jo, D < Do
(o,N)
Maximal performance given budget Fo and coefficient ô for trading off compute/data: No',N") = arg max (TAII(O,N)) s.t. C+ S Fo. (o,N)
The first part of Problem 4. seeks to allocate compute budget C. between N and o to minimize compute required to reach return Jo. From practitioner's perspective, the solution to this part should prescribe how to attain a target level of performance as efficiently as possible given a certain amount of GPU resources available for training. The second part aims to construct law that extrapolates to higher compute budgets and higher return. Instead extrapolating as a function of return, which can be arbitrary and not predictable, we follow Rybkin et al. [39] and extrapolate as function of budget F= C + 5. D. This allows the practitioners to achieve optimal return given the budget of resources available, where d denotes the cost of data relative to compute, expressed e.g. in wall-clock time.
3

---

Experimental setup. We use BRO [34] and SimbaV2 [26], approaches based on SAC [13] that use regularized residual network represent the Q-values and have been shown to scale well to high capacities. These prior works showed that scaling width gives better performance that scaling depth in TD-eearning Thus, to study the impact of model size, we vary only the network width in (256, 512, 1024, 2048, 4096). We consider batch sizes from 4 to 4096 (varied in powers of 2 and UTD ratios of I, 2. 4, 8. We keep other hyperparameters fixed across all tasks at values suggested by Nauman et al. [34]. For our initial study, we leverage the results from [34] on Deepmind Control suite [50]. Following prior work [14, 34], we separate these into medium difficulty tasks (referred to as DMC-medium) and 6 hard difficulty tasks (DMC-hard) hard). For these tasks, we fit averages of the tasks for the two suites respectively, building upon the protocol prescribed in Rybkin et al. [39]. to show generalization of our fits across tasks. We evaluate scaling on 4 more difficult tasks from DMC and HumanoidBench [42], where we make fits for each task individually to show applicability to single tasks. Further details are in Appendix B.
Analyzing the Interplay Between Model Size and Batch Size
Rybkin et al. [39] argues that the best batch size decreases a power law with respect to the UTD ratio a. However, this prior analysis holds model size N constant and does not consider its influence on batch size. We extend prior analysis [39] by considering how model size modulate lates the effective batch size under fixed UTD ratio, revealing a distinct form of overfitting unique to TD>-eeamiig.
5.1 Measuring Overfitting in TD-Learning
h1-crawi-vo, UTD=1
Train, Critic Width 512
Val, Critic Width 512
Following Rybkin et al. [39], which identifies overfitting as a key factor in selecting effective batch sizes for fixed model size, we begin our analysis by understanding how overfitting depends on model size. Unlike supervised learn- ing, where the target is fixed, TD-learning in- volves fitting to targets that evolve over time and depend on the network being trained. This makes overfitting in TD-eearnng fundamentally different. As a measure of generalization, we measure the TD-error on both the training data (i.e., transitions sampled from the replay buffer) and held-out validation set of transitions drawn i..dd from the same distribution. Further details are provided in Appendix B.
10a
Loss
Batch
10a
Batch 512
Batch 1024
0.0
0.5
1.0
0.0
0.5 Steps
1.0
Steps
1e6
Train, Critic Width
Val. Critic Width
10a
Loss
Observations on model size. We report training hi-crawl at the
0.0
0.5
1.0
0.0
0.5
1.0
Steps
1e6
Steps
1e6
and validation TD-errors end of training in Figure 2(() (see Appendix D.5 for complete loss curves). As model size in- creases, the final training TD-error decreases, consistent with increased model capacity. Inter- estingly, we find that increasing model capacity consistently leads Moreover, there is no clear sign of classical overfitting (i.e., low training error but high validation error). perhaps because TD-leaming rarely "fully" fits target values regardless of model size.
Figure 1: Mcasuring train and validation TD-errors for different batch sizes on hi-crawl. While the train- ing and validation TD-errors reduce model size in- creases, for smaller models a larger batch size results in a higher final TD-error. This illustrates the role of batch size in modulating overfitting with TD-Icarnngg
lower validation TD-error.
Observations on batch size. We next study the role of batch size in Figure 1 (when varying batch sizes for a fixed model size) and Figure 2(b, c). Perhaps as expected, larger batch sizes generally reduce training TD-error, likely because they provide a better low-variance estimate of the gradient. However, their impact on validation TD-error is more nuanced and depends the model size N. For smaller networks (widths (256, 512]), increasing the batch size often plateaus or increases the validation TD-error. This corroborates prior work [39], which identified larger batch sizes as source of overfitting when operating at networks with width 512. However, larger models allow us to use larger batch sizes without overfitting (Figure 2(d)). Why does this occur?
5.2 A Mental Model for TD-Overfitting
In supervised learning, overfitting occurs when reducing training loss further would primarily fit to noise or spurious correlations on the training dataset, in way that results in a higher loss

---

30
Validaaion loss Thaining Loss
Model Widh:: 236
Model wiskt: 256
Commonhh E3to Ste
2
biccremluu
Model (Withh 512
Model Whsth: 512
hlppoeesnn
Mode Wiith;
10losEon Loaz
Model wisth: 1024
stand vb
8
8
600K
21
9M
3CM
256 5:2 1024 2048
120
256
512
1024
20:00
183K
620K
2M
414
300
Model Sine
Betch
Baich Size
Model Stee
Figure 2: Efffct of batch size on TD-error for hi-cravl with σ 1. Left to right: (a) increasing model
size consistently lowers the best achieved validation TD-erro for fixed batch size; (b) Larger batch sizes reduce training TD-error. (c) However, beyond a certain threshold, larger batch sizes lead to increased validation TD-error, particularly for smaller models, indicating TD-overfitting (d) This overfitting threshold increases with model size: larger models can enable higher batch sizes, suggesting increased robustness to overfitting.
validation dataset distributed identically as the training data. Even though smaller networks overfit (Figure 2(c)), our experiments not in this regime since larger networks able to attain both lower training TD-error and lower validation error (Figures 2(b, c)).
We argue that this apparent deviation from classical overfitting is explained by the use of target networks. Regardless of whether a given network has sufficient capacity to reduce TD error on the current batch, TD methods would subsequently update the target network. This can lead to increase in TD-error on validation data at the next step. That is, TD-error may not reduce: (i) on validation state-action pairs or (ii) with respect updated target values.
Small model, small batch
Small model, large batch
plagpe
Large model, small batch
Large model, large batch
ytage
For conceptual understanding: low- capacity Q-networks entangle features used to predict Q-values across state-action pairs [1]. Target network updates in- evitably change target values on unseen transitions, potentially increasing the val- idation TD-error, as we observe empiri- cally in Figure 2(b, c) (full curves in Fig- ure 13). Larger batch sizes produce lower- variance gradients that exacerbate this prob- lem, as fitting the targets on some transi- tions comes at the expense of others with limited representational capacity.
Figure 3: A conceptual view of I'Doverfitting, Small mod- els cannot cope with large hatch sizes due to more directed gradient updates onto low-quality TD-targets, and might di- verge from the target optimal value function V*. Instead, they might perform better with smaller batch sizes, which result in noisy updates. Large models produce TD targets that are high-quality and benefit from regressing these targets better larger batch sizes.
In contrast, larger-capacito capacity models can more effectively decouple predictions across transitions, mitigating this issue and leading to improved generalization even at high batch sizes. This suggests a key observation: avoiding overfitting in TD-leaning requires either smaller batch sizes higher model capacity. We present this insight as an illustration in Figure 3. We note that high capacity model generally leads lower training and validation TD-errors (Figure 2(b, c)). We term this phenomenon TD-overfitting. to emphasize that it is driven not by memorization values on the training set but by the interplay between limited capacity and non-stationary targets, that exist uniquely in TD-learning.
Takeaway 1: Smaller models cannot utilize large batch sizes
The training TD-error decreases with higher batch size. With low model capacity, increasing batch size results in a higher validation TD-error, i.c. the maximum admissible batch size is small. Larger models enable the use of larger admissible batch size. This can be attributed to poor generalization of the TD-targets from smaller models.
5.3 The Role of Dynamic Programming in TD-Overfitting
We now conduct experiments to verify that indeed TD-targets are the root cause of the TD- overfitting phenomenon discussed above. To this end, conduct a diagnostic experiment inspired
5

---

by the setup in Ostrovski et al. [37]. We train a passive critic alongside the main Q-network, This passive critic is not in-
Average Slope: -0.55
18
Average Slope: -0.33
Validation Loss
Validdttion Loss
2x
volved in TD updates but is trained solely to fit the TD-
Passive: 512
Main: 512
10a
Passive: 1024
10a
Main: 1024
targets generated by the main Q-network (experimental de- tails in Appendix B). By ab- lating the network width used to parameterize the passive critic, we can decouple the con- tributions of (i) the main Q- network's i capacity and (ii) the quality of TD-targets, which we hypothesize drives -D-vverfittnn..
Passive: 2048
Main: 2048
512
1024
2048
512
1024
2048
Main Critic Width
Passive Critic Width
Figure 4: Validation TD-error w/ the passive critic. Increasing the model size for both the main and passive critic reduce the validation TD-error, increasing the main critic size is much more effective, showing that target quality is crucial for effective learning.
In Figure 4, we observe that when the TD-targets are generated by a low-capacity Q-network, the passive critic - regardless of its own capacity exhibits higher validation TD-error. This supports the main insight from our mental model: the TD overfitting phenomenon is driven primarily by the poor generalization of TD-targer ID-targets produced by low-capacity networks. While increasing the passive critic's s width improves its ability to fit lower quality targets (e.g., a width 2048 passive critic fits width = 512 targets slightly better than smaller passive critic), the validation TD-error can increase or slowly decrease over the course of training (see h1-crawl in Figure 14). Conversely, when the main Q-network generating the targets is larger (e-g., width 2048), even smaller passive critics (e.g., width = 512 or 1024) can match resulting TD-targets quite well, and validation error decreases over the course of training (see hi-crawl in Figure 14). This indicates that large portion of the overfitting dynamics of TD-learning is governed by the target values, and how they evolve during training, irrespective of the main critic (though the targets themselves depend on the main critic).
We also observe that if the passive critic is smaller than the target-network. it may underfit in its ability to fit the TD-targets the training data, leading to elevated training and validation TD-errors. However, this underfitting effect is much less severe than the overfitting observed when the TD-targets themselves are poor due the limited capacity of the Q-network, seen from the slope in Figure 4.
Takeaway Overfitting in TD-leaming is governed by TD- ID-targets
Overfitting in TD-leaming is less about fitting the TD-targets on limited data, but more about the quality of the TD- "D-targets themselves a direct consequence of model capacity and the fundamental nature of dynamic programming with deep neural networks.
6 Prescribing Batch Sizes Using Model Size and the UTD ratio
Using the insights developed above, we now construct a prescriptive rule to select effective batch sizes, which in turn allows us to estimate the tradeoffss between UTD ratio and compute in the next section for addressing Problem 4.1. Specifically, we aim to identify the largest batch size, denoted B, that can be used before the onset of TD overfitting, i.c., before validation TD-error begins to increase for given model size. From our insights in the previous section, we see that the largest such value of B increases as model size N increases. We also note Figure 5, that B decreases with increasing UTD ratio a,, aligned with the findings of Rybkin et al. [39]: for a fixed model size. larger values lead to TD-overfitting when batch size increases. Motivated by these empirical findings, we propose rule that models B asymmetrically between model size N, and UTD ratio follows:
B(a, N) ~ d + bB R.D N-8e
Q
(6.1)
where ag, by, az. .g > 0 take on values listed in Appendix D.I.
Scaling Observation I: Batch size selection
For best performance, batch size should increase with model size and decrease with the UTD ratio. This dependency can be modeled by a predictable function in Eq. (6.1).

---

batch size
2048
h1-crawi-v,, dm1
2048
h1-crawl-vO, d-2
2046
h1-crawl-v0, a=4
2048
hI-crawi-vO. aw8
512
512
512
512
128
128
128
128
619K
2M
9M
34M
619K
2M
9M
34M
619K
2M
9M
34M
2M
9M
34M
N: model size
model size
model size
N:: model size
Best batch size
2048
h1-crawi-vO, N=619K
2048
h1-crawl-vO, NM-MM
2048
h1-crawi-vO, N- 9M
2048
h1 h1-crawi-vO, N-34M
512
512
512
512
128
128
128
128
rical
32
32
32
32
2
0: UTD
UTD
UTD
UTD
Figure 5: A two-dimensional batch size fit B(,, N). Slices of the fit are shown at particular values of a and N. In line with our analysis of TDooveffitting we observe that larger models allow larger batch sizes. We build a fit that captures the intuition, but this effect does not continue indefinitely but instead asymptotes. We further extend prior work that observed that batch size needs to be decreased with UTD [39] and incorporate that in our 2-dimensional fit. Leveraging batch sizes from this fit allows us to better answer compute allocation questions.
Implications. As increases, the 0B term in the denominator dominates, yielding the approxima- tion N) x an/o"8, consistent with power law relation from prior work [39]. Conversely, as + 0+, i.e., when targets nearly static, B oo since updates are infrequent and TD- overfitting does not occur. Finally, our functional form increases with N, with an upward asymptote at 3 + ag/o"# when N + This reflects the intuition that low-capacity networks require smaller batch sizes to avoid -Dooeeffit,, whereas for sufficiently large models, the maximum admissable batch size is primarily constrained by the UTD ratio
Crucially note that our proposed functional form factorizes into a function of a and a function of N,
(6.2)
1+ 1+ N-
To evaluate the feasibility of such a relationship in our analysis, we had to run a 3-dimensional grid search B x N x However, the fact that this fit is effective subsequently allows practitioners to instead run two 2-dimensional grid searches on B x a and B x N, significantly reducing the amount of compute needed to estimate this fit.
Evaluation. In order to evaluate this fit, we compare to simple log-linear fit log B ~ (1, log a, log N), which would increase indefinitely in and N. Averaged over tasks, our rule achieves a relative eтrоr of 48.9% compared to the log-linear fit's 55.1% (see also Appendix D). One might wonder if relative error of 48.9 9% is actually large for batch size fit. We empirically observe that the error is large because, in many cases, there is a wide range of batch sizes that all attain good performance (see Appendix D.2).
7 Partitioning Compute Optimally Between Model Size and UTD
Equipped with the relationship between batch size, model size, the UTD ratio from Eq. (6.1), and the definition of compute in Eq. (4.1), we now answer the questions from Problem 4.1.'
7.1 Solving Problem 4.1, Part I: Maximal Data Efficiency under Compute Co
To solve this problem, we note that to maximize the data-efficiency we should operate in regime where the total compute, C := k-o o- -N----- ND,(a, .) S Co, where Co and k are constants not dependent on and N. We then require functional form for D,, We observe that extending the relationship from Rybkin al. [39], which modeled data efficiency as an inverse power law of the UTD ratio for a fixed, given model size, can also work well in our scenario when the model size is variable. Inspired by prior work on language models [19, 43], we augment the fit from Rybkin et al. [39] with additive power law term dependent on model size. Intuitively, this is sensible since the total amount of data needed to attain a given performance should depend inversely on the model size since
We also attempted to build fit for learning rate in our experiments, but found learning rate to generally not be as critical as the batch size for compute-optina scaling. Please see Appendix D.3 for more details.
7

---

DMC-medium. BRO
DMC-hard, BRO
bl-crawi-vO, BRO
h1-crawi-vO, SimbaV2
15
15
X1e5
5
2
2
0
1M 3M5M
26M
1M 3MSM
26M
.6M
2M
9M
34M
.6M
2M
9M
36M
Dy: Data until
humancid-stand BRO
h1-pole-vo, BRO
h1 h1-stand-v0,
h1-stand-v0, Simbav2
LTTD
62
2
12M 2M 9M 34M model
16M
2M
9M
34M
.6M 2M
9M
34M
.6M 2M 9M 36M
N:
Figure 6: Data efficiency fit (a, .V) on all domains, shown as iso-data contours. Each contour denotes the curve which attains the same data efficiency to attain given target performance J, with data efficiency denoted by color. The form of the fit allows closcd-form solution for optimal configurations, and show these as stars. These points lic power law. This law enables to scale compute while allocating UTD model size as we will discuss in subsequent results in this paper.
bigger models attain better sample-efficiency in TD-learning [34, 25J. Moreover, model size and the UTD ratio present two avenues for improving data efficiency, hence the additive nature of the proposed relationship. We find that the relationship can be captured as:
D)(a,N) =e Dyjaa + (4))
by N
+
(7.1)
where Dinin is a constant not dependent on a and N, and a,, ar, by. B5 are constants that depend on the return target J. With this relationship in place, we now able to answer Part 1:
bn
AN
o' (Do) =
(Do) = (Do
(7.2)
Do
Dmin
Dmin
where the coefficients can be computed from aJ, a., bJ. By (see details in Appendix A).
Scaling Observation 2: Partitioning compute optimally between model size and UTD
Optimal UTD and model size a predictable function of data budget D (alternatively, compute budget C), as power law in Eq. (7.2).
We visualize this solution in Figure 6. We plot iso-D contours, i.e. curves in (a, N) space that attain identical data efficiency, and find that these curves move diagonally to the top-right for smaller D values, way where both increasing the model size and the UTD ratio improves data efficiency. These contours curved such that there is single point each frontier that attains optimal compute efficiency C. We plot these points, which follow the solution in Eq. (7.2). This allows us to predict data efficiency for novel combinations of UTD and model size, which is crucial.
Evaluation. Our data efficiency coefficients are fitted against a grid of UTD ratios and model sizes. We evaluate our proposed data efficiency fits a grid of interpolated and extrapolated UTD ratios and model sizes using the fitted batch size. Averaged over tasks, our fit achieves a relative error of 10.0% against the ground truth data efficiency on fitted UTD ratios and model sizes, 14.9% on interpolation, and 18.0% on extrapolation. Experimental details are described in Appendix B.
We also compare our estimated UTD ratios and model size with other approaches for allocating unseen compute budgets in Table 1. We compare to the following alternate approaches: UTD-only scaling at compute budget C for given model size, (ii) model-only scaling at compute budget C for a given UTD, and (iii) our proposed compute-optimal UTD and model size, run with constant, fixed batch size not specifically designed for compute budget C. This constant fixed batch size corresponds to

---

the batch size prescribed by our fit for the first compute budget B(o N N (Cmia))) In Table 1, we observe that our compute-optimal scaling achieves the target performance using the least amount of data, whereas both o-nnly scaling and N-only scaling require substantially more data, as evaluated using the ratio of the total amount of data needed for the approaches and the total amount of data needed for our compute-optimal approach. The strategy of using constant batch size performs only marginally worse than our approach. However, as this comparison still relies on our proposed UTD ratio and model-size prescriptions, it primarily shows that these prescriptions are relatively robust to variations in batch size.
Implications. Our results show that appropriate choices of UTD and model size improve both data efficiency and compute utilization. At the same time, we find broad regions of near-equivalent per- formance: multiple (UTD, model-size) settings perform similarly well, so fully optimizing these hyperaarameters is often unnecessary to capture most of the gains (Figure 20 and Figure 21). Sim- ilarly, while the best configuration is environment- dependent, with some tasks benefiting from larger models to begin learning and others from higher UTD, scaling the model size paired with a mild increase in UTD is often good starting point. Our framework makes these trade-offs explicit and provides a principled approach to selecting good values for these hyperparameters.
Table 1:
Data efficiency ratios various ap- proaches to allocate compute to our approach of compute-optimal (a, N) scaling. All perform subpar compute-optinal UTD, model size prescrip- tions terms of data efficiency.
Approach
Mean
Median
Compute-optimal (ours)
1.00
1.00
Compute-optimal (ours)
fixed batch size
1.03
1.05
o-only scaling
1.26
1.18
NN-oly scaling
1.11
1.11
7.2 Solving Problem 4.1, Part 2: Resource Partitioning for Different Returns J
For the solution the problem to be practical, need prescribe solution that works for all values of J. However, J can be arbitrary and not smooth, which makes designing general law impossible. Instead, follow Rybkin al. [39] and use the notion of total budget. J = C + 5 DD as a substitute for J. Similarly to J, the budget JF increases as the complexity of policy learning increases.
That is, for a some unknown monotonic function of F. Using this intuition, we will now demonstrate solution to compute allocation that optimizes F. therefore also optimizing J. Similarly, we will be able to extrapolate solution to higher F, and thus higher J.
TD- algorithm with the "optimal" hyperparameters, will be
We produce a solution to Problem 4.1, part 2, by observing that C and D evolve predictably as function of F, in line with previous work [39]:
(FFFo)
ac F
ac
(F)
bc
Be
(7.3)
Fo
h1-crawi-v0
h1-pole-vo
h1-stand-vo
humanold-stand
x1ed
X1ed
Dee @4400
X1e4
J: P.Peforrance vell
Dy Optimal
32
700
80
600
28
600
24
600
64
500
400
600
16
87.0.99
48
R/=0.90 99
R4 99
Rfs1.00
400
300
14
200
24
4806440
xle15
240
x1e15
120
160x1e15
xle15
Optimal compute
-800
48
700
600
J: Performance
48
6600
32
600
12
400
500
32
R2-0.87
400
82-0.68
2
87=0.99
G 16
24
300
200
320 480640
x1e15
240
320 400
xle15
90
120
16Dx1e15
16
24 32 xle15
17: Optimal budget J
rf: Optimal budget
1ý: Ostimal budget for
1f: Optimal budget
Figure 7: Optimal data (FF) and compute C(Fo) fits for given budget Fo. Return J denoted in color, showing how increased budgets correspond to higher returns. Similar to [39], we are able to allocate resources across data and compute in a predictable way, while accounting for the effect of both model size and UTD.

---

Evaluation, We show that this dependency is predictable in Figure 7, including evaluating confidence interval and extrapolation to higher budgets for this fit. This allows to optimally allocate resources for higher values of budget or return across data and compute.
Scaling Observation 3: Optimal partitioning between data and compute
Optimal scaling for data C and compute are predictable functions of the total budget Fo, as power law in Eq. (7.3).
Now, we extend this analysis to allocating compute across UTD and model size as a function of the budget. We use the same power law form:
aF
by
(Fo)
=
(F)
Fo
(7.4)
Scaling Observation 4: Optimal partitioning of budget between UTD and model size
Optimal scaling for UTD a and model size N depends as a power law on the budget F, as in Eq. (7.3). We can estimate the optimal allocation trend using this power law, and estimate robustness of perfomance to allocation as the variance this trend.
Implications. We show results for two challenging tasks in Figure 8 and further results in Appendix D. We ob-
h1 11-ppole-vO
humanold-stand
8
Optimal UTD
R7m=0.06
R²=0 94
-800
600
serve the coefficients 0r, By for re- source allocation vary between tasks,
Performance
500
600
showing that for some tasks scaling model size UTD is more less important. Further, we observe that different tasks vary in the amount of variance, seen as the size of the con- fidence interval in Figure 8. This shows that for some tasks, precisely setting model size and UTD is impor-
400
300
x1e15
xle15
Nyy Optimal model size
x1e5
x1es
256
R?=0.32
-0-0.44
-800
600
J: Performance
192
500
9
600
tant; while other tasks allow to trade off model size and UTD without a big decrease in performance. Our exper- imental procedure enables practition- ers to make these workflow decisions based on the relationships that we fit in this paper.
128
400
400
300
240
320
400
xle15
16
24
32 xle15
:Optiiall budget for
15: Optimal budget for
Figure Optimal UTD (FF) and model size NFFu), with extrapolation to higher budgets or returns. While for some tasks it necessary to values precisely, other tasks allow some variation in model size and UTD indicated by variance.
8 Discussion
We have established scaling laws for value-based RL allowing compute scaling in an optimal manner. Specifically, we provide a way to scale batch size, UTD, model size, as well as data budget, and provide scaling laws that estimate tradeoffs between these quantities. These laws informed by our novel analysis of the impact of scaling on overfitting in TD-Icarning. We also saw that some environments several configurations of the hyperparameters compute-optimal, which reflected as a benign relative error in our fits. We were limited in how many variables we can study due to the necessity of running nigher-dimensional grid searches for every new variable. Building on our results, future work will study other important hyperparameters, such learning rate and the critic update ratio. Further, while our work is limited to challenging simulated robotic tasks, future work will study large scale domains such as visual and language domains using larger scale models. The analysis and the laws presented in this work a step towards training TD-leaming methods at a scale similar to other modern machine learning approaches.
studied could broadly be considered
10

---

Acknowledgements
We would like to thank Amrith Setlur, Seohong Park, Colin Li, and Mitsuhiko Nakamoto for feedback on an earlier version of this paper. We thank the TRC program at Google Cloud for providing TPU sources that supported this work. We thank NCSA Delta cluster for providing GPU resources that supported the experiments in this work. This research was supported by ONR under N00014-24- 12206, N00014-22- 1-2773, and ONR DURIP grant, with compute support from the Berkeley Research Compute, Polish gigh-performance computing infrastructure, PLGrid (HPC Center: ACK Cyfronet AGH), that provided computational resources and support under grant no. PLG/2024/017817. Pieter Abbeel holds concurrent appointments Professor at UC Berkeley and as an Amazon Scholar. This work was done at UC Berkeley and CMU, and is not associated with Amazon.
References
[1] Richard E Barlow and Hugh D Brunk. The isotonic regression problem and its dual. Journal of the American Statistical Association, 1972.
[2] Johan Bjorek, Carla P Gomes, and Kilian Q &Weinberger. Towards deeper deep reinforcement learning. arXiv preprint arXiv: 106.01 102.
[3] Xinyue Chen, Che Wang. Zijian Zhou, and Keith W Ross. Randomized ensembled double Q-learning: Learning fast without model. In International Conference on Learning Representations, 2020.
[4] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heck, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
[5] Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In International Conference on Learning Representations, 2023.
[6] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint, 2024.
[7] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning. 2024.
[8] Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga. Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, al. Stop regressing: Training value functions via classification for scalable deep arXiv preprint arXiv:2403.03955 2024.
[9] Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, and Shixiang Shane Gu. Why should i trust you, bellman? the bellman error is poor replacement for value error. arXiv preprint arXiv:2201.12417, 2022.
[10] Samir Yitzhak Gadre, Georgios Smyriis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang. Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks, arXiv preprint, 2024.
[11] Mattco Gallici, Mattic Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint, 2024.
[12] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning. 2023.
[13] Tuomas Haamoja, Aurick Zhou, Pieter Abbecl, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018
[14] Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint arXiv: 16828, 2023.
[15] Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint, 2023.

---

[16] Jordan Hoffimann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. Advances in Neural Information Processing Systems, 2023.
[17] Shengyi Huang. Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and JoAGo GM AraAšjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
[18] Matthew Thomas Jackson, Uljad Berdica, Jarek Liesen, Shimon Whiteson, and Jakob Nicolaus Foerster. clean slate for offline reinforcement learning. arXiv preprint arXiv:2504. 2025.
[19] Jared Kaplan, Sam McCandlish, Tom Henighan. Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint, 2020.
[20] Jakub Krajewski, Jan Ludziejewski. Kamil Adamczewski, Maciej Pióro, Michal Krutul, Szymon Antoniak. Kamil Ciebiera, Krystian Król, Tomasz OdrzygóždZ, Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024.
[21] Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- parameterization inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021.
[22] Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning, arXiv preprint vrXiv:2109.088A. 2021.
[23] Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR3: Value-based deep reinforcement learning requires explicit regularization. International Conference on Learning Representations, 2022.
[24] Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline Q-learning on diverse multi-task data both scales and generalizes. In International Conference Learning Representa- tions, 2023.
[25] Hojoon Lee, Dongyoon Hwang. Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. SimBa: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint, 2024.
[26] Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspherical normalization for scalable deep reinforcement learning. arXiv preprint arXiv:2502. 2025.
[27] Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting. In International Conference on Learning Reprexentations, 2023.
[28] Jan Ludziejewski, Jakub Krajewski, Kamil Adamezewski, Maciej Pióro, Michat Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóćdž. Piotr Sankowski, al. Scaling laws for fine-grained mixture of experts. In International Conference on Machine Learning, 2024.
[29] Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. International Conference on Machine Learning, 2023.
[30] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of bargh-batch training. arXiv preprint. 2018
[31] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, 2016.
[32] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 2023.
[33] Michal Nauman, Michat Bortkiewicz, Piotr Mitoš, Tomasz Trzcinski, Mateusz. Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: The bitter lesson of reinforcement learning. In International Conference on Machine Learning, 2024.
[34] Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miloš, and Marek Cygan. Bigger, regularized, optimistic: Scaling for compute and sample-efficiees continuous control. Advances in Neural Information Processing Systems, 2024.
12

---

[35] Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Puerre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, 2022.
[36] Johan Obando Ceron, Mare Bellemare, and Pahlo Samuel Castro. Small batch deep reinforcement learning. Advances in Neural Information Processing Systems, 36:26003 26024, 2023.
[37] Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems, 34:23283-23295, 2021.
[38] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:; Reliable reinforcement learning implementations. Journal of machine learning research, 22(268):1-8, 2021.
[39] Oleh Rybkin, Michal Nauman, Preston Fu, Charlic Snell, Pieter Abbeel, Sergey Levine, and Aviral Kumar. Value-based deep scales predictably, 2025. URL btctps://ariiv org. org/ htbb///02.0445
[40] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering Atari, Go, chess and Shogi by planning with learned model. Nature, 2020.
[41] Max Schwarzer. Johan Samir Obando Ceron, Aaron Courville, Mare Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level Atari with human-level efficiency. In International Conference on Muchine Learning, 2023.
[42] Carmelo Sfemazza,, Dun-Ming Huang. Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-bod body locomotion and manipulation. arXiv preprint arXiv:2403.10506 2024.
[43] Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, and Alaacldin El-Nouby. Scaling laws for native multimodal models scaling laws for native multimodal models. arXiv preprint arXiv:2504.0788 077511, 2025.
[44] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of Go with deep neural networks and tree search. Nature, 2016.
[45] Jayesh Singla, Ananye Agarwal, and Deepak Pathak. SAPG: Split and aggregate policy gradients. International Conference on Machine Learning, 2024.
[46] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, arXiv preprint arXiv:2408.03314 2024.
[47] Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch. Thomas Lampe. Philemon Brakel, Sarah Bechtle, Steven Kapturowski, Roland Hafner, et al. Offline actor-critic reinforcement lcarning scales to large models. International Conference on Machine Learning, 2024.
[48] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04611, 2022.
[49] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[50] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li. Diego de Las Casas, David Budden, Abbas Abdolmaleki. Josh Merel, Andrew Lefrancq, et al. DeepMind control suite. arXiv preprint, 2018.
[51] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa, dm_control: Software and tasks for continuous control. Software Impacts, 2020.
[52] Kevin Wang. Ishaan Javali, MichaL Bortkiewicz, Benjamin Eysenbach, et al. 1000 layer networks for self- supervised rl: Scaling depth can enable new goal-reaching capabilities. arXiv preprint arXiv:2503.14858. 2025.
[53] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lueas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104-12113. 2022.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided a proper justification given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] " is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer is often more nuanced, so please just your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
[Yes]
Justification:
The plots in the paper show the scaling laws.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
[Yes]
Justification:
in the conclusion section.
14

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
[Yes]
Jusiifictiionaasuuptionss and proofs for Prop 1 are stated in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
[Yes]
Justification: Appendix B provides extensive details and full code will be released for reference.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: in the supplement
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. /tipsscccppubllle wttpcccccppppppceeeeeeeeeceepeee htt/g//ddde//oeuuuuurrme wwwtanissssinnoiiiieeooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? [Yes]
Justification: We provide details in Appendix and follow prior work [34, 39] unless otherwise stated.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? [Yes]
Justification:
confidence intervals are provided where possible in the appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
[Yes]
Justification: In the appendix B
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttt::://nnurips.... cc cc/public/ htte/etiiiiiieeliiiieeeeeeeeeeee
[Yes]
Justification:
The paper represents theoretical work with only simulated results
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper represents theoretical work with only simulated results
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
The paper represents theoretical work with only simulated results
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: the appropriate papers always cited and the released code will acknowledge code that was reused.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets beyond code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: No LLMs used beyond standard purposes such as writing code, help with theoretical analysis, literature search and writing.
Guidelines:
The answer NA
that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: pestps://neurse //tpur/een.////ee/ cc/Conff c2/Conferencee/2225//////// for what should or should be described.
20

---

A Details on Deriving Scaling Fits
FLOPs calculation. We inherit the definition from Rybkin al. [39], so that
C(a, N) o N D(o, N)
(A.1)
for constant & not dependent and N. We follow the standard practice of updating the critic, target, and actor all a times for each new data point collected (Algorithm 1).
A.1 Maximal compute efficiency for data I
Do
As described in Section 7, the number of data points needed to achieve performance J is equal to
Du(a, N) Re D +
aJ
bj N
8,
(A.2)
where D aJ,, a., bJ, B3 > 0 are constants not dependent on and N. We first present a closed- form solution to the simpler optimization problem in Eq. (A.3). This will enable us to characterize the solution to Problem 4.1, part 1, which does not have closed-form solution in terms of Co but can be easily estimated.
Proposition A.1. ay or B. < 1, there exists unique optimum
(o*,N") arg min (a,N)
C)(a,N) s.t. Dy(a,N) Da-
(A.3)
Moreover,
1 + Lj 3y
1/aJ
R
1/8,
= a2
N = by
ay
(A.4)
Do Dmin
Do
Dmin
satisfy the following relation:
BIbös
1/3,
N =
a (0+)ar195.
(A.5)
Proof. For ease of notation, drop the subseript J throughout this derivation. Since there exist sufficiently large (a,N) for which D(a, N) < Dn. Slater's conditions are satisfied, and the KKT conditions necessary and sufficient for optimality. Let
N)a ko N D(a, N) + A(D(o,N) Do)
(A.6)
denote the Lagrangian. Now, we will solve for (ö, N, satisfying the KKT conditions. The stationarity conditions are
ØC
0 = (kō + - N)
(A.7)
de
0L
0 - (kō + Rb N N-8-1 kOD(o,
(A.8)
ON
Complementary slackness implies that
À(D(O, N) Dn)
(A.9)
We claim that Ā > 0. Assume for the sake contradiction that Ā 0. Substituting into Equa- tions (A.7) and (A.8), we obtain
a 5 336 Ñ D(ā, max faro .
N -)
(A.10)
But the last inequality contradicts < 1 or 3 < 1, concluding the claim.
21

---

It follows that N) = Do. Dividing Eq. (A.7) by Eq. (A.8), we obtain
a aa-1 N Bb N-B-1 δ
(A.11)
or equivalently
_b3
Ñ
0a/8
(A.12)
a.aa
Substituting into the active constraint D(ō, N) = Do. we obtain
1 1/a
1/3
1 +
Ñ = b
(A.13)
Do
Dmin
Do Diin
Thus, (ā, N) is the unique KKT solution, and thus the unique optima.
A.2 Maximal data efficiency for compute S Co
We are now equipped to solve the optimization problem presented in Problem 4.1, part 1. Although we cannot solve for the optimal (a", NN* directly, the following proposition shows that the set of optimal solutions obtained by varying the compute budget Co matches exactly the set of solutions obtained by varying the data budget Do in Proposition A. This equivalence reduces the original problem to simpler surrogate. Using Eq. (A.5), it is straightforward to compute the optimum numerically.
Proposition A.2. Suppose as < By < 1, and assume the data and compute formulations established in Equations (A.1) and (A.2). Let
Cyin
min C), .>NN50
(A.14)
For fixed budget Co 2 Cyii, write
(Pi)
D
min (o.N)
D)(a,N) s.. Cy(a,N) co
(A.15)
and
(Pa)
min (g,N)
C)(a,N) s.t. D"
(A.16)
Each problem admits unique solution (o*, ,N,, and these solutions coincide.
Proof. As before, we drop the subscript J. We first justify the existence of a global minimizer to (P1) over (0, oo)?. As > 0+ or N + 0+, then (a/a)a - oo or (b/N)8 - oc, hence D(a, N) - oo. If a, N > oo, then C(a, N) > ka NDmmin + co, contradicting C < Co- Thus, the feasible set {C Co} is coercive, and by continuity of C and D. (Pi) attains a global minimizer.
Proposition A.I shows that there exists À > 0 such that the KKT conditions for (Pi) hold,
OD
00" ,N) + (o" Nº = 0
(A.17)
do
do
0D
OC
aN (0 (o) +
ON (o" (o*,N*) =
(A.18)
C(a", N*) = Co.
(A.19)
For (P2), the KKT conditions imply that there exists μ 2 0 such that
ac
YD
do (a NT) + p 0o
(al, Nt) = 0
(A.20)
ac
3D
ON (ot, Nt) + ON
(ot, Nt) =0
(A.21)
D(ot, NI) = D*
(A.22)
22

---

If a < 1, then
0D
a0 0 a-1
0a-1 <0
(A.23)
do
OC o a- + kN min + t b 0,
(A.24)
Òo
so μ > O. In the other case, if B < 1, then
JD ON
< 0 and
àc ON 0, p> 0.
Since the first solution is additionally given satisfy and must be their solutions, (o", ,,, (at, Nt, 1/p). Uniqueness in proposition A. implies uniqueness in (P)).
= these systems are identical,
A.3 Maximal performance for budget S Fo
Performance level J is task-dependent and is not guaranteed to satisfy any general properties, so modeling part 2 of Problem 4. directly is impossible. However, given a particular value of J, we can compute the UTD ratio F,, and model size Ny, that uniquely minimize the total budget Fa N) = Cs(a, N) + Da(a N) (see Proposition A.3). We run this procedure for JI....,Jm Jm E [mmin; J as described in Appendix B.
We expect that a higher budget will ultimately yield higher performance under the best hyperparameter configuration. This procedure yields several points {(J,, } along the Pareto frontier J + min,,N N Fy N), . as shown in Figure Importantly, we do not directly model this curve, and only need its existence, continuity, and monotonically increasing nature for our fits. Consequently, its inverse continuous and monotonically increasing. Therefore, for given budget F,,, S S m, the performance level Jj is optimal for that budget, i.e.
(FF. = arg max (a.N)
J (TAIR(O,N)) N)) s.t. C+ F
This procedure yields m points along the solu- tion to Problem 4.1, part 2. Since data efficiency is predictable, we can therefore constrain the budget to model 0² N in Eq. (7.4).
Jm
Performance
Proposition A.3. Suppose 3y) E (0, 1). and fix 5 > 0. Consider the unconstrained minimization min,, F)(o,N) N). The optimum (o*, N*) is unique and satisfies Eq. (A.5).
J1
Proof. As either a + 0+ or N + 0+, the term 8D
As o, N + the term 2 No + co. By the same logic as the proof of Proposition A.2, global minimizer exists.
Fr. (a (3L (F.(
Fs.
Fim
Budget
Then, the objective is exactly the same as Eq. (A.6), with À replaced by and the Do constant offset removed. Thus, the same logic the proof of Proposition A.I applies, and we obtain the same relation Eq. (A.5).
Figure 9: A (hypothetical) depiction of the performance- budget Pareto frontier implicitly model. For each J., compute the budget-mininizing UTD ratio a² (F,,) model size N We can then discard the y- axis, leaving us with a relationship between budget + and (z)).
B Experiment Details
For our experiments, use total of 17 tasks from two benchmarks (DeepMind Control [5]]
and HumanoidBench [42]), listed in Table 2, with the BRO algorithm and architecture [34]. We additionally use tasks from HumanoidBench (hi-crawl, hi-stand) with SimbaV2 [26]. As described in Appendix C, we normalize our returns to [0, 1000]; optimal æ returns are pre-normalized. For HumanoidBench, we report the returns listed by authors as the "success bar," even though possible to achieve a higher return. Our experiments fit Da N) for 20 normalized performance thresholds J, spaced uniformly between Juin and Jmax: inclusive; 20 is an arbitrary choice that we made to obtain useful insights about our method while not overwhelming the reader.
23

---

Table 2: Tasks used in presented experiments.
Domain
Task
Optimal ा Returns
Juin
Jmax
8
HumanoidBench
h1 h1-crawl
700
450
780
2e12
h1-pole
700
300
680
Sell
h1-stand
800
200
660
5c11
DMC
humanoid-stand
1000
300
850
5e10
DMC-Medium
acrobot-swingup
1000
150
400
lel1
cheetah-run
1000
400
750
lell
tinger-turn-bard
1000
400
900
lell
fish-swim
1000
200
710
lell
hopper-hop
1000
150
320
lell
quadruped-run
1000
200
790
lell
walker-run
1000
350
730
lel1
DMC-Hard
dog-run
1000
100
270
lell
dog-trot
1000
100
580
lell
dog-stand
1000
100
910
lell
dog-walk
1000
100
860
lell
humanoid-ru
1000
75
190
lell
humanoid-walk
1000
200
650
lell
Table 3: Configurations for ORIGINAL 3-dimensional grid searches.
Task
UTD ratio
Critic width
Possible batch sizes
h1-crawl
1, 4,
256, 512, 1024, 2048 256, 512, 1024, 2048 256, 512, 1024, 2048
32, 64, 128, 256, 512, 1024, 2048 64, 128, 256, 512, 1024, 2048 128, 256, 512, 1024, 2048, 4096 64, 128, 256, 512, 1024
h1-pole
1, 4, 8
hi-stand
1,2, 4, 8 1,2,4,8
humanoid-stand
128, 256, 512, 1024, 2048
B.1 Hyperparameter Sweep Details
Out of the 17 tasks, we run full 3-dimensional grid search B x N x on 4 of them: 3 tasks from HumanoidBench and Humanoid-Stami from DMC. Due to the computational requirements of running a large grid search for obtaining the scaling fits, we use constant network depth (2 BroNet blocks [34]) and learning rate (3e-4) throughout our experiments and run at least random seeds per configuration. From these experiments, we follow the procedure described in Appendix C to estimate batch size rule (Figure 11). A superset the configurations run in this three-dimensional grid search are listed as ORIGINAL in Table 3. Out of the listed batch sizes in Table 3, 4 consecutive values of batch sizes for each (o, NN. such that the empirically most performant batch size is neither the minimum heuristic enables us to effectively reduce the total number of experiments we need to run to estimate batch size fits. For instance, for small model sizes and low UTD values h1-crawl, this amounts to simply running batch sizes up to 64, since performance decreases monotonically as the batch size increases.
at least
maximum of the range. Since full 3D-sweep is expensive, this
Based on these runs, we set Jmin and Jmax. as described in the following subsection. This enables us to establish batch size rule (Eq. 16),)), where the "best" batch size uses the least amount of data to achieve performance Jmax- To evaluate our batch size rule 3" (a, .N., we run a 2-dimensional sweep using proposed batch sizes on INTERPOLATED and EXTRAPOLATED UTD ratios and model sizes N. The configurations are listed in Table 4. Note that we did not study extrapolation of model size on humanoide ioinoodd-ttadd since already noticed that a width of 2048 performed worse than model width of 1024 at low UTD values.
Using various combinations of these measurements, we can fit data efficiency (Eq. (7.1)). InSec- tion 1. we evaluate the absolute relative error of the prediction with respect the ground truth data efficiency on each of the datasets, when the fit solely uses ORIGINAL. AT. data and is evaluated on ORIGINAL. INTERPOLATER and EXTRAPOL.ATED data. Our final D, as described elsewhere the paper, is fitted on all three datasets, ORIGINAL, INTERPOLATED, and EXTRAPOLATED.
24

---

Table Configurations for INTERPOLATED and EXTRAPOLATED.
Dataset
Task
UTD ratio o
Critic width
INTERPOLATED
h1 a1-crawl
3, 6. 12
368, 720, 1456 368. 720, 1456
hi-pole
3. 6, 12
h1 h1-stand
6, 12
368, 720, 1456
humanoid-stand
3, 12
176, 368, 720, 1456
N EXTRAPOLATED
hi-crawl
1, 8, 16
4096
h1-pole
1,2, 16
4096
hi-stand
1,2,4, 8, 16
4096
EXTRAPOLATED
hi-crawl
256, 512, 1024, 2048
hi-pole
16
256, 512, 1024, 2048
h1-stand
16
256, 512, 1024, 2048
humanoid-stand
16
128, 256, 512, 1024, 2048
The other 13 tasks are from DMC, which we group as DMC-medium and DMC-hard following Nauman et al. [34]. For obtaining these fits, we borrow the data directly from Nauman et al. [34]: the authors this prior work ran 10 random seeds at a constant batch size 128 and learning rate 3e-4 on several UTD (1, 2, 5, 10, 15) and model size (Table 7 in [34]) configurations. Due to the lack of appropriately set batch size these experiments borrowed from prior work, the data does not accurately represent the best achievable data efficiency, and in some cases increasing UTD or model size worsens performance. In these cases, fitting D per task can result in instability, where the exponents at,, B5 are driven to O. To counteract this, we use two approaches:
1. Share parameters a,, By of the fit over tasks follows:
9,
CnV
DSTHH No, D min
(B.1)
N
Conceptually, this forces the slope of the compute-optimal line prescribed by Eq. (A.5) to be shared across tasks within the same domain, but allows for different intercept. This results variance reduction in the fitting procedure.
2. Average over multiple tasks according to the procedure Appendix C. We present these fits in the main paper to improve clarity and reduce clutter (Figure 6). This method essentially treats the benchmark as single task and fits an average amount of data required to achieve some performance.
Selecting experimental constants. To select Jmax. we first group by the UTD ratio and model size N. Out of each group, we select the run with the highest final Monte-Carlo returns (over all batch sizes). Over these runs, set Jmax the highest return threshold that 80% of the runs reach.
We heuristically select Jmin as the lowest return threshold such that configurations that eventually reach performance Jmax "look sufficiently different," i.e. there are configurations with batch sizes B1, B2 such that their confidence intervals [D_.... a Jmin: D Jmm. J do not overlap. Here D denotes the true (not fitted) amount of data required to reach the performance level, and is the standard deviation given by the procedure described Appendix
We select ó in the budget formula F = C+dD so that o'D represents the real-time cost of environment
steps, as measured in FLOPs. Our procedure is as follows:
1. Pick the run that achieves performance Jmax within the lowest wall-clock time.
2. Based timing statistics from this run, set
FLOPs/ /grad steps env steps/sec
x grad steps/sec
å ~
(B.2)
The resulting expression for F is therefore proxy for wall clock time.
25

---

Algorithm Training loop drop-ins for any value-based algorithm
Initialize environment
2: Initialize replay buffer P
3: Initialize parameter vectors 0 (critic), 0 (target critic), ø (actor)
4: Initialize validation environment pd
Initialize validation replay bufferr p
size (Pl/
6: Initialize passive critic parameter vector (fessove
Il possibly different size than 0
7: for each iteration do
each environment step do
9:
ag ~ TT(aa((s)
10:
81-1 ~ p(sc411|s (t,
11:
P {(8;, de, (8t,, at), (t+1)}
12:
if mod == then
I1 do validation less frequently avoid overhead
13:
ai ~ TTlai (st
14:
st41 p (8 (s (stat at pvd -- pro U (8, at (ot) aa 88411))
val
15:
16:
end if
17:
end for
18:
each update
19:
Sample training batch æ ~ P for gradient steps do
20:
21:
i Meriic Ve
Le (2;; 0)
22:
gressive c grasine Mhritic V grenne Cenne (2; geavinv
23:
- Φ Thetor V Cacsor ())
24:
7
7)
25:
end for
26:
logging then
27:
Sample validation batch x: ~ pmm
28:
Cual f- Lenicc ( 0
29:
L +- Laiie (2; gressive
30:
end
31:
end for
32:
end for
B.2 Detailed Explanations for How to Obtain Main Paper Figures
Figure 2. Standard off-policy online RL trains on data sampled from a replay buffer, which regularly augmented with data from the environment. We construct held-out dataset of transitions following the same distribution as the training replay buffer. To do so, we create validation environment, which is identical to the training environment with different random seed, and correspouding validation replay buffer. This allows us to measure the validation TD-error, i.c. the TD-error of the critic against the target on data sampled from the validation replay buffer. Algorithmic details are described in Algorithm in blue.
Figure 4. The passive critic regresses onto the target produced by the main critic, and is trained using a similar procedure as the main critic. We report the TD-target validation data. Algorithmic details are described in Algorithm in green.
of the passive critic against the
Figure 5. We describe batch size fitting procedure in Appendix C.
Figure Circles represent the true data efficiencies on our ORIGINAL UTD ratios and model sizes. Using this data, fit batch size rule 3" (a, NN) (Eq. ().1)), and run experiments using batch size rule on INTERPOLATED
and EXTRAPOLATED UTD ratios and model sizes. Then, we fit data efficiency D Damas (a, .) (Eq. (7.1)) on all of the data, where Juux is listed in Table 2. The iso-data contours are predictions from the fit, and the log-log- containing compute-optimat points follows the formula in Eq. (7.2).
Figure 7. We fit D,, independently for each J.. Following Appendix A.3, we numerically solve for the optimum (FF,), (F,,)) We plot DD and C for these optima against FJ,- Out of these m 20 points, we fit line to the bottom 15 of them and mark the top as budget extrapolation. We record R² between the log-linear fit and log-y values over all 20 points.
26

---

hl-stand, N=2M, B=1024
h1-crawl, N191K, B=
humanold-oo standdddd pw8, N-2M, B-512
800
800
700
600
Return
600
400
400
200
Isotonic smoothing
Original data
200
Ongiinl data
Isotonic smoothing
Isotonic smoothing
Original data
Inclusiing resets
Gaussian smoothing
Gaussian smoothing
2 4
10 xle5
D: Data
Xle5
Figure 10: A demonstration of our MC returns preprocessing, Left: Full- Fall-parameter resets introduce variance in returns; we remove the dips before running isotonic regression. Middle: Gaussian smoothing can lead to under-smoothing the returns, making data efficiency difficult fit. Right: Gaussian smoothing can lead to over-smoothing the returns, e.g. at 625K env steps, Gaussian-smoothe returns are higher than the maximum returns achieved up to that point.
Figure 8, Same method as Figure 7.
C Additional Details on the Fitting Procedure
Preprocessing return values. Our fits require estimates of the data and compute needed by a given run to reach target performance level. The BRO algorithm [34] employs tull-parameter resets form of plasticity regularization [35], reinitializing the agent every 2.5M gradient steps to encourage better exploration and long-tern learning. However, these resets induce abrupt drops in Monte Carlo (MC) returns, which do not reflect a true degradation in learning quality. Instead, returns typically recover quickly and often surpass pre-reset levels. Including these transient dips in the MC curve would artificially inflate the estimated data and compute required to reach a given performance threshold. To obtain cleaner, post-reset return drops from our analysis (Figure 10, left). This allows us to more accurately model the intrinsic data efficiency of the algorithm, independent of resct-induced variance.
consistent signal of learning progress, we therefore remove
Following [39], we then process the return values with isotonic regression [1], which transforms the return values to the most aligned nondecreasing sequence of values that then be used to estimate D, (Figure 10, middle and right). This procedure enables us to fit the minimum number of samples needed to reach given performance level, regardless of whether the performance drops later in training. It also reduces variance compared to the naive approach of measuring the data efficiency directly each random seed.
Uncertainty-adjuste optimal batch size. We follow Rybkin et al. [39] to compute uncertainty- adjusted optimal batch sizes, since the precision of the fit B would otherwise be limited by the granularity of our grid search. We run K 100 bootstrap estimates by sampling n random seeds with replacement out of the original n random seeds, applying isotonic regression, and selecting the optimal batch size B₁ by data efficiency to threshold Jmax- Since these batch sizes can span multiple orders of magnitude (Table 3), report the mean of these bootstrapped estimates in log space the "best" batch size:
K log B
Buuotstrap = cxp
(C.1)
K
k=
Additionally, considering the set of bootstrapped data efficiencies to reach a given performance threshold J, this procedure also yields an estimate of the standard deviation of the data efficiency.
Fitting procedure. Prior work fits the data using a brute force grid search followed by LBFG-S [16, 39). Empirically, we found that the quality of the resulting fit is highly dependent on the initial point found by brute force, and the bounds of the brute force grid must be tuned per fit. To resolve these issues, we use the following procedure:
27

---

1. Normalize the inputs 2 to [e, h]
[0.5, 2] in log space via
Dog(max.1 x) log(min
(C.2)
log h log m (mim x) - 8 log E
(C.3)
log m
x = exp
(C.4)
and normalize the output by dividing by the mean, v' y/ī. This results in more numerically stable fitting procedure, since a E [1. 20) and E [le5, 2e8] are otherwise on very different scales.
2. Define 0' softplus(0) 0) - log(1 + exp(0)) for all "raw" parameters 0 E R. Softplus is smooth approximation to ReLU and forces fit parameters to be positive, and empirically tends to improve fitting stability. For example, to fit data efficiency, we optimize over [0pmin, fa: 06, ax 0g] R2, and extract e.g. Dinin softplus( (6
3. Use LBFG-S to optimize over raw parameters. We use MSE in log space as the objective: C(y, 9) = (log y log 0)
4. Apply softplus and correct the parameters for normalization.
Empirically, we find that initializing all raw parameters as zero generally works well.
Aggregate data efficiency. In Figure 6, we show data efficiency fits aggregated over multiple tasks. We follow Rybkin al. [39]: first, normalize the data efficiency D by intra-environment medians Din mod median (D (a, N)),,N To interpret the normalized data efficiency on the same scale
as the original data, we write Di median (DiTTo med)
cav' so that Dir morm := Dr
D
D ml
Finally, we fit all the normalized data together using the same functional form.
D Additional Experimental Results
D.1 Batch Size Fits B(a, N)
Refer Figure 11.
1680.64
h1-crawl
0.30 + 6.01e7 00030
1.122
4112.98
h1 hi-pole
0.24 + 1.45e1 0..24 N.0007
(D.1)
1458.10
hi-stand
0.27 + 1.33e N0.27
1160.40
humanoid-stand
0.40 + 2.77ee 0.49
N
D.2 Batch Size Fit Analysis
In Table 5, we group runs by UTD and model size, and bin runs based on batch sizes. Then, consider the data efficiency ratio between the runs appearing in bins with suboptimal batch sizes and runs with the predicted batch size, and average over UTDs and model sizes. We find that batch sizes within a interval around the best batch size B perform reasonably, and performance degrades significantly with larger intervals. Indeed, per this analysis, one cannot naively reuse the same batch size for small and large models: in Figure 5, we see a zs 40x range in bootstrap-optimal batch sizes across different model sizes UTD 8. However, the sensitivity of performance to the precise value of batch size is relatively low, which is good news for practitioners and which is why we observe high relative error the fit, which turns be benign.
D.3 Learning Rate Sensitivity Analysis
natural question whether learning rate affects performance in the compute-optimal regime or not. We found that there is a range of "reasonable" learning rates, which empirically always contains
28

---

Grouped by UTD ratio
Best batch size
h1-pole-v0, o=1
h1-pole-v0, o=2
h1-polee-vO, a=4
h1-pole-vO, o=8
2018
2048
2048
2048
512
512
512
512
128
128
128
128
32
32
32
32
619K
2M
9M
34M
619K
2M
9M
34M
619K
2M
34M
619K
2M
9M
34M
Best batch size
h1 stand vo, o=1
h1 11l-standvv vo, o=2
h1 stand vO, a=4
h1 stand v0, a=8
2048
2048
2048
2048
512
512
512
512
32
32
32
32
619K
34M
humanoid aw1
humanoid stand, o=2
humanaid standd,--dddd.ddiidd a-4
humanaid-sttnn stand, g=8
2048
batch
512
512
512
512
32
32
32
32
183H628K 2M
9M
34M
163H628K 2M 9M 34M
183K628K 2M
9M
34M
183H628K 2M 9M 34M
medel
Grouped by model size N
Best batch size
h1-pole-vo, N=619K
hl-pole-vo, N=2M
hl-pole-vo, N=9M
h1-pole-vO, N=3MM
2048
2048
2048
2048
512
512
512
512
128
128
128
128
32
32
32
32
2
2
8 Best batch size
h1-stand-or -v,, N=619K
h1 stastand v0, N=2M
h1 h11-stand-vO, N=9M
h1 mil-stand-ve v0, N:34M
2048
2048
2048
2048
512
512
512
512
32
32
32
32
humanoid stand, N-628K
humanoid stand, N=2M
humanoid stand, N=9M
humanaid stand, N-34M
2048
2048
Best batch
512
512
512
512
128
128
128
32
32
32
32
2
2
0:
0:
a:
a:
Figure 11: Two-dimensional batch size B(a, N) grouped by a and N, as a completion Figure 5, BRO algorithm and architecture [34]
29

---

Grouped by UTD ratio
Best batch size
h1-crawi-vo,
h1-crawi-vD,
h1-crawi-vD,
h1-crawi-vD,
2048
2048
2048
2048
1024
1024
1024
1024
256
256
256
256
604K
2M
9M
36M
604K
2M
9M
36M
604K
2M
9M
36M
604K
2M
9M
36M
Best batch szze
h1-stand-vD,
h1-stand-vv, o-2
a1-ssanndoo vv, a-4
h1-stand-v00,
2048
2048
2048
2048
256
256
256
256
2M
36M
2M N: model size
36M
model size
model size
N: model size
Grouped model size N
Best batch size
h1-crawi-v0,,
h1-crawi-vO,
h1-crawi-vD,
h1-crawi-vo,
2048
2048
2048
2048
1024
1024
1024
1024
256
256
256
256
604K
2M
9M
36M
604K
2M
9M
36M
604K
2M
9M
36M
604K
2M
9M
36M
Best batch s2e
h1-stand-vD, gw1
h1-stand-v0, o-2
h1-stand-vO, a-4
-t-sstnddvO,, a-8
2048
2048
2048
2048
256
256
256
256
36M
2M
36M
model size
N: model size
N: model size
N: model size
Figure 12: Analogous Figure 11, with the Simba architecture [26]
Table Batch size sensitivity over grid search. Batch sizes away from the predicted batch size perform poorly.
Batch size range
Data efficiency ratio
1/16 B" 1/8 B*)
1.52
1/8 B 1/4 B*
1.38
1/4 1/2
1.26
1.22
B"
1.00
B* 2
1.16
B 4B*
1.18
AB".8BB
1.19
8 8B,,66B*
1.30
30

---

Table 6: Learning rate sensitivity over grid search.
Learning rate range
Data efficiency ratio
1/41r" 1/
1.39
1 1/2 1/311r
1.35
12/3
1.04
Ir" 1.5
1.03
1.5 21r"] 21r,,4r"
1.18
1.27
Table 7: Bootstrap-optimal vs. default learning rates over compute-optimal (a, B).
Environment
Data efficiency ratio
h1-crawl
1,0118
h1 h1-pole
1,0000
h1-stand
1,0000
humanoid-ste
0.9504
"default" value of 3e-4. Crucially, this is the case for all model sizes and UTD ratio, meaning that practitioner can get away without setting learning rate carefully for a compute-optimal run as long as they utilize default value.
Grid search regime. We hyperparameter sweeps over (model size, UTD, learning rate) and (model size, batch size, learning rate), where E {1e-4, 2e-4, 3e-4, 6e-4). In this regime, we found that the empirically optimal learning rate only took on values f2e-4, 3e-4). We report the data efficiency ratio between the empirically optimal and default learning rates in Table 6. Since our default learning rate is in the range [Ir". Ir"), the overall effect on performance is minimal.
Although we observe smaller relative variation the best learning rate over UTD and model sizes compared batch size, find empirically that the best learning rate (i) decreases with increasing model size, correlation: -0.75, (ii) decreases with increasing UTD, correlation: -0.46, (iii) increases with increasing batch size, correlation: 0.42. With simple log-linear fits, we obtain a relative error of 37.5%:
h1-crawl
Ir ~ 4.4827e-4 (N/2.3e6)- -0.3122 0.1127
(B (B/511 (512))0.708
h1 h1-pole
Ir" 2.4727c-4 (NV2.3366) 0.2472
-0.3392
(B/512) 512) 0.2701
0512))22701
Despite the high relative error, we observe that data efficiency is similar within an interval of "reasonable" learning rates.
Compute-optimal regime. For each task and compute-optimal setting o (Co), N* (Cg) with fitted batch size (o N (Co),, we ran a sweep of learning rates over [lc-4, 2e-4, 3e-4, 4c-4, 5c-4]. Following Eq. (C.1), compute the bootstrap-optimall learning rate for each setting. then round to the nearest the five learning rates. In Table 7, we show that data efficiency is not improved significantly when using the rounded bootstrap-optimal learning rate, compared the "default" learning rate 3e-4. The table shows averages over compute budgets.
D.4 Target Network Update Rate Sensitivity Analysis
Value-based deep RL methods train a Q-network Qo by minimizing the TD-error against the target Q-network Q (Eq. (3.1)). The target network weights 0 are typically updated via Polyak averaging, (1- 70, where 7 is constant, the target network update rate. Small yield high-bias, low-variance targets; large 7 the opposite. Intuitively, 7 seems to be an important hyperparameter for modulating the dynamics of TD-Icaming, Empirically, however, we do not find a strong relationship between the model size, the target network update rate т, and training or validation TD error. We sweep
[5e-4, 1e-3, 2e-3, 5e-3, le-2, 2e-2, 5e-2, le- 2e-1]. Then, fit power law TD emor ~ 7h, and record the correlation and slope in Table 8. In general, we find that training and validation TD error increase with 7 (positive slope and correlation), but there not strong relationship between model size and the corresponding correlation or slope.
31

---

Table 8: Correlation between target update rate 7
Task
Metric
Critic width
Correlation
Slope (6)
h1-crawl
Critic loss
512
0.7365
0.1203
1024
0,9175
0.1348
2048
0.9302
0.1164
Validation critic loss
512
0.4639
0.0329
1024
0,6413
0,0244
2048
0.4751
0.0168
h1-stand
Critic loss
512
0,9056
0.3916
1024
0.9446
0,2035
2048
0.1857
0.0195
Validation critic loss
512
0.7777
0.1294
1024
0.3109
0.0196
2048
-0.6852
-0.0421
We additionally found that data efficiency is not very sensitive to our choice of т, as long as the value of 7 is reasonable. Following the same sensitivity analysis from Section 6, we find that varying by an order of magnitude from the bootstrapped optimal value only 19%. For comparison, varying the batch size by an order of magnitude yields data efficiency variation of up 52% (Table 5). Throughout the remainder of our experiments, we use "default" value of r 5e-3, which we find is within the "reasonable" interval and near the bootstrapped optimal value.
worsens the data efficiency by
D.5 Full TD-error curves
We provide full training and validation TD-error curves in Figure 13, completion to Figure 1. The summary statistics are marked with 'X' and correspond to the points used in Figure
D.6 Passive Critic Learning Curves
We provide the full validation TD error curves over training in Figure 14. In these plots the summary statistics are marked with 'X', and we provide Figure 15 as a completion to Figure
D.7 Data Efficiency Fits Jmme (a,N) N)
For the following four tasks, we fit data efficiency using the empirically best data efficiency for performance threshold Jmax across batch sizes for each (a, .) setting. Figures 16 and 17, show the fits for multiple values of J.
h1 h1-crawl
5.11e4
(2.5965/0) 0.15 (1.7067/N)01)
h1 h1-pole
9.43e3
[1+ (3.2226////
0.27
(2.50e12
(D.2)
h1 h1-stand
2.14e5
(1 (6.68e-1/o)
2.53
(1.7466/N)).97
0.97)
humanoid-stand
1.49e4
(1 (1.7866/0)
0.26
(3.75e/NN)
0.63
For the remaining tasks, use the available batch size.
32

---

h1-crawl-vo, B=256
h1-crawi-vo, B=512
h1-crawi-vo, B=1024
h1-crawi-vo, B=2048
Loos
10a
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
1e6
le6
1e6
1e6
h1-pole-vo, 8-256
h1-pole-v0, 8-512
h1-pole-vo, 8-1024
h1-pole-v0, B-2048
10a
Loos
10a
0.0
0.5
15
0.0
0.5
1.0
1.5
0.5
1.0
1.5
0.0
0.5
1.0
1.5
1c6
Steps
166
Steps
1c6
h1-stand-vD, B=256
h1-stand-vo,
h1-stand-v0, B=1024
h1-stand-vD, B=2048
10a
Loss
10a
1.00
1.00
Steps
Steps
Steps
Steps
humanold-stand, B=128
humanold-stand, B=256
humanoid-stand, B=512
humanold-stand, B=1024
10:
Loss
10
Steps
Steps
Steps
166
Steps
Training
Width=128
Width=512
Width=2048
Validation
Width=
Width=1024
training, grouped by critic width and passive critic width, UTD The summary statistics in Figure 2 are marked with 'X" and averages the last 10% training.
Figure 13: Training and validation TD TD-error
33

---

h1-crawi-vo, UTD-2
Crilic Width: 512
Critic Width: 1024
Critic Width: 2048
Volidation
Passive Width: 512
107
Passivv Width: 1024
Passive Width: 2048
200k
400k Stops
600k
800k
200k 400k
600k
800k
200k
400k Stops
600k
BO0K
Steps
h1-pole-vo, UTD=2
Critic Width: 512
Critic Width: 1024
Critic Width: 2048
Pessive Width: 512
Valldation Loss
Passive Width: 1024
Passive Width: 2048
20²
600k Steps
800k
600k
Steps
Steps
h1-stand-vo, UTD-2
Critic Width: 512
Critic Width: 1024
Critic Width: 2048
Passive Width: 512
Passive Width: 1024
Passive Width: 2048
Validation
1002
200k
400k
600k
600k
200k
400k
600k
800k
200k
400k Steps
600k
800k
Steps
Steps
Figure 14: Validation TD-error curves over training, grouped by critic width. The summary statistics in Figure are marked with 'X" and are averages over the last 10% of training.
DMC-medium, shared or,, By:
acrobot-swingup
4.29c5 (1 + (6.46c-1/o)) + (8.42c5/N) 1.39
cheetah-run
4.81e5 (1 + (4.30e-1/o) + (3.40e5/N) /N) 1.39
finger-turn
2.66e5 (1 + (1.08e0 /0)00 + (4.67e5/N)
fish-swim
6.28e5 (1 + (1.36e- (1.36e-1/o)u22 1/0)u.* + (2.7065/N) /) 1.99.
(D.3)
hopper-hop
3.52e5 (1 + (8.24e-1/o) + (3.1255/N)1.3)
quadruped-rum
1.39e5 (1 + (3.54e0/o) 0.98 + (1.6466/N)) 1.39)
walker-run
1.61e5 (1 + (2.85e0/o) 0.98 (6.0765/N)1.33))
DMC-medium, averaged environment:
DMC-medium averaged 3.72e5 (1 + (1.26e0/o)1.1 + (6.3365/N)0.80)
(D.4)
DMC-hard, shared a., 3y.
dog-run
4.45e5 (1
dog-stand
4.40e5
(199 //2) (.29
dog-trot
5.38e5 (1 [1+(6: (6.42e11-//) (7.53e5/N) /N 2)2
(D.5)
dog-walk
6.03e5 (1 + (3.096-11/0)07) (3.78o5) N)
humanoid-run
4.29e5 (1 + (2.04c0//)) (1.00066/N) 1.22
humanoid-walk
3.30c5 (1 + (1.1330 1.20)
0.73 +
(1.13c6/N)
34

---

h1-crawi-vD
3x20
Average Slope: 0055
3x
Average Slope: 00.3
Validation
2x100
2x10
Passive: 512
Main: 512
10a
Rossive:
10
Passive: 2048
Main: 2048
512
1024
2048
512
1024
2048
Main Critic Width
Passive Critic Width
h1-pole-v0
xx20
Average Slope: -0.27
Average Slope: 0.11
Validation
75200
Passive: 512
Main:
Passive: 1024
Main: 1024
Rossive: 2048
Main: 2048
512
1024 Critic Width
2048
512
1024
2048
Passive Critic Width
h1-stand-v0
Average Slope:
Averag age Slope:
Validation Loss
18:00
2x12
Possive: 512
Main: 512
Passive: 2048
Main: 2048
2x30
2x12
512
1024
2048
512
1024
2048
Main Critic Width
Passive Critic Width
Figure 15: Summary statistics for passive critic experiments, as completion Figure 4, run at UTD 2. Across multiple environments, increasing the main critic size is much more effective than increasing the passive critic size.
DMC-hard, averaged environment:
DMC-hard averaged
5.39e5 (1 +1+ (8.926-1/0)0.7 (6.6865/N)1.27)
(D.6)
D.8 Optimal Budget Partition
We provide plots analogous Figures 7 and in Figures 18 to 21 for DMC-medium and DMC-hard tasks. These data efficiency fits use the shared exponents a.,, By method described in Appendix B.
As shown in Figures 8, 20 and 21, however, the optimal UTD and model size for given budget Fo are unpredictable. We verify that these hyperparameters are fundamentally unpredictable this setting, running at least 50 seeds per UTD and model size at the fitted batch size, for a1-crawl, in Figure 22. Despite this, the for CF, achieves considerably lower uncertainty than Figure indicating that there is large range of "reasonable" hyperparameters corresponding to similar data and compute values.
35

---

h1-crawi-vo, oml
h1-crawi-vO, om2
h1-crawi-vo, amd
hl-craal-v0, um8
D: Dsta
8. Relturn
600
500
619K
34M
619K
619K
2M
34M
h1-pale-vO, a-1
h1-pole-vO, dw2
h1-pole-vO, awd
h1 h1-pole-v0, d=8
600
12
R.eeumm
21 Data
500
400
619K
9M
34M
619K
34M
619K
2M
34M
619K
2M
9M
34M
300
h1-stand-vvt, d-1
h1-stand-vo0, dw?
h1--stand-v0, awd
h1 h1-sstand-vO, o-8
600
500
D: Datata
400
300
619K
2M
9M
34M
619K
9M
34M
619K
2M
9M
34M
619K
2M
9M
34M
200
humaanidd-stand cm1
humanoid-sland, cm2
huraanidd-Saan. amd
humanoid-standd om
800
x1:5
700
D: Data
600
400
183K628K 2M
9M UTD
34M
163K628K 2M 9M
34M
183К626K 2M 9M
34M
183K628 2M
9M
34M
300
UTD
UTD
UTD
Figure 16: Data efficiency fits (o, N) for multiple performance thresholds J. grouped UTD ratio Each D, is fit independently.
h1-crawi-x0,
h1-crawi-vO,
h2-crawlivO, omd
h1-crawi-v0, a=
x3e5
Xle5
KId5
K165
700
D: Data
Reluum
2
2
600
500
619K
2M
BM
34M
619K
2M
9M
34M
619K
2M
9M
34M
619K
2M
9M
34M
h1-pole-vo, 9=1
h1-pole-vO,
h1-pole-v0, om
h1-pole-vO,
x3c5
xle5
xId5
K1:5
600
12
12
Dataa
8
500
400
619K
2M
9M
34M
619K
2M
9M
34M
619K
2M
9M
34M
619K
2M
9M
34M
h1-stand-v0,
h1-stand-v0,
h1 h1-stand-vv,, o-8
x3e5
Xlab
x1eb
600
500
0: D3ta
400
2
2
2
619K
2M
BM
34M
619K
2M
9M
34M
619K
2M
9M
34M
619K
2M
9M
34M
200
humanold stand,
humanoid stand,
humanoid nd, a
humanoid staddd--dddddd.m a=
800
x345
Xieb
xi46
K14b
700
D: Data
600
4
500
2
2
400
183K678K 34M
183K678K 2M 9M 34M
183K678K 2M
183K678K 2M
300
a: UTD
a. UTD
a. UTD
0r UTD
Figure 17: Same as Figure 16, but instead grouped by model size N.
36

---

DMC-medium
acrobot swingup
cheetah-run
finger
fish swim
Dyy Opdimal data
*
xbn
X1ed 32
X1e4
Perfoormacc level
350
800
64
32
32
600
250
R2-0.99
82-1.00
62.0.99
48
x1e15
xle15
32
64
hopperhop
qua guadruped-run
walker-run
Dyy Optimal data
x6dc
300
16
600
600
16
200
8
400
500
8--11.00
8200.99
02
400
150
200
16
32
48
xle15
32 xle15
8
16
32x1e15
17" Optimaal budget for
F
Optimal budget for
F Optimal budget for
DMC-hard
dog-run
dog-stand
dog-trot
dog-walk
Dy Optimal data
250
-800
750
6
6
Perforaacce
400
600
2
500
150
A2-1.00
250
3
200
3
3
100
200
6
x1e16
1
2
x1e16
8
x1e16
6
8
humanold-run
humanold-waik
Dy. Optimal data
x1e5
600
150
100
3
11.0000
#1m1.00
xle16 17: Optimal budget for
xlel6
1/: Optimal budget for
Figure 18: Optimal data D(Fo) for given budget Fo, completion of Figure 7.
37

---

DMC-medium
acrobot: swingup
cheetah
finger
fish swim
G Optimal compute
None
xioone
XNonne
XNone
Perfformance level
350
1016
800
10t6
600
600
250
8 ---0.9
2eset
- 0.98
500
R7
0.10
32
64
64
hopper-hop
quadruped-run
walker-run
Optimal compute
102
Nonn 1016
300
1040
600
600
200
400
500
0-0772
1015
400
150
200
16
32 48 xlel5
32
kle15
8
16
32x1e15
7y: Optimal budget for
Fy: Optimal budget for
Fj Optimal budget for
DMC-hat hard
dog-run
dog stand
dog-trot
dog -walk
Gyc Optimal compute
1e15
250
xleg
8800
16
750
16
12
Perfomance morommace
400
600
500
12
150
8
250
R2-0.90
200
8
100
200
8
x1e16
1
2
xle16
8
4
8
x1e16
humanold-run
humanold-walk
Cy: Optimal compuge
15111 1115
X1615
20
600
15
150
16
12
400
10
100
R' 96
#fe1.00
1y: Optimal budget for)
7): Optimal budget for]
Figure 19: Optimal compute C(Fo) for given budget Fo, completion of Figure 7.
38

---

DMC-medium
acrobot-swingup
cheetah-run
finger-tur
fish-swim
400
900
Perfol Performancc level
350
700
800
Oyy Optimal
700
8
-600
250
-500
-400
200
48
32
hopper-hop
guadruped-run
walker-run
(7-0..4
x7-0.16
17--0.14
700
Dry Optimal
16
600
600
200
400
-50
400
150
200
16
32
xle15
16
32
xle15
16
32 w1e15
7/: Optimal budget for
Optimal budget for
Optimal budget for
DMC-hard
dog
dog
dog
dog
Optimal UTD
R7--0
250
800
g7-0..0
800
Perfor monacacee lovel
200
400
400
300
150
400
200
100
1000
200
humesnidd-run
bu humanoid-walk
Oy.. Optimal UTD
Al 2.44
600
500
125
400
100
300
G1
75
200
: Op nottmaal budget for/
1y: Optimal budget
Figure 20: Optimal UTD ratio of (Fuu given budget Fu, completion of Figure 8.
39

---

DMC-medium
acrobot-swingup
cheetah-run
fingerturn
fishswii
Ny: Optimall mostll size
24
R7o0.17
400
14
700
-4
Rtwd. 36
900
42
600
Porffr oorforma ance level
770
16
300
600
20
250
16
500
16
Xiets
X1413
hopper-hop
quadruped-run
No: Optiinal model size
kie5
x1eg
xle5
700
600
250
24
200
400
400
150
200
16
32
48
xle15
8
16
32
x1e15
8
16
32
xle15
x/: Optimal budget
1y:
Optimal budget
7f: Optimal budget
DMC-har hard
dogirum
dog stand
dog trot
dog wogl
Nap: Optimnal model size
klo6
250
87+0.67
800
xich
500
R7-270
Perfor Performant mwmmmaancco lovell
200
600
600
400
300
150
400
200
200
R²
humanoid-run
humanoiddwwalk
No. Optimal model size
k1.
60.0.50
175
x1.6
g7a-1.0
600
500
400
100
300
75
200
8
xtwie
6
xlale
7/:
Optimal budget
fy: Optimal budget
Figure 21: Optimal model size (FF) for a given budget Fo, as a completion Figure
Cgg Optimal compuro
Dy: Optima
24
700
64
700
0gl Optimal
700
Wis Optimal model
00.4
700
16
600
600
600
600
6.00.00
500
07-3
500
8250.
500
500
32
0.5
320 48064D x1e15
320 480640 xle15
320
480 640
xle15
P/: Optimal budget for
77:
Optimal budget ]or
x): Optimal budget for
1): Optimal budgat for/
Figure 22: Optimal data, compute, UTD, and model size for given budget Fu, for 50+ seeds hi-crasl.
40


Reviews:
[{"review_id": "W1s0ES2a6V", "reviewer": "Reviewer_9ANn", "summary": "The paper investigates scaling laws in deep RL, focusing on the batch size and model capacity (layer widths). It is found that increasing the batch size only improves performance when model capacity is sufficently large but not when model capacity is small. A phenomenon called \"TD overfitting\" is posited, where using small networks produces larger TD errors on validation data when batch size is increased. The trend is reversed with larger netweorks.  Experiments are run with the BRO algorithm and continuous control benchmark tasks.", "questions": "Main questions:\n- The part starting at line 155 \"A mental model for TD-overfitting\": While the explanation makes intuitive sense, there are no experiments to support this specific mechanism. The idea seems to be related to having less interference between updates when the model is larger, it would be nice to investigate this hypothesis directly. For example, meausuring the amount of change in the value function prediction at validation states given small and large models. \nAlso, the findings in [1] could be relevant here, where they find that using TD updates seems to lead to more decoupled representations.  \n\n- It would seem like the target network update rate (for soft updates) or update frequency (for hard updates) would be an important factor affecting TD overfitting. In particular, it controls how fast the targets change which may induce more nonstationarity in the learning process. In line 180, nonstationarity is mentioned as a key part of the mechanism explaining TD overfitting. Have you explored this direction? Some investigations would be a great addition. This could be done with a single UTD ratio or other simplifying assumptions, given the computational costs. \n\n- Do you expect similar findings to hold for different learning algorithms such as Q-learning or PPO? The \"critic\" is used in a different way in those methods comapared to SAC.  (I do not expect additional experiments for this, just your opinion).\n\nClarification questions:\nThese impacted my understanding of the paper so answer to these would help properly evaluate it. \n- Equation 5.1: The notation is not clearly explained. What does $\\alpha$ and $\\beta$ refer to and how are the subsc\n\n- Fig. 3 shows curves fit to only 4 data points each. If there are 2 free variables in the equation, it's difficult to validate that the fitted curve is of the correct form. Many equations could fit reasonably well. It is mentioned in the text that a range of batch sizes work well for some of these settings. In that case, perhaps a heatmap/scatterplot similar to Fig. 4 with batch size vs. UTD on the axes and colors for the performance value could work better. We would be able to see which regions are good. Stars could denote the points with optimal performance for each UTD or model size.\n\n- In Fig. 4: How are the contours generated? Clearly, a continuum of parameters are not tested so there must be some method to interpolate and draw the curves.\n\n- Is the actor network also scaled up with the model size or is it only the critic? It seems like TD overfitting would be an issue mainly relevant to the critic.\n\n\n\n[1] \"Interference and Generalization in Temporal Difference Learning\" Bengio et al.", "limitations": "These are discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Generally, scaling deep RL agents is still challenging and improving our understanding is an interesting direction. As such, the paper tackles a topic releavnt to the community.\n\nThe presentation of the paper is nice and generally clear. They are some issues with unexplained notation and some parts to be clarified (as outlined in the Questions section).\n\nThe experimental findings about how model capacity can be an important factor to unlock batch size scaling is intriguing and fits well with results from prior works showing that scaling the critic is helpful. It's interesting to see another way that TD learning may lead to different learning dynamics than supervised learning, similar to previous work [1,2].\nI find this part to be the most interesting contribution of the paper and I would preferred a bit more investigation into the TD overfitting phenomena (see Questions).\n\nI am leaning towards acceptance currently and would be happy to raise my score given proper responses to the questions. \n\n[1] \"Learning Dynamics and Generalization in Reinforcement Learning\" Lyle et al.\n[2] \"Interference and Generalization in Temporal Difference Learning\" Bengio et al.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough and engaging feedback, and for a positive assessment of our work. We believe that the alternate perspectives and references that you discuss are quite relevant, and we will discuss them in the updated version of the paper.  We discuss your concerns below and we have also performed additional experiments to address some of the concerns.\n\n>  interference between updates\n\nThanks for pointing out this paper, we will cite it and discuss it in the updated version. Briefly, Bengio’20 discusses a phenomenon where TD learning leads to low interference due to a lack of temporal coherence in the value network. This is consistent with our analysis: a lack of temporal coherence suggests a poor target distribution, which in turn yields overfitting. A more detailed study of this phenomenon is warranted. However, a complicating factor is that measuring value changes can be challenging in continuous environments, and absolute value changes might not correspond to meaningful updates in the parameter space. We will perform proof-of-concept experiments on this and include them in a later version. \n\n> target network update rate\n\nIntuitively, target update rate should be important for the TD-overfitting phenomenon. In our preliminary experiments, we found that when holding other hyperparameters constant, TD-overfitting increases when setting the target update rate too high. That said, we saw that the effect of the target network update rate was overshadowed by setting a different learning rate and running training at a different UTD value & batch size. In other words, our initial experiments indicated that instead of changing the target network update rate we could find an alternate configuration of other hyperparameters that gave the same performance, and the target update rate was less necessary to account for. \n\nGiven a limited amount of computational resources, we had to make a conscious decision early on in the project regarding which hyperparameters to focus on and given the above preliminary findings, we decided to skip the target update rate. A full study of such effects is an exciting direction of future work, and we will rerun some of the preliminary experiments into the appendix of the final version of the paper. We do agree that there might be some nuanced effects of the target network update rate on TD-overfitting, which would be good to study in future. \n\n> Do you expect similar findings to hold for different learning algorithms\n\nWe ran 5 Atari-100K tasks with DQN at UTD=1, and for each $J$ estimated the data efficiency for the best batch size. Our preliminary results show that data efficiency improves with model size on average.\n\n| Model size |   $J$ = 0.16 |   $J$ = 0.27 |   $J$ = 0.38 | $J$ = 0.49   | $J$ = 0.60   |\n|-----------:|-------------:|-------------:|-------------:|:-------------|:-------------|\n|      150K  |        11880 |        19800 |        38010 | N/A          | N/A          |\n|      320K  |        10880 |        18130 |        26460 | 54130        | 82520        |\n|      790K  |        10310 |        17180 |        24050 | 40580        | 60010        |\n|     2.13M  |         8710 |        14520 |        20330 | 26800        | 36040        |\n|     6.47M  |         7860 |        13100 |        18340 | 23580        | 33270        |\n\n\nWe believe Atari is more susceptible to overfitting due to the deterministic nature of the environments. However, in general, since DQN also relies on TD-learning, we expect similar trends to apply. In contrast, PPO, when run with standard hyperparameter settings, relies on policy gradients more than TD-learning. Therefore, we expect that PPO scaling laws will be more similar to supervised learning scaling laws and simpler. Studying whether PPO is also affected by the TD-overfitting phenomenon we describe is an interesting direction of future work, particularly for applications using policy gradient methods, such as LLMs. \n\n> The notation is not clearly explained\n\nWe use $\\alpha, \\beta$ for the power coefficients and $a, b$ for multiplicative coefficients. The subscripts specify the equation that the terms appear in, e.g. $a_B$ in $B(., .)$, $a_J$ in $D_J(. ,.)$, $a_C$ in $C^*(.)$, etc. \n\n> Fig. 3 shows curves fit to only 4 data points each\n\nWe will provide heatmaps in the appendix. Note that the fit in Eq (5.1) has 4 variables, and is fit to 16 points. All 16 points are shown in Fig 3. We have considered different functional forms and selected this one as it had the best empirical performance and intuitive justification as discussed on l223. Specifically, with large $\\sigma$, it will dominate the denominator, requiring a small batch size, consistent with Rybkin’25. With small $\\sigma$, the model size term will dominate, consistent with Kaplan’20. \n\n> In Fig. 4: How are the contours generated?\n\nWe plot predictions from the fit (6.1). Because the fit is two-dimensional (it depends on $\\sigma$ and N), we are able to produce a continuum of predictions, which is necessary for proper extrapolation.\n\n> Is the actor network also scaled up\n\nIn line with prior work [1], we found that scaling critic-only leads to best performance. \n\nPlease let us know if there is anything else in the submission that you think we should improve or clarify for an improved score.\n\n[1] Nauman, Michal, et al \"Bigger, Regularized, Optimistic.\" NeurIPS 2024\n\n[2] Rybkin, Oleh et al. “Value-Based Deep RL Scales Predictably.”\n\n[3] Kaplan, Jared, et al. “Scaling Laws for Neural Language Models”. arXiv 2020"}, {"review_id": "MkzuFGHokd", "reviewer": "Reviewer_qAiA", "summary": "This paper introduces an empirical investigation of compute-optimal scaling laws for value-based deep reinforcement learning. The authors study the relationship between model size, batch size, and updates-to-data (UTD) ratio, propose empirical scaling laws (including power-law fits) for choosing these hyperparameters under compute or data budgets, and present a mechanistic explanation for “TD-overfitting” when batch size is too large for a given model size.", "questions": "1.\tHow would the proposed scaling laws change if learning rate were tuned jointly with batch size and UTD ratio? Could ignoring learning rate invalidate the power-law relationships claimed?\n\t2.\tHow robust are the observed scaling laws across different RL algorithms (e.g. SAC without BRO’s regularization, Simba, or discrete-action methods)? Is there evidence they generalize?", "limitations": "- The paper explicitly acknowledges (in Section 8) that learning rate and other hyperparameters were not included in the scaling analysis due to computational costs. This omission is non-trivial, since learning rate interacts strongly with batch size and UTD in standard practice.\n- The analysis is limited to a small number of continuous control tasks with one specific algorithm variant (BRO). The empirical fits may not generalize to other domains or algorithms.\n- The proposed scaling laws are based on empirical fits with relatively high error and may not provide precise prescriptions for practitioners. There is limited evidence that following these rules leads to improved final policy performance.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strenghts**\n\n- The paper tackles an important problem in RL practice: understanding the joint scaling behavior of batch size, model size, and UTD ratio under compute constraints.\n- It provides a clear empirical investigation that identifies a “TD-overfitting” phenomenon, offering intuition and diagnostics (e.g. the passive critic analysis) for why larger models tolerate larger batch sizes.\n- The authors propose empirical scaling laws (power-law fits) that could be useful in guiding hyperparameter choices for practitioners under budget constraints.\n\n**Weaknesses**\n\n- Learning rate is completely ignored. The paper even acknowledges in its limitations (Section 8) that learning rate is an important hyperparameter not included in the scaling analysis. This is a critical omission, since when changing batch size or UTD ratio, standard practice is to adjust learning rates (e.g. via linear scaling rules or other heuristics). The claimed power-law scaling rules are therefore incomplete and of limited practical value unless learning rate effects are explicitly controlled for.\n- Unclear evidence of improved performance. While the paper provides empirical fits and shows that model size and UTD ratio trade off along predicted contours, it does not clearly demonstrate that applying these scaling laws improves policy performance in practice compared to naive or baseline settings. The empirical fits are largely about validation TD-error, data efficiency metrics, and FLOPs—not about showing better returns in held-out or transfer tasks when using the proposed scaling rules.\n- Primarily empirical with limited scope. Despite claiming generality, the experiments focus almost entirely on the BRO variant of SAC and use a narrow set of continuous control benchmarks (DeepMind Control Suite and HumanoidBench). There are no experiments demonstrating that the scaling rules transfer to other RL architectures (e.g. standard SAC, Simba, or discrete-action methods like PQN). This limits the generality of the proposed laws.\n- Fit quality is mediocre. In multiple places the paper notes high relative error in the fits (e.g. ~50% uncertainty in batch size predictions). While they propose 2D fits to reduce grid search costs, this large error means that the scaling rules may not actually offer reliable guidance in practice.", "was_revised": true, "final_justification": "I raised my score since the authors could demonstrate the validity of their experiments including a discussion over the learning rate.", "rebuttal": "Thank you for the comprehensive and thoughtful response. We are glad to hear that our paper ‘provides a clear empirical investigation’ that ‘that could be useful […] for practitioners’. To address your points about improving the empirical evidence presented in the paper, we provide additional experiments to address the feedback. Due to large compute requirements and a delay in obtaining the required compute resources to rerun hyperparameter sweeps used to analyze scaling behaviors, some of our experiments are still running. We will report back and include them when they finish, within the author-reviewer discussion period.\n\nWe address the remainder of the concerns below. \n\n> Learning rate is completely ignored.\n\nWe have now run an additional analysis of learning rate dependencies. We find that (i) best learning rate decreases with increasing model size, correlation: -0.75, (ii) best learning rate decreases with increasing UTD, correlation: -0.46, (iii) best learning rate increases with increasing batch size, correlation: 0.42. With simple log-linear fits, we obtain a relative error of 37.5%. These results are consistent with existing literature McCandlish’18, Yang’22, Rybkin’25. We are running additional experiments to evaluate this hypothesis and will report back before the end of the discussion period.\n\nh1-crawl-v0:\n$R^2$ = 0.3417  \nRelative error: 43.5702%  \n$\\text{lr}^* ~ 4.4827\\text{e}-4 \\cdot (N/2.3\\text{e}6)^{-0.3112} \\cdot \\sigma^{-0.1273} \\cdot (B/512)^{0.3709}$  \n\nh1-stand-v0:\n$R^2$ = 0.5702  \nRelative error: 32.2800%  \n$\\text{lr}^* ~ 2.4727\\text{e}-4 \\cdot (N/2.3\\text{e}6)^{-0.2472} \\cdot \\sigma^{-0.2392} \\cdot (B/512)^{0.2701}$\n\nDespite the high relative error, we observe that data efficiency is similar within an interval of “reasonable” learning rates, which includes our fixed value; see the “high relative error” section below. By using the optimal learning rate fit in our results, we can improve the method’s performance.  We expect this method to follow the same data efficiency law as Eq (6.1), but have better coefficients $\\alpha_J, \\beta_J$, because Eq (6.1) is justified with asymptotic case analysis and is consistent with results from Rybkin’25. We are currently running experiments to fit this improved data efficiency law and will report them once they finish.\n\nA complete N-dimensional analysis of learning rate was not possible in this rebuttal period due to computational requirements: 4 learning rate values x 4 batch size values x 4 model size values x 4 utd values yields 256 required GPUs per environment. However, our initial analysis provides insight into how learning rate behaves and can be used by future studies to propose more effective scaling methods.\n\n> other domains or algorithms.\n\nIn addition to the two domains and 17 tasks in the original submission, we have now evaluated scaling laws on DQN in Atari. This is consistent with prior work on architectures [1,2] and RL scaling laws [3] which both showed findings generally applicable to many algorithms and environments. Interestingly, we observe that on Atari, batch size needs to be decreased with model size, which can be estimated from small-scale experiments using our workflow. We are investigating whether this is due to statistical overfitting or TD-overfitting and will report and include these results when they finish. Future work will focus on validating scaling laws on practical large-scale environments.\n\n| Model size |   $J$ = 0.16 |   $J$ = 0.27 |   $J$ = 0.38 | $J$ = 0.49   | $J$ = 0.60   |\n|-----------:|-------------:|-------------:|-------------:|:-------------|:-------------|\n|      150K  |        11880 |        19800 |        38010 | N/A          | N/A          |\n|      320K  |        10880 |        18130 |        26460 | 54130        | 82520        |\n|      790K  |        10310 |        17180 |        24050 | 40580        | 60010        |\n|     2.13M  |         8710 |        14520 |        20330 | 26800        | 36040        |\n|     6.47M  |         7860 |        13100 |        18340 | 23580        | 33270        |\n\nWe ran 5 Atari-100K tasks at UTD=1, and for each $J$ estimated the data efficiency for the best batch size. Our preliminary results show that data efficiency improves with model size on average.\n\n> Unclear evidence of improved performance.\n\nIn Figure 4, we show the compute-optimal settings for UTD and model size for each data budget. As we show in Appendix A.2, this scaling method is also data-optimal for each compute budget. For each compute budget $\\mathcal C$, we run the configuration with UTD $\\sigma^\\*(\\mathcal C)$, model size $N^\\*(\\mathcal C)$, and batch size $\\tilde B(\\sigma, N)$. We provide three baselines demonstrating the superior performance of compute-optimal scaling: for each compute budget, we can additionally run (i) UTD-only scaling at compute budget $\\mathcal C$ for a given model size, (ii) N-only scaling at compute budget $\\mathcal C$ for a given UTD, (iii) the compute-optimal UTD and model size and constant batch size (specifically, the fitted batch size from the lowest compute budget).\n\n| Aggregation | Compute optimal | UTD-only scaling | $N$-only scaling | Constant batch size |\n|-------------|-----------------|------------------|------------------|---------------------|\n| Average     | 1.00            | 1.26             | 1.11             | 1.03                |\n| Median      | 1.00            | 1.18             | 1.11             | 1.05                |\n\nIn the table, we record the ratio between the data efficiency in each baseline to the compute-optimal data efficiency, aggregated over environments and compute budgets. We show that compute-optimal scaling with the fitted batch size outperforms the baselines.\n\nWe further note that we expect the improvement from our approach to be more prominent as the gap between small-scale settings that we can readily experiment with and large-scale settings that we wish to find good hyperparameters for diverge from each other. For instance, tuning hyperparameters on a 1e6 parameter model and transferring to a 5e6 parameter model might be possible since the original hyperparameters are already in a good neighborhood; but transferring to a 2e9 parameter model (bare minimum for a language application) will likely work poorly unless a proper scaling law procedure is applied and hyperparameters are adjusted. As the only work establishing joint scaling laws for batch size, UTD, and compute, we believe future practitioners will rely on our results in deciding how to tune hyperparameters at large scale.\n\n> high relative error in the fits (50%)\n\nIndeed, we observe that our batch size fits have a moderate confidence interval of up to 50%. We observe this is because batch size in many cases has a wide range of values which all have near-optimal performance. Note that utilizing any of these batch values performs similarly, which means a practitioner can indeed use any value within this interval to obtain good performance and a moderate confidence interval is justified. In the table below, we group runs by UTD and model size, and bin suboptimal batch sizes. Then, we consider the data efficiency ratio between suboptimal and optimal runs, and average over UTDs and model sizes. We find that batch sizes within an interval around the best batch size $B^*$ perform reasonably, and performance degrades significantly with larger intervals. This is consistent with prior work on LLMs [4], and explains the relative error in the fit. In practice, this means that our fits, despite a large relative error, yield good results when using them to train models, which is anyways our ultimate goal.\n\n| Batch size range      | Data efficiency ratio |\n|-----------------------|-----------------------|\n| [1/16 B*, 1/8 B*]     | 1.52                  |\n| [1/8 B*, 1/4 B*]      | 1.38                  |\n| [1/4 B*, 1/2 B*]      | 1.26                  |\n| [1/2 B*, 2/3 B*]      | 1.22                  |\n| [1.5B*, 2B*]          | 1.16                  |\n| [2B*, 4B*]            | 1.18                  |\n| [4B*, 8B*]            | 1.19                  |\n| [8B*, 16B*]           | 1.30                  |\n\nSimilarly, we observe that there is a range of reasonable learning rates:\n\n| Learning rate range   | Data efficiency ratio |\n|-----------------------|-----------------------|\n| [1/4 lr*, 1/2 lr*]    | 1.39                  |\n| [1/2 lr*, 2/3 lr*]    | 1.35                  |\n| [2/3 lr*, lr*]        | 1.04                  |\n| [lr*, 1.5 lr*]        | 1.03                  |\n| [1.5 lr*, 2 lr*]      | 1.18                  |\n| [2 lr*, 4 lr*]        | 1.27                  |\n\nIn practice, we ran our main experiments at a constant learning rate of 3e-4. Running an additional sweep over learning rates of [1e-4, 2e-4, 6e-4] showed that only 2e-4 had superior performance in some cases. Since we are within the [lr*, 1.5 lr*] interval, the overall effect on performance is minimal.\n\nWe believe the submission is now much stronger after addressing your feedback. Please let us know if there is anything else in the submission that you think we should address for an improved score.\n\n[1] Nauman, Michal, et al \"Bigger, Regularized, Optimistic.\" NeurIPS 2024\n\n[2] Lee, Hojoon, et al. \"Simba: Simplicity bias for scaling up parameters in deep reinforcement learning.\" ICLR 2025\n\n[3] Rybkin, Oleh et al. “Value-Based Deep RL Scales Predictably.”\n\n[4] Grattafiori, Aaron et al. “The Llama 3 Herd of Models.”\n\n[5] McCandlish, Sam. “An Empirical Model of Large-Batch Training.”\n\n[6] Yang, Greg. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer"}, {"review_id": "4geCjpz1qv", "reviewer": "Reviewer_Vd6A", "summary": "This paper investigates optimal compute allocation strategies for value-based deep reinforcement learning methods using temporal-difference (TD) learning. The authors examine two primary scaling dimensions: model size and the updates-to-data (UTD) ratio. Through controlled experiments, they identify a novel phenomenon called \"TD-overfitting\" where smaller models exhibit poor generalization (higher validation TD-error) when trained with large batch sizes, while larger models benefit from increased batch sizes. The authors trace this phenomenon to poor-quality TD-targets generated by smaller networks and validate this hypothesis through passive critic experiments. Based on these insights, they develop empirical models for optimal batch size selection and provide guidance for jointly scaling model size and UTD ratio given fixed compute budgets.", "questions": "- Learning rate scaling: In supervised learning, learning rates are typically scaled proportionally with batch size (e.g., linear scaling rule). However, this study appears to use fixed learning rates across different batch sizes. Would the optimal batch size curves improve if learning rates were scaled proportionally with batch size? How might this affect the TD-overfitting threshold?\n- Double descent phenomenon: Supervised learning typically exhibits double descent, where performance initially degrades with increased model size before improving again. However, in your experiments, performance appears to improve gradually with model size without clear degradation phases. Have you observed any double descent phenomena in your scaling experiments across different batch sizes or UTD ratios? If not, what might explain the absence of this commonly observed pattern in the TD-learning setting?", "limitations": "- Offline RL applicability: The current analysis focuses on online RL settings where data collection and training occur simultaneously. In offline RL setups where the data budget is fixed and constrained, would these scaling relationships hold identically? The dynamics of TD-overfitting might differ when the dataset is static and finite.\n- Performance-optimal vs. compute-optimal scaling: In online RL with simulators, practitioners often prioritize finding the best performance while treating data collection as relatively inexpensive (assuming unlimited environment interactions). In such scenarios, how would one determine the optimal batch size and UTD ratio that maximizes final performance rather than compute efficiency? Would the identified scaling laws apply when the objective shifts from compute-optimal to performance-optimal training?\n- Extension to self-supervised learning: The TD-overfitting phenomenon stems from training on self-generated moving targets, which is not unique to RL. Self-supervised learning methods like BYOL, MoCO, and other contrastive approaches also rely on evolving target representations. The paper doesn't investigate whether similar \"self-target-overfitting\" occurs in these domains, limiting our understanding of how broadly this phenomenon applies to learning systems with non-stationary self-generated targets.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- Significant theoretical contribution: The TD-overfitting hypothesis provides a compelling explanation for seemingly contradictory practices in the RL community. While Atari-based studies typically employ very small batch sizes (e.g., 32), recent work on continuous control benchmarks using modern architectures like BRO, Simba, and MR.Q successfully leverages much larger batch sizes. The authors' insight that model capacity determines the optimal batch size regime elegantly reconciles these apparently conflicting approaches and suggests that architectural advances have enabled more aggressive batch size scaling.\n- Practical impact: This work directly addresses a critical challenge for practitioners who need to efficiently allocate compute resources when scaling RL training. The empirical models for joint optimization of model size and UTD ratio provide actionable guidance for real-world applications.\n\nWeaknesses:\n- Limited architectural generalization: While the focus on BRO provides depth, the transferability to other value-based architectures remains unclear. Different network designs, normalization schemes, and update mechanisms could alter the observed scaling relationships.\n- Broader implications for self-supervised learning: If the core issue stems from fitting to self-generated moving targets, this phenomenon might extend beyond RL to self-supervised learning methods like BYOL or MoCo, which also train on targets generated by their own evolving representations. The authors don't explore whether similar \"self-target-overfitting\" occurs in these domains, which could strengthen the theoretical foundation and broaden the impact of their insights.", "was_revised": true, "final_justification": "This paper provides a compelling TD-overfitting hypothesis that reconciles batch size practices across RL domains and offers practical scaling guidance. While its generality beyond BRO is uncertain, it makes a strong contribution to NeurIPS.", "rebuttal": "Thank you for the feedback and a positive assessment of our work. To address your concerns, we performed new experiments on alternative network designs for the critic and studied scaling of learning rates for training as reported below. Further, we would like to thank you for additional comments on self-supervised learning and double descent, which look at our work from new and interesting perspectives – we will make sure to add these into our discussion section. In this paper, we will focus on strengthening the existing claims, however, these alternative perspectives might lead to wider impact in future work. \n\n> Different network designs\n\nWe ran 5 Atari-100K tasks with DQN with a 2-layer MLP at UTD=1, and for each $J$ estimated the data efficiency for the best batch size. Our preliminary results show that data efficiency improves with model size on average.\n\n| Model size |   $J$ = 0.16 |   $J$ = 0.27 |   $J$ = 0.38 | $J$ = 0.49   | $J$ = 0.60   |\n|-----------:|-------------:|-------------:|-------------:|:-------------|:-------------|\n|      150K  |        11880 |        19800 |        38010 | N/A          | N/A          |\n|      320K  |        10880 |        18130 |        26460 | 54130        | 82520        |\n|      790K  |        10310 |        17180 |        24050 | 40580        | 60010        |\n|     2.13M  |         8710 |        14520 |        20330 | 26800        | 36040        |\n|     6.47M  |         7860 |        13100 |        18340 | 23580        | 33270        |\n\nWe agree that regularization can influence the behavior of TD error, and chose BRO to study TD overfitting and scaling in a setting that has shown consistent performance gain with critic scaling. Our findings and the fitted scaling laws should therefore be interpreted as conditional on an architecture that supports scalable TD learning. This type of conditional framing is consistent with prior work in model scaling, such as scaling laws for dense transformers [1], mixture-of-experts models [2], or state-space architectures [3], where scaling behavior is reported for architectures known to scale well. To clarify this for readers, we now emphasize in the main text that our scaling laws are specific to the BRO architecture, and the parameters of our fitted curves depend on regularization and architectural design.\n\n> Learning rate scaling\n\nWe have now run an additional analysis of learning rate dependencies. We find that (i) best learning rate decreases with increasing model size, correlation: -0.75, (ii) best learning rate decreases with increasing UTD, correlation: -0.46, (iii) best learning rate increases with increasing batch size, correlation: 0.42. These results are consistent with existing literature McCandlish’18, Yang’22, Rybkin’25. We are running additional experiments to evaluate this hypothesis and will report back before the end of the discussion period.\n\nBelow, we show simple log-linear fits of the best learning rate. We empirically find that the best learning scales less than linearly with the batch size.\n\nh1-crawl-v0:\n$R^2$ = 0.3417  \nRelative error: 43.5702%  \n$\\text{lr}^* ~ 4.4827\\text{e}-4 \\cdot (N/2.3\\text{e}6)^{-0.3112} \\cdot \\sigma^{-0.1273} \\cdot (B/512)^{0.3709}$  \n\nh1-stand-v0:\n$R^2$ = 0.5702  \nRelative error: 32.2800%  \n$\\text{lr}^* ~ 2.4727\\text{e}-4 \\cdot (N/2.3\\text{e}6)^{-0.2472} \\cdot \\sigma^{-0.2392} \\cdot (B/512)^{0.2701}$\n\n> Performance-optimal vs. compute-optimal scaling\n\nOur framework addresses this issue of finding the “reasonable batch size and optimal UTD ratio to attain the best final performance” as described in Problem 3.1, part 2 and Appendix A.3. Specifically, our framework can be applied in both ways: (i) given a certain performance, minimize the budget required to reach it, and (ii) given a certain budget, maximize performance that can be achieved. Both of these applications can be derived from the functional form of the scaling law that we study. Practitioners can solve for either question as they find necessary, we do not intend to imply that one question is more important than the other. \n\n> self-supervised learning methods like BYOL or MoCo\n\nIn this paper, we focus on a studying of SOTA value-based RL methods, which we believe is crucial for future work on large-scale RL. It is an interesting point that self-supervised learning can have some similarities with the analysis proposed in our paper. We would like to note that in value-based RL, an additional source of potential instability arises from the fact the actor is updated continuously, so the values change even faster and due to more factors than in self-supervised learning. It is likely that the value-based RL setting is more challenging. However, self-supervised learning is more mature and allows for larger-scale experiments than value-based RL, and so is an exciting direction of future work.\n\n> Double descent phenomenon\n\nThis is a great question! We have not observed a double descent phenomenon in the range of model sizes we evaluated. We would like to note that two complications appear in value-based RL that might impact the double descent presence. First, the notion of overfitting is different in value-based RL since overfitting can occur due to noise in the targets, as discussed in our paper. Second, the notion of network capacity is different, since the network must not only fit the targets at the end of training, but also all targets as they change during training, requiring more capacity. Because of this, further work is needed to analyse whether double descent is present in value-based RL. \n\n> Offline RL applicability\n\nWe expect that some of our findings transfer directly to offline RL as the phenomenon of TD-overfitting should be similarly present. However, due to increased distribution shift between the data and the policy, offline RL has other complicating factors such as the need to tune a ‘conservatism’ coefficient or a BC regularization loss. Addressing this is an exciting direction of future work. We will add this to the discussion section of the paper.\n\n[1] Kaplan, Jared, et al. “Scaling Laws for Neural Language Models”. arXiv 2020\n\n[2] Krajewski, Jakub, et al. \"Scaling laws for fine-grained mixture of experts.\" arXiv 2024\n\n[3] Gu, Albert and Dao Tri. \"Mamba: Linear-Time Sequence Modeling with Selective State Spaces.\" COLM 2024\n\n[4] Rybkin, Oleh et al. “Value-Based Deep RL Scales Predictably.”\n\n[5] Yang, Greg. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer\n\n[6] McCandlish, Sam. “An Empirical Model of Large-Batch Training.”"}, {"review_id": "DIA3aInxyr", "reviewer": "Reviewer_R4Hr", "summary": "This paper presents a systematic study on how model size, UTD, and batch size affect the performance of value-based deep reinforcement learning (RL) algorithms.\nThe key insight is the identification of \"TD-overfitting,\" where large batch sizes degrade generalization in small models but improve it in larger ones (e.g., Takeaway 2).\nBased on empirical observations, the authors propose practical scaling rules for optimal compute allocation across model size, UTD, and batch size (Scaling Observations 1-4).", "questions": "See weaknesses.", "limitations": "The empirical results are limited to robotic tasks, as described in Section 8. I think this limitation does not diminish the quality of this paper, as long as there is a solid motivation to study robotic tasks.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n* The paper is well-written, and the empirical results are generally strong.\n* The identification of \"TD-overfitting\" is a novel and insightful contribution. The resulting scaling rules are well-motivated by this finding and appear practical.\n\n**Weaknesses**\n\nThe experimental setup lacks clarity and persuasiveness.\n\n=====\n\nWhy do the authors focus specifically on the BRO architecture? While I understand that regularization is crucial in practice, it can also influence the behavior of TD error. I am concerned that this may affect the interpretation of the TD-overfitting phenomenon. If regularization significantly alters the behavior, this should be clearly discussed and reflected in the main takeaways.\n\n=====\n\nWhy do the authors focus primarily on robotic tasks? To convincingly demonstrate the TD-overfitting phenomenon, it would be more natural to include experiments based on value-iteration-style algorithms, such as DQN. Continuous action tasks should introduce additional approximation errors when evaluating the TD errors compared to discrete action settings, specifically when taking the expectation over $\\mathbb{E}_{a \\sim \\pi}$. \n\nIs there a specific motivation for studying robotic tasks in the context of TD-overfitting? If so, this rationale should be clearly articulated.\n\n=====\n\nThe TD-overfitting phenomenon seems to be evaluated only in the 1-step TD setting. Since $\\infty$-step TD corresponds to the unbiased return estimator, I expect that multi-step TD may be less susceptible to TD-overfitting than the 1-step version. I think evaluating multi-step TD and analyzing its behavior would offer stronger support for the TD-overfitting phenomenon.\n\nCould the authors provide either theoretical implications or empirical evidence regarding this point?\nI believe a small-scale experiment, such as one using MinAtar, would be sufficient to validate the scaling rules in alternative settings.\n\n* Rainbow: Combining Improvements in Deep Reinforcement Learning : https://arxiv.org/abs/1710.02298", "was_revised": true, "final_justification": "The response addressed my concerns about the experimental setup (e.g., model architecture, robotic task, and n-step TD).", "rebuttal": "Thank you for your positive and constructive feedback. To address your concerns, we are including several new experiments and discussion as requested, which we believe further strengthen our submission. In particular, we are adding new results on different architectures beyond BRO, n-step TD, and evaluations of scaling on Atari games, a domain quite distinct from robotics. **Please let us know if these responses address your concerns, and if so, we would be grateful if you are willing to raise your score.**\n\n> BRO architecture?\n\nOur decision to focus on the BRO architecture was motivated by findings in prior work [1,2], which show that naively scaling critic models often leads to degraded performance. BRO incorporates architectural and regularization components (i.e. BroNet) that have been shown to enable stable critic scaling with consistent performance gains. We agree that regularization can influence the behavior of TD error, and chose BRO to study TD overfitting and scaling in a setting where scaling is feasible. Our findings and the fitted scaling laws should therefore be interpreted as conditional on an architecture that supports scalable TD learning. This type of conditional framing is consistent with prior work in model scaling, such as scaling laws for dense transformers [3], mixture-of-experts models [4], or state-space architectures [5], where scaling behavior is reported for architectures known to scale well. To clarify this for readers, we now emphasize in the main text that our scaling laws are specific to the BRO architecture, and the parameters of our fitted curves depend on regularization and architectural design.\n\n> Robotic tasks?\n\nWe chose to focus primarily on robotic tasks because our goal is to investigate whether model scaling in value-based RL can yield predictable improvements, similar to trends observed in supervised learning. This is a non-trivial question as prior work [1,2,6,7] has highlighted that naively increasing model size or UTD often leads to unstable or degraded performance in RL. The considered robotic tasks are commonly used in recent RL model scaling studies [2,6], enabling direct comparison and contextualization of our results. \n\nThat said, we agree that evaluating scaling behavior in other domains, such as discrete action environments with value-iteration-style algorithms, is an important direction. To this end, we are currently running a subset of Atari-100K experiments with DQN, and preliminary results shown in the table below confirm that model scaling can yield monotonic improvements in performance in discrete-action tasks as well and we will expand this with the TD overfitting analysis in the camera-ready version. Furthermore, we have expanded the limitations section to explicitly state that our current findings may not generalize across all RL settings (e.g. goal-conditioned, sparse reward, or pixel-based tasks), and require further validation.\n\n| Model size |   $J$ = 0.16 |   $J$ = 0.27 |   $J$ = 0.38 | $J$ = 0.49   | $J$ = 0.60   |\n|-----------:|-------------:|-------------:|-------------:|:-------------|:-------------|\n|      150K  |        11880 |        19800 |        38010 | N/A          | N/A          |\n|      320K  |        10880 |        18130 |        26460 | 54130        | 82520        |\n|      790K  |        10310 |        17180 |        24050 | 40580        | 60010        |\n|     2.13M  |         8710 |        14520 |        20330 | 26800        | 36040        |\n|     6.47M  |         7860 |        13100 |        18340 | 23580        | 33270        |\n\n> Continuous action space?\n\nWe thank the reviewer for this suggestion. We agree that continuous action tasks introduce additional approximation challenges, particularly when estimating the TD target via action sampling or optimization (as opposed to exact maximization in discrete settings). This does make theoretical analysis of TD error more complex. However, our main motivation for focusing on robotic control tasks is to study TD overfitting in high-dimensional, realistic settings where RL is frequently deployed, e.g. in applications such as sim-to-real. To this end, we include experiments on challenging, high-dimensional environments, such as the simulated Unitree robot from HumanoidBench, to ground our analysis in realistic control settings. We agree complementing this with discrete action experiments would offer a useful theoretical contrast, and we are working toward that in parallel (see our response above on Atari-100K).\n\n> n-step TD?\n\nWe thank the reviewer for this insightful suggestion. We agree that, theoretically, increasing $n$ in multi-step TD should reduce the susceptibility to TD-overfitting - as $n \\rightarrow \\infty$, the learning target approaches a Monte-Carlo return estimate, effectively turning the problem into supervised regression on full returns rather than bootstrapped targets. In this sense, 1-step TD represents the most bootstrapped and thus most overfitting-prone setting. However, in the off-policy setting, multi-step TD introduces additional bias, especially when learning from older transitions that reflect outdated or suboptimal policies. In practice, this tradeoff between bias and variance can confound the expected reduction in TD-overfitting as well. To this end, our preliminary experiments with the BRO architecture suggested that n-step TD improves performance with small models, where variance reduction helps stability. However, performance at large n degrades with larger models, likely due to increased off-policy bias. We are currently running more comprehensive experiments to validate these observations and plan to include these results in the final version. \n\n\n[1] Bjorck, Nils, et al. \"Towards deeper deep reinforcement learning with spectral normalization.\" NeurIPS 2021\n\n[2] Nauman, Michal, et al \"Bigger, Regularized, Optimistic.\" NeurIPS 2024\n\n[3] Kaplan, Jared, et al. “Scaling Laws for Neural Language Models”. arXiv 2020\n\n[4] Krajewski, Jakub, et al. \"Scaling laws for fine-grained mixture of experts.\" arXiv 2024\n\n[5] Gu, Albert and Dao Tri. \"Mamba: Linear-Time Sequence Modeling with Selective State Spaces.\" COLM 2024\n\n[6] Lee, Hojoon, et al. \"Simba: Simplicity bias for scaling up parameters in deep reinforcement learning.\" ICLR 2025\n\n[7] D'Oro, Pierluca, et al. \"Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier.\" ICLR 2023"}]

Author rebuttal:

Meta-review: a) Scientific Claims and Findings: The paper investigates compute-optimal scaling for value-based deep RL. They charactrize TD-overfitting as small models perform poorly with large batch sizes while larger models benefit from them. The authors develop empirical scaling laws for optimally allocating compute between model size and updates-to-data ratio.

b) Strengths: The work addresses a critical scaling problem with novel theoretical insights that reconcile conflicting batch size practices across RL domains. The TD-overfitting hypothesis provides compelling mechanistic explanations supported by comprehensive empirical investigations and delivers practical scaling guidance.

c) Weaknesses and Missing Elements: Primary limitations include narrow focus on BRO architecture raising generalizability concerns, initial omission of learning rate analysis, and high relative errors in scaling law fits. The experimental scope was limited to continuous control with insufficient validation on discrete action spaces.

d) Decision Rationale: The AC recommends the paper for acceptance due to its novel TD-overfitting contribution and practical scaling guidance for an increasingly important problem. The authors demonstrated extensive additional experiments that substantially strengthened the paper.

e) Rebuttal Discussion Summary: Reviewers raised concerns about learning rate omission, mechanistic understanding, and architectural limitations. Authors addressed these through comprehensive learning rate analysis, temporal coherence experiments, and additional DQN results, leading three reviewers to maintain or raise scores.
Decision: Accept (poster)