Scaling Laws for Imitation Learning in Single-Agent Games
Authors: Jens Tuyls
Dhruv Madeka
Kari Torkkola
Dean Foster
Karthik R Narasimhan
Sham M. Kakade
Venue: Accepted by TMLR ()

Abstract: Imitation Learning (IL) is one of the most widely used methods in machine learning. Yet, many works find it is often unable to fully recover the underlying expert behavior, even in constrained environments like single-agent games. However, none of these works deeply investigate the role of scaling up the model and data size. Inspired by recent work in Natural Language Processing (NLP) where “scaling up” has resulted in increasingly more capable LLMs, we investigate whether carefully scaling up model and data size can bring similar improvements in the imitation learning setting for single-agent games. We first demonstrate our findings on a variety of Atari games, and thereafter focus on the extremely challenging game of NetHack. In all games, we find that IL loss and mean return scale smoothly with the compute budget (FLOPs) and are strongly correlated, resulting in power laws (and variations of them) for training compute-optimal IL agents. Finally, we forecast and train several NetHack agents with IL and find our best agent outperforms the prior state-of-the-art by 1.7x in the offline setting. Our work both demonstrates the scaling behavior of imitation learning in a variety of single-agent games, as well as the viability of scaling up current approaches for increasingly capable agents in NetHack, a game that remains elusively hard for current AI systems.

Published in Transactions on Machine Learning Research (12/2024)
Scaling Laws for Imitation Learning in Single-Agent Games
Jens Tuyls", ; Dhruv Madeka?, Kari Torkkola?, Dean P. Foster Karthik Narasimhan',
Sham Kakade2
Princcton University, AAmazon, HHrward Universiby, University of Pennsyloania
jtuylsBes princeton. wwwgllseessiinnooon..oo
Reviewed on OpenReview: https: / openreview. net/ forum? id= qQVchyF333
Abstract
Imitation Learning (IL) is one of the most widely used methods in machine learning. Yet. many works find it is often unable to fully recover the underlying expert behavior (Wen et al. 2020: Jacob et al. 2022). even in constrained environments like single-agent games (De Haan et al. 2019 Hambro al. 2022b). However, none of these works deeply investigale the role of scaling up the model and data size. Inspired by recent work in Natural Language Processing (NLP) (Kaplan et al. 2020 Hoffmann al. 2022) where "scaling up" has resulted in increasingly more capable LLMs, we investigate whether carefully scaling up model and data size can bring similar improvements in the imitation learning setting for single-agent games. We first demonstrate our findings on a variety of Atari games, and thereafter focus on the extremely challenging game of NetHack. In all games, we find that IT. loss and mean return scale smoothly with the compute budget (FT.OPs) and are strongly correlated, resulting power laws (and variations of them) for training compute-optimal IL agents. Finally, we forecast and train several HetHack agents with IL and find our best agent outperforms the prior trate-of-the-art by 1.7x in the offline setting. Our work both demonstrates the scaling behavior of imitation learning in a variety of single-agent games, as well as helps narrow the gap between the learner and the expert in NetHack, a game that remains elusively hard for current AI systems!
1 Introduction
While conceptually simple, imitation learning has powered some of the most impressive feats of AI in recent years. AlphaGo (Silver et al., 2016) used imitation on human Go games to bootstrap its Reinforcement Learning (RL) policy. Cicero, an agent that can play the challenging game of Diplomacy, used an IL-based policy as an anchor to guide planning (Jacob et al.] (022). Go-Explore, a method for hard-cxploration problems which solved all previously unsolved Atari games, used self-imitation learning in its robustification phase Feofffe et al. 2021
Despite its prevalence, several works have pointed out some of the limitations of IL. De Haan al. (0119) and Wen et al. (2020) call out the issue of causol confusion, where the IL policy relies on spurious correlations to achieve high training and held-out accuracy, but performs far worse than the data-generating policy, even in single-agent Atari games. Jacob et al. (2022) have mentioned similar issues for policies learning from human games: they consistently underperform the data- nata-gennrottng policy. However, many of these works, the role of model and data size is not deeply investigated. This is especially striking considering the increasingly impressive capabilities that recent language models have exhibited, mostly
consequence of scale. In a series of papers trying to characterize these improvements with scale starting with Hest.ness et al. 20177) and RRoeenflld et al. (2019), it has been shown language modeling loss (i.e. cross-entropy) scales smoothly with model size and number of training tokens (Kaplan et al. 2020 Hoffmann et al.) (002).. If we think of language models as essentially performing "imitation learning" on text, then a natural next question is
!Code:
in-ganes

---

Published in Transactions on Machine Learning Research (12/2024)
NetHack
Battle Zone
Breakout
1.5
2.7
1.0
1e15
2013
Ke15
1.14
1e16
2014
0.5
Dev loss
0.5
2035
Selb
5210
loss
1.0
2034
2014
2455
52:14
2014
0.6
2235
5033
5:17
0.1
3235
1019
letn
10'
30'
10'
30*
10
10
10
10
10
10"
30
10*
Parameters
Parameters
Parameters
(a) IsoFLOP curves
2.0
1.0
20647.69
0.11
3.90e" C 47 120e
2 11..ee.---014..00 4
0.5
Dev Loss
1.0
Dov
0.6
0.1
0.1
1015
1014
10 FLOPs
10
0
10
1014 FLOPs
10
1014
1014
1015
10kG FLOPs
100
199
(b) Optimal loss vs. FLOPs
10"
10a
10a
10²
Parameters
10a
Parameters
1.5
10²
10a
10f
log
0.61 C
log/== 10.667 1
10'
log
-1..22 loge
101
10th
10
10
10
103
2014
10
10 FLOPs
10
10
1014
10t
10ta
10
10
FLOPs
FLOPs
(c) Optimal parameters vs. FLOPs
100
10
10"
Samples
Samples
Samples
100
10a
10'
logo
Cog
kgg0
0.51-109C
log 00:00-0.33-000 log
104
10th
1014
10
10
105
10
10x
10
1014"
16
30t
1014
10
109
FLOPs
FLOPs
FLOPs
(d) Optimal samples FLOPs
Figure BC loss scaling. We train a wide range of model sizes across several orders of magnitudes of FLOP budgets. We plot the validation loss for each model, with fitted parabolas per IsoFLOPs curve (a). We then regress the loss minima (b), the loss-optimal number of parameters (c), and the loss-optimal number of samples (d) their corresponding FTOPP budgets. We find clear power law trends for Nethack (first column), Battle Zone (middle column), and Breakout (last column). The full list of Atari results can be found in Appendix J
whether these results extend to IL-based agents in games, and whether scale could provide similar benefits and alleviate some of the issues mentioned earlier on.
2

---

Published in Transactions on Machine Learning Research (12/2024)
In this paper, we ask the following question: How does compute in terms of model and data size affect the performance of agents trained with imitation learning in the single-agent game selling? We first focus on several Atari games with dense rewards which allows us to demonstrate our findings on a variety of games. However, since Atari games have all been solved at this point, there is not much room for further improvement in terms of scaling up. To demonstrate the potential of scaling up II, our core focus will be on the extremely challenging game of NetHack, a roguelike vidco game released especially well-suited and interesting domain to study for several reasons. First, it is procedurally generated and highly stochastic, disqualifying approaches relying heavily on memorization instead of generalization, such as Go-Explore (Ecoffet et al., 2021). Second, the game is partially observed, requiring the use of memory, potentially for thousands of steps due to the game's long-term dependencies. Finally, the game is extremely challenging for current AI systems, with current agents reaching scores nowhere close lo average human performance The best agent on NetHack is a purely aule-based system called AutoAscend (Hambro et al. 2022a), with RI. approaches lagging behind (Hambro et al.] 2022b; Küttler et al.] 2020 Mu ct al.] 20022 Mazoure et al. (023). Even just recovering this system hard, with Hambro et al. (2022b); Piterbarg el al. [2023a b) all reporting that the best neural agents still far underperform the system's 's mean return in the environment, causing the authors to call for significant research advances. We instead investigate whether simply scaling up IL can help close some of this gap.
1987. NetHack is an
Contributions. We train suite of neural Atari and NetHack agents with different model sizes using Behavioral Cloning (BC) to imitate expert policies and analyze the loss and mean return isoFT.OP profiles. We find the optimal cross-entropy loss scales as a power law plus constant in the compute budget, and we use two different methods to derive scaling laws for the loss-optimal model and data sizes. We then relate the cross-entropy loss of our trained BC agents to their respective mean return when rolled out in the environment, and find that the mean return follows an inverse power law plus constant with respect to the optimal cross-entropy loss, showing improvements in loss generally translate in better performing agents. We use our scaling law derivations to forecast the training requirements of several compute-optimal neural BC agent for NetHack. These forecasts are then verified to achieve the loss mean return predicted by our scaling laws. Furthermore, our best agent outperforms prior neural NetHack agents by at least 1.7x in all settings, showing scale can provide dramatic improvements in performance for BC. We briefly exlend our results to the RL setting, where we also train a suite of NetHack agents using IMPALA (Espeholt et al. 2018 and again find that model and data size scale as power laws in the compute budget. Finally, we also study the effect of partial observability on BC loss. Our results demonstrate that the improvements in imitation learning performance for single-agent games with dense rewards can be described by power laws (and variations of them). This suggests carefully scaling up model and data size can provide big boosts to imitation learning performance in single-agent games, helping to narrow the gap between the learner and the expert.
2 Preliminaries
We now introduce the formal setup for behavioral cloning. We assume the environment can be described by a Partially Observable Markov Decision Process (POMDP) (S, T, R, 7), with states S, transition function T, action set A, possible observation emissions O, reward function R(s, a), and discount factor 7.
In the behavioral cloning setup, we don't assume access to the rewards but instead assume access to dataset D consisting of trajectories by multiple (possibly sub-optimal) demonst acting in the environment. However, in this work, they are assumed to all come from the same expert policy т. The goal is to recover this expert policy. To do this, a Iearner Tg will optimize the following cross-cntropy loss:
(So, dus $1, ..,...) of states and actions. These trajectories can be generated
[(0)= -B(hh [log (T(aa
(1)
where by can include part or the entircty of the history of past states and actions.
2The average overall human performance is around 127k Hambrr et al. 2022b). while the current best performing NetHack agent gets score of 10k. 3 Pllease refer section for a discussion of why we need this requirement.
3

---

Published in Transactions on Machine Learning Research (12/2024)
3 Experimental setup
We analyze the scaling behavior of agents trained with BC in two domains: (1) Atari and (2) NetHack. The former serves to test the validity of the scaling laws in a range of games, while the latter tests the performance gains of scaling in an extremely challenging and unsolved game.
Whenever we report FLOP or paramcter counts, we are referring their effective counts, which we define only including the parts of the network that are being scaled, similar to Hilton (2023) (see Appendix E for full details). Please see Appendix G for details on all hyperparameters.
3.1 Atari
We chose the following set of 8 Atari games: Battle Zone, Q*bert, Name This Game, Phoenix, Space Invaders, Bank Heist, Boxing, and Breakout. For detailed discussion how these games were selected, please refer to [ppendix F] We then perform the following steps for each game. First, train CNN-based agent with PPO (Schulman et al., 2017) in order to get an expert agent. Second, we gather dataset of about 1B samples consisting of rollouts of the expert agent. We then train a family of CNN-based agents on this dataset using BC, varying the width of the core CNN and the final linear layer (see Appendix E The total number of parameters ranged from 1 to 5M.
3.2 NetHack
We train Transformer-based agents on the NLD-A dataset (Hambro et al. 2022b) varying both the width and depth (i.e. number layers) of the model (see [Appendix E). The total number of parameters ranged from 200k 200M. While the original NLD-AA dataset already contains around 3B samples, we extended the dataset to around 55B samples (NT.D-A--L) by collecting more rollouts from AutoAscend (i.e. the data-generating policy).
4 Scaling up imitation learning
This section is structured as follows. We first investigate the role of model size and number of samples with respect to cross-entropy loss (subsection 4.1p. While intuitively it feels like lower loss should result in a better agent, we verify this by directly investigating the role of model size and number of samples with respect to the environment return (subsection 4.2], and relating these results to the loss results. Finally, we also show a possible extension of our analysis to the RL setting (subsection 4.3).
4.1 Scaling laws for BC loss
To investigate the role of model size and number of samples with respect to cross-entrrop loss, we follow two approaches that are similar to the ones used in Hoffmann et al. (2022)
Approach #1: isoPLOP profiles. For Atari, we train up 12 different model sizes, ranging from 5M. For NetHack, we train 10 different model sizes, ranging from 200k to 200M. For all domains, we train FLOP budgets as low as lel4 and up 5e18. In Figure we plot the loss evaluated on held-out set of about 100 (for Atari) and 10k (for NetHack) trajectories against the parameter count for each FLOP budget. Similarly to Hoffmann et al. (022),, we observe clear concave-up parabolas with well-defined minima at the optimal model size for given compute budget in all games. We also find that loss decreases with increasing FLOP budgets. We take these loss-optimal data points to fit three regressions: one that regresses the parameters on the FLOPs, another that regresses the samples on the FLOPs, and a final one that regresses the loss on the FLOPs. These regressions give risc the following power laws plus a constant:
Nopt = Cx + by Dopt = = +bp Lopt C² + bL.
(2)
where Nopt indicates the loss loss-optimal model size, Dopt the loss-optimal number of training samples, Lopt the minimal validation loss, and C. the compute budget in FLOOPs. The a, and b, variables indicate additional
4

---

Published in Transactions on Machine Learning Research (12/2024)
NetHack
Battle Zone
Breakout
1+1
563
163
Returns 20
Retums
522
Returns
202
162
Enpert
2015
142.4
34.55
56.1
2013
Eqert
2035
Espert
2e1
3034
5e17
2017
Se18
1015
2016
2055
1el
105
10f
10"
106
10a
10a
10a
10F
107
10
10f
105
Parameters
Parameters
Parameters
(a) TsoFT.P curves
16a
Se²
Returts
Returns
Returs
20
Zea
1n3
10
Expert
Expert
Expert
5ua
13.86e c-0om 1,-36-11-
(3.25 C-34 1y10
A
(1.5
+2444e -y-
1014
1044
10
10
1044
106
1014
1027
1044
104s
10t4
189
FLOPSs
FLOPS
FLOPS
(b) Optimal return vS. FLOPs
3ea
100
Farameter
101
Parameters
1.0
100
log
4,92
0.51 logc
10²
log
16,82 0.76
llgg
10a
log
21.59 0.0000000000000 0.00-
165
10
1014
1027
FLOPss
FLOPs
FLOPs
(c) Optimal parameters vs. FLOPs
1017
10
10
20l4
Samples
1015
Samples
Samples
3.0
log 3.12 0.69 CogC
log - 4.19 + lgc
log0 4.41 + 0.35 Ing.
1003
1034
10
10
109
10
1027
10a
FLOPs
FLOPs
FLOPs
(d) Optimal samples vs. FLOPs
Figure 2: BC return scaling. We train a wide range of model sizes across several orders of magnitudes of FLOP budgets (same models as in Figure lab and plot their average return in the environment. (a). We then regress the optimal returns (b), the return-optimal number of parameters (c), and the return-optinal number of samples (d) on their corresponding FLOP budgets. We find mostly clear power law trends for Nethack (left), Battle Zone (middle), and Breakout, (right). Full Atari results can be found in Appendix
fitted paramcters. Note that sometimes we set the b, variables to since they model the threshold of the power laws, which we don't always observe. For many of the loss power laws however, we do observe it and hence fit by well (e.g. see left. plot of Figure 1bp. We refer to the legends of Figure 1c. Figure 1d.
5

---

Published in Transactions on Machine Learning Research (12/2024)
Table 1: Fitted power law coefficients NetHack. We list the scaling coefficients for model size (a) and number of samples (3) for all three settings. 95% Cls are noted in parentheses, where the delta method was used for the parametric fit parameters (see Appendix H).
IsoFLOP profiles
Parametric fit
Setting
at
3
a
1. BC Loss
0.61 (0.52, 0.71)
0.39 (0.29, 0.48)
0.590 (0.587, 0.593)
0.410 (0.407, 0.413)
2. BC Return
0.51 (0.43, 0.59)
0.49 (0.41, 0.57)
0.605 (0.601, 0.610)
0.395 (0.390, 0.399)
and Figure 1b for sample values of the power law exponents a, 3, and 7, respectively, These figures indicate that increase FLOPs, we expect loss to decrease according to power law when appropriately increasing the size of the model and the size of the data according to their own power laws well.
Approach #2: parametric fit. Instead of only fitting the loss-optimal points as was done in approach #1 above, also fit all points from Figure la to the following quadratic form:
log L(N, D) = 8o + BN log N + BD log
+ (log N)² + AND log N log D + BD (log D)
(3)
If we only look at the linear terms here, notice that this loss has the form of Cobb-Douglas production function:
L(N, D) exp(3n) N D&D,
(4)
where we can think of parameters and samples D as inputs that affect how much output (i.e. loss) gets produced. We then take the functional form in Eqqation and minimize the loss subject to the constraint that FLOPPs D) 6NTT To do this, we used the method of Lagrange multipliers to get the following functional forms for Nopt and Dopt (see Appendix A for full derivation):
' Dopt - G
C
Nopt G )
BD - 3N
where
G = exp
(5)
20 23ND + 23
We find that
28 n2 AND
and B
23p2 23 +28 We compare the two approaches for NetHack
28 22 8yp
28 2 +2
in Table
We find both approaches to give similar scaling components for model and data size.
4.2 Scaling laws for BC return
Note that the analysis in the previous section was all in terms of cross-entropy loss. However, in the imitation learning setting, we almost never care directly about this quantity. Instcad, we care about the average return of the resulting agent in the environment. To investigate how this quantity scales, roll out every model from Figure la in the corresponding Atari NetHack environment and average their score across 100 rollouts in Atari, and across at least 500 rollouts in NetHack. The results in Figure show we get mirrored image of the loss case: we observe clear concave-down parabolas with well-defined maxima at the optimal model size for a given compute budget in all games. We also see that when FLOOPs are increased, the returns
NNte that this FLOPs equation is only valid for NetHack experiments, since the model there is Transformer-basee.. To carry similar analysis for Atari, where the models CNN-based, this formula needs to be adjusted. We only perform the analysis for NetHack due to the simplicity of the FLOPss equation. While past work has pointed they still recommend using it as a proxy measure progress in the game.
the NetHack score is not necessarily aligned with winning the game Kütttler al. 2020

---

Published in Transactions on Machine Learning Research (12/2024)
16
50
2e
Retum
Retur
22f
5el
R (5.0888e-) (2.))-11) 5.020-11--1
R=12.07e 4 (211)-MM -1580-1p11
R (2.58e-1 (2/1)-110 2.51e-
Loss
(a) NetHack
(b) BattleZone
(c) Breakout
Figure 3: BC return vs. optimal loss. We investigate the relationship between the optimal loss BC agent and the mean return. We find they are highly correlated for all games.
go up. We then follow a similar procedure as in subsection 4. and perform the same three regressions, giving rise to the following power laws plus a constant Figure 2c Figure 2d and Figure 2bp:
Nopt Cx + bb Dopt aDC + bp Rapt (arC² + +bR)
(6)
where Nopt indicates the return-optimal model size, Dopt the return-optimal data size, Ropt the maximal return, and C the compule budget in FLOPs. We refer to the legends of [Figure 2cl Figure 2d. and Figure 2h for sample values of a, 3, and 7, respectively. These figures indicate that we increase FLOPs, the policy returns improve according to power law when appropriately increasing the size of the model and the size of the data according to their own power laws as well. Note that the functional form for Rops is simply the inverse a power law plus constant. When looking at Figure 26 we find that for the Atari games the scaling laws hold all the way until expert performance. For NetHack, we find more FLOPs (and maybe other improvements, see the end of section 5) for discussion on this) will be required to reach the expert score of ~17k.
Additionally, we can take the functional form in Equation 3 and simply replace loss with mean return. We can then solve the same constrained optimization problem resulting in the exact same expressions as found in Equation 5] We list the resulting coefficients for NetHack in (Tbll I] Unlike was the case for BC loss, we find the scaling exponents for the two methods to somcwhat differ for BC return. While the paramctric fit indicates scaling model and data size similarly to the loss case, the isoFLOP profiles indicate scaling model and data size equally instead. In general, we recommend performing rolling vrossvvaliation and picking the method with the lowest RMSE.
To investigate the relationship between loss and mean return, regress the loss-optimal returns the corresponding loss values. We find the relationship can be well described by the inverse of a power law plus a constant, i.e. It = (aæz (1/Lopt)* + bRR) as shown in Figure 3 The fit in the figure shows optimal loss and mean return are highly correlated all games, indicating we can expect return to increase smoothly as we make improvements in loss, rather than showing sudden jumps.
4.3 Extension to reinforcement learning
Given the stark trends we found for BC the previous sections, we investigate whether similar trends can be found for RL. We explore this briefly for the game of NetHack since several works the past years have attempted RL-based approaches for Het (Küttler et. al., 2020 Hambro et al. 2022b) without coming close to solving the game, unlike is the case for Atari. We investigate the role of model size and environment interactions using approaches 1 and 2 from subscction 4.1 applied to IMPALA (Espcholt et al. 2018).
While learning curves in RL lend to have high variance, Figure 4 suggests that compute-optimal agents should increase both the number of parameters and number of environment interactions as the FLOP budgets are scaled up. We also find that the NetHack game score varies smoothly with FLOPs and hence can be seen as natural performance metric (Hilton al. 2023). We provide complete details of our setup and results in Appendix
7

---

Published in Transactions on Machine Learning Research (12/2024)
1/2
Sant
234
2011
287
5.00 lgg
2.43-105
-6.30-0.56.0p 1gg
407
10
10F
30'
32
241
10
10
234
30
10
10"
10
3341
12-0
13
332
302
10
189
(a) IsoFLOP
(b) Return FLOPs
(c) Params vs. FLOPs
(d) Samples vs. FLOPs
Figure 4: RL return scaling. We train wide range of model sizes across several orders of magnitude of FLOP budgets and plot the average return when rolled out in the environment at the end of training (a). We then regress the return-optimal average returns (b); parameters (c), and samples (d) on their corresponding FLOP budgets. We train 1 seed per point on the isoPLOP profile.
5 Forecasting compute-optimal BC agents
The isoFLOP profiles and scaling laws shown in Figure 1 and Figure 2 allow us to predict the loss or return we can expect to achieve for a given compute budget, when allocating FLOPs optimally between model and data size. For all our Atari games (except for Space Invaders), we already trained agents that reached the expert return (see the black lines in Figure 2bj, and we don't expect the agents to improve further with more compute, so these games not very suitable to test scaling laws for extrapolation. Hence, we will turn to NetHack to test the predictive power of our scaling laws, using the following extrapolation tests:
1. Loss @ 40B: Given 40B samples, predict the loss-optimal model size, and check if the resulting agent achieves loss predicted by our loss scaling law.
2. Return @ 40B & 55B: Similar to the the loss prediction, we predict the return-optimal model size for both 40B samples and 55B samples. Then, we check if the resulting agents achieve a return predicted by our return scaling law.
While the predictions above start with a dataset size and calculate the optimal FLOPs and model size from there, one could just as well have started from a desired loss or return to achieve, a set model size, or a given FLOPs budget. Given any one of the aforementioned, one can compute any of the others (assuming want to run in the compute optimal regime).
To compute the optimal model size for the predictions mentioned earlier, we use the scaling laws derived from the isoFLOP profiles as follows. We first plug in D = 40B into the regression in Figure 1d (for loss) or Figure 2d (for return) to solve for CAOB, the FLOP budget corresponding to a dataset size of 40B samples. Then, we plug CLoB into the regression in Figure 1c (for loss) or Figure 2c (for return) to get NAoR, the compute-optinal model size when using 40B samples. This way, we find that the model size for the loss prediction @40B, the return prediction @40B, and the return prediction @55B should be 138M, 32M, and 45M, respectively.
We trained the three forecasting settings above which took ~ days for the @40B models, and ~6.5 days for the @55B models. The results can be found in Figure 3) For the loss prediction, we find our scaling law matches the actual loss almost perfectly. Somewhat surprisingly, the return models performed a bit better than predicted by our scaling law. This could be due to using cosine learning rate schedule for the predictions, while using constant learning rate for the isoFLOP profiles from which the scaling laws are derived (see section for bit more discussion
this). In Table we compare best agent with offline RL methods (DON-Offline, CQL, and IQL) well as other past BC methods (CDGPT5, Transformer, and diff History LM). For completeness, we also list methods that use additional RL fine-tuning on top of offline
We could have also used the parametric fit, and in practice we recommend performing rolling cross validatio for both methods and choosing the method with the lowest RMSE.
8

---

Published in Transactions on Machine Learning Research (12/2024)
0.7
Train Test
Table 2: Comparison with baselines. We com- pare best BC model with previous models in the NetHHackChaleegg--00 environment and find it out- performs all of them on the human monk role in the offline sctting. "Exact scores not reported. See Ap- pendix for full results with standard errors.
0.5
Dev Loss
Models
Human Monk
Offline only
0.1
DQN-Offline
0.0
1045
10t6
10
10
10
FLOPs
CQL
56
IQL
201
(a) Loss predictions
BC (CDGPT5)
1058
BC (Transformer)
1974
lef
diff History LM
4504
7ea
Scaled-BC (ours)
7784
Sea
Offline + Online
Returns
Kickstarting + BC
2090
2ea
APPO + BC
2809
LDD*
2100
lea
BC Fine-tuning KS
10588
Dataset Average
17274
Se2
Train
Test
1015
1036
10
10a8
10
FLOPs
(b) Return predictions
Figure Forecasting Results. In Table 2, we compare our models prior work and find it sets new SOTA in the offline setting. In (a), we show the loss-optimal model matches the prediction. In (b), show the return-optimal models are close to our predictions.
training. Our agent gets score of 7784 on the Human Monk role in NetHack well beyond the next best offline method of 4504, which additionally relied on a pretrained initialization. This indicates BC performance can be boosted significantly with scale.
Discussion While our return power law initially continues to improve with scale, we find it will plateau well before expert performance NetHack (but not in Atari). There could be various reasons for this early plateauing of the power law, one of which partial observability, which we study further in section
6 Limitations
Natural performance metrics. There is no reason in general to expect game scores to scale smoothly. If they do, Hilton el al. (2023) define them as natural performance metrics. Hence, one way of viewing our results is as a confirmation of the score functions for NetHack and Atari as natural performance metrics for IL. We expect that for any game score to be a natural performance metric, needs to be least somewhat. dense so tracks learning progress, which is why we focused on environments with relatively dense rewards in this paper"] It's possible results extend to highly sparse reward settings well, but may need to introduce alternative proxy metrics (e.g. intrinsic performance (Hilton et al. 2023 in that case.
Thiss however, does guarantee we will observe scaling laws these environments when using TL.!

---

Published in Transactions on Machine Learning Research (12/2024)
Fully abserved Inventory
0.28
Ctx 256
Ctx len 512
0.23
0.26
Ctx
Ctx 2048
Ctx len 4096
Dev Loss
0.22
0.24
Ctx len 8192
Dev
0.22
0.21
0.20
0.20
0.18
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Samples
Samples
10
(a) Effect of context length
(b) Elffec of missing features
Figure 6: Effects of partial obscrvability. In (a) observe context lengths up to 4096 give significant improvements in loss, suggesting long contexts may be needed to fully model the expert. In (b) show that including the inventory, which has been commonly left out in past work but upon which the expert relies, further improves the loss.
Experimental setup. Previous works have pointed to the importance of tuming hyperparameeers for every run on the isoFLOP profile. In particular, Hoffmann et al. (2022) recommend using a separate cosine learning rate schedule for every FL.PP budget on the isoFLOP profile However, to limit computational cost, we used a constant learning rate for every model size so we could leverage "snapshots" of the same run to evaluate different FLOP budgets for the same model size. While we did lune this constant learning rate pretty carefully (see Appendix GJ. there will nevertheless be some uncertainty in the exact values of all our power law cocfficients. However, expect the overall trends to still hold.
Availability of data. While some game environments might already come with large datasets of trajec- tories (Hambro et al.) 2022b), this is generally not the case. Hence, depending on the game, the expert, and the scaling law, we might find ourselves a data-constrained sctting. If the game has a fast simulator and a computationally cheap expert (as the case this paper), then data availability may be less of a problem since we can simply collect more data by rolling out many trajectories in parallel using the expert policy. However, if the game simulator itself is slow or the expert expensive (e.g. a human), then data availability may become a bottleneck. Finally, the specific scaling law for game dictates how much data we actually need (assuming we'll need trillions of data points, we might be data-constrained somewhat irrespective of the computational requirements of running the game and the expert. The opposite is also true, however: for simple games (like Atari), the scaling laws seem to usually indicate we don't need that much data (< IB), which means we might still be able to get away with slightly more computationally demanding game simulators and experts. One interesting direction for future work could be extending our results the data-consttained setting, similar to Muennighoff et al. (2024).
want to run in the compute-optimal regime). Hence, if the scaling law indicates
Comparing across environments and architectures. While we find the functional form of the scaling laws to be the same across all environments (Atari and NetHack) and architectures (Transformer and CNN), we did not investigate the influence of different environment propertics or architectures on the scaling coefficients. We leave such compa-comsssssons of different environments and architectures to future work.
7 Partial observability
In this section, we discuss how partial observability can affect how well we can model the expert. Specifically, focus two dimensions: (1) context length, and (2) information parity.
10

---

Published in Transactions on Machine Learning Research (12/2024)
Context length. While Atari games are mostly Markovian, NetHack is highly partially observed, making the use of memory crucial. Astend extensively keeps track of the past, making it important for our learner to also take into account the history. However, knowing exactly how much history is necessary to model AutoAscond is hard since it is a rule-based policy and hence it's not immediately clear what the "effective context length" of AutoAscend is. In Figure 6a] we study a wide range of context. lengths and find that loss improves considcrably up to lengths as long as 4096, after which improvements seem to be marginal. This suggests that past work trying to do BC on AutoAscend has either leveraged models that are known to struggle with modeling long contexts (Hambro et al. 2022a (e.g. LSTMs) or simply is using context lengths
that are too short (Piterbarg et al. 2023a b).
Information parity. We define information parity as the case when the learner has access to the same features of the environment as the expert does. Similarly as with context length, this is not relevant in the Atari games we study since the observation consists of a simple image of pixels, which both the learner and expert have full access to. In NotHack, however, the picture is murkier again because there is many observation features that NLE exposes of which AutoAscend makes extensive use, while prior work typically only uses subset. In F Figure 6b, we study leaving out the inventory features, which have been commonly left out in past work (Hambro et al.
2022a Wolczyk et al. (024) as they were not available in the original NLD dataset. We observe that including the inventory does in fact improve validation loss, suggesting that it is necessary to fully model the expert.
Our results on context length and information parity suggest that there may be hidden assumptions to our scaling laws, namely that they will be affected if the expert either keeps track of more history than the learner can, or is relying on features that the learner does not have access to. We leave more extensive investigation of exactly how the scaling laws will be affected in the case of partial observability to future work.
8 Related work
NetHack. Work on NetHack has been somewhat limited so far, with carly work establishing the NLE benchmark (Küttler et al. )020), evaluating symbolic vs. neural agents (Hambro et al., 2022a), and creating large-scale datasets based off of rule-based and human playthroughs for methods aiming to learn from demonstrations (Hambro et al.] 2022b). Later work has either focused on better reward signal supervision and sample efficiency through proxy metrics and contrastive pre-training (Mazoure et al. 2023 Bruce et al. 2023) or leveraged dynamics models with language descriptions in order to improve sample efficiency and generalization (Mu et al., 2022). [Piterbarg et al. (2023b) also investigates the gap between neural methods and AutoAscend, but focuses on leveraging an action hicrarchy, improvements in architecture, and finc-tuning with RL. More recent work includes building long-context language agents (Piterbarg el. al. [022a] and investigating RL finetuning techniques to boost the performance of pretrained models (Wolczyk et al. 2024h.
Scaling laws. Hestness et al. (2017) and Rosenfeld et al. (2019) to characterize empirical scaling laws for deep learning. Kaplan et al. (020) and Hoffmann et al. (2022) specifically focus on training compute-optimal language models, finding similar trends as presented in this paper. While in the imitation learning setting, our agents also minimize cross-entropy loss, we additionally show that the eventual performance of the agent as measured by the average return in the environment scales smoothly with the loss. Other works focus more broadly on generative modeling (Henighan 2020), or analyze specific use cases such as acoustic modeling (Droppo & Elibol, [021]). Clark et al. (2022) investigate scaling laws for routing worksrks, and Hernandez et al. (2021) study scaling laws for transfer, finding the effective data transferred (the amount of extra data required to match pre-trained model from scratch) follows a power-law in the low-data regime. More recent works have also tried to extend these scaling law results to multi-modal learning (Cherti et al.] 2022 Aghajanyan et al. 2023) Caballero et al. (2022) introduce broken neural scaling lawss, which allow modeling of double descent and sharp inflection points.
of the carliest works that try
al.
Perhaps the closest work to our paper is that of Hilton et al. (2023), who characterize scaling laws in RI. However, they don't consider IL, and they do not evaluate on Atari or NetHack, the latter of which we consider an especially interesting environment because of its challenging nature.
11

---

Published in Transactions on Machine Learning Research (12/2024)
9 Discussion
Beyond singlc-agent games. We have shown that in the imitation learning sctting (and to some extend in the RL selling), scaling up model and data size provides predictable improvements, as demonstrated in a variety of Atari games and in the full game of NetHack. While we do not extend our analysis beyond singlo-agent games in this paper, we believe there are scveral key takeaways to take from our work when performing BC on a new domain. Firsl, our results show that the same scaling laws show up across games with various properties (stochasticity, partial observability, pixel-based, etc.), suggesting they may show up for other domains as well. Second, poor performance (relative the expert) in new domain might be explained at least in part by scale. This is powerful since one might be able to stick with BC as method instead of turning to alternatives as long as one is careful about the model and data size. Third, one has to be careful about partial observability. A gap in context length or information parity between the learner and the expert could potentially hurt BC performance.
Leveraging human data. In this work, we did not consider analyzing the scaling relationships when using human trajectories (e.g. from NLD-NAO (Hambro el al.) 2022bl) instead of those from AuloAscend (NLD-AA (Hambro et al. 2022b)). ) This is because extra care must be taken to handle the lack of actions in the human datasot, requiring techniques such as BCO (Torabi ct al. 2018). Investigating scaling laws here could be especially interesting since: (1) the human dataset is more diverse, containing trajectories from many different. players with varying level of skill, and (2) it contains many examples of trajectories that ascend (i.e. win the game). (1) could shed perspective on the role of scaling when the data includes
many different and potentially suboptimal demonstrations, similar to Beliaev et al. (2022). (2) could provide
insight into the viability of methods such as Video PreTraining (Baker et al.] 2022 since these rely heavily on being able to clone the expert data well.
The effect of expert quality. The experts used in our paper are of different qualities for different domains. For example, while AutoAscend is pretty poor compared to a human expert, the expert we use for Breakout is superhuman (Schwarzer et al. 2021). Nevertheless, we observe scaling laws in all cases, suggesting that the quality of the expert does not affect the eristence of our scaling law. Second, what we do find to vary based on the quality of the expert is the upper bound or ceiling that the scaling laws converge to. Specifically, for all Atari games, we find that the scaling laws converge exactly to the mean return of the expert. Hence, we expect that as the expert quality improves or deteriorates for a particular environment, the corresponding scaling law will change to reflect this shift in the ceiling (i.e. it will now plateau at a different mean return).
10 Conclusion
In this work, we find that imitation learning loss and mean return follow clear scaling laws with respect FLOPSs, as demonstrated in Atari and in the challenging game of NetHack. In addition, we find loss and mean return to be highly correlated, meaning improvements in loss translate in improved performance in the environment. Using the found power laws, we forecast the compute requirements (in terms of model and data size) to train compute-optinal agents aimed at recovering the underlying expert. In NetHack, we find the performance improves substantially, surpassing prior SOTA by 1.7x in the offline setting. We also briefly extend our results to the reinforcement learning setting, and find similar power laws for model size and number of interactions in NetHack. Our results demonstrate that scaling up model and data size can provide big boosts to imitation learning performance in single-agent games. More broadly, they also call for work in the larger imitation learning and reinforcement learning community to more carefully consider and study the role of scaling laws, which could provide large improvements in many other domains.
Broader Impact Statement
While we do not see a direct path towards any negative applications, we note that scaling up could have unforescen consequences. As scaling results in imitation and reinforcement learning agents that are increasingly more capable and influential in our lives, it will be important to keep them aligned with human values.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Acknowledgements
We thank Alexander Wettig, Ameet Deshpande, Dan Friedman, Howard Chen, Jane Pan, Mengzhou Xia, Khanh Nguyen, Shunyu Yao, and Vishvak Murahari from the Princeton NLP group for valuable leedback, comments, and discussions. We are also grateful to Riccardo Savorgnan, Sohrab Andaz, Tessa Childers- Day, Carson Eisenach, Kenny Shirley, and others from the Amazon SCOT Forecasting team for helpful discussions and encouragement. We thank Kurtland Chua for helpful feedback. Finally, we give special thanks to Eric Hambro for answering any questions we had about the NetHack environment and datasets throughout the project. Sham Kakade acknowledges funding from the Office of Naval Research under award N00014-22-145 and the National Science Foundation Grant under award #CCF-2212841. JT and KN acknowledge support from the National Science Foundation under Grant No. 2107048. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
References
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hanbardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. arXiv preprint arXiv:2301.0377 2023.
Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning enviromment down to five games. In International Conference Machine Learning, pp.
PMLR, 2023.
Bowen Baker, lige Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24663 2022.
Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimating expertise of demonstrators. In International Conference on Machine Learning, pp. 1732 1748. PMLR, 2022.
Jake Bruce, Ankit Anand, Bogdan Mazoure, and Rob Fergus. Learning about progress from experts. In International Conference on Learning Representations, 2023.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws, arxie preprint arXiv:2210.1480) 2022.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ihharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. arXiv preprint arXiv:2212 07144,, 2022.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake A. Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, T. W. Hennigan, Matthew G. Johnson, Katic Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, L. Sifre, Simon Osindero, Oriol Vinyals, Jack W. Rae, Erich Elsen, Koray Kavukeuoglu, and Karen Simonyan. Unified scaling laws for routed language models. In International Conference on Machine Learning, 2022.
Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Advances in Neural Information Processing Systems, 32, 2019.
Jasha Droppo and Oguz Elibol. Scaling laws for acoustic models. arXio preprint Nii:21066099 2021.
Adrien Ecoffet, Joost Huizinga, Joel Lchman, Kenneth O. Stanlcy, and Jeff Clune. First return, then explore. Nature, 5507 Feb 2021. ISSN 1476-4687. doi: 10. 10.1038/s411 0003///1156-00000311111 URL http:p//......ooo//o....00////20
13

---

Published in Transactions on Machine Learning Research (12/2024)
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dumning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, In International conference on machine learning, pp. 1407 1416. PMLR, 2018.
Eric Hambro, Sharada Mohanty, Dmitrii Babaev, Minwoo Byeon, Dipam Chakraborty, Edward Grefenstette, Minqi Jiang, Jo Dacjin, Anssi Kanervisto, Jongmin Kim, et al. Insights from the neurips 2021 nethack challenge. In NearIPS 2021 Competitions and Demonstrations Track, pp. 41 52. PMLR, 2022a.
Eric Hambro, Roberta Raileanu, Danielle Rothermel, Vegard Mella, Tim Rocktäschel, Heinrich Küttler, and Naila Murray. Dungeons and data: A large-scale nethack dataset. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 24864-24878. Curran Associates, Inc., 2022b.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling, arXiv preprint ar Xrv:220000. 14701, 2020.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprinl Xiv:22102. 01293, 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Pat- wary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint Xie: 00409, 2017.
Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint arXi::2301.144 13442, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint 0rXiv:2203.15555 2022.
Athul Paul Jacob, David J Wi, Gabricle Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Jacob Andreas, and Noam Brown. Modeling strong and like gameplay with kl-regularized search. In International PMLR, 2022.
Conference on Machine Learning, pp.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 3rXii:2001.0820 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1422.6900 69800, 2014.
Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikacl Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arX1:22310. 001666 2023.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-700 7684, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
Bogdan Mazoure, Jake Bruce, Doina Precup, Rob Fergus, and Ankit Anand. Accelerating exploration and representation learning with offline pre-training. arXiv preprint arXiv:2304.00046, 2023.
Jesse Mu, Victor Zhong, Roberta Raileamu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, and Edward Grefenstette. Improving intrinsic exploration with language abstractions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 33947-33960. Curran Associates, Inc., 2022.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Newral Information Processing Systems, 36, 2024.
Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. diff history for long-context language agents. or preprint arXiu:231..07540, 2023a.
Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. Nethack is hard to hack. arXiv preprint 1rXiv:2305119240 2023b.
Antonin Raflin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 8, 2021. URL http::/jmlis httt:://jml...or/pppeeee.r. http:/ppprs//v2//2000 1364. html
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. arXiv preprint 0rXii:1909.12670 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiviv preprint wrXiv:177700334 2017.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations, 2021.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Autonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, llya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, (75 489, Jan 2016. ISSN 1476-4687. doi: (0.103/nnaturell URL https: //doi hwtpo/oorg/....000//aatttteeeeee
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 4950-4957, 2018.
Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting copycat agents in behavioral cloning from observation historics. Advances in Neural Information Processing Systems, 33: 2020.
Maciej Wolczyk, Bartlomiej Cupial, Mateusz Ostaszewski, Michal Bortkiewicz, Michal Zajac, Razvan Pascanu, Lukasz Kucinski, and Piotr Miloš. Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem. In Forty-first International Conference on Machine Learning, 2024.
15

---

1 Published in Transactions on Machine Learning Research (12/2024)
A Parametric fit: derivation of power law
In this section, we show the derivation resulting in Equation 5 We first restate Equation 3 for convenience:
log L(N, D) - S + 3N log N + BD log
+ 8x² log" N + AND log N log D + SDa log" D.
Now, to minimize this equation with the constraint that FLOPs(N, D) = z 6ND, we use Lagrango multipliers. We first write down the Lagrangian:
C(N,D, D) L(N Ng(NN, C),
where gNN, D) 6ND. Now, setting VL 0 get:
Vogg iNN, D) = AV(6ND) and 6ND C.
The former results in the following system of equations:
Oog Í(N, D) ((GND)
ON a log i(N, D) aD
ON
0(6ND)
0D
This means that
8N + BND log D
+ 23 log N
N 26D
N
N
1
BD D AND log N
22 log
A6N.
D
Multiplying the top equation by N and the bottom by D have that
8x + BND log D + 23 log N =8p + 3ND log
+ 2B log D.
Recalling C 6, we solve for N and D in terms of C, giving the results listed in Equation 5)
Full set of forecasting results for NetHack
The full set of forecasting results for Het be found in Table 3 DQN-Ofline, CQL, and IQL all offline RL methods, while BC (CDGPT5), BC (Transformer), and diff History LM are all based on behavioral cloning.
C Full set of isoFLOP figures for NetHack
We show the full set of isoFLOP curves in NetHack for all settings in Figure
Architecture details
D.1 Atari
Our architecture for BC experiments in Atari is simple. It consists of the following layers:
convolutional layer with x 8 filter size and stride 4, followed by a ReLU activation layer.
convolutional layer with x 4 filter size and stride 2, followed by a ReLU activation layer.
convolutional layer with x 3 filter size and stride 1, followed by a ReLU activation layer.
final linear layer that maps the flattened output dimension of the CNN layers above to the number of actions in the respective Atari game.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Table 3: Forecasting results (full). Table with added standard errors. Results from Hambro et al, (2022b) use 10 seeds, while those from Piterbarg ct al. (2023b) and ours are seed. "Exact scores not reported original work.
Human Monk
Offline only
DQN-Offine HHmbro et al.
2022b
0.0 + 0.0
CQL Hambro el. al
2022b
56 + 28
IQL
HHmmro et al.
2022b
201 27
BC
(CDGPT55 mbro al.
2022b
1058 159
BC (Transformer Piterbarg et al. 2023b
1974
diff History LM Piterbarg et al.
2023a
4504 +
Scaled-BC (ours)
7784 +
Offlin + Online
Kickstarting BC (Hambro
2022b
2090 123
APPO BC (Hambro et al. 2022b)
2809 + 102
LDD" Mu et al.) 2022)
2100 -
BC + Fine-tuning + KS (Wolozyk et al. 2024
10588 + 672
15
164
563
562
S6.15
14:55
263
163
Dev lozs
Returns
1.0
262
Sel
2033
3037
2r1.3
-+-
5c34
2e1
5e35
2337
5437
+8m 3635
-go 1615
-4- 1437
Parameters
Parameters
(a) BC Loss IsoFL.PP curves
(b) BC Return TsoFL.OP curves
(c) RT. Return TsoFL.P curves
Figure 7: Full isoFLOP curves. (a) similar to Figure la but including the full set points. (b) is similar
F Figure? 2a but including the full set of points. (c) is similar to Figure 4a but including the full set points.
D.2 NetHack
We use two main architectures for all our experiments, one for the BC experiments and another for the RL experiments.
BC architecture. The NLD-AA dataset Hambro et al. 2022b) is comprised of ttyrec-ormatted trajectories, which are 24 x 80 ASCII character and color grids (one for each) along with the cursor position. To further reduce partial observability, we regenerate the dataset to also store the dungeon and inventory glyphs. To encode the ASCII characters and colors along with the glyphs, we modify the architocture used in Hambro et al. 2022b) resulting in the following:
Dungeon encoder. This component encodes the main observation in the game, which is a 21 x 80 grid per time step. Note the top row and bottom two rows are cut off as those are fed into the message and bottom line statistics encoder, respectively. We embed each character, color, and glyph an embedding lookup table, concatenate them together, and feod them through one lincar layer, after which we put them in their respective positions in the grid. We then feed this embedded grid into ResNet, which consists of two identical modules, each using convolutional layer followed
17

---

Published in Transactions on Machine Learning Research (12/2024)
by max pooling layer and two residual blocks (of two convolutional layers each), for total of 10 convolutional layers, closely following the setup in Espeholt et al. (2018)
Message encoder. The message encoder takes the top row of the grid, converts all ASCII characters into one-hot vector, and concatenates these, resulting in 80 x 256 20,480 dimensional vector representing the message. This vector is then fed into two-layer MLP, resulting in the message representation.
Bottom line statistics. To encode the bottom line statisties, we flatten the bottom two rows of the grid and create a nirmaacler-normalizedd (subtract 32 and divide by 96) and "digits-normalized (subtract 47 and divide by 10, mask out ASCIT characters smaller than 45 or larger than 58) input representation, which we then stack, resulting in a 160 x 2 dimensional input. This closely follows the Sample Factory model used in Hambro et al. (022bb
Inventory. Unlike prior work (Hambro et al. 2022a we also include the inventory glyphs in our network to increase information parity with respect to the expert (see section 7). We use the same glyph embedding table as used for the dungeon encoder followed by a linear projection. Then, we concatenate all inventory items along the hidden dimension and feed them through two-layer MLP. Note that we only use these inventory features for IL, not for RL.
After the components above are encoded, we concatenate all of them together. Additionally, we also concatenate the previous frame's action representation (coming from an embedding lookup table), and a crop representation (a 9 crop around the player, processed by five-layer CNN). We then feed this combined representation into two-layer MLP, after which Transformer (in case of IL) LSTM (in case of RL) processes the representation further. Finally, we have two linear heads top of the sequence model, one for the policy and one for the value (not used for BC).
RL architecture. We modify the architecture from Küttler et al. 2020 to also include a five-layer one-dimensional CNN that processes the message, as well as another five-layer two-dimensional CNN that processes x crop of the dungeon grid around the player.
E
FLOP and parameter counting
As mentioned in the main text, we only count FLOPs and parameters for the parts of the model being scaled.
E.1 Atari
The model network consists of three convolutional layers interleaved with ReLUs (see Appendix for details), followed by a linear layer (i.e. the policy head). We simply scale the width of all layers. This means we scale the channels of the convolutional network and the width of the linear head on top.
Since we scale the whole network, count the FLOPs from all convolutional and linear layers. We compute the forward FLOPs of a convolutional layer as 2 hamt Wont Cout P, where hout is the height of the output shape, Wont is the width of the output shape, Cout are the number of output channels, and p indicates the number of parameters in one filter of the current layer (without counting bias). Hence, p = k2 Cins where k is the kernel size and Gin the number of input channels. Following prior work (Hilton al. 2023) we
assume the backward pass takes about twice the number of FLOPs from the forward pass.
E.2 Net
For all our BC and RL experiments, we only scale the following parts of the model and keep the rest fixed:
The hidden size the two-layer MLP right before the Transformer (in the case of BC) or the LSTM (in the case of RL).
httpp:://github..cm//iii http:///aaaaaaaaaaaaaaaaaaaaaaaa factry--beaattr
18

---

Published in Transactions on Machine Learning Research (12/2024)
The hidden size of the Transformer (in the case of BC) and the LSTM (in the case of RL).
The input size of the two linear layers for the actor and critic respectively (i.e. the policy and value heads).
Similar to prior work (Kaplan et al. 2020 Hoffmann et al. 2022). we found 6ND to be a good approximation (for both the Transformer and the LSTM) for the number of FLOPs used based on model size N and number of samples D. This is because we found there to be about 2ND FLOPs in the forward pass, and we assume the backward pass takes about the twice the number of FLOPs from the forward pass.
For the RL experiments, there is slight change in the way count FLOPs, which is that we count every forward pass number of FLOPs from the learner twice, since there is corresponding forward pass from an actor. Hence, for RL our formula becomes SND.
F Game Selection for Atari Games
We chose the following set of 8 Atari games: Battle Zone, Q*bert, Name This Game, Phoenix, Space Invaders. Bank Heist, Boxing, and Breakout. The first 4 games were picked from the Atari-5 subset (Aitchison et al.) 2023), which trics to condense the full set of Atari games to a subset of represontative games. We had some trouble training a good (i.e. substantially better than random) expert for Double Dunk (the 5th of the Atari-5 subset), so instead we added 4 other games, chosen at random from the full set of Atari games (though excluding Montezuma's Revenge due to its extremely sparse rewards). We didn't include more than 8 games due to constraints both compute and experimenter time, but strongly suspect our results will generalize to most games in the Atari set. Note that the results could even hold true for Montezuma' 's revenge as well, but since we didn't specifically seek out very sparse reward games, we do not make that claim and instead convey this as a potential limitation.
G
Training details
G.1
Atari
We list the hyperparameters for all our BC experiments in Atari in Table 4a and the hyperparameters to train expert policies for each Atari game in Tabble 4b To train these expert policies, we used the Stable Baselines3 (Raffin et al. 2021 implementation of PPO (Schulman et al. 2017). We use Adam (Kingma & Ba 2014) as our optimizer for both settings. All training experiments were done on NVIDIA GPUs (a mix of GeForce RTX 3090, GeForce RIX 2080 Ti, RIX A5000, and RIX A6000) and took about - days depending on the game and FLOP budget.
Table 4: Hyperparameters for all experiments in Atari. We list the hyperparameters for all our BC experiments (a) well as the ones used to train the PPO expert agent for each game (b).
Hyperparameter
Value
Hyperparameters
Value
Learning rate
2.5c
Learner batch size
256
Learning rate
0.0001
Entropy cost
0.01
Batch size
128 x (1 GPU)
Bascline cost
0.5
Unroll length
32
Total timesteps
5e
Data workers
30
Discount factor
0.99
GAE lambda
0.95
(a) BC hyperparameters
Clip range
0.2
(b) PPO hyperparameterrs
19

---

Published in Transactions on Machine Learning Research (12/2024)
G.2 NetHack
We use AdamW (Loshchilov Hutter 2019) as our optimizer for all BC experiments. For RI, we use RMSprop. Please find all hyperparameters for both settings in Table 5 all of which were manually tuned taken from prior work. All NetHack BC experiments were experiments were
NVIDIA H100 80GB GPUs. All Atari BC mixture of NVIDIA A5000 and A6000 GPUs. The RL experiments were run V100 32GB GPUs. All experiments took anywhere from few hours to 4 days to run. The one exception to this is the forecasted BC models which took up to 6.5 days to run.
The NLD-AA dataset (Hambro et al. 2022b is released under the NetHack General Public License and can be found https://github: com http://///.....o/ooo//////aaaaaa
In Figure Sa we investigate the optimal learning rale for both cosine and constant learning rate schedules in NetHack. We find 0.0005 works best for the constant schedule and hence use this for generating the isoFLOP profiles in the paper. For the prodictions, we use cosine schedule since Figure Sal shows it works better. In addition, in Figure 8b we test width vs. depth scaling and find that keeping the aspect ratio (i.e. hidden dimension divided by the number of layers) is best kept at a little over 100, which we stick to when generating our isoFLOP profiles.
Table 5: Hyperparameters for all experiments in NetHack. We list the hyperparameters for all our BC experiments (a) as well as the ones for our RL experiments (b). For BC, we always use a constant learning rate except for the predictions in section 5) for which we use a cosine schedule with warmup.
Hyperparameter
Value
Learning rate
0.0002
Hyperparameters
Value
Learner batch size
32
Unroll length
80
Learning rate
0.0005
Entropy cost
0.001
Batch size
32
Baseline cost
0.5
Unroll length
8192
Maximum episode steps
5000
Tlyrec workers
30
Reward normalization
yes
Reward clipping
none
(a) BC hyperparameters
Number of actors
90
Discount factor
0.99
(b) IMPALA hyperparaneters
Derivation of delta method
Following standard lincar regrossion assumptions, have that vn(ß (3) D N' E), where B is vector containing all regression coefficients, i.e.
B = - BN, Br, BN2, BND, 3p²)
We also rewrite a from Equation 5 more explicitly as a function
2,9 3ND
h(p)
2,3 23 D + 28
Approximating h()) with a first-order Taylor expansion around B we get:
h(A) M(3) + Vh(3)T (B 33)
20

---

Published in Transactions on Machine Learning Research (12/2024)
0.44
Constant
Constant (1000)
0.230
0.4
Constant (2000)
-
Constant (3000)
0.229
Dev Loss
0.36
Cosine (2000)
(1000)
Dev Loss
0.228
--- Cosine (300D)
0.32
0.227
0.226
0.28
0.225
0.24
0.224
40M params
6AM params
0.22
0.223
1e-5
Ze-5
5e-5
1e-4
2e-4
5e-4
1e-3
50
100
150
200
250
300
Learning Rate
Aspect Ratio
(a) Learning rate sensitivity
(b) Scaling depth width
Figure 8: IsoFLOP learning rate & aspect ratio in NetHack. In (a), we test both cosine and constant learning rate schedules with varying warmup periods (specified in mumber of gradient steps). We find 0.0005 to work best when using a constant learning rate. In (b), we investigate the optimal aspect ratio and find it remains constant across model sizes.
Computing the variance of h(ß) then gives g
Var[h(B)] Var(h(B) + Vh(B)² (B- - B)] Var(h(B) + Vh(B)T . Vh(3)T 3)
VarlVh(B) |
VI(B)² Cov(3) Vhh(3)
VI(A)T Vv((8)
Now the Standard Error (SE) can be found as
SE = Vh(3)T Vh(B).
Finally, we approximate the SE above by using Vh(33) instead of Vh(3), giving the following confidence interval:
CI h(ß) + 1 SE.
Scaling laws for reinforcement learning in NetHack
Table 6: Model and data size predictions for RL scaling laws in NetHack.
RL Aug. Human (127k)
1. IsoFLOP profiles
4.4B
13.2T
2. Parametric fit
67B
0.93T
We train LSTM-based agents the NetHackScoreev0 environment. NetHackScore-vO features the full game of NetHack but has a reduced action space, starting character fixed to human monk, and automatic
We follow very similar derivation https::/ee. httppp////esss.ppppp........ooor https///eeeettteeeeeaaaaartrrrea deltta_ethhhd
21

---

Published in Transactions on Machine Learning Research (12/2024)
menu skipping. We also experimented with the NetHHackChallenge-vo environment but found the results too noisy at the FLOP budgets we were able to run. However, we expect similar results will hold for this environment at larger FLOP budgets.
IsoFLOP profiles. We train 9 different model sizes ranging from 100k to 50M using IMPALA (Fspeholt et al. 2018), each with a FLOP budget ranging from 1e13 to le17. For each of these models, we evaluate the model at the end of training by rolling out 1k times in the environment and reporting the average return. While learning curves in RL tend to have high variance, we generally st.ill find that compute-optimal models should increase both the number of parameters and number of environment interactions the FLOP budgets are scaled up (see Figure 4). We also find that the NetHack game score varies smoothly with FLOPs and hence can be seen as natural performance metric (Hilton et al., 2023), as discussed in more detail in section G We again follow a similar procedure as in subsection 4.1 resulting in power laws as listed in Equation We find a = 0.43, 3 = 0.56, and 7 = 0.32.
Parametric fit We take the functional form in Equation 3 and replace loss with mean return. We can then solve the same constrained optimization problem resulting in the exact same expressions as found in Eqquaiin (the denominator of 6 is replaced with 8 due to slight difference in FLOP counting for RL, see Appendix E After fitting, we find = 0.6 and 3 = 0.4. Note dropped the low flop budgets when per performing this regression, as we found this greatly improved the fit.
Forecasting human performance. Hambro et al. 2022b) report that average human performance is around 127k. Basod on the two approaches discussed above, we forecast the compute requirements for training an RL agent from scratch to achieve human-level performance on NetHack, listed in [Tale 6 For the isoFLOP profile approach, we first use Figure 4b to solve for C127a- Then we plug this into the power laws from Figure 4c and Figure 4d] For the parametric fit, we instead plug C127A into the power laws from Equation 5] with the correct a and B from above, where the denominator of 6 is replaced with 8 mentioned earlier. In Table 6. we find the parametric fit to put significantly more emphasis on model size, which could be possible due to dropping of the low FLOP budgets (optimal model size tends to shift more clearly in larger FLOP budgets). Due to computational constraints, we leave testing the limits of this prediction to future work.
RL with pretraining. All our scaling law results on the RL side in this paper are with policies trained from scratch. However, some of the most promising neural methods for NetHack and other domains leverage pre-trained (c.g. through imitation learning) policy that is then finetuned with RL. It would be very interesting to analyze the scaling behaviors for these kind of kickstarted policies, and see whether they scale differently than the ones trained from scratch. We leave this to future work.
J Full results for Atari
F Figure 9 Figure 10 F Figure 11 and Figure 12 list the full set of Atari results with respect to cross entropy
loss. F'igure 13 Figure 14
Figure 15] and Figure 16 list the full set of Atari results with respect to environment
return. Finally, Figure 17 lists the full set of Atari results relating environment return and optimal loss. Note that for the return results, we can see that Space Invaders is the only Atari game where didn't run high
enough FLOP budgets to reach expert performance.
22

---

Published in Transactions on Machine Learning Research (12/2024)
1.5
2.7
1.0
2.0
Dev loss
Dev loss
1.0
0.1
2014
2616
2614
2016
Se1d
Se14
5636
0.05
0.6
2615
2617
Se15
S617
Se15
5e17
Ie16
10a
10a
10a
10a
107
10a
10a
10
10
107
Parameters
Parameters
(a) Bank Heist
(b) Battle Zone
1.0
0.5
Dev loss
Dev
0.2
2e14
2016
5e14
Sc14
2016
1e15
1617
1615
S6.6
2015
2617
0.2
1616
10a
10
10a Parameters
10
107
10
10a
Parameters
Boxing
Breakout
0.7
0.5
Dev loss
0.5
Dev loss
2c14
2016
2c14
2636
Sel4
S636
Se14
5236
1e15
2015
5e15
S6
Se15
5017
1e16
0.2
2016
10a
104
10a
10f
10a
10c
109
Parameters
Parameters
(e) Name This Game
(f) Phoenix
2.0
1.5
Dev loss
0.5
Dev loss
1.0
1e14
-am
0.2
Se30
2e14
1e15
2617
1e15
1617
2e15
Se37
2615
2617
0.05
10a
10
10
10a
107
0.6
10a
10a
10
10c
Parameters
Parameters
(g) ** *ert
(h) Space Invaders
Figure 9: BC isoFLOP curves. Full results for all Atari games.
23

---

Published in Transactions on Machine Learning Research (12/2024)
1.0
2.0
6.75e c 6.4 2
3.90e C 017 1.2e
0.5
Dev Loss
0.2
Dev Loss
1.0
0.1
0.05
0.6
0.02
1024
10
10t6
10a7
10
1034
10
10t
10
10tH
FLOPs
FLOPs
(a) Bank Heist
(b) Battle Zone
1.0
L 1.38e C-024 + 6.50e
Lm 2.51e . C- + 4.99e
0.5
Dev Loss
Dev
0.2
0.1
0. 1024
10t5
10th
10a7
10:4
10t5
10th FLOPs
10
1028
FLOPs
Boxing
(d) Breakout
0.7
1.64e C 025 1.88e
0.5
1.740 026 1.39e
0.5
Dev Loss
0.2
0.2
10
10t5
10
10
10
10
10t5
10
109
10
FLOPs
FLOPs
(e) Name This Game
(f) Phoenix
1.5
L=927e CC000 1
L=322e CC-010 122ee
Dev Loss
0.5
Dev Loss
1.0
0.2
0.05
10a
10
10 FLOPs
1017
10
0
1015
10 FLOPs
10 10th
(g) Q
bert
(h) Space Invaders
Figure 10: BC optimal loss vs. FLOPs curves. Full results for all Atari games.
24

---

Published in Transactions on Machine Learning Research (12/2024)
10a
10a
Parameters
Parameters
log
11 26 + 0.60 llog
log N 10.67 + 0.58 llgg
1024
10
10th
1027
10th
1024
10:3
10tt
10a
10
FLOPs
FLOPs
(a) Bank Heist
(b) Battle Zone
10c
10a
Parameters
10a
Parameters
205
10a
10a
log 15
0.77 C
102
10
10
10
10a
log 16.32 0.74 log
FLOPs
1034
10ts
10th
10t
104
FLOPs
Boxing
(d) Breakout
10a
10c
Parameters
Parameters
105
logN=
llg
log
log
103
102 1014
1024
10
10
1027
10
10ts
10tH FLOPs
10
10
FLOPs
(e)
Name This Game
(f) Phoenix
Se
10a
Parameters
Parameters
10a
100
10a
10a
No 14.55 + 0.69 logC
log - 17.54 0 llg
1034
10
10t6
10
10
101
1015
1016
10
10
FLOPs
FLOPs
(x) Q bbertt
(h) Space Invaders
Figure 11: BC optimal parameters FLOPs curves. Full results for all Atari games.
25

---

Published in Transactions on Machine Learning Research (12/2024)
16a
16P
10a
10a
Samples
Samples
log 0.57 + 0.50 llog
log D
1.06 + 0.51 log
1024
10
104h
1027
10th
1024
10:3
10th
10a
10
FLOPs
FLOPs
(a) Bank Heist
(b) Battle Zone
10a
109
10a
Samples
10a
Samples
107
107
0.34 llogC
105
1
10
log = 0.37 + 0.45 log
FLOPs
106
1014
10a5
10th FLOPs
10t
104
Boxing
(d) Bruakout
Samples
Samples
107
107
log
log
logo
log
10a
10a
10a4
10:3
10
1027
10
1014
101s
10th FLOPs
10
10
FLOPs
Name This Game
Phoenix
10a
10a
10a
10a
Samples
Samples
10a
107
log D = 0.54 + 0.49 logC
log0 - 1.46 0.42 log
1034
10
10t6
10
10
10
1015
101h
10
10
FLOPs
FLOPs
(x) Q
bbet
(h) Space Invaders
Figure 12: BC optimal samples FLOPs curves. Full results for all Atari games.
26

---

Published in Transactions on Machine Learning Research (12/2024)
700
5e
200
20
Returns
Returns
100
10
50
Se²
Export
2015
Se36
Expert
mdn 1e15
lel6
2624
5615
1e17
2e14
2e15
2e10
Sel4
1e16
2e17
10
10a
10
10a
10a
107
10a
10
10a
10e
Parameters
Parameters
(a) Bank Heist
(b) Battle Zone
40
50
Returns
20
Returns
Expert
10
2614
2e15
2e16
Expert
2015
Se16
5e14
Set5
Se16
2014
20
10a
10
10
10
107
1815
1:16
2616
5e17
Parameters
5
10a
10
10a
Parameters
(c) Boxing
(d) Breakout
Export
2014
Se14
2e
1e15
Returns
Returns
Se
10P
1e16
2616
5e16
Sea
Expert
1615
le16
5617
2w14
2015
2616
Se14
So15
Sw16
3ea
2e
10a
10
10a
10a
10a
10*
10a
10'
Parameters
Parameters
(e) Name This Game
(f) Phoenix
2ea
2ea
Expert
-gm 1el6
10a
2014
2016
Se16
10a
2c15
2o17
Returns
2e
Returns
3615
5e17
103
Se
5e²
Expert
le15
1616
1017
2c14
5c14
5615
102
2e
10a
10
10a
10c
107
10a
10
10
10f
Parameters
Parameters
(4) bebrtt
(h) Space Invaders
Figure 13: BC return isoFLOP curves. Full results for all Atari games.
27

---

Published in Transactions on Machine Learning Research (12/2024)
Se
Returns
Ze*
Returns
2e
10a
Sel
Expert
Expert
R (5.23eM. Sanel..--.11... --
R= (3.25e CC- 1.19
1024
10th
10ts
1024
10t
10s
10a
FLOPs
FLOPs
(a) Bank Heist
(b) Battle Zone
50
50
Returss
Returns
20
20
Expert
10
(1.30e45 C-293 + -2
Expert
10
10
R (1.53e
C- + 21-1
FLOPs
1034
10ts
10
10a
FLOPs
(c) Boxing
(d) Breakout
2e
Returns
Returns
10a
Se'
Expert
Expert
(1.09e*
+120e-f-1
(6.61e² C-203 -195e-
1014
1015
10
10
10ts
109
FLOPs
FLOPs
(e)
Name This Game
Phoenix
2ea
2ea
10a
10a
Returns
Returns
2e
10a
Expert
Expert
= (1.97el) C-359 + 5.86e 5y-1
R = (4.14p) C-228 +2.92 +1
10a4
10
10tc
10
101
10
10
10)7
10
FLOPs
FLOPs
(x) Q bbbrtt
(h) Space Invaders
Figure 14: BC optimal returns FLOPs curves. Full results for all Atari games.
28

---

Published in Transactions on Machine Learning Research (12/2024)
16f
Parameters
10a
Parameters
10
10*
10
10a
log 11.35 0.66-
log
16.82 + 0.76 log
103
10:4
10
10a
104
10th
10
10
FLOPs
FLOPs
(a) Bank Heist
(b) Battle Zone
5e3
Parameters
10a
10a
log
10a
10a5 FLOPs
21.559 0.88 log
101
10
10
FLOPs
Boxing
(d) Breakout
10c
Parameters
Parameters
10a
10f
10a
log 12.45 C
10a
log 4.54 + 0.21 Cog
10:4
10
t0
10
10t5
10ah
10
FLOPs
FLOPs
(e)
Name This Game
(f) Phoenix
10a
10*
Parameters
10a
Farameters
10f
10a
log 17.49 + 0.77 logo
log
6.99 + 0.50 -lg-
10a
10
1015
10.6
10
10
10
10
1017
10
FLOPs
FLOPs
(g] bert
(h) Space Invaders
Figure 15: BC return-optimal parameters FLOPs curves. Full results for all Atari games. Except For Phoenix, all slope coefficients are statistically significant.
29

---

Published in Transactions on Machine Learning Research (12/2024)
16a
16P
10a
10a
Samples
Samples
107
107
log - 0.34 + 0.47 llog
log - 4.19 + 0.36 log
10:4
10t3
10th
1024
10t
10th
10a
FLOPs
FLOPs
(a) Bank Heist
(b) Battle Zone
10a
10P
10a
Samples
108
Samples
10a
107
logD = 0.73 0.45
10
5
log = 4.41 + 0.35 log
FLOPs
106
1014
10
10t
10a
FLOPs
Boxing
(d) Brea kout
10"
10a
Samples
Samples
107
109
log
log
log 14.59 log
10a
10:4
10t5
10:*
10:4
10ts
10
109
FLOPs
FLOPs
(e)
Name This Game
Phoenix
10P
10a
10a
Samples
Samples
107
10a
10a
logo - 1.64 0.43 logC
log D 6,34 0.66 log
10
10fs
10t6
10
10f4
101F
1Ht
1017
10
FLOPs
FLOPs
(x)
Q +bbrt
(h) Space Invaders
Figure 16: BC return-optimal samples FLOPs curves. Full results for all Atari games.
30

---

Published in Transactions on Machine Learning Research (12/2024)
Se
Retur
Retur
102
10a
R= 177722--.121 151
(1/11-244
R= (2.07e (0.1)----111-1111 + 1)
1.5
2.0
5.0
10.0
0.6
1.0
155
Loss
(a) Bank Heist
(b) Battle Zone
50
50
Return
Retur
20
20
10
R (1.64e-1 (1/1) -
R (1.58e -. (1..)-11111 2-51
1 Loss
1 Loss
(c) Boxing
(d) Breakout
2e
Return
Return
10a
Sea
(3.99e-1. (122---(11))22))) 1-022--9--1
(3.24e- (1/1)-115 1--4
2.0
3.0
3.0
4.0
1/ Loss
// Loss
(e) Name This Game
Phoenix
2e
2ea
10
10a
Return
Return
2e
10a
(1.576 (1/1)) + 6.330 55-1
R = (2. (2.466-3. 2661, 1.50e
Sea
1.0
2.0
5.0
0.8
1.0
1.5
Loss
1/ Loss
(x) *bbert
(h) Space Invaders
Figure 17: BC return optimal loss. Full results for all Atari games.
31

---

Published in Transactions on Machine Learning Research (12/2024)
K
Descriptions of observation and action space
For Atari, the obscrvation space consists of an 84 x 84 image of pixel values between 0 and 255. The action space discrete and varies per game. See below for the size of the action space per game:
Battle Zone: 18
<bbert:
Bank Heist: 18
Boxing: 18
Breakout: 4
Name This Game:
Phoenix: 8
Space Invaders:
For NetHack, the observation space consists of 24 x 80 ASCII character and color grids (one for each), along with the cursor position (consisting of 2 coordinates). The action space discrete and consists of 121 actions for the full game (i.e. NetHachchallenge-vv) and 23 actions for the simplified version (i.e. NetHHackScrre-v)..
L Reward densities per game
Below provide for every environment the average number of steps the expert takes before reward is received from the environment:
NetHack:
2. Phoenix: 76
BattleZone: 20
4. Breakout: 10
BankHeist: 8
NamoThisGame: 5
7. SpaceInvaders:
Boxing:
9. Qbert: 3
32


Reviews:
[{"review_id": "ewe4wRHy9u", "reviewer": "Reviewer_pNmf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* The paper is well-written, and an extensive batch of experiments was performed.\n* The scaling laws highlight a smooth dependence of loss/rewards on controllable parameters: data size, compute, and the number of parameters.\n* Prediction for BC indicates a better-performing NH agent in the offline setting.\n\nWeaknesses: \n* The biggest weakness of the paper is the missed opportunity to showcase the application of scaling laws to RL settings (i.e., when RL agents are trained online from scratch). This line of questions is perhaps the most interesting one that could be asked in the paper and which is missing. In particular,\n\t* Figure 4 is only marginally discussed in the paper.\n\t* Section 4.3 does not use predictions for reward-optimal models in RL.\n\t* Appendix H sets a reward target out of reach for the current AI models, i.e., 127k of an average human score (vs 10k of a state-of-the-art AI model). This yields scaling law predictions of model-data size pairs (4.4B, 13.2T) and (67B, 0.93T), respectively. These sizes of models are not only a challenge for current RL methods but also require significant computational resources (in comparison, the biggest model trained for NetHack in the paper has 200M parameters). The Authors use this as an excuse to discard such an analysis as a future work. \n\t* However, one could ask for optimal-model predictions for RL rewards within the range of the currently available methods (say score in the rage 1k-20k). The answers would either result in new SOTA models or highlight a gap between predictions and the actual results. I would expect there to be a breaking point for the validity of predictions, which would open up an interesting set of research hypotheses. For instance, how good is observed BC/RL behavior in predicting unobserved RL rewards, are there natural barriers e.g., related to computational complexity, what are the conclusion for LLMs, reasoning, etc.\n* The paper does not discuss some well-known issues with NetHack's reward structure being misleading.\n* Models between training modes and environments are not comparable. Indeed, the NetHack BC version uses transformers, the RL version uses LSTM, and Atari uses CNNs. \n* The paper could provide more discussion concerning the shape of scaling-laws curves, i.e., their behavior with respect to the changes in the variables of interest ($N$, $D$, and $C$).\n\nOther questions and remarks:\n* Did the authors verify the hypothesis that \"for any game score to be a natural performance metric, it needs to be at least somewhat dense\". For example, are the results for Atari games that do not satisfy the \"density of rewards\" criterion not scale smoothly?\n* Did the Authors try to explain the difference between BC methods vs Offline RL (as shown in Table 2)? In particular, some guidance in this direction can be gained in [1], which formulates a number of \"practical observations.\"\n* Although realized in a different manner, the approach on a high-level resembles upside-down RL [2], where rewards are mapped into actions. \n* In Figures 1-2, (b) is not aligned with (c)-(d), which makes it inconvenient to read out the values for extrapolation.\n\nDemonstrating that BC loss, BC rewards, and RL rewards adhere to scaling laws (scale smoothly model size, number of parameters, and compute) is a nice empirical finding. Therefore, both `Yes` in the categories `Claims And Evidence` and `Audience` below have to be treated as borderline positive sentiment. However, the interested audience would be much bigger if the claims had an impact on predicting the agent performance in the RL setting, or the lack thereof, thereby opening a set of interesting questions and hypotheses. \n\n[1] Kumar, A., et al. When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?, ICLR 2022.\n\n\n[2] Schmidhuber, J. Reinforcement Learning Upside Down: Don’t Predict Rewards - Just Map Them to Actions, 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces scaling laws for eight Atari games (selected according to the density of rewards) and NetHack in three variants: behavioral cloning (BC) loss, BC returns, and reinforcement learning (RL) returns. Scaling laws are empirical findings that show smooth dependence of the quantity of interest (such as validation loss or rewards), which is typically unknown before training the model on three quantities that are known and controllable at the start of the training (number of model parameters, data size, and computational budget). To arrive at the scaling laws, 12 models were trained for Atari (ranging from 1k to 5M parameters) and 10 for NetHack (ranging from 200k to 200M parameters). The computational budgets span between 1e14 and 5e18 FLOPS. The paper showcases the use of such scaling laws by predicting (via extrapolation) the loss/reward optimal model size, given a fixed number of samples (40B and 55B). The resulting model reaches the score of 7.8K as a Human Monk in NetHack in the offline setting, beating the previous score of `diff History LM` by 1.7$\\times$.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yc5JTmM6lZ", "reviewer": "Reviewer_j1NP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The article is well focused on a specific question. It is mostly experimentally driven, and I think the authors do a reasonable job of exploration: without regard for practicality more data and more domains would always be better, but there are practical limits for any single effort and the experiments used here seem adequate for suggesting a general trend. The best-fit trends proposed by the authors has utility: they show that selecting a choosing a larger set of parameters suggested for a larger compute budget allows the authors to train a better imitation learning agent than seen in prior work.\n\nThere were two things that I hoped the article would address, both relating to helping a reader generalize these results beyond NetHack and 6 Atari games.\nFirst, some more discussion of how to interpret the results. The authors do a good job of describing what they're doing, and present the results, but leave analysis and interpretation to the reader. What conclusions should a reader take, and hope to transfer to their own domain? What should a reader take from the similarity in loss scaling between the IsoFLOP data and the parametric fit, but the dissimilarity in return scaling? Should they just use the parametric model to estimate the scaling in their own domain, as the parametric model is slightly more informative? Should a reader expect the demonstrated scaling to transfer at all to a new domain, or is the only general conclusion \"bigger is better, with the right parameter / data balance\"?\n\nSecond, some discussion availability of data. There are a couple of ways that IL in games differs from LLMs. Compared to written text for LLMs, It is much more likely that there is a limited amount of expert data available for a specific game. Another difference is that the expert might be a trained policy: how does cloning the expert scale compared to training the expert?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work looks at scaling laws for imitation learning in games: exploring a wide range of model sizes and training set sizes, and showing that for a given compute budget the optimal choices are well-fit by an exponential relationship. It looks at behavior cloning of a PPO agent in six Atari games and a rule-based expert system in NetHack, varying the cloned model's size and the data available to the BC training process.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Repeating the comments from above, I think the paper could be improved by trying to draw a few more conclusions from the data.\n\n\nMinor comments\nWhat were the selection criteria for the six Atari games?\n\n\"imitation learning has powered some of the most impressive feats of AI in recent years. AlphaGo (Silver et al., 2016) used imitation on human Go games\"\nAt least personally, this particular example brings up a slightly awkward association, where the subsequent AlphaZero work noted that IL was a detriment to performance. If the authors have other well known examples, those examples might avoid that association. This is just noting my personal impression: I have no issues with the authors being aware of this possible association and preferring to leave it as is, with the familiarity of AlphaGo as an example being viewed as a larger benefit.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7TrnJfiAnb", "reviewer": "Reviewer_Fv4n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n1. **Innovative Approach:** The paper takes an innovative approach by applying scaling principles from NLP to the domain of Imitation Learning. This cross-disciplinary inspiration is valuable and opens new avenues for research in IL.\n2. **Comprehensive Experiments:** The authors conduct extensive experiments across Atari and NetHack. The variety of environments provides a robust evaluation of their hypothesis.\n3. **Clear Findings:** The paper presents clear and actionable findings, demonstrating that IL loss and mean return follow scaling laws with respect to FLOPs. This finding is significant as it provides a new understanding of how compute resources impact IL performance.\n4. **State-of-the-Art Results:** In the challenging environment of NetHack, the paper shows that their approach outperforms the previous state-of-the-art (SOTA) by 1.7x, indicating the practical effectiveness of scaling in IL.\n5. **Forecasting Compute Requirements:** By deriving scaling laws, the authors not only analyze current performance but also forecast the compute requirements for training optimal IL agents. This predictive capability is useful for future research and practical applications.\n\n### Weaknesses:\n1. **Limited Scope of Environments:** The paper focuses on environments with relatively dense rewards. While this is acknowledged as a limitation, it restricts the generalizability of the findings to other environments where rewards might be sparse or noisy.\n2. **Dependence on Expert Quality:** The experiments rely on expert agents trained with PPO in environments like Atari and NetHack. The quality of these experts significantly impacts the performance of the IL agents, but the paper does not deeply investigate how variations in expert quality affect the scaling laws.\n3. **Scalability Concerns:** While scaling up model and data sizes is shown to be effective, it raises concerns about the practicality of this approach in real-world applications where compute resources are limited. The paper could benefit from a discussion on the trade-offs between scaling and resource efficiency.\n4. **Overemphasis on Correlation:** The strong correlation between IL loss and mean return is emphasized, but the paper does not fully explore the causality or potential confounding factors that could influence this relationship. A deeper analysis of these correlations would strengthen the conclusions.\n5. **Sparse Discussion on Generalization:** The paper mainly focuses on the immediate performance improvements in specific games. However, it does not provide sufficient discussion on how these findings might generalize to other types of games or tasks within IL. Expanding on this would enhance the impact of the research.\n6. **Limited RL Exploration:** The paper briefly extends its exploration to the RL setting by examining the impact of model size and environment interactions on RL performance; however, the discussion remains quite limited. The authors note that compute-optimal agents in RL also benefit from scaling up model size and interactions, but this observation is based on a single RL algorithm (IMPALA) applied to only one game environment. Considering the diverse nature of RL problems with varying dimensionalities and complexities, the paper would benefit from a deeper investigation into how these scaling laws hold across different RL settings, including both online and offline learning scenarios. Additionally, the need for extensive tuning specific to different problem settings is an important consideration that is not fully addressed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores the impact of scaling up model and data sizes on the performance of agents trained using Imitation Learning (IL) in single-agent games. The authors are motivated by the success of scaling in Natural Language Processing (NLP) and aim to investigate whether similar improvements can be achieved in IL, particularly in constrained environments like single-agent games. They conduct experiments on Atari games and the complex game NetHack, discovering that both IL loss and mean return exhibit scaling laws with respect to the compute budget (measured in FLOPs). The study concludes that scaling up models and data size is a viable approach for developing more capable agents, especially in challenging environments like NetHack.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see weaknesses above.", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MYqAD00jbT", "reviewer": "Reviewer_nPQe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n(1) Timely research question. This is the first work to attempt to characterize the scaling laws of imitation learning ala “Scaling Laws for Neural Language Models” by Kaplan et al for interactive settings, where performance metrics like reward reflect the effects of compounding errors.\n\n(2) Experimentation is thorough. The authors perform evaluations across a large set of single-agent game environments.\n\n(3) The authors achieve state-of-the-art performance for offline-trained, learned agents in NetHack through their experiments.\n\n(4) The writing is clear throughout the paper.\n\n\n----\n\n### Weaknesses\n\n**Is Mean Return Scaling Well-Defined?**\nIn single-agent games, reward functions typically reflect external heuristics for progress that have been hand-engineered by game designers. The quality of these heuristics may be variable, even when they rewards are “dense.” \n\nAs a result, it is not obvious that the “mean return” metric for which the authors compute scaling laws is necessarily well-defined, unlike the “single-step” metrics of IL validation loss or perplexity.\n\nTo be more precise, the observed scaling behaviors of return distributions as a function of model capacity and training FLOPs may be confounded by a shared functional property of the tested reward. Hilton et al. [1] call reward functions “natural performance metrics” if the return of a policy trained with RL (under AlphaZero) “follows a power law in training compute.” In light of this definition, one way to interpret the authors’ return scaling results may be as a confirmation of the IL-variant of this property for the tested Atari and NetHack reward functions. Though the authors cite Hilton et al., they do not discuss this point.\n\n**Some Inconsistency Between Strength of Contribution Claims and Results/Discussion**\nIn the “Discussion” paragraph of Section 5, the authors find mean return scaling to plateau in NetHack. \n\nThis result directly conflicts with the following statement in the “contribution” summary provided in the introduction: “This suggests carefully scaling up model and data size can provide a promising path towards increasingly capable game agents [in NetHack], as well as potentially boost performance in other imitation learning settings.” Similar language can be found in the conclusion.\n\nThe authors have convincingly shown through their experiments that IL agents’ NetHack score follow a power-law in training compute + model capacity in the earliest stages of the game. However, their results suggest the opposite overall conclusion of the stated one; namely, that IL may not be a tenable method for beating, or even matching, the performance of the AutoAscend bot. Similar findings have been reported in existing works studying IL + RL for NetHack [3, 4].\n\n\n---\n\n[1] Hilton, Jacob, Jie Tang, and John Schulman. \"Scaling laws for single-agent reinforcement learning.\" arXiv preprint arXiv:2301.13442 (2023).\n\n[2] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361 (2020).\n\n[3] Wołczyk, Maciej, et al. \"Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem.\" arXiv preprint arXiv:2402.02868 (2024).]\n\n[4] Piterbarg, Ulyana, Lerrel Pinto, and Rob Fergus. \"Nethack is hard to hack.\" Advances in Neural Information Processing Systems 36 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors conduct an ambitious analysis of imitation learning scaling laws in solved (Atari) and unsolved (NetHack) single-agent games. Their contributions are twofold. First, the authors show that imitation learning metrics such as BC validation loss and return may scale with power-laws as a function of model capacity and training flops in certain environments and certain learning regimes, including several Atari games and the “early-game” portion of NetHack. These scaling relationships echo those previously observed in single-agent RL and in the supervised training of language models. Second, their investigations produce a state-of-the-art, offline-trained agent for NetHack.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The first weakness described above can be addressed by an amendment to Section 6. I believe that this change would strengthen the work.\n\nThe second weakness can be addressed by softening the language used to describe the contributions of the paper in the introduction and the conclusion of the paper, aligning it closer with the experimental results of the body. This change is critical to securing my recommendation for acceptance. I believe that there is already a sufficient discussion of possible explanations for the plateau-effect in IL and RL agent mean NetHack returns in the results and discussion sections of the paper.", "broader_impact_concerns": "I have no concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Motivated by the work in the language modeling domain, where it is shown that there are scaling laws that govern how the number of compute FLOPS should scale with the dataset size, this paper explores the existence of scaling laws in imitation learning (behavioral cloning). Several cases with the scaling behaviors are shown for both Atari games and NetHack environment and the results are briefly extended to reinforcement learning. Overall, the paper presents rich experiments that could be useful for future investigation. However, the reviewers and myself find that the results still don't capture sufficient insights that could have potentially been extracted from these experiments. Unlike language, IL is very much dependent on the environment, and therefore the scaling laws in this paper are unlikely to generalize beyond the environments that are studied in this paper in their current form. I find this is not a blocker for publication at TMLR but I think the authors ought to release all the generated data so that future work can look into extracting more insights from these experiments so that hopefully insights that are generalizable for solving imitation learning problems could be extracted from them. The paper is recommended to be accepted with minor revisions that respond to these comments and those of the reviewers as much as possible, quoted below. Congratulations to the authors!

**Reviewer pNmf**: It feels that with the results of the paper, the Authors could offer the readers more insights. The reluctance of using RL scaling laws for prediction, indicates that there might be a fundamental gap between IL and RL setups, an observation that could lead to multiple interesting research directions.

 **Reviewer Fv4n**: To help the paper reach its full potential, I suggest the authors address the following in future work:

* Conduct a more detailed analysis of the IL loss and mean return correlation, and explore how the results generalize across different domains.
* Clarify the comparability of models across different environments and training architectures.
* Expand the application of scaling laws to RL settings to explore their broader relevance.

**Reviewer nPQe**: While the results in the work are novel, in that no prior work has not examined scaling laws for imitation learning settings, the tested environments cover only the extremes of the task space of interest -- NetHack is extremely difficult, while Atari is relatively simple.

**Reviewer j1NP**: Are the claims and made in this submission supported? Yes, because the paper is very hesitant to make any claims. Like other reviewers I want to see something more. Experiment run, analysis done to the point of data fitting and standard error, but hesitance to draw any conclusions. I think the experiments are well done, but I still remain unsure what I or other readers should do with the results. Even directly asking what should readers take away from this paper, the authors' clarified takeaway message is (1) performance might scale with data or model size, maybe (2) maybe your behavior cloning run would a different amount of data or a different model size (3) missing feature hurts agent performance.

**Reviewer j1NP**: It would be nice to get a more precise statement than "like Montezuma's Revenge" but the selection process described in the author response does not make me think there's something hiding in the selection of games. If I was more inclined to recommend the article be published in close-to-current state, I guess I might have been more inclined to get rid of that last ambiguity and ensure that same description of game selection was in the article text.
Decision: Accept (poster)