Small batch deep reinforcement learning
Authors: Johan Samir Obando Ceron
Marc G Bellemare
Pablo Samuel Castro
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Deep Reinforcement Learning
Value based
Batch Size
TL;DR: We uncover the the surprising finding that decreasing the batch size improves the performance of many standard deep RL agents.

Abstract: In value-based deep reinforcement learning with replay memories, the batch size parameter specifies how many transitions to sample for each gradient update. Although critical to the learning process, this value is typically not adjusted when proposing new algorithms. In this work we present a broad empirical study that suggests reducing the batch size can result in a number of significant performance gains; this is surprising, as the general tendency when training neural networks is towards larger batch sizes for improved performance. We complement our experimental findings with a set of empirical analyses towards better understanding this phenomenon.

Small batch deep reinforcement learning
Johan Obando-Ceron' Mila, Université Montréal j Jobaddo0730ggail... com
Marc G. Bellemare Mila, Université de Montréal bellemam@mila, quebeb
Pablo Samuel Castro
Google DeepMind Mila, Université de Montréal
pacâgoogle.com
Abstract
In value-based deep reinforcement learning with replay memories, the batch size parameter specifies how many transitions to sample for each gradient update. Although critical to the learning process, this value typically not adjusted when proposing new algorithms. In this work we present a broad empirical study that suggests reducing the batch size can result in number of significant performance gains; this is surprising, as the general tendency when training neural networks is towards larger batch sizes for improved performance. We complement experimental findings with a set of empirical analyses towards better understanding this phenomenon.
Introduction
One of the central concerns for deep reinforcement learning (RL) is how to efficiently make the most use of the collected data for policy improvement. This is particularly important in online settings, where RL agents learn while interacting with an environment, interactions can be expensive. Since the introduction of DQN [Mnih al., 2015], one of the core components most modern deep algorithms is the use of finite replay memory where experienced transitions are stored. During learning, the agent samples mini-batches from this memory to update its network parameters.
Since the policy used to collect transitions is changing throughout learning, the replay memory contains data coming from mixture of policies (that differ from the agent's current policy), and results in what is known as off-policy learning. In contrast with training data for supervised learning problems, online RL data highly non-stationary. Still, any point during training the replay memory exhibits distribution over transitions, which the agent samples from at each learning step. The number of sampled transitions each learning step known as the batch size, and is meant to produce an unbiased estimator of the underlying data distribution. Thus, in theory, larger batch sizes should be more accurate representations of the true distribution.
Some in the supervised learning community suggest that learning with large batch sizes leads to better optimization [Shallue et al., 2019], since smaller batches yield noisier gradient estimations. Contrastingly, others have observed that larger batch sizes tend to converge to "sharper" optimization landscapes, which can result worsened generalization [Keskar et al., 2017]; smaller batches, on the other hand, seem to result in "flatter" landscapes, resulting in better generalization.
Learning dynamics in deep RL drastically different than those observed in supervised learning, in large part due to the data non-taatinnarity mentioned above. Given that the choice of batch size will have direct influence better understanding of its impact is critical. Surprisingly, to the best of our knowledge there have been studies exploring the impact of the choice of batch size deep RI. Most recent works have focused on related questions, such as the number of gradient updates per environment step [Nikishin al., 2022, D'Oro al., 2023, Sokar et al., 2023], but have kept the batch size fixed.
the agent's 's sample efficiency and ultimate performance, developing
"Work done during an internship at Google DeepMind
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

196 Improvement
10'4
Batch Size improvement over when using QR-DQN
10 Human Norma maddeaddddddde Score
1.8
QR-DQN
10
+37%
1.2
Batch Size:
0.6-
16
32 (dcfault)
0..0
25
50
75
100
Number of Frames (in millions)
Figure 1: Evaluating QR-DQN [Dabney et al., 201 [0] with varying batch sizes over 60 Atari 2600 games. (Left) Average improvement obtained when using batch size of 8 over 32 (default); (Right) Aggregate Interquantile Mean [Agarwal et al., 2021] of human normalized scores. All games run for 3 seeds, with shaded areas displaying 95% stratified bootstrap confidence intervals.
In this work conduct broad empirical study of batch size in online value-based deep reinforcement learning. We uncover the surprising finding that reducing the batch size seems to provide substantial performance benefits and computational savings. We showcase this finding variety of agents and training regimes (section 3), and conduct in-depth analyses of the possible causes (section 4). The impact of our findings and analyses go beyond the choice of the batch size hyper-parameter, and help us develop a better understanding of the learning dynamics in online deep RL.
2 Background
reinforcement learning problem is typically formulated as a Markov decision process (MDP), which consists of a 5-tuple (S, A, P. R, 7.). where S denotes the state space, A denotes the actions, P x - (S) encodes the transition dynamics, R x A + R is the reward function, and y E [0, 1) is a discount factor. The aim is to learn a policy : S > A parameterized by such that the sum of discounted returns Es, [2te 7'rr] is maximized; here, the state-action trajectory (So, ao, $1, ......) is obtained by sampling 8:41 P (- Sz, ar) at each decision step f, and r; ~ R R. Sr, a:).
action a na (- 5z) and reaching state
In value-based methods. the policy is obtained as the argmax of a learned Q-function: Ta(s) = arg m (s,s, a). This function aims to approximate the optimal state-action values Q", defined via the well-known Bellman recurrence: (s, a,) = max, E Rs,,, a 7Q" (s a,+1) ), and typically learned using Q-learning [Watkins and Dayan, 1992 Sutton and Barto, 2018).
To deal with large state spaces, such as all possible images in an Atari 2600 game, Mnih al. [2015] introduced DQN, which combined Q-learning with deep neural networks to rep- resent Q9. A large replay buffer D is maintained to store experienced transitions, from which mini-batches are sampled perform learning updates [Lin, 1992]. Specifically, temporal dif- ference learning is used to update the network parameters with the following loss function: L(0)= E .y ( (>nn((((r) + y max (st41, (04441444+1)) a++1)) - Qo a.) Here Qa is a tar- get network that is a delayed copy of Qo. with the parameters synced with Q9 less frequently than Qo is updated.
Since the introduction of DQN, there have been a number of algorithmic advances in deep RL agents, in particular those which make use of distributional RL [Bellemare et al., 2017], introduced with the C51 algorithm. The Rainbow agent combined C51 with other advances such as multi-step learning and prioritized replay sampling [Hessel et al., 2018]. Different ways of parameterizing return distributions were proposed in the form of the IQN [Dabney et al., 201 8b] and QR-DQN [Dabney al., 2018al algorithms. For reasons which will be clarified below, most of our evaluations and analyses were conducted with the QR-DQN agent.
3 The small batch effect on agent performance
In this section we showcase the performance gains that arise when training with smaller batch sizes. We do so first with four standard value-based agents (63.1), with varying architectures ($3.2), agents optimized for sample efficiency ($3.3), and with extended training ($3.4). Additionally, we explore the impact of reduced batch sizes exploration ($3.5) and computational cost ($3.6).

---

DON
1.25
0R-DON
1.5
I2N
1.00
3.0
0.6
0.5
0.5
32 (defoat)
0.25
0.0
0.00
0.0
20 05
60
80
105
20
46
50
80
100
20
45
65
6b
100
20
40
60 60 100
Number of Frames (In millions)
Number of Frames million))
Number of Frames (In milliors)
Number Frames (In millions)
Figure 3: IQM for human normalized scores for DQN, Rainbow, QR-DQN, and IQN over 20 games. All games run with 3 independent seeds, shaded areas representing 95% confidence intervals.
We use the Jax implementations of RL agents, with their default hyper- parameter values, provided by the Dopamine library [Castro al., 2018]² and applied the Arcade Learning Environment (ALE) [Bellemare et al., 2013].² is worth noting that the default batch size is 32, which we indicate with a black color in all the plots below, for clarity. We evaluate our agents on 20 games chosen by Fedus et al. [2020] for their analysis of replay ratios, picked to offer a diversity of difficulty and dynamics. To reduce the computational burden, we ran most of our experiments for 100 million frames (as opposed to the standard 200 million). For evaluation, we follow the guidelines of Agarwal et al. 12021]. Specifically, we run independent seeds for each experiment and report the Humman-normalizee
Experimental setup:
interguantile mean (IQM), aggregated over the 20 games, configurations, and seeds, with the 95% stratified bootstrap confidence intervals. Note that this means that for most of the aggregate results presented here, we are reporting mean and confidence intervals over 60 independent seeds. All experiments were run on NVIDIA Tesla P100 GPUs.
3.1 Standard agents
We begin by investigating the impact reducing the batch size can have on four popular value-based agents, which were initially benchmarked on the
JOM Human Normaltzed Score
2.0
ORIDON
2.0
ORIOON
Ronn
2.5-
1.5
ALE suite: DQN [Mnih et al., 2015], Rainbow [Hessel et al., 201 8] (Note that Dopamine uses "compact" version of the original Rainbow agent, including only multi-step updates, prioritized re- play, and C51), OR-DQN [Dabney al., (a],
Defasdt CHN
Default CRN
2.0-
1.0
Defauit CNN
Detault CAN
0.5-
0.5
and IQN [Dabney et al., 201 8b]. In Figure 3 we can observe that, in general, reduced batch size results in improved performance. The notable exception is DQN, for which we provide an analysis and ex- planation for why this is the case below. To verify that our results are not a consequence of the set of 20 games used in analyses, (where the effect is most observed) over the full 60 games in the suite and report the results in Figure 19, Remarkably, batch size of 8 results in significant gains on 38 out of the full 60 games, for an average performance improvement of 98.25%
0.00
0.0
0
50
100
50
100
Number
millions
Figure 2: IQM for human normalized scores with varying neural network architectures over 20 games, with 3 seeds per experiment. Shaded areas represent 95% stratified bootstrap confi- dence intervals.
ran QR-DQN
3.2 Varying architectures
Although the CNN architecture originally introduced by DQN [Mnih et al., 2015] has been the backbone for most deep RL networks, there have been some recent works exploring the effects of varying architectures [Espeholt et al., 2018, Agarwal et al., 2022, Sokar et al., 2023]. We investigate the small batch effect by varying the QR-DQN architecture in two ways: (1) expanding the convolutional widths by 4 times (resulting in substantial increase in the number of parameters), and (2) using the Resnet architecture proposed by Espcholt et al. [2018] (which results in a similar number of parameters the original CNN architecture, but is deeper network). In Figure 2 we can observe that not only do reduced batch sizes yield improved performance, but they are better able to leverage the increased number of parameters (CNNx4) and the increased depth (Resnet).
Dooamine code available at 'Dopamine sticky actions by default [Machado et al., 2018].
3

---

3.25
2.0
+16%
1:25
112%
ON Human Wermaltzes
1.00
+58%
1.00
0.75
0.75
1.0
0.50
Butch Size
0.50
0.25
32 (3dfaut)
0.00
0.0
0.00
35
0.01 c:55 0.11
20
0.1 3.00 0.11
Number of Trammessc (in Millios))
Number of fremes (In Millons)
Number of Fraiees in Millinss
Figure 4: Measured IQM of Goman-normalized scores on the 26 100k benchmark games, with varying batch sizes, of DER, SPR, and DrQ(c). We evaluate performance 100k agent steps (or 400k environment frames), and at 30 million environment frames, run with 6 independent seeds for each experiment, and shaded areas display 95% confidence intervals.
3.3 Atari 100k agents
There has been an increased interest in evaluating Atari agents on very few environment interactions, for which Kaiser et al. [2020] proposed the 100k benchmark'. We evaluate the effect of reduced batch size on three of the most widely used agents for this regime: Data-efficient Rainbow (DER), a version of the Rainbow algorithm with hyper-parameet et al., 2019]: DrQ(e), which is variant of DQN that uses data augmentation [Agarwal et al., 202 I]; and SPR, which incorporates sele-supe learning improve sample efficiency [Schwarzer et al., 2020]. For this evaluation we evaluate on the standard 26 games for this benchmark [Kaiser et al., 2020], aggregated over independent trials.
tuned for faster early learning [van Hasselt
In Figure we include results both at the 100k benchmark (left side of plots), and when trained for 30 million frames. Our intent evaluate the batch size effect on agents that were optimized for a different training regime. We can see that although there is little difference in 100k, there is a much more pronounced effect when trained for longer. This finding suggests that reduced batch sizes enables continued performance improvements when trained for longer.
3.4 Training Stability
LOM Human Normalized Score
QR
Size
To further investigate whether reduced batch sizes enables continual improvements with longer train- ing, we extend the training of QR-DQN up to the standard 200 million frames. In Figure 5 we can that training performance tends to plateau for the higher batch sizes. In contrast, the smaller batch sizes to be able to continuously improve their performance.
1.5
32
+93%
1.0
0.5
1OOM
0.0
0
50
100
150
200
Number of Fromes (in millions)
3.5 Impact on exploration
Measuring IQM for human- normalized scores when training for 200 mil- lion frames. Results aggregated over 20 games, where each experiment was run with 3 inde- pendent seeds and report 95% confidence intervals.
Figure 5:
The simplest and most widely used approach for exploration to select actions randomly with a probability c, as opposed to selecting them greedily from the current Qo estimate. The increased vari- ance resulting from reduced batch sizes (as will explore in more depth below) may also result a natural form of exploration. To investigate this, we set the target value to 0.0 for QR-DQN' In Figure we compare performance across four known hard exploration games [Bellemare et al., 2016, Taiga et al., 2020] and observe that reduced batch sizes tends to result in improved performance for these games.
'Here. 100k refers agent steps, 400k environment frames, due skipping frames in the standard training setup.
"Note that we follow the training schedule of Mnih et al. [2015] where the value begins at 1.0 and is linearly decayed to its target value over the first million environment frames.

---

MoneecumaRevenge
Venture
Freeway
Gravitar
100 Human Womme Scom
0N/DONie -
1.25-
2000
1.0
20
Batch Size:
0.75
1000
0.5-
0.5
16
10
(default)
0.25
04
0.0
0.0
5D
160
50
160
50
100
50
100
Number of Frames tnn milli ns!
Figure Left: Performance of QR-DQN on four hard exploration games with target value of 0.0, and with varying batch sizes. Right: Aggregate IQM of numan-normalized scores over 20 games with target e value of 0.0. In all the plots independent seeds were used for each game/batch-size configuration, with shaded areas representing 95% confidence intervals.
Many methods have been proposed to address the ieniitation-eploratatin dilemma, and some techniques emphasize exploration by adding noise directly to the parameter space of agents [Fortunato al., 2018, Plappert et al., 2018, Hao et al., 2023, Eberhard al., 2023], which inherently adds variance to the learning process. Our analyses show that increasing variance by reducing the batch size may result in similar beneficial exploratory effects, as the mentioned works suggest.
3.6 Computational impact
Empirical advances in deep reinforcement learning are generally measured with respect to sample efficiency; that is, the number of environment interactions required before achieving a certain level of performance. It fails to capture computational differences between algorithms. have the same performance with respect to environment interactions, but one takes twice as long to perform each training step, one would clearly opt for the faster of the two. This important distinction, however, is largely overlooked in the standard evaluation methodologies used by the DRL community.
two algorithms
Keraaoo Scor
1.5
Nainbow
CR DGN
1.5
KIN
1.0
1.0-
0.
0.5
0.5
40
75
Time (hours)
Time (hours)
(hours)
Time (hours)
Figure 7:
Measuring wall-time versus IQM of normalize scores when varying batch sizes in DQN (with n-step set to 3), Rainbow, QR-DQN, and IQN over 20 games. Each experiment had independent runs, and the confidence intervals show 95% confidence intervals.
We have already demonstrated the performance benefits obtained when reducing batch size, but an additional important consequence is the reduction computation wall-time. Figure 7 demonstrates that not only can we obtain better performance with reduced batch size, but we can do so at fraction of the runtime. As concrete example, when changing the batch size of QR-DQN from the default value of 32 to 8, we achieve both 50% performance increase and a 29% speedup in wall-time. may seem surprising that smaller batch sizes have faster runtime, since larger batches presumably make better use of GPU parallelism. However, pointed out by Masters and Luschi 12018), the speedups may be a result of a smaller memory footprint, enabling better machine throughput.
Considering the unsuitable increase in computational requirements, progress with deep learning efficient training methods. natural direction is to eliminate algorithmic inefficiencies in the learning process, aiming to reduce time, energy consumption and carbon footprint associated with training these models [Bartoldson et al., 2023, Chen et al., 2021]. Figure 14 illustrates the wall-time reduction when using high-capacity neural networks and smaller batch size value. This motivates a fundamental trade-off in the choice of batch size, and the way of how we benchmark deep reinforcement learning algorithms.
demands more compute-efficient
5

---

Key observations reduced batch sizes:
They generally improve performance, evaluated across variety of agents and network architectures.
When trained for longer, the performance gains continue, rather than plateauing.
They seem to have beneficial effect on exploration.
They result in faster training, measured by wall-time.
4 Understanding the small batch effect
Having demonstrated the performance benefits arising from reduced batch size across wide range of tasks, in this section we seek to gain some insight into possible causes. We will focus on QR-DQN, as this the agent where the small batch effect is most pronounced (Figure 3). We begin by investigating possible confounding factors for the small batch effect, and then provide analyses on the effect of reduced batch sizes on network dynamics.
IOM Normalized Score
1.2
Batch Size
4.1 Relation to other hyperparameters
0.8
(dettaultt
Learning rates It is natural to wonder whether an im- proved learning rate could produce the same effect simply reducing the batch size. In Figure 8 we explored variety of different learning rates and observe that, although performance is relatively stable with batch size of 32, it unable reach the performance gains obtained with a batch size of 8 or 16. Figure 8 shows that the smaller the learning rate, the larger batch size needs to be, and thus the longer training takes. This result aligns well with the findings of Wilson and Mar- tinez [2003].
64
0.4
0.0
5e-06
5e-05
5e-04
Learning rate paramater
Figure 8: Varying batch sizes for different learning values. Results aggregated IQM of human-normalized scores over 20 games for QR-DQN.
Second order optimizer effects All experiments, like most modern RL agents, use the Adam optimizer [Kingma and Ba, 2015], a variant of stochastic gradient descent (SGD) that adapts its learning rate based the first- and second-order moments of the gradients, estimated from mini-batches used for training. It is thus possible that smaller batch sizes have a second-order effect on the learning-rate adaptation that benefits agent performance. To investigate this we evaluated, for each training step, performing multiple gradient updates on subsets of the original sampled batch; we detine the parameter Batch as the number of gradient updates and dividing factor (where value of 1 is the default setting). Thus, for Batch Divisor of 4, we would perform 4 gradient updates with subsets of size 8 instead of single gradient update with mini-batch of size 32. With an optimizer like SGD this has no effect (as they are mathematically equivalent), but we may see differing performance due Adam's 's adaptive learning rates. Figure demonstrates that, while there are differences, these are not consistent nor significant enough to explain the performance boost.
Astwalx
Spaceinueders
0.0
1400
6000
Batch Divisor
1000-
Retums
1200
0.6
1000
4000-
750
0.4-
Mini Sptihhs
800-
2000
500-
0.2
600
25
50
75
100
0
25
50
75
100
250-
100
0.0
25
50
75
100
Number of frames (in millions)
Number frames millions)
Figure 9: Varying the number of gradient updates per training step. for fixed batch size of 32. Left: Performance of QR-DQN on three games with different BatchDivvise value. Right: Results aggregated IQM of Humaa-normalizee scores over 20 games for QR-DQN.
Relationship with multi-step learning In Figure we observed that DQN was the only agent where reducing batch size did not improve performance. Recalling that the Dopamine version of

---

Adding
Rainbow: Remeving
DON
Default
Prforiized Distributional
1.00
Mukti step
0.75
1.0
1.0
0.10
Batch Sbe
0.25
0.5
0.5
16
32 Batch size
8
16
64
0.00
Batch size
Number of frames millions)
1.5
DQN
1.5
Rainbow
1.5
QR-DQN
1.0-
1.0
1.0
54ke7 Siz
0.5-
0.5-
0.5
0.0
0.0
0.0
13
11
5
NStep
Figure 10: Measured IQM human normalized scores over 20 games with 3 independent seeds for each configuration, displaying 95% stratified bootstrap confidence intervals. Top left: Adding components to DQN; Top center: Removing components from Rainbow. Top right: Aggregate DQN performance with n-step 3. 3. Bottom: Varying batch sizes and n-steps in DQN (left), Rainbow (center), and QR-DQN (right).
Rainbow used is simply adding three components to the base DQN agent, we follow the analyses of Hessel al. [2018] and Ceron and Castro [2021]. Specifically, in Figure 10 (top row) we simultaneously add these components to DQN (top left plot) and remove these components from Rainbow (top center plot). Remarkably, batch size inversely correlated with performance only when multi-step returns are used. Given that DQN is the only agent considered here without multi-step learning, this finding explains the anomalous findings Figure Indeed, as the right panel of Figure 10 (top row) shows, adding multi-step learning to DQN results in improved performance with smaller batch sizes. To further investigate the relationship between batch size and multi-step returns, in Figure 10 (bottom row) we evaluate varying both batch sizes and n-step values for DQN, Rainbow, and QR-DQN. We can observe that smaller batch sizes suffer less from degrading performance as the n-step value is increased.
Key insights:
The small batch effect does not to be consequence of sub-optimal choice of learning rate for the default value 32.
The small batch effect does not arise due to beneficial interactions with the Adam optimizer.
The small batch effect appears to be more pronounced with multi-step learning.
When increasing the update horizon multi-step learning, smaller batches produce better results.
4.2 Analysis of network optimization dynamics
In this section we will focus on three representative games (Asteroids, DemonAttack, and SpaceIn- vaders), and include results for more games in the supplemental material. In Figure 11 we present the training returns as well as a variety of metrics we collected for our analyses. We will discuss each in more detail below. The first column in this figure displays the training returns for each game, where we can observe the inverse correlation between batch size and performance.
Variance of updates Intuition suggests that as we decrease the batch size, we will observe an increase in the variance our updates as our gradient estimates will noisier. This is confirmed in the second column of Figure 11, where we see increased variance with reduced batch size. A natural question whether directly increasing variance results in improved performance, thereby (partially) explaining the results with reduced batch size. To investigate, we added Gaussian noise (at

---

Returns
LossVariance
GrodientNorm
Representationtorm
2.0
Asteroids
2.5-
40
10-
2.0-
20-
5-
0-
50
100
50
100
50
100
50
100
50
100
50
X10
X10*
X10'
20
DemenAttack
2
10
10
0.5-
0-
50
100
0
50
100
50
100
160
50
100
50
Spacelnvaders
20
4+
1.
10
20
50 1.00
50
100
50
100
100
50
100
Number of Frames (in millions)
Figure 11: Empirical analyses for three representative games with varying batch sizes. From left right: training returns, aggregate loss variance, average gradient norm, average representation norm, srank [Kumar et al., 2021a], and dormant neurons [Sokar et al., 2023]. All results averaged over seeds, shaded areas represent 95% confidence intervals.
varying scales) to the learning target Qi (see section 2 for definition). As Figure 12 demonstrates, simply adding noise the target does provide benefits, albeit with some variation across games.
Asteroids
DemonAttack
Spacelnvaders
Stoee
0.8-
1500
Target Naide Scale
3000
Returs
10000
1000
2000
TargetNo Scale
None
5.00
1000-
1.0
0.04
25
50
75
100
25
50
75
100
25
50
75
100
0
25
50
75
100
Number of frames (in millions)
Number of Frames (in millions)
Figure 12: Adding noise of varying scales to the learning target with the default batch size of 32. Left: Performance of QR-DQN on three games with different target noise scale values. Right: Results aggregated IQM of human- normaiizr scores over 20 games for QR-DQN.
Gradient and representation norms Keskar et al. [2017] and Zhao et al. [2022] both argue that smaller gradient norms can lead to improved generalization and performance, in part due to less "sharp" optimization landscapes. In Figure 11 (third column) correlated with gradient norms, which may be an important factor in the improved performance. In Appendix D, we conducted experiments trend: better performance is achieved with smaller batch sizes and gradient norms.
that batch size is, in fact,
different subset of games, and observed consistent
There have been number of recent works suggesting RL representations, taken to be the output of the convolutional layers in our networks yield better agent performance when their norms smaller. Gogianu et al. [2021] demonstrated that normalizing representations yields improved agent performance as a result of a change to optimization dynamics; Kumar et al. [202 Ib] further observed that smaller representation norms can help mitigate feature co-adaptation, which can degrade agent performance the offline setting. As Figure 1I (fourth column) shows, the norms of the representations correlated with batch size, which aligns well with the works just mentioned.
Effeet on network expressivity and plasticity Kumar et al. [2021a] introduced the notion of the ef fective rank of the representation sranks(o)) and argued that it is correlated with network's expres-
"This is common interpretation used recently, for example, by Castro et al. [2021]. Gogianu et al. [2021]. and Farebrother et al. [2023] 78 is threshold parameter. We used the same value of 0.01 used by Kumar et al. [2021a].

---

sivity: a reduction in effective rank results in an implicit under-pan parameterization. The authors provide evidence that bootstrapping the likeliest cause for effective rank collapse (and reduced performance). Interestingly, in Figure 11 (fifth column) we see that with smaller batch sizes srank collapse occurs earlier in training than with larger batch sizes. Given that there is mounting evidence that deep RL networks tend to overfit during training [Dabney et al., 2021, Nikishin et al., 2022, Sokar et al., 2023], it is possible that the network is better able to adapt to an earlier rank collapse than to a later one.
Spas
Sae
Figure 13: Gradient covariance matrices for As- teroids (left) and SpaceInvaders (right). In envi- ronments where smaller batch size significantly improves performance, it also induces weaker gra- dient correlation R and less gradient interference.
To further investigate the effects on network ex- pressivity, we measured the fraction of dormant neurons (neurons with near-zero activations).
Sokar et al. [2023] demonstrated that deep RL agents suffer from increase in the number of dormant neurons in their network; further, the higher the level of dormant neurons, the worse the performance. In Figure 11 (rightmost column) we can see that, although the relationship with batch size is not as clear as with some of the other metrics, smaller batch sizes appear to have a much milder increase in their frequency. Further, there does appear be close relationship with the measured srank findings above. Lyle et al. [2023] evaluated the covariance structure of the gradients to revisit the network's loss landscape, and argue that weaker gradient correlation and less gradient interference improve performance. We observe similar results in the gradient covariance heat maps shown in Figure 13 and Figure 16, where gradients appear be largely colinear when using larger batch size values.
Key insights:
Reduced batch sizes result in increased variance of losses and gradients. This increased variance can have a beneficial effect during training.
Smaller batch sizes result in smaller gradient and representation norms, which tend to result in improved performance.
Smaller batch sizes seem to result in networks that are both more expressive and with greater plasticity.
5
Related work
There is considerable amount of literature on understanding the effect of batch size supervised learning settings. Keskar et al. [2016) presented quantitative experiments that support the view that large-batch methods tend to converge to sharp minimizers the training and testing functions, and as has been shown in the optimization community, sharp minima tends to lead to poorer generalization. Masters and Luschi [2018] support the previous finding, presenting an empirical study of stochastic gradient descent's 's performance, and reviewing the underlying theoretical assumptions surrounding smaller batches. They conclude that using smaller batch sizes achieves the best training stability and generalization performance. Additionally, Golmant al. [2018] reported that across wide range of network architectures and problem domains, increasing the batch size yields no decrease in wall-clock time convergence for either train or test loss.
Although batch size is central to deep reinforcement learning algorithms, has not been extensively studied. One of the few results in this space is the work by Stooke and Abbeel [2018]. where they argued that larger batch sizes can lead to improved performance when training in distributed settings. Our work finds the opposite effect: smaller batch sizes tends to improve performance; this suggests that empirical findings may not directly carry over between single-agent and distributed training scenarios. Islam et al. [201]] and Hilton et al. [2022] have investigated the role of batch size in on-policy algorithms. The latter demonstrates how to make these algorithms batch size-invarian invariant, aiming sustain training efficiency small batch sizes.
Dark red color refers to high negative correlation, and dark blue one high positive correlation.

---

Lahire et al. [2021] cast the replay buffer sampling problem as an importance sampling one, allowing it to perform well when using large batch. Fedus al. [2020] presented a systematic and extensive analysis of experience replay in Q-learning methods, focusing on two fundamental properties: the replay capacity and the ratio of learning updates to experience collected (e.g. the replay ratio). Although their findings are complementary to ours, further investigation into the interplay of batch size and replay ratio is an interesting avenue for future work. Finally, there have been a number of recent works investigating network plasticity [Schwarzer et al., 2023, D'Oro et al., 2023, Sokar et al., 2023, Nikishin et al., 2022], but all have kept the batch size fixed.
Wolczyk and Krutsylo [2021] investigate the dynamics of experience replay in online continual learning, and focus on the effect of batch size choice when sampling from a replay buffer. They find that smaller batches are better at preventing forgetting than using larger batches, contrary to the intuitive assumption that it is better to recall more samples from the past to avoid forgetting. Additionally, the authors show that this phenomenon does not disappear under learning rate tuning. Their settings are similar to those used to generate Figure 3 in [Sokar et al., 2023], and suggest that target non-stationarity (e.g. bootstrapping) may have a role to play in explaining the small batch size effect we are observing.
Conclusions
In online deep RL, the amount of data sampled during each training step is crucial to agent's learning effectiveness. Common intuition would lead one to believe that larger batches yield better estimates of the data distribution and yield computational savings due to data parallelism on GPUs. Our findings here suggest the opposite: the batch size parameter generally alters the agent's learning curves in surprising ways, and reducing the batch size below its standard value is often beneficial.
From a practical perspective, our experimental results make clear that the effect of batch size on performance is substantially more complex than in supervised learning. Beyond the obvious performance and wall-time gains we observe, changing the batch size appears to have knock-on effects exploration as well as asymptotic behaviour. Figure& hints at complex relationship between learning rate and batch size, suggesting the potential usefulness of "scaling laws" for adjusting these parameters appropriately.
Conversely, our results also highlight a number of forcement learning. For example, one would naturally expect that decreasing the batch size should increase variance, and eventually affect prediction accuracy. That its effect on performance, both transient and asymptotic, should so critically depend the degree to which bootstrapping occurs (as in re-teep returns; Figure 10), suggests that gradient-based temporal-differt difference learning algorithms need a fundamentally different analysis from supervised learning methods.
effects in deep rein-
Future Work Our focus in this paper has been on value-based online methods. This raises the question of whether our findings carry over actor-critic methods, and different training scenarios such as offline RL [Levine et al., 2020] and distributed training [Stooke and Abbeel, 2018]. While likely for actor-critic methods, the dynamics sufficiently different in offline
similar findings RL and in distributed training that it would likely require a different investigative and analytical approach. It is also interesting direction to explore adaptive schemes that dynamically varies the batch size during training. Our experiments used a constant batch size, so further research needed determine whether it is advantageous to reduce the batch size over time in practice, as well as how quickly it should be reduced.
Our work has broader implications than just the choice of the batch size hyper-parameter. For instance, our findings on the impact of variance on performance suggest a promising avenue for new algorithmic innovations via the explicit injection of variance. Most exploration algorithms are designed for tabular settings and then adapted for deep networks; our results in section 3.5 suggest there may be opportunities for exploratory algorithms designed specifically for use with neural networks. We hope our analyses can prove useful for further advances in the development and understanding of deep networks for reinforcement learning.
10

---

Acknowledgements. Many thanks to Georg Ostrovski and Gopeshh Subbaraj for their feedback on an earlier draft of this paper. We also acknowledge Max Schwarzer, Adrien Ali Taiga, Rishabh Agarwal and Jesse Farebrother for useful discussions, well the rest of the DeepMind Montreal team for their feedback on this work. The authors would also like to thank the anonymous reviewers for useful feedback this paper. Last but not least, we would also like to thank the Python community [Van Rossum and Drake Jr, 1995, Oliphant, 2007] for developing tools that enabled this work, including NumPy [Harris et al., 2020], Matplotlib [Hunter, 2007] and JAX [Bradbury et al., 2018].
Broader impact Although the work presented here is mostly of an academic nature, it aids in the development of more capable autonomous agents. While our contributions do not directly contribute to any negative societal impacts, urge the community to consider these when building our research
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In ThirrtyFifth Conference on Neural Information Processing Systems, 2021.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Beyond tabula rasa: Reincarnating reinforcement learning. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
Brian Bartoldson, Bhavya Kailkhura, and Davis Blalock. Compute-efficient deep learning: Algo- rithmic trends and opportunities. Journal of Machine Learning Research, 24:1-77, 2023.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013. doi: 10. 10.161 3/jai.....3
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 29, Curran Associates, Inc., 2016. URL https://procoeddnns neurips.cc/ppppe cc/paper_ .df.
Mare G. Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, page 449-458, 2017.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, al. Jax: compos- able transformations of python+ numpy programs. 2018.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforeement Learning. 2018. URL http: // //arxiv.. werriii.rgg/eeeee 1abs//812.066 06110.
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. MICo: Learning improved representations via sampling-hased state similarity for Markov decision processes. In Advances in Neural Information Processing Systems, 2021.
Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more in-
sightful and inclusive deep reinforcement learning research. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1373-1383. PMLR, 18-24 Jul 2021. URL attps://proceedingg
mlr. press/ V131///eeoo//0 html.
Lili Chen, Kimin Lee, Aravind Srinivas, and Pieter Abbeel. Improving computational efficiency in visual reinforcement learning via stored embeddings. Advances Neurul Information Processing Systems, 14426779-26799, 2021.

---

W. Dabney, M. Rowland, Mare G. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In AAAI, 2018a.
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1096-1105. PMLR, 2018b.
Will Dabney, Andre Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G. Bellemare, and David Silver. The value-improvement path: Towards better representations for reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023. URL https: //wppe/eeee..neeeeeee.eeeeeeeeee wwwp://eee/oommmmtttooeom JeBBBJ..
Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https: https::/ http:////ppeeeeeeeeeeeeeeeeeeeee http//ormmmicc hQQV5QN27eS..
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning). ICML' 18, 2018.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Mare G Bellemare. Proto-value networks: Scaling representation learning with auxiliary tasks. In The Eleventh International Conference on Learning Representations, 2023.
URL https: //ppn//eeeeeeeeeeeeeemeeeeeeeeee net//forummid=oDDSSee...
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International Conference Machine Learning, pages 3061-3071 PMLR, 2020.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alexander Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In Proceedings of the International Conference Representation Learning (ICLR 2018), Vancouver (Canada), 2018
Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: An optimisation perspective. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3734-3744. PMLR, 18-24 Jul 2021.
Noah Golmant, Nikita Vemuri, Zhewei Yao, Vladimir Feinberg, Amir Gholami, Kai Rothauge, Michael W Mahoney, and Joseph Gonzalez. On the computational inefficiency of large batch sizes for stochastic gradient descent. arXiv preprint 1811v:1811..2999 2018
Jianye Hao, Tianpei Yang. Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: From single-agent to multiagent domain. IEEE Transactions on Neural Networks and Learning Systems, pages 1-21, 2023. doi: 10.1109/tnnls.2003
URL httpp:://aee //doi. http://ooooooog//// 10. 1109%2Ftnnls 2023. 3236361.
Charles R Harris, K Jarrod Millman, Stéfan Van Der Walt, Ralf Gommers, Pauli Virtanen, David Counaaeaa,, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel Smith, et al. Array program- ming with numpy. Nature, 585(7825):3573622 2020.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
12

---

Jacob Hilton, Karl Cobbe, and John Schulman. Batch size-invariance for policy optimization. Advances in Neural Information Processing Systems, 35:17086-17098, 2022.
John Hunter. Matplotlib: 2d graphics environment. Computing in science & engineering. 9(03): 90-95, 2007.
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of bench- marked deep reinforcement learning tasks for continuous control, 2017.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blažej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2020. URL https:// openreview meet/forumiddS111 JHtDB.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04833 2016.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL http::::/oppereeieeeeeeeee//
forum? forum?id=Hl.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL attp:://arxe arxiv. .rgg /a//ab/////111 1412.6980..
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- parametrrizatiioon inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021a. URL https: hatpaa/:////evii///e/e
http:///aweeeeeeeeeee///e//////a het//forumiidd000mniast
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In International Conference Learning Representations, 2021b.
Thibault Lahire, Matthieu Geist, and Emmanuel Rachelson. Large batch experience replay. In International Conference on Machine Learning, 2021. URLpttps: https///pppp..ssicsseieeeeeeeeee www.ntiischoll.. org prrg/orrpuIID: 238259488.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0/643 2020.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Mach. Learn., 8(3 (0:344)2933333 3211, May 1992.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney, Understanding plasticity in neural networks. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 23 23191-23211. PMLR, 23-29 Jul 2023. URL https://proceedingg mir. mlr. cress/v202/lyye233 html.
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Artif. Int. Res., 61(1):523-562 jan 2018. ISSN 1076-9757.
Dominie Masters and Carlo Luschi. Revisiting small batch training for deep neural networks. ArXiv, abs/1804.076 12, 2018
13

---

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 February 2015.
Exgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning- In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 16828-16847 PMLR, 17-23 Jul 2022.
Travis E. Oliphant. Python for scientific computing. Computing in Science & Engineering. 9()): 10-20, 2007. doi: 10.11099MCCE22207..55
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations, 2018. URL net (alorrumidd-yBBA220
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self- self-predictive representations. In International Conference Learning Representations, 2020.
Max Schwarzer, Johan Samir Ohando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference Machine Learning, pages 30365-30380. PMLR, 2023.
Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20( 12):1-49, 2019. URL http://jmlr http:///mmm.o..rgeeeeeee htrgpapprss/v2// 188-789... html.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phe- nomenon in deep reinforcement learning. In ICML, 2023.
Adam Stooke and Pieter Abbeel. Accelerated methods for deep reinforcement learning. CoRR, 0010020301111 2018. URL httt:://rxiv. http://arwvv.rggabba////asa/a 1803.8033000 02811.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Adrien Ali Taiga, William Fedus, Marlos C. Machado, Aaron Courville, and Marc G. Bellemare. On bonus based exploration methods in the arcade learning environment. In International Conference net net//frwmriddBBeelyyson
on Learning Representations, 2020. URL
Hado P van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in re- inforcement learning? In H. Wallach, H. Larochelle, A. Beygelzimer, d' A'chéé Buc, Fox, and R. Gareet,, editors. Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https: / http://ppoceee...ceee// wwwpiiinnnnneuiss... wwtpseet.eee...pppeeeeeeeeeeeeee wttpc//pe////9////.... 1b742ae215adf 197544966622224922222 Paper pdf.
Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning. 8(3):279-299) 1992.
D.Randall Wilson and Tony R. Martinez. The general inefficiency of batch training for gradient descent learning. Neural Networks, 16(10): ((110):442-14511 2003. ISSN 0893-6080. doi: https://doi. 0t articllepp1//00996000033333
URL ststp::/www..ccieneee http:///weeeeeeetccccceeeeeeeeee com/science/
Maciej Wotezyk and Andrii Krutsylo. Remember more by recalling less: Investigating the role of batch size in continual learning with experience replay (student abstract). 35(18):15923--1924 2021.
14

---

Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 26982- 26992. PMLR, 17-23 Jul 2022. URL https: h http////poceeenns..e.eee./.esses mlr. wwwppprssssi..e..eee v112//1622//2 222222. html.
15

---

and we specify the A Code availability
Our experiments were built on open source code, mostly from the Dopamine repository. The root directory for these is https://gtthub.com httppittuu..ooo//./ honpgggg///pammmm/////aattrrrooo subdirectories below (with clickable links):
DQN, Rainbow, QR-DQN and IQN agents from /jax/agents/
Atari-100k agents from lubs/atari-100k
Batch size from jaaa//ggnskqqaatllloooiiiiigaagr (line 36)
Exploration from /awp///gnnkaaaaiiilooo//////a/// (line 16)
Resnet from / (line 108)
Dormant neurons metric from /labs/redo/
For the srank metric experiments we used code from:
geneeralizationreepeeenetttaioii http://ii....aiacco/////oooooone
Atari 2600 games used B
Most our experiments were run with 20 games from the ALE suite [Bellemare al., 2013]. as suggested by Fedus et al. [2020]. However, for the Atari 100k agents (subsection 3.3), we used the standard set of 26 games [Kaiser al., 2020] to be consistent with the benchmark. Finally, also ran some experiments with the full set of 60 games. The specific games are detailed below.
20 game subset: AirRaid, Asterix, Asteroids, Bowling, Breakout, DemonAttack, Freeway, Gravitar, Jamesbond, MontezumaRevenge, MsPacman, Pong, PrivateEye, Qbert, Seaquest, Spacelnvaders, Venture, WizardOrWor, YersRe, Zaxxon.
26 game subset: Alien, Amidar, Assault, Asterix, BankHeist, BattleZone, Boxing, Breakout, Chop- perCommand, CrazyClimber, DemonAttack, Freeway, Frostbite, Gopher, Hero, Jamesbond, Kanga- roo, Krull, KungFuMaster, MsPacman, Pong. PrivateEye, Qbert, RoadRunner, Seaquest, UpNDown.
60 game set: The 26 games above in addition to: AirRaid, Asteroids, Atlantis, BeamRider, Berzerk, Bowling, Carnival, Centipede, DoubleDunk, ElevatorAAcion,, Enduro, FishingDerby, Gravitar, Ice- Hockey, JourneyEscape, MontezumaRevengge NameTTisGame,, Phoenix, Pitfall, Pooyan, Riverraid, Robotank, Skiing, Solaris, Spacelnvaders, StarGunner, Tennis, TimePilot, Tutankham, Venture, VideoPinball, WizardO/Wor, YarsRevenge, Zaxxon.
C Wall-time versus IQM of human-normalized
QR-DQN (CNNx4)
QR-DQN (Resnet)
2.0
2.0
1.5
1.5
1.0
1.0
Batch Size:
0.5
0.5
(default)
0.0
0.0
0
20
40
0
20
40
Time (hours)
Time (hours)
Figure 14: Measuring wall-time versus IQM of human-normalized scores when varying batch sizes and neural network architectures over 20 games in QR-DQN. Each experiment had 3 independent runs, and the confidence intervals show 95% confidence intervals.
16

---

D Average gradient norm
AirRaid
Asteric
Poin
WizendORWor
Venture
20
1.0
Returs
2-
Batch Size
0-
0.5-
1-
16
0.5-
-10
64
0-
-20
0.0
0.0-
0
50
100
50
100
50
100
S0
100
50
100
AirRaid
Asterix
WizersorWor
50-
Venture
60-
5-
60
40-
4-
Gradient
40-
40-
3-
20
30-
20-
20-
2-
10-
10-
0-
0-
50
100
50
100
100
100
100
Number of Frames (in millions)
Figure 15: Empirical analyses for representative games with varying batch sizes. Top: training returns, Bottom: average gradient norm. Results averaged over seeds, shaded areas represent % confidence intervals.
E
Gradient covariance
Batch Sew
Betch Size
Batch
Betch Sizw 12
Balth
Batch Size 52
Bendor 33
NemsOn 13
IOrEST TO
Www.
Temtio 13
DemonAttck
Batch
Figure 16: Gradient covariance plots for 6 representative games, which highlight the role of the gradient structure with varying batch sizes. We find that smaller batch size significantly improves performance and induces less gradient interference and weaker gradient correlation.
17

---

Second order optimizer effects
Roturos
1.000-
1250
60
4000-
1000-
5000-
2000-
750
20-
14G
186
150
treokout
Demonirtiv
Frenway
Raturns
60-
e0c0-
4020
20-
1000
20
10--
180
50
100
50
1000
30
150
jamwshood
MertenanaBeerrge
Prng
Returns
1000
2000-
Betch Divbor
204
1000-
2000
30
100
30
100
100
30
100
Returns
2000
15000
6000
2000-
10000
4000
2000-
2000
100
10o
166
Menture
Yanahinaage
Zexeen
3000-
75000-
7500
Retumaa
5000
25000
50
1.6
50
100
100
50
1.6
Number of frammes Im million))
Figure 17: Evaluating multiple gradient updates per training step on QR-DQN, training curves for all games. Results averaged over seeds, shaded areas represent 95% confidence intervals.
G Variance of updates.
AirRaid
Asterix
Asterolos
Bowling
Retume
1500
60-
10000-
1000
40-
5000-
20-
50 Broakout
160
50
100
100
50 Gravitar
150
DemonAttack
Freeway
20004
Returns
1000
10-
50
100
50
160
160
50 Pong
1.00
Jamesboond
MontezuumaReveenge
Raturss
1000-
Tarelt Vetuo Scale
204
5.0
2000
0.0-
20
PriswbeEye
Qert
Seaquest
Spacelnvaders
150
3000
Ratums
2000
3000-
10000
2000
1000
100
100
100
Ver
Witar WotrrdoWerr
YarsRewinge
Za0ken
Retums
3000-
75000-
7500
5000
50000
s000.
2500
2500
50
100
50
100
100
50
100
Number frames In millons)
Figure 18: Evaluating the effect adding target noise to QR-DQN, learning curves for all games. Results averaged over 3 seeds, shaded areas represent 95% confidence intervals.
18

---

H
Results on the full ALE suite
We additionally provide complete results for all games using QR-DQN agent in Figure 19.
St
BeamRider
Boxing
B reakout
Camlivll
Centipede
MaoparrComarre
CrazyCliber
DemonAttack
Doubh Dunkk
Elevatoniecion
Endtur
Fish Derinyerrty
Freeway
Batch Size:
Frostbite
Capher
Gravilar
Harr
Lottoc
Jamesbond
Ape
JoumeyEscape
KungFuMaster
Mclerman
Barch Size
(debail)
Pooyan
Obort
Riverrald
RoadRunnee
Robotank
Seaquest
Sking
[delauR]
Solaris
Spaceinvaders
StariQurner
Ternis
TimePlot
Tutanicaam
UpNDown
Venture
Pinbali
WizaroOFWtr
YarsRevenge
Zaxoon
Betch Stee:
Figure 19; Training curves for QR-DQN agent. The results for all games are over 3 independent runs.
19

---

I Varying architectures
40000
Retu
16030
2001
Raturns
2037-
1000
20000
Weturbb
35000
6030
6033
Returrs
103.0-
3590
3900
Demonicctad
Witardo
46021-
110031-
30000
Recurno
12091
Dolcs
50800
5000
-10
300
Kumber framms (tn (lemn))
Figure 20: Evaluating the effect of CNNx4 QR-DQN, learning curves for all games. Results averaged over 3 seeds, shaded areas represent 95% confidence intervals.
30000
200000
110
110
Trewwy
Grevior
Vartec.rashrraao
Returns
1200
20
1/6
25
1.5
16
Obert
Saace Ewineer
40009
2300-
10090-
2.00
20
60
166
06
130
20
330
366
20
Beturne
100009
33000
5000-
Figure 21: Evaluating the effect of Resnet to QR-DQN, learning curves for all games. Results averaged over 3 seeds, shaded areas represent 95% confidence intervals.
20

---

Training Stability
IQN
IQM Human Normalized Score
1.5
1.0
Batch Size
0.5
100M
8
16
0.0
50
100
150
200
Number of Frames (in millions)
Figure 22: Measuring IQM for human-normalized scores when training for 200 million frames using IQN [Dabney et al., 201 Results aggregated over 20 games, where each experiment was run with 3 independent seeds and we report 95% confidence intervals.
AirRald
Adteiix
McPacman
Breakcout
3000e-
Returni
15000
200-
4000-
10000
50.0
2000
50
1do
130
300
103
150
250
5.
1.00
1:0
205
50
1:00
150
200
Freeway
Jamesbond
MontssumaRoveng
Returns
2000
4000
10:0
2000
50
150
150
200
103
150
250
1.00
1:0
200
5.0
3.20
152
200
PrivateEye
15000
Batch Size
43000
Returns
40000
30000-
16
50.0
20000
32
10000
10000-
5000
60
130
100
200
100
350
200
1:0
3:0
205
50
130
180
200
1500
15000
Returns
1000
10000-
2000
500
5000-
1000
20-
50
100
150
200
100
150
200
150
50
150
150
200
Demenntttack
YbrsRevenge
20-
Returns
15000
50000
50000
15000
20-
5000
50
130
200
103
Number frames millions)
Figure 23: Learning curves for individual games, when trained for 200 million frames using IQN [Dabney al., 2018b]. Results aggregated over seeds, reporting 95% confidence intervals.
21

---

5000
150
Returns
4000
3000
20000
20000
2000-
1000-
156
78a
103
150
250
Freeway
Jamesbond
MontensmaRovenge
2000-
Returns
15.0
2000
2000
10:0
1000
1000
150
250
PrivateEyc
Spaceinvoders
6000
60000-
Batch
8000-
6000-
40000
16
4000
2500
Ventiam
Bowling
1500
104
15000-
Returns
10:0
2000
5.0
200
Demonstttick
Pong
YarsRevenge
WizandOWWor
100000
15000
Returns
30-
50000
10000
25000
5000
Number of fromes (in millions)
Figure 24: Learning curves for individual games, when trained for 200 million frames using QR-DQN [Dabney et al., 2018a]. Results aggregated over seeds, reporting 95% confidence intervals.
22


Reviews:
[{"review_id": "M0fQ0GXwXg", "reviewer": "Reviewer_KbLx", "summary": "The authors study how batch size affects RL performance, and argue that a reduced batch size might (quite surprisingly) bring better performance improvement in a number of settings, in particular for QR-DQN, a smaller batch can lead to much better performance (almost doubling the performance). Different batch sizes are tested in Atari environments together with other hyperparameter changes. The authors also point out a benefit of using a smaller batch size is the reduction in wall clock time. The paper focus on empirical study and analysis, and provide a list of interesting findings.", "questions": "Suggestions as discussed in weakness section: \n- spend more effort on related work and explain discrepancies from conclusions in the previous literature. After looking at these previous findings, why do you think your results are different from these findings that mostly say larger batch sizes are better? \n- go through the arguments made in the paper, modify them to be more accurate or bring in better evidence to support them.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**originality**\n- the paper dedicates to bring a better understanding of effect of smaller batch sizes with extensive empirical studies, although there are already past works that study the effect of batch sizes, the results in this paper bring some new findings and observations and can be considered novel contribution. \n\n**quality**\n- overall presentation is good, but some arguments can be improved. \n- extensive experiments and ablations are great\n\n**clarity**\n- Overall paper is clear to read. And the structure is easy to follow. \n\n**significance**\n- the observations presented in the paper can be interesting to the research community and help us better understand the effect of different batch size settings \n- the authors argue that a smaller batch size can bring wall-clock time reduction and given the results they also have potential to bring better performance, which is a result that can be helpful towards better algorithms", "weaknesses": "Major concerns: \n\n**related work**\n- the authors claim \"Surprisingly, to the best of our knowledge there have been no studies exploring the impact of the choice of batch size in deep RL.\" well, there are indeed some works that touch on this issue, for example to list a few: \n  - Accelerated Methods for Deep Reinforcement Learning by Adam Stooke and Pieter Abbeel. \n  - Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control by Islam et al. \n  - Shallow updates for deep reinforcement learning by Levine et al. \n  - An Empirical Model of Large-Batch Training by McCandlish et al. \n- these are older papers, please do some search on google scholar and have a better discussion of related works. Some of these works found larger batch size to be more beneficial. The authors need to spend a bit more effort in looking at related work and try to explain the discrepancy. \n\n\n**Arguments and conclusions made in the paper**\n- Some of the arguments can be improved, for example line 95: \"In Figure 3 we can observe that, in general, reduced batch size results in improved performance.\" I don't think this is true, in Figure 3 there are 8 curves in 4 figures that have lower batch sizes than default, and 4 out of these 8 curves have weaker performance than the default, while the other 4 one can argue they are stronger or slightly stronger than default. One can argue in QR-DQN batch size of 8 is really good, but from this figure alone I don't see how it's a general trend. \n- Line 226 Figure 11 (third column), I am not convinced there is a clear correlation between batch size and gradient norm. The variance is high, and for Asteroids and SpaceInvaders it seems in late stage of training they start to get higher gradient norms. \n- Line 242 authors argue that \" it is possible that the network is better able to adapt to an earlier rank collapse than to a later one.\" I am not sure how this argument is made, figure 11 col 5 shows that Srank does not positively correlate with performance at all. It seems this collapse has no negative effect on the training or even indicates stronger performance, which is again kind of going against what has been argued in previous literature. And the argument that \"Smaller batch sizes seem to result in networks that are both more expressive and with greater plasticity.\" seems to be entirely wrong to me, as figure 11 shows on SpaceInvaders, small batch size of 8 has lowest srank and highest percentage of dormant neurons. \n- Overall, I found a number of the points made in paper to be only supported by very weak evidence and they are not convincing. The authors might want to either modify the arguments into more accurate ones, or try to find better evidence to support the arguments. If the evidence is weak, the conclusion might not hold at all and it could be really coming from randomness or because of excessive fine-tuning on a particular algorithm on some particular environments. \n\nThe paper is currently lacking in these 2 aspects, but can be a good paper if these concerns are properly addressed. \n\nMinor concerns:\n- **originality** the novelty of the work is reduced by the fact that it is focused on studying existing methods, but mitigated by the novel empirical results, ablations and analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for a careful read of our submission, and concrete suggestions for improving it. We address each point separately below, referencing the PDF attached to the general rebuttal at the top. We hope our responses, and the amendments we will make to our paper based on the points raised, are sufficient to address your concerns. Please let us know if you feel something was not properly addressed.\n\n## Related work\nWe thank the reviewer for pointing out these papers, which had escaped our attention (with the exception of S&A, which we were already citing). They are quite relevant to our work, so we will expand our discussion  to include them accordingly (specifics below). We will also soften the wording regarding no prior studies on batch size in RL. \n\nSome discussion points on the works suggested:\n* S&A focus on distributed training, which have very different learning dynamics than those considered in our work. Indeed, Stooke & Abeel emphasize _increasing_ batch size to obtain better performance. \n* Islam et al. focus on two policy-gradient methods (DDPG and TRPO), which again have quite different learning dynamics than the value-based methods we consider. Their findings seem to suggest that _larger_ batch sizes yield better performance, which contrasts with our findings.\nTo investigate further, we ran some experiments on MPO [1] (an off-policy value-based method that still shares some of the benefits from on-policy policy-gradient methods like TRPO, see Figure 3 in rebuttal PDF). Our results suggest there is also a tendency towards improved performance with smaller batch sizes; we suspect this advantage is due to it being value-based, but more investigation is necessary. The evident differences on the effect of batch size on value-based versus policy-gradient methods are certainly worth discussing, and we will add a discussion to this effect, along with our new results.\n*  Levine et al. focus on \"shallow\" RL (e.g. using linear approximators instead of deep networks), and find that larger batch sizes yield better performance.\n*  McCandlish et al. seems to focus on very large batch sizes; the smallest batch size considered is 64, and they go as high as the millions for Dota 5v5. For their Atari results (most comparable to ours), the authors focus on A2C (another asynchronous policy gradient method) and find that the best batch sizes are 100-1000 early in training, and 400-8000 later in training, which are substantially larger than what we considered.\n\nBased on the reviewer suggestion, we will also be referencing and comparing our work to [2], which explores the connection between batch size and importance sampling, and [3,4] which discuss the impact of gradient norms on training (which our analyses show are connected to batch size choice).\n\n## On 4/8 curves having weaker performance than default\nWhile true, it is worth noting that DQN in this figure does not have multi-step returns; as we discuss and analyze in section 4, we don’t observe improved performance with smaller batch sizes in DQN without multi-step returns. Indeed, as can be observed in Figure 10, both smaller-than-default batch sizes yield improved performance for DQN when multi-step updates are used. We will clarify this point in our discussion.\n\n## Correlation between batch size & gradient norm\nFor computational reasons, we focused on three games for our submission. However, we ran this analysis on five extra games (see Figure 1 in rebuttal PDF).  These show a stronger correlation between batch size and gradient norm. Nonetheless, we will modify our discussion to soften the claim, given your concerns, and include the extra figures in the appendix as further evidence of the connection.\n\n## Connection between srank, plasticity, batch size, & performance\nThe point we were making was based on the observation that in all three games we observed improved performance with reduced batch sizes, _and_ in all three games we observed an early collapse in srank. This is of course a correlation, and not necessarily a causal relationship, but we felt it was worth remarking on.\n\nThe comment about plasticity was with regards to the discussion in the text: “although the relationship with batch size is not as clear as with some of the other metrics, smaller batch sizes appear to have a much milder increase in their frequency”. As Sokar et al. [5] showed, the level of dormant neurons increases (mostly) monotonically throughout training (as the black line for batch_size=32 shows); with smaller batch sizes we have a milder increase throughout training, although we do see them start with a higher fraction of dormant neurons.\n\nIn both cases we can soften our claims and add the points made above to the discussion.\n\n## Why are our findings different than those saying larger batch sizes are better?\nAs discussed above, there appears to be a correlation between the use of distributed and/or policy gradient methods and improved performance due to larger batch sizes. Our work explores non-distributed value-based methods. As mentioned in the discussion above, we will highlight this point further in our discussion.\n\n# References\n*  [1] A. Abdolmaleki, J.T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maximum a posteriori policy optimisation. ICLR 2018\n*  [2] T. Lahire, M. Geist, and E. Rachelson. \"Large batch experience replay.\" arXiv preprint arXiv:2110.01528 (2021)\n*  [3] P. Mi, L. Shen, T. Ren, Y. Zhou, X. Sun, R. Ji, and D. Tao. \"Make sharpness-aware minimization stronger: A sparsified perturbation approach.\" NeurIPS (2022)\n*  [4] X. Zhang, R. Xu, H. Yu, H. Zou, and P. Cui. \"Gradient Norm Regularizer Seeks Flat Minima and Improves Generalization.\" (2022)\n*  [5] G. Sokar, R. Agarwal, P.S. Castro, and U. Evci. The dormant neuron phenomenon in deep reinforcement learning. In ICML, 2023"}, {"review_id": "0rlSq3ONBa", "reviewer": "Reviewer_oeF6", "summary": "The work investigates the influence of replay batch size in experience replay for online reinforcement learning. The key finding is that reducing the batch size can be more beneficial, which contradicts common knowledge about regular deep learning.", "questions": "-", "limitations": "Not applicable", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper expands the analysis of an underinvestigated observation that can potentially change the default settings for experience replay and reduce the computational cost of further experiments. The authors provide new insights into the computational impact of batch size in experience replay and analyze network optimization dynamics. One of the strengths of the paper is its extensive experiments conducted with different settings and architectures.", "weaknesses": "Experience replay is also used in continual learning. It is surprising that the authors missed the paper that already drew the same conclusion [1], but kept it underinvestigated. Nonetheless, it should be mentioned in the related work section.\n\n[1] Wołczyk, M., & Krutsylo, A. (2021). Remember More by Recalling Less: Investigating the Role of Batch Size in Continual Learning with Experience Replay (Student Abstract). AAAI Conference on Artificial Intelligence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful review of our submission, and for pointing us to the paper on experience replay in continual learning, it is indeed quite related.\n\nWołczyk, M., & Krutsylo, A. (2021) investigate the dynamics of experience replay in online continual learning, and focus on the effect of batch size choice when sampling from a replay buffer. They find that smaller batches are better at preventing forgetting than using larger batches, contrary to the intuitive assumption that it is better to recall more samples from the past to avoid forgetting. Additionally, the authors show that this phenomenon does not disappear under learning rate tuning. Their settings are similar to those used to generate Figure 3 in Sokar et al., 2023 [1], and suggest that target non-stationarity (e.g. bootstrapping) may have a role to play in explaining the small batch size effect we are observing. We will add this to our discussion.\n\n# References\n[1] Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In ICML, 2023."}, {"review_id": "DMYHffAsIN", "reviewer": "Reviewer_scQb", "summary": "This work studies the effect of reducing the batch size in value-based deep RL algorithms. Surprisingly, the authors find that smaller batch sizes generally improve learning performance and speed up training in terms of wall-clock time. Towards understand this \"small batch effect\", they empirically investigate how batch size relates to e.g. multistep learning, variance of gradient updates, network capacity, network plasticity, etc. ", "questions": "1. Fig. 1: Can the authors provide any intuition on why 22/60 tasks performed worse with a smaller batch size with QR-DQN? \n2. In Fig. 8, a larger batch size improves performance only for learning rate = 5e-06. Can the authors provide explanation?\n", "limitations": "See Weaknesses", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Overall, the writing is extremely clear and well-organized. While reading, I found myself asking questions that the authors then answered in later section of the paper (e.g. Lines 125-129 prompted questions on how batch size relates to plasticity and network capacity)\n2. The empirical evaluation is thorough; the authors consider a wide range of Atari tasks and investigate how batch size relates to a variety of learning factors.\n3. The relationships uncovered in this work relate to many RL research areas (e.g. exploration, continual learning), and I believe they will spur interesting future research.", "weaknesses": "1. The study only considers visual tasks with discrete actions. Does a small batch size improve data efficiency if you use the non-visual RAM observations in a few representative tasks? Do the same trends observed in Fig. 11 still hold? Since the paper is scoped to focused on value-based algorithms, I believe it is sufficient to state the discrete action limitation in the conclusion.\n2. Since a smaller batch size seems to come with a variety of benefits (e.g. smaller gradient norms), it isn't clear to me if the observed benefits in Fig. 6 are due to improved exploration via higher variance gradient updates. Since it is likely not feasible to isolate exploration, can the authors instead clarify how these figures show improved exploration?  \n\n**Minor comments:**\n2. Lines 59-60: \"r\" -> \"r_t\"\n3. I believe line 265 should say \"**decreasing** the batch size should increase variance\"\n3. Lines 266-268: \"That it's effect...\" This sentence is difficult to read and would benefit from rephrasing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful review of our paper, and suggestions for improvement. We address your concerns and questions below, referencing the PDF attached to the general response above. We will also correct all the minor issues pointed out in our submission.\n\n## The study only considers visual tasks with discrete actions.\nAlthough our work focused on value-based algorithms, we agree it is worth investigating whether the effect is present in non-visual and continuous control tasks. We ran some experiments with DQN and Rainbow on 2 classic control environments (where inputs are state vectors) (Figure 2 in rebuttal PDF), as well as MPO [1] on DM-control environments (Figure 3 in rebuttal PDF). In both cases, we see a general trend towards improved performance when using smaller batches. In the classic control case we also see confirmation of the findings presented in section 4.1 of our submission: the small-batch effect is more pronounced with multi-step learning.\n\n## Where is improved exploration coming from?\nMany methods have been proposed to address the exploitation-exploration dilemma, and some techniques emphasize exploration by adding noise directly to the parameter space of agents Fortunato et al., 2018 [2], Hao et al., 2023 [3], Plappert et al., 2017 [4], Gupta et al., 2018 [5], which inherently adds variance to the learning process. Noise perturbation is another approach that has been taken to induce exploration [6].\n\nLike these works, our analyses show that increasing variance by reducing the batch size may result in similar beneficial exploratory effects, as the mentioned works suggest. As the reviewer rightly points out, it is difficult to isolate the direct impact on exploration; however, the improved performance observed on all the hard exploration games in Atari, as well as in MountainCar (which is considered to be a hard exploration classic control environment, shown in Figure 2 in the rebuttal PDF) suggests that improved exploration may be an advantageous consequence of the variance induced by reduced batch size. As we discussed in section 5.1, we believe further work exploring the impact of variance injection in deep RL algorithms is necessary, and will add these points to our discussion.\n\n## Why 22/60 of the games performed worse?\nIt is very rare for an agent/algorithm to outperform the baselines on all games considered. For example, C51 [7] improved on 44/57, Rainbow [8] on 26 out of 57 games, QR-DQN [9] on 52/57, and Munchausen-IQN [10] on 40/60 games. Reporting aggregate performance, as is the norm with this benchmark, does mean we (as a community) gloss over some of these per-game differences. We agree this is an important problem the community should pay more attention to and will add a note to this effect in our discussion.\n\n## Why larger batch size only improves for smaller learning rate?\nThe default learning rate used (5e-05) was one optimized by prior work for a batch size of 32. It has been previously shown that one should reduce learning rates when increasing batch sizes [11], which is consistent with our findings (e.g. reducing the learning rate can be beneficial when increasing the batch size).\n\n# References\n*  [1] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Ried-miller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations,2018.\n*  [2] Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I.,Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin,O., Blundell, C., and Legg, S.  Noisy networks for exploration.  InProceedings of the International Confer-ence on Representation Learning (ICLR 2018), Vancou-ver (Canada), 2018.\n*  [3] Jianye Hao, Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, and Zhen Wang.Exploration in deep reinforcement learning:  From single-agent to multiagent domain.IEEE Transactions on Neural Networks and Learning Systems, pages 1–21, 2023.\n*  [4] Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen,R. Y., Chen, X., Asfour, T., Abbeel, P., and Andrychow-icz, M.   Parameter space noise for exploration. arXivpreprint arXiv:1706.01905, 2017\n*  [5] Abhishek  Gupta,  Russell  Mendonca,  YuXuan  Liu,  Pieter  Abbeel,  and  Sergey  Levine.   Meta-reinforcement learning of structured exploration strategies, 2018\n*  [6] Eberhard, O., Hollenstein, J., Pinneri, C., and Martius, G.Pink noise is all you need:  Colored noise exploration in deep reinforcement learning. InDeep ReinforcementLearning Workshop NeurIPS 2022, 2022\n*  [7] Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 449–458, 2017.\n*  [8] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.\n*  [9] W. Dabney, M. Rowland, Marc G. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In AAAI, 2018.\n*  [10] Nino Vieillard, Olivier Pietquin, Matthieu Geist. Munchausen Reinforcement Learning. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020).\n*  [11] D.Randall Wilson and Tony R. Martinez. The general inefficiency of batch training for gradient descent learning. Neural Networks, Volume 16, Issue 10, December 2003, Pages 1429-1451."}, {"review_id": "QNzxMTD5e9", "reviewer": "Reviewer_vSto", "summary": "The paper demonstrates the advantages of employing smaller batch sizes in value-based RL algorithms. It reveals that utilizing a smaller batch size can moderately or significantly enhance performance across several value-based RL algorithms, with the exception of DQN, where there are no improvements. Nonetheless, the paper reveals that incorporating a deeper network or employing n-step returns can restore the benefits of smaller batch sizes in the case of DQN.\n\nTo examine the underlying reasons behind the benefits of smaller batch sizes, the paper conducts a comprehensive set of experiments. These experiments investigate various factors, such as increased update variance, reduced representation norm, and improved network expressivity, shedding light on how smaller batch sizes can help performance.", "questions": "I suggest explaining the intuition on how the n-step return interplays with the batch size. My guess is the n-step return has a larger variance than a single-step return, and a smaller batch size can somehow further magnify this variance.", "limitations": "I don't see any clear limitations in this study.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper presents a comprehensive and robust series of experiments, providing evidence for the effectiveness of smaller batch sizes in various value-based RL algorithms.\n- The analysis conducted in the paper offers clear insights into the reasons why smaller batch sizes yield improvements in performance.\n- The paper also delves into an in-depth investigation to understand why smaller batch sizes do not yield benefits in the case of DQN, providing valuable insights and explanations for this observed phenomenon.", "weaknesses": "While the result may not be technically novel, I agree that a comprehensive study on the topic of the benefits of smaller batch sizes is still highly valuable to the research community. Although some researchers are aware that smaller batch sizes can be beneficial, as seen in certain implementations of existing algorithms, such as IMPALA (e.g., https://github.com/facebookresearch/torchbeast uses a default batch size of 8), there is a lack of thorough analysis on this matter. Thus, the paper fills this gap and provides valuable insights and understanding of the issue. I appreciate the significance and value that the paper brings to the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and appreciate the positive evaluation.\n\nIndeed, our intuition matches yours: the increase in variance from both fronts (smaller batch size and multi-step updates) seems to have a positive effect on performance. It is possible that they are adding different types of variance, which have a complementary effect on learning. This finding is further confirmed by Figure 2 in the PDF included with the general rebuttal at the top, which evaluates on classic control environments and shows that the small batch effect is also more pronounced when combined with multi-step updates."}]

Author rebuttal:
Dear reviewers and (S)ACs, we are attaching a PDF with three figures that we reference in each of our reviewer-specific rebuttals. The figures are:

**Figure 1:** Gradient variance analysis (with corresponding reward curves) for five extra Atari 2600 games, that help strengthen the claim of correlation between batch size, gradient variance, and agent performance. This is mostly in response to reviewer KbLx.

**Figure 2:** Effect of batch size on two classic control environments, which are state-based (as opposed to pixel-based, like Atari 2600 games). This is mostly in response to reviewer scQb.

**Figure 3:** Effect of batch size on MPO evaluated on DM-control. This is mostly in response to reviewers KbLx and scQb.

Meta-review: This paper considers deep value-based RL in the small-batch regime, performing a controlled and in-depth study of observed benefits upon considering batch sizes smaller than those usually taken for granted in the literature, finding that variance is beneficial. The reviewers agree that the empirical study is quite high-quality, and that the phenomenon is interesting.

One reviewer raised concerns about the relationship with prior literature, which was amply addressed in the response.
Decision: Accept (poster)