SlowGuess/NeurIPS_2023_OCR huggingface dataset:
varying scales) to the learning target  $Q_{\bar{\theta}}$  (see section 2 for definition). As Figure 12 demonstrates, simply adding noise to the target does provide benefits, albeit with some variation across games.
Figure 12: Adding noise of varying scales to the learning target with the default batch size of 32. Left: Performance of QR-DQN on three games with different target noise scale values. Right: Results aggregated IQM of human-normalized scores over 20 games for QR-DQN.
Gradient and representation norms Keskar et al. [2017] and Zhao et al. [2022] both argue that smaller gradient norms can lead to improved generalization and performance, in part due to less "sharp" optimization landscapes. In Figure 11 (third column) we can see that batch size is, in fact, correlated with gradient norms, which may be an important factor in the improved performance. In Appendix D, we conducted experiments on a different subset of games, and observed a consistent trend: better performance is achieved with smaller batch sizes and gradient norms.
There have been a number of recent works suggesting RL representations, taken to be the output of the convolutional layers in our networks $^6$ , yield better agent performance when their norms are smaller. Gogianu et al. [2021] demonstrated that normalizing representations yields improved agent performance as a result of a change to optimization dynamics; Kumar et al. [2021b] further observed that smaller representation norms can help mitigate feature co-adaptation, which can degrade agent performance in the offline setting. As Figure 11 (fourth column) shows, the norms of the representations are correlated with batch size, which aligns well with the works just mentioned.
Effect on network expressivity and plasticity Kumar et al. [2021a] introduced the notion of the effective rank of the representation  $s r a n k_{\delta}(\phi)^{7}$ , and argued that it is correlated with a network's expres
This is a common interpretation used recently, for example, by Castro et al. [2021], Gogianu et al. [2021], and Farebrother et al. [2023]
${}^{7}\delta$  is a threshold parameter. We used the same value of 0.01 as used by Kumar et al. [2021a].
sivity: a reduction in effective rank results in an implicit under-parameterization. The authors provide evidence that bootstrapping is the likeliest cause for effective rank collapse (and reduced performance).
Interestingly, in Figure 11 (fifth column) we see that with smaller batch sizes srank collapse occurs earlier in training than with larger batch sizes. Given that there is mounting evidence that deep RL networks tend to overfit during training [Dabney et al., 2021, Nikishin et al., 2022, Sokar et al., 2023], it is possible that the network is better able to adapt to an earlier rank collapse than to a later one.
To further investigate the effects on network expressivity, we measured the fraction of dormant neurons (neurons with near-zero activations).
Figure 13: Gradient covariance matrices for Asteroids (left) and SpaceInvaders (right). In environments where smaller batch size significantly improves performance, it also induces weaker gradient correlation<sup>8</sup> and less gradient interference.
Sokar et al. [2023] demonstrated that deep RL agents suffer from an increase in the number of dormant neurons in their network; further, the higher the level of dormant neurons, the worse the performance. In Figure 11 (rightmost column) we can see that, although the relationship with batch size is not as clear as with some of the other metrics, smaller batch sizes appear to have a much milder increase in their frequency. Further, there does appear to be a close relationship with the measured  $srank$  findings above. Lyle et al. [2023] evaluated the covariance structure of the gradients to revisit the network's loss landscape, and argue that weaker gradient correlation and less gradient interference improve performance. We observe similar results in the gradient covariance heat maps shown in Figure 13 and Figure 16, where gradients appear to be largely colinear when using larger batch size values.
- Reduced batch sizes result in increased variance of losses and gradients. This increased variance can have a beneficial effect during training.
- Smaller batch sizes result in smaller gradient and representation norms, which tend to result in improved performance.
- Smaller batch sizes seem to result in networks that are both more expressive and with greater plasticity.
There is a considerable amount of literature on understanding the effect of batch size in supervised learning settings. Keskar et al. [2016] presented quantitative experiments that support the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions, and as has been shown in the optimization community, sharp minima tends to lead to poorer generalization. Masters and Luschi [2018] support the previous finding, presenting an empirical study of stochastic gradient descent's performance, and reviewing the underlying theoretical assumptions surrounding smaller batches. They conclude that using smaller batch sizes achieves the best training stability and generalization performance. Additionally, Golmant et al. [2018] reported that across a wide range of network architectures and problem domains, increasing the batch size yields no decrease in wall-clock time to convergence for either train or test loss.
Although batch size is central to deep reinforcement learning algorithms, it has not been extensively studied. One of the few results in this space is the work by Stooke and Abbeel [2018], where they argued that larger batch sizes can lead to improved performance when training in distributed settings. Our work finds the opposite effect: smaller batch sizes tends to improve performance; this suggests that empirical findings may not directly carry over between single-agent and distributed training scenarios. Islam et al. [2017] and Hilton et al. [2022] have investigated the role of batch size in on-policy algorithms. The latter demonstrates how to make these algorithms batch size-invariant, aiming to sustain training efficiency at small batch sizes.
8 Dark red color refers to high negative correlation, and dark blue one high positive correlation.
Lahire et al. [2021] cast the replay buffer sampling problem as an importance sampling one, allowing it to perform well when using large batch. Fedus et al. [2020] presented a systematic and extensive analysis of experience replay in Q-learning methods, focusing on two fundamental properties: the replay capacity and the ratio of learning updates to experience collected (e.g. the replay ratio). Although their findings are complementary to ours, further investigation into the interplay of batch size and replay ratio is an interesting avenue for future work. Finally, there have been a number of recent works investigating network plasticity [Schwarzer et al., 2023, D'Oro et al., 2023, Sokar et al., 2023, Nikishin et al., 2022], but all have kept the batch size fixed.
Wołczyk and Krutsylo [2021] investigate the dynamics of experience replay in online continual learning, and focus on the effect of batch size choice when sampling from a replay buffer. They find that smaller batches are better at preventing forgetting than using larger batches, contrary to the intuitive assumption that it is better to recall more samples from the past to avoid forgetting. Additionally, the authors show that this phenomenon does not disappear under learning rate tuning. Their settings are similar to those used to generate Figure 3 in [Sokar et al., 2023], and suggest that target non-stationarity (e.g. bootstrapping) may have a role to play in explaining the small batch size effect we are observing.
In online deep RL, the amount of data sampled during each training step is crucial to an agent's learning effectiveness. Common intuition would lead one to believe that larger batches yield better estimates of the data distribution and yield computational savings due to data parallelism on GPUs. Our findings here suggest the opposite: the batch size parameter generally alters the agent's learning curves in surprising ways, and reducing the batch size below its standard value is often beneficial.
From a practical perspective, our experimental results make it clear that the effect of batch size on performance is substantially more complex than in supervised learning. Beyond the obvious performance and wall-time gains we observe, changing the batch size appears to have knock-on effects on exploration as well as asymptotic behaviour. Figure 8 hints at a complex relationship between learning rate and batch size, suggesting the potential usefulness of "scaling laws" for adjusting these parameters appropriately.
Conversely, our results also highlight a number of theoretically-unexplained effects in deep reinforcement learning. For example, one would naturally expect that decreasing the batch size should increase variance, and eventually affect prediction accuracy. That its effect on performance, both transient and asymptotic, should so critically depend on the degree to which bootstrapping occurs (as in  $n$ -step returns; Figure 10), suggests that gradient-based temporal-difference learning algorithms need a fundamentally different analysis from supervised learning methods.
Future Work Our focus in this paper has been on value-based online methods. This raises the question of whether our findings carry over to actor-critic methods, and different training scenarios such as offline RL [Levine et al., 2020] and distributed training [Stooke and Abbeel, 2018]. While similar findings are likely for actor-critic methods, the dynamics are sufficiently different in offline RL and in distributed training that it would likely require a different investigative and analytical approach. It is also an interesting direction to explore adaptive schemes that dynamically varies the batch size during training. Our experiments used a constant batch size, so further research is needed to determine whether it is advantageous to reduce the batch size over time in practice, as well as how quickly it should be reduced.
Our work has broader implications than just the choice of the batch size hyper-parameter. For instance, our findings on the impact of variance on performance suggest a promising avenue for new algorithmic innovations via the explicit injection of variance. Most exploration algorithms are designed for tabular settings and then adapted for deep networks; our results in section 3.5 suggest there may be opportunities for exploratory algorithms designed specifically for use with neural networks. We hope our analyses can prove useful for further advances in the development and understanding of deep networks for reinforcement learning.
Acknowledgements. Many thanks to Georg Ostrovski and Gopeshh Subbaraj for their feedback on an earlier draft of this paper. We also acknowledge Max Schwarzer, Adrien Ali Taiga, Rishabh Agarwal and Jesse Farebrother for useful discussions, as well as the rest of the DeepMind Montreal team for their feedback on this work. The authors would also like to thank the anonymous reviewers for useful feedback on this paper. Last but not least, we would also like to thank the Python community [Van Rossum and Drake Jr, 1995, Oliphant, 2007] for developing tools that enabled this work, including NumPy [Harris et al., 2020], matplotlib [Hunter, 2007] and JAX [Bradbury et al., 2018].
Broader impact Although the work presented here is mostly of an academic nature, it aids in the development of more capable autonomous agents. While our contributions do not directly contribute to any negative societal impacts, we urge the community to consider these when building on our research
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Beyond tabula rasa: Reincarnating reinforcement learning. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
Brian R Bartoldson, Bhavya Kailkhura, and Davis Blalock. Compute-efficient deep learning: Algorithmic trends and opportunities. Journal of Machine Learning Research, 24:1-77, 2023.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013. doi: 10.1613/jair.3912.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_files/paper/2016/file/afda332245e2af431fb7b672a68b659d-Paper.pdf.
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 449-458, 2017.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. Jax: composable transformations of python+ numpy programs. 2018.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. *Dopamine: A Research Framework for Deep Reinforcement Learning*. 2018. URL http://arxiv.org/abs/1812.06110.
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. MICo: Learning improved representations via sampling-based state similarity for Markov decision processes. In Advances in Neural Information Processing Systems, 2021.
Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1373-1383. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/ceron21a.html.
Lili Chen, Kimin Lee, Aravind Srinivas, and Pieter Abbeel. Improving computational efficiency in visual reinforcement learning via stored embeddings. Advances in Neural Information Processing Systems, 34:26779-26791, 2021.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 26982-26992. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/zhao22i.html.
A Code availability
Our experiments were built on open source code, mostly from the Dopamine repository. The root directory for these is https://github.com/google/dopamine/tree/master/dopamine/, and we specify the subdirectories below (with clickable links):
- DQN, Rainbow, QR-DQN and IQN agents from /jax/agents/
- Atari-100k agents from /labs/atari-100k/
- Batch size from /jax/agents/quantile/configs/quantile.gin (line 36)
- Exploration  $\epsilon = 0$  from /jax/agents/quantile/configs/quantile.gin (line 16)
- Resnet from /labs/offline-r1/jax/networks.py (line 108)
- Dormant neurons metric from /labs/redo/
For the srank metric experiments we used code from: https://github.com/google-research/google-research/blob/master/ generalization Representations_r1_aistats22/coherence/coherence_compute.py
B Atari 2600 games used
Most of our experiments were run with 20 games from the ALE suite [Bellemare et al., 2013], as suggested by Fedus et al. [2020]. However, for the Atari 100k agents (subsection 3.3), we used the standard set of 26 games [Kaiser et al., 2020] to be consistent with the benchmark. Finally, we also ran some experiments with the full set of 60 games. The specific games are detailed below.
20 game subset: AirRaid, Asterix, Asteroids, Bowling, Breakout, DemonAttack, Freeway, Gravitar, Jamesbond, MontezumaRevenge, MsPacman, Pong, PrivateEye, Qbert, Seaquest, SpaceInvaders, Venture, WizardOfWor, YarsRevenge, Zaxxon.
26 game subset: Alien, Amidar, Assault, Asterix, BankHeist, BattleZone, Boxing, Breakout, ChopperCommand, CrazyClimber, DemonAttack, Freeway, Frostbite, Gopher, Hero, Jamesbond, Kangaroo, Krull, KungFuMaster, MsPacman, Pong, PrivateEye, Qbert, RoadRunner, Seaquest, UpNDown.
60 game set: The 26 games above in addition to: AirRaid, Asteroids, Atlantis, BeamRider, Berzerk, Bowling, Carnival, Centipede, DoubleDunk, ElevatorAction, Enduro, FishingDerby, Gravitar, Ice-Hockey, JourneyEscape, MontezumaRevenge, NameThisGame, Phoenix, Pitfall, Pooyan, Riverraid, Robotank, Skiing, Solaris, SpaceInvaders, StarGunner, Tennis, TimePilot, Tutankham, Venture, VideoPinball, WizardOfWor, YarsRevenge, Zaxxon.
C Wall-time versus IQM of human-normalized
Figure 14: Measuring wall-time versus IQM of human-normalized scores when varying batch sizes and neural network architectures over 20 games in QR-DQN. Each experiment had 3 independent runs, and the confidence intervals show  $95\%$  confidence intervals.
D Average gradient norm
Figure 15: Empirical analyses for 5 representative games with varying batch sizes. Top: training returns, Bottom: average gradient norm. Results averaged over 3 seeds, shaded areas represent  $95\%$  confidence intervals.
E Gradient covariance
Figure 16: Gradient covariance plots for 6 representative games, which highlight the role of the gradient structure with varying batch sizes. We find that smaller batch size significantly improves performance and induces less gradient interference and weaker gradient correlation.
F Second order optimizer effects
Figure 17: Evaluating multiple gradient updates per training step on QR-DQN, training curves for all games. Results averaged over 3 seeds, shaded areas represent  $95\%$  confidence intervals.
G Variance of updates.
Figure 18: Evaluating the effect of adding target noise to QR-DQN, learning curves for all games. Results averaged over 3 seeds, shaded areas represent  $95\%$  confidence intervals.
H Results on the full ALE suite
We additionally provide complete results for all games using QR-DQN agent in Figure 19.
Figure 19: Training curves for QR-DQN agent. The results for all games are over 3 independent runs.
I Varying architectures
Figure 20: Evaluating the effect of  $\mathrm{CNNx4}$  to QR-DQN, learning curves for all games. Results averaged over 3 seeds, shaded areas represent  $95\%$  confidence intervals.
Figure 21: Evaluating the effect of Resnet to QR-DQN, learning curves for all games. Results averaged over 3 seeds, shaded areas represent  $95\%$  confidence intervals.
J Training Stability
Figure 22: Measuring IQM for human-normalized scores when training for 200 million frames using IQN [Dabney et al., 2018b]. Results aggregated over 20 games, where each experiment was run with 3 independent seeds and we report  $95\%$  confidence intervals.
Figure 23: Learning curves for individual games, when trained for 200 million frames using IQN [Dabney et al., 2018b]. Results aggregated over 3 seeds, reporting  $95\%$  confidence intervals.
Figure 24: Learning curves for individual games, when trained for 200 million frames using QR-DQN [Dabney et al., 2018a]. Results aggregated over 3 seeds, reporting  $95\%$  confidence intervals.
Number of frames (in millions)