Variance Double-Down: The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning
Authors: Johan Samir Obando Ceron
Marc G Bellemare
Pablo Samuel Castro
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement Learning
Deep Reinforcement Learning
Value based
Batch Size
Multi step learning

Abstract: State of the art results in reinforcement learning suggest that multi-step learning is necessary. However, the increased variance that comes with it makes it difficult to increase the update horizon beyond relatively small numbers. In this paper, we report the counterintuitive finding that decreasing the batch size substantially improves performance across a large swath of deep RL agents. It is well-known that gradient variance decreases with increasing batch sizes, so obtaining improved performance by increasing variance on two fronts is a rather surprising finding. We conduct a broad set of experiments to better understand this variance double-down phenomenon.

Under review as a conference paper at ICLR 2023
VARIANCE DOUBLE-DOWN: THE SMALL BATCH SIZE ANOMALY IN MULTISTEP DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
In deep reinforcement learning, multi-step learning is almost unavoidable achieve state-of-the performance. However, the increased variance that multi- step learning brings makes it difficult to increase the update horizon beyond rel- atively small numbers. In this paper, we report the counterintuitive finding that decreasing the batch size parameter improves the performance of many standard deep RL agents that use multi-step learning. It is well-known that gradient vari- ance decreases with increasing batch sizes, obtaining improved performance by increasing variance on two fronts is a rather surprising finding. We conduct a broad set of experiments better understand what we call the variance double- down phenomenon.
INTRODUCTION
Deep reinforcement learning (DRL), which combines traditional reinforcement learning (RL) tech- niques with neural networks, has had a number of recent successes, including achieving superhuman performance on challenging games (Mnih et al., 2015; Schrittwieser et al., 2020; Perolat et al., 2022), overcoming difficult robotics challenges (Andrychowiez et al., 2020; Smith et al., 2022), and being successfully applied to large-scale real-world tasks (Bellemare et al., 2020; Degrave et al., 2022). Yet successful application of DRL to new problems remains a challenge, in large part due to the difficulty in understanding how neural network training is affected by the vast number of hyper- parameters involved. Despite a number of recent works developing a greater understanding of the dynamics of training neural networks for reinforcement learning (Ceron & Castro, 2021; Araújo et al., 2021; Nikishin et al., 2022; Ostrovski et al., 2021; Schaul et al., 2022), the relationship be- tween particular hyper-paramettr configurations and performance given environment remains hard to predict.
One generally held desire in training neural networks is reduce the variance of gradient updates, so as to avoid unstable and unreliable learning. For example, in the reinforcement learning literature there has been growing trend to use multi-step (or n-step) learning (Hessel et al., 2018; Schwarzer et al., 2020; Kapturowski et al., 2018; Agarwal et al., 2022) for improved performance. Despite their demonstrated advantage, researchers have been limited small values of to avoid performance collapse, in part due to the increased variance arising from larger n.
The supervised learning literature suggests that an effective mechanism for mitigating variance is through the choice of batch size: Shallue et al. (2019) empirically demonstrate that larger batch sizes result in reduced variance and increased performance. In this paper, we report the counter- intuitive finding that reducing the batch size can help avoid performance collapse with larger n-step updates. This is effectively doubling down on increased variance for improved performance. We showcase this anomaly in a broad set of training regimens and value-based RL agents, and conduct an empirical analysis to develop a better understanding of its causes. Additionally, we demonstrate that reduced batch sizes also results in reduced overall computation time during training. In Ap- pendix A we provide background on deep reinforcement learning, including a description of n-step updates and batch sizes.

---

Under review as a conference paper at ICLR 2023
DQN
Rainbow
IQM Human Normalized Score
Batch
IQM Human Normalized Score
1.4
1.2
1.2
32 (default)
1.0
64
3.4
0.2
0.2
0.0
2.0
40
B0
100
40
100
Number of Frames (in millions)
Number of Frames (in millions)
QR-DQN
IQN
IQM Human Norm alizzdd Score
IQM Human Normalized Score
1.4
1.0
1.0
0.8
0.8
0.4
3.2
0.0
8.0
40
60
89
100
49 60 B5
100
Number of Frames (in millions)
Number of Frames (in millions)
Figure Varying batch sizes DQN, Rainbow, QR-DQN, and IQN.
CASE STUDY: THE ARCADE LEARNING ENVIRONMENT
Advances deep reinforcement learning (DRL) often build on prior algorithms, network archi- tectures, and hyper-parameter selections. Given the large number of options, new work typically re-tunes only those components necessary for the new methods being considered. Thus, we have accumulated set of, mostly static, parameters upon which new ideas are tested (this may be form of the "social dynamics of research" hypothesized by Schaul et al. (2022)). One the static parameters for training single-agent value-based agents has been the choice of batch size.
Since the introduction of DQN by Mnih et (2015), single-agent training on the Arcade Learning Environment (ALE, Bellemare al., 2013) has used batch size of 32, where this value care- fully tuned by the authors for performance. Since then, this value has rarely been changed, save for distributed agent training (Kapturowski al., 2018; Espeholt et al., 2018). 8). If one takes the general advice from the supervised learning literature, we should aiming to increase the batch size so reduce variance and improve performance (Shallue al., 2019). We focus the effect of changing the batch size, while keeping all else equal.
2.1 EXPERIMENTAL SETUP
For this case study, we use JAX implementations of agents provided by the Dopamine library (Castro al., 2018) and applied to game-playing in the ALE (Bellemare et al., 2013). For computational reasons, we evaluate our agents 20 games chosen by Fedus et al. (2020) in their analysis of replay ratios; these were picked offer a diversity of difficulty and dynamics. Similarly, we run each learning trial for 100 million frames (as opposed to the standard 200 million). In exploratory experiments, we determined that for our purposes there are unsubstantial differences at 100M and 200M frames. The four agents we consider are: DQN (Mnih et al., 2015), Rainbow (Hessel et al.,
Dopamine uses sticky actions by default (Machado al., 2018).
2

---

Under review as a conference paper at ICLR 2023
2018);, QR-DQN (Dabney et al., 2018a), and IQN (Dabney al., 2018b). These all use the default values given Dopamine. All experiments were run with 3 independent seeds on NVIDIA Tesla PI00 GPUs.
For evaluation, we follow the robust evaluation guidelines of Agarwal et al. (2021). Specifically, we report the numan-normalizes median, interquantile mean (IQM), mean, and optimality gap, aggre- gated over the 20 games. For all plots we report the mean with 95% stratified bootstrap confidence intervals. Agarwal al. (2021) suggest IQM as the more robust of these metrics, we place stronger emphasis on it. Throughout the main paper we present the main findings, but include extra figures in the appendix.
2.2 WHEN DO SMALLER BATCH SIZES IMPROVE PERFORMANCE?
We first varied the batch size for all agents (Figure 1). There are two surprising observations from this result. The first that aggregate agent performance is relatively stable with respect to changing batch sizes. The second, and perhaps more surprising, result is that agent performance seems to improve with reduced batch size. Indeed, we can observe that the default batch size is in fact not optimal for any of the agents and, with the exception of DQN, all agents seem to benefit from reduced batch size.
The four agents considered differ in number of respects. Three important considerations are that, of the 4, DQN is the only agent without distributional training (Bellemare et al., 2017), prioritized experience replay, and the only one without n-step returns. To get better sense for whether either of these components is responsible for the reduced batch size performance boost, we performed ablation studies similar to those conducted by Ceron & Castro (2021). Since the version of Rain- bow provided with the Dopamine library (Castro et al., 2018) is effectively DQN with three added components, we can investigate the changing dynamics as these components are added or removed from DQN and Rainbow, respectively. Figure 2 depicts the outcome of this ablation study. We find a striking pattern: while the four variants that use 1-step learning see their performance increase with greater batch sizes, might be expected, the relationship is almost completely reversed for the variants using 3-step learning. Additionally, the other two components do not seem to present such relationship with batch size. See Appendix B for further results QR-DQN and IQN.
Humen Hormalized Score
1.5
DQN: Adding companents
Rainbow: Removing components
Default
Multi step
orittized
Distributional
0.5
0.5
8
16
64
16
Batch size
Batch size
Figure 2: Left: Adding components to DQN; Right: Removing components from Rainbow.
The last results demonstrated there is a strong performance relationship between batch size and up- date horizon. We systematically explored this by evaluating various choices of these two parameters for three of the agents. As Figure shows, the optimal batch size decreases n increases. This is most stark in QR-DQN, where simply reducing the batch size to 8 improves performance by close 70% on the subset of games we consider." With Rainbow a batch size of 8 is able to maintain performance for n-step values as high as 9; in contrast, performance for the default batch size of 32 collapses beyond an n-step of 3.
'Dopamine uses a "compact" version of original Rainbow agent, including only n-step updates, priori- tized replay, and categorical-distribatte RI. In Dopamine, QR-DQN update horizon of by default.
3

---

Under review as a conference paper at ICLR 2023
8
16
32
64
1.0
1.0
1.5
10M
3
9
11
15
11
15
5
11
13
NStep
MStep
NStep
Figure 3: Varying batch sizes and rt-steps in DQN (left), Rainbow (center), and QR-DQN (right),
UNDERSTANDING THE PHENOMENON
Having observed the advantages that can come when combining reduced batch sizes with increased update horizons, we would like to better understand the phenomenon. Given the dramatic change in performance observed in QR-DQN with an n-step value of 3 when reducing the batch size to (Figure 3), we focus on this agent and on the game Asterix, where the effect is prominent. We provide results on additional games in Appendix C., where we observe the same qualitative findings.
3.1 THE EFFECT ON VARIANCE
Reducing batch size and increasing the update horizon are generally thought to increase variance. We thus start by confirming this through measuring the variance throughout learning.
In Figure 4 we measure, from left to right, the training returns, the variance of the TD-loss, the variance the last layer's weight gradients, and the variance of the last layer's bias gradients. The first observation is that, in aggregate, we see increased variance with both reduced batch size and increased update horizon. Analyzing the results in more detail, we can observe the following:
Loss variance: As expected, this type of variance is correlated with the update horizon, but inversely correlated with batch size. In both cases, the variance seems have an upward trend throughout training.
Weight gradient variance: While we see a reduction variance with increased batch size (as expected), seem to observe reduction in variance with increased update horizon, which is the opposite what was expected. In all cases, the variance goes down as training progresses. Bias gradient variance: Here have the expected relationship: reduced batch size and increased update horizon both bring higher variance. In contrast to the other two types of variance, this one seems stabilize relatively early in training.
thus appears that, in aggregate, the performance boost is correlated with increased variance on both these fronts; we are dubbing this the variance double-down phenomenon. The curious behaviour of the reduced variance of the weight gradients when increasing n-steps may be an important compo- nent of this phenomenon: perhaps is generally advantageous to have a certain degree of variance in the weight gradients, and thus the reduced batch size helps counteract the effect of increased update horizon. Nevertheless, unlikely to be the only cause, so we investigate number of additional possible causes for the phenomenon below.
Do adaptive learning rates work better with lower batch sizes? All our experiments, like most modern RL agents, use the Adam optimizer (Kingma & Ba, 2015), variant of stochastic gradient descent (SGD) that adapts its learning rate based on the first- and second-order moments of the gradients, estimated from mini-batcho sizes have a second-order effect on the learning-rate adaptation that benefits agent performance, To investigate this we evaluated, for each training step, performing multiple gradient updates on subsets of the original sampled batch; the parameter M ini MiniBllhh Batch.SSll defines the number of gradient steps and dividing factor (where a value of 1 is the default setting). Thus, for Min BatchSplit of 4, we would perform 4 gradient updates with subsets of size instead of single gradient update with the full mini-batch of size 32. With an optimizer like SGD this has no effect (as they are mathematically
used for training. It is thus possible that smaller batch
We measured the variance of the other layers but they were qualitatively the same as the final layer.
4

---

Under review as a conference paper at ICLR 2023
Return
Loss
WeightGradient
BlasGradient
BatchSize
32
64
NStep
Returns
Frames (X1M)
Frames (x1M)
Frames (xIM)
Frames (x11M
Figure 4: Measuring the variance during training while varying batch size with n-step value equal to (top row), and varying n-step with batch size equal to 32 (bottom row) on Asterix.
equivalent), but may see differing performance due to Adam's adaptive learning rates. Indeed, the left panel in Figure 5 suggests that while there are differences, these are not significant enough to explain the performance boost observed in Figure when reducing the batch size to 8. We provide further analyses on all games in Appendix C.2.
M MisBarchhpilts
TaggehokusSiclle
Policy Policy Chum
0.012
33009
2.001
Beturn
12500
0.1
2000
0.
0.004
5008
Frammss Mx)
Batch
Size
46
(x030)
300
Evaluating multiple gradient updates per training step; Center: measuring policy churn during training; Right: adding noise to the target update.
Figure 5: Left:
Is it side effect of policy churn? Schaul et al. (2022) highlighted the phenomenon of policy churn, whereby the greedy action of a value network is rapidly and consistently changing through- training. This effect was hypothesized exploration. the middle panel of Figure 5 we measured the average policy churn throughout training. We can see that policy churn increases with batch size. This is somewhat expected, as increased batch sizes effectively increases the replay ratio, which was shown to be correlated with policy churn. However, given that Schaul et al. (2022) found increased policy churn to be benefi- cial, it is likely that the reduced policy churn is correlated with. but not the cause of, the improved performance observed with reduced batch size.
provide training benefit in the form of implicit
Is variance all one needs? The main thesis of this work is the performance benefit obtained from the variance double-down phenomenon. Although we have focused on increasing variance "structured",, indirect, way by varying update horizon and batch size, one can also increase variance more direct ways. We explore this by adding noise to the target values used by the Bellman update during learning. Specifically, we sample zero mean isotropic Gaussian matching the shape of the target values, scale by a multiplier, and add to the target values. In the right panel of Figure 5 compare the performance when using different scaling values. Rather surprisingly, scaling the target noise by 5.0 gives a dramatic performance boost on the game Asterix. Although this striking result does not hold across all games (see Appendix C.3), it does hold for many. This
5

---

Under review as a conference paper at ICLR 2023
suggests that indeed, prediction variance may play incompletely understood, yet beneficial role in deep reinforcement learning.
4 EFFECT UNDER DIFFERENT LEARNING REGIMES
In the preceding sections we established that learning from minibatches that are much smaller than standard results in improved performance across the gamut of Atari 2600 games. Specifically, our analysis takes place in the 100-million frames regime. At this point, one may wonder whether the root cause of the double-down phenomenon is due to peculiarities of this regime, for example the interplay between online exploration and neural network predictions (what Ostrovski et al. (2021) call the tandem effect). To investigate this further, affects performance in other learning regimes, and how this relates to the degree to which multi-step learning is used by high-performing algorithms for this regime.
now study how the batch size parameter
4.1 THE LOW DATA REGIME
We first consider algorithms designed for the low data regime, specifically the Atari 100k benchmark introduced by Kaiser al. (2020).5 This is of particular interest to us as all algorithms that achieve competitive performance on this benchmark do so by increasing the parameter beyond what is used for longer training periods. Here we consider three methods, all of which use n = 10: Data-efficient Rainbow (DER), a version of the Rainbow algorithm with hyper-parameters tuned for faster early learning (van Hasselt et al., 2019); SPR, which incorporates self-supervised learning to improve sample efficiency (Schwarzer et al., 2020); and DrQ(e), which in addition uses data augmentation (Agarwal et al., 2021). Our results in this section evaluate performance on 26 games (the standard for this setting), aggregated over 6 independent trials.
Figure 6 (top) depicts the performance of all three agents after 100,000 agent steps. measured in terms of interquartile mean (IQM) of human- normalized scores (Agarwal et al., 2021). We observe that DER exhibits the same trend as the preceding experiments: reducing the batch size from its default value of 32 transitions improves performance. This is expected given that DER is tuned version of Rainbow. The trend is less clear for SPR and DrQ(e), although in the the former case similar performance is achieved for smaller batch size.
These results concern published agents, whose hyper-parameters (including n and, to some extent, the batch size) have been tuned to maximize performance in the 100k regime. To understand whether the double- down relationship between n and the batch size parameter also holds in this regime, we evaluated these agents on wider range of parameter values (Figure 6, middle). We find that indeed, DER exhibits this double-down relationship, with the optimal batch size varying as a function of n; the result is also present, to a lesser extent, with SPR and appears to be absent from DrQ(e). This suggests that additional algorithmic components present in the latter may obviate the need for reducing the batch size in this particular regime.
Given that performance in the 100k regime depends on number of considerations and is difficult to measure precisely, in a further experiment we trained DER and DrQ for the longer duration of 30M frames. This provides interesting in bebeeeeen the low data regime and the classic 100M regime. On this longer time frame, we indeed find that both algorithms exhibit the double-down phenomenon (Figure 6, bottom): for DER, a batch size of 8 performs substantially better, while for DrQ batch size of 16 slightly outperforms the default value. Combined with the other results presented here, this suggests that the double-down phenomenon may only emerge with longer training regimes.
4.2 THE OFFLINE REGIME
We next turn our attention to the offline reinforcement learning regime (Gulcehre et al., 2020; Levine al., 2020), 6 where we are given dataset of sample transitions from which we would like to obtain a policy that performs well. Compared to the online regime, learning offline is more challenging as there is more room for overfitting to the fixed dataset, and there is no possibility for the agent to
$ For comparison with the 100M frames regime presented earlier, note that "100k" this context refers to 100,000 agent steps, or 400,000 frames.
'Or batch reinforcement learning (Ernst al., 2005).

---

Under review as a conference paper at ICLR 2023
DER
SPR
DrQ(eps)
32 (default)
16
8
4
0.20
0.24
0.28
0.25
0.30
0.35
0.18
0.24 0.30
IQM Human Normalized Score
16
32
DER
SPR
DrQ(e)
0.4
0.4
0.3
0.3
0.2
0.2
0.2
Human
0.1
0.1
0.1
0.0
5
10
15
20
0.0
5
10
20
0.0
30 N
15
N
N
DER
DrQ(eps)
(QM Human Normalized Score
(QM Human Normalized Score
1.0
2.4
2.2
2.6
16
32 (default)
15 Number of Frames (in millions)
20
30
10
13
30
Number of Frames (in millions)
Figure 6: Varying batch sizes using the default settings of DER, SPR, and DrQ(e), evaluated at 100k (top and middle) and 30M frames (bottom).
correct its estimation mistakes by interacting with the environment (as argued by Ostrovski et al. (2021)).
We study the effect of varying n and the batch size parameter for three algorithms: DQN, CQL (Kumar al., 2020), and CQL+DR3 (Kumar et al., 2021). Except for DQN, these algorithms specifically tailored to the offline regime, incorporating among other things a penalty to mitigate value overestimation. We follow the training scheme of Kumar et (2021): each agent trained on 17 games from the ALE for 200 iterations (where each iteration consists of 62.5K gradient steps), and after each iteration the agent is evaluated for 125K steps on the environment. The offline dataset consists of the transitions experienced during the full training of a DQN agent (Agarwal et al., 2020).
Figure illustrates the impact jointly varying our two parameters of interest on performance. In the case of CQL+DR3 (the highest-performin method), it is clear that when is increased from to 3, it is beneficial to also reduce the batch size (from to or 8), in line with our previous findings. For CQL alone, the relative performance gap between batch sizes is reduced. We find a similar trend for DQN. One might argue that reducing the batch size without additional training effectively mitigates overestimation, simply because each transition trained fewer times. A closer look at the learning curves (Fig 29 in the appendix) suggests reduced overfitting is not the main factor explaining results, least regarding CQL and CQL+DR3.
RELATED WORK
There has been a growing interest in developing a better understanding of reinforcement learning dynamics with neural networks, and our work falls in this category. Ceron & Castro (2021) demon- strated the surprising finding that simple switch of optimizer and loss can dramatically improve the performance of DRL agents. Andrychowicz et al. (2021) performed broad examination of the

---

Under review as a conference paper at ICLR 2023
CQL+DR3
CQL
DQN
64
32 (default)
16
8
4
0.75
0.90
1.05
0.2
0.4
0.6
0.8
0.03 0.06 0.09 0.12
IQM Human Normalized Score
CQL+DR3
CQL
DQN
64
32 (default)
16
8
0.90
1.05
1.20
0.6
0.7
0.8
0.06 0.12 0.18 0.24
IQM Human Normalized Score
Figure 7: Varying batch sizes for offline experiments. Top: All agents use and Bottom: all agents use n 3.
impact varying et al. (2021) performed similar analysis, but for value-based agents. Lyle al. (2021) identified a mechanism by which non-stationary prediction targets can prevent learning progress in deep RI. agents. They term this phenomenon as capacity loss. Wang et al. (2022) extensively and systemati- cally investigated the properties representations learned by deep reinforcement learning systems. Fujimoto al. (2022) studied the relationship between the Bellman error and the accuracy of value functions through theoretical analysis and empirical study. They found that the Bellman error is poor proxy for the accuracy of the value function.
can one agent performance, for volicy-gradiemt methods; Araújo
The use of overparameterized deep models in value based RL still exhibits mysteries in stability and performance. To better understand the utility of deep models in RL, Xiao et al. (2022) presented an anaylsis of recursive value estimation using overparameterized linear representations. Nikishin (2022) identified the primacy bias in deep RI., a damaging tendency of artificial agents to overfit early experiences. They proposed resetting mechanism allowing the agent forget part of its knowledge. Fedus et al. (2020) conducted depth study of how replay affects performance in value-based deep RL agents. Lahire al. (2021) I); Stooke & Abbeel (2018) studied the key role of batch size in deep RL agents over a significant range of classical RL benchmarks.
DISCUSSION
The long-term goal of reinforcement learning research is to develop generally capable agents that can adapt to uncertain environments. Although theoretical results spanning multiple decades have given us a crisp insight into the mathematical properties of these algorithms, these theories unfortu- nately do not hold for non-linear function approximators such neural networks. Given that neural networks have played a key role in the impact RL has had since 2015, it behooves the community to develop a better understanding of the the interplay of the various components and how changes can affect learning dynamics.
Our work has revealed the striking finding that doubling down on variance by increasing and reducing batch size seems to, overwhelmingly so, produce improved performance. This flies in the face of traditional beliefs from the supervised learning community that reduced variance is best. Indeed, the remarkable performance gains obtained from simply adding noise to the target values (right panel Figure 5) suggest that our relationship with learning variance needs to be better understood. One important aspect of our findings is that it seems to be limited to the of deep neural networks: indeed, in Appendix we find that the phenomenon not present when using linear function approximators.
8

---

Under review as a conference paper at ICLR 2023
Rainbow
QR-DQN
1QM Human Normalized Score
1.4
IQM Human Normal lizd Score
1.0
0.6
Batch Size:
0.4
0.4
8
16
0.2
32 (default)
64
0.0
0.0
10
20
40
50
60
70
16
20
30
36
60
Time (hours)
Time (hours)
Figure 8: Measuring runtime versus performance when varying batch sizes in Rainbow and QR-
DQN with n-step equal 3.
One may wonder whether the same effect can be observed if one were to increase the stochasticity of action selection by, for instance, increasing the value of c of the r-greedy strategy used by all these agents. The results in Appendix H suggest otherwise: decreasing seems to be more beneficial. It is interesting to observe that the variance double-down phenomenon remains present despite varying values of
One natural area for further exploration is to explore this phenomenon actor-critic methods (Haarnoja et al., 2018; Fujimoto al., 2018) those agents which their return estimator is based on multi-step (Tang et al., 2022), such as Q(A) (Harutyunyan et al., 2016), Retrace (Munos et al., 2016) emphatic algorithms (Jiang al., 2021). We present that suggests the variance double-down phenomenon is present in these scenarios. Additionally, because this paper focuses mainly an empirical investigation of the doubling down phenomenon, an exciting future work it is to develop a theoretical understanding about the nature and dynamics of this phenomenon.
preliminary findings in Appendix G
6.1 COMPUTATIONAL CONSEQUENCES
Empirical advances in deep reinforcement learning are generally measured with respect to sample efficiency; that is, the number of environment interactions required before achieving a certain level of performance (as we have done throughout this paper). While valid metric, it fails capture computational differences between algorithms.
If two algorithms have the same performance with respect environment interactions, but takes twice as long to perform each training step, one would clearly opt for the faster of the two. This important distinction, however, is largely overlooked in the standard evaluation methodologies used by the DRL community.
Many of our results have demonstrated the performance benefits obtained when reducing the batch size, but an additional important consequence is the reduction in computation wall-time. Figure 8 demonstrates that not only can we obtain better performance with reduced batch size, but we can do so at a fraction of the runtime.
We invite the reader to revisit the results presented above under this lens. For example, when evalu- ated with respect to environment interactions, the top row of Figure 6 would suggest that there is no real advantage to reducing the batch size from 32 to 16 for SPR; however, if evaluated with respect to computation time, the advantages of using reduced batch would become apparent.
As argued by Ceron Castro (2021), the ALE as a benchmark proves quite onerous for communities with limited access compute; thus, computational gains like the one presented here help reduce this barrier to entry. We encourage others to consider not just sample efficiency, but also computational efficiency, when evaluating new methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. In International Conference Learn ing Representations, 2018. URL https: /oppneeeeeeeeeeeeeeeeeeee net/ forum 1ideiidss1im 11d=$1ANXQOO0.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114, PMLR, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Beyond tabula rasa: Reincarnating reinforcement learning. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
Marcin Andrychowiez, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? large-scale study. In International Conference on Learning Representations, 2021. URL hwtpt:ttpp....m opw.revie.....e.e www.eiiddnnnddnnidieeiii
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
João Guilherme Madeira Araūjo, Johan Samir Obando Ceron, and Pablo Samuel Castro. Lifting the veil on NeurlPS 2021, 2021.
for value-bases deep reinforcement learning. In Deep RL Workshop
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013. doi: 10. 10.161 /613//air..912. URL https:/ / do org/ 10. C 1611 1616611111252 wa..saaaaarrr3..... /0.
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, pp. 458, 2017.
Mare G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos Machado, Sub- hodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric bal- loons using reinforcement learning. Nature, 588(7836):77-82 2020.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http:
/ //arxiv.org/aaaaa abs /11...111110 06110.
Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more in- sightful and inclusive deep reinforcement learning research. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1373-1383. PMLR, 18-24 Jul 2021. URL https::/proceediin
www..ppees///e. /v139/ http//11199///eee0// ceron2la .hmll.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural genera- tion to benchmark reinforcement learning. arXiv preprint 192.1:191..01588, 2019.
W. Dabney, M. Rowland, Mare G. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In AAAI, 2018a
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1096-1105. PMLR, 2018b.
10

---

Under review as a conference paper at ICLR 2023
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Mag- netic control of tokamak plasmas through deep reinforcement learning. Nature, 60217897):414- 419, 2022.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importanee weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning), ICML' 18, 2018.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney, Revisiting fundamentals of experience replay. In International Con- ference Machine Learning, pp. 3061-3071. PMLR, 2020.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alexander Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In Proceedings of the International Conference on Repre- sentation Learning (ICLR 2018), Vancouver (Canada), 2018.
Scott Fujimoto. Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, and Shixiang Shane Gu. Why should i trust you, bellman? The Bellman error is poor replacement for value error. In Kama- lika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 6918-6943, PMLR, 17-23 Jul 2022. URL https: petps://proceedinie
mlr prwss/ 62/ fu fujimoto22a html.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning, Advances Neural Information Process- ing Systems, 33:7248-7259, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. PMLR, 2018.
Anna Harutyunyan, Mare G Bellemare, Tom Stepleton, and Rémi Munos. (A) with off-policy corrections. In International Conference on Algorüthmic Learning Theory, pp. 305-320. Springer, 2016.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski. Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Ray Jiang, Tom Zahavy, Zhongwen Xu, Adam White, Matteo Hessel, Charles Blundell, and Hado Van Hasselt. Emphatic algorithms for deep reinforcement learning. In International Conference on Machine Learning. pp. 5023-5033. PMLR, 2021.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Mitos, Blažej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2020. URL https:// openreview wttrrriwwwweeeeeeeeeeeeeee.eeeee forum? iieeiiddsssss JB.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent perience replay in distributed reinforcement learning. International conference on learning representations, 2018
11

---

Under review as a conference paper at ICLR 2023
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: / //arxiv.org////o.o abs 1412. 6980.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-11999 2020.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Representations, 2021.
Thibault Lahire, Matthieu Geist, and Emmanuel Rachelson. Large batch experience replay. arXiv preprint arXiv:2110.015268 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Mach. Learn., 8(3 (0:544)22333353 -21,, May 1992.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in rein- forcement learning. In International Conference on Learning Representations, 2021.
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Artif. Int. Res., 61(1):523-562, jan 2018. ISSN 1076-9757.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, February 2015.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16828-16847. PMLR, 17-23 Jul 2022. URL https: httts::/proceediine mml.. .tt.ingssms...sssss//.... w162/nikishin222 hhtml.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL ptttps://operreviee net met/Eorumt B.d.nPAAAGGGGC
Julien Perolat, Bart de Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. arXiv preprint arXiv:2206.1533 2022.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In Proceedings of ICLR, 2016, 2016.
Tom Schaul, André Barreto, John Quan, and Georg Ostrovski. The phenomenon of policy churm. Advances in Neural Information Processing Systems, 35:4235-423 4246, 2022.
12

---

Under review as a conference paper at ICLR 2023
Julian Schrittwesse, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588 (7839):604-600) dec 2020. doi: 10. 10.1038/s41 38000004116-00000051---4- URL https: /doi org/
10. 10.1003822Fsa 58 020003001---000
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficien reinforcement learming with self-predictive representations. In International Conference on Learning Representations, 2020.
Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the effects of data parallelism neural network training. Journal of Machine Learning Research, 20 12):1-1))9 2019. URL http: jmlr. org/ https/pppps////// 18-789. html.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning walk in 20 minutes with model-free reinforcement learning. arXiv preprint vrXiv:2208.07860 2022.
Adam Stooke and Pieter Abbeel. Accelerated methods for deep reinforcement learning. arXiv preprint arXiv:1803 0.0001,,.1,, 2018.
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning.
3(1):9 44, August 1988.
Yunhao Tang, Mark Rowland, Rémi Munos, Bernardo Ávila Pires, Will Dabney, and Mare G Belle- mare. The nature of temporal difference errors in multi-step distributional reinforcement learning. arXiv preprint vrXiv:220..07570 2022.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillierap, and Martin A. Riedmiller. Deepmind control suite. arXiv,, 2018.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Proceedings of the Thirthieth AAAI Conference On Artificial Intelligence (AAAI), 2016, 2016. cite arxiv:1509. 011.9906666600 Comment: AAAI 2016.
Hado P Hasselt, Matteo Hessel, and John Aslanides. When forcement learning? In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL https: /ttp::::://pccceee //tpc//eeccccccccc/////c. neurips..... cttppppscc/ppppr/2/////// pdf.
parametric models in rein-
1b742ae2 5ad 8b 28w55499666222 2D2.5499662222792--4222222
Han Wang, Erfan Miahi, Martha White, Marlos Machado, Zaheer Abbas, Raksha Kumaraswamy. Vincent Liu, and Adam White. Investigating the properties of neural network representations in reinforcement learning. arXiv preprint arXiv: 159555 2022.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanetot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48, pp. 1995-2003, 2016.
Chenjun Xiao, Bo Dai, Jincheng Mei, Oscar A Ramirez, Ramki Gummadi, Chris Harris, and Dale Schuurmans. Understanding and leveraging overparameterization in recursive value es- timation. In International Conference on Learning Representations, 2022. URL https: / //openreview net/forum wwwprrummidddhhhhdeh shbAgghhhha.kk...
13

---

Under review as a conference paper at ICLR 2023
A BACKGROUND
A.1 DQN
Mnih et al. (2015) introduced DQN, which combined Q-learning with deep neural networks. Some of the most important design contributions are: (1) The Q function is represented using a feed forward neural network consisting of three convolutional layers followed by two fully connected layers. Two copies of the Q-network are maintained: a target network (parameterized by 0). The online network is updated via the learning process described below, while the target network remains fixed and is synced with the online weights at less frequent (but regular) intervals. (2) A large replay buffer D is maintained to store experienced transitions (s,0,T,8) (Lin, 1992). (3) The temporal difference update is implemented using the following loss function to update the online network: L(0) = e(
online network (parameterized by 0) and
DQN - -Qo(s, (s. a)) using YDQN = (r + y E. Qa(8', .a)) and mini-batches of size 32 sampled from the replay buffer D.
A.2 RAINBOW
In this section we briefly present the enhancements to DQN that were combined by Hessel et al. (2018) for the Rainbow agent.
Hessel et al. (2018) proposed Rainbow, a combination of several incremental improvements top of DQN (Mnih et al., 2015) that increased sample efficiency, stability and final performance of the DQN algorithm. Rainbow rests upon years of model-free RL research and has been adapted to work well in data-limited regimes. Rainbow agent is composed of six components: double Q- learning (van Hasselt al., 2016), prioritized experience replay (PER) (Schaul et al., 2016), dueling networks (Wang et al., 2016), multi-step learning (Sutton, 1988), noisy nets (Fortunato et al., 2018), and distributional reinforcement learning (Bellemare al., 2017).
The two main components are (PER) and multi-step learning, removing either component caused a large drop in median performance (Hessel et al., 2018; Ceron & Castro, 2021). PER proposed to sample a trajectory & = (s, r, s') with probability ps proportional to the temporal difference error instead of sampling uniformly from the replay buffer. In multi-step learning, instead of com- puting the temporal difference error using a single-step transition, one can use multi-step targets instead Sutton (1988), where for a trajectory (so, do, To, $1, a.....) and update horizon n: R(") := Ex * yielding the multi-step temporal difference: r2(") + + max Q0(sttn) a')
a0(
A.3 DISTRIBUTIONAL RL AGENTS
The experiments in the previous sections were conducted using the JAX Dopamine DQN and Rain- bow implementations. However, we did not evaluate if low batch size values behave similar in Distributional Q-learning variants. In this section, we investigate the interaction of Quantile Re- gression for Distributional RL QR-DQN (Dabney et al., 201 8a) and Implicit Quantile Networks IQN (Dabney et al., 2018b) with low batch sizes values on 20 Atari games (same setup of previous experiments).
QR-DQN (Dabney et al., 2018a) computes the return quantile values for N fixed, uniform prob- abilitics. This has no restrictions or bound for value, as the distribution of the random return is approximated by a uniform mixture of Diracs: Zz( a) = & "i' 6o.(x.)), with each 0, as- signed quantile value trained with quantile regression. IQN uses implicit quantile networks (IQN) as the parameterization of the return distribution Dabney et al. (2018b). IQN learns to transform a base distribution (typically a uniform distribution in J0, 1)) to the quantile values of the return dis- tribution. This can result in greater representation power in comparison to QR-DQN, as well as the ability to incorporate distortion risk measares.
14

---

Under review as a conference paper at ICLR 2023
B THE EFFECT OF REDUCED BATCH SIZE
In this appendix we provide additional results investigating the effect of reduced batch size, comple- menting the results presented in section 2.
DQN(n=1, default)
DON(n=3)
Rainbow(n=1)
Rainbow(n=3, default)
32 (default)
16
1.05
1.20
0.90
1.05
1.20
1.35
0.90
1.05
1.20
0.90
1.05
1.20
1.35
Human Normalized Score
QR-DQN(n=1)
QR-DON(n=3, default)
IQN(n=1)
IQN(n=3, default)
64
32 (default)
16
0.45
0.60
0.75
0.8
1.0
1.2
0.6
0.8
1.0
1.2
1.0
1.2
1.4
Human Normallzed Score
Figure 9: Evaluating different batch sizes values with two common update horizon values for DQN, Rainbow, QR-DQN and IQN.
QR-DON
QR-DON
1OM Human Normalizee Score
JOM Human Normalizer aadd Score
0.8
0.1
0.4
16
32 (default)
02
64
0.0
5.0
20
25
mB
66
Number of Frames (in millions)
Number of Frames (in millions)
Figure 10: Evaluation of different batch size values for Quantile Regression agent with n-step equal to (left) and (right).
Median
IQM
Mean
Optimality Gap
64 32 (default)
16
8
0.4
0.6
0.8
0.45
0.60
0.75
0.8
1.2
1.6
2.00.36
0.42
0.48
0.54
Human Normalized Score
Median
IQM
Mean
Optimality Gap
64
32 (default)
16
0.6
0.8
1.0
0.8
1.0
1.2
0.9
1.2
1.5
1.8
0.30
0.35
0.40
0.45
Normalized Score
Figure 11: Evaluation of different batch size values for Quantile Regression agent with n-step equal to (top) and (bottom).
15

---

Under review as a conference paper at ICLR 2023
IQN
IQN
(QM Human Normalized rimalizeed Score
Batch Size:
IQM Human Normalized aanzlllzdd Score
144
32 (default)
1.0
64
8.4
02
0.0
2.0
20
300
25
60
10
130
Number of Frames (in millions)
Number of Frames (in millions)
Figure 12: Evaluation of different batch size values for Implicit Quantile Regression agent with n-step equal to (left) and (right).
Median
Optimality Gap
64 32 (default)
16
0.75
0.90
1.05
0.6
0.8
1.0
1.2
1.5
1.8
2.1
0.30
0.35
0.40
0.45
Human Normalized Score
Median
IQM
Mean
Optimality Gap
64
32 (default)
16
8
0.90
1.05
1.20
1.0
1.2
1.4
1.5
2.0
2.5
0.27
0.30
0.33
0.36
Human Normalized Score
Figure 13: Evaluation of different batch size values for IQN with n equal (top) and (bottom).
16

---

Under review as a conference paper at ICLR 2023
C UNDERSTANDING OF THE VARIANCE DOUBLE-DOWN PHENOMENON
In this section we provide additional results to complement those presented in section
C.1
THE EFFECT ON VARIANCE
Retur
Loss
WeightGradient
BiasGradient
Returns
Variance
BatchSize
16
64
Returns
Variance
MStep
Frames (x1M)
Frames (x1M)
Frames (x1M)
Frames (x1M)
Figure 14: Measuring the variance during training while varying batch size with n-step value equal to (top row), and varying n-step with batch size equal to 32 (bottom row) on Pong.
Return
Loss
WeightGradient
BiasGradient
BatchSize
32
64
Returns
9 N S
Variance
Frames 61M)
Frames (1M)
Frames 6x1M)
Frames (1M)
Figure 15: Measuring the variance during training while varying batch size with n-step value equal to (top row), and varying n-step with batch size equal to 32 (bottom row) Breakout.
17

---

Under review as a conference paper at ICLR 2023
Retum
Loss
WeightGradient
BiasGradient
Variance
BatchSize
32
Returns
Variance
NStep
Frames (x1M)
Frames (x1M)
Frames (xIM)
Frames (x1M)
Figure 16: Measuring the variance during training while varying batch size with n-step value equal
to (top row), and varying n-step with batch size equal to 32 (bottom row) MsPacman.
Retur
Loss
WeightGradient
BlasGradient
BatchSize
Variance
NStep
Returns
Variance
Frames (x1M)
Frames (x1M)
Framies (x1M)
Framnes (x1M)
Figure 17: Measuring the variance during training while varying batch size with n-step value equal to (top row), and varying n-step with batch size equal to 32 (bottom row) on SpaceInvaders.
18

---

Under review as a conference paper at ICLR 2023
C.2
Do ADAPTIVE LEARNING RATES WORK BETTER WITH LOWER BATCH SIZES?
IQM Human Normalized Score
0.8
0.6
0.4
MiniBatchSplits
0.2
0.0
0
20
40
60
80
100
Number of Frames (in millions)
Figure 18: Evaluating multiple gradient updates per training step on QR-DQN, aggregated over all 20 games.
200
Figure 19: Evaluating multiple gradient updates per training step on QR-DQN, training curves for all games.
19

---

Under review as a conference paper at ICLR 2023
C.3 Is VARIANCE ALL ONE NEEDS?
1QM Human Normalized Score
0.8
0.6
Talge
0.4
None
0.2
1.0
5.0
0.0
0
20
40
60
80
Number of Frames (in millions)
Figure 20: Evaluating the effect of adding target noise to QR-DQN, aggregated over all games.
Arta
Rlaiiny in
Graskon:
Tayych
Tepechoccuat
6:000
+0.04
Standar
Tanych anaiuatt
Tanyal
Pino
Envenbye
Sear crt
1.25
Tourth
Yarsheverge
20000n
Figure 21: Evaluating the effect of adding target noise to QR-DQN, learning curves for all games.
20

---

Under review as a conference paper at ICLR 2023
D EFFECT UNDER DIFFERENT LEARNING REGIMES
In this appendix we provide additional results to complement those presented in section 4.
D.1 Low DATA REGIME
DER
1QM Human Normalized Score
Batch Size:
JQM Human Normalized Score
0.30
Batch Size:
0.25
16
0.023
32 (default)
(default)
0.015
0.35
0.012
0.30
0.003
0.000
-0.003
Agent Steps 10k)
Agent Steps (x 10k)
Figure 22: Varying batch sizes in DER with n-step equal to (left) and 10 (right).
Median
Mean
Optimality Gap
32 (default)
16
8
0.016 0,024 0.032
0.018 0.024 0.030
0.05
0.10
0.93
0.96
0.99
1.02
Human Normalized Score
Median
IQM
Mean
Optimality Gap
32 (default)
16
8
0.15
0.20
0.25
0.30
0.20
0.24
0.28
0.32
0.36
0.40
0.44
0.63
0.66
0.69
0.72
Human Normalized Score
Figure 23:
Aggregate metrics on Atari 100k based on 26 games. Evaluation of different batch sizes DER with n-step equal to (top) and 10 (bottom).
SPR
SPR
MM Human Normalized Score
0.10
IQM Human Normalized Score
Batch Size:
0.09
0.25
8.06
(default)
0.20
3.04
0.15
0.30
3.02
0.06
Agent Steps (x 10k)
Agent Steps (x 10K)
Figure 24: Varying batch sizes in SPR with n-step equal to (left) and 10 (right)
21

---

Under review as a conference paper at ICLR 2023
Median
IQM
Mean
Optimality Gap
(default)
16
0.02
0.04
0.06
0.08
0.025
0.050
0,075 0.100
0.1
0.2
0.3
0,40.78
0.84
0.90
0.96
Human Normalized Score
Median
IQM
Mean
Optimallty Gap
(default)
16
0.2
0.3
0.4
0.25
0.30
0.35
0,48
0.56
0.64
0.57
0.60
0.63
0.66
Human Normalized Score
Figure 25: Aggregate metries when varying batch sizes in SPR with n-step equal to (top) and 10 (bottom)
Median
IQM
Mean
Optimality Gap
32 (default)
8
0.60
0.75
0.90
0.75
0.90
1.05
0.8
1.0
1.2
1.4
0.32
0.36
0,40
Human Normalized Score
Median
IQM
Mean
Optimality Gap
32 (default)
16
8
0.96
1.02
1.08
1.14
1.20
1.35
1.50
1.65
1.8
2.0
2.2
2.4
0.2100.225 0.200 0.240..255
Human Normalized Score
Figure 26: Aggregate metrics on Atari 30M based on 26 games. Evaluation of different Varying batch sizes for DER (top) and DrQ(c) (bottom) using n-step equal 10.
DrQ(eps)
IQM Human dormalized Score
0.35
Batch Size:
4
0.25
16
32 (default)
0.15
0.10
0.05
0.00
2
6
10
Agent Steps (x 10k)
Figure 27: Varying batch sizes in DrQ(e) with n-step equal to 10.
22

---

Under review as a conference paper at ICLR 2023
0.6
Batch Soes
16
32 (detault)
IQM Human Normalized Poomalized Score
128
0.4
0.3
0.2
5
10
15
20
25
30
Number of Frames (in millions)
Median
Optimality Gap
128
64
32 (default)
0.2
0.3
0.4
0.5
0.3
0.4
0.5
0.5
1.0
1.5
2.0
0.48
0.56
0.64
Human Normalized Score
Figure 28: Evaluation of different batch size values for DER agent Atari 30M based the remaining 34 Atari games. Top: IQM curves; Bottom: Aggregate metrics.
23

---

Under review as a conference paper at ICLR 2023
D.2
OFFLINE RL
1.2
1.0
3.8
0.00
2.92
0,25
2.0
0.00
50
100
150
205
10
100
150
200
255
1.4
1.0
9.35
1.2
ops Scotre
9.30
3225
3.8
0.6
0.15
0.4
10M
20.00
0.
0.00
0.0
32
100
1.0
200
5g
1.00
150
200
103
150
220
Figure 29: Varying batch sizes for CQL+DR3 (left), CQL (center), and DQN (right), with n-step equal to (top row) and 3 (bottom row).
24

---

Under review as a conference paper at ICLR 2023
E
COMPUTATIONAL CONSEQUENCES
DON
Rain
1QM Human Normalized Score
IQM Human Normalized Score
1.0
0.0
0.6
0.6
0.4
0.4
0.2
0.2
40
60
80
10
20
40
50
50
70
80
Time (hours)
Time (hours)
QR-DON
IQN
IQM Human Normalizer lized Score
IQM Human Normalized Score
1.0
0.6
Batch Size:
8
0.4
(default)
Time (hours)
Time (hours)
Figure 30: Measuring runtime versus performance when varying batch sizes in DQN, Rainbow, QR-DQN, and IQN (from left to right), all with n-step equal to 3.
29.0 hoars
82.0 hours
66. hoars
52.0
43.0 hours
50k Pettado
Kammora Deieils
Agents
Agents
Figure 31: Computational savings: evaluating different batch sizes in DQN (left) and Rainbow (right), with n-step equal to and n-step equal to 3.
25

---

Under review as a conference paper at ICLR 2023
F
EVALUATION WITH LINEAR FUNCTION APPROXIMATORS
monttacc
acrobct
R4tum
Setur
22 odefouki
64 123
(teburt)
[teburt)
campooe
Reau
(defach)
(cebaut)
126
Figure 32: Evaluation of different batch values for DQN agent using Fourier series (Konidaris et al., 201 1) as simple linear function approximation. Right: DQN(n 1) and Left: DQN(n 3).
26

---

Under review as a conference paper at ICLR 2023
G IS THE PHENOMENON PRESENT IN OTHER ENVIRONMENT TYPES?
Although not as comprehensive as the results our main paper, we found that the variance double- down phenomenon does appear be present in continuous control tasks (Figure 33), well in procedurally generated environments (Figure 35).
1900
801-
330
830
500
433
Rrgentum
contral hopprctoe
1000
Exhadd batum
302
400
200
222
400
100
controfcciitt mossiaaay
contrstonlllarrrn
000
700
64
000
300
605
433
1024
200
755
256 (default)
140
140
360
Divironment Steps
Figure 33: Evaluation of different batch size values for MPO (Abdolmaleki al., 2018) agent DM-control suite environments (Tassa et al., 2018), with n=5.
Median
Mean
Optimality
1024
512
256 (default)
64
0.80
0.85 0.90
0.84
0.88
0.92
0.72
0.78
0.84
0.12
0.18
0.24
0.30
Normalized Score
Median
IQM
Mean
Optimality Gap
1024
256 (default)
128
64
0.84
0.88
0.92
0.84 0.87 0.90
0.93
0.75
0.80
0.85
0.15
0.20
0.25
0.30
Normalized Score
Figure 34: Varying batch sizes in MPO (Abdolmaleki et al., 2018) agent on DM-control suite envi- ronments (Tassa al., 2018) with n-step equal to (top) and 5 (bottom)
27

---

Under review as a conference paper at ICLR 2023
64
128
256
512
0.45
IQM
0.40
0.35
0.30
1
5
10
NStep
Figure 35: Evaluation of different batch size values for Rainbow agent on the ProcGen suite (Cobbe al., 2019).
28

---

Under review as a conference paper at ICLR 2023
H EXPERIMENTS WITH VARYING EXPLORATION VALUES
0.001
0.01
0.1
1.2
5.8
120 Haamar Nattre Witmaited teed St0
0.3
5.6
00
0.5
129 Hamaa
0.2
10H
0.0
0.0
0.001
0.01
2.00
2.75
trere
2.75
(OMtenmur Numuluel Soae trsi
1.75
Batch Size
125
1.25
32
1.0
3.00
64
2.73
0.75
030
2.30
225
IQ
3.25
3.0
100
Namber Framms milloral
Number Nunbor orarmmas millona
Nambst of fromes in millooal
Figure 36: Evaluation of different batch size values (colored lines), n--teps (rows), and exploration
values (columns) on QR-DQN. The default values are 32, and 0,01, respectively.
29

---

Under review as a conference paper at ICLR 2023
1
EXPERIMENTS WITH VARYING LEARNING RATES
Leamiingaae Se-06
LearningRate 5e-05
Leamiingaate 0.0005
0.
0.3
0.0
0.0
0.7
0.3
9.2
62
9.2
0.3
03
0.0
0.0
LearningRate Se-06
LearningRate 5e-05
LearningRate 0.0005
Batch Size
1.0
0.8
64
8.6
6.6
9.6
0.4
0.0
0.0
0.0
20
Number
Kumber Trames millone)
Number Frames millant
Figure 37: Evaluation of different batch size values (colored lines), n-steps (rows), and learning rates (columns) on QR-DQN. The default values are 32, 3, and 1e-5, respectively.
30

---

Under review as a conference paper at ICLR 2023
J ON-POLICY EVALUATION
Most of our experiments are in the online, off-policy setting, so it is informative to evaluate whether it is present an on-policy situation. To do so, populate the replay buffer with fixed policy i coming from pre-trained agent, and do policy estimation with the same learning rule (based on the Bellman update). Figure 38 plots the Q-estimation error for set of five games. The results suggest that the phenomenon is somewhat present, but only mildly so.
Breakout (n=3) 1)
Breakout (n 3)
Asterix (n=1) 1)
Asterix (n 3)
Q-estimation error
Pong (n=1)
Pong (n=3)
Qbert (n=1)
Qbert (n=3)
Q-estimation error
Seaquest (n=11)
Seaquest (n=3)
BatchSize
16
Number of frames (millions) Number of frames (millions)
Figure 38: Evaluation of on-policy estimation errors for different batch size values (colored lines) and n-steps (in title) with QR-DQN.
31


Reviews:
[{"review_id": "mWhQrLPYR1B", "reviewer": "Reviewer_iTrT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper perform an exhaustive investigation of the role of batch size and the steps used in temporal difference learning. Unlike the conventional intuition in supervised learning that increasing batch size would lead to variance reduction, thus increase the overall performance. In RL (Deep Q Learning based algorithms), decrease the batch size and increase the steps for Q-function learning would improve the performance, which is a counterintuitive finding in the literature. ", "summary_of_the_review": "Overall I think in this paper, the authors studied a very interesting question, and it is worth investigating. However, one of the major drawbacks of the paper is that we don't know why the claim is drawn in the RL settings, and also, there is no theoretical analysis or further understanding of the phenomenon. ", "strength_and_weaknesses": "**Strength**\n- The authors conducted extensive experiments to show the phenomena under all kind of different settings (such as different Q-learning algorithms, DQN, Rainbow, DQN-QR, IQN). In standard Q learning settings,  the empirical experiments support the authors' counterintuitive claim. \n- The paper is well written, and the experiments conducted are easy to follow. \n- The claim is interesting, which is beneficial for empirical hyper-parameter selection. \n\n**Weakness**\n- It is still unclear why this claim is drawn in RL, rather than standard supervised Learning settings. There is no theoretical analysis or deeper understanding why this phenomena occurs. \n- For the offline setting, why there are some counterexample that does not support your claim? \n- I personally think it would be really great if the authors could point out what perspective that reducing batch size and increase the n-step in Q learning would improve the performance, especially:\n  - **Exploration**. Intuitively decreasing the batch size would increase the variance for gradient updates, which potentially increase the exploration benefits.\n  - **Optimization / Estimation for Q-function**. At first I think under the offline setting it would be very easy to validate whether reducing batch size and increase n in Q learning would help optimization, but from your experiments, we just see not consistent results. Besides, I think by performing an off-policy evaluation experiments (fix $\\pi$ and learning Q) would check whether these two strategy would help Q estimation. \n  - **Generalization**. There are previous work identifies that smaller batch size would possibly improve the performance of generalization. I wonder if your two strategy would also observe similar benefits under the RL settings. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "c86xqE2q2C", "reviewer": "Reviewer_74JP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "### Summary\n\n&nbsp;&nbsp; This research examines the effect of batch size on performance in multi-step reinforcement learning. In supervised-learning experiments, several researchers have found that the variance of neural networks during training has a significant effect on learnability and convergence, since the neural networks' stochasticity pushes them to global optimality regions. Typically, non-linearity properties of networks, such as activation functions and skip-connection, direct a large number of inflection points to loss surfaces with respect to parameter spaces. Various researchers have conducted technical and empirical studies to demonstrate loss surface of neural networks and escape of local optimal points by utilizing stochasticity of learning. \n\nIn contrast, in deep reinforcement learning, there are several hyper-parameters, hence the influence of batch size is rarely investigated. In addition, the multi-step approach that assesses value-functions as the sum of those on the last visited states has been extensively employed for empirical performance enhancement. However, there was no actual research demonstrating how these characteristics impact agents' performance. In this context, this work analyzes the impact of batch size and the number of steps of value-function on both online and offline reinforcement learnings. This study asserts that increasing variances by decreasing batch size enhances overall performance, and the bigger the number of steps that are likewise predicted to have a high variance does not ensure excellent performance. \n\nIn order to make a fair comparison, this research presented the Dopamine framework [1] that built the ALE framework and conducted extensive experiments to substantiate their hypotheses. There are a few results that do not accord with their claims, but in general, as batch size drops, network performance and variances of loss and gradient values rise. Instead, there are no global trends for the amount of steps, but there is a sweet spot for performance. For instance, the majority of studies stated that three or five is the optimal number of multi-steps like Dopamine framework.\n\n[1] Machado, Marlos C., et al. \"Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents.\" Journal of Artificial Intelligence Research 61 (2018): 523-562.", "summary_of_the_review": "Overall, the paper marginally show perfirmance enhancement using a batch-size and the number of steps in multi-step reinforcement learnings. In order to provide more general insights, this should be addressed by either improved theoretical statement or extensive experimental results across various environments. ", "strength_and_weaknesses": "### Strength\n\n- Numerous experimental findings altering hyper-parameters such as batch sizes and step-sizes of multi-step value functions are presented in this study. \n- This article shown that ablation research varies each component of deep value-based RL. \n- By changing the batch size, they have repeatedly shown that agents get higher results in the ALE framework. \n- Motivated by the stochasticity of networks in supervised learning, they studied the gradient values of networks based on the number of batch-sizes.\n\n### Weakness\n\n- This study provides neither a technical nor a theoretical explanation of batch size and multi-step in deep reinforcement learning. I was unable of digesting a high-level summary of this research. Specifically, I disagreed with the author's statement that multi-step reinforcement learning should be examined independently. Numerous empirical research have already demonstrated that the multi-level value function does not exhibit a consistent trend across diverse situations. As the authors noted out in their paper, past research has, as a rule of thumb, set the value of multi-level performance to three. At this time, I was unable to understand why this work manipulates and reveals the effect of multi-levels and its impact on batch-size by up to 15. \n\n- Numerous theoretical works [1, 2, 3] have examined how the stochasticity of gradients is obtained from batch components. These studies also asserted that random mini-batches (shuffling) demonstrate networks that are more generalized than static mini-batches. I guess that these research are incompatible with online learning, but I believe they might give insightful perspectives on the exploration of batch size for off-line learning. In particular, in theoretical RL, the majority of publications might be demonstrating that such an algorithm can create a transitions matrix from provided datasets that include batches. At least at this stage, I believe the authors should provide a high-level statement regarding batch size to grasp the stochasticity of learners rather than give basic experiments to prove their hypothesis. \n\n- There are several hyper-parameters in RL, such as $\\epsilon$-greedy, learning rate, and update frequency. However, this article does not consider the exploration impact in online learning. Exploration and exploitation must be balanced very carefully during the decision-making research. However, this article only discussed the stochasticity of networks and its association to performance; there is no investigation of the exploration impact. To validate your hypothesis, you must offer consistent experimental results with modifying these hyper-parameters related to exploration. \n\n- The experimental setting was inconsistent in this work. In Figure 1, DQN, Rainbow, and IQN were employed as baselines, but in Figure 2, just DQN and Rainbow were used for the ablation investigation. Specifically, I do not understand why the mini-batch split and random noise to target-value experiment results were reported. I felt that these trials were not really relevant to the paper's core argument. Simply, the author preferred to emphasize stochasticity. As previously said, I believe the link between exploration capabilities and batch size warrants more investigation. \n\n[1] Yun, Chulhee, Shashank Rajput, and Suvrit Sra. \"Minibatch vs local SGD with shuffling: Tight convergence bounds and beyond.\" arXiv preprint arXiv:2110.10342 (2021).  \n\n[2] Ahn, Kwangjun, Chulhee Yun, and Suvrit Sra. \"SGD with shuffling: optimal rates without component convexity and large epoch requirements.\" Advances in Neural Information Processing Systems 33 (2020): 17526-17535.  \n\n[3] Yun, Chulhee, Suvrit Sra, and Ali Jadbabaie. \"Open Problem: Can Single-Shuffle SGD be Better than Reshuffling SGD and GD?.\" Conference on Learning Theory. PMLR, 2021.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E--kmTjCmt", "reviewer": "Reviewer_fBLk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows that the default batch-size of 32 is not optimal for typical RL algorithms (DQN, Rainbow, QR-DQN, IQN) and that decreasing the batch-size, or also increasing the batch-size, can give better human-normalized returns for some algorithms.", "summary_of_the_review": "This paper investigates a phenomenon where the increased variance due to smaller batch-sizes seems to lead to improved performance for RL methods. This is an experimental paper but there does not seem to be enough evidence for this claim either way. The authors should do more controlled experiments to study this phenomenon more systematically.", "strength_and_weaknesses": "+ The paper observes that a larger variance caused by using a smaller batch-size leads to higher returns for multi-step objectives in RL. This is a phenomenon that jives well with existing empirical results, e.g,. the fact that on-policy methods also have a huge variance in their TD loss but do seem to achieve a good return. So the effect pointed in this paper is perhaps real.\n+ The paper has extensive experiments whether this phenomenon is checked for the low-data regime, for offline learning,\n\n-- All experiments were conducted over 3 seeds, how large are the error bars for more seeds?\n-- For example, in Fig 2 left how do we control for the fact that the multi-step objective is quite different and therefore other parameters such as learning rate and weight decay also determine how well the function is learned. In this sense, I think the main premise of this study is misguided. A more reasonable hypothesis would be: for the best value of other hyper-parameters, do training runs with smaller batch-size and multi-step objective work better?\n-- “the performance boost is correlated with increased variance on both these fronts; we are dubbing this the variance double-down phenomenon” I don’t think there is enough evidence yet in this paper to ascribe to the improved returns (esp. since the error bars are quite large in Fig. 4 for 3 seeds) to improved variance. It is understood that this is a non-convex objective and therefore it is difficult to make such a claim rigorously.\n-- “It is thus possible that smaller batch sizes have a second-order effect on the learning-rate adaptation that benefits agent performance” This sentence makes sense, can the authors elaborate?\n-- The results on low-data and offline regimes seem to throw a spanner in the main hypothesis; they are quite inconclusive.\n-- “One might argue that reducing the batch size without additional training effectively mitigates overestimation, simply because each transition is trained on fewer times” — there does not seem to be any evidence in  the paper to make this claim.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject