Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper shows that the default batch-size of 32 is not optimal for typical RL algorithms (DQN, Rainbow, QR-DQN, IQN) and that decreasing the batch-size, or also increasing the batch-size, can give better human-normalized returns for some algorithms.
strength_and_weaknesses: + The paper observes that a larger variance caused by using a smaller batch-size leads to higher returns for multi-step objectives in RL. This is a phenomenon that jives well with existing empirical results, e.g,. the fact that on-policy methods also have a huge variance in their TD loss but do seem to achieve a good return. So the effect pointed in this paper is perhaps real.
+ The paper has extensive experiments whether this phenomenon is checked for the low-data regime, for offline learning,

-- All experiments were conducted over 3 seeds, how large are the error bars for more seeds?
-- For example, in Fig 2 left how do we control for the fact that the multi-step objective is quite different and therefore other parameters such as learning rate and weight decay also determine how well the function is learned. In this sense, I think the main premise of this study is misguided. A more reasonable hypothesis would be: for the best value of other hyper-parameters, do training runs with smaller batch-size and multi-step objective work better?
-- “the performance boost is correlated with increased variance on both these fronts; we are dubbing this the variance double-down phenomenon” I don’t think there is enough evidence yet in this paper to ascribe to the improved returns (esp. since the error bars are quite large in Fig. 4 for 3 seeds) to improved variance. It is understood that this is a non-convex objective and therefore it is difficult to make such a claim rigorously.
-- “It is thus possible that smaller batch sizes have a second-order effect on the learning-rate adaptation that benefits agent performance” This sentence makes sense, can the authors elaborate?
-- The results on low-data and offline regimes seem to throw a spanner in the main hypothesis; they are quite inconclusive.
-- “One might argue that reducing the batch size without additional training effectively mitigates overestimation, simply because each transition is trained on fewer times” — there does not seem to be any evidence in  the paper to make this claim.

clarity,_quality,_novelty_and_reproducibility: The paper is very well-written and very clear. This paper is original.
summary_of_the_review: This paper investigates a phenomenon where the increased variance due to smaller batch-sizes seems to lead to improved performance for RL methods. This is an experimental paper but there does not seem to be enough evidence for this claim either way. The authors should do more controlled experiments to study this phenomenon more systematically.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: ### Summary

&nbsp;&nbsp; This research examines the effect of batch size on performance in multi-step reinforcement learning. In supervised-learning experiments, several researchers have found that the variance of neural networks during training has a significant effect on learnability and convergence, since the neural networks' stochasticity pushes them to global optimality regions. Typically, non-linearity properties of networks, such as activation functions and skip-connection, direct a large number of inflection points to loss surfaces with respect to parameter spaces. Various researchers have conducted technical and empirical studies to demonstrate loss surface of neural networks and escape of local optimal points by utilizing stochasticity of learning. 

In contrast, in deep reinforcement learning, there are several hyper-parameters, hence the influence of batch size is rarely investigated. In addition, the multi-step approach that assesses value-functions as the sum of those on the last visited states has been extensively employed for empirical performance enhancement. However, there was no actual research demonstrating how these characteristics impact agents' performance. In this context, this work analyzes the impact of batch size and the number of steps of value-function on both online and offline reinforcement learnings. This study asserts that increasing variances by decreasing batch size enhances overall performance, and the bigger the number of steps that are likewise predicted to have a high variance does not ensure excellent performance. 

In order to make a fair comparison, this research presented the Dopamine framework [1] that built the ALE framework and conducted extensive experiments to substantiate their hypotheses. There are a few results that do not accord with their claims, but in general, as batch size drops, network performance and variances of loss and gradient values rise. Instead, there are no global trends for the amount of steps, but there is a sweet spot for performance. For instance, the majority of studies stated that three or five is the optimal number of multi-steps like Dopamine framework.

[1] Machado, Marlos C., et al. "Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents." Journal of Artificial Intelligence Research 61 (2018): 523-562.
strength_and_weaknesses: ### Strength

- Numerous experimental findings altering hyper-parameters such as batch sizes and step-sizes of multi-step value functions are presented in this study. 
- This article shown that ablation research varies each component of deep value-based RL. 
- By changing the batch size, they have repeatedly shown that agents get higher results in the ALE framework. 
- Motivated by the stochasticity of networks in supervised learning, they studied the gradient values of networks based on the number of batch-sizes.

### Weakness

- This study provides neither a technical nor a theoretical explanation of batch size and multi-step in deep reinforcement learning. I was unable of digesting a high-level summary of this research. Specifically, I disagreed with the author's statement that multi-step reinforcement learning should be examined independently. Numerous empirical research have already demonstrated that the multi-level value function does not exhibit a consistent trend across diverse situations. As the authors noted out in their paper, past research has, as a rule of thumb, set the value of multi-level performance to three. At this time, I was unable to understand why this work manipulates and reveals the effect of multi-levels and its impact on batch-size by up to 15. 

- Numerous theoretical works [1, 2, 3] have examined how the stochasticity of gradients is obtained from batch components. These studies also asserted that random mini-batches (shuffling) demonstrate networks that are more generalized than static mini-batches. I guess that these research are incompatible with online learning, but I believe they might give insightful perspectives on the exploration of batch size for off-line learning. In particular, in theoretical RL, the majority of publications might be demonstrating that such an algorithm can create a transitions matrix from provided datasets that include batches. At least at this stage, I believe the authors should provide a high-level statement regarding batch size to grasp the stochasticity of learners rather than give basic experiments to prove their hypothesis. 

- There are several hyper-parameters in RL, such as $\epsilon$-greedy, learning rate, and update frequency. However, this article does not consider the exploration impact in online learning. Exploration and exploitation must be balanced very carefully during the decision-making research. However, this article only discussed the stochasticity of networks and its association to performance; there is no investigation of the exploration impact. To validate your hypothesis, you must offer consistent experimental results with modifying these hyper-parameters related to exploration. 

- The experimental setting was inconsistent in this work. In Figure 1, DQN, Rainbow, and IQN were employed as baselines, but in Figure 2, just DQN and Rainbow were used for the ablation investigation. Specifically, I do not understand why the mini-batch split and random noise to target-value experiment results were reported. I felt that these trials were not really relevant to the paper's core argument. Simply, the author preferred to emphasize stochasticity. As previously said, I believe the link between exploration capabilities and batch size warrants more investigation. 

[1] Yun, Chulhee, Shashank Rajput, and Suvrit Sra. "Minibatch vs local SGD with shuffling: Tight convergence bounds and beyond." arXiv preprint arXiv:2110.10342 (2021).  

[2] Ahn, Kwangjun, Chulhee Yun, and Suvrit Sra. "SGD with shuffling: optimal rates without component convexity and large epoch requirements." Advances in Neural Information Processing Systems 33 (2020): 17526-17535.  

[3] Yun, Chulhee, Suvrit Sra, and Ali Jadbabaie. "Open Problem: Can Single-Shuffle SGD be Better than Reshuffling SGD and GD?." Conference on Learning Theory. PMLR, 2021.  
clarity,_quality,_novelty_and_reproducibility: **Clarity** : This study gives experimental findings on the ALE framework. If the authors wish to assert that a small number of batches increases performance by increasing stochasticity, they should examine more environment like Mujoco. If not, at least, the ALE-like Procgen should be chosen to support this notion. As just one setting employs ALE, its utilization is marginal. 

**Quality** : This document appears to lack a logical structure. In addition, this study does not present the state-of-the-art performance on ALE, nor does it present consistent empirical evidence for substantially direct performance increase. Simply said, adjusting mini-batches has a minor effect on performance, however not all algorithms adhere to this hypothesis. If the authors give an examination of algorithms by modifying the batch size in RL, they must develop more high-level technical statements that enable comprehension of the exploration and exploitation balance, since this is a fundamental foundation for comprehending decision making areas. Otherwise, they should at least characterize the entropy of of batches for learners during training. 

**Novelty** : The authors discover that the batch size of RL has a positive impact on performance. Previously, the majority of RL articles focused on network architectures and manipulation, therefore this discovery is expected to give new axes for improving RL algorithms. However, as previously said, I felt that this research lacked sufficient results to conclude these finding in ICLR2023. 

**Reproducibility** : In the manuscript, the authors described all algorithms follow the Dopamine framework, however, I believe that it did not meet the ICLR standard for reproducting experimental results. The authors do not provide ant source-codes and checkpoints to validate this results. Specifically, due to the fact that this paper employs empirical way to reasoning, I though that providing source-code and checkpoints was crucial, however it does not. 
summary_of_the_review: Overall, the paper marginally show perfirmance enhancement using a batch-size and the number of steps in multi-step reinforcement learnings. In order to provide more general insights, this should be addressed by either improved theoretical statement or extensive experimental results across various environments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper perform an exhaustive investigation of the role of batch size and the steps used in temporal difference learning. Unlike the conventional intuition in supervised learning that increasing batch size would lead to variance reduction, thus increase the overall performance. In RL (Deep Q Learning based algorithms), decrease the batch size and increase the steps for Q-function learning would improve the performance, which is a counterintuitive finding in the literature. 
strength_and_weaknesses: **Strength**
- The authors conducted extensive experiments to show the phenomena under all kind of different settings (such as different Q-learning algorithms, DQN, Rainbow, DQN-QR, IQN). In standard Q learning settings,  the empirical experiments support the authors' counterintuitive claim. 
- The paper is well written, and the experiments conducted are easy to follow. 
- The claim is interesting, which is beneficial for empirical hyper-parameter selection. 

**Weakness**
- It is still unclear why this claim is drawn in RL, rather than standard supervised Learning settings. There is no theoretical analysis or deeper understanding why this phenomena occurs. 
- For the offline setting, why there are some counterexample that does not support your claim? 
- I personally think it would be really great if the authors could point out what perspective that reducing batch size and increase the n-step in Q learning would improve the performance, especially:
  - **Exploration**. Intuitively decreasing the batch size would increase the variance for gradient updates, which potentially increase the exploration benefits.
  - **Optimization / Estimation for Q-function**. At first I think under the offline setting it would be very easy to validate whether reducing batch size and increase n in Q learning would help optimization, but from your experiments, we just see not consistent results. Besides, I think by performing an off-policy evaluation experiments (fix $\pi$ and learning Q) would check whether these two strategy would help Q estimation. 
  - **Generalization**. There are previous work identifies that smaller batch size would possibly improve the performance of generalization. I wonder if your two strategy would also observe similar benefits under the RL settings. 

clarity,_quality,_novelty_and_reproducibility: **Quality** 

Overall in this paper, the authors studied a very interesting problem in RL; the phenomena that the authors observe is counterintuitive, which could be potentially valuable for empirical performance tuning and new perspectives of theoretical understanding of deep RL. 

**Clarity**

The authors should have a deeper discussion to clarify what perspectives lead to performance improvement, which could be more valuable for future follow-up research. 

**Originality**

As far as I know, there is no previous work pointing out the counterintuitive claim, which makes this work valuable. 
summary_of_the_review: Overall I think in this paper, the authors studied a very interesting question, and it is worth investigating. However, one of the major drawbacks of the paper is that we don't know why the claim is drawn in the RL settings, and also, there is no theoretical analysis or further understanding of the phenomenon.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the effect of batch size in multistep RL. The central finding in the paper is that reducing batch size and increasing number of steps for TD learning simultaneously can be helpful in many cases. The somewhat surprising finding is that both changes increase the variance of the loss and gradients. There was a rich discussion around this work. On the positive side, reviewers appreciated the potential impact of the study and the extra experiments done during the rebuttal stage. However, there are some pending concerns with the current work. First, the experimental protocol of keeping other hyperparameters fixed while varying batch size fixed is strange. As the reviewers pointed out, varying of learning rates and exploration hyperparameters should be in conjunction with batch size for best performance, as they affect the variance as well. In this regard, the new results e.g., Figure 37 cast somewhat of a shadow on how meaningful is the main claim of the paper, as tuning other params for higher batch sizes can also improve their performance. Second, the somewhat mixed performance in the offline setting was another concern whether the claimed observation about batch sizes needs further assumptions / restricted to certain environments and algorithms -- amongst the 3 algorithms considered, CQL is the most common in practice and the claimed trend does not hold there. Finally, some of the reviewers felt that even if the trend was highly robust (which it does not seem given the above points), a more detailed understanding of the phenomena would improve the work. I encourage the authors to focus on this feedback for a future submission.
justification_for_why_not_higher_score: Issues with experimental design and analysis of the central phenomena studied in the paper.
Decision: Reject