Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work systematically tests a number of possible reasons for off-policy DRL methods not performing as well as they potentially could.
From this, the authors conclude that statistical overfitting is a key contributor to the poor performance.
Based on this insight, they propose a method for keeping statistical overfitting low. This method is shown to successfully reduce overfitting and improve RL performance.
strength_and_weaknesses: This is a high-quality paper.
It begins with a clear starting question that is thoroughly addressed.
The experiments performed for this are clear and compelling.
Based on the investigation, a concrete conclusion is drawn and a novel method is produced.
This method is novel and well-motivated.

The generated insights have the potential to be the foundation for further methods.
clarity,_quality,_novelty_and_reproducibility: Overall, this work is a good example of all four of these qualities.

In addition to the three causes mentioned in the abstract, "high variance" is mentioned in the introduction. However, this is not addressed to the same degree within the paper.

The authors correctly note that the other possible reasons may still contribute but are not the primary culprit.

The experiment in Section 3.2 answers a core question and is a great inclusion.

The relationship to overfitting in the offline RL setting is glossed over.

Minor Comments:
- End of Section 1: "[AVTD] attempts to counteract automatically select regularization schemes by hill-climbing" is missing words.
- Start of Section 2: The MDP is defined in terms of R, but then r is used later.
- The authors could induce overfitting to varying degrees and compare the ranking of agents by overfitting vs ranking of agents by performance.
- Section 3: "has also been used previously [to] stabilize" (missing word)
- Section 3: "bellman" -> "Bellman"
- Commas are generally missing

Questions:
- How crucial is feature normalization to the results in Figure 3?
summary_of_the_review: This is a high-quality paper. It begins with a clear starting question that is thoroughly addressed.
Based on the investigation, a concrete conclusion is drawn and a novel method is produced.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: ["Long" summary]

The paper deals with sample efficiency in deep reinforcement learning (DRL). In particular, there was a known problem in DRL with high update-to-data (UTD) ratio, addressed by several recent studies with different regularizers. The authors aim to understand why DRL usually fail with high UTD and how we can overcome this problem in general. Addressing this problem is a vital step toward data-efficient DRL. The paper first examined possible reasons that cause the failure, including data quality, non-stationarity and s distribution shift. The empirical results show that any reason alone cannot explain the failure. Therefore, the authors conjectured that the main cause is statistical overfitting.

Then, to solve the statistical overfitting the paper proposed. An algorithm was proposed, in which multiple DRL models with various regularizers (each proposed by a previous study) are simultaneously trained while sharing one replay buffer. At each step, the model with lowest validation TD error is selected to compute the policy. Experiments were conducted to show the effectiveness of the algorithm. Furthermore, some additional experiments results are shown in Sec.5 to support the design choices.

[In short]

The first contribution of the paper is the empirical results for better understanding the failure with high UTD, which are helpful and necessary for data-efficient DRL.

The second is the introduction of an algorithm to adaptively select the agent with proper regularizer to handle high UTD.
strength_and_weaknesses: [Pros]
- The paper provides important insight about why DRL fail when trying to exploit the data by high UTD ratios. This is helpful and necessary toward data-efficient DRL.
- The experimental evaluations are relatively comprehensive and well-defined.


[Cons]
- If I understand correctly, AVTD (Algorithm 1) needs to train all the off-policy agents, which means the computation cost is almost the sum of the individuals (K is the number of agents with different regularizers).
- The overall performance of AVTD (Figure 9, aggregated normalized score) is almost the same as WD=0.01 and only marginally outperforms Reset. In particular, AVTD performed the worst in Acrobot swing-up. Practically saying, why should we use AVTD since it is less computationally efficient while not significantly better than baseline methods.


[Questions]
- What does "AVTD" stand for?
- Page 13, SN, we follow the implementation of WHAT?
- Is there any possibility to theoretical investigate the high UTD failure?

clarity,_quality,_novelty_and_reproducibility: The paper was written in good quality. The ideas are addressed clearly with support from rich empirical results. While no theoretical analysis is involved, the paper provides novel and significant empirical insights.
summary_of_the_review: The paper touches an interesting and essential problem in DRL: the failure with high UTD ratio using off-policy algorithms. I appreciate the paper because it provides insightful empirical results toward understanding the failure. Meanwhile, the proposed algorithm, AVTD, which aims to overcome the failure, does not demonstrate satisfying performance though it costs more computation. I think at least the first contribution of the paper is significant to DRL community. Hopefully the authors can figure out a better way to overcome the difficulty in the near future. Overall, I lean toward an acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In the paper under review, it is shown that the TD for validation data (validation TD) correlates with the performance of the RL + regularization methods in high UTD (replay ratio) settings. 
Based on this finding, a method that selects the regularization method with the smallest validation TD from multiple regularization methods is proposed. 

strength_and_weaknesses: # Strengths:  
[Novelity] [Quality]:  
* The comparison of several regularization methods in addition to methods such as Reset, DroQ, and Spectrum normalization, was performed on several benchmark tasks. This comprehensive comparison result would be a good experiment stack. 

* Through many experiments in several settings (eg., approx. offline learning based on the tandem learning framework), it is shown that validation TD correlates with actual performance. A unifying method (AVTD) in which the regularization method is selected in accordance with its validation TD score is proposed. It is shown that the performance of this unifying method is better than that of each regularization method. This kind of unification of RL methods is very important in practice, because, currently, the success of RL methods (+ hyperparameter values) is highly task-dependent.


# Weaknesses: 
[Clarity] [Quality]:  
The paper is basically well-written, but there are typos and unclear points. 

> RL algorithm has been a important 

-> RL algorithm has been an important


> Concretely, our method, AVTD, trains several off-policy RL agents on a shared replay buffer where each agent applies a different overfitting regularize.

Since AVTD appears for the first time here, the name in its non-abbreviated form should be provided as well: our method, <non-abbreviated form> (AVTD)

> In theory, these off-policy algorithms can be made very sample efficient by making sure the Q-network fits the current replay buffer well,

needs reference. 

> Our analysis analyzes a standard SAC agent in the high UTD regime. 

-> We analyze a standard SAC agent in the high UTD regime. 

> we plot the training and validation TD errors as well as the standardized TD gap (STD gap),

Why introduce a Standardized TD gap? (Why not just use validation TD?) Validation TD also correlates with performance, and the AVTD algorithm uses validation TD. So I don't see any motivation to introduce Standardized TD gap. 

> Figure 2

Why not merge it with Figure 1 and arrange the figures horizontally? In the current version, the legend and some captions are duplicated and consume space. 


> We would also highlight that the regularizers that achieve the lowest validation TD error offline are usually one of the top performing methods online (Figure 5)

It would help to understand the correlations more easily if there were figures summarizing the results shown in Figures 5 and 12. 
e.g., how many times did the method with the lowest TD error rank in 1st place in actual performance? and how many times did it rank in 2nd or lower places?


> The solution of (Nikishin et al., 2022) which

-> The solution of Nikishin et al. (2022) which

> (Khadka et al., 2019) also

-> Khadka et al. (2019) also


> Algorithm 1. 10: After every nepisode episodes, collect a heldout trajectory and add to Dheldout with the same action selection strategy above for D.

What is the size of the Heldout dataset \mathcal{D}_{\text{heldout}}? If the size is large, this impairs overall sample efficiency since we need to consume non-trivial numbers of samples for validation.



> Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs/1801.01290, 2018a. URL http://arxiv.org/abs/1801.01290.

> Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861–1870. PMLR, 2018b.

> Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. The impact of non-stationarity on generalisation in deep reinforcement learning. arXiv preprint arXiv:2006.05826, 2020a.

> Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. arXiv preprint arXiv:2006.05826, 2020b.

> Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, 2019a.

> Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052–2062, 2019b.

> Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Modelbased policy optimization. Advances in Neural Information Processing Systems, 32, 2019a.

> Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Modelbased policy optimization. In Advances in Neural Information Processing Systems, pp. 12498–12509, 2019b.

Duplicate citations.

> Spectral normalization (SN). For spectral normalization, we follow the implementation of

-> Spectral normalization (SN). For spectral normalization, we follow the implementation of Gogianu et al. (2021)? Bjorck et al. (2021)?? or Miyato et al. (2018)???

clarity,_quality,_novelty_and_reproducibility: See my comments on strengths and weaknesses.

summary_of_the_review: I am leaning toward acceptance.
The main strength of this paper is proposing a framework for unifying multiple RL methods (more precisely, regularization methods) and performing an empirical analysis of the framework.
Currently, the success or failure of RL methods (+ hyperparameter settings) is highly task-dependent. Thus, the framework to unify individual methods is very important. 

The presentation of the paper seems to be not very much polished. 
More careful modification of the presentation would further improve the paper's quality.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper analyzes why off-policy DRL methods do not work well in high UTD regimes and proposes a solution to reduce overfitting. While the reviewers question the applicability of the proposed method because it is computationally expensive and provides very few improvements for the extra compute, they also agree that the analysis in the paper is very interesting and useful to the community. It is for this reason, that I recommend accepting this paper.

Zhixuan raised a lot of valid questions and the authors revised the paper to address most of their concerns. I recommend the authors add the extra clarifications and experiments that are still missing in the revision to the final version of the paper.
justification_for_why_not_higher_score: While the paper is definitely interesting to be published, it is not very clear in terms of its definitions and the usage of classical terms like statistical overfitting. I would not highlight this paper as a spotlight or oral.
justification_for_why_not_lower_score: The paper is not perfect. But the analysis in the paper is very useful to the community and hence worth publishing.
Decision: Accept: poster