Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper provides a unified way to provide robust statistics in evaluating RL algorithms in experimental research. Though I don't believe the metrics are particularly novel, I believe this work would be useful to the broader community and was evaluated on a number of environments. I do have a few concerns, however, about experimental performance per environment being omitted from both the main paper and the appendix.

Comments: 

+ I think this is a valuable work and the ideas/metrics are useful, though I'm not sure I would call them novel (CVar and the like have been seen before).  I think the value comes in the unification of the metrics to give more robust pictures of algorithmic performance. 
+ The details of all of these evaluations and individual performance should be provided in the appendix, however, it seems only MuJoco curves were included. Moreover, it says that a blackbox optimizer was used to find hyperparameters, but these hyperparameters were not provided in the appendix or anywhere else as far as I can tell. I think it's important for a paper which recommends evaluation methodology in particular to be more explicit regarding all details within the appendix. I hope to see additional details in future revisions -- including per-environment performance.
+ I believe clustering results across environments can be potentially misleading. Say that we have an environment where the algorithm always fails but is very consistent and an environment where it excels. These are blended together in the current Figures. While it requires more space, I believe it is important to separate these two. I am concerned that a recommendation paper like this one will set a precedent for only including the combined metrics of algorithmic performance across environments, masking effects.  I would suggest splitting out results per environment as well and pointing out particular cross-environment phenomena. 

There is a missing discussion of prior work on statistical testing of RL evaluation:
+ Colas, Cédric, Olivier Sigaud, and Pierre-Yves Oudeyer. "A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms." arXiv preprint arXiv:1904.06979 (2019).
+ Colas, Cédric, Olivier Sigaud, and Pierre-Yves Oudeyer. "How many random seeds? statistical power analysis in deep reinforcement learning experiments." arXiv preprint arXiv:1806.08295 (2018).

EDIT: Score boosted after significant updates to the paper.

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: *Summary*

Authors proposed a variety of metrics to measure the reliability of an RL algorithm. Mainly looking at Dispersion and Risk across time and runs while learning, and also in the evaluation phase. 
Authors have further proposed ranking and also confidence intervals based on bootstrapped samples. They also compared the famous continuous control and discrete actions algorithms on Atari and OpenAI Gym on the metrics they defined.

*Decision*

I believe the paper is discussing a very important issue, and some possible solutions to it, even if not perfect it's an important step toward paying more attention to maybe similar metrics. I am in favor of the paper in general,  but I have some concerns.

1. My main concern is that why authors think that community will adopt these metrics and report them? I like how authors have proposed different metrics, but having one or two easy to compute metric is much more likely to be adopted, than 6 different metrics, which I’m not sure how easy it is to use the python package? It’s of main importance, because if community don’t use these metrics in the future, the contribution of the paper is minimal. 

2. There is no question of the importance of reliability of RL algorithms, but we need to be careful that RL algorithms are not optimizing for metics like CVaR, so maybe a better learning algorithm (in the sense of expectation learning) might not have better reliability metrics because it is not the main objective.  
So following this, how would authors think their metrics can be used to design a more reliable algorithms? For example there is good literature on CVaR learning for safe policies. Do you think there exists a proxy for metrics you introduced that can be used to for the objective of the optimization?

3. Another main concern is the effect of exploration strategy: All these metrics can be highly affected  by different exploration strategy in different environments. For example if an environment has a chain like structure, then given the exploration strategy you may have an extremely high CVaR or IQR. How do authors think they can strip off this effect? (Running all algorithm with the same exploration strategy is not sufficient, since the interplay of learning algorithm and exploration may be important)

4. Generalizability: How do authors think these metrics are generalizable. For example if algorithm A has better metrics than algorithm B on open AI Gym task for continuous control, how much we expect the same ranking applies while learning on a new environment. I am asking this, because to me, some of these metrics are very environment dependent, and being reliable in some environments may not imply reliability in other environments.


*Note*:
Code and modularity of it: The main contribution of this paper will be shown when other researchers start using it and report the metric, if the code is hard to use, the contribution of the paper is hard to be significant. 

==== Post Rebuttal ====
Thanks for the responses authors posted, I think there is a good chance that the community will benefit from this experimental metrics in the future, so I increase my rating to accept.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors study an important problem in the area of reinforcement learning (RL). Specifically, the authors focus on how to evaluate the reliability of RL algorithms, in particular of the deep RL algorithms. The paper is well motivated by providing convincing justification of evaluating the RL algorithms properly. In particular, the authors define seven specific evaluation metrics, including 'Dispersion across Time (DT): IQR across Time', 'Short-term Risk across Time (SRT): CVaR on Differences', 'Long-term Risk across Time (LRT): CVaR on Drawdown', 'Dispersion across Runs (DR): IQR across Runs', 'Risk across Runs (RR): CVaR across Runs', 'Dispersion across Fixed-Policy Rollouts (DF): IQR across Rollouts' and 'Risk across Fixed-Policy Rollouts (RF): CVaR across Rollouts', from a two-dimension analysis shown in Table 1.

Moreover, the authors apply the proposed evaluation metrics to some typical RL algorithms and environments, and provide some insightful discussions and analysis.

Overall, the paper is well presented though it is somehow different from a typical technical paper.

review_title: Official Blind Review #3

Meta-review: Main content:

This paper provides a unified way to provide robust statistics in evaluating the reliability of RL algorithms, especially deep RL algorithms. Though the metrics are not particularly novel, the investigation should be useful to the broader community as it compares seven specific evaluation metrics, including 'Dispersion across Time (DT): IQR across Time', 'Short-term Risk across Time (SRT): CVaR on Differences', 'Long-term Risk across Time (LRT): CVaR on Drawdown', 'Dispersion across Runs (DR): IQR across Runs', 'Risk across Runs (RR): CVaR across Runs', 'Dispersion across Fixed-Policy Rollouts (DF): IQR across Rollouts' and 'Risk across Fixed-Policy Rollouts (RF): CVaR across Rollouts'. The paper further proposed ranking and also confidence intervals based on bootstrapped samples, and compared against continuous control and discrete actions algorithms on Atari and OpenAI Gym.

--

Discussion:

The reviews clearly agree on accepting the paper, with a weak accept coming from a reviewer who does not know much about this subarea. Comments are mostly just directed at clarifications and completeness of description, which the authors have addressed.

--

Recommendation and justification:

This paper should be accepted due to its useful contributions toward doing a better job of measuring performance of RL.
Decision: Accept (Spotlight)