Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary

The paper proposes an off-policy evaluation dataset (DOPE) for various control tasks.
The authors include various baselines for off-policy evaluation and evaluate them with additional metrics to MSE, namely regret@k and rank correlation.

## Writeup

The writeup is good. It is clear what the "features" of the proposed benchmark are, and what problems the benchmark addresses that the other benchmarks didn't address before. I don't really like that in different images form  dm-control and gym environments pop up in the paper. Since these environments as self-standing were not the contribution of the paper I would remove these images, unless they are used to show a specific point. The space should be used rather to illustrate the point of the paper, that additional metrics are needed for evaluating algorithms / better benchmark is needed.

## Pros

I think that using additional evaluation metrics for off-policy are indeed necessary. The authors show that not one single metric summarizes performance over all of the metrics. This is going to allow researchers to do a more thorough analysis of offline reinforcement learning algorithms.

## Cons

Instead of using the space for "nice pictures", use it to actually support the claims of the paper, maybe shifting some figures from the appendix to the main text would be more informative. 
The paper is not making any improvement over the methods or suggesting any reason why there is such a disparity in performance between different metrics, as such, there are no deeper insights.
A large chunk of the paper is previous work.

## Comments

In general, the experimental evaluation is thorough (there are a lot of experiments in the appending with a lot of tables and figures). 

To me it is a bit questionable to which extent this data might count as high-dimensional and complicated, in comparison to real-world of robotics, this data is still quite low dimensional. It would be nice to see this dataset contain image data for learning from image input.

I am highly suspicious about the way the data is generated for the ant-maze environment. Specifically, you use expert data for ant-maze-large because DAPG didn't work. Please specify where does the expert data come from.

I didn't try to run the code, but I looked at it. I didn't find the code for the algorithms themselves. I would expect to have the training code to reproduce the experiments. With the current code I can only evaluate saved policies?

Can you comment on what kind of normalization was done on the reward function? Ideally it should be normalized across all environments in the benchmark, such that performance is comparable. 

Maybe also writing down the equations for regret@k and rank correlation would be helpful. It is sometimes simpler to understand the metric once one sees the actual equation. 


















review_title: It's a Necessary Benchmark Paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: This article proposes a benchmark of off-policy evaluation, which provides different metrics for policy ranking, evaluation and selection. Offline metrics are provided by evaluating the value function of logged data, and then evaluating absolute error, rank correlation and regret. Verify the effectiveness of different offline evaluation methods. This article provides two evaluation scenarios, one is DOPE RL unplugged, and the other is D4RL. In the experiment, the author verified the benchmark proposed in this article in the MuJoCo environment to evaluate the effectiveness of different offline evaluation methods.

The paper’s key strengths:

1.	I think the research direction of this article is very meaningful, because many scenarios in the real world only have offline data, which requires a simulated environment or other offline evaluation methods for strategy evaluation and model selection. This is very crucial for model improvement and evaluation. And the author proposed absolute error and rank correlation in the benchmark to evaluate the strategy. I think this is very innovative.
2.	The author has applied a large number of offline evaluation methods in the experiment to verify in different metrics, and in the appendix has conducted experiments and verifications for many MuJoCo scenarios. I think the experimental results are very rich and very confirmatory.

The paper’s key weaknesses:
1.	The biggest problem with this article is that the contribution of the article is insufficient and lacks originality. This article proposes a benchmark for off-policy evaluation and verifies different OPE methods, but this article does not compare with other similar benchmarks to verify whether the benchmark proposed in this article is effective.
2.	In the experimental part, this paper verifies different metrics for different OPE methods. However, in Figure 4 and Figure 5, the different methods in the two sets of benchmarks proposed in this article are quite different in different OPE methods. I hope the author can give some comments on the differences between the two sets of evaluation methods.
3.	The author uses the value function in formula 1 to estimate the effect of the strategy. I doubt this method. Because of the attenuation factor here, I think it has an impact on the final value calculated by different methods. Because bellman equation is only an estimate of the value of a certain state, not an absolute strategy benefit, I hope the author will give some explanations here.
review_title: The review of benchmarks for deep off-policy evaluation

final_score: rating: 7: Good paper, accept
review_content: 1. Significance
- I view this paper has high significance level. As more and more OPE methods arise, papers rely on different datasets and metrics, which makes them hard to compare, and especially see the "success" and "failure" mode of this method. We need a benchmark dataset on this domain and this paper moves an important step towards this by providing standardized logged dataset, and various policies in different challenging, complex domains.

2. Novelty
- There is a very similar work by Voloshin et.al (2019), which gives the empirical evaluation of popular OPE methods across various domains. Though not providing the logged data, they give the logging policy to logged the data, I am concerned about the novelty of this work compared with Voloshin et.al (2019). It would be great if the author could discuss the novelty/improvement over that work.

3. Quality & Clarity
- This paper has a great structure, metrics/choice of datasets/design of policies are well motivated and clearly stated, it is a very easy to read paper.

- This work gives a clear description of the logged datasets/policies/and the performance of various off-policy method. However, I feel it would be great if the authors could also think, and if possible make the following discussions along with this paper:

(1). it would be great if the authors could give some statistics to measure the property of the datasets, such as a measure of the missing support of logging policy, how difference are the logging policy and target policy?

(2). I know the authors discuss some properties of the environment in words, such as sparse/dense rewards, stochasticity of the environment, etc. It would be great to summarize these environment info into a single table.

(3). A very important class  of estimators in OPE is these kind of interpolated estimators between value-based and weighting-based, which contains a trade-off of bias-variance, such as the work by Phillip and Emma (2016). It also shows in Voloshin et.al (2019)'s empirical evaluations and shows very promising performance. However, it is missing in this paper and it would be excited to see how it works in the D4RL tasks.

(4). A further extension from (1) is that the authors did a great job in doing lots of empirical evaluations in various domains. I saw aggregated performance in main paper and individual task performance is listed in Appendix. It would be great to classify them into groups which depends on  the MDP property, the metric measuring difference of logging/target policy, and etc... This will definitely give some further directions like which method perform well in which setting. This is also shows in Voloshin et.al (2019)'s paper but kind of missing in this work.

(5). One difference compared with Voloshin et.al (2019) is that it seems in the logged dataset, they know the propensity. However, I feel the dataset released along the paper does not have propensity. Will is possible to also add some datasets with propensity included?

Overall, I feel this paper addressing an important problem, however, I feel there is still room for improvement to make them have more impact. I am willing to adjust my score if the authors address some of these issues. 

Ref: 
1.  Philip S. Thomas, Emma Brunskill  Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning.
2. Cameron Voloshin, Hoang M. Le, Nan Jiang, Yisong Yue Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning.

review_title: This paper gives a standardized benchmark for off-policy evaluation, which contains fixed logged datasets and various target policies to cover wide range of challenges.

final_score: rating: 7: Good paper, accept
review_content: This work describes a set of benchmarks for offline policy evaluation and selection.  The area is important and well-motivated in the introduction.  

The work builds upon existing benchmark tasks for offline RL.  However, the increment is meaningful.  By providing the triad of data, trained policies, and online evaluation results, authors have isolated the offline evaluation component, controlled for ancillary aspects, and facilitated experimentation.  In other words, experiments are both easier and more useful.

The work as presented makes little attempt to connect with representation learning which is the focus of the ICLR conference.  It could be improved in this regard, if authors make connections between estimation and representation (e.g., how are model-based estimators related to or employing representation learning?   what about model-free estimators that have to fit a propensity model to the behaviour policy?  how, if at all, do the baselines described in section 5.1 utilize representation learning, or how could they be improved via representation learning techniques?)

The name of the work is highly unfortunate.  There is nothing deep about the benchmark in the representation learning sense, and the use of the term deep contributes to the general degradation of the term into a meaningless yet popular adjective.  Unfortunately, les jeux sons faits ... do better next time!


review_title: Great stuff, I plan on using this myself, but no connection to representation learning as presented

Meta-review: This paper proposes benchmark tasks for offline policy evaluation.  The proposed benchmark tasks evaluate the policy with batch data with respect to three metrics, including the standard mean squared error. The paper also evaluate several baseline offline reinforcement learning methods with the benchmark tasks, which will serve as standard baselines.  All of the reviewers are in favor of the paper.
Decision: Accept (Poster)