Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This works aims at dealing with the reality gap of offline RL benchmarks, containing datasets from different domains (i.e. robotics, industrial control, finance trading, and city management) with controlled data amount.
Evaluation of some representative methods is conducted based on the proposed dataset.

strengths: 1.	Data size and quality are split into different levels, making it possible for a more detailed analysis of models.
2.	The construction of this benchmark reveals some problems of existing off-line RL methods, providing insights for future model design.

weaknesses: 1.	Comparison with other works is not detailed enough. 
2.	Motivation for constructing this dataset should be justified with more evidence.

correctness: The claims made in the paper and the supporting experiments/evaluation methods look correct to me.
clarity: I could follow, but further revision is suggested.
relation to prior work: The difference from previous contributions should be further explained and clarified.
additional feedback: 
This benchmark could support 4 selected tasks and the authors claim that in the future they will investigate more tasks, so what are the underlying rules to choose included tasks? What are the rules?
review_title: The aim of building this benchmark is clear but there are several problems.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper proposes a new benchmark dataset for batch reinforcement learning, including different simulated tasks such as industrial control, financial trading, etc. The paper show results of existing algorithms under more constrained settings such as conservative actions, limited data and offline policy selection and show that many existing batch RL algorithms does not outperform behaviour cloning or the deterministic behaviour policy.
strengths: The paper provides a set of new tasks and show extensive benchmarking results of existing algorithms. Additionally, the paper shows clearly that with offline policy selection, the performance of most batch RL algorithms degrades significantly.
weaknesses: 1. The overall novelty of the work is limited. While a set of new tasks are proposed, given the existing benchmarks such as D4RL and RL Unplugged, it is not clear what are the main properties of the newly proposed tasks that set them apart. The differences in data properties and policy selection as described in Table 1 could also be done for the existing benchmark. The only difference to previous tasks that the authors claim is its high stochasticity and non-stationarity. But the importance of these properties are not discussed or empirically shown in the paper. Additionally, offline policy selection has also been proposed in previous work (RL Unplugged).

2. The paper claims that one difference of NeoRL dataset from previous benchmark is the conservative data collection. However, the paper still trains an RL policy in the original environment and use it as the behaviour policy. How is the conservative data collection implemented in these tasks? How much does this factor affect the relative performance of different algorithms? 
correctness: See the above section.
clarity: The introduction could be improved by explaining more clearly the difference to previous benchmarks and the motivation for new tasks. There are numerous grammar and language issues in the paper and I would suggest the authors get editing help from someone with full professional proficiency in English.
relation to prior work: See the weakness section.
additional feedback: No additional feedback
review_title: More justification on the need of new tasks for benchmarking batch RL is needed

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a new benchmarking dataset for offline RL algorithms, with a focus to close the reality gap in the dataset. They point out the following reality gaps in previous offline RL benchmarkings: 1) The use of a too exploratory policy to collect the dataset, while real-world policies are usually conservative to ensure safety; 2) The size of the dataset is too large, while real-world applications only usually have very limited historical data in the magnitude of dozens; 3) Online evaluation to select the best policy, while directly deploying the policies in the real-world test environment could be dangerous and undesirable.

The proposed new benchmarking contains industrial control, financial marketing, and city energy management scenarios. It solves these gaps by 1) using three-level size of datasets of 10^2, 10^3 and 10^4 trajectoreis so there is limited data; 2) using fully or partially trained sac policies to collect the data so the dataset is not overly exploratory; 3) besides online selection, also using offline policy evaluation to select the best policy based on a leave-out test dataset. 

The paper further dose a comprehensive evaluation of some of the state-of-art offline RL methods on the proposed benchmarkings, and gives a detailed analysis of the findings. 

strengths: - Offline RL is an important research area with wide potential applications and benchmarkings can help better algorithmic development
- The evaluation experiments of state-of-the-art offline RL methods on the proposed datasets are pretty comprehensive, i.e., 7 methods on a diversity of environments and a diversity of datasets;
- This paper takes seriously the problem of off-policy evaluation for selecting the best policy which is important for really deploy offline RL method in the real world, and does a good experiment on comparing off-policy evaluation with online policy selection that demonstrates a disagreement between these two evaluation method, which can shed light on further algorithmic development of offline RL and offline policy evaluation. 
weaknesses: - The proposed gym-mujoco environment has an overlap with the D4RL dataset, e.g., the 50% sac variant seems to be identical to the medium dataset in D4RL;
- Some of the key properties of the propsoed benchmarking, e.g., limited data, conservative policy, are not very well justified (see questions in the Correctness section below)
- The writing of the paper can be fairly improved.

correctness: I have the following questions:
- Is the dataset of 10^3 or 10^4 trajectories really small? According to the appendix, all environments has a maximal trajectory length of 10^3 maginitude, so 10^3 trajectories already are 10^6 transitions, which is what D4RL's dataset size. So in this sense, only the 10^2 trajectory dataset size can be considered as "limited data". Please clarify on this.
- Can the authors add more justifications that why their way of collecting the dataset, i.e., using trained sac of 25%, 50%, 75% , 100% of the maximal performance is conservative/not overly exploratory? (especially the 25% one) 
- How is WIS run since only a dataset but not the behaviour policy is given?


clarity: The paper can be further improved in terms of the writing. Below are some examples:  
- line 166 - 168: "... to mimic these gaps ... " -> "to fill these gaps"? 
- line 170: "... to mimic the above issues ... " -> "to address such issues"?
- line 181 - 185, the paper first states "In real scenarios, the historical datasets may not contain rewards..." on line 184 it then states that "while our default datasets contain original environment rewards for fair comparisons", which confuese me since it seems to be  contradictory to the previous statement.
- line 259, 'commonly' -> 'Commonly'
- line 291, 'Although model-free methods leverage in offline RL algorithms and are easy to use' -> 'Although model-free methods leveraged in offline RL algorithms are easy to use'
- line 365 - 366 "From the normalized scores three evaluations," -> "From the normalized scores over three evaluations,"
- Please explain how average rank is computed. 
- In section 7.1.1 a converged expert sac is listed as the baseline, but its performance was not reported in table 2. 
relation to prior work: It is clearly discussed how this work differs from previous contributions.

additional feedback: All my questions are listed above.
review_title: Review of NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning

Meta-review: This paper tries close the reality gap of offline RL benchmarks. The benchmark has datasets from several simulated domains (i.e. robotics, industrial control, finance trading, and city management) with controlled data amount. They have evaluated some offline RL baselines on their datasets.

The paper is not very well-written it feels a bit rushed. It seems like the reviewers were confused about the main points of the paper as well. The comparisons with other works is not detailed enough. There are some missing citations such as RWRL [1], also claims related to the OPE are not clear to me, what actually is different here compared to DOPE or offline policy selection tasks in RL Unplugged.

Also the paper claims that the behavior policies in NeoRL are more conservative and less explorative. But I am not fully convinced with that, it is something the paper claims without any solid backings.

The reviewer's scores also point this paper towards rejection. I would recommend the author's to consider rewriting the paper with clarifications and addressing the concerns raised by the reviewers and resubmit it to a future venue.

[1] Dulac-Arnold, Gabriel, et al. "An empirical investigation of the challenges of real-world reinforcement learning." arXiv preprint arXiv:2003.11881 (2020).


Decision: Reject