Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
In this paper a test suite of data sets and corresponding benchmarks for offline reinforcement learning is introduced.
Several existing RL benchmarks are used, the results of several algorithms are presented.
The authors claim that the benchmarks were specifically designed for the offline setting and are guided by the key properties of datasets in real-world applications of offline RL.

Strong points:
The present paper has already been cited and the benchmarks suite has already been used by other publications. Obviously there is a need for offline RL test suites.

Weak points:
The authors' claim that such a benchmark for offline RL should "be composed of tasks that reflect challenges in real-world applications of data-driven RL" is only partially met by the paper in its present form. The area of robotics, with deterministic dynamics, is comparatively well represented, but there is no real, industrial application. In particular it seems that so far no benchmark has been included that has the ambition to have the characteristics and complexity of a real application. 

Recommendation:
On the one hand, the really realistic benchmarks are missing, so that a publication seems premature. On the other hand, the current status is already used by the research community, since there seems to be no test suite for offline RL apart from "RL unplugged: Benchmarks for offline reinforcement learning". I therefore recommend to accept the paper.

Questions:
To what extent are the current benchmarks stochastic?
Are there bi- or multimodal transition probabilities?

Additional feedback with the aim to improve the paper:
In Table 2 and Table 3 average results are reported over only 3 random seeds. This seems to me to be clearly too little, especially since the policy performance of Q-function based algorithms often fluctuates strongly. Since no uncertainties, e.g. in the form of standard error, are given, the reliability of the results cannot be assessed.
The way policies are selected before they are tested should be described more clearly. My impression was that in each case the policy is used that results for a considered algorithm and random seed after 500K training iterations or gradient steps. Since different algorithms require different computational efforts this approach does not seem to be in the sense of a real-world application. In most cases there should be the willingness to use much more computational effort for especially good policies. According to the motto: computing power is cheap, data is expensive. 

I like the formulation “Effective offline RL algorithms must handle […] data collected via processes that may not be representable by the chosen policy class.“ This expresses the, in my opinion, correct view of the real situation well, while the assumption that there is a "behavior policy" that generated the data is not true in general. It may have been different people at different times who performed the actions while the data set was recorded.

Please check the bibliography for accidental lower case, like „markov“, „adobeindoornav“


----------------------------------
(Dec 3) Taking into account the other reviews, the authors' responses and the changes made by the authors, as well as the extensive and controversial discussion, I rate the paper still with a score of 6.

review_title: Premature or high time?

final_score: rating: 6: Marginally above acceptance threshold
review_content: D4RL: Datasets for Deep Data-Driven Reinforcement Learning
review:

summarization:

In this paper, the authors consider the problems of offline reinforcement learning problems,
 and has a focus of dataset and shareable code base.
While no novel algorithms are proposed in this project,
a systematic evaluation of existing algorithms (offline RL algorithms) is proposed.

Pros:
1. Offline RL is a hot topic this year, 
with a lot of papers exploring efficient and robust ways to utilize offline collected data.
This paper, while using simulated data, 
provides a general platform to benchmark these algorithms.

2. The project provides a comprehensive evaluation and discussion on existing considerations in offline RL.
It provides several interesting directions in offline RL.

3. The paper is well-written.
It is very clear what the purpose of the project is,
and it is very clear why the authors make the dataset in the way they did.

Cons:

1. The dataset is mostly simulated.
While I understand the difficulty of collection real-data, 
it does raise some concerns that a simulated dataset can be generated by researchers themselves.

Summary:
While the dataset is simulated,
I do think there’s value in the dataset and the shared code-base to facilitate recent progress in offline RL research.

review_title: Review from reviewer 3

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a standardized benchmark for offline RL research. The data collection is well motivated from real world scenarios covering many important design factors. I really appreciate that human demonstration and hand-crafted controllers are also included. The evaluation protocol and the API looks clear and easy to use. The benchmark of existing methods is thorough and provides many useful insights. I believe this work will have a high impact on the offline RL community. I can expect this benchmark will be used by many papers in the future and will function as the starting point for many offline RL research.

However, I notice that this dataset may not be accessible for underrepresented groups. I therefore vote to reject. As the authors note in the paper, each task consists of a dataset for training and a simulator for evaluation. In my understanding, half of the six tasks (Maze2D, AntMaze, Gym-mujoco) depend heavily on the MuJoCo simulator, which is a commercial software and is not free even for academic use. A personal MuJoCo license costs 500 USD per year. I am concerned that MuJoCo is not accessible for most underrepresented researchers. It is not clear when MuJoCo becomes a dominating benchmark for online RL research, though there are indeed free, open-sourced alternatives, e.g. PyBullet (https://github.com/bulletphysics/bullet3). In online RL, we need the simulator for training. One reason MuJoCo becomes popular may be because it's more stable and faster than PyBullet. However, in offline RL, a simulator is used only for evaluation not for training. So the high reliability of MuJoCo may no longer be so necessary. I therefore view offline RL as a good opportunity for the community to get rid of commercial simulation softwares, making RL research more accessible for underrepresented groups. If accepted, this paper will indeed greatly promote the use of MuJoCo given its potential high impact, making RL more privileged.  

Overall, I really enjoy reading the paper and am glad to see a standardized benchmark for offline RL. I am happy to raise my score if the accessibility issue is addressed, e.g., by using PyBullet as the physical engine.

====================

(Nov 24) I appreciate the effort put into the Bullet reimplementation and therefore increase my score from 3 to 6.

review_title: Concerns about accessibility for underrepresented groups 

final_score: rating: 2: Strong rejection
review_content: This paper offers a new set of challenges for batch reinforcement learning coupled with a set of benchmarks, including autonomous robotics and driving domains. All domains also come with simulation environments.

While the dataset provided by the authors may be a useful resource for researchers in offline-RL, this paper does not introduce any new or novel ideas. Overall the main contribution of this work is the gathering of offline data in one place, reducing the time needed for other researchers to do so. It does not seem as if the authors did any non-trivial work other than annotating the data. Furthermore, there have been many previous work which have already collected offline datasets as part of their work.

I do not underestimate the importance the authors' hard work, nor the importance of the provided datasets to the RL community. Nevertheless, I do not believe this work should be published in a high-end conference without presenting any ideas that are not trivial or known to other researches. 

The authors propose to use the following design factors in their offline datasets: narrow distributions, multi-task data, sparse rewards, suboptimal data, and partially observable policies. While these factors may indeed be good for testing offline-RL algorithms, they do not provide a complete picture of real world datasets. Real datasets present many more real-world problems, including:
1. Non-stationary data. Many real datasets are non-stationary. The non-stationary behavior could be mimicked or simulated from real behavior.
2. Real policies. Real datasets don't involve policies that were trained by RL agents. The authors could create datasets that are constructed by real human beings (for example, humans playing atari games, with mixed or different expertise). In a controlled setup, the datasets could be constructed so that policies are categorized (e.g., "level of non Markovianess`"). 
3. Causal structures. Real policies may act according to some causal structure in the background that is not necessarily known.
4. Reduction from real datasets. One could collect or use large amounts of high quality datasets from the real world and add certain corruptions to the data as to lower its quality (e.g., removing certain trajectories). If the initial data is of high quality, the corrupted data could be controlled well.
5. Changing and/or very large action sets. Real world datasets have changing and large datasets. As an example, consider ad placement, or text based tasks.
6. Non-robotic environments, including games but also real world problems.

If the authors bring together datasets that generate original ideas that have never been previously explored, then I believe it's more likely that this paper could be accepted in future venues.




review_title: A collection of offline datasets

Decision: Reject