D4RL: Datasets for Deep Data-Driven Reinforcement Learning
Authors: Justin Fu
Aviral Kumar
Ofir Nachum
George Tucker
Sergey Levine
Venue:  (2021)
Keywords: reinforcement learning
deep learning
benchmarks

Abstract: The offline reinforcement learning (RL) problem, also known as batch RL, refers to the setting where a policy must be learned from a static dataset, without additional online data collection. This setting is compelling as it potentially allows RL methods to take advantage of large, pre-collected datasets, much like how the rise of large datasets has fueled results in supervised learning in recent years. However, existing online RL benchmarks are not tailored towards the offline setting, making progress in offline RL difficult to measure. In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. Examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multi-objective datasets where an agent can perform different tasks in the same environment, and datasets consisting of a mixtures of policies. To facilitate research, we release our benchmark tasks and datasets with a comprehensive evaluation of existing algorithms and an evaluation protocol together with an open-source codebase. We hope that our benchmark will focus research effort on methods that drive improvements not just on simulated tasks, but ultimately on the kinds of real-world problems where offline RL will have the largest impact. 

Under review as a conference paper at ICLR 2021
D4RL: DATASETS FOR DEEP
REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The offline reinforcement learning (RL) setting (also known full batch RL). where policy is learned from a static dataset, is compelling as progress enables RL. methods take advantage of large, previously-collectee
datasets, much like how the rise of large datasets has fueled results in supervised learning. However, existing online RL benchmarks are not tailored towards the offline setting and ex- isting offline RL benchmarks are restricted to data generated by partially-trained agents, making progress in offline RL difficult to measure. In this work, we in- troduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RI. With fo- cus dataset collection, examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multitask datasets where agent performs different tasks the same environment, and datasets collected with mixtures of policies. By moving beyond simple benchmark tasks and data collected by partially-tran -rrinne RL agents, we reveal important and unappreciated deficiencies of existing algorithms. To facilitate research, we have released our benchmark tasks and datasets with a comprehensive evaluation of existing algo- rithms, evaluation protocol, and open-source examples. This serves com- mon starting point for the community to identify shortcomings in existing offline RL methods and collaborative route for progress in this emerging area.
INTRODUCTION
Impressive progress across a range of machine learning applica- tions has been driven by high-capacity neural network models with large, diverse training datasets (Goodfellow et al.) 2016). While reinforcement learning (RL) algorithms have also benefited from deep learning (Mnih et al. 2015), active data collection is typi- cally required for these algorithms succeed, limiting the extent to which large. previously-collected datasets can be leveraged. Of- fline RL (Lange al. 2012) (also known full batch RL), where agents learn from previously-collected datasets, provides a bridge between RL and supervised learning. The promise of offline RL is leveraging large, previously-collected datasets in the context of sequential decision making, where reward-driven learning can pro- duce policies that reason over temporally extended horizons. This could have profound implications for a range of application domains, such as robotics, autonomous driving, and healthcare.
Figure 1: A selection of pro- posed benchmark tasks.
Current offline RL methods have not fulfilled this promise yet. While recent work has investigated technical reasons for this (Fujimoto et al. 2018a Kumar al. 2019 Wu al. 2019), a major challenge in addressing these issues has been the lack of standard evaluation benchmarks. Ideally, such benchmark should: a) be composed of tasks that reflect challenges in real-world applications of data-driven RL, b) be widely accessible for researchers and define clear evaluation protocols for
Website with code, examples, tasks, and data is available https:: //s tes tes. le. com com/views / d4r1 ancnymous/

---

Under review as a conference paper at ICLR 2021
reproducibility, and e) contain range of difficulty to differentiate between algorithms, especially challenges particular to the offline RL setting.
Most recent works (Fujimoto al., 20188; Wu et al. 2019 Kumar et al. 2019 Peng et al. 2019; Agarwal et al., 2019b) use existing online RL benchmark domains and data collected from training runs of online RL methods. However, these benchmarks were not designed with offiin RL in mind and such datasets do not reflect the heterogenous nature of data collected in practice. [u al. (2019) find that existing benchmark datasets are not sufficient differentiate between simple baseline approaches and recently proposed algorithms. Furthermore, the aforementioned works do not propose a standard evaluation protocol, which makes comparing methods challenging.
Why simulated environments? While relying on (existing) real-world datasets is appealing. evalu- ating candidate policy is challenging because it weights actions differently than the data collection and may take actions that were not collected. Thus, evaluating candidate policy requires either col- lecting additional data from the real-world system, which is hard to standardize and make broadly available, or employing off-policy evaluation, which is not yet reliable enough (e.g., the NeurlPS 2017 Criteo Ad Placement Challenge used off-policy evaluation, however, in spite of an unprece- dentedly large dataset because of the variance in the estimator, top entries were not statistically distinguishable from the baseline). Both options are at odds with a widely-accessible and repro- ducible benchmark. As a compromise, we use high-quality simulations that have been battle-tested in prior domain-speciin work, such as in robotics and autonomous driving. These simulators allow researchers to evaluate candidate policies accurately.
Our primary contribution is the introduction of Datasets for Deep Data-Driven Reinforcement Learning (D4RL): suite of tasks and datasets for benchmarking progress in offline RL. We fo- cus our design around tasks and data collection strategies that exercise dimensions of the of- fline RL problem likely to occur in practical applications. such as partial observability, passively logged data, or human demonstrations. To serve as a reference, we benchmark state-of-the- art offline RL algorithms (Haarnoja et al. 20186] Kumar et al.] 2019; Wu al.) 2019 Agar- wal et al., 2019b, Fujimoto et al. 201 8al Nachum et al. 2019 Peng et al.) 2019] Kumar et al.) 2020) and provide reference implementations as a starting point for future work. While previ- ous studies (e.g.. (Wu et al. 2019)) found that all methods including simple baselines performed well on the limited set of tasks used in prior work, we find that most algorithms struggle to per- form well on tasks with properties crucial to worldw applications such passively logged data, narrow data distributions, and limited human demonstrations. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, reveal important and un- appreciated deficiencies of existing algorithms. To facilitate adoption, we provide an easy-to-use API for tasks, datasets, and collection of benchmark implementations of existing algorithms (https: //sites. google..ooooooooooooooooooo... botom/view///// anri--nnoommss// This is a common starting point for the community to identify shortcomings in existing offline RL methods, and provides meaningful metric for progress in this emerging area.
2 RELATED WORK
Recent work in offline RL has primarily used datasets generated by a previously trained behavior policy, ranging from a random initial policy to near-expert unline-trained policy. This approach has been used for continuous control for robotics (Fujimoto et al. 2018a; Kumar al., 22199] Wul et al. 2019 Gulcehre et al. 2020). navigation (Laroche et al. 2019), industrial control (Hein et al., 2017. and Atari video games (Agarwal et al., 2019b). To standardize the community around com- mon datasets, several recent works have proposed benchmarks for offline RL algorithms. |Agarwal jet al. (2019b): Fujimoto et al. (2019) propose benchmarking based on the discrete Atari domain. Concurrently to our work, Gulcehre et al. (2020) proposed a benchmarkf based on locomotion and manipulation tasks with perceptually challenging input and partial observability. While these are important contributions, both benchmarks suffer from the same shortcomings prior evaluation protocols: they rely on data collected from online RL training runs. In contrast, with D4RI., in addition to collecting data from online RL training runs, we focus on range of dataset collection
Note that the benchmark proposed by Gulcehre et al. 2020 contains the Real-World RL Challenges henchmark (Dulac-Amold et al. 2020) hased on (Dulac-Amold et al. 2019) and also uses data collected from partially-trained RI. agents.
2

---

Under review as a conference paper at ICLR 2021
procedures inspired by real-world applications, such human demonstrations, exploratory agents, and hand-coded controllers. As alluded to by [W et al. (2019) and as we show in our experiments, the performance of current methods depends strongly on the data collection procedure, demonstrat- ing the importance of modeling realistic data collection procedures in benchmark.
Offline RL using large, previously-colles datasets has been successfully applied to
sys- tems such as in robotics (Cabi et al.) 2019), recommender systems (Li et 2010) |Strehl et al. 2010)
Thomas et al., 2017), and dialogue systems (Henderson et al.] 2008 Pietquin et al. [01 Jaques
et al., 2019). These successes point to the promise of offline RL, however, they rely on private real-world systems or expensive human labeling for evaluation which is not scalable or accessible for a benchmark. Moreover, significant efforts have been made to incorporate large-scale datasets into off-policy RI. (Kalashnikov et al. 2018: Mo et al. 2018 Gu et al., 2017). but these works use large numbers of robots to collect online interaction during training. Pure online systems for real- world learning have included model-based approaches (Hester et al., 201T), or approaches which collect large amounts of data in parallel (Gu et al., 2017). While we believe these are promising directions for future research, the focus of this work is to provide a varied and accessible platform for developing algorithms.
3 BACKGROUND
The offline reinforcement learning problem statement is formalized within a Markov decision pro- cess (MDP), defined by a tuple AS, P, R. po, (o,7), where S denotes the state space, A denotes the action space, P(s'ss, a) denotes the transition distribution, Pn()) denotes the initial state dis- tribution, R(s, a) denotes the reward function, and y E (0, denotes the discount factor. The goal in RL is to find policy T[als) that maximizes the expected cumulative discounted rewards ((T) = P,, IE=0 7" RR(s,, a.)], also known as the discounted returns.
In episodic RL, the algorithm is given access the MDP via trajectory samples for arbitrary of the algorithm's 's choosing. Off-policy methods may use experience replay (Lin 1992) to store these trajectories in replay buffer D of transitions (st, ag, 8, Te), and use off oif-policy algorithm such as Q-learning (Watkins & Dayan 1992) to optimize æ. However, these methods still iteratively collect additional data, and omitting this collection step can produce poor results. For example, running aftt--of-thh-an off-policy RL algorithms on trajectories collected from an expert policy can result diverging Q-values (Kumar et al.) 2019))
In offline RL, the algorithm no longer has access to the MDP, and is instead presented with a fixed dataset of transitions D. The (unknown) policy that generated this data is referred to as behavior policy gB. Effective offline RL algorithms must handle distribution shift, as well as data collected via processes that may not be representable by the chosen policy class. Levine et al. 2020) provide comprehensive discussion of the problems affecting offline RI..
4 TASK DESIGN FACTORS
In order to design a benchmark that provides a meaningful measure of progress towards realistic applications of offline RL, we choose datasets and tasks cover range of properties designed to challenge existing RL algorithms. We discuss these properties as follows:
Narrow and biased data distributions, such as those from deterministic policies, are problematic for offline RL algorithms and may cause divergence both empirically (Fujimoto et al., 2018a Kumar
al. 2019) and theoretically (Munos) 2003, Farahmand et al.) 2010 Kumar et al. 2019: Agarwal et al. hand-cratted policies. An important challenge in offline RI. is to be able to gracefully handle diverse data distributions without algorithms diverging producing performance worse than the provided behavior. A common approach for dealing with such data distributions is to adopt a conservative approach which tries keep the behavior close to the data distribution (Fujimoto et al. 2018a Kumar et al. 2019 Wu et al. 2019).
2019a Du et al.) 2020). Narrow datasets may arise in human demonstrations, or when using
Undirected and multitask data naturally arises when data is passively logged, such recording user interactions on the internet or recording videos of a car for autonomous driving. This data may not necessarily be directed towards the specific task one is trying to accomplish. However,
3

---

Under review as a conference paper at ICLR 2021
pieces of trajectories still provide useful information to learn from. For example, may be able to combine sub-trayy tibetrajectories to accomplish a task. In the figure to the upper-right, an agent is given trajectories from A-B and B-C in dataset (left image), it can form trajectory from A- C by combining the corresponding halves the original trajectories. We refer to this property as stitching, since the agent can portions of existing trajectories in order to solve task, rather than relying on generalization outside of the dataset.
Sparse rewards. Sparse reward problems pose chal- lenges to traditional RL methods due to the difficulty of credit assignment and exploration. Because offline RL considers fixed datasets without exploration, sparse re- ward problems provide an unique opportunity to isolate the ability of algorithms to perform credit assignment de- coupled from exploration.
An example of stitching to- gether subtrajectories to solve a task.
Figure 2:
Suboptimal data. For tasks with clear objective, the datasets may not contain behaviors from optimal agents. This represents a challenge for approaches such as imitation learning, which generally require expert demonstrations. We note prior work (e.g., (Fujuoto et al., 2018a Kumar et al. 2019 Wu et al. 2019)) predominantly uses data with this property.
Non-represe behavior policies, non-Markovian behavior policies, and partial observ- ability. When the dataset is generated from partially- agent, we ensure that the behavior policy can be realized within our model class. However, real-life behavior may not originate from a policy within our model class, which can introduce additional representational errors. For example, data generated from human demonstrations or hand-crafted controllers may fall outside of the model class, More generally, non-Markovian policies and tasks with partial observability can introduce additional modeling eπrоrs when we estimate action probabilities under the assumption that the data was generated from a Markovian policy. These errors can cause additional bias for offline RL al- gorithms, especially in methods that assume access to action probabilities from a Markovian policy such importance weighting (Precup al. 20000).
Realistic domains. As we discussed previously, real-world evaluation is the ideal setting for bench- marking offline RL, however, it at odds with widely-accessible and reproducible benchmark. To strike balance, we opted for simulated environments which have been previously studied and are broadly accepted by the research community. These simulation packages (such as MuJoCo, Flow, and CÁRLA) have been widely used to benchmark online RL methods and are known to fit well into that role. Moreover, on several domains we utilize human demonstrations or mathematical models of human behavior in order provide datasets generated from realistic processes. However, this did have the effect of restricting our choice of tasks. While recent progress has been made on simulators for recommender systems (e.g., (Ie al. 2019)), they use "stylized" user models and they have not been thoroughly evaluated by the community yet. In the future as simulators mature, we hope to include additional tasks.
In addition, we include a variety of qualitatively different tasks to provide broad coverage of the types of domains where offline RL could be used. We include locomotion, traffic management, autonomous driving, and robotics tasks. We also provide tasks with a wide range in difficulty. from tasks current algorithms can already solve to harder problems that are currently out of reach. Finally, for consistency with prior works, we also include the OpenAI Gym robotic locomotion tasks and similar datasets used by Fujimoto et al. (0018)); Kumar et al. (2019); et al. 20199.
TASKS AND DATASETS
Given the properties outlined in Section we assembled the following tasks and datasets. All tasks consist of an offline dataset (typically I steps) of trajectory samples for training, and simu- lator for evaluation. The mapping not one-to-one several tasks use the same simulator with different datasets. Appendix C lists domains and dataset types along with their sources and Ap- pendix conotaiis a more comprehensive table of statistics such as size. Our code and datasets have been released open-source and are on our website at lhtps: ttops://siiie //sites. google. www/ciemieveee d4r1- all--anooooou/s/
4

---

Under review as a conference paper at ICLR 2021
Maze2D. Nonnmmaroovian policies, undirected and multitask data) The Maze2D domain a navigation task requiring a 2D agent to reach fixed goal location. The tasks are designed to provide a simple test of the ability of offline RI. algorithms to stitch together previously collected subtrajectories to find the shortest path to the evaluation goal. Three maze layouts are provided. The "umaze" and "medium" mazes are shown to the right, and the "large" maze is shown below.
The data is generated by selecting goal locations at random and then using a planner that generates sequences of waypoints that are followed using PD controller. In the figure on the left, the waypoints, represented by circles, are planned from the starting location (1) along the path to goal (2). Upon reach- ing a threshold distance to waypoint, the controller updates its internal state track the next waypoint along the path to the goal. Once a goal is reached, a new goal is selected (3) and the process continues. The trajectories in the dataset are visualized in Appendix G Because the controllers memorize the reached waypoints, the data collection policy is -or--aakov..n.
AntMaze. (Non-markovian policies, sparse rewards, undirected and multitask data) The AntMaze domain is a navigation domain that replaces the 2D ball from Maze2D with the more complex 8-DoF "Ant" quadraped robot. We introduce this domain to test the stitching challenge using morphologically complex robot that could mimic real-world robotic navigation tasks. Additionally, for this task we use a sparse 0-1 reward which is activated upon reaching the goal.
The data is generated by training a goal reaching policy and us- ing it in conjunction with the same high-level waypoint genera- tor from Maze2D to provide subgoals that guide the agent to the goal. The same maze layouts are used: "umaze", "medium". and "large". We introduce three flavors of datasets: 1) the ant commanded to reach a specific goal from fixed start location fantmaze-uma um-ve- 2) in the "diverse" datasets, the ant is commanded random goal from a random start location, 3) the "play" datasets, the ant is commanded to specific hand-picked locations in the maze (which are not necessarily the goal at evaluation), starting from a different set of hand-picker start loca- tions. As in Maze2D, the controllers for this task are non-Markovian they rely on tracking visited waypoints. Trajectories in the dataset are visualized in Appendix G
Gym-MuJoCo. (Suboptimal agents, narrow data distributions) The Gym-MuJoCo tasks (Hopper, HalfCheetah, Walker2d) are popular benchmarks used in prior work in offline deep RL (Fujimoto al. 2018a]Kumar Kumar et al.] 2019 Wu et al.] (2019). For consistency, we provide standardized datasets similar previous work, and additionally propose mixing datasets to test the impact of heterogenous policy mixtures. We expect that methods that rely on regularizing to the behavior policy may fail when the data contains poorly performing trajectories.
The "medium" dataset is generated by first train- ing policy online using Soft Actor-Critic (Haarnoja et al. (018a), early-stopping the training, and collect- ing 1M samples from this partially-traines policy. The "random" datasets are generated by unrolling a ran- domly initialized policy on these three domains. The "medium- replay" dataset consists of recording all samples in the replay buffer observed during train- ing until the policy reaches the "medium" " level of performance. Datasets similar to these three have been used in prior work, but in order to evaluate algorithms on mixtures of policies, we further intro- duce a "medium-exper data, generated via partially trained policy or by unrolling
dataset by mixing equal amounts of expert demonstrations and suboptimal policy.
Adroit. -No--representabl policies, narrow data distributions, sparse rewards, realistic) The Adroit domain (Rajeswaran et al.) 2018) (pictured left) involves controlling a 24-DoF simulated Shadow Hand robot tasked with hammering nail, opening door, twirling pen, picking up and
5

---

Under review as a conference paper at ICLR 2021
moving ball. This domain selected to measure the effect of a narrow expert data distributions and human demonstrations on a sparse reward, high-dimensional robotic manipulation task.
While Rajeswaan et al. (2018) propose utiliizing human demonstra- tions, in conjunction with online RL fine-tuning, our benchmark adapts these tasks for evaluating the fully offline RL setting. We include three types of datasets for each task, two of which are in- cluded from the original paper: small amount of demonstration data from a human ("human") (25 trajectories per task) and large amount of expert data from fine-tuned RL policy ("expert"). To mimic the use-case where a practitioner collects a small amount of additional data from policy trained on the demonstrations, we in- troduce third dataset generated by training imitation policy on the demonstrations, running the policy, and mixing data at a 50-50 ratio with the demonstrations, referred to as "cloned." The Adroit domain has several unique properties that make it qualitatively different from the Gym MuJoCo tasks. First, the data is collected in from human demonstrators. Second, each task is difficult to solve with online RL, due sparse rewards and exploration chal- lenges, which make cloning and online RL alone insufficient. Lastly, the tasks are high dimensional, presenting representation learning challenge.
FrankaKitchen. (Undirected and multitask data, realistic) The Franka Kitchen domain, first proposed by Gupta et al. (2019), in- volves controlling 9-DoF Franka robot in kitchen environment containing several common household items: a microwave. kettle, overhead light, cabinets, and oven. The goal of each task is to interact with the items in order to reach a desired goal configu- ration. For example, one such state is to have the microwave and sliding cabinet door open with the kettle on the top burner and the overhead light on. This domain benchmarks the effect of multitask behavior on a realistic, non-navigation environment in which the "stitching" challenge is non-trivial because the collected trajectories are complex paths through the state space. As result, algorithms must rely on generalization to unseen states in order to solve the task, rather than relying purely on trajectories seen during training.
In order to study the effect of "stitching" and generalization, we use 3 datasets of human demon- strations, originally proposed by Gupta al. (2019). The "complete" dataset consists of the robot performing all of the desired tasks in order. This provides data that is easy for an imitation learning method solve. The "partial" and "mixed" datasets consist of undirected data, where the robot performs subtasks that not necessarily related to the goal configuration. In the "partial" dataset, a subset of the dataset is guaranteed to solve the task, meaning an imitation learning agent may learn by selectively choosing the right subsets of the data. The "mixed" dataset contains no tra- jectories which solve the task completely, and the RI. agent must learn to assemble the relevant trajerrajctooies. This dataset requires the highest degree of generalization in order succeed.
Flow, (Non-representable policies, realistic) The Flow benchmark (Vinitsky et al., 2018) is a framework for study- ing traffic control using deep reinforcement learning. We use two tasks the Flow benchmark which involve controlling autonomous vehicles maximize the flow of traffic through ring or merge road configuration (left).
We use the Flow domain in order to provide task that simulates real-world traffic dynamics. A large challenge in autonomous driving is to be able to directly learn from human behavior. Thus, we include "human" data from agents controlled by the intelligent driver model (IDM) (Treiber al.) 2000), hand-designed model of human driving behavior. In order to provide data with a wider distribution as reference, we also include "random" data generated from an agent that commands random vehicle accelerations.
Offline CARLA. (Partial observability non-representable policies. undirected and multitask data, realistic) CARLA (Dosovitskiy et al. 2017 is high-fidelity autonomous driving simulator

---

Under review as a conference paper at ICLR 2021
that has previously been used with imitation learning approaches (Rhinehart et al., 2018 Codevilla jet al. 2018)) from large, static datasets. The agent controls the throttle (gas pedal). the steering, and the break pedal for the car, and receives 48x48 RGB images from the driver's s perspective as obser- vations. We propose two tasks for offiin RL: lane following within a figure eight path (shown to the right, top picture), and navigation within small town (bottom picture). The principle challenge of the CARLA domain is partial observability and visual complexity, as all observations are provided as first-person RGB images.
The datasets in both tasks are generated via hand-designed controllers meant to emulate human driving the loll-wollo task uses simple heuristics to avoid cars and keep the car within lane boundaries, whereas the navigation task layers an additional high-level controller on top that takes turns randomly at intersections. Similarly to the Maze2D and AntMaze do- mains, this dataset consists of undirected navigation data in order to test the "stitching" property, however, it is in more perceptually challenging domain.
Evaluation protocol. Previous work tunes hyperparameters with online evaluation inside the sim- ulator, and as Wu al.] (2019) show, the hyperparameters have large impact on performance. Unfortunately, extensive online evaluation is not practical in real-world applications and this leads to over optimistic performance expectations when the system is deployed in a truly offline setting. To rectify this problem, we designate subset of tasks in each domain as "training" tasks, where hy- perparameter tuning is allowed, and another subset as "cvaluation" tasks on which final performance measured (See Appendix D Table 5)
To facilitate comparison across tasks, we normalize scores for each environment rovironmenttrouglly to the range
score-randos score between 0 and 100, by computing normal i zzd score 100 expect score-rari random scoze" A normalized score of 0 corresponds to the average returns (over 100 episodes) of agent taking tions uniformly at random across the action space. score of 100 corresponds to the average returns of domain-specific expert. For Maze2I), and Flow domains, this corresponds to the performance of the hand-designed controller used to collect data. For CARLA, AntMaze, and FrankaKitchen, we used an estimate of the maximum score possible. For Adroit, this corresponds to policy trained this
with behavioral cloning on human-demone demonstrommsstttiionnn and fine-tuned with RL. For corresponds to a soft-actor critic (Haamoja et al. 2018b) agent.
6 BENCHMARKING PRIOR METHODS
We evaluated recently proposed offline RL algo- rithms and several baselines benchmarks. This evaluation (1) provides base- lines reference for future work, and (2) iden- tifies shortcomings rithms in order to guide future research. The average normalized performance for all tasks is
offline RL
Sa
existing offline RL algo-
plotted in the figure to the right. It is clear that as we move beyond simple tasks and data collection strategies, differences between algo- rithms are exacerbated and deficiencies in all al- gorithms are revealed. See Appendix Table normalized results for all tasks, Appendix Table experimental details.
BMCV
SAC-011
20 Averaae Normalized Performance
3 for unnormalized scores, and Appendix for
We evaluated behavioral cloning (BC), online and offline soft actor-critic (SAC) (Haarnoja et al.) 2018b), bootstrapping error reduction (BEAR) (Kumar et al. 2019), and behavior-regullrizeed actor- critic (BRAC) (Wu et al.) 2019). advantage- seeantage-weighted regression (AWR) ) (Peng et al. 2019), batch- constrained Q-learning (BCQ) (Fujimoto et al.] 2018a), continuous action random ensemble mix- tures (cREM) (Agarwal et al. 2019b), and AlgaeDICE (Nachum et al. 2019). We note that REM was originally designed for discrete action spaces, and the continuous action version has not been developed extensively. In most domains, expect online SAC to outperform offline algorithms
7

---

Under review as a conference paper at ICLR 2021
when given the same amount of data because this baseline is able to collect on-policy data. There are a few exceptions, such as for environments where exploration challenges make it difficult to find high-reward states, such the Adroit and maze domains.
Overall, the benchmarked algorithms obtained the most success on datasets generated from an RL- trained policy, such as in the Adroit and Gym-MuJoCo domains. In these domains, offline RL algorithms are able to match the behavior policy when given expert data, and outperform when given suboptimal data. This positive result is expected, as is the predominant setting in which these prior algorithms have been benchmarked in past work.
Another positive result comes in the form of sparse reward tasks. In particular, many methods were able to outperform the baseline online SAC method on the Adroit and AntMaze domains, This indicates that offline RL is a promising paradigm for overcoming exploration challenges, which has also been confirmed in recent work (Nair et al. 2020). We also find that conservative methods that constrain the policy to the dataset, such as BEAR, AWR, CQL, and BCQ, are able handle biased and narrow data distributions well on domains such as Flow and Gym-MuloCo.
Tasks with undirected data, such as the Maze2D, FrankaKitchen, CARLA and AntMaze domains, are challenging for existing methods. Even in the simpler Maze2D domain, the large maze provides surprising challenge for most methods. However, the smaller instances of Maze2D and AntMaze are very much within reach of current algorithms. Mixture distributions (a form of rerreerreeet policies) were also challenging for all algorithms we evaluated. For MuJoCo, even though the medium-expert data contains expert data, the algorithms performed roughly on-par with medium datasets, except for hopper. The same pattern was found in the cloned datasets for Adroit, where the algorithms mostly performed on-par with the limited demonstration dataset, even though they had access to additional data.
We find that many algorithms were able to succeed to some extent tasks with controller-generrteed data, such as Flow and carla-lane. We also note that tasks with limited data, such as human demon- strations in Adroit and FrankaKitchen, remain challenging. This potentially points the need for more sample-efficient methods, as big datasets may not be always be available.
7 DISCUSSION
We have proposed an open-source benchmark for offline RL. The choice of tasks and datasets were motivated by properties reflected in real-world applications, such as narrow data distributions and undirected, logged behavior. Existing benchmarks have largely concentrated on robotic control using data produced by policies trained with RL. (Fujimoto et al,, 2018a Kumar et al. 2019 Wu et al. 2019 Gulcehre et al. 2020 Dulac-Arnold et al. (020). This can give misleading sense of progress, as we have shown in our experiments that many of the more challenging properties that we expect real-world datasets to have appear result in substantial challenge for existing methods.
We believe that offline RL holds great promise as a potential paradigm to leverage vast amounts of existing sequential data within the flexible decision making framework of reinforcement learning. We hope that providing a benchmark that is representative of potential problems in offline RL, but that still can be accessibly evaluated in simulation, will greatly accelerate progress in this field and create new opportunities to apply RI in many real-world application areas.
There are several important properties exhibited in some real-world applications of RL that not explored in-depth in our benchmark. One stochasticity of the environment, which can occur in systems such as financial markets, healthcare, or advertisement. And while we explore domains with large observation spaces, some domains such recommender systems can exhibit large action spaces. Finally, our tasks are predominantly in the robotics, autonomous driving, and traffic control. There are several other areas (c.g. in finance, industrial, or operations research) which could provide simulated, yet still challenging, benchmarks.
8

---

Under review as a conference paper at ICLR 2021
Ultimately, would like offline RL applications move from simulated domains to domains, using real-world datasets. This includes exciting areas such as recommender systems, where user behavior can be easily logged, and medicine, where doctors must keep complete medical records for each patient. A key challenge in these domains is that reliable evaluation must be done in real system using off-policy evaluation (OPE) methods. We believe that both reliable OPE methods and real-world benchmarks which are standardized across different research groups, will be important to establish for future benchmarks in offline RL.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. arXiv preprint arXiv: vrrXiv//008..0 1000001,08.,026,,, 2019a.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity off-po deep reinforcement learning. CoRR, abs/1907.04544, 2019b. URL http::/arxiv.co abs/1907.04544
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 2016.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. A framework for data-driven robotics. arXiv preprint arXiv: 1909. 12200, 2019.
Felipe Codevilla, Matthias Miiller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End- to-end driving via conditional imitation learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-9. IEEE, 2018.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings the 1st Annual Conference on Robot Learning, pp. 1-16, 2017.
Simon S. Du, Sham M. Kakade, Ruosong Wang, and Lin F. Yang. Is good representation suf- ficient for sample efficient reinforcement learning? International Conference on Learning Representations, 2020. htttps:///pperreeeeese wwwpee.rreee....eee...e. net.
wwt.wewww..weeeeeeeeee/ fwr.mriruiiier genAVKPB
Gabriel Duloc-Arold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12900 2019.
Gabriel Gulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learn- ing. arXiv preprint arXiv:2003. 11881, 2020.
Amir-massout Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. In Advances Neural Information Processing Systems, pp. 568-576, 2010.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv:1812.02900, 2018a.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th In- ternational Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587-1596. PMLR, 2018b.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint 1911v:1910.01700 2019.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA), pp. 3389-3396. IEEE, 2017.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Kon- rad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, Gabriel Dulac- Arnold, Jerry Li, Mohammad Norouzi, Matt Hoffman, Ofir Nachum, George Tucker, Nicolas Heess, and Nando de Freitas. RI unplugged: Benchmarks for offline reinforcement learning, 2020.
10

---

Under review as a conference paper at ICLR 2021
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving ong-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956. 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, 0oss/880.001990
20. http://t: http://www.rggbb/////o0/ 01290
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv: 01000, 2018b.
Daniel Hein, Steffen Udluft, Michel Tokic, Alexander Hentschel, Thomas A Runkler, and Volkmar Sterzing. Batch reinforcement learning on the industrial benchmark: First experiences. In 2017 International Joint Conference on Neural Networks (IJCNN), pp. 4214 IEEE, 2017.
James Henderson, Oliver Lemon, and Kallirroi Georgila. Hybrid reinfforceemeen/sppprvveeed learning of dialogue policies from fixed data sets. Computational Linguisticx, 34444)4873344 2008.
Todd Hester, Michael Quinlan, and Peter Stone. A real-time model-based reinforcement learning architecture for robot control. arXiv preprint arXiv: 105. 1749, 2011.
Eugene Ie, Chih-wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. Recsim: A configurable simulation platform for recommender systems. arXiv preprint
2019.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. Way off off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, htts/1907.00456, 2019. URL org/ brgg/bs/// /907 00456
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforce- ment learning for vision- basodd robotic manipulation. In Conference on Robot Learning, pp. 651- 2018.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019. URL http:://aaii..orrttt/t/
htsp/bbs/19900 00949
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 24770799,,,,777, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning (ICML), 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Lihong Li, Wei Chu, John Langford, and Robert Schapire. contextual-bandi approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pp. 661-670, 2010.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching Machine learning, 8(3-4):293-32) 1992.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human- control through deep reinforcement learning. Nature, 518((54)))559555 533, February 2015. ISSN 00280836.

---

Under review as a conference paper at ICLR 2021
Kaichun Mo, Haoxiang Li, Zhe Lin, and Joon- Joon-Young Lee. The adobeindoornav dataset: Towards deep reinforcement learning based real-world indoor robot visual navigation. 2018.
Rémi Munos. Error bounds for approximate policy iteration. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning, pp. 560-567. AAAI Press, 2003.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:/9/2.02074. 2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.0933) 09359, 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine Advantage-weiighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00117,, 2019.
Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Hervé Frezza-Buet. Sample- efficient batch reinforcement learning for dialogue management optimization. ACM Transactions on Speech and Language Processing (TSLP), 7(3):1-21, 2011.
Doina Precup, Richard S Sutton, and Satinder Singh. Eligibility traces for off off-policy policy evalua- tion. In JCML '00 Proceedings of the Seventeenth International Conference on Machine Learning, 2000.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems, 2018.
Nicholas Rhinehart, Rowan McAllister, and Sergey Levine. Deep imitative models for flexible inference, planning, and control. arXiv preprint arXiv: urXiv:1810.06544 2018.
Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from logged implicit explo- ration data. In Advances in neural information processing systems, pp. 2217-2225, 2010.
Philip S Thomas, Georgios Theocharous, Mohammad Ghavamzadeh, Ishan Durugkar, and Emma Brunskill. Predictive off-policy policy evaluation for nonstationary decision problems, with ap- plications to digital marketing. In AAAI, pp. 4740-4745, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observa- tions and microscopic simulations. Physical review E, 62(2):1805, 2000.
Eugene Vinitsky, Aboudy Kreidieh, Luc Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M Bayen. Benchmarks for reinforcement learning in mixed-autonomy traffic. In Conference on Robot Learning, pp. 399,409, 2018.
Christopher J.C.H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8:279-292, 1992.
Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen. Flow: Architecture and benchmarking for reinforcement learning in traffic control. arXiv preprint 170.0555. pp. 10, 2017.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 111,11191,, 2019.
12

---

Under review as a conference paper at ICLR 2021
Appendices
A
TASK PROPERTIES
The following is full list of task properties and dataset statistics for all tasks the benchmark. Note that the full dataset for "carla-town" requires over 30GB of memory to store, so we also provide subsampled version of the dataset which we used in our experiments.
Domain
Task Name
Controller Type
# Samples
maze2d-umen -maaze
Planner
10
mare2d-medium
Planner
2 10
Maze2D
maze2d-large
Planner
4 10
umtmaze-umaze
Planner
10
antmare-umazz-ddivrree
Planner
106
AntMaze
antmaze-mecdium-play
Planner
10
antmaze-medium -inecdiium-diveeee
Planner
10c
antmaze-large-play
Planner
10c
antmaze-large-divers
Planner
10
hopper-random
Policy
10
hopper-medium
Policy
10
hopper-mediiun-repla
Policy
200920
hopper-mediumm-xpeer
Policy
2 10
halfcheetahh-random
Policy
10
Gym-MuJoCo
nalfchectah-medingg
Policy
10
Policy
101000
Policy
2 x 10" 10
walker2d- randdoom
Policy
walker2d-men me-dmu
Policy
10c
walker2dd-eedium-rppla
Policy
100930
walker2d-meddium-xxpo
Policy
2 10" 5000
pen-human
Human
pen-cloned
Policy
5 10
pen-expert
Policy
5 10
hammer-human
Human
11310
hammer-cloned
Policy
10
Adroit
hammer-expert
Policy
10c
door-human
Human
6729
door-cloned
Policy
10
door-expert
Policy
10
relocate-hume human
Human
9942
relocate-cloned
Policy
10
relocate-expert
Policy
106
Now-ring-random
Policy
10
www-ring-contrlll
Policy
106
Flow
Row-merge-random
Policy
106
now-meerge-onntroleer
Policy
10c
kitchen-compii
Policy
3680
FrankaKitchen
kitchen-partial
Policy
136950
kitchen-mixed
Policy
136950
carla-lane
Planner
10"
CARLA
carla-town
Planner
2 10 full
10a subsampled
Table I: Statistics for each task in the benchmark. For the controller type, "planner" refers to desi-de navigation planner, "human" refers to human demonstrations, and "policy" refers to random or neural network policies. The number of samples refers to the number of environment transitions recorded in the dataset.
13

---

Under review as a conference paper at ICLR 2021
B RESULTS BY DOMAIN
The following tables summarize performance for each domain (excluding CARLA, due to all algo- rithms performing poorly), sorted by the best performing algorithm to the worst.
000
8558
0C-
BR4C
80-
BMCC p
SAcell
Avarage Normallzed
Average
Gym
Adrolt
800
86A4
8564
26R
SAC
54C otf
SRAG
300
20
Averge Normeizeed Sertormane
Average kormallowe Pertorancc
Franka Nithen
80-
800
800
10.Ce
BMCV
30ICE
32M
BMC-9
Normaluel ferfumarce
malued nes
14

---

Under review as a conference paper at ICLR 2021
Table 2: Normalized results comparing online & offline SAC (SAC, SAC-off), bootstrapping error reduction (BEAR), behavior- wwwivvioorrrggaaee regularized actor critic with policy
(BRAC-p) or value (BRAC-v) regularization. behavioral cloning (BC), advantage-Wn weeeggg--wwieedd regression (AWR), batch-constraines Q-learning (BCQ), continuous random ensemble mixtures (cREM), and AlgacDICE (aDICE). Average results are reported over seeds, and normalized score between 0 (random) and 100 (expert).
COL
5.7
5.0
12.5
74.0
84.0
61.2
53.7
15.8
14.9
35.4
7.0
10.8
44.4
79.2
58.0
46.2
26.7
48.6
62.4
111.0
98.7
37.5
4.4
9.9
0.2
39.2
2.1
0.4
-0.1
107.0
86.7
101.5
95.0
$2.0
87.9
157.2
40.6
43.8
49.8
51.0
20.9
-2.6
aDICE
-15.7
10.0
-0.1
0.0
0.0
0.0
0.0
0.0
0.0
-0.3
0.5
0.9
-2.2
0.3
1.2
-2.1
0.6
1.1
-0.8
0.4
1.1
-3.3
0.3
-0.0
-0.1
-2.9
0.3
0.0
-0.3
-3.5
0.3
0.0
83.6
196.4 6.
4.7
0.0
0.0
2.5
-1.2
-11.2
BCQ
12.8
8.3
6.2
78.9
53.0
0.0
0.0
6.7
2.2
2.2
10.6
40.7
53.1
54.5
38.2
15.0
33.1
64.7
57.5
110,9
68.9
0.5
-0.0
-0.1
44.0
0.4
0.0
-0.3
114.9
107.2
99.0
41.6
76,2
94.6
114.8
28.2
8.1
18.9
8.1
-0.1
1.9
<REA
-15.8
0.9
-2.2
0.0
0.0
0.0
0.0
0.0
0.0
-2.6
-0.3
-0.2
0.6
-3.0
40.2
0.8
-2.6
-0.2
0.7
3.5
0.2
-0.
-0.2
3344
0.2
4.1
-0.2
0.3
0.2
-0.2
47.4
-87.4
183.2
31.2
0.0
0.0
0.0
0.1
-0.9
AWR
1.0
7.6
23.7
56.0
70.3
0.0
0.0
0.0
0.0
2.5
1.5
10.2
37.4
17.4
35.9
40.3
15.5
28.4
52.7
53.8
27.1
12.3
1.2
0.4
-0.0
28.0
0.4
0.0
-0.2
111.0
39,0
102.9
91.5
75.2
80.4
152.7
99.6
0.0
15.4
10.6
-0.4
1.9
BRAC-+
-6.00
33.8
40.6
70.0
70.0
0.0
0.0
0.0
0.0
31.2
1.9
12.2
46.3
81.1
31.1
47.7
0.9
0.6
41.9
81.6
0.8
0.6
0.2
-0.3
-0.3
-2.5
0.3
-0.1
-0.3
-3.0
0.3
-0.3
.91.2
78.6
143.9
27.3
0.0
0.0
0.0
19.6
-2.6
BRAC-P
4.7
32.4
10.4
50.0
40.0
0.0
0.0
0.0
0.0
24.
-0.2
11.0
43.8
77.5
32.7
45.4
-0.3
0.6
44.2
76.9
1.9
0.3
-0.3
-0.3
1.6
0.3
-0.1
-0.3
-3.5
0.3
-0.3
-12.3
95.7
129. 8
146.2
0.0
0.0
0.0
18.2
4.6
BEAR
3.4
29.0
4.6
73.0
61.0
0.0
8.0
0.0
0.0
25.1
11.4
41.7
59.1
52.1
38.6
19.2
33.7
53.4
40.1
96.3
-1.0
-0.3
-0.3
26.5
0.3
-0.1
-0.3
105.9
127.3
103.4
98.6
62.7
103.5
150.4
-2066
0.0
13.1
47.2
-0.2
-2.7
SAC-oFf
58.2
26.1
-1.9
0.0
0.0
0.0
0.0
0.0
30.5
11.3
-1.3
0.9
0.8
-2.4
1.9
3.5
1.8
-0.1
1.6
6.3
3.9
0.0
23.5
0.2
0.0
-0.2
6.1
25.2
40.
70.0
111.6
-40.1
15.0
0.0
2.5
0.1
-1.8
BC
3.8
30.3
5.0
65.0
55.0
0.0
0.0
0.0
0.0
2.1
1.6
9.8
36.1
6.6
29.0
38.4
11.3
11.3
35.8
6.4
111.9
34.4
1.5
0.5
0.0
56.9
0.8
-0.1
-0.1
85.1
125.6
34,9
101.3
.5.00
94.9
114.1
-17.1
33.8
33.8
47.5
31.8
-1.8
SAC
62.7
21.3
2.7
0.0
0.0
0.0
0.0
0.0
0.0
100.0
100.0
100.0
100.0
100.0
100.0 .
100.0
100.0
100.0
100.0
100.0
21.6
0.2
0.2
-0.2
21.6
0.2
-022
-0.2
21.6
0.2
-02
-0.2
100.7
100.7
121.5
121.5
0.0
0.6
0.0
00.8
14
balfcheetalt-nn www.atmddduummeeee expeee
mazeld-Jin weceed-umma ee-uaaze
muzeld medium
maze2d-lan 2a-egggeee large
antmaze- umaze
antmaze wwwza-aaaaeeeeeee diverse
antmaze- medium--pii meay play
antmaze-mediumi meaizz--ddium-m-ie
madiim-ddivvers -divvrse
antmaze- Sarmazz-larg--aaee large---eeee play
antmazee large- diverse
hall malfcheethh--. random
walker 2d d--random
walker2d-mmeiee m--meduum
hopper medium
hulfchectah--in malt-medium--eppl
walker 2ll-melium--eeeee menetiu-rrrlpayy replay
hopper- pecr-mmed pere-meddiu--p-pee hum--rpplyy
wal 2d aed--m-du--eeeee
memamm-eeeeeeee expert
hopper -eed-meduum expper eeeppre--
hammer human
relocate- human human
commer--llneed
relocate-clome clened
hammer-- comeer--xperr
relocate- expert door-c hoor-expert
flow-ring- ring-------n controller
flow-ring- -in---in-runooo
flow -erge-rreeeee merge-controllee
controller
flow merge-rreeeeeeee random
kitchen complete
kitehen-partin partial
Task Name
hopper- peepper-rannieee ran
door- human
pen-cloned
door-cloned
kite kitehen-mixe
carla-lane lane carla-ton town
pen- pon-humm So-hhumn
pen-expert
Frunka Kitchen
Maze 2D
AntMaze
CARLA
Adroit
Gym
Flow
15

---

Under review as a conference paper at ICLR 2021
Table 3: The raw, un un-normalized scores for cach task and algorithm are reported in the table below. These scores represent the undiscounted return obtained from executing a policy in the simulator, averaged over 3 random seeds.
COL
31.7
26
40.0
0.7
0.8
0.5
0.2
0.1
4114.8
322.9
331.2
5232.1
2664.2
2557.3
5455.6
1227.3
1227.3
7466.9
5097.3
3192.0
1214.0
300.2
234.3
2.0
1264.6
-0.41
-44.76
-10.66
3286.2
11062.4
2926.8
4019,9
-66.5
15.1
450.9
204.6
1.8
1.9
2.0
213.2
-180.379
AlgacDICE
2.2
39.6
6.5
0.0
0.0
0.0
0.0
0.0
-31.00
26.5
10.0
-551.6
15.5
17.5
-540.8
31.0
14.0
3377.6
19.7
15.5
-0,7
-34488
-56.5
-10.8
8.6
-233.1
-56.4
-18.8
6.9
-255.2
-56.5
-8.7
136.3
-6.8
533.9
128.7
0.0
0.0
0.1
-13.4
-400.2
BCQ
41..5
35.0
23.2
0.8
0.6
0.0
0.0
0.1
0.0
-1.3
228.0
323.9
4767.9
2441.0
1752.4
4463.9
688.
1057.8
7750.8
2640.5 3
3588.5
2149.0
210.5
-56.6
-8.
1407.8
-224.4
-56.
-17.5
3521.3
13731.5
2850.7
1759.6
-20.7
14.2
3614
178.3
0.3
0.8
0.3
-1.4
-66.6
cREM
2.1
15.6
0.8
0.0
0.0
0.0
0.0
-97.6
-12.7
4.0
-600.0
99.3
0.3
-655.7
-8.7
5.8
-600
..00
3.1
199.7
243.22
-58.7
-14.1
66.5
-245.8
-5.0
-16.9
104.0
-247.1
-60.9
-11.7
-255.0
-330.9
505 805.8
52.7
0.0
0.0
0.0
-0.0
-137.4
AWR
25.2
33.2
70.1
0.6
0.7
0.0
0.0
0.0
36.3
71.5
312.4
4366.1
800.7
1149.5
4727.4
712.5
904.0
6267.3
2469.7
862.0
463.1
-15.3
-4.4
-7.2
931.3
-226.9
-5.11
-16.6
3406.0
4822.9
2964.5
3875.5
-2.66
-12.7
441.4
329,3
0.0
0.4
-65.8
BRAC-V
1.7
102.4
115.2
0.7
0.7
0.0
0.0
0.0
3590.
87. 77.4
376.3
5473 3.8
3725
990.
564006
44.5
-0.8
4926
3747.5
5.1
114.7
-24388
-66.4
-19.7
22.2
-236.9
55.00
19.44
6.4
-241.1
-66.6
-21.4
-33.22
-16.2
422.9
176.3
0.0
0.0
199
-18155
BRAC-P
30.4
98.8
34.5
0.5
0.4
0.0
0.0
0.0
2713.6
-7.2
337.5
5158.8
3559,9
1044.0
5350,8
-11.5
0.5
5208.
3533,1
42.6
339.0
239.77
-66.5
-19,7
143.4
-236.7
-58.7
-19.8
-7.8
-211.4
-66.4
-20.6
1188.5
16.2
392.9
427.6
0.0
0.0
0.0
186.
-231.6
BEAR
28.6
89.8
19.0
0.7
0.6
0.0
0.1
0.0
0.0
2831.4
336.3
349.9
4897.0
2717.0
1674.5
4517.
883.8
1076.8
6349.6
1842.7
3113.5
66.3
222.0
-66.4
-899
885.4
-21.11
-60.9
-1766
3254.
16359.7
2980.1
4173.8
-66.3
31.0
436.5
75.1
0.0
0.5
1.9
-3.0
-182.5
SAC-off
145.6
82.0
1.5
0.0
0.0
0.0
0.0
0.0
0.0
3502.0
192.0
347.7
-808.6
44.2
5.7
-58
87.8
93.3
-55.7
-5.1
32.9
284.8
214.2
57.2
.4.5
797.6
-24.11
-56.3
-16.1
277.4
3019.5
163.8
-1..2
147.7
-32.4
354.6
33,9
0.6
0.0
-159.9
BC
29.0
93.2
20.1
0.7
0.6
0.0
0.0
0.0
0.0
17.9
73.0
299.4 4
4196.4
304,8
923.5
4492.
S18.6
364.4
4169.4 4
297.0
3621.2
1121.9
-82.4
41.7
-5.6
1791.8
-175.
-60.7
-10.1
2633.7
16140.8 8
969 4
4289.3
-73.3
14.7
359,8
82.6
1.4
1.4
1.9
324.7
-161.5
SAC
110.4
69.5
14.1
0.0
0.0
0.0
0.0
0.0
0.0
12135.0
4592.3
3234.3
121.35.0
4592.3
3234.3
12135.0
4592.3
3234.3
12135.0
4592.3
3234.3
739.3
-24.77
-61.8
-13.7
739.3
-248.7
-61.8
-13.7
739.3
-248.7
-61.8
-13.7
25.8
25.8
375.4
375,4
0.0
0.0
0.0
-79.7
halfcheetah- medium replay
halfcheetah- medium expert
antmaze- -ivuudidrvvse
walker2. (ilkker2----eiile mediim--eeeple numlryllay
walker2d-mes 2wd-ddddm-----oo pxpt
antmaaze umaze- umaze-diverss diverse
antmaze- antmaze-mmdiune mwdiumm-d-mmmmmm play
antmaze- large- play
antmaze-lan large-drrresss diverse
halfcheetah- sestthh-randdm random
halfchcetah--ie weaa--meddng medium
hopper- medimm--relaay
hopper- Sespper-mediungg mediumm--ae dededuee-eppprr expert
flow-rin ring controller
flow- www.-mer-e-oee controller
flow- www--meeeeeaaaae-- random
Task Name
maze2d-unn 2d-azed-mmzzze od-umaze
mmnzeddmmedimm
maze2d- large
antmaze-umen umaze
walker2d-rande ad-random
hopper- random
walker2d-mediing
hopper medium
pen-humun
haninee human
door-human
relocate-Bie hummnn
pen-cloned
hammer--clome cloned
door-cl cnne
relocate- Mlocate-loondd cloned
pen expert
hammer-ox expert
door door-expert expert
relocate expert
now-ring-randowe
kitchen- complete
kitchen- parttall kit kitchen--ne
curla- lame carla-
Offline CARLA
KricheKitchen
Domain
Maze2D
AntMaze
Adroit
Gym
Flow
16

---

Under review as a conference paper at ICLR 2021
C TASK AND DATASETS
The following table lists the tasks and dataset types included in the benchmark, including sources for each.
Domain
Source
Dataset Types
Maze2D
N/A
UMaze, Medium, Large
AntMaze
N/A
UMaze, Diverse, Play
Gym-MuloCo
Brockman et al. (0166)
Expert, Random, Medium (Various) Medium-Expert, Medium-Replay
Todorov et 2012)
Adroit
Rajeswaran et al.
(2018)
Human, Expert (Rajeswaran et al. 2018
Cloned
Flow
Wu et al. (0117)
Random, Controller
Franka Kitchen
Gupta et al. (2019)
Complete, Partial, Mixed (Gupta el al.
2019)
CARLA
Dosovitskly al. 2017)
Controller
Domains and dataset types contained within our benchmark. Maze2D and AntMaze are new domains we propose. For each dataset, we also include references to the source if orig- inally proposed in another work. Datasets borrowed from prior work include MuJoCo (Expert, Random, Medium), Adroit (Human, Expert), and FrankaKitchen (Complete, Partial, Mixed), All other datasets are datasets proposed by this work.
Table 4:
D TRAINING AND EVALUATION TASK SPLIT
The following table lists recommended protocol for hyperparameter tuning. Hyperparameters should be tuned on the tasks listed on the left in the "Training" column, and algorithms should be evaluated without tuning the tasks in the right column labeled "Evaluation".
Domain
Training
Evaluation
maze2d-umaze
maze2dl-vall-umal
Maze2D
maze2d-medium
maze2d-eval-meddiun
maze2d-large
maze2d-eval-large
ant-umaze
ant-eval-umaze
ant-urazzediverse
unt-eval-mmaze-divers
ant-medium-play
autleval-meddium-lay
AntMaze
ant-medium-diierse
lil-eval-medium-diii
ant-large-plas
ant-eval-lant large-play
lrggeereeeee
ant-eval large-diverse
pen-human
hammer-human
pen-cloned
hammer-cloned
pen-expert
hammer-exper expert
Adroit
door-human
glocate-human
door-cloned
relocate-cloned
door-expert
relocate-cxpert
halfcheetah-randon
hopper-random
halfcheetah-meede medium
hopper-medium
halfchectah-me mixed
hopper-mixed
halfchectah-meed expert
hopper-medium-expp
Gym
wuller22-randoo
ant-random
walker2d-medium
ant-medium
walker2d-mixed walkerrdd-medium-exper and-medium-expprt
ant-mixed
Table 5: Our recommended partition of tasks into "training" tasks where hyperparameter tuning is allowed, and "evaluation" tasks where final algorithm performance should be reported.
17

---

Under review as a conference paper at ICLR 2021
E EXPERIMENT DETAILS
For all experiments, we used default hyperparameter settings and minimal modifications to public implementations wherever possible, using 500K training iterations gradient steps. The code bases we used for evaluation are listed below. The most significant deviation from original published algorithms was that we used an unofficial continuous-action implementation of REM (Agarwal al. 2019b), which was originally implemented for discrete action spaces. We ran our experiments using Google cloud platform (GCP) on ol-sstame -standadddd machines.
BRAC and AIgaeDICE:
gpogle-reseane
AWR: AWW.AAttss////aaaabcc/////pppppp
SAC: Scc.hhtt..///tthhb...o///ttttt//
BEAR: BBAA4//tttsss///nno..oooo/ooarrd
Continuous-action REM:
mujoco
CQL: htw.://p./gggaa........//////aaa
F
ASSESSING THE FEASIBILITY OF HARD TASKS
Few prior methods were able to successfully solve carla- town the larger AntMaze tasks. While including tasks that present a challenge for current methods important ensure that our benchmark has room for improvement, also important to provide some confidence that the tasks are actually solvable. We verified this in two ways, First, we ensured that the trajectories observed in these tasks have adequate coverage of the state space. An il- lustration of the trajectories in the CARLA and AntMaze tasks are shown below, where trajectories are shown different colored lines and the goal state is marked with star. We see that in carla-town and AntMaze, each lane or corridor is traversed multiple times by the agent.
(a) carla-town
Second, the data in AntMaze was generated by having the ant follow the same high-level planner in the maze in Maze2D. Because Maze2D solvable by most meth- ods, would expect this to mean that AntMaze is po- tentially solvable as well. This While the dynamics of the ant itself are much more complex, its walking gait is rel- atively regular periodic motion, and since the high-level waypoints are similar, we would expect the AntMaze data to provide similar coverage as in the 2D mazes, as shown in the figures on the right. While the Ant has more erratic motion, both datasets cover the the ma- jority of the maze thoroughly. A comparison of the state coverage between Maze2D and AntMaze on all tasks is shown in the following Appendix section
CC
(b) antmaze-large
(c) maze2d-large
18

---

Under review as a conference paper at ICLR 2021
G MAZE DOMAIN TRAJECTORIES
In this section, we visualized trajectories for the datasets in the Maze2D and AntMaze domains. Each image plots the states visited along each trajectory different colored line, overlaid on top of the maze. The goal state is marked as white star.
Figure 4: Trajectories visited in the Maze2D domain. From left-to-right: maze2d-umaze, maze2d- medium, and maze2d-large.
Figure 5: Trajectories visited in the AntMaze domain. Top row, from left-to-right: antmaze-umaa umaze, mutmaze-medium-plae and antmaze- laruuz--are-ppay. Bottom row, from left-to-right: cilmaze-ummzze diverse, antmare-meeduum-divere. and antmaze-large-divere.
19


Reviews:
[{"review_id": "woE-2zxOdE0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "D4RL: Datasets for Deep Data-Driven Reinforcement Learning\nreview:\n\nsummarization:\n\nIn this paper, the authors consider the problems of offline reinforcement learning problems,\n and has a focus of dataset and shareable code base.\nWhile no novel algorithms are proposed in this project,\na systematic evaluation of existing algorithms (offline RL algorithms) is proposed.\n\nPros:\n1. Offline RL is a hot topic this year, \nwith a lot of papers exploring efficient and robust ways to utilize offline collected data.\nThis paper, while using simulated data, \nprovides a general platform to benchmark these algorithms.\n\n2. The project provides a comprehensive evaluation and discussion on existing considerations in offline RL.\nIt provides several interesting directions in offline RL.\n\n3. The paper is well-written.\nIt is very clear what the purpose of the project is,\nand it is very clear why the authors make the dataset in the way they did.\n\nCons:\n\n1. The dataset is mostly simulated.\nWhile I understand the difficulty of collection real-data, \nit does raise some concerns that a simulated dataset can be generated by researchers themselves.\n\nSummary:\nWhile the dataset is simulated,\nI do think there’s value in the dataset and the shared code-base to facilitate recent progress in offline RL research.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iBlZrFsxYPE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this paper a test suite of data sets and corresponding benchmarks for offline reinforcement learning is introduced.\nSeveral existing RL benchmarks are used, the results of several algorithms are presented.\nThe authors claim that the benchmarks were specifically designed for the offline setting and are guided by the key properties of datasets in real-world applications of offline RL.\n\nStrong points:\nThe present paper has already been cited and the benchmarks suite has already been used by other publications. Obviously there is a need for offline RL test suites.\n\nWeak points:\nThe authors' claim that such a benchmark for offline RL should \"be composed of tasks that reflect challenges in real-world applications of data-driven RL\" is only partially met by the paper in its present form. The area of robotics, with deterministic dynamics, is comparatively well represented, but there is no real, industrial application. In particular it seems that so far no benchmark has been included that has the ambition to have the characteristics and complexity of a real application. \n\nRecommendation:\nOn the one hand, the really realistic benchmarks are missing, so that a publication seems premature. On the other hand, the current status is already used by the research community, since there seems to be no test suite for offline RL apart from \"RL unplugged: Benchmarks for offline reinforcement learning\". I therefore recommend to accept the paper.\n\nQuestions:\nTo what extent are the current benchmarks stochastic?\nAre there bi- or multimodal transition probabilities?\n\nAdditional feedback with the aim to improve the paper:\nIn Table 2 and Table 3 average results are reported over only 3 random seeds. This seems to me to be clearly too little, especially since the policy performance of Q-function based algorithms often fluctuates strongly. Since no uncertainties, e.g. in the form of standard error, are given, the reliability of the results cannot be assessed.\nThe way policies are selected before they are tested should be described more clearly. My impression was that in each case the policy is used that results for a considered algorithm and random seed after 500K training iterations or gradient steps. Since different algorithms require different computational efforts this approach does not seem to be in the sense of a real-world application. In most cases there should be the willingness to use much more computational effort for especially good policies. According to the motto: computing power is cheap, data is expensive. \n\nI like the formulation “Effective offline RL algorithms must handle […] data collected via processes that may not be representable by the chosen policy class.“ This expresses the, in my opinion, correct view of the real situation well, while the assumption that there is a \"behavior policy\" that generated the data is not true in general. It may have been different people at different times who performed the actions while the data set was recorded.\n\nPlease check the bibliography for accidental lower case, like „markov“, „adobeindoornav“\n\n\n----------------------------------\n(Dec 3) Taking into account the other reviews, the authors' responses and the changes made by the authors, as well as the extensive and controversial discussion, I rate the paper still with a score of 6.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_Sn87qXh3el", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a standardized benchmark for offline RL research. The data collection is well motivated from real world scenarios covering many important design factors. I really appreciate that human demonstration and hand-crafted controllers are also included. The evaluation protocol and the API looks clear and easy to use. The benchmark of existing methods is thorough and provides many useful insights. I believe this work will have a high impact on the offline RL community. I can expect this benchmark will be used by many papers in the future and will function as the starting point for many offline RL research.\n\nHowever, I notice that this dataset may not be accessible for underrepresented groups. I therefore vote to reject. As the authors note in the paper, each task consists of a dataset for training and a simulator for evaluation. In my understanding, half of the six tasks (Maze2D, AntMaze, Gym-mujoco) depend heavily on the MuJoCo simulator, which is a commercial software and is not free even for academic use. A personal MuJoCo license costs 500 USD per year. I am concerned that MuJoCo is not accessible for most underrepresented researchers. It is not clear when MuJoCo becomes a dominating benchmark for online RL research, though there are indeed free, open-sourced alternatives, e.g. PyBullet (https://github.com/bulletphysics/bullet3). In online RL, we need the simulator for training. One reason MuJoCo becomes popular may be because it's more stable and faster than PyBullet. However, in offline RL, a simulator is used only for evaluation not for training. So the high reliability of MuJoCo may no longer be so necessary. I therefore view offline RL as a good opportunity for the community to get rid of commercial simulation softwares, making RL research more accessible for underrepresented groups. If accepted, this paper will indeed greatly promote the use of MuJoCo given its potential high impact, making RL more privileged.  \n\nOverall, I really enjoy reading the paper and am glad to see a standardized benchmark for offline RL. I am happy to raise my score if the accessibility issue is addressed, e.g., by using PyBullet as the physical engine.\n\n====================\n\n(Nov 24) I appreciate the effort put into the Bullet reimplementation and therefore increase my score from 3 to 6.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_H97HrjdmeR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper offers a new set of challenges for batch reinforcement learning coupled with a set of benchmarks, including autonomous robotics and driving domains. All domains also come with simulation environments.\n\nWhile the dataset provided by the authors may be a useful resource for researchers in offline-RL, this paper does not introduce any new or novel ideas. Overall the main contribution of this work is the gathering of offline data in one place, reducing the time needed for other researchers to do so. It does not seem as if the authors did any non-trivial work other than annotating the data. Furthermore, there have been many previous work which have already collected offline datasets as part of their work.\n\nI do not underestimate the importance the authors' hard work, nor the importance of the provided datasets to the RL community. Nevertheless, I do not believe this work should be published in a high-end conference without presenting any ideas that are not trivial or known to other researches. \n\nThe authors propose to use the following design factors in their offline datasets: narrow distributions, multi-task data, sparse rewards, suboptimal data, and partially observable policies. While these factors may indeed be good for testing offline-RL algorithms, they do not provide a complete picture of real world datasets. Real datasets present many more real-world problems, including:\n1. Non-stationary data. Many real datasets are non-stationary. The non-stationary behavior could be mimicked or simulated from real behavior.\n2. Real policies. Real datasets don't involve policies that were trained by RL agents. The authors could create datasets that are constructed by real human beings (for example, humans playing atari games, with mixed or different expertise). In a controlled setup, the datasets could be constructed so that policies are categorized (e.g., \"level of non Markovianess`\"). \n3. Causal structures. Real policies may act according to some causal structure in the background that is not necessarily known.\n4. Reduction from real datasets. One could collect or use large amounts of high quality datasets from the real world and add certain corruptions to the data as to lower its quality (e.g., removing certain trajectories). If the initial data is of high quality, the corrupted data could be controlled well.\n5. Changing and/or very large action sets. Real world datasets have changing and large datasets. As an example, consider ad placement, or text based tasks.\n6. Non-robotic environments, including games but also real world problems.\n\nIf the authors bring together datasets that generate original ideas that have never been previously explored, then I believe it's more likely that this paper could be accepted in future venues.\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes benchmark tasks for offline reinforcement learning.  The paper has major strength and weakness, and it has resulted in very active discussion among reviewers, authors, and other participants.

The major strength includes the following:
- The proposed benchmark is already heavily used in the community
- Offline reinforcement learning is very important to solve reinforcement learning tasks in the real world
- The paper covers a range of tasks and provides through evaluation of existing methods to be used as baselines

The major weakness is that it is not sufficiently convincing that the methods that perform well in the proposed benchmark tasks will perform well in the offline reinforcement learning tasks in the real world.  

This is partly due to the nature of the benchmark tasks of offline reinforcement learning, which require simulators to evaluate the policies learned with offline reinforcement learning.  This means that one cannot simply collect datasets from real world tasks and provide them as benchmark datasets.  

Although one cannot do much about simulators, benchmark tasks for offline reinforcement learning still have many design choices.  In particular, how should the datasets in the benchmark be collected (i.e., behavior policies)?

While the datasets in the proposed benchmark are collected with various behavior policies including humans, it is not necessarily convincing that the resulting benchmark tasks are good for the purpose of evaluating offline reinforcement learning to be used in the real world.

In addition to the suggestions given by the reviewers, a possible direction to improve the paper is to focus on the choice of behavior policies used to generate the datasets in the proposed benchmark.  One might then be able to provide some convincing arguments as to why performing well in the benchmark might imply good performance in the real world by relating it to the choice of behavior policies.
Decision: Reject