Introducing completely free datasets for data-driven deep reinforcement learning | by Takuma Seno | Towards Data Science

In this blog post, I’m introducing the new datasets for data-driven deep reinforcement learning, which are available for completely free!
Data-driven reinforcement learning (RL) is a paradigm that RL algorithms achieve policies to maximize rewards within the offline data, unlike online RL that optimizes its policy through exploration and exploitation trials. This data-driven RL is getting attention for its practicality and potential impacts on machine learning systems. Data-driven RL is also called “offline” RL and “batch” RL. There are very nice articles explaining this.
medium.com
bair.berkeley.edu
It is important that we share a standard dataset to evaluate our new algorithms. For example, in the visual processing domain, there are many datasets including ImageNet, which have been pushing machine learning researches.
In data-driven deep RL, there is the d4rl dataset, which is the first standardized dataset for this domain.
github.com
The dataset consists of several tasks.
The dataset is actually great because this dataset already made researchers compare their algorithms in a standardized way. Almost all recently published data-driven RL papers use this dataset.
The d4rl is great, however, there was a discussion on the use of MuJoCo simulator in OpenReview for ICLR2021.
openreview.net
One of the reviewers criticized that most of the d4rl datasets are collected with tasks in the MuJoCo simulator, which prevents many people from joining this domain due to its expensive license. So, even if you train your algorithms with d4rl, you need to get the license of MuJoCo to evaluate your models.
In my opinion, this sort of benchmark should be available with ease of access for reproducibility and openness of the research. I believe that the MuJoCo license is one of the barriers that make it hard for people to work on deep RL. And, many people still need more accessible ones.
Therefore, I’ve released d4rl-pybullet, the completely free version of the d4rl.
github.com
The dataset is collected with PyBullet, which provides the suits of reproductions of MuJoCo tasks. This means that you can train your data-driven RL algorithms with d4rl-pybullet, and directly evaluate them on PyBullet tasks.
Here is the use of this dataset.
import gymimport d4rl_pybullet# dataset will be automatically downloaded into ~/.d4rl/datasetsenv = gym.make('hopper-bullet-mixed-v0')# interaction with its environmentenv.reset()env.step(env.action_space.sample())# access to the datasetdataset = env.get_dataset()dataset['observations'] # observation data in N x dim_observationdataset['actions'] # action data in N x dim_actiondataset['rewards'] # reward data in N x 1dataset['terminals'] # terminal flags in N x 1
The API is compatible with the original d4rl that is very simple and intuitive.
The dataset consists of the 3 categories as d4rl does:
random: the dataset is collected with the random policy.
mixed: the dataset is collected through the Soft Actor-Critic training.
medium: the dataset is collected with the medium-level trained policy.
And, there are 4 tasks in each category:
HopperBulletEnv-v0
AntBulletEnv-v0
Walker2DBulletEnv-v0
HalfCheetahBulletEnv-v0
Overall, there 12 datasets and you can see more details here.
This is an example benchmark result.
As you can see above, the SAC, an online algorithm, does not learn well due to the overestimation problem. The CQL, a state-of-the-art data-driven RL algorithm, successfully achieves better performance than the best episode in the dataset.
I’ve also released a useful wrapper for Atari datasets released by Google.
github.com
The Atari dataset is also often used to evaluate data-driven algorithms for discrete-action space. However, you’ll notice that you need to manually download the dataset, and it’s designed for the dopamine framework.
Thus, I’ve made this. d4rl-atari provides d4rl-style API, and you can start your experiments without any effort. Here is the use of d4rl-atari.
import gymimport d4rl_atarienv = gym.make('breakout-mixed-v0', stack=False) # -v{0, 1, 2, 3, 4} for datasets with the other random seeds# interaction with its environment through dopamine-style Atari wrapperenv.reset() # observation is resized to 1x84x84env.step(env.action_space.sample())# dataset will be automatically downloaded into ~/.d4rl/datasets/[GAME]/[INDEX]/[EPOCH]dataset = env.get_dataset()dataset['observations'] # observation data in 1000000x1x84x84dataset['actions'] # action data in 1Mdataset['rewards'] # reward data in 1Mdataset['terminals'] # terminal flags in 1M
The greatest thing is that the dataset is automatically downloaded from the Google repository, and if it does already exist, it will be loaded from local storage. Also, it’s extremely easy to use and it’s designed in a framework-agnostic way.
Finally, d4rl-atari provides a useful Atari wrapper that does frame skipping , random initialization andtermination on loss of life , which are standardized procedures across deep RL researches.
This is an example benchmark result with the data consisting of the last 1M steps of Breakout. The CQL achieves much better performance than the QR-DDQN.
I’ve introduced the 2 dataset libraries, d4rl-pybullet and d4rl-atari. Both of them have very simple and intuitive API, which enables you to become a data-driven researcher today even if you’re not affodable to MuJoCo license. Also, I share the sample benchmark result with the state-of-the-art algorithm in each library to show how good they are on the offline training scheme.
It is still unclear if the deep RL researches rely on MuJoCo benchmarking. But, I believe these works can help many people to join this exciting domain and push the research forward. If you have any questions and requests, please contact me on GitHub. Thank you!