Introducing completely free datasets for data-driven deep reinforcement learning
By: Takuma Seno
Tags: Machine Learning, Reinforcement Learning, Deep Learning

In this blog post, I’m introducing the new datasets for data-driven deep reinforcement learning, which are available for completely free!

What is data-driven deep reinforcement learning?

Data-driven reinforcement learning (RL) is a paradigm that RL algorithms achieve policies to maximize rewards within the offline data, unlike online RL that optimizes its policy through exploration and exploitation trials. This data-driven RL is getting attention for its practicality and potential impacts on machine learning systems. Data-driven RL is also called “offline” RL and “batch” RL. There are very nice articles explaining this.

Datasets for benchmarking data-driven RL

It is important that we share a standard dataset to evaluate our new algorithms. For example, in the visual processing domain, there are many datasets including ImageNet, which have been pushing machine learning researches.

In data-driven deep RL, there is the d4rl dataset, which is the first standardized dataset for this domain.

The dataset consists of several tasks.

Figure 1 from https://arxiv.org/abs/2004.07219

The dataset is actually great because this dataset already made researchers compare their algorithms in a standardized way. Almost all recently published data-driven RL papers use this dataset.

Discussion on the use of MuJoCo

The d4rl is great, however, there was a discussion on the use of MuJoCo simulator in OpenReview for ICLR2021.

One of the reviewers criticized that most of the d4rl datasets are collected with tasks in the MuJoCo simulator, which prevents many people from joining this domain due to its expensive license. So, even if you train your algorithms with d4rl, you need to get the license of MuJoCo to evaluate your models.

In my opinion, this sort of benchmark should be available with ease of access for reproducibility and openness of the research. I believe that the MuJoCo license is one of the barriers that make it hard for people to work on deep RL. And, many people still need more accessible ones.

Completely free datasets for benchmarking

Therefore, I’ve released d4rl-pybullet, the completely free version of the d4rl.

The dataset is collected with PyBullet, which provides the suits of reproductions of MuJoCo tasks. This means that you can train your data-driven RL algorithms with d4rl-pybullet, and directly evaluate them on PyBullet tasks.

Here is the use of this dataset.

import gym

import d4rl_pybullet



# dataset will be automatically downloaded into ~/.d4rl/datasets

env = gym.make('hopper-bullet-mixed-v0')



# interaction with its environment

env.reset()

env.step(env.action_space.sample())



# access to the dataset

dataset = env.get_dataset()

dataset['observations'] # observation data in N x dim_observation

dataset['actions'] # action data in N x dim_action

dataset['rewards'] # reward data in N x 1

dataset['terminals'] # terminal flags in N x 1

The API is compatible with the original d4rl that is very simple and intuitive.

The dataset consists of the 3 categories as d4rl does:

random: the dataset is collected with the random policy.

mixed: the dataset is collected through the Soft Actor-Critic training.

medium: the dataset is collected with the medium-level trained policy.

And, there are 4 tasks in each category:

HopperBulletEnv-v0

AntBulletEnv-v0

Walker2DBulletEnv-v0

HalfCheetahBulletEnv-v0

Overall, there 12 datasets and you can see more details here.

This is an example benchmark result.

The evaluation result on hopper-bullet-mixed-v0

As you can see above, the SAC, an online algorithm, does not learn well due to the overestimation problem. The CQL, a state-of-the-art data-driven RL algorithm, successfully achieves better performance than the best episode in the dataset.

Appendix: d4rl-atari

I’ve also released a useful wrapper for Atari datasets released by Google.

The Atari dataset is also often used to evaluate data-driven algorithms for discrete-action space. However, you’ll notice that you need to manually download the dataset, and it’s designed for the dopamine framework.

Thus, I’ve made this. d4rl-atari provides d4rl-style API, and you can start your experiments without any effort. Here is the use of d4rl-atari.

import gym

import d4rl_atari



env = gym.make('breakout-mixed-v0', stack=False) # -v{0, 1, 2, 3, 4} for datasets with the other random seeds



# interaction with its environment through dopamine-style Atari wrapper

env.reset() # observation is resized to 1x84x84

env.step(env.action_space.sample())



# dataset will be automatically downloaded into ~/.d4rl/datasets/[GAME]/[INDEX]/[EPOCH]

dataset = env.get_dataset()

dataset['observations'] # observation data in 1000000x1x84x84

dataset['actions'] # action data in 1M

dataset['rewards'] # reward data in 1M

dataset['terminals'] # terminal flags in 1M

The greatest thing is that the dataset is automatically downloaded from the Google repository, and if it does already exist, it will be loaded from local storage. Also, it’s extremely easy to use and it’s designed in a framework-agnostic way.

Finally, d4rl-atari provides a useful Atari wrapper that does frame skipping , random initialization and termination on loss of life , which are standardized procedures across deep RL researches.

This is an example benchmark result with the data consisting of the last 1M steps of Breakout. The CQL achieves much better performance than the QR-DDQN.

The evaluation result on breakout-expert-v0.

Conclusion

I’ve introduced the 2 dataset libraries, d4rl-pybullet and d4rl-atari. Both of them have very simple and intuitive API, which enables you to become a data-driven researcher today even if you’re not affodable to MuJoCo license. Also, I share the sample benchmark result with the state-of-the-art algorithm in each library to show how good they are on the offline training scheme.

It is still unclear if the deep RL researches rely on MuJoCo benchmarking. But, I believe these works can help many people to join this exciting domain and push the research forward. If you have any questions and requests, please contact me on GitHub. Thank you!