Package: d3rlpy
d3rlpy: An offline deep reinforcement learning libraryd3rlpy is an offline deep reinforcement learning library for practitioners and researchers.importd3rlpydataset,env=d3rlpy.datasets.get_dataset("hopper-medium-v0")# prepare algorithmsac=d3rlpy.algos.SACConfig().create(device="cuda:0")# train offlinesac.fit(dataset,n_steps=1000000)# train onlinesac.fit_online(env,n_steps=1000000)# ready to controlactions=sac.predict(x)Documentation:https://d3rlpy.readthedocs.ioPaper:https://arxiv.org/abs/2111.03788[!IMPORTANT]
v2.x.x introduces breaking changes. If you still stick to v1.x.x, please explicitly install previous versions (e.g.pip install d3rlpy==1.1.1).Key features:zap: Most Practical RL Library Everoffline RL: d3rlpy supports state-of-the-art offline RL algorithms. Offline RL is extremely powerful when the online interaction is not feasible during training (e.g. robotics, medical).online RL: d3rlpy also supports conventional state-of-the-art online training algorithms without any compromising, which means that you can solve any kinds of RL problems only withd3rlpy.:beginner: User-friendly APIzero-knowledge of DL library: d3rlpy provides many state-of-the-art algorithms through intuitive APIs. You can become a RL engineer even without knowing how to use deep learning libraries.extensive documentation: d3rlpy is fully documented and accompanied with tutorials and reproduction scripts of the original papers.:rocket: Beyond State-of-the-artdistributional Q function: d3rlpy is the first library that supports distributional Q functions in the all algorithms. The distributional Q function is known as the very powerful method to achieve the state-of-the-performance.Installationd3rlpy supports Linux, macOS and Windows.PyPI (recommended)$ pip install d3rlpyAnaconda$ conda install conda-forge/noarch::d3rlpyDocker$ docker run -it --gpus all --name d3rlpy takuseno/d3rlpy:latest bashSupported algorithmsalgorithmdiscrete controlcontinuous controlBehavior Cloning (supervised learning):white_check_mark::white_check_mark:Neural Fitted Q Iteration (NFQ):white_check_mark::no_entry:Deep Q-Network (DQN):white_check_mark::no_entry:Double DQN:white_check_mark::no_entry:Deep Deterministic Policy Gradients (DDPG):no_entry::white_check_mark:Twin Delayed Deep Deterministic Policy Gradients (TD3):no_entry::white_check_mark:Soft Actor-Critic (SAC):white_check_mark::white_check_mark:Batch Constrained Q-learning (BCQ):white_check_mark::white_check_mark:Bootstrapping Error Accumulation Reduction (BEAR):no_entry::white_check_mark:Conservative Q-Learning (CQL):white_check_mark::white_check_mark:Advantage Weighted Actor-Critic (AWAC):no_entry::white_check_mark:Critic Reguralized Regression (CRR):no_entry::white_check_mark:Policy in Latent Action Space (PLAS):no_entry::white_check_mark:TD3+BC:no_entry::white_check_mark:Implicit Q-Learning (IQL):no_entry::white_check_mark:Decision Transformer:white_check_mark::white_check_mark:Gato:construction::construction:Supported Q functionsstandard Q functionQuantile RegressionImplicit Quantile NetworkBenchmark resultsd3rlpy is benchmarked to ensure the implementation quality.
The benchmark scripts are availablereproductionsdirectory.
The benchmark results are availabled3rlpy-benchmarksrepository.ExamplesMuJoCoimportd3rlpy# prepare datasetdataset,env=d3rlpy.datasets.get_d4rl('hopper-medium-v0')# prepare algorithmcql=d3rlpy.algos.CQLConfig().create(device='cuda:0')# traincql.fit(dataset,n_steps=100000,evaluators={"environment":d3rlpy.metrics.EnvironmentEvaluator(env)},)See more datasets atd4rl.Atari 2600importd3rlpy# prepare dataset (1% dataset)dataset,env=d3rlpy.datasets.get_atari_transitions('breakout',fraction=0.01,num_stack=4,)# prepare algorithmcql=d3rlpy.algos.DiscreteCQLConfig(observation_scaler=d3rlpy.preprocessing.PixelObservationScaler(),reward_scaler=d3rlpy.preprocessing.ClipRewardScaler(-1.0,1.0),).create(device='cuda:0')# start trainingcql.fit(dataset,n_steps=1000000,evaluators={"environment":d3rlpy.metrics.EnvironmentEvaluator(env,epsilon=0.001)},)See more Atari datasets atd4rl-atari.Online Trainingimportd3rlpyimportgym# prepare environmentenv=gym.make('Hopper-v3')eval_env=gym.make('Hopper-v3')# prepare algorithmsac=d3rlpy.algos.SACConfig().create(device='cuda:0')# prepare replay bufferbuffer=d3rlpy.dataset.create_fifo_replay_buffer(limit=1000000,env=env)# start trainingsac.fit_online(env,buffer,n_steps=1000000,eval_env=eval_env)TutorialsTry cartpole examples on Google Colaboratory!offline RL tutorial:online RL tutorial:More tutorial documentations are availablehere.ContributionsAny kind of contribution to d3rlpy would be highly appreciated!
Please check thecontribution guide.CommunityChannelLinkIssuesGitHub IssuesProjects using d3rlpyProjectDescriptionMINERVAAn out-of-the-box GUI tool for offline RLSCOPE-RLAn off-policy evaluation and selection libraryRoadmapThe roadmap to the future release is available inROADMAP.md.CitationThe paper is availablehere.@article{d3rlpy,
  author  = {Takuma Seno and Michita Imai},
  title   = {d3rlpy: An Offline Deep Reinforcement Learning Library},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {315},
  pages   = {1--20},
  url     = {http://jmlr.org/papers/v23/22-0017.html}
}AcknowledgementThis work started as a part ofTakuma Seno's Ph.D project at Keio University in 2020.This work is supported by Information-technology Promotion Agency, Japan
(IPA), Exploratory IT Human Resources Project (MITOU Program) in the fiscal
year 2020.