Package: zjkdemo2
Deep Reinforcement LearningShyamal H Anadkat | Fall '21BackgroundHello! This is a repository for AIPI530 DeepRL final project.The goal is to build a pipeline for offline RL. The
starter code has been forked fromd3rlpy(see citation at the bottom)
Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data.Before diving in, I would recommend getting familiarized with basic Reinforcement Learning.
Here is a link to my blog post on Reinforcement Learning to get you started:RL PrimerThe blog post briefly covers the following:What is reinforcement learning ?What are the pros and cons of reinforcement learning ?When should we consider applying reinforcement learning (and when should not) ?What's the difference between supervised learning and reinforcement learning ?What is offline reinforcement learning ? What are the pros and cons of offline reinforcement learning ?When should we consider applying offline reinforcement learning (and when should not) ?Have an example of offline reinforcement learning in the real-worldsource:https://bair.berkeley.edu/blog/2020/12/07/offline/Getting Started(please read carefully)This project is customized to training CQL on a custom dataset in d3rlpy, and training OPE (FQE) to evaluate the trained
policy. Important scripts:cql_train.py: at the root of the project is the main script, used to train cql & get evaluation scoresplot_helper.py: utility script to help produce the plots requiredHow do I install & run this project ?1. Clone this repositorygit clone https://github.com/shyamal-anadkat/offlinerl2. Installpybulletfrom source:pip install git+https://github.com/takuseno/d4rl-pybullet3. Install requirements:pip install Cython numpy 
pip install -e .Executecql_train.pyfound at the root of the projectDefault dataset ishopper-bullet-mixed-v0Default no. ofepochsis10. You can change this via custom args--epochs_cql&--epochs_fqeFor example if we want to run for 10 epochs:python cql_train.py --epochs_cql 10 --epochs_fqe 10(see colab example below for more clarity)Important Logs:Estimated Q values vs training steps (CQL):d3rlpy_logs/CQL_hopper-bullet-mixed-v0_1/init_value.csvAverage reward vs training steps (CQL):d3rlpy_logs/CQL_hopper-bullet-mixed-v0_1/environment.csvTrue Q values vs training steps (CQL):d3rlpy_logs/CQL_hopper-bullet-mixed-v0_1/true_q_value.csvTrue Q & Estimated Q values vs training steps (FQE):d3rlpy_logs/FQE_hopper-bullet-mixed-v0_1/..Note:I created my own scorer to calculate the true q values. Seescorer.py(true_q_value_scorer) for
implementation details)For plotting, I wrote a utility script (at root of the project) which can be executed like sopython plot_helper.pyNote: you can provide arguments that correspond to the path to the logs or it will use the default.If you're curious here's
thebenchmark/reproductionOther scripts:Format:./scripts/formatLinting:./scripts/lintSample Plots (with 100 epochs):Note: logs can be found in/d3rlpy_logsExamples speak more:Walkthrough:Background on d3rlpyd3rlpy is an offline deep reinforcement learning library for practitioners and researchers.Documentation:https://d3rlpy.readthedocs.ioPaper:https://arxiv.org/abs/2111.03788How do I install d3rlpy?d3rlpy supports Linux, macOS and Windows. d3rlpy is not only easy, but also completely compatible with scikit-learn API,
which means that you can maximize your productivity with the useful scikit-learn's utilities.PyPI (recommended)$ pip install d3rlpyMore examples around d3rlpy usageimportd3rlpydataset,env=d3rlpy.datasets.get_dataset("hopper-medium-v0")# prepare algorithmsac=d3rlpy.algos.SAC()# train offlinesac.fit(dataset,n_steps=1000000)# train onlinesac.fit_online(env,n_steps=1000000)# ready to controlactions=sac.predict(x)MuJoCoimportd3rlpy# prepare datasetdataset,env=d3rlpy.datasets.get_d4rl('hopper-medium-v0')# prepare algorithmcql=d3rlpy.algos.CQL(use_gpu=True)# traincql.fit(dataset,eval_episodes=dataset,n_epochs=100,scorers={'environment':d3rlpy.metrics.evaluate_on_environment(env),'td_error':d3rlpy.metrics.td_error_scorer})See more datasets atd4rl.Atari 2600importd3rlpyfromsklearn.model_selectionimporttrain_test_split# prepare datasetdataset,env=d3rlpy.datasets.get_atari('breakout-expert-v0')# split datasettrain_episodes,test_episodes=train_test_split(dataset,test_size=0.1)# prepare algorithmcql=d3rlpy.algos.DiscreteCQL(n_frames=4,q_func_factory='qr',scaler='pixel',use_gpu=True)# start trainingcql.fit(train_episodes,eval_episodes=test_episodes,n_epochs=100,scorers={'environment':d3rlpy.metrics.evaluate_on_environment(env),'td_error':d3rlpy.metrics.td_error_scorer})See more Atari datasets atd4rl-atari.PyBulletimportd3rlpy# prepare datasetdataset,env=d3rlpy.datasets.get_pybullet('hopper-bullet-mixed-v0')# prepare algorithmcql=d3rlpy.algos.CQL(use_gpu=True)# start trainingcql.fit(dataset,eval_episodes=dataset,n_epochs=100,scorers={'environment':d3rlpy.metrics.evaluate_on_environment(env),'td_error':d3rlpy.metrics.td_error_scorer})See more PyBullet datasets atd4rl-pybullet.How about some tutorials?Try a cartpole example on Google Colaboratory:official offline RL
tutorial:CitationThanks toTakuma Senoand his work ond3rlpyThis wouldn't have been possible without it.Seno, T., & Imai, M. (2021). d3rlpy: An Offline Deep Reinforcement Learning LibraryConference paper. 35th Conference on Neural Information Processing Systems, Offline Reinforcement Learning Workshop, 2021@InProceedings{seno2021d3rlpy,
  author = {Takuma Seno, Michita Imai},
  title = {d3rlpy: An Offline Deep Reinforcement Library},
  booktitle = {NeurIPS 2021 Offline Reinforcement Learning Workshop},
  month = {December},
  year = {2021}
}