Package: tianshou
⚠️️Transition to Gymnasium: The maintainers of OpenAI Gym have recently releasedGymnasium,
which is where future maintenance of OpenAI Gym will be taking place.
Tianshou has transitioned to internally using Gymnasium environments. You can still use OpenAI Gym environments with
Tianshou vector environments, but they will be wrapped in a compatibility layer, which could be a source of issues.
We recommend that you update your environment code to Gymnasium. If you want to continue using OpenAI Gym with
Tianshou, you need to manually install Gym andShimmy(the compatibility layer).Tianshou(天授) is a reinforcement learning platform based on pure PyTorch. Unlike existing reinforcement learning libraries, which are mainly based on TensorFlow, have many nested classes, unfriendly API, or slow-speed, Tianshou provides a fast-speed modularized framework and pythonic API for building the deep reinforcement learning agent with the least number of lines of code. The supported interface algorithms currently include:Deep Q-Network (DQN)Double DQNDueling DQNBranching DQNCategorical DQN (C51)Rainbow DQN (Rainbow)Quantile Regression DQN (QRDQN)Implicit Quantile Network (IQN)Fully-parameterized Quantile Function (FQF)Policy Gradient (PG)Natural Policy Gradient (NPG)Advantage Actor-Critic (A2C)Trust Region Policy Optimization (TRPO)Proximal Policy Optimization (PPO)Deep Deterministic Policy Gradient (DDPG)Twin Delayed DDPG (TD3)Soft Actor-Critic (SAC)Randomized Ensembled Double Q-Learning (REDQ)Discrete Soft Actor-Critic (SAC-Discrete)Vanilla Imitation LearningBatch-Constrained deep Q-Learning (BCQ)Conservative Q-Learning (CQL)Twin Delayed DDPG with Behavior Cloning (TD3+BC)Discrete Batch-Constrained deep Q-Learning (BCQ-Discrete)Discrete Conservative Q-Learning (CQL-Discrete)Discrete Critic Regularized Regression (CRR-Discrete)Generative Adversarial Imitation Learning (GAIL)Prioritized Experience Replay (PER)Generalized Advantage Estimator (GAE)Posterior Sampling Reinforcement Learning (PSRL)Intrinsic Curiosity Module (ICM)Hindsight Experience Replay (HER)Here are Tianshou's other features:Elegant framework, using only ~4000 lines of codeState-of-the-artMuJoCo benchmarkfor REINFORCE/A2C/TRPO/PPO/DDPG/TD3/SAC algorithmsSupport vectorized environment (synchronous or asynchronous) for all algorithmsUsageSupport super-fast vectorized environmentEnvPoolfor all algorithmsUsageSupport recurrent state representation in actor network and critic network (RNN-style training for POMDP)UsageSupport any type of environment state/action (e.g. a dict, a self-defined class, ...)UsageSupport customized training processUsageSupport n-step returns estimation and prioritized experience replay for all Q-learning based algorithms; GAE, nstep and PER are very fast thanks to numba jit function and vectorized numpy operationSupport multi-agent RLUsageSupport bothTensorBoardandW&Blog toolsSupport multi-GPU trainingUsageComprehensive documentation, PEP8 code-style checking, type checking and thoroughtestsIn Chinese, Tianshou means divinely ordained and is derived to the gift of being born with. Tianshou is a reinforcement learning platform, and the RL algorithm does not learn from humans. So taking "Tianshou" means that there is no teacher to study with, but rather to learn by themselves through constant interaction with the environment.“天授”意指上天所授，引申为与生具有的天赋。天授是强化学习平台，而强化学习算法并不是向人类学习的，所以取“天授”意思是没有老师来教，而是自己通过跟环境不断交互来进行学习。InstallationTianshou is currently hosted onPyPIandconda-forge. It requires Python >= 3.8.You can simply install Tianshou from PyPI with the following command:$pipinstalltianshouIf you use Anaconda or Miniconda, you can install Tianshou from conda-forge through the following command:$condainstalltianshou-cconda-forgeYou can also install with the newest version through GitHub:$pipinstallgit+https://github.com/thu-ml/tianshou.git@master--upgradeAfter installation, open your python console and typeimporttianshouprint(tianshou.__version__)If no error occurs, you have successfully installed Tianshou.DocumentationThe tutorials and API documentation are hosted ontianshou.readthedocs.io.The example scripts are undertest/folder andexamples/folder.中文文档位于https://tianshou.readthedocs.io/zh/master/。Why Tianshou?Comprehensive FunctionalityRL PlatformGitHub Stars# of Alg.(1)Custom EnvBatch TrainingRNN SupportNested ObservationBackendBaselines9:heavy_check_mark: (gym):heavy_minus_sign:(2):heavy_check_mark::x:TF1Stable-Baselines11:heavy_check_mark: (gym):heavy_minus_sign:(2):heavy_check_mark::x:TF1Stable-Baselines37(3):heavy_check_mark: (gym):heavy_minus_sign:(2):x::heavy_check_mark:PyTorchRay/RLlib16:heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:TF/PyTorchSpinningUp6:heavy_check_mark: (gym):heavy_minus_sign:(2):x::x:PyTorchDopamine7:x::x::x::x:TF/JAXACME14:heavy_check_mark: (dm_env):heavy_check_mark::heavy_check_mark::heavy_check_mark:TF/JAXkeras-rl7:heavy_check_mark: (gym):x::x::x:Kerasrlpyt11:x::heavy_check_mark::heavy_check_mark::heavy_check_mark:PyTorchChainerRL18:heavy_check_mark: (gym):heavy_check_mark::heavy_check_mark::x:ChainerSample Factory1(4):heavy_check_mark: (gym):heavy_check_mark::heavy_check_mark::heavy_check_mark:PyTorchTianshou20:heavy_check_mark: (Gymnasium):heavy_check_mark::heavy_check_mark::heavy_check_mark:PyTorch(1): access date: 2021-08-08(2): not all algorithms support this feature(3): TQC and QR-DQN insb3-contribinstead of main repo(4): super fast APPO!High quality software engineering standardRL PlatformDocumentationCode CoverageType HintsLast UpdateBaselines:x::x::x:Stable-Baselines:x:Stable-Baselines3:heavy_check_mark:Ray/RLlib:heavy_minus_sign:(1):heavy_check_mark:SpinningUp:x::x:Dopamine:x::x:ACME:heavy_minus_sign:(1):heavy_check_mark:keras-rl:heavy_minus_sign:(1):x:rlpyt:x:ChainerRL:x:Sample Factory:heavy_minus_sign::x:Tianshou:heavy_check_mark:(1): it has continuous integration but the coverage rate is not availableReproducible and High Quality ResultTianshou has its tests. Different from other platforms,the tests include the full agent training procedure for all of the implemented algorithms. It would be failed once if it could not train an agent to perform well enough on limited epochs on toy scenarios. The tests secure the reproducibility of our platform. Check out theGitHub Actionspage for more detail.The Atari/Mujoco benchmark results are underexamples/atari/andexamples/mujoco/folders.Our Mujoco result can beat most of existing benchmark.Modularized PolicyWe decouple all of the algorithms roughly into the following parts:__init__: initialize the policy;forward: to compute actions over given observations;process_fn: to preprocess data from replay buffer (since we have reformulated all algorithms to replay-buffer based algorithms);learn: to learn from a given batch data;post_process_fn: to update the replay buffer from the learning process (e.g., prioritized replay buffer needs to update the weight);update: the main interface for training, i.e.,process_fn -> learn -> post_process_fn.Within this API, we can interact with different policies conveniently.Quick StartThis is an example of Deep Q Network. You can also run the full script attest/discrete/test_dqn.py.First, import some relevant packages:importgymnasiumasgymimporttorch,numpyasnp,torch.nnasnnfromtorch.utils.tensorboardimportSummaryWriterimporttianshouastsDefine some hyper-parameters:task='CartPole-v0'lr,epoch,batch_size=1e-3,10,64train_num,test_num=10,100gamma,n_step,target_freq=0.9,3,320buffer_size=20000eps_train,eps_test=0.1,0.05step_per_epoch,step_per_collect=10000,10logger=ts.utils.TensorboardLogger(SummaryWriter('log/dqn'))# TensorBoard is supported!# For other loggers: https://tianshou.readthedocs.io/en/master/tutorials/logger.htmlMake environments:# you can also try with SubprocVectorEnvtrain_envs=ts.env.DummyVectorEnv([lambda:gym.make(task)for_inrange(train_num)])test_envs=ts.env.DummyVectorEnv([lambda:gym.make(task)for_inrange(test_num)])Define the network:fromtianshou.utils.net.commonimportNet# you can define other net by following the API:# https://tianshou.readthedocs.io/en/master/tutorials/dqn.html#build-the-networkenv=gym.make(task)state_shape=env.observation_space.shapeorenv.observation_space.naction_shape=env.action_space.shapeorenv.action_space.nnet=Net(state_shape=state_shape,action_shape=action_shape,hidden_sizes=[128,128,128])optim=torch.optim.Adam(net.parameters(),lr=lr)Setup policy and collectors:policy=ts.policy.DQNPolicy(net,optim,gamma,n_step,target_update_freq=target_freq)train_collector=ts.data.Collector(policy,train_envs,ts.data.VectorReplayBuffer(buffer_size,train_num),exploration_noise=True)test_collector=ts.data.Collector(policy,test_envs,exploration_noise=True)# because DQN uses epsilon-greedy methodLet's train it:result=ts.trainer.offpolicy_trainer(policy,train_collector,test_collector,epoch,step_per_epoch,step_per_collect,test_num,batch_size,update_per_step=1/step_per_collect,train_fn=lambdaepoch,env_step:policy.set_eps(eps_train),test_fn=lambdaepoch,env_step:policy.set_eps(eps_test),stop_fn=lambdamean_rewards:mean_rewards>=env.spec.reward_threshold,logger=logger)print(f'Finished training! Use{result["duration"]}')Save / load the trained policy (it's exactly the same as PyTorchnn.module):torch.save(policy.state_dict(),'dqn.pth')policy.load_state_dict(torch.load('dqn.pth'))Watch the performance with 35 FPS:policy.eval()policy.set_eps(eps_test)collector=ts.data.Collector(policy,env,exploration_noise=True)collector.collect(n_episode=1,render=1/35)Look at the result saved in tensorboard: (with bash script in your terminal)$tensorboard--logdirlog/dqnYou can check out thedocumentationfor advanced usage.It's worth a try: here is a test on a laptop (i7-8750H + GTX1060). It only uses3seconds for training an agent based on vanilla policy gradient on the CartPole-v0 task: (seed may be different across different platform and device)$python3test/discrete/test_pg.py--seed0--render0.03ContributingTianshou is still under development. More algorithms and features are going to be added and we always welcome contributions to help make Tianshou better. If you would like to contribute, please check outthis link.Citing TianshouIf you find Tianshou useful, please cite it in your publications.@article{tianshou,
  author  ={Jiayi Weng and Huayu Chen and Dong Yan and Kaichao You and Alexis Duburcq and Minghao Zhang and Yi Su and Hang Su and Jun Zhu},
  title   ={Tianshou: A Highly Modularized Deep Reinforcement Learning Library},
  journal ={Journal of Machine Learning Research},
  year    ={2022},
  volume  ={23},
  number  ={267},
  pages   ={1--6},
  url     ={http://jmlr.org/papers/v23/21-1127.html}}AcknowledgmentTianshou was previously a reinforcement learning platform based on TensorFlow. You can check out the branchprivfor more detail. Many thanks toHaosheng Zou's pioneering work for Tianshou before version 0.1.1.We would like to thankTSAILandInstitute for Artificial Intelligence, Tsinghua Universityfor providing such an excellent AI research platform.