Package: rllte-core
RLLTE: Long-Term Evolution Project of Reinforcement LearningPaper|Documentation|Tutorials|Forum|Benchmarks|English|中文|ContentsOverviewQuick StartInstallationFast Training with Built-in AlgorithmsOn NVIDIA GPUOn HUAWEI NPUThree Steps to Create Your RL AgentAlgorithm Decoupling and Module ReplacementFunction List (Part)RL AgentsIntrinsic Reward ModulesRLLTE EcosystemAPI DocumentationCite the ProjectHow To ContributeAcknowledgmentMiscellaneousOverviewInspired by the long-term evolution (LTE) standard project in telecommunications, aiming to provide development components for and standards for advancing RL research and applications. Beyond delivering top-notch algorithm implementations,RLLTEalso serves as atoolkitfor developing algorithms.An introduction to RLLTE.WhyRLLTE?🧬 Long-term evolution for providing latest algorithms and tricks;🏞️ Complete ecosystem for task design, model training, evaluation, and deployment (TensorRT, CANN, ...);🧱 Module-oriented design for complete decoupling of RL algorithms;🚀 Optimized workflow for full hardware acceleration;⚙️ Support custom environments and modules;🖥️ Support multiple computing devices like GPU and NPU;💾 Large number of reusable benchmarks (RLLTE Hub);👨‍✈️ Large language model-empowered copilot (RLLTE Copilot).⚠️ Since the construction of RLLTE Hub requires massive computing power, we have to upload the training datasets and model weights gradually. Progress report can be found inIssue#30.See the project structure below:For more detailed descriptions of these modules, seeAPI Documentation.Quick StartInstallationPrerequisitesCurrently, we recommendPython>=3.8, and user can create an virtual environment bycondacreate-nrlltepython=3.8with piprecommendedOpen a terminal and installrlltewithpip:pipinstallrllte-core# basic installationpipinstallrllte-core[envs]# for pre-defined environmentswith gitOpen a terminal and clone the repository fromGitHubwithgit:gitclonehttps://github.com/RLE-Foundation/rllte.gitAfter that, run the following command to install package and dependencies:pipinstall-e.# basic installationpipinstall-e.[envs]# for pre-defined environmentsFor more detailed installation instruction, seeGetting Started.Fast Training with Built-in AlgorithmsRLLTEprovides implementations for well-recognized RL algorithms and simple interface for building applications.On NVIDIA GPUSuppose we want to useDrQ-v2to solve a task ofDeepMind Control Suite, and it suffices to write atrain.pylike:# import `env` and `agent` modulefromrllte.envimportmake_dmc_envfromrllte.agentimportDrQv2if__name__=="__main__":device="cuda:0"# create env, `eval_env` is optionalenv=make_dmc_env(env_id="cartpole_balance",device=device)eval_env=make_dmc_env(env_id="cartpole_balance",device=device)# create agentagent=DrQv2(env=env,eval_env=eval_env,device=device,tag="drqv2_dmc_pixel")# start trainingagent.train(num_train_steps=500000,log_interval=1000)Runtrain.pyand you will see the following output:On HUAWEI NPUSimilarly, if we want to train an agent on HUAWEI NPU, it suffices to replacecudawithnpu:device="cuda:0"->device="npu:0"Three Steps to Create Your RL AgentDevelopers only need three steps to implement an RL algorithm withRLLTE. The following example illustrates how to write an Advantage Actor-Critic (A2C) agent to solve Atari games.Firstly, select a prototype:fromrllte.common.prototypeimportOnPolicyAgentSecondly, select necessary modules to build the agent:fromrllte.xploit.encoderimportMnihCnnEncoderfromrllte.xploit.policyimportOnPolicySharedActorCriticfromrllte.xploit.storageimportVanillaRolloutStoragefromrllte.xplore.distributionimportCategoricalRun the.describefunction of the selected policy and you will see the following output:OnPolicySharedActorCritic.describe()# Output:# ================================================================================# Name       : OnPolicySharedActorCritic# Structure  : self.encoder (shared by actor and critic), self.actor, self.critic# Forward    : obs -> self.encoder -> self.actor -> actions#            : obs -> self.encoder -> self.critic -> values#            : actions -> log_probs# Optimizers : self.optimizers['opt'] -> (self.encoder, self.actor, self.critic)# ================================================================================This will illustrate the structure of the policy and indicate the optimizable parts. Finally, merge these modules and write an.updatefunction:fromtorchimportnnimporttorchasthclassA2C(OnPolicyAgent):def__init__(self,env,tag,seed,device,num_steps)->None:super().__init__(env=env,tag=tag,seed=seed,device=device,num_steps=num_steps)# create modulesencoder=MnihCnnEncoder(observation_space=env.observation_space,feature_dim=512)policy=OnPolicySharedActorCritic(observation_space=env.observation_space,action_space=env.action_space,feature_dim=512,opt_class=th.optim.Adam,opt_kwargs=dict(lr=2.5e-4,eps=1e-5),init_fn="xavier_uniform")storage=VanillaRolloutStorage(observation_space=env.observation_space,action_space=env.action_space,device=device,storage_size=self.num_steps,num_envs=self.num_envs,batch_size=256)dist=Categorical()# set all the modulesself.set(encoder=encoder,policy=policy,storage=storage,distribution=dist)defupdate(self):for_inrange(4):forbatchinself.storage.sample():# evaluate the sampled actionsnew_values,new_log_probs,entropy=self.policy.evaluate_actions(obs=batch.observations,actions=batch.actions)# policy loss partpolicy_loss=-(batch.adv_targ*new_log_probs).mean()# value loss partvalue_loss=0.5*(new_values.flatten()-batch.returns).pow(2).mean()# updateself.policy.optimizers['opt'].zero_grad(set_to_none=True)(value_loss*0.5+policy_loss-entropy*0.01).backward()nn.utils.clip_grad_norm_(self.policy.parameters(),0.5)self.policy.optimizers['opt'].step()Then train the agent byfromrllte.envimportmake_atari_envif__name__=="__main__":device="cuda"env=make_atari_env("PongNoFrameskip-v4",num_envs=8,seed=0,device=device)agent=A2C(env=env,tag="a2c_atari",seed=0,device=device,num_steps=128)agent.train(num_train_steps=10000000)As shown in this example, only a few dozen lines of code are needed to create RL agents withRLLTE.Algorithm Decoupling and Module ReplacementRLLTEallows developers to replace settled modules of implemented algorithms to make performance comparison and algorithm improvement, and both
built-in and custom modules are supported. Suppose we want to compare the effect of different encoders, it suffices to invoke the.setfunction:fromrllte.xploit.encoderimportEspeholtResidualEncoderencoder=EspeholtResidualEncoder(...)agent.set(encoder=encoder)RLLTEis an extremely open framework that allows developers to try anything. For more detailed tutorials, seeTutorials.Function List (Part)RL AgentsTypeAlgo.BoxDis.M.B.M.D.M.P.NPU💰🔭On-PolicyA2C✔️✔️✔️✔️✔️✔️✔️❌On-PolicyPPO✔️✔️✔️✔️✔️✔️✔️❌On-PolicyDrAC✔️✔️✔️✔️✔️✔️✔️✔️On-PolicyDAAC✔️✔️✔️✔️✔️✔️✔️❌On-PolicyDrDAAC✔️✔️✔️✔️✔️✔️✔️✔️On-PolicyPPG✔️✔️✔️❌✔️✔️✔️❌Off-PolicyDQN✔️❌❌❌✔️✔️✔️❌Off-PolicyDDPG✔️❌❌❌✔️✔️✔️❌Off-PolicySAC✔️❌❌❌✔️✔️✔️❌Off-PolicySAC-Discrete❌✔️❌❌✔️✔️✔️❌Off-PolicyTD3✔️❌❌❌✔️✔️✔️❌Off-PolicyDrQ-v2✔️❌❌❌❌✔️✔️✔️DistributedIMPALA✔️✔️❌❌✔️❌❌❌Dis., M.B., M.D.:Discrete,MultiBinary, andMultiDiscreteaction space;M.P.: Multi processing;🐌: Developing;💰: Support intrinsic reward shaping;🔭: Support observation augmentation.Intrinsic Reward ModulesTypeModulesCount-basedPseudoCounts,RNDCuriosity-drivenICM,GIRM,RIDEMemory-basedNGUInformation theory-basedRE3,RISE,REVDSeeTutorials: Use Intrinsic Reward and Observation Augmentationfor usage examples.RLLTE EcosystemExplore the ecosystem of RLLTE to facilitate your project:Hub: Fast training APIs and reusable benchmarks.Evaluation: Reasonable and reliable metrics for algorithm evaluation.Env: Packaged environments for fast invocation.Deployment: Convenient APIs for model deployment.Pre-training: Methods of pre-training in RL.Copilot: Large language model-empowered copilot.API DocumentationView our well-designed documentation:https://docs.rllte.dev/How To ContributeWelcome to contribute to this project! Before you begin writing code, please readCONTRIBUTING.mdfor guide first.Cite the ProjectIf you useRLLTEin your research, please cite this project like this:@article{yuan2023rllte,title={RLLTE: Long-Term Evolution Project of Reinforcement Learning},author={Mingqi Yuan and Zequn Zhang and Yang Xu and Shihao Luo and Bo Li and Xin Jin and Wenjun Zeng},year={2023},journal={arXiv preprint arXiv:2309.16382}}AcknowledgmentThis project is supported byThe Hong Kong Polytechnic University,Eastern Institute for Advanced Study, andFLW-Foundation.EIAS HPCprovides a GPU computing platform, andHUAWEI Ascend Communityprovides an NPU computing platform for our testing. Some code of this project is borrowed or inspired by several excellent projects, and we highly appreciate them. SeeACKNOWLEDGMENT.md.Miscellaneous↳ Stargazers, thanks for your support!↳ Forkers, thanks for your support!↳ Star History