Package: emote-rl
🍒 emoteEmbark'sModularTrainingEngine - a flexible framework for
reinforcement learning🚧 This project is very muchwork in progress and not yet ready for production use.🚧What it doesEmote provides a way to build reusable components for creating reinforcement learning algorithms, and a
library of premade componenents built in this way. It is strongly inspired by the callback setup used
by Keras and FastAI.As an example, let us see how the SAC, the Soft Actor Critic algorithm byHaarnoja et al.can be written using Emote. The main algorithm in
SAC is given inSoft Actor-Critic Algorithms and Applicationsand
looks like this:Using the components provided with Emote, we can write this asdevice=torch.device("cpu")env=DictGymWrapper(AsyncVectorEnv(10*[HitTheMiddle]))table=DictObsTable(spaces=env.dict_space,maxlen=1000,device=device)memory_proxy=TableMemoryProxy(table)dataloader=MemoryLoader(table,100,2,"batch_size")q1=QNet(2,1)q2=QNet(2,1)policy=Policy(2,1)ln_alpha=torch.tensor(1.0,requires_grad=True)agent_proxy=FeatureAgentProxy(policy,device)callbacks=[QLoss(name="q1",q=q1,opt=Adam(q1.parameters(),lr=8e-3)),QLoss(name="q2",q=q2,opt=Adam(q2.parameters(),lr=8e-3)),PolicyLoss(pi=policy,ln_alpha=ln_alpha,q=q1,opt=Adam(policy.parameters())),AlphaLoss(pi=policy,ln_alpha=ln_alpha,opt=Adam([ln_alpha]),n_actions=1),QTarget(pi=policy,ln_alpha=ln_alpha,q1=q1,q2=q2),SimpleGymCollector(env,agent_proxy,memory_proxy,warmup_steps=500),FinalLossTestCheck([logged_cbs[2]],[10.0],2000),]trainer=Trainer(callbacks,dataloader)trainer.train()Here each callback in thecallbackslist is its own reusable class that can readily be used
for other similar algorithms. The callback classes themselves are very straight forward to write.
As an example, here is thePolicyLosscallback.classPolicyLoss(LossCallback):def__init__(self,*,pi:nn.Module,ln_alpha:torch.tensor,q:nn.Module,opt:optim.Optimizer,max_grad_norm:float=10.0,name:str="policy",data_group:str="default",):super().__init__(name=name,optimizer=opt,network=pi,max_grad_norm=max_grad_norm,data_group=data_group,)self.policy=piself._ln_alpha=ln_alphaself.q1=qself.q2=q2defloss(self,observation):p_sample,logp_pi=self.policy(**observation)q_pi_min=self.q1(p_sample,**observation)# using reparameterization trickalpha=torch.exp(self._ln_alpha).detach()policy_loss=alpha*logp_pi-q_pi_minpolicy_loss=torch.mean(policy_loss)assertpolicy_loss.dim()==0returnpolicy_lossInstallationFor installation and environment handling we usepdm. Install it frompdm. Afterpdmis set up, set up and activate the emote environment by runningpdminstallor for a full developer installation with all the extra dependencies:pdminstall-d-G:allCommon problemsTorch won't install:Check that your python version is correct. Try deleting your.venvand recreating it withpdmvenvcreate3.8
pdminstall-G:allBox2d complains:Box2d needs swig and python bindings. On apt-based systems trysudoaptinstallswig
sudoaptinstallpython3.8-devPython 3.8 is tricky to install:For Ubuntu based distros try adding the deadsnakes PPA.ContributionWe welcome community contributions to this project.Please read ourContributor Guidefor more information on how to get started.
Please also read ourContributor Termsbefore you make any contributions.Any contribution intentionally submitted for inclusion in an Embark Studios project, shall comply with the Rust standard licensing model (MIT OR Apache 2.0) and therefore be dual licensed as described below, without any additional terms or conditions:LicenseThis contribution is dual licensed under EITHER OFApache License, Version 2.0, (LICENSE-APACHEorhttp://www.apache.org/licenses/LICENSE-2.0)MIT license (LICENSE-MITorhttp://opensource.org/licenses/MIT)at your option.For clarity, "your" refers to Embark or any other licensee/user of the contribution.