Package: fast-rl
Fast_rlThis repo is not affiliated with Jeremy Howard or his course which can be foundhere.
We will be using components from the Fastai library for building and training our reinforcement learning (RL)
agents.Our goal is for fast_rl to be make benchmarking easier, inference more efficient, and environment compatibility to be
as decoupled as much as possible. This being version 1.0, we still have a lot of work to make RL training itself faster
and more efficient. The goals for this repo can be seen in theRoadMap.An important note is that training can use up a lot of RAM. This will likely be resolved as more models are being added. Likely will be resolved by off loading to storage in the next few versions.A simple example:fromfast_rl.agents.dqnimportcreate_dqn_model,dqn_learnerfromfast_rl.agents.dqn_modelsimport*fromfast_rl.core.agent_coreimportExperienceReplay,GreedyEpsilonfromfast_rl.core.data_blockimportMDPDataBunchfromfast_rl.core.metricsimportRewardMetric,EpsilonMetricmemory=ExperienceReplay(memory_size=1000000,reduce_ram=True)explore=GreedyEpsilon(epsilon_start=1,epsilon_end=0.1,decay=0.001)data=MDPDataBunch.from_env('CartPole-v1',render='human',bs=64,add_valid=False)model=create_dqn_model(data=data,base_arch=FixedTargetDQNModule,lr=0.001,layers=[32,32])learn=dqn_learner(data,model,memory=memory,exploration_method=explore,copy_over_frequency=300,callback_fns=[RewardMetric,EpsilonMetric])learn.fit(450)More complex examples might involve running an RL agent multiple times, generating episode snapshots as gifs, grouping
reward plots, and finally showing the best and worst runs in a single graph.fromfastai.basic_dataimportDatasetTypefromfast_rl.agents.dqnimportcreate_dqn_model,dqn_learnerfromfast_rl.agents.dqn_modelsimport*fromfast_rl.core.agent_coreimportExperienceReplay,GreedyEpsilonfromfast_rl.core.data_blockimportMDPDataBunchfromfast_rl.core.metricsimportRewardMetric,EpsilonMetricfromfast_rl.core.trainimportGroupAgentInterpretation,AgentInterpretationgroup_interp=GroupAgentInterpretation()foriinrange(5):memory=ExperienceReplay(memory_size=1000000,reduce_ram=True)explore=GreedyEpsilon(epsilon_start=1,epsilon_end=0.1,decay=0.001)data=MDPDataBunch.from_env('CartPole-v1',render='human',bs=64,add_valid=False)model=create_dqn_model(data=data,base_arch=FixedTargetDQNModule,lr=0.001,layers=[32,32])learn=dqn_learner(data,model,memory=memory,exploration_method=explore,copy_over_frequency=300,callback_fns=[RewardMetric,EpsilonMetric])learn.fit(450)interp=AgentInterpretation(learn,ds_type=DatasetType.Train)interp.plot_rewards(cumulative=True,per_episode=True,group_name='cartpole_experience_example')group_interp.add_interpretation(interp)group_interp.to_pickle(f'{learn.model.name.lower()}/',f'{learn.model.name.lower()}')forgininterp.generate_gif():g.write(f'{learn.model.name.lower()}')group_interp.plot_reward_bounds(per_episode=True,smooth_groups=10)More examples can be found indocs_srcand the actual code being run for generating gifs can be found intestsin
eithertest_dqn.pyortest_ddpg.py.As a note, here is a run down of existing RL frameworks:Intel CoachTensor ForceOpenAI BaselinesTensorflow AgentsKerasRLHowever there are also frameworks in PyTorch:HorizonDeepRLSpinning UpInstallationfastai (semi-optional)Install Fastaior if you are using Anaconda (which is a good idea to use Anaconda) you can do:conda install -c pytorch -c fastai fastaifast_rlFastai will be installed if it does not exist. If it does exist, the versioning should be repaired by the the setup.py.pip install fastaiInstallation (Optional)OpenAI all gyms:pip install gym[all]Mazes:git clone https://github.com/MattChanTK/gym-maze.gitcd gym-mazepython setup.py installInstallation Dev (Optional)git clone https://github.com/josiahls/fast-reinforcement-learning.gitcd fast-reinforcement-learningpython setup.py installInstallation IssuesMany issues will likely fall underfastai installation issues.Any other issues are likely environment related. It is important to note that Python 3.7 is not being tested due to
an issue with Pyglet and gym do not working. This issue will not stop you from training models, however this might impact using
OpenAI environments.RoadMapWorking on1.0.0Base version is completed with working model visualizations proving performance / expected failure. At
this point, all models should have guaranteed environments they should succeed in.1.1.0Working onMore Traditional RL modelsWorking onAdd PPOWorking onAdd TRPOAdd D4PGAdd A2CAdd A3C1.2.0 HRL modelsPossibly might change version to 2.0 depending on SMDP issuesAdd SMDPAdd Goal oriented MDPs. Will Require a new "Step"Add FeUdal NetworkAdd storage based DataBunch memory management. This can prevent RAM from being used up by episode image frames
that may or may not serve any use to the agent, but only for logging.1.3.0Add HACAdd MAXQAdd HIRO1.4.0Add h-DQNAdd Modulated Policy HierarchiesAdd Meta Learning Shared Hierarchies1.5.0Add STRategic Attentive Writer (STRAW)Add H-DRLNAdd Abstract Markov Decision Process (AMDP)Add conda integration so that installation can be truly one step.1.6.0 HRL Options modelsPossibly will already be implemented in a previous modelOptions augmentation to DQN based modelsOptions augmentation to actor critic modelsOptions augmentation to async actor critic models1.8.0 HRL SkillsSkills augmentation to DQN based modelsSkills augmentation to actor critic modelsSkills augmentation to async actor critic models1.9.02.0.0 Add PyBullet Fetch Environments2.0.0 Not part of this repo, however the envs need to subclass the OpenAIgym.GoalEnv2.0.0 Add HERContributionFollowing fastai's guidelines would be desirable:GuidelinesWhile we hope that model additions will be added smoothly. All models will only be dependent oncore.layers.py.
As time goes on, the model architecture will overall improve (we are and while continue to be still figuring things out).StyleSince fastai uses a different style from traditional PEP-8, we will be followingStyleandAbbreviations. Also we will use RL specific abbr.ConceptAbbr.Combination ExamplesRLStatestActionacnBoundsbbSame as Bounding BoxExamplesReward GraphsModelDQNDueling DQNDouble DQNDDDQNFixed Target DQNDQNDueling DQNDouble DQNDDDQNFixed Target DQNDDPGDDPGDDPGAgent StagesModelGif(Early)Gif(Mid)Gif(Late)DDPG+PERDoubleDueling+ERDoubleDQN+ERDuelingDQN+ERDoubleDueling+PERDQN+ERDuelingDQN+PERDQN+PERDoubleDQN+PERDDPG+PERDDPG+ERDQN+PERFixedTargetDQN+ERDQN+ERFixedTargetDQN+PERDoubleDQN+ERDoubleDQN+PERDuelingDQN+ERDoubleDueling+PERDuelingDQN+PERDoubleDueling+ERDDPG+ERDDPG+PER