Package: yaaf
Yet Another Agents FrameworkAn RL research-oriented framework for agent prototyping and evaluationIntroductionInstallationExamplesDQN for Space InvadersDQN for Cart PoleA3C (on GPU) for Space InvadersDQN from scratch for CartPoleMarkov SubmoduleCitingRoadmapContributingAcknowledgmentsIntroductionYAAF is a reinforcement learning, research-oriented framework designed for quick agent prototyping and evaluation.At its core, YAAF follows the assumptions that:Agents execute actions upon the environment (which yields observations and rewards in return)Environments follow the interface fromOpenAI GymAgents follow a clearagent interfaceAny deep learning framework can be used for deep rl agents (even though it comes packed withPyTorch tools)As a simple example, suppose you'd want to evaluate an agent following a random policy on the Space Invaders environmentimportgymfromyaaf.agentsimportRandomAgentfromyaaf.evaluationimportAverageEpisodeReturnMetricfromyaaf.executionimportEpisodeRunnerfromyaaf.visualizationimportLinePlotenv=gym.make("SpaceInvaders-v0")agent=RandomAgent(num_actions=env.action_space.n)metric=AverageEpisodeReturnMetric()runner=EpisodeRunner(5,agent,env,[metric],render=True).run()plot=LinePlot("Space Invaders Random Policy",x_label="Episode",y_label="Average Episode Return",num_measurements=5)plot.add_run("random policy",metric.result())plot.show()Quick Disclaimer:YAAF is not yet another deep reinforcement learning framework.If you are looking for high-quality implementations of state-of-the-art algorithms, then I suggest the following libraries:OpenAI Spinning Up (2020)Acme (2020)Stable-Baselines (2019)RLLib (2019)InstallationFor the first installation I suggest setting up new Python 3.7 virtual environment$ python -m venv yaaf_test_environment
$ source yaaf_test_environment/bin/activate
$ pip install --upgrade pip setuptools
$ pip install yaaf  
$ pip install gym[atari] # Optional - Atari2600Examples1 - Space Invaders DQNimportgymfromyaaf.environments.wrappersimportDeepMindAtari2600Wrapperfromyaaf.agents.dqnimportDeepMindAtariDQNAgentfromyaaf.executionimportTimestepRunnerfromyaaf.evaluationimportAverageEpisodeReturnMetric,TotalTimestepsMetricenv=DeepMindAtari2600Wrapper(gym.make("SpaceInvaders-v0"))agent=DeepMindAtariDQNAgent(num_actions=env.action_space.n)metrics=[AverageEpisodeReturnMetric(),TotalTimestepsMetric()]runner=TimestepRunner(1e9,agent,env,metrics,render=True).run()2 - CartPole DQNimportgymfromyaaf.agents.dqnimportMLPDQNAgentfromyaaf.executionimportEpisodeRunnerfromyaaf.evaluationimportAverageEpisodeReturnMetric,TotalTimestepsMetricenv=gym.make("CartPole-v0")layers=[(64,"relu"),(64,"relu")]agent=MLPDQNAgent(num_features=env.observation_space.shape[0],num_actions=env.action_space.n,layers=layers)metrics=[AverageEpisodeReturnMetric(),TotalTimestepsMetric()]runner=EpisodeRunner(100,agent,env,metrics,render=True).run()3 - Asynchronous Advantage Actor-Critic on GPU(my multi-task implementation, requires tensorflow-gpu)https://research.nvidia.com/publication/reinforcement-learning-through-asynchronous-advantage-actor-critic-gpufromyaaf.environments.wrappersimportNvidiaAtari2600Wrapperfromyaaf.agents.hga3cimportHybridGA3CAgentfromyaaf.executionimportAsynchronousParallelRunnernum_processes=8envs=[NvidiaAtari2600Wrapper("SpaceInvadersDeterministic-v4")for_inrange(num_processes)]hga3c=HybridGA3CAgent(environment_names=[env.spec.idforenvinenvs],environment_actions=[env.action_space.nforenvinenvs],observation_space=envs[0].observation_space.shape)hga3c.start_threads()trainer=AsynchronousParallelRunner(agents=hga3c.workers,environments=envs,max_episodes=150000,render_ids=[0,1,2])trainer.start()whiletrainer.running:continuehga3c.save(f"hga3c_space_invaders")hga3c.stop_threads()4 - CartPole DQN from scratchimportgymfromyaaf.agents.dqnimportDQNAgentfromyaaf.agents.dqn.networksimportDeepQNetworkfromyaaf.executionimportEpisodeRunnerfromyaaf.evaluationimportAverageEpisodeReturnMetric,TotalTimestepsMetricfromyaaf.models.feature_extractionimportMLPFeatureExtractor# Setup envenv=gym.make("CartPole-v0")num_features=env.observation_space.shape[0]num_actions=env.action_space.n# Setup modelmlp_feature_extractor=MLPFeatureExtractor(num_inputs=num_features,layers=[(64,"relu"),(64,"relu")])network=DeepQNetwork(feature_extractors=[mlp_feature_extractor],num_actions=num_actions,learning_rate=0.001,optimizer="adam",cuda=True)# Setup agentagent=DQNAgent(network,num_actions,discount_factor=0.95,initial_exploration_steps=1000,final_exploration_rate=0.001)# Runmetrics=[AverageEpisodeReturnMetric(),TotalTimestepsMetric()]runner=EpisodeRunner(100,agent,env,metrics,render=True).run()Markov Sub-moduleTODOCiting the ProjectWhen using YAAF in your projects, cite using:@misc{yaaf,
  author = {João Ribeiro},
  title = {YAAF - Yet Another Agents Framework},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/jmribeiro/yaaf}},
}RoadmapDocumentationCode cleanupMore algorithmsContributingIf you want to contribute to this project, feel free to contact me bye-mailor open an issue.AcknowledgmentsYAAF was developed as a side-project to my research work and its creation was motivated by work done in the projectAd Hoc Teams With Humans And Robotsfunded by the Air Force Office of Scientific Research, in collaboration with PUC-Rio.