Package: adeptRL
adept is a reinforcement learning framework designed to accelerate research
by providing:a modular interface for using custom networks, agents, and environmentsbaseline reinforcement learning models and algorithms for PyTorchmulti-GPU supportaccess to various environmentsbuilt-in tensorboard logging, model saving, reloading, evaluation, and
renderingproven hyperparameter defaultsThis code is early-access, expect rough edges. Interfaces subject to change.
We're happy to accept feedback and contributions.Read MoreInstallationQuickstartFeaturesPerformanceDocumentationArchitecture OverviewModularNetwork OverviewResume trainingEvaluate a modelRender environmentExamplesCustom Network (stub| example)Custom SubModule (stub|example)Custom Agent (stub|example)Custom Environment (stub|example)InstallationDependencies:gymPyTorch 1.xPython 3.5+We recommend CUDA 10, pytorch 1.0, python 3.6From source:Follow instructions forPyTorch(Optional) Follow instructions forStarCraft 2gitclonehttps://github.com/heronsystems/adeptRLcdadeptRL# Remove mpi, sc2, profiler if you don't plan on using these features:pipinstall.[mpi,sc2,profiler]From docker:docker instructionsQuickstartTrain an AgentLogs go to/tmp/adept_logs/by default. The log directory contains the
tensorboard file, saved models, and other metadata.# Local Mode (A2C)# We recommend 4GB+ GPU memory, 8GB+ RAM, 4+ Corespython-madept.applocal--envBeamRiderNoFrameskip-v4# Distributed Mode (A2C, requires NCCL)# We recommend 2+ GPUs, 8GB+ GPU memory, 32GB+ RAM, 4+ Corespython-madept.appdistrib--envBeamRiderNoFrameskip-v4# IMPALA (requires mpi4py and is resource intensive)# We recommend 2+ GPUs, 8GB+ GPU memory, 32GB+ RAM, 4+ Corespython-madept.appimpala--agentActorCriticVtrace--envBeamRiderNoFrameskip-v4# StarCraft 2 (IMPALA not supported yet)# Warning: much more resource intensive than Ataripython-madept.applocal--envCollectMineralShards# To see a full list of options:python-madept.app-h
python-madept.apphelp<command>Use your own Agent, Environment, Network, or SubModule"""my_script.pyTrain an agent on a single GPU."""fromadept.scripts.localimportparse_args,mainfromadept.networksimportNetworkModule,NetworkRegistry,SubModule1Dfromadept.agentsimportAgentModule,AgentRegistryfromadept.environmentsimportEnvModule,EnvRegistryclassMyAgent(AgentModule):pass# ImplementclassMyEnv(EnvModule):pass# ImplementclassMyNet(NetworkModule):pass# ImplementclassMySubModule1D(SubModule1D):pass# Implementif__name__=='__main__':agent_registry=AgentRegistry()agent_registry.register_agent(MyAgent)env_registry=EnvRegistry()env_registry.register_env(MyEnv,['env-id-1','env-id-2'])network_registry=NetworkRegistry()network_registry.register_custom_net(MyNet)network_registry.register_submodule(MySubModule1D)main(parse_args(),agent_registry=agent_registry,env_registry=env_registry,net_registry=network_registry)Call your script like this:python my_script.py --agent MyAgent --env  env-id-1 --custom-network MyNetYou can see all the argshereor how to implement
the stubs in the examples section above.FeaturesScriptsLocal (Single-node, Single-GPU)Best place tostartif you're trying to understand code.Distributed (Multi-node, Multi-GPU)Uses NCCL backend to all-reduce gradients across GPUs without a parameter
server or host process.Supports NVLINK and InfiniBand to reduce communication overheadInfiniBand untested since we do not have a setup to test on.Importance Weighted Actor Learner Architectures, IMPALA (Single Node, Multi-GPU)Our implementation uses GPU workers rather than CPU workers for forward
passes.On Atari we achieve ~4k SPS = ~16k FPS with two GPUs and an 8-core CPU."Note that the shallow IMPALA experiment completes training over 200
million frames in less than one hour."IMPALA official experiments use 48 cores.Ours: 2000 frame / (second * # CPU core) DeepMind: 1157 frame / (second * # CPU core)Does not yet support multiple nodes or direct GPU memory transfers.AgentsAdvantage Actor Critic, A2C (paper|code)Actor Critic Vtrace, IMPALA (paper|code)NetworksModular Network Interface: supports arbitrary input and output shapes up to
4D via a SubModule API.Stateful networks (ie. LSTMs)Batch normalization (paper)EnvironmentsOpenAI GymStarCraft 2 (unstable)Performance~ 3,000 Steps/second = 12,000 FPS (Atari)Local Mode64 environmentsGeForce 2080 TiRyzen 2700x 8-coreUsed to win aDoom competition(Ben Bell / Marv2in)Trained for 50M Steps / 200M FramesUp to 30 no-ops at start of each episodeEvaluated on different seeds than trained onArchitecture:Four Convs(F=32)
followed by anLSTM(F=512)Reproduce withpython -m adept.app local --logdir ~/local64_benchmark --eval  -y --nb-step 50e6 --env <env-id>AcknowledgementsWe borrow pieces of OpenAI'sgymandbaselinescode. We indicate where this
is done.