Package: agilerl
AgileRLReinforcement learning streamlined.Easier and faster reinforcement learning with RLOps. Visit ourwebsite. Viewdocumentation.Join theDiscord Serverto collaborate.NEW: AgileRL now introduces evolvableContextual Multi-armed Bandit Algorithms!This is a Deep Reinforcement Learning library focused on improving development by introducing RLOps - MLOps for reinforcement learning.This library is initially focused on reducing the time taken for training models and hyperparameter optimization (HPO) by pioneering evolutionary HPO techniques for reinforcement learning.Evolutionary HPO has been shown to drastically reduce overall training times by automatically converging on optimal hyperparameters, without requiring numerous training runs.We are constantly adding more algorithms and features. AgileRL already includes state-of-the-art evolvable on-policy, off-policy, offline, multi-agent and contextual multi-armed bandit reinforcement learning algorithms with distributed training.AgileRL offers 10x faster hyperparameter optimization than SOTA.Global steps is the sum of every step taken by any agent in the environment, including across an entire population, during the entire hyperparameter optimization process.Table of ContentsBenchmarksGet StartedTutorialsAlgorithms implementedTrain an agentCiting AgileRLBenchmarksReinforcement learning algorithms and libraries are usually benchmarked once the optimal hyperparameters for training are known, but it often takes hundreds or thousands of experiments to discover these. This is unrealistic and does not reflect the true, total time taken for training. What if we could remove the need to conduct all these prior experiments?In the charts below, a single AgileRL run, which automatically tunes hyperparameters, is benchmarked against Optuna's multiple training runs traditionally required for hyperparameter optimization, demonstrating the real time savings possible. Global steps is the sum of every step taken by any agent in the environment, including across an entire population.AgileRL offers an order of magnitude speed up in hyperparameter optimization vs popular reinforcement learning training frameworks combined with Optuna. Remove the need for multiple training runs and save yourself hours.AgileRL also supports multi-agent reinforcement learning using the Petting Zoo-style (parallel API). The charts below highlight the performance of our MADDPG and MATD3 algorithms with evolutionary hyper-parameter optimisation (HPO), benchmarked against epymarl's MADDPG algorithm with grid-search HPO for the simple speaker listener and simple spread environments.Get StartedInstall as a package with pip:pipinstallagilerlOr install in development mode:gitclonehttps://github.com/AgileRL/AgileRL.git&&cdAgileRL
pipinstall-e.Demo:cddemos
pythondemo_online.pyor to demo distributed training:cddemos
acceleratelaunch--config_fileconfigs/accelerate/accelerate.yamldemos/demo_online_distributed.pyTutorialsWe are in the process of creating tutorials on how to use AgileRL and train agents on a variety of tasks.Currently, we havetutorials for single-agent tasksthat will guide you through the process of training both on and off-policy agents to beat a variety of Gymnasium environments. Additionally, we havemulti-agent tutorialsthat make use of PettingZoo environments such as training DQN to play Connect Four with curriculum learning and self-play, and also for multi-agent tasks in MPE environments. We also have atutorial on using hierarchical curriculum learningto teach agents Skills. We also have files for a tutorial on training a language model with reinforcement learning using ILQL on Wordle intutorials/Language. If using ILQL on Wordle, download and unzip data.ziphere.Our demo files indemosalso provide examples on how to train agents using AgileRL, and more information can be found in ourdocumentation.Evolvable algorithms implemented (more coming soon!)DQNRainbow DQNDDPGTD3PPOCQLILQLMADDPGMATD3NeuralUCBNeuralTSTrain an agent to beat a Gym environmentBefore starting training, there are some meta-hyperparameters and settings that must be set. These are defined inINIT_HP, for general parameters, andMUTATION_PARAMS, which define the evolutionary probabilities, andNET_CONFIG, which defines the network architecture. For example:INIT_HP={'ENV_NAME':'LunarLander-v2',# Gym environment name'ALGO':'DQN',# Algorithm'DOUBLE':True,# Use double Q-learning'CHANNELS_LAST':False,# Swap image channels dimension from last to first [H, W, C] -> [C, H, W]'BATCH_SIZE':256,# Batch size'LR':1e-3,# Learning rate'EPISODES':2000,# Max no. episodes'TARGET_SCORE':200.,# Early training stop at avg score of last 100 episodes'GAMMA':0.99,# Discount factor'MEMORY_SIZE':10000,# Max memory buffer size'LEARN_STEP':1,# Learning frequency'TAU':1e-3,# For soft update of target parameters'TOURN_SIZE':2,# Tournament size'ELITISM':True,# Elitism in tournament selection'POP_SIZE':6,# Population size'EVO_EPOCHS':20,# Evolution frequency'POLICY_FREQ':2,# Policy network update frequency'WANDB':True# Log with Weights and Biases}MUTATION_PARAMS={# Relative probabilities'NO_MUT':0.4,# No mutation'ARCH_MUT':0.2,# Architecture mutation'NEW_LAYER':0.2,# New layer mutation'PARAMS_MUT':0.2,# Network parameters mutation'ACT_MUT':0,# Activation layer mutation'RL_HP_MUT':0.2,# Learning HP mutation'RL_HP_SELECTION':['lr','batch_size'],# Learning HPs to choose from'MUT_SD':0.1,# Mutation strength'RAND_SEED':1,# Random seed}NET_CONFIG={'arch':'mlp',# Network architecture'h_size':[32,32],# Actor hidden size}First, useutils.utils.initialPopulationto create a list of agents - our population that will evolve and mutate to the optimal hyperparameters.fromagilerl.utils.utilsimportmakeVectEnvs,initialPopulationimporttorchdevice=torch.device("cuda"iftorch.cuda.is_available()else"cpu")env=makeVectEnvs(env_name=INIT_HP['ENV_NAME'],num_envs=16)try:state_dim=env.single_observation_space.n# Discrete observation spaceone_hot=True# Requires one-hot encodingexceptException:state_dim=env.single_observation_space.shape# Continuous observation spaceone_hot=False# Does not require one-hot encodingtry:action_dim=env.single_action_space.n# Discrete action spaceexceptException:action_dim=env.single_action_space.shape[0]# Continuous action spaceifINIT_HP['CHANNELS_LAST']:state_dim=(state_dim[2],state_dim[0],state_dim[1])agent_pop=initialPopulation(algo=INIT_HP['ALGO'],# Algorithmstate_dim=state_dim,# State dimensionaction_dim=action_dim,# Action dimensionone_hot=one_hot,# One-hot encodingnet_config=NET_CONFIG,# Network configurationINIT_HP=INIT_HP,# Initial hyperparameterspopulation_size=INIT_HP['POP_SIZE'],# Population sizedevice=device)Next, create the tournament, mutations and experience replay buffer objects that allow agents to share memory and efficiently perform evolutionary HPO.fromagilerl.components.replay_bufferimportReplayBufferfromagilerl.hpo.tournamentimportTournamentSelectionfromagilerl.hpo.mutationimportMutationsfield_names=["state","action","reward","next_state","done"]memory=ReplayBuffer(action_dim=action_dim,# Number of agent actionsmemory_size=INIT_HP['MEMORY_SIZE'],# Max replay buffer sizefield_names=field_names,# Field names to store in memorydevice=device)tournament=TournamentSelection(tournament_size=INIT_HP['TOURN_SIZE'],# Tournament selection sizeelitism=INIT_HP['ELITISM'],# Elitism in tournament selectionpopulation_size=INIT_HP['POP_SIZE'],# Population sizeevo_step=INIT_HP['EVO_EPOCHS'])# Evaluate using last N fitness scoresmutations=Mutations(algo=INIT_HP['ALGO'],# Algorithmno_mutation=MUTATION_PARAMS['NO_MUT'],# No mutationarchitecture=MUTATION_PARAMS['ARCH_MUT'],# Architecture mutationnew_layer_prob=MUTATION_PARAMS['NEW_LAYER'],# New layer mutationparameters=MUTATION_PARAMS['PARAMS_MUT'],# Network parameters mutationactivation=MUTATION_PARAMS['ACT_MUT'],# Activation layer mutationrl_hp=MUTATION_PARAMS['RL_HP_MUT'],# Learning HP mutationrl_hp_selection=MUTATION_PARAMS['RL_HP_SELECTION'],# Learning HPs to choose frommutation_sd=MUTATION_PARAMS['MUT_SD'],# Mutation strengtharch=NET_CONFIG['arch'],# Network architecturerand_seed=MUTATION_PARAMS['RAND_SEED'],# Random seeddevice=device)The easiest training loop implementation is to use ourtrain_off_policy()function. It requires theagenthave functionsgetAction()andlearn().fromagilerl.training.train_off_policyimporttrain_off_policytrained_pop,pop_fitnesses=train_off_policy(env=env,# Gym-style environmentenv_name=INIT_HP['ENV_NAME'],# Environment namealgo=INIT_HP['ALGO'],# Algorithmpop=agent_pop,# Population of agentsmemory=memory,# Replay bufferswap_channels=INIT_HP['CHANNELS_LAST'],# Swap image channel from last to firstn_episodes=INIT_HP['EPISODES'],# Max number of training episodesevo_epochs=INIT_HP['EVO_EPOCHS'],# Evolution frequencyevo_loop=1,# Number of evaluation episodes per agenttarget=INIT_HP['TARGET_SCORE'],# Target score for early stoppingtournament=tournament,# Tournament selection objectmutation=mutations,# Mutations objectwb=INIT_HP['WANDB'])# Weights and Biases trackingCiting AgileRLIf you use AgileRL in your work, please cite the repository:@software{Ustaran-Anderegg_AgileRL,author={Ustaran-Anderegg, Nicholas and Pratt, Michael},license={Apache-2.0},title={{AgileRL}},url={https://github.com/AgileRL/AgileRL}}