Package: marllib
MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning LibraryNews:
We are excited to announce that a major update has just been released. For detailed version information, please refer to theversion info.Multi-agent Reinforcement Learning Library (MARLlib)isa MARL librarythat utilizesRayand one of its toolkitsRLlib. It offers a comprehensive platform for developing, training, and testing MARL algorithms across various tasks and environments.Here's an example of how MARLlib can be used:frommarllibimportmarl# prepare envenv=marl.make_env(environment_name="mpe",map_name="simple_spread")# initialize algorithm with appointed hyper-parametersmappo=marl.algos.mappo(hyperparam_source='mpe')# build agent model based on env + algorithms + user preferencemodel=marl.build_model(env,mappo,{"core_arch":"gru","encode_layer":"128-256"})# start trainingmappo.fit(env,model,stop={'timesteps_total':1000000},share_policy='group')# ready to controlmappo.render(env,model,share_policy='group',restore_path='path_to_checkpoint')Why MARLlib?Here we provide a table for the comparison of MARLlib and existing work.LibrarySupported EnvAlgorithmParameter SharingModelPyMARL1 cooperative5shareGRUPyMARL22 cooperative11shareMLP + GRUMAPPO Benchmark4 cooperative1share + separateMLP + GRUMAlib4 self-play10share + group + separateMLP + LSTMEPyMARL4 cooperative9share + separateGRUMARLlib12no task mode restriction18share + group + separate +customizableMLP + CNN + GRU + LSTMLibraryGithub StarsDocumentationIssues OpenActivityLast UpdatePyMARL:x:PyMARL2:x:MAPPO Benchmark:x:MAlibEPyMARL:x:MARLlibkey features:beginner: MARLlib offers several key features that make it stand out:MARLlib unifies diverse algorithm pipelines with agent-level distributed dataflow, allowing researchers to develop, test, and evaluate MARL algorithms across different tasks and environments.MARLlib supports all task modes, including cooperative, collaborative, competitive, and mixed. This makes it easier for researchers to train and evaluate MARL algorithms across a wide range of tasks.MARLlib provides a new interface that follows the structure of Gym, making it easier for researchers to work with multi-agent environments.MARLlib provides flexible and customizable parameter-sharing strategies, allowing researchers to optimize their algorithms for different tasks and environments.:rocket: Using MARLlib, you can take advantage of various benefits, such as:Zero knowledge of MARL: MARLlib provides 18 pre-built algorithms with an intuitive API, allowing researchers to start experimenting with MARL without prior knowledge of the field.Support for all task modes: MARLlib supports almost all multi-agent environments, making it easier for researchers to experiment with different task modes.Customizable model architecture: Researchers can choose their preferred model architecture from the model zoo, or build their own.Customizable policy sharing: MARLlib provides grouping options for policy sharing, or researchers can create their own.Access to over a thousand released experiments: Researchers can access over a thousand released experiments to see how other researchers have used MARLlib.InstallationNote:
Currently MARLlib supports Linux only.Step-by-step  (recommended)install dependenciesinstall environmentsinstall patches1. install dependencies (basic)First, install MARLlib dependencies to guarantee basic usage.
followingthis guide, finally install patches for RLlib.$condacreate-nmarllibpython=3.8# or 3.9$condaactivatemarllib
$gitclonehttps://github.com/Replicable-MARL/MARLlib.git&&cdMARLlib
$pipinstall-rrequirements.txt2. install environments (optional)Please followthis guide.3. install patches (basic)Fix bugs of RLlib using patches by running the following command:$cd/Path/To/MARLlib/marl/patch
$pythonadd_patch.py-yPyPI$pipinstall--upgradepip
$pipinstallmarllibGetting startedPrepare the configurationThere are four parts of configurations that take charge of the whole training process.scenario: specify the environment/task settingsalgorithm: choose the hyperparameters of the algorithmmodel: customize the model architectureray/rllib: change the basic training settingsBefore training, ensure all the parameters are set correctly, especially those you don't want to change.Note:
You can also modify all the pre-set parameters via MARLLib API.*Register the environmentEnsure all the dependencies are installed for the environment you are running with. Otherwise, please refer toMARLlib documentation.task modeapi examplecooperativemarl.make_env(environment_name="mpe", map_name="simple_spread", force_coop=True)collaborativemarl.make_env(environment_name="mpe", map_name="simple_spread")competitivemarl.make_env(environment_name="mpe", map_name="simple_adversary")mixedmarl.make_env(environment_name="mpe", map_name="simple_crypto")Most of the popular environments in MARL research are supported by MARLlib:Env NameLearning ModeObservabilityAction SpaceObservationsLBFcooperative + collaborativeBothDiscrete1DRWAREcooperativePartialDiscrete1DMPEcooperative + collaborative + mixedBothBoth1DSMACcooperativePartialDiscrete1DMetaDrivecollaborativePartialContinuous1DMAgentcollaborative + mixedPartialDiscrete2DPommermancollaborative + competitive + mixedBothDiscrete2DMAMuJoCocooperativeFullContinuous1DGRFcollaborative + mixedFullDiscrete2DHanabicooperativePartialDiscrete1DMATEcooperative + mixedPartialBoth1DGoBiggercooperative + mixedBothContinuous1DEach environment has a readme file, standing as the instruction for this task, including env settings, installation, and
important notes.Initialize the algorithmrunning targetapi exampletrain & finetunemarl.algos.mappo(hyperparam_source=$ENV)develop & debugmarl.algos.mappo(hyperparam_source="test")3rd party envmarl.algos.mappo(hyperparam_source="common")Here is a chart describing the characteristics of each algorithm:algorithmsupport task modediscrete actioncontinuous actionpolicy typeIQL*all four:heavy_check_mark:off-policyPGall four:heavy_check_mark::heavy_check_mark:on-policyA2Call four:heavy_check_mark::heavy_check_mark:on-policyDDPGall four:heavy_check_mark:off-policyTRPOall four:heavy_check_mark::heavy_check_mark:on-policyPPOall four:heavy_check_mark::heavy_check_mark:on-policyCOMAall four:heavy_check_mark:on-policyMADDPGall four:heavy_check_mark:off-policyMAA2C*all four:heavy_check_mark::heavy_check_mark:on-policyMATRPO*all four:heavy_check_mark::heavy_check_mark:on-policyMAPPOall four:heavy_check_mark::heavy_check_mark:on-policyHATRPOcooperative:heavy_check_mark::heavy_check_mark:on-policyHAPPOcooperative:heavy_check_mark::heavy_check_mark:on-policyVDNcooperative:heavy_check_mark:off-policyQMIXcooperative:heavy_check_mark:off-policyFACMACcooperative:heavy_check_mark:off-policyVDACcooperative:heavy_check_mark::heavy_check_mark:on-policyVDPPO*cooperative:heavy_check_mark::heavy_check_mark:on-policy*all four: cooperative collaborative competitive mixedIQLis the multi-agent version of Q learning.MAA2CandMATRPOare the centralized version of A2C and TRPO.VDPPOis the value decomposition version of PPO.Build the agent modelAn agent model consists of two parts,encoderandcore arch.encoderwill be constructed by MARLlib according to the observation space.
Choosemlp,gru, orlstmas you like to build the complete model.model archapi exampleMLPmarl.build_model(env, algo, {"core_arch": "mlp")GRUmarl.build_model(env, algo, {"core_arch": "gru"})LSTMmarl.build_model(env, algo, {"core_arch": "lstm"})Encoder Archmarl.build_model(env, algo, {"core_arch": "gru", "encode_layer": "128-256"})Kick off the trainingsettingapi exampletrainalgo.fit(env, model)debugalgo.fit(env, model, local_mode=True)stop conditionalgo.fit(env, model, stop={'episode_reward_mean': 2000, 'timesteps_total': 10000000})policy sharingalgo.fit(env, model, share_policy='all') # or 'group' / 'individual'save modelalgo.fit(env, model, checkpoint_freq=100, checkpoint_end=True)GPU acceleratealgo.fit(env, model, local_mode=False, num_gpus=1)CPU acceleratealgo.fit(env, model, local_mode=False, num_workers=5)Training & rendering APIfrommarllibimportmarl# prepare envenv=marl.make_env(environment_name="mpe",map_name="simple_spread")# initialize algorithm with appointed hyper-parametersmappo=marl.algos.mappo(hyperparam_source="mpe")# build agent model based on env + algorithms + user preferencemodel=marl.build_model(env,mappo,{"core_arch":"mlp","encode_layer":"128-256"})# start trainingmappo.fit(env,model,stop={"timesteps_total":1000000},checkpoint_freq=100,share_policy="group")# renderingmappo.render(env,model,local_mode=True,restore_path={'params_path':"checkpoint_000010/params.json",'model_path':"checkpoint_000010/checkpoint-10"})Benchmark resultsAll results are listedhere.Quick examplesMARLlib provides some practical examples for you to refer to.Detailed API usage: show how to use MARLlib api in
detail, e.g. cmd + api combined running.Policy sharing cutomization:
define your group policy-sharing strategy as you like based on current tasks.Loading model and rendering:
render the environment based on the pre-trained model.Incorporating new environment:
add your new environment following MARLlib's env-agent interaction interface.Incorporating new algorithm:
add your new algorithm following MARLlib learning pipeline.TutorialsTry MPE + MAPPO examples on Google Colaboratory!More tutorial documentations are availablehere.Awesome ListA collection of research and review papers of multi-agent reinforcement learning (MARL) is available. The papers have been organized based on their publication date and their evaluation of the corresponding environments.Algorithms:Environments:CommunityChannelLinkIssuesGitHub IssuesRoadmapThe roadmap to the future release is available inROADMAP.md.ContributingWe are a small team on multi-agent reinforcement learning, and we will take all the help we can get!
If you would like to get involved, here is information oncontribution guidelines and how to test the code locally.You can contribute in multiple ways, e.g., reporting bugs, writing or translating documentation, reviewing or refactoring code, requesting or implementing new features, etc.PaperIf you use MARLlib in your research, please cite theMARLlib paper.@article{hu2022marllib,
  title={MARLlib: Extending RLlib for Multi-agent Reinforcement Learning},
  author={Hu, Siyi and Zhong, Yifan and Gao, Minquan and Wang, Weixun and Dong, Hao and Li, Zhihui and Liang, Xiaodan and Chang, Xiaojun and Yang, Yaodong},
  journal={arXiv preprint arXiv:2210.13708},
  year={2022}}