SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\lambda_{e}$ ,  $\lambda_{D}$  and  $\lambda_{d}$  are three positive coefficients. The team reward ensures the learned skills are useful for the team performance, while the intrinsic rewards are used to optimize the diversity term in Eq. 3. We adopt the popular PPO [33] objective to optimize both the high-level skill coordinator and the low-level skill discoverer. Given a policy  $\pi(a|x)$  with parameters  $\bar{\theta}$ , a value function  $V(y)$  with parameters  $\phi$  and a reward function  $r$ , we write a generic template for PPO objective as:
$$
\mathcal {L} _ {\mathrm {P P O}} \left\{\pi (a | x), V (y), r \right\} = \mathcal {L} (\theta) + \lambda_ {c} \mathcal {L} (\phi),
$$
$$
\mathcal {L} (\theta) = - \mathbb {E} _ {t} \left[ \min  \left(\frac {\pi \left(a _ {t} \mid x _ {t}\right)}{\pi_ {o l d} \left(a _ {t} \mid x _ {t}\right)} \hat {A} _ {t}, \operatorname {c l i p} \left(\frac {\pi \left(a _ {t} \mid x _ {t}\right)}{\pi_ {o l d} \left(a _ {t} \mid x _ {t}\right)}, 1 - \epsilon , 1 + \epsilon\right) \hat {A} _ {t}\right) \right], \tag {5}
$$
$$
\mathcal {L} (\phi) = \mathbb {E} _ {t} \left[ \max \left\{\left(V (y _ {t}) - \hat {R} _ {t}\right) ^ {2}, \left(V _ {o l d} (y _ {t}) + \operatorname {c l i p} \left(V (y _ {t}) - V _ {o l d} (y _ {t}), - \epsilon , \epsilon\right) - \hat {R} _ {t}\right) ^ {2} \right\} \right],
$$
where  $\hat{A}_t$  is the advantage computed using GAE [34],  $\hat{R}_t = \hat{A}_t + V_{old}(y_t)$  and  $\lambda_c$  is the coefficient of value loss. Then, we write the overall objective for the high-level skill coordinator as:
$$
\begin{array}{l} \mathcal {L} _ {h} \left(\theta_ {h}, \phi_ {h}\right) = \mathcal {L} _ {\mathrm {P P O}} \left\{\pi_ {h} \left(Z | \hat {s}, \hat {o} ^ {1: n}, Z ^ {0}\right), V _ {h} (\hat {s}), r _ {t} ^ {h} \right\} + \sum_ {i = 1} ^ {n} \mathcal {L} _ {\mathrm {P P O}} \left\{\pi_ {h} \left(z ^ {i} | \hat {s}, \hat {o} ^ {1: n}, Z ^ {0}, Z, z ^ {1: i - 1}\right), V _ {h} (\hat {o} ^ {i}), r _ {t} ^ {h} \right\} \\ \left. \right. - \lambda_ {h} \left(\mathbb {E} \left[ \mathcal {H} \left(\pi_ {h} \left(Z | \hat {s}, \hat {o} ^ {1: n}, Z ^ {0}\right)\right)\right] + \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \mathcal {H} \left(\pi_ {h} \left(z ^ {i} | \hat {s}, \hat {o} ^ {1: n}, Z ^ {0}, Z, z ^ {1: i - 1}\right)\right)\right]\right), \tag {6} \\ \end{array}
$$
where  $\mathcal{H}(\pi_h(\cdot))$  is the entropy of the high-level policy that aims to optimize the skill entropy term in Eq. 3, and  $\lambda_{h}$  is the high-level entropy coefficient.  $\theta_{h}$  and  $\phi_h$  denote the parameters of the policy and
(a) Alice and Bob
(b)  $Z = 0,z^{1} = 3,z^{2} = 2$
Figure 4: (a) The Alice_and_Bob game. The objective is to collect both diamonds, where each diamond is allowed to be collected only when an agent is standing on the button with the same color. (b) At timestep  $t = 0$ , HMASD selects the team skill of collecting the blue diamond ( $Z = 0$ ) and two individual skills for reaching the blue button ( $z^1 = 3$ ) and blue diamond ( $z^2 = 2$ ). (c) After collecting the blue diamond, the team skill transitions to collecting the red diamond ( $Z = 1$ ) with two individual skills of reaching the red button ( $z^1 = 1$ ) and red diamond ( $z^2 = 0$ ) at timestep  $t = k$ , where  $k$  is the skill interval.
(c)  $Z = 1,z^{1} = 1,z^{2} = 0$
value function in the skill coordinator, respectively. Similarly, the overall objective for the low-level skill discoverer is:
$$
\mathcal {L} _ {l} \left(\theta_ {l}, \phi_ {l}\right) = \sum_ {i = 1} ^ {n} \mathcal {L} _ {\mathrm {P P O}} \left\{\pi_ {l} \left(a ^ {i} \mid o ^ {i}, z ^ {i}\right), V _ {l} (s, Z), r _ {t} ^ {i} \right\} - \lambda_ {l} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \mathcal {H} \left(\pi_ {l} \left(a ^ {i} \mid o ^ {i}, z ^ {i}\right)\right) \right], \tag {7}
$$
where  $\mathcal{H}(\pi_l(\cdot))$  denotes low-level policy entropy and optimizes the action entropy term in Eq. 3, and  $\lambda_{l}$  is the low-level entropy coefficient.  $\theta_{l}$  and  $\phi_l$  represent the parameters of the low-level policy and value function, respectively. The skill discriminator is trained in a supervised manner with the categorical cross-entropy loss:
$$
\mathcal {L} _ {d} \left(\phi_ {D}, \phi_ {d}\right) = - \mathbb {E} _ {\left(s, Z\right) \sim \mathcal {D}} \left[ \log q _ {D} \left(Z \mid s\right) \right] - \sum_ {i = 1} ^ {n} \mathbb {E} _ {\left(o ^ {i}, Z, z ^ {i}\right) \sim \mathcal {D}} \left[ \log q _ {d} \left(z ^ {i} \mid o ^ {i}, Z\right) \right], \tag {8}
$$
where  $\mathcal{D} = \{(s,Z,o,z^{1:n})\}$  is a dataset storing the state-skill pairs during training.  $\phi_D$  and  $\phi_d$  are the parameters of team discriminator and individual discriminator, respectively. The pseudo code of our method is shown in Appendix A.
During the execution phase, we only use the centralized high-level policy  $\pi_h(Z,z^{1:n}|s,o)$  and the decentralized low-level policy  $\pi_l(a^i |o^i,z^i)$ . For every  $k$  timesteps, the high-level policy  $\pi_h(Z,z^{1:n}|s,o)$  first chooses a team skill  $Z$  (i.e., team strategy) from a global perspective and then sequentially assigns complementary individual skills  $z^{1:n}$  to agents based on the team strategy  $Z$ . With the assigned individual skill  $z^i$ , each agent  $g^i$  selects an action  $a^i$  according to the low-level policy  $\pi_l(a^i |o^i,z^i)$  at every timestep for execution. Therefore, our method performs one timestep of centralized execution and  $k - 1$  timesteps of decentralized execution in every  $k$  timesteps. Such periodic and spaced centralized execution can coordinate agents more efficiently from a global view compared to the fully decentralized execution, and a small amount of centralized execution is acceptable in many multi-agent tasks. For example, during a basketball game, the coach can call a timeout and gather all players to adjust the team strategy and each player's individual strategy. In short, our method achieves a balance between fully centralized execution [31, 35] and fully decentralized execution [4, 36-39].
In this section, we evaluate the effectiveness of our method. We first conduct a case study to show how HMASD effectively learns diverse useful skills and combines them to complete the task. Then, we compare HMASD with strong MARL baselines on two challenging sparse reward multi-agent benchmarks, i.e., SMAC [40] with 0-1 reward and Overcooked [41]. We further perform ablation studies for HMASD to confirm the benefits of components in our method. We select MAPPO [5], MAT [31] and MASER [42] as our baselines. MAPPO and MAT are two strong policy-based MARL algorithms that achieve state-of-art performance on various multi-agent tasks [40, 43-45]. MASER
Figure 6: Performance comparison between HMASD and baselines on SMAC with 0-1 reward.
Figure 7: Performance comparison between HMASD and baselines on Overcooked.
presents a goal-conditioned method for solving sparse reward MARL. For all methods, we show the mean and variance of the performance across five different random seeds. The hyperparameter setting can be found in Appendix E.
4.1 Case Study
In this subsection, we design a toy game Alice_and_Bob to demonstrate how our method works. As shown in Fig. 4(a), the Alice_and_Bob game is an  $8 \times 8$  grid world environment surrounded by walls. There are two agents Alice and Bob with random initial positions, two buttons at the top and two diamonds at the bottom. The goal of the game is to collect both diamonds, where each diamond is allowed to be collected only when an agent is standing on the button with the same color.
Alice and Bob can receive a non-zero team reward only after they cooperatively collect both diamonds. We set the number of team skills  $n_Z = 2$  and the number of individual skills  $n_z = 4$  for HMASD. Fig. 4(b) and (c) show the learned skills of our method. We can observe that the two team skills  $Z = 0, 1$  correspond to collecting the blue diamond and red diamond for the whole team, respectively, and the four individual skills  $z^i = 0, 1, 2, 3$  guide the individual agent to reach the red diamond, red button, blue diamond and blue button, respectively. We observe similar skills for agents with different initial positions in Appendix C. By learning these diverse and useful skills, our method achieves a higher success rate of task completion than baselines as shown in Fig. 5. These results demonstrate that HMASD could discover significant
Figure 5: Performance comparison between HMASD and baselines on Alice_and_Bob.
team and individual skills, and effectively combine them to accomplish the task.
4.2 Performance on Sparse Reward Multi-Agent Benchmarks
In this subsection, we first test our method on a widely-used MARL benchmark, SMAC [40], where we learn to control a team of ally units against a team of enemy units controlled by a built-in strategy. The ally team wins the game only if all enemy units are killed within the time limit of an episode. Our objective is to maximize the win rate for the ally team. The default reward setting of SMAC contains many dense rewards, such as the unit's health and damage. These dense rewards enable many MARL algorithms like MAT [31] and MAPPO [5] to attain almost  $100\%$  win rate on all scenarios. However, designing useful dense rewards is usually expensive and adjusting the weights between rewards is
Figure 8: Ablation studies regarding components of HMASD.
time-consuming. In this work, we consider a more general reward setting, i.e., SMAC with 0-1 reward, which returns a non-zero team reward of 1 only at the last timestep if the ally team wins. As shown in Fig. 6, HMASD significantly outperforms baselines on SMAC with 0-1 reward, where MAT and MAPPO don't work at all. MASER only works on the scenario  $3\mathrm{m}$  and fails on the other two scenarios  $2\mathrm{s\_vs\_1sc}$  and  $2\mathrm{m\_vs\_1z}$ . This demonstrates the effectiveness of HMASD and the high efficiency of skill discovery for solving sparse reward multi-agent tasks.
Next, we evaluate HMASD on another sparse reward multi-agent task called Overcooked, a popular cooperative cooking simulation game. We follow a simplified version of the game proposed in [41], the objective of which is to deliver the soup as fast as possible. Each soup requires agents to place 3 onions in a pot, cook them for 20 timesteps, put the cooked soup in a dish, and deliver it to the service desk. All agents will receive a team reward of 20 after delivering a soup. In Fig. 7, we compare HMASD with baselines on three Overcooked scenarios. It can be observed that HMASD could also achieve a superior performance over baselines on Overcooked, which further confirms that HMASD can discover useful skills to effectively complete the task.
Moreover, we visualize the learned skills and find that HMASD uses only a few skills to complete the complex task after training. This is because HMASD will encourage different skills to explore different state-observation spaces, but only a small part of the state-observation space can result in a non-zero team reward in the complex sparse reward multi-agent task. For example, on the SMAC scenario, there is usually a big map, we observe that most of the learned skills explore areas of the map without enemies and thus don't contribute to the team reward. These results indicate that when the state-observation space is large, HMASD can discover diverse skills but maybe only some of them are useful for the team reward. More fine-grained results can be found in Appendix F.
Additionally, we compare HMASD with an exploration bonus MARL baseline [46] in Appendix H. The results show that the performance improvements of HMASD mainly come from diverse skill discovery and effective skill combination, rather than implicit exploration.
In this subsection, we conduct ablation studies to investigate the impact of three main components in HMASD: (1) discovering both team and individual skills, (2) using a combination of extrinsic team reward and intrinsic rewards for the low-level reward and (3) employing the high-level skill coordinator to assign skills to agents. To test component (1), we introduce two variants of HMASD, denoted as HMASD_NoTeam and HMASD_NoIndi, respectively. HMASD_NoTeam only learns individual skills for agents, while HMASD_NoIndi only lets all agents learn a shared team skill. To evaluate component (2), we consider HMASD_NoExRew and HMASD_NoInRew, which set  $\lambda_{e} = 0$  and  $\lambda_{D} = \lambda_{d} = 0$  in Eq. 4, respectively. As for component (3), we adopt HMASD_NoHigh, which removes the high-level policy and randomly assigns skills to agents at the start of each episode. As shown in Fig. 8, HMASD performs better than both HMASD_NoTeam and HMASD_NoIndi, which emphasizes the importance of discovering both team and individual skills in multi-agent tasks. After removing the extrinsic team reward or intrinsic rewards in the low-level reward, the performance of HMASD has a large drop. Especially without intrinsic rewards, HMASD can't work on most scenarios. This highlights the considerable contributions of optimizing the diversity term in Eq. 3. Besides, the performance comparison between HMASD and HMASD_NoHigh reveals the necessity of the high-level policy. More ablations for HMASD are shown in Appendix D.
Skill Discovery in MARL Skill discovery is a promising approach to solving complex tasks for its ability to discover diverse skills even without any extrinsic reward. Recently, this approach has been extended to MARL. MASM [9] learns coordinating skills adversarially by setting an information bottleneck. VMAPD [10] constructs a diverse Dec-POMDP to learn diverse skills as different solutions for multi-agent tasks. However, these two methods focus on learning a shared team skill for all agents, and use only one skill throughout an episode. To combine different skills to tackle multi-agent tasks, HSD [11] employs a hierarchical structure, where the high-level policy selects low-level skills for agents based on local observations. HSL [12] follows a similar hierarchical structure and introduces a skill representation mechanism to enhance skill learning. Nevertheless, these two methods only learn individual skills from each agent's individual perspective, without considering team skill learning. Besides, in these two methods, each agent selects skills only based on its local observation, leading to potential skill duplication between agents with similar observations. ODIS [47] applies skill discovery to offline MARL, which discovers generalizable individual skills across different tasks from offline multi-task data, and trains a coordination policy to assign skills to agents with the centralized training and decentralized execution paradigm [4]. However, this decentralized skill assignment may also lead to skill duplication. Chen et al. [48, 49] learn joint options (i.e., skills) by approximating the joint state space as the Kronecker product of the state spaces of individual agents. In this work, we propose to discover both team skills and individual skills. Moreover, we assign skills to agents from a global perspective to enable complementary skill selection and prevent skill duplication effectively.
Sparse Reward MARL The issue of sparse reward poses a significant obstacle when applying RL to solve real-world problems, which will be exacerbated in multi-agent tasks. Several approaches have been proposed to address the sparse reward problem in MARL. SEAC [50] leverages experience sharing among agents to enable efficient exploration. CMAE [51] promotes cooperative exploration by selecting a shared goal for agents from state spaces. VACL [52] tackles the sparse reward problem in MARL by utilizing an automatic curriculum learning algorithm that incrementally expands the training tasks from easy to hard. More recently, MASER [42] introduces a goal-conditioned method that generates subgoals for agents from the experience replay buffer. In this work, we present a hierarchical MARL algorithm that discovers underlying skills and effectively combines these skills to solve sparse reward multi-agent tasks.
It is an efficient way to learn a set of skills and combine them properly to tackle complex tasks. In this study, we take advantage of skill discovery to address MARL problems. We propose to discover latent team and individual skills by embedding them into a probabilistic graphical model. In this way, we formulate multi-agent skill discovery as an inference problem, and derive a variational lower bound as the optimization objective. We then design a practical MARL method called HMASD to optimize the lower bound, where different team skills explore different global state spaces and different individual skills explore different local observation spaces. The empirical results show that HMASD significantly improves performance on challenging sparse reward multi-agent tasks by learning diverse team and individual skills with efficient skill combination.
This work is supported by National Key R&D Program of China under Contract 2022ZD0119802, and National Natural Science Foundation of China under Contract 61836011. It was also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC. What's more, it is funded by Collective Intelligence & Collaboration Laboratory (Open Fund Project No. QXZ23014101) and by Young Elite Scientists Sponsorship Program by CAST 2022QNRC001.
[1] D. Rizvi and D. Boyle, "Multi-agent reinforcement learning with action masking for uav-enabled mobile communications," arXiv preprint arXiv:2303.16737, 2023.
[2] B. Xiao, R. Li, F. Wang, C. Peng, J. Wu, Z. Zhao, and H. Zhang, "Stochastic graph neural network-based value decomposition for marl in internet of vehicles," arXiv preprint arXiv:2303.13213, 2023.
[3] T. Wu, P. Zhou, K. Liu, Y. Yuan, X. Wang, H. Huang, and D. O. Wu, "Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks," IEEE Transactions on Vehicular Technology, vol. 69, no. 8, pp. 8243-8256, 2020.
[4] T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson, “Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,” in Proceedings of the International Conference on Machine Learning. PMLR, 2018, pp. 4295–4304.
[5] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, "The surprising effectiveness of ppo in cooperative multi-agent games," Advances in Neural Information Processing Systems, vol. 35, pp. 24611-24624, 2022.
[6] J. She, J. K. Gupta, and M. J. Kochenderfer, "Agent-time attention for sparse rewards multi-agent reinforcement learning," arXiv preprint arXiv:2210.17540, 2022.
[7] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine, “Diversity is all you need: Learning skills without a reward function,” arXiv preprint arXiv:1802.06070, 2018.
[8] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman, "Dynamics-aware unsupervised discovery of skills," arXiv preprint arXiv:1907.01657, 2019.
[9] S. He, J. Shao, and X. Ji, "Skill discovery of coordination in multi-agent reinforcement learning," arXiv preprint arXiv:2006.04021, 2020.
[10] S. Huang, C. Yu, B. Wang, D. Li, Y. Wang, T. Chen, and J. Zhu, "Vmapd: Generate diverse solutions for multi-agent games with recurrent trajectory discriminators," in IEEE Conference on Games. IEEE, 2022, pp. 9-16.
[11] J. Yang, I. Borovikov, and H. Zha, "Hierarchical cooperative multi-agent reinforcement learning with skill discovery," arXiv preprint arXiv:1912.03558, 2019.
[12] Y. Liu, Y. Li, X. Xu, Y. Dou, and D. Liu, “Heterogeneous skill learning for multi-agent tasks,” Advances in Neural Information Processing Systems, vol. 35, pp. 37 011–37 023, 2022.
[13] L. Gyarmati, H. Kwak, and P. Rodriguez, “Searching for a unique style in soccer,” arXiv preprint arXiv:1409.0308, 2014.
[14] S. Levine, “Reinforcement learning and control as probabilistic inference: Tutorial and review,” arXiv preprint arXiv:1805.00909, 2018.
[15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in Neural Information Processing Systems, vol. 30, 2017.
[16] F. A. Oliehoek and C. Amato, A concise introduction to decentralized POMDPs. Springer, 2016.
[17] K. Gregor, D. J. Rezende, and D. Wierstra, “Variational intrinsic control,” arXiv preprint arXiv:1611.07507, 2016.
[18] J. Achiam, H. Edwards, D. Amodei, and P. Abbeel, “Variational option discovery algorithms,” arXiv preprint arXiv:1807.10299, 2018.
[40] M. Samvelyan, T. Rashid, C. S. De Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr, J. Foerster, and S. Whiteson, “The starcraft multi-agent challenge,” arXiv preprint arXiv:1902.04043, 2019.
[41] M. Carroll, R. Shah, M. K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan, “On the utility of learning about humans for human-ai coordination,” Advances in Neural Information Processing Systems, vol. 32, 2019.
[42] J. Jeon, W. Kim, W. Jung, and Y. Sung, "Maser: Multi-agent reinforcement learning with subgoals generated from experience replay buffer," in Proceedings of the International Conference on Machine Learning. PMLR, 2022, pp. 10041-10052.
[43] C. S. de Witt, B. Peng, P.-A. Kamienny, P. Torr, W. Böhmer, and S. Whiteson, “Deep multi-agent reinforcement learning for decentralized continuous cooperative control,” arXiv preprint arXiv:2003.06709, vol. 19, 2020.
[44] Y. Chen, T. Wu, S. Wang, X. Feng, J. Jiang, Z. Lu, S. McAleer, H. Dong, S.-C. Zhu, and Y. Yang, "Towards human-level bimanual dexterous manipulation with reinforcement learning," Advances in Neural Information Processing Systems, vol. 35, pp. 5150-5163, 2022.
[45] K. Kurach, A. Raichuk, P. Stańczyk, M. Zajac, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet et al., "Google research football: A novel reinforcement learning environment," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, 2020, pp. 4501-4510.
[46] T. Wang, J. Wang, Y. Wu, and C. Zhang, "Influence-based multi-agent exploration," arXiv preprint arXiv:1910.05512, 2019.
[47] F. Zhang, C. Jia, Y.-C. Li, L. Yuan, Y. Yu, and Z. Zhang, "Discovering generalizable multi-agent coordination skills from multi-task offline data," in Proceedings of the International Conference on Learning Representations, 2023.
[48] J. Chen, J. Chen, T. Lan, and V. Aggarwal, "Learning multi-agent options for tabular reinforcement learning using factor graphs," arXiv preprint arXiv:2201.08227, 2022.
[49] J. Chen, J. Chen, T. Lan, and V. Aggarwal, "Scalable multi-agent covering option discovery based on kronecker graphs," Advances in Neural Information Processing Systems, vol. 35, pp. 30406-30418, 2022.
[50] F. Christianos, L. Schäfer, and S. Albrecht, "Shared experience actor-critic for multi-agent reinforcement learning," Advances in Neural Information Processing Systems, vol. 33, pp. 10707-10717, 2020.
[51] I.-J. Liu, U. Jain, R. A. Yeh, and A. Schwing, "Cooperative exploration for multi-agent deep reinforcement learning," in Proceedings of the International Conference on Machine Learning. PMLR, 2021, pp. 6826-6836.
[52] J. Chen, Y. Zhang, Y. Xu, H. Ma, H. Yang, J. Song, Y. Wang, and Y. Wu, "Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems," Advances in Neural Information Processing Systems, vol. 34, pp. 9681-9693, 2021.
A Pseudo Code of Hierarchical Multi-Agent Skill Discovery