SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1: Hierarchical Multi-Agent Skill Discovery
Initialize: Skill coordinator parameters  $\theta_{h}$  and  $\phi_h$  ,Skill discoverer parameters  $\theta_{l}$  and  $\phi_l$  ,Skill discriminator parameters  $\phi_D$  and  $\phi_d$  , High-level replay buffer  $B_{h}$  , Low-level replay buffer  $B_{l}$  State-skill pair dataset  $\mathcal{D}$    
for episode  $= 0,1,\dots ,K$  do # The inference phase for  $t = 0,1,\dots ,T$  do Collect global state  $s_t$  and agents' partial observations  $\mathbf{o}_t = (o_t^1,\dots ,o_t^n)$  from environments. if  $t$  mod  $k = 0$  then Sample skills  $Z,z^{1:n}\sim \pi_h(Z,z^{1:n}|s_t,\mathbf{o}_t)$  end for each agent  $g^{i}\in \mathcal{N}$  do Sample action  $a_{t}^{i}\sim \pi_{l}(a_{t}^{i}|o_{t}^{i},z^{i})$  end Execute joint actions  $a_{t} = (a_{t}^{1},\dots ,a_{t}^{n})$  in environments and collect the team reward  $r_t$  next state  $s_{t + 1}$  and next observations  $\mathbf{o}_{t + 1}$  . for each agent  $g^{i}\in \mathcal{N}$  do Compute low-level reward  $r_t^i = \lambda er_t + \lambda_D\log q_D(Z|s_{t + 1}) + \lambda_d\log q_d(z^i |o_{t + 1}^i,Z)$  Store  $(s_t,Z,o_t^i,z^i,a_t^i,r_t^i)$  into  $\mathcal{B}_l$  end if t mod  $k = k - 1$  then Store  $(s_{t - k - 1},\mathbf{o}_{t - k - 1},Z,z^{1:n},\sum_{p = 0}^{k - 1}r_{t - p})$  into  $\mathcal{B}_h$  end Store  $(s_{t + 1},Z,\mathbf{o}_{t + 1},z^{1:n})$  into  $\mathcal{D}$    
end # The training phase Sample a random minibatch of data from  $\mathcal{B}_h$  to update  $\theta_h$  and  $\phi_h$  by minimizing  $\mathcal{L}_h(\theta_h,\phi_h)$  Sample a random minibatch of data from  $\mathcal{B}_l$  to update  $\theta_l$  and  $\phi_l$  by minimizing  $\mathcal{L}_l(\theta_l,\phi_l)$  Sample a random minibatch of data from  $\mathcal{D}$  to update  $\phi_D$  and  $\phi_d$  by minimizing  $\mathcal{L}_d(\phi_D,\phi_d)$
B Derivation of the Variational Lower Bound
In this section, we adopt the structured variational inference to formulate our objective based on the PGM in Fig. 2(b). In structured variational inference, approximate inference is performed by optimizing the variational lower bound [14]. Let  $\tau = (s_{0:T}, o_{0:T}, a_{0:T}, Z, z^{1:n})$ , the variational lower bound is given by:
$$
\begin{array}{l} \log p (\mathcal {O} _ {0: T}) = \log \int p (\mathcal {O} _ {0: T}, \tau) d \tau \\ = \log \int p \left(\mathcal {O} _ {0: T}, \tau\right) \frac {q (\tau)}{q (\tau)} d \tau \\ = \log \mathbb {E} _ {\tau \sim q (\tau)} \left[ \frac {p \left(\mathcal {O} _ {0 : T} , \tau\right)}{q (\tau)} \right] \\ \geq \mathbb {E} _ {\tau \sim q (\tau)} \left[ \log p \left(\mathcal {O} _ {0: T}, \tau\right) - \log q (\tau) \right], \\ \end{array}
$$
(a)  $Z = 1,z^{1} = 1,z^{2} = 0$
(c)  $Z = 0,z^{1} = 3,z^{2} = 2$
(e)  $Z = 0,z^{1} = 2,z^{2} = 3$
(b)  $Z = 0,z^{1} = 3,z^{2} = 2$
(d)  $Z = 1,z^{1} = 1,z^{2} = 0$
(f)  $Z = 1,z^{1} = 1,z^{2} = 0$
Figure 9: Visualization of learned team and individual skills on Alice_and_Bob. (a-b), (c-d), (e-f) show the skills learned by the agents with three different initial positions, respectively.
where the inequality in the last line is derived using Jensen's inequality. We assume that the action prior distribution  $p(\pmb{a}_t)$  is a uniform distribution. Then, we have:
$$
p \left(\mathcal {O} _ {0: T}, \tau\right) = p \left(s _ {0}\right) \prod_ {t = 0} ^ {T} p \left(\boldsymbol {o} _ {t}\right) p \left(s _ {t + 1} \mid s _ {t}, \boldsymbol {a} _ {t}\right) p \left(\mathcal {O} _ {t} \mid s _ {t}, \boldsymbol {a} _ {t}\right) p (Z \mid s _ {t}) p \left(z ^ {1} \mid o _ {t} ^ {1}, Z\right) \dots p \left(z ^ {n} \mid o _ {t} ^ {n}, Z\right)
$$
$$
q (\tau) = q (s _ {0}) \prod_ {t = 0} ^ {T} q (\boldsymbol {o} _ {t}) q (s _ {t + 1} | s _ {t}, \boldsymbol {a} _ {t}) q (Z | s _ {t}) q (z ^ {1} | o _ {t} ^ {1}, Z) \dots q (z ^ {n} | o _ {t} ^ {n}, Z) q (a _ {t} ^ {1} | o _ {t} ^ {1}, z ^ {1}) \dots q (a _ {t} ^ {n} | o _ {t} ^ {n}, z ^ {n}).
$$
Similar to [14], we fix  $q(s_0) = p(s_0), q(\pmb{o}_t) = p(\pmb{o}_t), q(s_{t+1}|s_t, \pmb{a}_t) = p(s_{t+1}|s_t, \pmb{a}_t)$  and let  $p(\mathcal{O}_t|s_t, \pmb{a}_t) = \exp(r(s_t, \pmb{a}_t))$ . The bound reduces to:
$$
\begin{array}{l} \log p \left(\mathcal {O} _ {0: T}\right) \geq \mathbb {E} _ {\tau \sim q (\tau)} \left[ \log p \left(\mathcal {O} _ {0: T}, \tau\right) - \log q (\tau) \right] \\ = \mathbb {E} _ {\tau \sim q (\tau)} \left[ \sum_ {t = 0} ^ {T} \left(r \left(s _ {t}, \boldsymbol {a} _ {t}\right) + \log p (Z | s _ {t}) + \sum_ {i = 1} ^ {n} \log p \left(z ^ {i} \mid o _ {t} ^ {i}, Z\right) \right. \right. \\ \left. \left. - \log q (Z | s _ {t}) - \sum_ {i = 1} ^ {n} \log q (z ^ {i} | o _ {t} ^ {i}, Z) - \sum_ {i = 1} ^ {n} \log q (a _ {t} ^ {i} | o _ {t} ^ {i}, z ^ {i})\right) \right]. \\ \end{array}
$$
C Visualization of skills on Alice_and_Bob
We visualize the learned team and individual skills on Alice_and_Bob by the agents with different initial positions in Fig. 9. With different initial positions, agents can learn similar team and individual skills. Specifically, team skills  $Z = 0$  and  $Z = 1$  guide the whole team to collect the blue diamond and the red diamond, respectively. Individual skills  $z^i = 0$ ,  $z^i = 1$ ,  $z^i = 2$  and  $z^i = 3$  guide the individual agent to reach the red diamond, red button, blue diamond and blue button, respectively.
(b)
Figure 10: More ablations for HMASD. (a) The performance of HMASD with different skill intervals. (b) The performance of HMASD with different numbers of team skills. (c) The performance of HMASD with different numbers of individual skills.
D More Ablations
We conduct more ablations about the skill interval  $k$ , the number of team skills  $n_Z$  and the number of individual skills  $n_z$  in Fig. 10. Results reveal that HMASD performs poorly with too short or too long skill intervals, which demonstrates that an appropriate number of timesteps is necessary for each skill to be learned effectively. Furthermore, it can be observed that the performance of HMASD varies greatly with different numbers of team skills and individual skills, which shows that HMASD is not very stable on sparse reward multi-agent tasks. The limitations of our work are discussed in Appendix G.
E Hyperparameter Setting
For all baselines, we use the open source code of the original paper. We implement HMASD based on the codebase of MAT and MAPPO. The hyperparameters for different tasks are presented in Table 1-3. In particular, we want to emphasize the setting of  $\lambda_{e}$ , i.e., the weight of team reward in the low-level reward. It guides the skills to be useful for the team performance. On the simple Alice_and_Bob, the state-observation space is small, and it's easy for skills to explore those states that contribute to the team reward. So we set  $\lambda_{e} = 0$ . On the complex SMAC with 0-1 reward and Overcooked, the state-observation space is large, and it's very difficult to explore those rare states that induce positive team reward. We set  $\lambda_{e} = 100$ , which is much larger than the weights of intrinsic rewards  $\lambda_{D}$  and  $\lambda_{d}$ . In this way, the team reward can dominate the low-level reward when agents encounter positive team reward, which guides the agents to learn to complete the task more quickly.
Table 1: Common hyperparameters used for HMASD, MAT and MAPPO across all tasks.
hyperparameters	value	hyperparameters	value	hyperparameters	value
training threads	16	rollout threads	32	hidden size	64
use valuenorm	True	use orthogonal	True	gain	0.01
optimizer	Adam	optimizer epsilon	1e-5	weight decay	0
ppo epoch	15	clip param	0.2	num mini batch	1
value loss coef	1	use gae	True	gae lambda	0.95
gamma	0.99	use huber loss	True	huber delta	10
Table 2: Common hyperparameters used for HMASD, MAT and MAPPO in different tasks.
task	lr	episode length	num env steps	eval episodes	eval rollout threads
Alice_and_Bob	5e-4	100	3e6	100	1
SMAC with 0-1 reward	1e-4	100	2e6	100	4
Overcooked	1e-4	400	1e7	32	8
Table 3: Different hyperparameters used for HMASD in different scenarios.
scenario	k	nZ	nz	λh	λl	λe	λD	λd
Alice_and_Bob	50	2	4	0.1	0.01	0	0.1	0.2
3m	25	3	3	0.001	0.01	100	0.1	0.5
2s_and_1sc	25	2	5	0.03	0.01	100	0.1	1
2m_and_1z	50	3	2	0.005	0.01	100	0.1	1
cramped_room	25	3	3	0.01	0.01	100	1	0.5
asymmetric_advantages	25	3	3	0.01	0.01	100	0.1	0.1
coordination_ring	10	3	3	0.1	0.01	100	0.15	0.1
(a) Individual skill 0
(b) Individual skill 1
Figure 11: Visualizations of 3 learned individual skills on the SMAC scenario  $3\mathfrak{m}$ . To visualize the  $i^{th}$  individual skill, we set all agents' individual skills to be  $i$ , i.e.,  $z^1 = z^2 = z^3 = i$ , where  $i = 0,1,\dots ,n_z - 1$ . (a) For individual skill 0, agents learn to spread out to both sides to form an arc formation and focus fire on the same enemy at the same time, which can quickly reduce the number of enemies and win the game. (b) For individual skill 1, agents learn to explore the bottom left area of the map, where there are no enemies. In the end, agents lose the game for not killing all enemies within the time limit. (c) For individual skill 2, agents learn to move right to approach enemies but not attack them in time, which leads to losing the game.
(c) Individual skill 2
(a) Individual skill 0
Figure 12: Visualizations of 2 learned individual skills on the SMAC scenario  $2\mathrm{m\_vs\_1z}$ . (a) For individual skill 0, agents learn to take advantage of attack range to kite the enemy. The agent closer to the enemy will run away, and the other agent will attack the enemy. In this kiting way, the two agents take turns attacking the enemy and ultimately win the game. (b) For individual skill 1, agents learn to explore the bottom area of the map, while the enemy is in the upper area. Finally, agents fail to find the enemy and lose the game.
(b) Individual skill 1
F More Fine-grained Results
Percentage of skills useful for the team reward We count the average percentage of useful individual skills among all learned individual skills on the SMAC. For each SMAC scenario, we conduct 5 different runs with different random seeds. We learn 3, 5, 2 individual skills for each run on  $3\mathrm{m}$ ,  $2\mathrm{s\_vs\_1sc}$ ,  $2\mathrm{m\_vs\_1z}$ , respectively. Therefore, we learn  $3*5 + 5*5 + 2*5 = 50$  individual skills on three SMAC scenarios. After our test, only 12 individual skills are useful for the team performance. In other words, only  $24\%$  of the learned individual skills are useful for completing the task on average after training.
Percentage of runs that learn meaningful behavior For every scenario, we conduct 5 runs. For each run on the SMAC scenario, once agents discover useful skills, the performance will quickly
increase to 1. So the final performance is either 1 or 0, which leads to a large variance among 5 runs. There are 7 scenarios in this work. Among  $5 * 7 = 35$  runs, 26 runs could learn meaningful behavior. Therefore, the percentage of the model to learn meaningful behavior among all trials is  $26 / 35 = 74.3\%$ . One of our future goals is to increase this percentage and reduce the variances of our method.
Visualization of the learned skills We visualize the learned individual skills on SMAC as shown in Fig. 11 and Fig. 12. We can observe that only one skill can result in a non-zero team reward on both scenarios, and the remaining skills explore the state-observation spaces that don't contribute to the team reward. As for the learned skills on Overcooked, we find that in those high-performance runs, all skills are useful and each of them can perform well individually. This may be because Overcooked is a grid world task and has a smaller state-observation space than SMAC. Thus those states with positive rewards are easier to be explored by all skills. These visualizations reflect the potential inefficiency of HMASD in solving complex tasks with large state-observation space.
In this section, we discuss three potential limitations of HMASD. Firstly, when the state-observation space is large, HMASD can discover diverse skills but maybe only some of them (about  $24\%$  on SMAC) are useful for the team reward. There are a large percentage of skills are assigned to explore those zero-reward states. Secondly, the number of team skills and individual skills should be carefully adjusted. When applying HMASD to a new task, it requires to adjust several hyperparameters as shown in Table 3. Thirdly, HMASD can only learn team skills for the entire team, which lacks flexibility in multi-agent tasks that require team skills within sub-teams. Our future work aims to improve these limitations and make HMASD better at solving sparse-reward multi-agent tasks.
H Comparison of HMASD with the Multi-Agent Exploration Baseline
In this section, we compare HMASD with an exploration bonus MARL baseline, EITI/EDTI [46], on Overcooked. EITI/EDTI proposes exploration strategies where agents start with decentralized exploration driven by their individual curiosity (i.e., individual exploration), and are also encouraged to coordinate their exploration (i.e., joint exploration). The final reward for each agent is the sum of team reward, individual exploration bonus and joint exploration bonus. EITI and EDTI provide 2 ways to calculate the joint exploration bonus. The comparison of HMASD with EITI/EDTI on Overcooked is shown in Table 4.
We can see that HMASD outperforms EITI and EDTI on all Overcooked scenarios. Although EITI/EDTI encourages both individual exploration and joint exploration, it doesn't formulate the relationship between individual exploration and joint exploration. These two explorations may have conflicts, causing neither exploration to work well. In our work, we propose to discover both team and individual skills for solving sparse-reward multi-agent tasks. Importantly, we build a probabilistic graphical model to formulate the relationship between team skill and individual skill, and then derive a lower bound as our objective. The results show that our method could discover significant team and individual skills, and effectively combine them to accomplish the sparse-reward multi-agent task.
Table 4: The final episode reward of HMASD and EITI/EDTI on Overcooked.
Scenario	EITI	EDTI	HMASD
cramped_room	0.122 ± 0.013	0.119 ± 0.015	236.0 ± 8.0
asymmetric_advantages	0.008 ± 0.002	0.004 ± 0.002	295.1 ± 84.5
coordination_ring	0.0 ± 0.0	0.0 ± 0.0	80.0 ± 97.9