SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mingyu Yang $^{1}$ , Yaodong Yang $^{2\dagger}$ , Zhenbo Lu $^{3\dagger}$ , Wengang Zhou $^{1,3}$ , Houqiang Li $^{1,3}$
<sup>1</sup>University of Science and Technology of China, <sup>2</sup>Institute for AI, Peking University  
<sup>3</sup>Institute of Artificial Intelligence, Hefei Comprehensive National Science Center ymy@mail.ustc.edu.cn, yaodong.yang@pu.edu.cn luzhenbo@iai.ustc.edu.cn, {zhwg, lihq}@ustc.edu.cn
Skill discovery has shown significant progress in unsupervised reinforcement learning. This approach enables the discovery of a wide range of skills without any extrinsic reward, which can be effectively combined to tackle complex tasks. However, such unsupervised skill learning has not been well applied to multi-agent reinforcement learning (MARL) due to two primary challenges. One is how to learn skills not only for the individual agents but also for the entire team, and the other is how to coordinate the skills of different agents to accomplish multi-agent tasks. To address these challenges, we present Hierarchical Multi-Agent Skill Discovery (HMASD), a two-level hierarchical algorithm for discovering both team and individual skills in MARL. The high-level policy employs a transformer structure to realize sequential skill assignment, while the low-level policy learns to discover valuable team and individual skills. We evaluate HMASD on sparse reward multi-agent benchmarks, and the results show that HMASD achieves significant performance improvements compared to strong MARL baselines.
Multi-agent reinforcement learning (MARL) has recently demonstrated remarkable potential in solving various real-world problems, such as unmanned aerial vehicles [1], autonomous driving [2] and traffic light control [3]. Despite its broad applications, current MARL algorithms [4, 5] typically require well-crafted team or individual rewards to guide the agents to learn policies for efficient coordination. This limitation hinders the generalization of MARL to the sparse reward multi-agent tasks, where agents receive a non-zero reward only when they coordinate to achieve a challenging goal. Compared to single-agent tasks, the sparse reward problem in multi-agent tasks poses more challenges. On the one hand, the joint state and action spaces of multi-agent tasks increase exponentially with the number of agents, which exacerbates the difficulty for agents to explore those valuable but rare states. On the other hand, we need to distribute the received sparse reward not only to different timesteps but also to different agents [6].
To solve sparse reward multi-agent problems, a promising approach is to discover underlying skills within the multi-agent task and effectively combine these skills to achieve the final goal. For example, in a football game, there are various skills involved such as dribbling, passing, receiving and shooting. A common training strategy for a football team is to let each player learn these fundamental skills first and then train players together to cooperatively use these skills for scoring the goal, which is more efficient than directly training players without any skills together. Similarly, in a sparse reward multi-agent task, it's difficult for agents to cooperatively achieve a challenging goal from scratch. We can first try to discover the underlying useful skills within the task, which is much easier than solving the entire task under the sparse reward setting [7, 8]. Although these discovered skills may
†Corresponding authors: Yaodong Yang and Zhenbo Lu
not independently achieve the goal or induce non-zero rewards, we can learn to effectively combine them to accomplish the final task. In short, it is possible to decompose a sparse reward multi-agent task into a combination of different skills, which greatly simplifies the task complexity.
Previous works mainly follow two paradigms to discover skills in MARL. One is to let all agents learn a shared team skill [9, 10], which promotes team cooperation behaviors but suffers from high complexity. The other does the opposite and learns skills only from the perspective of individual agents [11, 12], which is more efficient since individual skills are usually easier to learn than team skills. However, merely learning individual skills may be insufficient to achieve team objectives.
Taking football as an example again, individual skills can refer to the technical abilities of individual players, such as dribbling and passing, while team skills refer to the ability of players to work together as a whole, i.e., the overall team tactics, such as wing-play and tiki-taka [13]. As shown in Fig. 1, the goal of a football team is to let all players coordinate to master useful team skills from a global perspective. Directly learning the team skill is usually too complex. A better way is to decompose the team skill into different individual skills for players from an individual perspective, and ensure that the joint behavior of all players can form the team tactic. Both team skills
and individual skills are important to a successful team. Therefore, a primary challenge for multiagent skill discovery is how to simultaneously learn the individual skill for each individual agent and the team skill for the whole team, and the second challenge is how to combine these skills to accomplish multi-agent tasks.
To this end, we present a new paradigm for discovering both team and individual skills in MARL. Inspired by probabilistic inference in RL [14], we embed the multi-agent skill discovery problem into a probabilistic graphical model (PGM). With the PGM, we can formulate multi-agent skill discovery as an inference problem, allowing us to take advantage of various approximate inference tools. Then, we derive a variational lower bound as our optimization objective by following the structured variational inference [14], and propose Hierarchical Multi-Agent Skill Discovery (HMASD), a practical two-level hierarchical MARL algorithm for optimizing the derived lower bound. Specifically, the high-level policy employs a transformer [15] structure to assign skills to agents, with inputs consisting of the sequence of global state and all agents' observations, and outputs consisting of the sequence of team skill and agents' individual skills. In this way, an agent's individual skill depends on the team skill and all previous agents' individual skills, allowing agents to choose complementary skills and achieve better skill combination. The low-level policy chooses primitive actions for each agent to interact with the environment conditioned on the assigned skills. Moreover, we introduce two skill discriminators to generate intrinsic rewards for agents to learn diverse and distinguishable skills. Finally, we show that HMASD achieves superior performance on sparse reward multi-agent benchmarks compared to strong MARL baselines. To our best knowledge, our work is the first attempt to model both team skills and individual skills with the probabilistic graphical model in MARL.
In this section, we first introduce the problem formulation and notations for cooperative MARL. Then, we describe the concept of skill and a mutual information based objective for skill discovery in unsupervised RL. We finally present the framework of representing RL as a probabilistic graphical model. More related works are discussed in Sec. 5.
In this work, we consider a fully cooperative MARL problem, which is usually described as a decentralized partially observable markov decision process (Dec-POMDP) [16]. A Dec-POMDP is
Figure 1: Illustration of team skill and individual skill in football.
(b)
Figure 2: (a) RL can be cast as an inference problem by considering a basic probabilistic graphical model consisting of states  $s_t$ , actions  $\pmb{a}_t$  and optimality variables  $\mathcal{O}_t$ . (b) Similarly, we formulate multi-agent skill discovery as an inference problem by augmenting the basic probabilistic graphical model with team skills  $Z$ , individual skills  $z^i$  and observations  $o_t^i$ .
defined by a tuple  $G = \langle \mathcal{N},\mathcal{S},\mathcal{A},P,r,\Omega ,O,\gamma \rangle$ , where  $\mathcal{N}\equiv \{g^1,g^2,\dots ,g^n\}$  is the set of  $n$  agents and  $S$  is the global state space of the environment. All agents share the same action space  $\mathcal{A}$  and the joint action space is  $\mathcal{A}^n$ . At timestep  $t$ , each agent  $g^{i}\in \mathcal{N}$  chooses an action  $a_{t}^{i}\in \mathcal{A}$ , where  $i\in \{1,2,\dots ,n\}$  is the agent identity. The actions of all agents form a joint action  $\pmb {a}_t\in \mathcal{A}^n$ . By executing  $\pmb{a}_{t}$ , the environment transitions to the next global state  $s_{t + 1}\sim P(s_{t + 1}|s_t,\pmb {a}_t)$ , and all agents receive a shared team reward  $r(s_{t},\pmb{a}_{t})$ . Each agent  $g^{i}$  can only observe a partial observation  $o_{t}^{i}\in \Omega$  according to the observation probability function  $O(s_{t},g^{i})$ . The joint observation of all agents is denoted as  $o_{t}$ .  $\gamma \in [0,1)$  is the discount factor. The objective is to learn a joint policy  $\pi$  that maximizes the expected global return  $\mathbb{E}\left[\sum_{t = 0}^{\infty}\gamma^{t}r_{t}|\pi \right]$ . In particular, this paper focuses on the sparse reward setting, where the team reward  $r_t$  is zero for most timesteps.
2.2 Mutual Information based Skill Discovery
Skill discovery is a popular paradigm in unsupervised RL, which enables agents to discover diverse skills without a reward function. A skill is represented by a latent variable  $z$ , which is used as an additional input to the policy and results in a latent-conditioned policy  $\pi(a|s,z)$ . The intention behind skill discovery is to expect the skill to control which states the agent visits and different skills to visit different states. This can be achieved by a simple objective of maximizing the mutual information between state  $s$  and skill  $z$ , i.e.,  $\mathcal{I}(s;z) = \mathcal{H}(z) - \mathcal{H}(z|s)$ . It's challenging to directly optimize the mutual information. Therefore, a majority of skill discovery methods [17-20] derive a variational lower bound for the mutual information as follows:
$$
\mathcal {I} (s; z) = \mathbb {E} _ {s, z \sim p (s, z)} \left[ \log p (z | s) - \log p (z) \right] \geq \mathbb {E} _ {s, z \sim p (s, z)} \left[ \log q _ {\phi} (z | s) - \log p (z) \right], \tag {1}
$$
where  $q_{\phi}(z|s)$  is a learned model parameterized by  $\phi$  to approximate the intractable posterior  $p(z|s)$ . Then, we can treat  $\log q_{\phi}(z|s) - \log p(z)$  as the intrinsic reward to optimize the latent-conditioned policy  $\pi (a|s,z)$ , where different  $z$  correspond to different skills.
2.3 Probabilistic Graphical Model for RL
The probabilistic graphical model (PGM) is a powerful tool for modeling complex and uncertain systems. It provides a graphical representation of the relationship between variables in a system, where nodes correspond to random variables and edges represent conditional dependencies. In recent years, PGM has been widely used in RL to model complex decision-making tasks [14, 21-24]. In this work, we follow the basic PGM for RL [14], which embeds the control problem into a graphical model and formulates it as an inference problem. As shown in Fig. 2(a), it first models the relationship among states, actions, and next states based on the dynamics  $P(s_{t + 1}|s_t,\pmb {a}_t)$ . To incorporate the reward function, it introduces a binary random variable  $\mathcal{O}_t$  called optimality variable into the model, where  $\mathcal{O}_t = 1$  denotes timestep  $t$  is optimal, and  $\mathcal{O}_t = 0$  indicates timestep  $t$  is not optimal. The probability distribution over  $\mathcal{O}_t$  is  $p(\mathcal{O}_t = 1|s_t,\pmb {a}_t) = \exp (r(s_t,\pmb {a}_t))$ . Refer to [14], we then perform structured variational inference to derive the final objective, which is to optimize a variational lower bound (also called evidence lower bound). The evidence is that  $\mathcal{O}_t = 1$  for all  $t\in \{0,\dots ,T\}$ . We will use  $\mathcal{O}_t$  to denote  $\mathcal{O}_t = 1$  for conciseness in the remainder of this paper.
The variational lower bound is given by:
$$
\log p \left(\mathcal {O} _ {0: T}\right) \geq \mathbb {E} _ {s _ {0: T}, \boldsymbol {a} _ {0: T} \sim q \left(s _ {0: T}, \boldsymbol {a} _ {0: T}\right)} \left[ \sum_ {t = 0} ^ {T} r \left(s _ {t}, \boldsymbol {a} _ {t}\right) - \log q \left(\boldsymbol {a} _ {t} \mid s _ {t}\right) \right], \tag {2}
$$
where  $q(\boldsymbol{a}_t|s_t)$  is the learned policy. Optimizing this lower bound corresponds to maximizing the cumulative reward and the policy entropy at the visited states, which differs from the standard RL objective that only maximizes reward. The entropy term can promote exploration and prevent the policy from becoming too deterministic. This type of RL objective is sometimes known as maximum entropy RL [25, 26].
In this section, we present our solution for learning both team and individual skills in MARL. We first model the multi-agent skill discovery problem with a PGM and derive a tractable variational lower bound as the optimization objective. We then propose a practical MARL algorithm to optimize the derived lower bound.
3.1 Multi-Agent Skill Discovery as an Inference Problem
In this work, we study the skill discovery problem in multi-agent tasks. One way for multi-agent skill discovery is to treat all agents as one big virtual agent and directly learn skills from the perspective of the whole team, which can improve teamwork but suffer from high complexity. Another way is to learn skills from the perspective of each individual agent, which reduces the complexity but lacks collaboration. To combine the advantages of two ways, we propose to learn skills not only from the individual perspective but also from the team viewpoint.
Specifically, we utilize a latent variable denoted by  $Z \in \mathcal{Z}$  to represent the skill of the entire team, referred to as team skill. And we use a latent variable denoted by  $z^i \in \mathcal{X}$  to represent the skill of agent  $g^i$ , referred to as individual skill. The individual skills of all agents are represented by  $z^{1:n} \in \mathcal{X}^n$ . In this work, both the team skill space  $\mathcal{Z}$  and the individual skill space  $\mathcal{X}$  are defined as discrete spaces consisting of finite latent variables with one-hot encoding, where the number of team skills and individual skills are denoted as  $n_Z$  and  $n_z$ , respectively. The team skill  $Z$  is acquired from the global view and is expected to control the global states that the whole team visits, while the individual skill  $z^i$  is developed through an individual perspective and is intended to control the partial observations accessed by agent  $g^i$ . Besides, since individual behaviors are typically based on the overall team strategy, the individual skill  $z^i$  should depend on the team skill  $Z$ . According to these intuitions, we employ a PGM illustrated in Fig. 2(b) to model the multi-agent skill discovery problem, where the team skill  $Z$  is conditioned on the global state  $s_t$  and the individual skill  $z^i$  is conditioned on both the team skill  $Z$  and agent  $g^i$ 's partial observation  $o_t^i$ . With the PGM, we formulate multi-agent skill discovery as an inference problem. Then, we perform structured variational inference to derive our objective, which is to optimize a variational lower bound as follows:
$$
\begin{array}{l} \log p \left(\mathcal {O} _ {0: T}\right) \geq \mathbb {E} _ {\tau \sim q (\tau)} \left[ \sum_ {t = 0} ^ {T} \left(r \left(s _ {t}, \boldsymbol {a} _ {t}\right) + \underbrace {\log p (Z \mid s _ {t}) + \sum_ {i = 1} ^ {n} \log p \left(z ^ {i} \mid o _ {t} ^ {i} , Z\right)} _ {\text {d i v e r s i t y t e r m}}\right) \right. \tag {3} \\ \underbrace{-\log q(Z|s_{t}) - \sum_{i = 1}^{n}\log q(z^{i}|o_{t}^{i},Z)}_{\text{skill entropy term}} - \underbrace{-\sum_{i = 1}^{n}\log q(a_{t}^{i}|o_{t}^{i},z^{i})}_{\text{action entropy term}})\Big], \\ \end{array}
$$
where  $\tau = (s_{0:T},\pmb{o}_{0:T},\pmb{a}_{0:T},Z,z^{1:n})$  is the joint trajectory containing states, observations, actions and skills. The detailed derivation of Eq. 3 is shown in Appendix B.
Optimizing this lower bound is to maximize the team reward  $r(s_{t}, \boldsymbol{a}_{t})$  and three terms. The diversity term shows the probability of skills on their corresponding states and observations, which can be maximized to encourage different skills to visit different states and observations, and is a crucial element for learning diverse skills. The skill entropy term and the action entropy term reflect the entropy of skills and actions at the visited states and observations, respectively. Optimizing the two entropy terms can enhance the exploration during skill learning.
Figure 3: The overall framework of HMASD. At the high level, the skill coordinator adopts a transformer structure to assign team skill  $Z$  and individual skills  $z^{1:n}$  to agents. At the low level, the skill discoverer chooses primitive action  $a^i$  for agent  $g^i$  conditioned on the assigned skill  $z^i$  and forms a joint action  $\pmb{a}$  to interact with the environment. The environment returns global state  $s$ , joint observation  $o$  and team reward  $r$ . To make the skills diverse and distinguishable, two skill discriminators are employed to generate intrinsic rewards  $r_{in}$  for the skill discoverer.
3.2 Hierarchical Multi-Agent Skill Discovery
In this subsection, we present Hierarchical Multi-Agent Skill Discovery (HMASD), a practical two-level hierarchical MARL algorithm designed to optimize the derived lower bound in Eq. 3. To estimate the probability distributions in the lower bound, we utilize four approximate functions. Specifically, we first employ a team skill discriminator  $q_{D}(Z|s_{t})$  to approximate  $p(Z|s_t)$ . Then, we use an individual skill discriminator  $q_{d}(z^{i}|o_{t}^{i},Z)$  to approximate  $p(z^i |o_t^i,Z)$ . Moreover, a skill coordinator  $\pi_h(Z,z^{1:n}|s_t,\pmb {o}_t)$  is utilized to approximate both  $q(Z|s_t)$  and  $q(z^{i}|o_{t}^{i},Z)$ , while a skill discoverer  $\pi_l(a_t^i |o_t^i,z^i)$  is applied to approximate  $q(a_{t}^{i}|o_{t}^{i},z^{i})$ . As shown in Fig. 3, these approximate functions are integrated into a two-level hierarchical structure for multi-agent skill discovery. At the high level, the skill coordinator assigns team skill  $Z$  and individual skills  $z^{1:n}$  to agents every  $k$  timesteps, where  $k\in \mathbb{N}^{+}$  is the number of timesteps between two consecutive skill assignments and is called skill interval. At the low level, the skill discoverer employs a latent-conditioned policy for each agent to explore the assigned skills using intrinsic rewards generated by the skill discriminators. Below, we describe how to learn these approximate functions for optimizing the derived lower bound.
Skill Coordinator Transformer [15] has recently shown great potentials in MARL [27-31]. In this work, we take inspiration from MAT [31], which applies a transformer structure to map the agents' observation sequence into the agents' action sequence. Similar to MAT, we employ a transformer structure for the skill coordinator in our method as shown in Fig. 3. Specifically, we take the sequence of state and observations  $(s,o^1,o^2,\dots ,o^n)$  as inputs and embed them into vectors with the same dimension. Then, the embedded sequence is passed through a state encoder, which contains several encoding blocks. Each encoding block consists of a self-attention mechanism, a multi-layer perceptron (MLP) and residual connections. The state encoder outputs a sequence denoted as  $(\hat{s},\hat{o}^{1},\hat{o}^{2},\dots ,\hat{o}^{n})$ , which encodes  $(s,o^1,o^2,\dots ,o^n)$  into informative representations and is used to approximate the high-level value function  $V_{h}$ . After encoding the state and observations, we assign skills in an auto-regressive way. We start with an arbitrary symbol  $Z^0$ , which is embedded and then fed into a skill decoder containing several decoding blocks. Each decoding block consists of two masked self-attention mechanisms, an MLP and residual connections, where the query of the second masked self-attention mechanism is the output of state encoder. The output of skill decoder is then fed to an MLP to generate the high-level policy  $\pi_h(Z|\hat{s},\hat{o}^{1:n},Z^0)$ . We sample a team skill  $Z$  from the policy and insert it back into the decoder to generate  $z^1$ . By analogy, after  $n + 1$  decoding rounds, we get team skill  $Z$  and all agents' individual skills  $z^{1:n}$ . In this way, agent  $g^{i}$ 's individual skill  $z^{i}$
depends on team skill  $Z$  and all previous agents' individual skills  $z^{1:i-1}$ , which can prevent skill duplication and allow agents to choose complementary individual skills based on the team skill.
Skill Discoverer After assigning skills sequentially, we learn a shared skill discoverer to explore the assigned skills for agents. The skill discoverer consists of a decentralized actor and a centralized critic as shown in Fig. 3. The decentralized actor takes the partial observation  $o^i$  and individual skill  $z^i$  as inputs, and outputs the low-level policy  $\pi_l(a^i | o^i, z^i)$  that chooses action  $a^i$  for each agent  $g^i$ . All agents share a centralized critic to approximate the low-level value function  $V_l(s, Z)$  based on the global state  $s$  and team skill  $Z$ . In our implementation, both the actor and the critic consist of an MLP, a GRU [32] and another MLP. The actor aims to learn the assigned individual skill  $z^i$  for each agent  $g^i$ , while the critic takes a global view and expects to guide the joint behavior of all agents to discover the assigned team skill  $Z$ .
Skill Discriminator To make the learned skills diverse and distinguishable, we learn two skill discriminators, a team discriminator and an individual discriminator. The team discriminator inputs the global state  $s$  and outputs the probability of each team skill  $Z \in \mathcal{Z}$ , denoted as  $q_{D}(Z|s)$ . In addition, we employ an individual discriminator  $q_{d}(z^{i}|o^{i},Z)$ , which takes the partial observation  $o^i$  and team skill  $Z$  as inputs and then outputs the probability of each individual skill  $z^i \in \mathcal{X}$ . Both discriminators are composed of an MLP. The team discriminator aims to discriminate the team skill based on the global state, while the individual discriminator is designed to discriminate the individual skill given the observation and team skill. The two skill discriminators are utilized to generate intrinsic rewards  $\log q_{D}(Z|s)$  and  $\log q_{d}(z^{i}|o^{i},Z)$ , respectively.  $\log q_{D}(Z|s)$  is to reward all agents to jointly explore global states that are easy to discriminate, while  $\log q_{d}(z^{i}|o^{i},Z)$  rewards each agent  $g^{i}$  for visiting those easily distinguishable observations given the team skill  $Z$ . In other words, it encourages different skills to explore different areas of the state-observation space. Since if two skills explore the same state, this state will be hard to discriminate and then lead to low intrinsic rewards. Therefore, the intrinsic rewards can guide agents to learn diverse and distinguishable skills.
Overall Training and Execution According to Eq. 3, we need to optimize the team reward, the diversity term, the skill entropy term and the action entropy term. At each episode, the high-level skill coordinator assigns skills to agents every  $k$  timesteps, then the low-level skill discoverer learns to explore the assigned skills for agents during these  $k$  timesteps. We define the sum of team rewards over these  $k$  timesteps as the single-step reward for the high-level policy, i.e., the high-level reward can be written as  $r_t^h = \sum_{p=0}^{k-1} r_{t+p}$ . For the low-level policy, we use a combination of extrinsic team reward and intrinsic rewards, i.e., the low-level reward for agent  $g^i$  is:
$$
r _ {t} ^ {i} = \lambda_ {e} r _ {t} + \lambda_ {D} \log q _ {D} (Z | s _ {t + 1}) + \lambda_ {d} \log q _ {d} (z ^ {i} | o _ {t + 1} ^ {i}, Z), \qquad (4)
$$