Heterogeneous Skill Learning for Multi-agent Tasks
Authors: Yuntao Liu
Yuan Li
Xinhai Xu
Yong Dou
Donghong Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
multi-agent reinforcement learning
cooperative multi-agent system
mutual information

Abstract: Heterogeneous behaviours are widespread in many multi-agent tasks, which have not been paid much attention in the community of multi-agent reinforcement learning. It would be a key factor for improving the learning performance to efficiently characterize and automatically find heterogeneous behaviours. In this paper, we introduce the concept of the skill to explore the ability of heterogeneous behaviours. We propose a novel skill-based multi-agent reinforcement learning framework to enable agents to master diverse skills. Specifically, our framework consists of the skill representation mechanism, the skill selector and the skill-based policy learning mechanism. We design an auto-encoder model to generate the latent variable as the skill representation by incorporating the environment information, which ensures the distinguishable of agents for skill selection and the discriminability for the skill learning. With the representation, a skill selection mechanism is invented to realize the assignment from agents to skills. Meanwhile, diverse skill-based policies are generated through a novel skill-based policy learning method. To promote efficient skill discovery, a mutual information based intrinsic reward function is constructed. Empirical results show that our framework obtains the best performance on three challenging benchmarks, i.e., StarCraft II micromanagement tasks, Google Research Football and GoBigger, over state-of-the-art MARL methods.

Heterogeneous Skill Learning for Multi-agent Tasks
Yuntao Liu
Yuan Li Academy of Military Science Beijing, China liyuan@nudt.cm edu. cn
Xinhai Xu
Academy Military Science Beijing, China wtt--yytfoxmmai..mmm
Academy Military Science Beijing, China xuxinhai @nudt edu. cn
Yong Dou
Donghong Liu
National University of Defense Technology
Academy of Military Science
Hunan Changsha, China douyongōnudt. edu.ttteddccc..
Beijing, China liu_ Liu_donghhong@sina.com com
Abstract
Heterogeneous behaviours are widespread in many multi-agent tasks, which have not been paid much attention in the community of multi-agent reinforcement learn- ing. It would be a key factor for improving the learning performance to efficiently characterize and automatically find heterogeneous behaviours. In this paper, we introduce the concept of the skill to explore the ability of heterogeneous behaviours. We propose novel skill based multi-agent reinforcement learning framework to enable agents master diverse skills. Specifically, our framework consists of the skill representation mechanism, the skill selector and the skill-based policy learning mechanism. We design an auto-encoder model to generate the latent variable as the skill representation by incorporating the environment information, which ensures the distinguishable of agents for skill selection and the discriminability for skill learning. With the representation, skill selection mechanism is invented to real- ize the assignment from agents to skills. Meanwhile, diverse skill-based policies are generated through novel skill-basee policy learning method. To promote efficient skill discovery, a mutual information based intrinsic reward function is constructed. Empirical results show that framework obtains the best perfor- mance on three challenging benchmarks, i.e., StarCraft II micromanagement tasks, Google Research Football and GoBigger, over state-of-the-arr MARL methods.
Introduction
The cooperative multi-agent reinforcement learning (MARL) has obtained great improvements in many multi-agent systems, such as real-time strategy games (2), artificial swarms robots [I3] and autonomous vehicles [4]. The original solution is the independent learning method, in which each agent treats other agents as part the environment. The main drawback that simultaneous learning of all agents will result in non-stationary environment from the perspective of each agent 3.. To avoid this problem, fully centralized learning approaches [18] have been studied, which aim to learn an optimal joint action of agents in response to the joint observation. However, the solution space grows exponentially with the number of agents, making it hard to find a good policy in limited time [4] Later, following the centralized training with decentralized execution (CTDE) paradigm, value decomposition based MARL methods have been proposed which attracts wide attention 21 33 B1] The main idea is that each agent leams its policy based local observation while a centralized mechanism is designed to guide the training of all agents. In most such methods,
"Corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

agents share their neural networks to compute actions, which is known as the parameter sharing technique. In this way, the training efficiency significantly improved due to the reduction of the number of parameters, and enough samples shared by all agents in the centralized training procedure. However, heterogeneous behaviours are hard to be generated for such kind of methods when facing complex multi-agent tasks.
To address this challenge, some recent methods learn different roles for agents based on their observations and actions. Agents are grouped into different roles which carry out different tasks.
ROMA [2 assumes a Gaussian distribution for learning roles and RODE [2]] uses pre-trained model with random samples to restrict the action space for each role. However, roles are sometimes not easy to be distinguished based on only observation-action pairs. Moreover, the diversity of heterogeneous behaviours is hard to be fully explored. Recently. CDS [4] introduces diversity into the MARL approach by adding an extra local network for each agent to the shared network. It tries to make balance between diversity and the high learning efficiency brought by parameter sharing. However, extra introduced networks for all agents hinder the application of CDS on large-scale tasks, which will be illustrated in the experiment.
Different from previous works, we consider studying heterogeneous behaviours in multi-agent tasks by introducing the skill learning [20]. The main idea is to enable agents to learn diverse skills, and then select proper skills for agents to complete certain tasks. For instance, in Google Research Football game. if multiple useful skills such as off-the-ball moving, dribbling with the ball, passing and shooting can be learned, agents can later use these skills to easily cooperate to win the match in a short policy search time. Introducing skill learning into MARI. approaches has two advantages. On one hand, distinct skill representation will help to generate diverse policies and enable agents to select heterogeneous behaviours. On the other hand, the skill learning process is compatible with the parameter sharing technique, which does not bring much overhead for training efficiency. Then three questions arise: (I) how skills are represented; (2) how skills are learned:(3) how agents select proper skills.
There are some works studying the problem of skill generation in the single-agent area. Early works used the dessnnde reward function to discover skills [.]. However, expert knowledge is often difficult to acquire. Autonomous acquisition of useful skills without human-supervised signals can be divided into two types. One is adopting mutual information maximization for skill discovery [6]. It uses model-free unsupervised RL. methods to learn distinguishable skills which can perform distinct tasks. Other works [5] 19 consider skill discovery with intrinsic motivation which drives the exploration of agents. However, for those works, skills are learned one by one from different single-agent scenarios. This is quite different from multi-agent tasks, in which different kinds of skills must be learned in one scenario. Moreover, multiple agents should make decisions on the selection of skills and the decision will change as the game goes on. For multi-agent tasks, HSD (32) studies the problem of skill discovery with hierarchical RL method. However, it does not make full consideration of distinguishable and diverse skill discovery, leading to poor performance for heterogencous problems.
To learn effective skills in aalti- tasks, we propose a novel skill-based learning framework. We design an auto-encoder model to generate the latent variable as the skill representation. To reflect the potential characters of agents and ensure diversity of skills, a set of skill-ids as well as observations and actions of agents are used as the input while the reward of the environment and the next observations are used as the output. With this representation, we devise skill selection mechanism that assigns an agent with a skill. Due to characters encoded in the skill representation, this mechanism could select proper skills for agents based on their observations. Finally, we introduce the information -hhoortionthhoorttii paradigm of mutual information to discover distinguishable and diverse skills. The optimization objective of mutual information is further represented as an intrinsic reward, which together with the environment reward is used to perform skill-based policy learning. For skill-based policy learning, we still adopt the parameter sharing technique, but we further compute the latent variable conditioned policy for each skill. In this way, our framework learns distinguishable skills and encourages each skill to explore the environment and perform its task more effectively.
We make extensive experiments to evaluate our framework on three challenging benchmarks, i.e., StarCraft II micromanagement environment, Google Research Football and GoBigger. To our best knowledge, the proposed framework achieves state-of-ther performance in all tasks.
2

---

Skill Represeclation
Skill Selector
Skill-based Pollcy Larning
Andigumatt
SuiL Pool
Skilk
Ageat!
Solli
Revard
fo Seetrttentt, 2ttent, (.ooaae))) /kl,of/kkityj 12....11
Figure 1: Schematics of HSL. It consists of three parts: skill representation, skill selector and skill-based policy learning. The skill representation generates latent variables z_latentfor the skill selector. The skill selector is equivalent to assignment module, which helps select skills for agents. The last part is responsible to generate diverse skills.
Background
Multi-agent Markov Games: We consider a partially observable multi-agent game, which can
be modeled as multi-agent extension of Markov Decision Processes (34). It is denoted by is the state space, Ô the partial observation space of agents, A, is the action of agent i, R; S A1 X ... x Ax - R the reward function of agent i and = x A x ... x AN x S + [o, 1] is the state transition function. A popular solution approach for multi-agent games is the value decomposition based MARL. Each agent associated with a neural network, which is used to compute the Q-value Q; based its local observation o,. During the training process, global Q-value Qtot computed as a function of Q;, for example, the summation used VDN 24 and the mixing network used in QMIX 21
(N, S, O, {A (.. where N a finite set n agents.
Mutual information: Mutual information is a core quantity in the information-theory that measures the dependence between random variables using a Shannon entropy- -baedd approach, defined as Equation
I(x;)= x(x) -((()))
(1)
According the definition, maximizing the mutual information I(::) means maximizing the entropy H(X and minimizing the conditional entropy H(X{Y) at the same time.
In the context of reinforcement learning. X refers to the function of the state and Y is the function of actions. Maximizing the mutual information encourages the entropy of the state high, encouraging exploration in policy learning. Skill-based RL methods such as 6 [ adopt the mutual information to perform diverse skill discovery.
3 Method
In this section, we present a novel heterogeneous skill learning MARL framework (HSL) method for multi-agent tasks. The framework is shown in Figure 1. We introduce the concept skill which is latent-conditioned (Z conditioned) policy (A|S, 2) to enable agents to perform diverse behaviours. Suppose there are M skills, of which ids are encoded in one-hot manner, and they are denoted by 2j.j 1, ..M. We first encode the environment information, i.e, observations and actions of agents, with skill-ids as latent skill variables 2 sllatenty.J representation. Based on latent variables as key features, we design a skill assignment mechanism, which associates each agent with a skill such that agents could learn heterogeneous policies. With skill representations and skill assignments, diverse skill policies are learned through the skill learning mechanism.
1, 1,2, ....M This process is called skill
3

---

3.1 Skill Representation Learning
Skill representation learning generates latent skill variables, which are used for skill assignment and skill learning. On one hand, generated latent skill variables should reflect the characters of different agents, which helps agents make proper choices over different skills. On the other hand, they provide key information that makes the learned skills as different as possible.
To achieve the aforementioned goals, we utilize key information from the environment such as the environmental state transition function and the reward function for skill representation. We design an auto-encoder model to generate skill variables. The encoder is denoted by fe (-; fe) where 8g represents parameters of the model. It encodes variable = =_latentj = fe(=;; 0e). There are two decoders with the same input which is concatenation of the observation og, the action a, of agent and all latent skill variables = =_latent = (aatteentt==aaaatitaaaaaaaaa [=_latent,l) = 1, 2.... M ). The output of one decoder, denoted by far( 0ar), is the estimation of environment reward while that of the other decoder, denoted by f:o(;; ddo), is the estimation of next observation of agent i. The learning objective of the auto-encoder model is shown Equation (2). With the first two terms, the auto-encoder model could generate latent skill variables which reveal the effect of different skills observations, actions and the environment. Consequently, it is beneficial for the skill selector to distinguish agents when their states and actions are similar.
one-hot vector of skill-id 2j to the latent skill
(re, Ode Odo) = (KLL(fao(=llttet o,, a.))(()))
+ Ef or, a.) - -r) >2E5s cos cos((a()))) (2)
where A1, X2 are the scaling factors.
The third term in Equation 2 aims to maximize the cosine distance between any two skills, which encourages diverse policy learning for the skill learning module. The skill representation learning mechanism is first trained in the early exploration of our framework. After the early exploration is finished, we fix the parameters of the encoder in this module to generate representation features for all skill ids.
3.2
Skill Selector
Here, we introduce a skill selection mechanism that assigns each agent a skill. This procedure essentially assigns multiple skills to multiple agents. It is different from the single-agent skill-based RL problem, which only assigns multiple skills to one agent.
Relying only observation features may result in the skill selector assigning the same skill to agents with similar observation features. Therefore, as shown the middle part of Figure 1, the skill selector chooses skills based on latent observation features of agents and the latent skill variables generated in Section 3.1 The latent observation feature o o_latent; is obtained through an MLP and a GRU unit based the observation o,, which is similar to other MARL methods. Then o latent, is disposed by another MLP module, of which the output multiplied by the latent skill variables x j_ matrices, i.e., o __latent and 2 __lteent and get the matrix of local Q-values Q" of all agents. We Ω represent the state space of the skill selector and A to represent the action space. The Q-values of all agents' actions are denoted by QT, Qi,.., 0% In the execution mode, the index corresponds to the maximum value is the skill that should be assigned the agent For the training of the skill selector, the learning objective is shown as Equation (3. Similar to the work in 21]. mixing network is introduced compute oi Note that the assignment is computed every k steps, thus the reward for each assignment computed as Ek=breee. Moreover, in the early training stage, we encourage exploration and incentivize the skill selections to be diverse as possible. The policy with high entropy forces the skill selector to explore distinguished states for agents and chooses different skills for agents, which benefits the training procedure. Therefore, maximizing the policy entropy is also a goal G = H[Q in our objective. We replace it with an intrinsic reward r - [log Q: (o_latent; =_ latent:)) and add into Equation B for maximizing the expectation g.
1, 2.., M. Then
apply matrix multiplication of these two
L, |>*=]" + =Br ymax Qui Se (Re4k, AT (oe, A,
(3)

---

3.3 Skill-based Policy Learning
In this section, we introduce skill learning mechanism, which consists of skill discovery process and skill-basee policy learning process. The main idea of skill discovery includes two aspects. The first is that skills should do effective exploration in the environment, which means that different skills should explore effectively and access different states of the environment. Ideally, several skills should cover as much of the exploration space of the environment as possible. The second aspect that skills should be learned as diverse as possible. In order to avoid duplication policies, each skill should compute the action that differs significantly from the others. For example, in cooperative football games, it is clear that the two skills of passing and shooting the ball should compute different actions so that the respective tasks can be completed.
We introduce mutual information from information theory to realize the aforementioned objectives. To achieve the first goal. we encourage each skill 2 to associate with the observation o, i.e. maximizing the mutual information I(o; 2) between skills and observations, where Z represents the mutual information. As for the diversity of skills, we hope that different skills generate actions as differently as possible. Therefore, we can strengthen the correlation between the action and the skill through maximizing the mutual information Z(a; conditioned on the given state o. Finally, we can improve the diversity of skills by maximizing the policy entropy Hfaao, z]), where H[] is the Shannon entropy. In summary, the final maximization objective is shown as Equation (4):
pm = [(;; 2) + I(a; 2|0) + H[
(H[=) H(20o]) + (H[alo) H(alo, =]) H[alo,
(4)
For term ② in Equation
4 using the toolkit of variational inference and applying the bound from
[ we can construct the variational lower bound which is shown in Equation (5) based on Theorem 1 in Appendix. (the proof is shown in Appendix A).
log p(zla, o)
H(a H(alo, =] > Ep(a.s.0)
H[a
(5)
p(a, 2|0)
Further, we can transform ()) [Tog
p(=la,0) pla,=lo)
into Equation (6)
- Eppa, [H[p(z]a, o)]] E (p
log p(=(o)
+ E Ept.,) [H[p(al2,o)) o)]]
(6)
Inserting Equation and 66 into Equation 4 yields Equation are eliminated (see Appendix A for details).
Note that H[[2|o] in term ! and
T > H[[[(=]] + Ep(=.0) [H[p(a|z, l]] - Ep(a, [H[p(z|a,o]]
(7)
p(z) is the skill prior distribution. We fix this prior distribution to be uniform to ensure its maximiza- tion entropy inspired by (6. To make full use of the information contained in all actions, inspired by [10), we the Boltzmann Softmax distribution consistent with the original Q function with the greedy policy, i.e., p(a) 2, o) = Softmax(qe. (a|z, o)). of the information. Since is In this way, it takes into account all actions and makes maximum difficult to exactly compute p(z)a, o) covering all states, actions and skills, we make a variational approximation to p(z(a,o) and optimize the lower bound of the variational posterior distribution qe, (2|a, ja, o).
generate policy distribution, which is
The final optimization objective is shown Equation S
m E (oa log Softmax (al2, 0:2 log qo (=la, o)]
The discovered skills be chosen by agents to perform skill-based policy learning. Each agent computes its skill-based policy Q;(a,, o,(2,) where 2ÿ is the representation the selected skill. To perform policy learning based on the global reward from the environment, mixing network to mix all individual Q-values into a global Q-value Qtotal. The TD-loss of the skill-based policy learning be described as Equation (9)
QMIX-style
CTD = EE (1+ max Qio (o', a') - (tor(o, a))
(9)
5

---

XMM2Z
3652 v2 3.62
27m 30m
130
82
C01
Tel- Win
63
Too Wax
50
20
20-
20
100
corrdor
100
ves
1.0
023
RGBC
Ter
Tal
60
1ED
1,0
0.0
1.0
3.0
5.0
0.00
0.25
1.00
(mi)
1 [xxil)
1 (mmm))
Figure 2: Performance comparison on SMAC.
3.4 Overall Optimization Objective
Finally, we describe how skill selector, skill discovery and skill-based policy learning are combined in our framework. It organized as a bi-level learning structure.
Inspired by [6) the optimization goal r" of skill discovery embedded as an intrinsic reward in Cp for skill-basee policy learning, which is described as follows.
Cp = r + 3m + max Qioeld', a') - (ooe(o, ai)
(10)
a'
We train our framework end-to-end by optimizing the following objective function:
L n,C, + Cp
(11)
where Cg is the objective function for the skill selector and 1/s is the scaling factor in Section 3.2
Experiments
In this section, we the performance of our framework on three challenging multi-agent tasks, i.e.,
StarCraft II micromanagement multi-agent challenge (SMAC) [22), Google Research Football (GRF)
15 and GoBigger [8] We compare our approach (HSL) with classical multi-agent value decomposi- tion methods. i.c., QMIX [21] and QPLEX (26), role-based methods, i.e., ROMA (00MAA27)RRDD)))))
RODE 28).
diversity- based method CDS and skill-based method HSD [32 All experiments are conducted over five random seeds. The detailed setting of the experimental setup is described in Appendix B.2.
4.1 Performance
SMAC: SMAC is cooperative multi-agen task, in which each agent cooperates with teammates to kill enemies. Maps in SMAC can be classified as Easy, Hard and Super Hard. In this section, we concentrate one Hard scenario 5m_vs_ and all Super Hard scenarios, i.e., MMM2, 3852_vs_3sd6 27m_vs saiivv__tt0 33m0,, corridor, 6h_vs_8t.
Figure shows comparing results of the proposed method, HSL, with other baseline methods. As we can see, HSL achieves the best performance over all scenarios. the first three scenarios, even if there exists baseline that could achieve satisfactory win rate, HSL still has a much faster convergent speed. In the last three scenarios, HSL obtains the best win rate. Especially for 6h_vs__ _s_________________________ and 5m__vs_____ 6m, HSL improves the win rate by around 10% and 20% respectively, compared to CDS.
It is interesting to see that CDS does not perform well in the scenario 27m 10m0 It is consistent with the explanation in the introduction that CDS is not suited for large-scale scenarios. CDS introduces extra neural networks for each agent. All these extra neural networks do not share

---

parameters in order to ensure diversity. Although HSL also introduces one extra neural network for learning skills, different skill representations ensure diversity and thus parameter sharing still could be applied.
Another note is that the role-based MARI., i.e., RODE, performs better than CDS in the first two scenarios. The reason is that agents in the two scenarios are heterogeneous, which is casy for RODE to distinguish roles based on observations and actions of agents. HSL is still better than RODE by introducing skills. For the last three scenarios in which agents are homogeneous, it is difficult for RODE distinguish roles. HSL could obtain the best performance by discovering diverse skills. In addition, we can observe that HSD cannot win in Super Hard scenarios. The skill discovery mechanism based only on each agent's local observation makes difficult to learn winning strategies effectively.
ve kneper
Croster-ttack
Cermer
Mynann AMN
60-
68
R00L
M
57
QeR
20
15-
0.5
1.0
2.
10
0.6
2.0
1.0
0.0
10.00
2.0
2.0
4.3
6.3
10.0
Figure 3: Performance comparison on GRE
GRF: In GRE, agents work with each other to decide when and where to move and keep the ball as long as possible maximize the chances of scoring. We consider three scenarios, i.e., 3 vs with keeper, Hard Counter-attack and Corner.
Results are shown Figure and again HSL performs the best. clear see that classical value decomposition MARL methods, i.c., QMIX and QPLEX, perform poorly in these scenarios due to the lack of disposing of heterogeneous tasks. Role-based MARL methods cannot achieve satisfactory performance because agents in these scenarios usually get similar states and actions, making it hard to distinguish roles for agents. Due to the effective skill representation and skill selector mechanism, HSL performs better than role-based MARL methods. Further, we could see that CDS is not able to find effective strategies to win on Corner. Still, the reason that the number of agents this scenario is much higher than that in the other two scenarios, which leads to an adverse effect on training extra networks in CDS.
Golbiiper
Cobigger with thars
L23
RCM
0M4
(xih) 12.0
T 1xuh
Figure 4: Performance comparison GoBigger.
GoBigger: In GoBigger tasks, agents control one or more circular balls and cooperate to become large as much as possible by eating food balls and other smaller balls in the environment. Meanwhile, each agent needs to prevent itself from being caten by other larger enemy balls. Agents can move, eject, split and stop, which could be discretized as 13 actions, including moving, ejecting and splitting in four orthogonal directions and stopping. For each episode, the team with the largest overall size will win at the end of the game.
We show the performance comparison in Figure 4 We can observe that these two maps in GoBigger are quite difficult. Only HSL and CDS can learn winning policies. Role-based MARL methods do not work at all due to the reason explained before. In GoBigger, becomes quite hard for role-based MARL methods to find roles based on observations and actions. Due to the effective skill representation and skill assignment, agents in HSL. can select useful skills to cooperate with each other to cat more food and eliminate enemies.
7

---

4.2
Ablation Study
HaF-HHud Counterr stteck
Gabogger: wa
1.93
ba lccr
Mug Megamnn
bis Samun
Seoad
300
10
10-
1.0 Tix40
2.2
2.
4.3
12.0 2.
6.0
12.0
15.0
Figure 5: Ablation studies.
In this section, we conduct ablation experiments evaluate the influence of the three components: (A) Skill representation learning; (B) Skill selector; (C) Skill-based policy learning. To test component A, we replace representations of skills with raw states, which denoted by IISL No Skill Repr. For evaluating component B, we introduce HSL Random Skill, where each agent randomly selects a skill. Also, we remove the intrinsic reward in the learning objective of the skill selector, which associated with HSL No Skill Intri. Reward. To test component C, we consider HSL No State which removes term O. and HSL No Action which removes term 2 and in Equation
Results are shown in Figure 5 HSL. Random Skill obtains the worst performance, which indicates that the skill selector plays key role in the performance of HSL. Compared with HSL Random Skill, HSL No Skill Repr. achieves higher performance in SMAC:óh_us ___________ j__z and GRF:Hard Counter-attack, since the raw state of agents could also work for skill assignment. However, it still performs worse than our skill representation technique. It becomes obvious in the task Gobigger: _s __v__, SM No Skill Repr. does not work at all. HSL No Action achieves higher performance than HSL No State does in SMAC:óh_ vs 8z and GRF:Hard Counter-attacd which shows that the term ! in (4) is important in these scenarios. In contrast, other terms (4) are important in HSL No State because the performance is higher in this scenario. As for the intrinsic reward in the skill selector, we can observe that the performance gap between HSL. and HSL. Skill Intri. Reward in GRF:Hard Counter-attack and Gobigger:3_we 3_vs_3 is larger than that in SMAC:6h vs__t. The reason is that dense rewards in SMAC:6h _165__s___________________ _s__ss_sssss help the skill selector learn effective skill selection policies. Conversely, the sparse reward in Hard Counter-attack and the inefficient reward in Gobigger:3 _s_ increase the policy learning difficulty the skill selector in HSL No Skill Intri. Reward.
4.3 Skill Demonstration
In this section, conduct case study on scenario SMAC: 6h_ to demonstrate what our skill-based framework learns. We carry out an experiment with the obtained strategy and give a skill demonstration in Figure 6 It is clear to see that our framework has learned 3 skills including escaping (red), attacking (green) and kiting (yellow). The middle part shows the skill selection of agents during the game. For example, Agent 1 selects the red skill at the beginning and the yellow skill until dies. The top and the bottom part give visualizations of discovered skills.
In the first two graphs, two agents select the escaping skill to absorb damage and other agents select the attack skill to produce damage to enemies. In the third graph, one agent close to enemies selects escaping skill. Four other agents select the kiting skill to run in the opposite direction to make damage to enemies. In the fourth graph and fifth graph, two of the alive agents still choose the kiting skill. Agent 3 first chooses the attacking skill since it is far away from enemies, and soon it chooses to escape when enemies are approaching. In the last graph, an escaped agent is died, while the remaining agents select the kiting skill to eliminate enemies. Results GRF and GoBigger benchmarks are detailed in Appendix B.3.
5 Related Work
Deep MARL algorithms have drawn broad attention in recent years. Most works take advantage of the CTDE paradigm 99] for cooperative policy learning for agents. Policy-based MARL algorithms design a centralized critic to compute gradients for decentralized actors such as MADDPG(T6). COMA 77] and DOP [30]. Value-based MARL algorithms achieve centralized optimization and

---

3k Kite
Skink Escape
Skillz:Attaee Attack
54222.0
Skill2: Attack
SHILL
Skill3: Kite
Skill3:
Skill3: Kite
Figure 6: Skill demonstration on SMAC: 6b_vs_8t.
decentralized execution via value decomposition. These methods introduce a mixing network to mix the local value functions of each agent to estimate the joint value function. VDN (25). QMIX (21) and QTRAN [23] have gradually improved the decomposition capabilities of the mixing networks. Qatten refines the representation of the joint value function by implementing multi-head attention formation to approximate the value decomposition procedure. Qplex (26) designs a duplex dueling
34
network to encode the IGM principle for higher representation capabilities of the IGM class.
Although many MARL works under the CTDE paradigm have achieved promising results, most of them apply the parameter sharing technique to share knowledge and experience among agents to compute cooperative actions. However, when faced with complex multi-agent tasks, centralized training and parameter sharing lead to difficulties in generating heterogeneous behavior. Therefore, a series of works have been developed to learn heterogeneous policy learning in MARL. ROMA introduces the concept of role to compute independent actions for different agents. RODE 28 decomposes the action space into sub-groups in order to reveal proper roles for agents for heterogeneous policy learning. Recent works introduce diversity enable agents learn diverse policies to solve complex tasks. MAVEN 17 designs a latent space to learn heterogeneous policies and encourage exploration. EITI and EDIT 20 to perform independent exploration to optimize team performance. EMC [35] proposes a curiosity- driven exploration method make full use of the explored valuable experiences for each agent's efficient policy learning. FOI 12 learns each agent's 's policy via the combination of the gradient from the intrinsic value function and the joint value function. CDS 4 adds an extra local policy network for each agent and tries to make balance between the high learning efficiency brought by diversity and parameter sharing. The most relevant work HSD 32) which encourages skill discovery in MARL settings. It proposes livle hierarchy to discover cooperative skills for each agent. However, HSD selects skills for agents only based on historical observations, which leads to poorly distinguishable skill discovery. Skill-based policy learning mechanism in HSD cannot learn diverse skills because it is also conditioned on skill features. By comparison, our approach does not totally rely on trajectories, but instead designs skill representation mechanism and introduces intrinsic rewards for skill selection. Furthermore, our approach computes intrinsic targets for skill discovery and biled policy learning via mutual information maximization.
27
use mutual information encourage each agent

---

Conclusion
This paper proposes novel skill-based MARL framework deal with the heterogeneous problem in complex cooperative multi-agent tasks. The concept of skill is introduced, which is represented as latent variables generated by an auto-encoder model. The representation reflects the characters of agents and ensures the diversity of skills, which play important role in generating and selecting skills for agents. The skill selection mechanism outputs skill for an agent based on its observation and skill representation. The selection is dynamically changed with the real-time game state, which helps agents use proper skills to adapt to complex situations. To ensure efficient policy learning, we propose skill-based policy learning mechanism based on mutual information optimization under the information theory paradigm. Empirical results on three challenging MARL benchmarks show that our framework significantly pushes forward the performance of state-of-the-ar MARL methods. We expect that our approach could provide valuable guidance for future MARL in using skills to solve complex cooperative problems in the real world.
7 Acknowledge
This work was partially supported by the National Natural Science Foundation of China (No.61902425) and the Open Fund of Science and Technology on Parallel and Distributed Pro- cessing Laboratory (WDZC20205500004)) We would like to thank the anonymous reviewers for their valuable suggestions.
References
[1] David Barber Felix Agakov. 2004. The im algorithm: variational approach to information maximization. (2004).
[2] Kai Arulkumaran, Antoine Cully, and Julian Togelius. 2019. Alphastar: An evolutionary com- putation perspective. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 314-315.
[3] Michael Bowling and Manuela Veloso. 2000, An analysis of stochastic game theory for multiagent reinforcement learning. Technical Report. Carnegie-Mellon Univ Pittsburgh Pa School of Computer Science.
[4] Li Chenghao, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjic Zhang 2021. Celebrating Diversity in Shared Multi- Aaeti-Agent Reinforcement Learning. Advances in Neural Information Processing Systems 34 (2021).
[5] Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. 2021. Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning. arXiv preprint arXiv:2106.011 01444 (2021).
[6] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. 2018. Diversity is All You Need: Learning Skills without a Reward Function. International Conference on Learning Representations.
[7] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson.
2018. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
[8] Go-Bigger. 2021. Go-Bigger: Multi-Agent Decision Intelligence Environment. https:// github. com/ cawabbbcooopeendlaaiii Arrendilab/GoBiigee (2021).
[9] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. 2017. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems. Springer, 66-83.
[10] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. 2017. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning. PMLR, 1352-1361.
10

---

[11] Dylan Modfield-Menelle Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca D Dragan. 2017. Inverse reward design. Proceedings of the 31st International Conference on Neural Information Processing Systems. 9768-6777.
[12] Jiechuan Jiang and Zongqing Lu. 2021. Conference on Machine Learning. PMLR, 4992-5001.
The emergence of individuality. In International
[13] Zahi Kakish, Karthik Elamvazhuthi, and Spring Berman. 2021. Using reinforcement learn- ing to herd robotic swarm to a target distribution. In International Symposium Distributed Autonomous Robotic Systems. Springer, 401-414.
[14] Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. 2021. Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions on Intelligent Transportation Systems (2021).
[15] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. 2019. Google re- search football: A novel reinforcement learning environment. arXiv preprint arXiv: (2019).
[16] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-compettivve environments. Advances in neural information processing systems 30 (2017).
[17] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. 2019. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems 32 (2019).
[18) Laetitia Matignon, Guillaume Laurent, and Nadine Le Fort-Piat. 2012. Independent reinforce- ment learners in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Review 27, 1 (2012), 1-31.
[19] Simone Parisi, Victoria Dean, Deepak Pathak, and Abhinav Gupta. 2021. Interesting Object, Cu- rious Agent: Learning Task-Agnostic Exploration. Advances in Neural Information Processing Systems 34 (2021).
[20] Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J Lim. 2021. Guided Reinforcement Learning with Learned Skills. In Self-Supervision for Reinforcement Learning Workshop-ICLR 2021.
[21] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. 2018. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning. PMLR, -295-4304,
[22] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nan- tas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. 2019. The starcraft multi-agent challenge. arXiv preprint 1eXiv:1902.04:00 (2019).
[23] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. 2019. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning. PMLR, 5887-5896.
[24] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et 2017. Value- decomposition networks for cooperative multi-agent learning. arXiv preprint 170600299 (2017).
[25] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et 2018. Value- Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. 2085-2087.

---

[26] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang, 2021. QPLEX: Duplex Dueling Multi-Agent Q-Learning. In International Conference on Learning Representations.
[27] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. 2020. ROMA: Multi-Agent Re- inforcement Learning with Emergent Roles. In International Conference Machine Learning. PMLR,
[28] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. 2021. RODE: Learning Roles to Decompose Multi-Agent Tasks. International Conference on Learning Representations.
[29] Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. 2020. selluence--Bsed Multi-Agent Exploration. In International Conference on Learning Representations.
[30] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. 2020. Dop: Off-policy multi-agent decomposed policy gradients. In International Conference on Learning Representations.
[31] Zifan Wu, Chao Yu, Deheng Ye, Junge Zhang. Hankz Hankui Zhuo, et al. 2021. Coordinated Proximal Policy Optimization. Advances Neural Information Processing Systems 34 (2021).
[32] Jiachen Yang, Igor Borovikov, and Hongyuan 7ha. 2020. Hierarchical Cooperative Multi- Agent Reinforcement Learning with Skill Discovery. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems. 1566-1574.
[33] Yaodong Yang. Jianye Hao, Guangyong Chen, Hongyao Tang, Yingfeng Chen, Yujing Hu, Changjie Fan, and Zhongyu Wei. 2020. Q-value path decomposition for deep multiagent reinforcement learning. In International Conference on Machine Learning. PMLR, 10706- 10715.
[34] Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. 2020. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03933 (2020).
[35] Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. 2021. Episodic Multi-agent Reinforcement Learning with Curiosity-driven Exploration. Advances in Neural Information Processing Systems 34 (2021).
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix D.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix E.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
3.3 and Appendix A.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main exper- imental results (either in the supplemental material or as URL)? [No] The code is proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix B.2.
12

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See all results in Section 4
(d) Did you include the total amount compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix B.4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasngg new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes) See Appendix B.3.
(b) Did you mention the license of the assets? [Yes] See [22], 115 and [8)
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "dNOBm-1eKFv", "reviewer": "Reviewer_6BRg", "summary": "The paper proposes a framework for multi-agent reinforcement learning with the goal of selecting heterogeneous behaviors and allocating them to agents so that an optimal policy can be achieved. The proposed framework represents skills as latent variables which are used to assign skills so that in the end agents can learn heterogeneous policies. The proposed framework is tested on three different multi-agent tasks and compared with other methods. In addition, an ablation study is included.", "questions": "In addition to commenting to the two points raised in \"Strengths and weaknesses\" -- i.e., the non direct applicability to complex scenarios and the comparison with non-MARL classic methods:\n\nA few details that could make the paper clearer:\n- the methods compared appear to be from the same group; it is worth to include and/or discuss the choice for the comparison. For example, MAVEN could be used as comparison.\n- the differences between the scenarios in each multi-agent task are not clear, as only some arbitrary naming is used. \n- it is worth to make examples of skills and actions as the way the paper discusses them they appear to be the same.\n\nThere are a few language problems, including:\n- \"attracts widely attention\" -> \"attracts wide attention\"\n- space after ; -- e.g., \"are represented;(2)\" -> \"are represented; (2)\"\n- \"dispose of\" -> \"consider\" \n- \"With the representation\" -> \"With this representation\"\n- \"one-hot\" -> \"one-shot\"\n- \"should explore differently and access different states of the environment.\" There is different repeated, and \"explore differently\" could be more specific.\n- \"described as follows Equation (10).\" -> \"described as follows.\"\n- \"comparing results\" -> \"comparative results\"\n- \"Another notification\" -> \"Another note\"\n- \"sparce\" -> \"sparse\"", "limitations": "The appendix includes a section where the main limitations are discussed and providing some some potential venues of future work. It is worth to discuss how the framework could scale to more complex real-world scenarios.\n\nThe section about the impact discusses that overall the method itself should not have a direct negative outcome, however notes that in practical applications they could occur. Some examples and ways to address them could be discussed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper appears to inspiration from a few other papers that are cited, including [4] and [32], but the view taken by the paper in identifying skills to improve MARL is significant to potentially achieve cooperative tasks. The paper is also overall well written, providing the intuition on the main choices made for the proposed framework and presenting sound technical details. It is appreciated also the experiments in three different problems and the evaluation with other methods, as well as the ablation study.\n\nWhile the paper shows an interesting direction, the tasks are relatively constrained making the significance of the proposed framework more limited. Indeed, as noted also in the limitations, the framework does not appear to be able to learn appropriate policies when more skills are present.\n\nIn addition, the paper could discuss not only the literature on MARL, but also classic methods of heterogeneous multi-agent task allocation, which is currently missing, including\n- Wu, H., Ghadami, A., Bayrak, A. E., Smereka, J. M., & Epureanu, B. I. (2021). Impact of Heterogeneity and Risk Aversion on Task Allocation in Multi-Agent Teams. IEEE Robotics and Automation Letters, 6(4), 7065-7072.\n- Korsah, G. A., Stentz, A., & Dias, M. B. (2013). A comprehensive taxonomy for multi-robot task allocation. The International Journal of Robotics Research, 32(12), 1495-1512.\n- Emam, Y., Mayya, S., Notomista, G., Bohannon, A., & Egerstedt, M. (2020, May). Adaptive task allocation for heterogeneous multi-robot teams with evolving and unknown robot capabilities. In 2020 IEEE International Conference on Robotics and Automation (ICRA) (pp. 7719-7725). IEEE.\n- Schillinger, P., Bürger, M., & Dimarogonas, D. V. (2018). Simultaneous task allocation and planning for temporal logic goals in heterogeneous multi-robot systems. The international journal of robotics research, 37(7), 818-838.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O7IxswKA8zs", "reviewer": "Reviewer_amao", "summary": "The authors proposed a skill-based MARL framework to enable agents to master diverse skills. The framework consists of the skill representation mechanism with an auto-encoder model, the skill selector to realize the assignment from agents to skills, and the skill-based policy learning mechanism with a mutual information-based intrinsic reward function. Experimental results show that the framework obtains the best performance on three challenging benchmarks, i.e., StarCraft II micromanagement tasks, Google Research Football and GoBigger, over state-of-the-art MARL methods.\n", "questions": "* L45- “However, extra introduced networks for all agents hinder the application of CDS on large-scale tasks, which will be illustrated in the experiment.” The proposed HSL requires extra networks to obtain the skill diversity. Since HSL outperformed CDS in MARL performance, the problem of CDS can be mentioned differently. \n* L248- In association with the above, “Still, the reason is that the number of agents in this scenario is much higher than that in the other two scenarios, which leads to an adverse effect on training extra networks in CDS”. I can understand the former (“the number of agents in this scenario is much higher than that in the other two scenarios”), but the latter (“which leads to an adverse effect on training extra networks in CDS”) was unclear. What did the authors try to mention? \n* Figure 1: How did the authors create the Skill_ID pool? I cannot find this. In particular, I want to know whether this is created in a rule-based or data-driven manner. \n* L142- “This procedure is essentially a many-to-many assignment problem”. However, I cannot find how to solve this problem. How did the authors solve this problem? \n* Appendix A: Eq. (12) is different from eq. (4). If s can be replaced with o, but what is x? Where is the 3rd term of eq. (4)?\nIn Appendix Table 2, n_skill of Hard Counter-attack in GRF was 3, but in Appendix C.1 Skill Demonstration on GRF, the authors trained HSL with 4 skills. Which is correct? \n", "limitations": "The limitations and potential negative societal impact were described in Appendices D and E, respectively. However, the latter was not concrete (“Therefore, we do not anticipate a direct negative outcome. In practical applications involving our method, potentially negative outcomes might occur.”).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of the paper is as follows:\n* Novelty: The authors proposed the skill representation mechanism with an auto-encoder model, the skill selector to realize the assignment from agents to skills, and the skill-based policy learning mechanism with a mutual information-based intrinsic reward function.  \n* Clear results: Experimental results show that the framework obtains the best performance on three challenging benchmarks, i.e., StarCraft II micromanagement tasks, Google Research Football and GoBigger, over state-of-the-art MARL methods.\n\nThe weakness of the paper is as follows (please also see below “Questions):\n* The presentation was sometimes inconsistent.\n* The method descriptions were sometimes unclear. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EPlkwVwUcmz", "reviewer": "Reviewer_j8Gw", "summary": "The authors suggest a skill-based multi-agent learning algorithm that generates diverse skills for finding heterogeneous behaviours. In the proposed mehtod, there are three mechanisms for capturing heterogeneous skills: skill representation, skil selector and skill-based policy learning. Experiment results show that the proposed method outperformed than other multi-agent methods on three multi-agent RL benchmarks.\n", "questions": "   1. Paragraphs on lines 58-70 should go to the relevant work section.\n\n   2. There is no description how to compute the local $Q_i^v$ in Secion 3.2\n\n   3. The loss function for $q_{\\theta_z}$ is missing.\n\n   4. As previously explained, the reviewer believes that the skill-id is sufficient to be used as a latent skill variable. Could you provide the results of using skill-ids as skill varaibles instead of skill representation method.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts\n   \n  1. The proposed method (HSL) suggests two types of intrinsic rewards for skill selector and policy learning to enable effective exploration on both skill selection and heterogeneous behaviour.\n\n  2. Experiment results show the significant improvements on several benchmarks.\n\nWeakenesses\n  1. The reviewer does not understand why the skill representation mechanism is needed in the proposed framework. \n     1) Two decoders can be trained without the latent skill varialbe, since two decodes already take observation and actions. So I could not agree that the auto-encoder generates latent skill variables that reveal different effect of different skills.\n     2) In equation (2), the cosine distance is used to distinguish latent skill varialbes, but I think the skill-id is already enough for distinguishability. \n  \n  2. The motivation of introducing $H[a|o,z]$ for intrinsic reward $r_m$ is not acceptable. The reviewer thinks that the effect of maximizing policy entorpy is just effective exploration not discrimability of skill.\n\n  3. There is something wrong with constructing the lower bound of the intrinsic reward $r_m$. \n\n        In equation 4, two terms 1 and 2 are conditional entropy, but the author use these conditional entropy as just common shannon entropy.\n        1) $H[a|z, o] =  \\mathbb{E}_{p(z|o)}[H(p(a|z,o))] \\neq H(p(a|z,o))$ is the conditional entropy between random variable $a$ and $z$ given $o$.\n        2) $H[z|o] =\\mathbb{E}_{p(o)}[H(p(z|o))] \\neq H(p(z|o)) $  is the conditional entropy between random variable $z$ and $o$", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept