Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperation
Authors: Sicong Liu
Yang Shu
Chenjuan Guo
Bin Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Multi-Agent Reinforcemenr Learning
Primary area: reinforcement learning
TL;DR: Learning generalizable multi-agent policy from offline multi-task data by improved skill learning

Abstract: Learning cooperative multi-agent policy from offline multi-task data that can generalize to unseen tasks with varying numbers of agents and targets is an attractive problem in many scenarios. Although aggregating general behavior patterns among multiple tasks as skills to improve policy transfer is a promising approach, two primary challenges hinder the further advancement of skill learning in offline multi-task MARL. Firstly, extracting general cooperative behaviors from various action sequences as common skills lacks bringing cooperative temporal knowledge into them. Secondly, existing works only involve common skills and can not adaptively choose independent knowledge as task-specific skills in each task for fine-grained action execution. To tackle these challenges, we propose Hierarchical and Separate Skill Discovery (HiSSD), a novel approach for generalizable offline multi-task MARL through skill learning. HiSSD leverages a hierarchical framework that jointly learns common and task-specific skills. The common skills learn cooperative temporal knowledge and enable in-sample exploitation for offline multi-task MARL. The task-specific skills represent the priors of each task and achieve a task-guided fine-grained action execution. To verify the advancement of our method, we conduct experiments on multi-agent MuJoCo and SMAC benchmarks. After training the policy using HiSSD on offline multi-task data, the empirical results show that HiSSD assigns effective cooperative behaviors and obtains superior performance in unseen tasks.

Published as a conference paper at ICLR 2025
LEARNING GENERALIZABLE SKILLS FROM OFFLINE TASK DATA FOR MULTI-AGENT COOPERATION
Sicong Liu, Yang Shu, Chenjuan Guo, Bin Yang
East China Normal University
sicongliul01 1maail1111mmmm... com, {yshu, sjgue, byngg,, ecnu, edu, cn
ABSTRACT
Learning cooperative multi-agent policy from offline multi-task data that can gen- eralize to unseen tasks with varying numbers of agents and targets is an attrac- tive problem in many scenarios. Although aggregating general behavior patterns among multiple tasks as skills improve policy transfer a promising approach, two primary challenges hinder the further advancement of skill learning in of- fline multi-task MARL. Firstly, extracting general cooperative behaviors from various action sequences as common skills lacks bringing cooperative temporal knowledge into them. Secondly, existing works only involve common skills and not adaptively choose independent knowledge as task-specific skills in each task for fine-grained action execution. To tackle these challenges, we propose Hierarchical and Separate Skill Discovery (HiSSD), a novel approach for gen- eralizable offline multi-task MARL through skill learning. HiSSD leverages a hierarchical framework that jointly learns common and task-specifie skills. The common skills learn cooperative temporal knowledge and enable in- in-sample explo- ration for offline multi-task MARL. The task specific skills represent the priors of each task and achieve task-guided fine fine-grained action execution. To verify the advancement of our method, we conduct experiments on multi-agent MuJoCo and SMAC benchmarks. After training the policy using HiSSD on offline multi-task data, the empirical results show that HiSSD assigns effective cooperative behav- iors and obtains superior performance in unseen tasks. Source code is available at httpp:: thub.. www/ wwtp//ooriiioo cAnna /HiSSD)
INTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) has drawn great attention to many attrac- tive problems such games, intelligent warehouses. automated driving, and social science (Vinyals)
jet al. 2019 Yun et al. 2022 Gronauer & Diepold 2022] Tian et al., 2025)] Shentu et al., 2025 Wu al. 2025 When it comes large-scale tasks, the MARL method yields superior performance compared to the traditional control techniques. In most real-world applications, however, building high-fidelity simulators or deploying online interaction can be costly or even infeasible. Meanwhile, multi-agent systems are expected to perform flcxibility among tasks with varying numbers of agents and targets. To address these issues, training multi-agent policies that can transfer across tasks with various numbers of agents under limited experience has become an attractive direction to tackle real-world multi-agent applications (Wu et al. 2019 Kumar et al.).
Although training the multi-agent policy on single task and fine-tuning the target task is simple way for policy transfer, it has the following drawbacks (Wen al. 2022 Hu al., 2021 Yang et al.)
2022 Long et al., 2020): (i) the fine-tuning stage still requires costly interaction. (ii) it lacks the capacity to handle tasks with various numbers of agents and targets. To overcome these issues, existing works leverage Transformer (Vaswani et al.] 2017) to enable flexible nopulation-invarian framework (Long et al. 2020 Wang et al. 2020). They also discover general cooperative behavior patterns common skills from offline multi task data to improve multi-agent policy transfer. ODIS
"Corresoonding author

---

Published as a conference paper at ICLR 2025
(Zhang et al. 2023) conducts two-stage offline multi-task MARL to discover generalizable multi- agent common skills. They pre-train the common skills from a global view and then optimize the policy by discovering the filleemaximizeed skills the multi-task data. HyGen (Zhang et al. 2024) integrates online and offline learning to ensure both multi-task generalization and training- efficiency. These methods obtained convincing improvement by learning generalizable common skills and reduced interaction costs during policy transfer.
However, existing works only learn general and reusable cooperation behaviors by aggregating co- operative actions from multi-task data. Equipping offline multi-task MARL with skill learning to improve policy transfer remains an issue. Firstly, extracting general cooperative behaviors from var- ious cooperative actions as common skills lack bringing cooperative temporal knowledge into them. Existing works have demonstrated the significance of learning temporal knowledge in multi-agent cooperation (Xu et al. 2022b Song et al. 2023). Secondly, existing works in literature mainly focus on discovering task-irrelevant common knowledge. Yet, few of them consider learning task specifce knowledge which is also beneficial for policy transfer in offline multi-task MARL (Yang al. 2024
Xu et al. 2022a Bose et al. 2024 Ishfaq et al. 2024).
In light of these issues, we propose Hierarchical and Separate Skill Discovery (HiSSD), novel approach for generalizable offline multi-task MARL through skill learning. HiSSD distinguishes knowledge derived from multi-task data into common skills and task-specific skills, utilizing a hi- erarchical framework that concurrently learns both categories. Concretely, the common skill rep- resents the general cooperation patterns that involve cooperative temporal knowledge and enable in-sample dynamics exploration for offline multi-task MARL. The task-specific skill represents the unique knowledge of action execution different tasks and achieves task-guided fine-grained ac- tion execution. Therefore. HiSSD effectively bridges offline multi-agent policy improvement and adaptive multi-task action execution with common and task-specific skills learning.
Overall, our contributions can be summarized as following points: (i) multi-task MARL method that leverages the hicrarchical framework and jointly learns common and task-specific skills. (ii) HiSSD is proposed to learn common skills representing cooperative be- haviors among multiple tasks for offline multi-agent policy exploration and action guidance. (iii) Meanwhile, HiSSD adaptively abstracts task-specific skills for each task to achieve task-guided fine-grained imitation. (iv) We conduct experiments on the SMAC and multi-agent MuJoCo bench- marks. After training policy using HiSSD on offline multi-task data, the empirical results show that HiSSD assigns effective cooperative behaviors and obtains superior performance in unseen tasks.
We present HiSSD, an offline
2
PRELIMINARIES
2.1 COOPERATIVE MULTI-AGENT REINFORCEMENT LEARNING
Cooperative multi-agent reinforcement learning is formulated as decentralized partially observable Markov decision process (Dec-POMDP) Olichook et al. 2016), as the problem is defined by a tuple G S, O, R, 7). Here, K [I....,k }} is the set of agents. The global observation 8 E S is unobservable to each agent in the centralized training and decentralized execution (CTDE) pipeline. Cooperative agent uses local observation Og Ω drawn from the observation function Oss,ki k) to sample actions dg from the actions space A. During interaction, the joint action u = {a), 02, ..., ] leads next state s' P(s'ls, u) and global reward r. The training goal is to learn cooperative multi-agent system to maximize the cumulative reward R. We use 7 = {πI .2., T denote the trajectory of each agent, and the policy evaluation used to estimate the performance of the joint policy π (u|TT) is normally defined by rewards in infinite-horizon tasks,
20
R E
Σ
('r, Us 88+1 1 (T)
where 7 E 1) is the discount factor, F' denotes the reward at time t.
2.2 LEARNING GENERALIZABLE POLICY FROM OFFLINE MULTI-TASK DATA
While cooperative multi-agent reinforcement learning has achieved advancement in many scenarios, it is still hard to transfer the policy to unseen tasks without additional interaction (Hu et al. 2021
2

---

Published as a conference paper at ICLR 2025
Iqbal et al. knowledge across tasks to improve policy transfer. In multi-task learning, tasks within the same task set possess identical types of units while exhibiting varying distributions. For instance, the quantity of agents and targets may differ across these tasks. Denote M as overall tasks, the whole data D-M divided into source task data DSouree (or D") and target task data 'D DTarget, where the target tasks are unseen during training. The goal is to train the multi-agent policy on the source task data that can be transferred unseen tasks in the same task set without additional interaction.
2021). An effective solution is leveraging multi-task learning, extracting generalizable
3 METHOD
In this section, we introduce the Hierarchical and Separable Skill Discovery (HiSSD) framework, a novel approach for addressing offline multi-task multi-agent reinforcement learning problems. Our main solution is leveraging the hierarchical skill learning framework and jointly learning com- mon and task- specific skills among multiple cooperation tasks. We begin by illustrating the overall framework of our offiine multi-lask MARL. We then detail the high-level planner with common skills and the low-level controller with task-specific skills. Finally, we describe the overall objective and training pipeline.
3.1 OFFLINE MULTI-TASK MARL WITH COOPERATIVE SKILL LEARNING
Skill is series of latent variables representing general and reusable knowledge among tasks guide action execution (Zhang et al.] 2023 2024). Besides the solution proposed by existing works, we give two insights into multi-task skill learning for further advancement in policy transfer. Firstly, integrating cooperative temporal knowledge into common skills helps decision-making. It gives both dynamics transition information and global perspective into multi-agent policy. Secondly, learning task-specific skills to guide action execution is beneficial to transfer policy adaptively. It brings each task's unique knowledge into the controller and adjusts the output action distribution. In this way, we propose offline multi-task MARL method that jointly learns common and task-specific skills to improve policy transfer. Figure provides brief illustration of the proposed framework.
Specifically, our method can be divided into two parts: (a) The high-level planner and (b) The low- level controller. The high-level planner contains common skills encoder TH, a forward predictor fo. and a value net V/tot, We feed local observation of:K into Te, to extract common skills ct-K fa receives common skills to output the predicted next global state s and local information I1:K The local information can be seen as the local observation's 's embedding. The value net receives oftK and [i:K to approximate the accumulated reward Σ yr,. The low-level controller includes a task-specific skills encoder 9w and an action decoder TØ The task-specific skills encoder infers task-specific skills 2! K!:K using local observation of The action decoder utilizes local observation, common skills, and task- task-specific skills to generate the real actions {att Te, (-lof of 2 {
Integrating cooperative temporal knowledge into common skills indicates that these skills maintain perceptibility with respect to global dynamic transitions. The transition falls into two parts, the global state transition and the value estimation. Therefore, the common skill is trained to minimize the prediction error to the real next global state $ and maximize the accumulated reward. This training objective facilitates offline exploration while effectively integrating cooperation temporal knowledge into generalizable skills through a local-to-global transition prediction mechanism. As for achieving an adaptive policy transfer, the major request is to distinguish each task's specific knowledge. The task-specific skills are trained through distribution matching with each task priors, enabling adaptive action execution guidance across diverse tasks. Moreover, our method does not require global information during execution, which differs from previous work (Liu et al. 2021 We will describe the details of the proposed skill learning in sections 3.2 and 3.3
3.2 LEARNING HIGH-LEVEL PLANNER WITH COMMON SKILLS
In this subsection, we introduce a high-level planning framework designed for extracting transfer- able skill representations from offline multi-task MARL datasets. We start with the probabilistic inference MARL and construct a training objective to integrate cooperative temporal knowledge into common skills.
3

---

Published as a conference paper at ICLR 2025
(8) High-L evel Planner
(b) Low-Level Controllr
Mirraze
Minimioo
Tax Danitation Approemation
f-1
Value Net Voc
Kawwords
d'
fe
Action Deccder m
2/
Common Skils Encocir fi.
Tisk Spok-Spacff: Skils Frocder Siu
Figure 1: Overall framework HiDS HiSSD utilizes a hierarchical framework that jointly learns common and task-specific skills from offline multi-task data to improve multi-agent policy transfer. (a) The high-level planner with common skills. HiSSD integrates cooperative temporal knowledge into common skills and enables offline exploration. (b) The low-level controller with task-specific skills. The task-specific knowledge can guide the action execution adaptively among tasks. HiSSD) uses an implicit Q-learning objective to train the value network.
Inspired by Levine (2018), we define the problem of discovering the optimal high-level planner with its common skill encoder Ti match the trajectory p(T) given by Eq. 2 Here, p(T) indicates to
maximize K agents' accumulated reward Σ yr in each transition p(sc+ lot e le1::),
T p(T) p(s1) II 1(s ==1
exp E Σ Er(ss,
l:K ..
((
2)
where 8t denotes the global state at time step t, o.KK indicate all agents' local observation, and of represent common skills generated by the encoder cq: ~ Tř.- Matching p(T) means that the common skill encoder To, in our learned planner needs generate common skills clKK and roll out trajectories ((T) that minimize the KL--divergence DEL (p(T) [[p())) Meanwhile, learning from offline data requires the planner to be constrained and conservative. The common skill generated by the planner must lead to the next state close to the offline dataset's distribution. Therefore we conduct the learned planner as a one-step predictor and formulate p(T) by,
T II
i(T) = p(s1) II p( 1:K o ):K)
where cf ~ Ta, (o] is the common skill inferred by the learned planner. 81+1 represent the ground-truth global next state and ((c indicates that we use forward predictor q predict the next global state using all agents' common skills. In this way, we could derive the KL-divergence and formulate objective as below,
T
C(Oh. = DKL -DKL((())(P(T)) E Bio .e44)) 1.K ~orT (
r(se, cl: 1: log g( ((55411177 (
(4)
Exploration
Prediction
where On denote the parameters of the common skill encoder Ti in the planner, DT represents
the multi-task dataset. The full derivation can be found in Appendix A.I Eq. 4 is deployed on all source tasks and aims to extract reusable cooperation knowledge as common skills !.:K" This objective divides the trajectory matching into trade-ofi off between exploration and prediction which fulfills the requirement of integrating cooperative temporal knowledge into common skills. The Exploration term guarantees value-maxine maximizaaiinioon perspective in common skills to guide the action execution. The Prediction term not only achieves conservative planner in offline learning but also brings the global state information into common skills, which different from existing works.
To approximate each term in Eq. we introduce a forward predictor fo for one-step global state prediction and a value net Viot for reward estimation. So receives common skills cf to predict the next global state s/ and next local information 4 The local information local observation's 's embedding. Viot uses the current observation 1:K or the local information l1:K
be seen the
4

---

Published as a conference paper at ICLR 2025
to estimate the accumulated reward. The goal of Eq. 4 is to maximize the estimated reward and minimize the prediction error. To train the value net from the offline dataset, we utilize the Implicit Q-learning objective (Kostrikov et al. 2021) given by,
CIQL (E) DE Lá (1r ( (ott1) Ve (o.ttt) where L5(n) - 1(n 0) (5)
where e E (0, 1) and the target value function Pr is the momentum version of Vi DJ is the multi-task dataset. This objective down weights the contributions of the TD-residual smaller than 0 while giving more weight to larger values. By substituting f& and V into Eq. we could rewrite the empirical objective for learning the high-level planner with common skills as below,
Cplanuer 6) = =B SUIDD. of Noo (o):))
[V2 (4-41 a log fo(((+2lc)
(6)
where 11-1 fo(-letiK) the predicted local information. The weight serves as the trade-off between guiding to space with high-reward and space that the execution policy ought to have a correct imitation. Inspired by Xu et al. (2022a). we introduce an alternative objective that implicitly involves the behavior constraint by using the TD-residual (r+ V/too] as the imitation weight,
r r 7
too
K
(o!
Cllanner (0p. ②) Eo (en of NTo (o)
exp
log fols64alct
a
Following this objective, the common skill acquires global cooperative perspective and is more likely to represent behavior patterns with high rewards from offline multi-task data.
3.3 LEARNING LOW-LEVEL. CONTROLLER WITH TASK-SPECIFIC SKILLSS
After constructing the objective of common skills, we now turn to the low-level controller with task-specific skills =!'ik" where i E T denotes the current task. Here omit the subscript of timestep t to simplify the notation. The main function of our proposed controller to generate real actions following the skills' guidance. Meanwhile, the job of the task-specific skill is to rec- ognize the current task and guide the policy adaptively. To be more specific, we denote the low- and the action decoder To, (a::K 2 of:KK c K. We adopt an objective based on 3-VAE (Higgins et al. 2017) to learn action execution and utilize Self-Supervised learning to learn task-specific representations for reg- ularization. This learning objective given by,
level controller's components task-specific skills encoder as 9 (2. 2.tK of
CController (6), = -E
DIET zor=gu((o))
[log Te, (a.:Klo !o!:K J:K 21: - 3DL [Ip(D))) [p(")))
(8)
where the T denotes the task number, i denotes the current task, O2 and represent the parameters of action decoder To, and task-specific skills encoder In respectively, and , is the regularization coefficient, The left term requires the action decoder тe, to maximize the likelihood of the real action a.:k from offline data, the right term minimizes the KL-divergence between the task-specific skills 2! and the task priors p(D') as a regularization. We show the full derivation in Appendix A.2 Notably, the task-specific skill 2i:K is regularized to approach the current task priors p(D*). ). We show that the problem can be formulated
contrastive learning objective, with its theoretical lower bound provided Theorem 3.1 Training the skill encoder for task discrimination enables the integration of task- knowledge into the skill embeddings z!:K
Theorem 3.1. Denote a of training tasks with their offline data DT_ Dí is the data of the sampled task. Let random variables x be some observations sampled from Di, skill ~ (2(x))
h(x, z) (slm)) p(Di) the prior distribution of current task i, then we have p(=)
h(z,x) x)
Epica s.. log
2 EEEDD [Pkn(0(()))lp(D]]]
EDIEDT h(2, xJ]
where x6 are observations sampled from task Di and (1,2,... .).
5

---

Published as a conference paper at ICLR 2025
We leave the proof of Theorem 3.1 to Appendix A.3 In practice, utilize the exponential cosine similarity ex to approximate h22, z) and randomly sample two agents' observations {or". "o"".00 of in the same task as the positive pairs (g. k+) and regard trajectories from other tasks the negative samples hi_. We introduce Oa the momentum version of Iw to optimize the contrastive loss,
DT
exp(gw(0) 9 Ok+)/o)) log exp(gw(1) go (k+ )/0) D exp(ew(1)
Lg(w) = Σ D.
B (a.ky JaDi ~DT
(10)
(xplo>(()-900kk)11/ (k-))//
where DT is overall source tasks, D' and DT denote the current task and other tasks in DT, respectively. a denotes the temperature and ga updated by the exponential moving average w' yju + (1 7)).. This training pipeline empowers the policy to embed task-specific knowledge and achieve adaptive action execution.
In summary, integrate Eqs. and 10 into Eq. [ to obtain the empirical objective of training the low-level controller. We propose to learn the prior of the current task using Eq. and replace the DKL term in Eq. g with it. The objective is given by,
Conntollerr (6),w)
Σ
[log Te, (a. (o lo! 1:K 2 [=]]] BLg(w).
(11)
DIEDT
3.4 TRAINING AND EVALUATION
HiSSD is fully trained offline and can be trained end-to-end. During training, each task's data D' in the source dataset 2T will be chosen for training. In every training step, we sequentially optimize Eqs. 5 and 11 train the value net, the planner, and the controller, respectively. At the test time, we only use local information to perform decentralized execution. The planner To, infers the common skill c.:KK at each time step. The controller тe, first generates task-specific skill z and then feed {o ct = k= into action decoder to generate the real action. Due to space limitations, we leave the pseudocode in Algorithm| in Appendix B
EXPERIMENTS
4.1 BENCHMARKS AND DATASETS
The StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al. 2019) is popular MARL benchmark and evaluate multi-task learning policy transfer methods. We follow the experimental settings by Zhang et al. (023) and use the offline dataset they collected. Similar the D4RL benchmark (Fu al. 2020), there four dataset qualities labeled as Expert. Medium, Beddium-Expert, and Sedium-Replay We construct task sets Marine-Easy and Marine-Hard In each task set, units in different tasks have the same type and various numbers, all algorithms are trained on offline data from multiple source tasks and evaluated on wide range of unseen tasks without additional data. Details are referred to the Appendix
SMAC
MAMuJoCo Multi-Agent MuJoCo (MAMuJoCo) is a benchmark for continuous multi-agent robotic control, based on the MuJoCo environment. To fulfiil the requirement of offline multi-task learning, we follow Wang et al. (2023a) and collect multi-task dataset in HalfCheetah-12 using HAPPÓ (Kuba et al. 2022) algorithm. We partition the robotic into six agents and construct the individual task by disabling each agent. Each task's 's name corresponds to the joint controlled by disabled agent. Algorithms are trained on multiple source tasks evaluated on unseen tasks without additional data. Details of the dataset are presented in Appendix C
4.2 BASELINES
SMAC To evaluate the capacity of policy transfer using HiSSD, we introduce several comparable baselines from prior works: (i) ODIS (Zhang al. 2023), an effective offline multi-task MARL method for cooperative skill discovery. (ii) UPDeT-m. an offline variant of UPDeT (Hu et al. 202T) by adopting the tranformer-basee Q mixing network. (iii) Transformer-bast (BC-t) method and its variants with return-to-go information (BC-r). We average Hi-SSD's perfor- mance 5 random seeds and report the best score for each task.
behavior cloning

---

Published as a conference paper at ICLR 2025
Table 1: Average test win rates of the best policies over five random seeds in the task set Marine- Hard with different data qualities. For simplicity, the asymmetric task names are abbreviated. For example, the task name "5m6m" denotes the SMAC map "5m_vs_6m". Results of BC-best stands for the best test win rates between BC-t and BC-r.
Tasks
Expert
Medium
BC-best
UPDxT-m
ODIS
HiSSD (Ours)
BC-best
UPD:T-m
ODIS
HiSSD (Ours)
Source Tasks
97.7.1
82.8.116.0
99.5.1
51.21
85.9
62.7.1:
5móm
50.4+ 2.3
17.24428.0
$3.94 5.1
66.1
7.0
21.9 3.4
6.34 4.9
22.74 7.1
26.41 3.8
9m10m
95.3+ 1.6
3.14
80.4
95.5=
13.8+10.9
28.5+10.2
78.1
73.9
Unseen Tasks
4mn
92.11 3.5
33.0.127.1
95.311 3.5
99.2.1
1.2
48.8121111
14.1 5.2
61.71:1777
77.3 10.2
33.6.140.22
99.2:
14.64::4.1
67.2421.3
10m
90.54 3.8
54.7:144.4
93.81 2.2
98.4
0.8
56.241200.6
3291111.
61.31:11.3
98.0.1 0.3
70.8:15522
17.2
58.6.1:1.88
75.5. 19.7
20.0:1:0.5
3.2 3.8
35.9.1 8.1
86.41: 6.0
7m8m
18.84
0.0:
25.0=11..1
35.3:
1.6: 1.6
0.0:
28.11:22.0
1422
8m9m
15.8+ 3.3
0.0 0.0
19.6 6.0
47.0= 6.2
3.14 3.8
2.3 2.6
4.7+ 2.7
15.3+ 2.8
10m11m
8.9
10ml 2m
1.0.1 1.5
0.0.1 0.0
1.61 1.6
14.5
9.1
0.0.1
0.0
0.0.1
0.0
1.6. 1.6
0.6.1: 0.5
0.0+
1.34
0.64 1.3
1.4:
Medium- Expert
Medium- Replay
Source Tasks
67.74-23.7
85.2:1179
73.6::22.
86.6::
8.8
41.41:00.1
83.61:4.0
78.8::
Sın6m
31.311 6.3
1.6.1 1.6
9.4.: 2.2
41.9 9.7
25.0.L 3.1
0.8.L. 1.4
16.61: 4.7
25.3 10.3
18.7
31.3::4.55
83.6
33.4
0.8
34.4
Unseen Tasks
4m
81.3
33.9::39.00
828::13.5
91.11 6.1
61.5+ 9.0
35.9:::2.6
55.6::143
77.34 1.9
98.31
75.0:124.2
61.71:20.3
10m
78.14 6.7
32.8118..1
82.8-16.8
96.4 2.1
82.4 8.2
11.01 7.8
84.4 15.1
94.7- 2.6
64.8424.3
9.4+
81.3-20.66
88.4
4.5
2.3
84.4
90.3+
7m88
13.3.1 4.5
2.3.
15.61 4.4
30.511 10.4
7.31
6.4
1.6.1
9.41
21.7.1 4.7
8m9m
10.24 4.6
9.54 8.6
10.9 4.7
38.2 18.3
11.5t
3.9
0.8 1.4
11.75
8.7
14.5 4.0
10mlIm
26.6
54.7=
6.6
42.5=
10m122
0.0.L 0.0
0.0:1 0.0
162 1.6
2,5 1.0
1.6 2.7
0.0:
0.0
2.34
1.4
0.5: 0.3
13ml5m
1.6.1
1.6
MAMuJJCCo For the continuous robotic control task, we compare HiSSD with four recent offline MARL algorithms: (i) Behavior cloning (BC) method, the multi-agent version of (ii) IQL Kostrikov et al. (021). (iii) TD3-BC (Fujimoto & Gu 2021). and (iv) ODIS (Zhang et al., 2023) reproduced by ourselves. All algorithms are evaluated over 32 independent runs, with 4 different random seeds employed during training to ensure reproducibility. Notably, all algorithms use the same architec- ture. and the details for the implementations and hyperparameters of algorithms in MAMuJoCo are shown in Appendix
4.3
MAIN RESULTS
We evaluate Hi-SSD and baselines on SMAC and present the average test win rates in the Marine-Hard task set Table BC-best represents the highest win rates between BC-t and BC-r. Below are the key results: (i) HiSSD achieves top performance
SMAC
over half of the tasks.
This indicates that using skills to guide action execution benefits policy transfer. (ii) Compared to ODIS, which only discovers common skills from offline multi-task data, our method outperforms medium and near-optimal data qualities, showing advancement in learning task-specific skills. Due to space limitation, we leave results other task sets in appendix
MAMuJoCo Table 2 shows the mean and standard deviation of average returns for the offline multi-task learning on our HalfCheetah-v2 task set. The results show that HiSSD outperforms all baselines and achieves sate-of-the-an tte-art performance in most tasks. Compared to baselines with only behavior cloning (BC), HiSSD outperforms in a wide range. We also find that HiSSD outperforms ODIS and TD3-BC when transferring unseen tasks. It indicates that learning cooperative temporal knowledge and task-specific skills is beneficial for learning generalizable multi-agent policy from offline multi-task data.

---

Published as a conference paper at ICLR 2025
Table 2: Average scores HallCheetah-v2 multi-task datasets in MAMuJoCo.
Tasks
BC
IQL
ITD3-BC
ODIS
HISSD(ours)
Source Tasks
complete
3188. 56626666668
4384.23-198.55
4365, 10:: 72.92
3677.66=t174.82
4450.57=
back thigh
3324.22-#: 58.49
3675.91 18.99
3685 82: 40.84
23811624:966
3698.38
13.98
foot
3989. 21:11422
2713,40:1195.63
3197.83:1
Front thigh
1861.53
53:11415..0
2744.63 329.00
2700.
17.1-07.44
2684,99 29.44.99
1948.741
81.24
front
1819.94-273.96
3048.43-:36.79
4158.15::1180
3944.78::21:72
3468 32.4 290.72
Unseen Tasks
back shin
1964.555 266.1:26666624
1974.62::31.33
1690.40::251.77
3217.59:1844.1
3472.12: 91.95
1683.161141942
39.0.82.L342.16
4175.29
Table 3: Ablation studies HiSSD. We report average test win rates of the best policies over five random seeds in the task set Marine-Hard with different data qualities.
Data Qualities
w/o Planning
wko Predicting
Half Negative
L2 Loss
BC best
IRSSD
Source Täsks
Expert
80.7::22.4
85.9::16.4
87.0+-15.3
8088+212
81.1-21.8
84.5+17.1
Medium
52.7::25.5
53.14-20.0
24.3++20.6
42.3=2188
50.3420.11
$5.9::19.2
Medium Expert
56.0:1.2.4
63.5.1.6..1
70.7
57.8127.1
41.7118.5
64.211.3.5
Medium Repily
51.1:128.2
47.71225.1
50.0:122.9
44.7.128.1
46.51226.7
51.9:124.00
Target
Expert
45.8::39.0
55.0::37.4
60.9::37.11
53.24338.9
46.8:133.8
61.2:::7.44
Medium
38.41:37.5
37.643342
46.74:38.0
42.4238.
25 64:26.8
48.6:1:0.11
Medium-Expee Expert
47.2:t36.9
51.24:34.3
55.8::37.6
51.0==8.9
38.8133.00
57.6
Medium Replay
47.91:40.3
41.24:3.77
48.71139.0
4.4 ABLATION STUDY
In this section, we provide additional empirical analysis of HiSSD. First, we demonstrate the ef- fectiveness of our skill learning approach that jointly encodes cooperative temporal patterns and task-specific knowledge. Next, we identify key factors in task-specific skill acquisition. Finally, we present skill visualizations for deeper insights.
Common Skills Analysis We conduct experiments SMAC's Marine-Hard task set demon- strate the effects of our proposed skills learning paradigm HiSSD and present results in Table We implement variants of HiSSD to show the impact of three factors HiSSD's skills learn- ing, respectively: (i) w/o Planning. HiSSD only learns task-specific skills among tasks. (ii) w/o Prediction. HiSSD trains the planner without the next global state prediction. The results indicate that learning common skills improves policy transfer, and learning to predict the next global state acquires further advancement.
Task-Specific Skills Analysis To investigate what is the essential factor of learning task-specific skills, we conduct experiments on SMAC's Marine-Hard task set with three variants of HiSSD: (i) Half-Negative. We reduce to half of the negative samples during contrastive learning. (ii) LosLoss. The objective replaced with one similar to Grill et al. (2020) which does not require negative samples. According to the results in 3 learning to distinguish the difference between tasks plays a key role in learning task-specific skills. Meanwhile, increasing the number of negative samples further improves the capacity of multi-agent policy transfer.
4.5 VISUALIZATION OF LEARNED SKILLS
To investigate the effectiveness of our proposed method more clearly. we evaluate HiSSD on multi- ple tasks SMAC and visualize the learned skills using t-SNE (Hinton Roweis] 2002).
Common Skills In Figure 2, we visualize the chosen common skills among multiple tasks. Neigh- boring points in the same distribution represent similar common skills. To show the skill flow, we partition the collected trajectories into four sub-parts by timestep. The plots show that common
8

---

Published as a conference paper at ICLR 2025
0-5 stops
5-10 steps
10-15 steps
15-20 steps
Figure 2: Visualization of learned common skills. We use HiSSD to collect trajectories on five tasks in SMAC and partition these trajectories into four time windows. Plots each figure represent the distribution of chosen common skills. We use multiple time windows to indicate the task flow.
0-5 steps
5-10 steps
10-15 steps
15-20 steps
Figure 3: Visualization of learned task-specific skills. We use HiSSD to collect trajectories five tasks in SMAC and partition these trajectories into four time windows. Plots represent the distribution of chosen task-specific skills. We use multiple time windows indicate the task flow.
skills are mapped into multiple clusters each containing skills from different tasks. Points with the same color in each distribution represent the collaboration between agents the corresponding task. The results indicate that HiSSD acquires the capability to learn task inse-inreeevant common skills.
Task-Specific Skills Figure 3 represented the chosen task-specific skills during evaluation. We collect trajectories on multiple tasks in SMAC using HiSSD and partition these trajectories into four sub-parts by timestep. From the plots, task-specific skills chosen in small-scale tasks (i.e., 3m, 4m, and Sm) are mapped into different distributions. However, the chosen skills in large-scale tasks (i.e.. 10m and 12m) overlap with each other. The results indicate that HiSSD efficiently generalizes to small-scale tasks similar to source domains, learning task discrepancies effectively. However, for large-scale tasks (10m, 12m), it struggles to capture significant distribution shifts, leading to overlapping. Notably, our method adaptively reduces distribution distance over time, demonstrating dynamic transition learning across tasks. As episodes progress, task representations converge due to agent attrition, where multi-agent tasks naturally decompose into simpler sub-tasks during episode execution. This demonstrates HiSSD's ability to: (i) effectively encode task task-specific knowledge, and (ii) adaptively distinguish between tasks through learned policies.
5 RELATED WORK
5.1 OFFLINE MARI.
Training policies from offline experience without interaction effectively reduce the trial and error costs when implementing RL in real-world scenarios (Levine et al. 2020 Zhang et al.) (02T). Due to the distribution shift in offline learning (Fujimoto et al.] (019), training a policy from static datasets faces unexpected extrapolation errors when estimating unseen data eWu al., 2019 Kumar ct al.). Therefore, previous works consider learning behavior-conssrainee policies (Kumar et al.
2020
Kostrikov et al. 1021), which can be extended to the MARL paradigm. They aim at adopting
suffiiient conservatism to current online MARL methods (Yang et al. 202 [Jing & Lu 2021 Pan
et al.
2022). training policies by value function decomposition (Sunehag al. [2018] Rashid al.
2020
Yang et al.] 20206b] Wang et al. (021). or Maali--geetttie policy gradient algorithms (Lowe et al,
2017
Foerster et al. 2018 Iqbal & Sha 2019 Yu et al. 2022 Kuba et al. 2022). Another effective

---

Published as a conference paper at ICLR 2025
way for offline learning is leveraging the powerful transformer- model (Chen et al. 2021 Lee
et al. 2022 Meng et al. 2023 Liu et al. 2023) or diffusion model (Janner et al. 2022 Pearce et al.
2023
He et al.] (223)). ). Yet the problem of combining generative models with the policy improve-
ment paradigm of reinforcement learning remains an issue (Zheng et al. 2022 Wang et al. 2023b
Kang al. 2023)
5.2 MULTI-TASK MARL
Multi-tass learning plays a key role in improving data-efficiency and generalization in MARL. highlights knowledge reuse (Da Silva & Costa 2016 [Shen et al.] 2021; [Sodhani al. 2021
which is beneficial for transferable multi-agent collaboration. This paradigm requires the policy to hold a flexible structure for deploying agents across tasks with varying input dimensions (Agarwal et al.) 22000; Wang et al.] [2000; Hu et al. 2021] Zhou et al.) [021). Recent works consider mul- tiple ways to realize multi-task adaptations such as policy representations learning (Grover et al.) 2018), evolutionary-bare baaed curriculum learning (Long et al.) (020), randomized entity-wise factor- ization (Iqbal et al. (02T), high-level cooperation strategy reusing (Liu et al. [202T), 1) and training transformer-basee
inqulation-invariann policies (Hu et al. 2021 Wen et al., 2022 Liu et al. 2023
Although these methods relieve the need for learning from scratch during transferrin, generalizing policies without simultaneous learning fine-tuning remains challenging Zhang al. (2023).
5.3 MARL WITH SKILL LEARNING
Hicrarchical MARL with skill learning is a practical way to solve complex decision-making tasks.
This paradigm embeds behavior patterns in skill space, promoting the exploration of cooperative
multi-agents with state empowerment from information theory (Barto & Mahadevan, 2003
Eysen-
bach et al.] 2019 Yang et al., 2023 He al.) 2020 Liu et al.. (022). MASD (He al.
20200
introduces an information bottleneck for cooperation patterns discovery. HSD (Yang et al.
2020a)
and HSL (Liu al.
2022) utilize hierarchical architectures to discover diverse behaviors.
HMASD
Yang et al. (023) treats skill discrimination as a sequential modeling problem. However, their
framework requires global information during execution. VO-MASD (Chen et al. 2024) discov-
ers hherarchy-hiieee like cooperation skills in pre-training stage to speed up the online learning. ODIS
(Zhang et al. 2023) combines offline multi-task learning with hierarchical MARL learn general- izable multi-agen policy. Although they discover skills following the value function decomposition, they only consider the common skill among tasks. HyGen (Zhang et al. 2024) follows ODIS and in- tegrates online exploration to further improve transfer capability, especially in middle data qualities. In this article, we propose a hierarchical policy that jointly learns common and task-specific skills data, further enhancing the capacity of multi-agent policy's 's generalization.
from offline
6 CONCLUSION
In this paper, propose new hierarchical multi-agent policy that jointly learns common and task-specific skills from offline multi-task data, further improving the capacity of policy transfer in offline multi-task MARL. We analyze the primary issues in current offline multi-task MARL methods and propose novel objectives to overcome these issues. We compare HiSSD to SOTA methods on popular MARL benchmarks and certify that it acquires promising improvement. One limitation of Hi-SSD is its training stability, and we consider it as our future work. Hopefully, our proposed skill learning pipeline can lead to new branch for offline multi-task learning MARL.
REPRODUCIBILITY STATEMENT
Source code is available at hhttpss://ittii com/ of our proposed objective and theorem is presented in Appendix A| The pseudocode is presented in Appendix B We leave the detailed description of the used benchmarks and datasets in Appendix C and present the implementation details in Appendix
wwt.cccmmmoo///..ee coom/mooriinn wtt.///oro.cnnn/is55500 The full derivation
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGMENT
This work was supported by National Natural Science Foundation of China (624061 12, 62372179).
REFERENCES
Akshat Agarwal, Sumit Kumar, Katia Sycara, and Michael Lewis. Learning transferable cooper- ative behavior multi-agent teams. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1741-1743, 2020.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13:341-379, 2003.
Avinandan Bose, Simon Shaolei Du, and Maryam Fazel. Offline multi-task transfer rl with repre- sentational penalization. arXiv preprint arXiv:2402. 12570, 2024.
Jiayu Chen, Bhargav Ganguly, Tian Lan, and Vaneet Aggarwal. Variational offline multi-agent skill discovery. arXiv preprint arXiv:2405. 16386, 2024.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin I Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement Icarminn via sequence modeling. Advances in neural information processing systems, 34:15584-55097, 2021.
Felipe Leno Da Silva and Anna Helena Reali Costa. Transfer learning for multiagent reinforcement learning systems. In Proceedings of the twenty-fifth international joint conference artificial intelligence, pp. 3982-3983. 2016.
Benjamin Eysenbach, Abhishek Gupta, Julian lbarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference on Learning Representa- tions, 2019.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21284 2020.
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: survey, Artificial Intelligence Review, 55(2):895 943, 2022.
Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations multiagent systems. International conference on machine learning, pp. 1802-1811. PMLR, 2018
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xue- long Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Shuncheng He, Jianzhun Shao, and Xiangyang Ji. Skill discovery of coordination in multi-agent reinforcement learning. arXiv preprint XXiv:2006.0402// 2020.

---

Published as a conference paper at ICLR 2025
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, 2017.
Geoffrey E Hinton and Sam Roweis. Stochastic neighbor embedding. In S. Becker, S. Thrun, and K. Obermayer (eds.), Advances Neural Information Processing Systems, volume 15. MIT Press, 2002.
Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. UPDeT: Universal multi-agent RL via policy decoupling with transformers. In International Conference Learning Representations, 2021.
Shariq Iqbal and Fei Sha. Actor-ateention-critic for multi-agent reinforcement learning. In Interna- tional conference on machine learning, pp. 2961-2970. PMLR, 2019.
Shariq Iqhal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning, In Inter- national Conference on Machine Learning, pp. 4596-4606. PMLR, 2021.
Haque Ishfaq, Thanh Sguyen-Tang, Songtao Feng, Raman Arora, Mengdi Wang, Ming Yin, and learnin
Doina Precup. Offline multitask representation learning for reinforcement learning. arXiv preprint arXiv:2403. 15574, 2024.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pp. 9902-9991991 PMLR, 2022.
Jiechuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2 100.00830.. 2021.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng YAN. Efficient diffusion policies for offline reinforcement learning. In Thirtyy-eeenthh Conference on Neural Information Processing Systems, 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In Deep RL Workshop NeurlPS 2021, 2021.
Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. In International Conference on Learning Representations, 2022.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing poolcy q-learning via bootstrapping error reduction. URL httl::ttt:::riiieeeee http///www..orgaaa////.....0 00949
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 13:1179-1191, 2020.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang. Lisa Lee, Daniel Freeman, Sergio Guadar- rama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi game decision trans- formers. Advances in Neural Information Processing Systems, 35:27921-22 177921, 2022.
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review, 2018. Stttta///aaxll..o2////////0... 00909
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164. 2020.
Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandkumar. Coach-player multi-agent reinforcement learning for dynamic team composition. International Conference Machine Learning, pp. 6860-6870. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Jie Liu, Yinmin Zhang, Chuming Li, Chao Yang, Yaodong Yang. Yu Liu, and Wanli Ouyang. Masked pretraining for multi-agent decision making. arXiv preprint arXiv:2310. 11846, 2023.
Yuntao Liu, Yuan Li, Xinhai Xu, Yong Dou, and Donghong Liu. Heterogeneous skill learning for multi-agent tasks. Advances in Neural Information Processing Systems, 35:37011 2022.
Qian Long, Zihan Zhou, Abhinav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Evolutionary popu- lation curriculum for scaling multi-agent reinforcement learning. In International Conference on Learning Representations, 2020.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed coperative-competttie environments. Advances in neural informa- fion processing systems, 30, 2017.
Linghui Meng, Muning Wen, Chenyang Le, Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Yaodong Yang. et al. Offline pre-trained multi-agent decision trans- former. Machine Intelligence Research, 20(2):233-248, 2023.
Frans A Oliehoek, Christopher Amato, al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In International conference ma- chine learning, pp. 17221-17237. PMLR, 2022.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Ser- gio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Im- itating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, 2023.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gra- dients. Advances in Neural Information Processing Systems, 34:12208-12221, 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agen reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 21 8188,,88, 2019.
Jiayi Shen, Xiantong Zhen, Marcel Worring, and Ling Shao. Variational multi-task learning with gumbel-softtax priors. Advances in Neural Information Processing Systems, 34:21031-21042, 2021.
Qichao Shentu, Beibu Li, Kai Zhao, Yang Shu, Zhongwen Rao, Lujia Pan, Bin Yang, and Chen- juan Guo. Towards general time series anomaly detector with adaptive bottlenecks and dual adversarial decoders. In The Thirteenth International Conference Learning Representations, 2025.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. In International Conference on Machine Learning, pp. 9767-9779. 9767-9779.PMLR, 2021.
Haolin Song. Mingxiao Feng, Wengang Zhou, and Houqiang Li. Ma2cl: masked attentive con- trastive learning for multi-agent reinforcement learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 4226 4226-4234, 2023.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. deooposscooooiition networks for cooperative agett learning based on team reward. In Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, pp. 2085-2087 2018
13

---

Published as a conference paper at ICLR 2025
Jindong Tian, Yuxuan Liang, Ronghui Xu, Peng Chen, Chenjuan Guo, Aoying Zhou, Lujia Pan, Zhongwen Rao, and Bin Yang. Air quality prediction with physics-informer dual neural ODEs in open systems. In The Thirteenth International Conference on Learning Representations, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Micha"el Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-333 2019.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations, 2021.
Weixun Wang. Tiampei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. Action semantics network: Considering the effects of actions in multiagent systems. In International Conference on Learning Representations, 2020.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforcement learning with implicit global-to-local value regularization. Advances in Neural Information Pro- cessing Systems, 2023a.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023b.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Infor- mation Processing Systems, 15:116509-65222 2022.
Xingjian Wu, Xiangfei Qiu, Zhengyu Li, Yihang Wang, Jilin Hu, Chenjuan Guo, Hui Xiong, and Bin Yang. CATCH: Channel-aware multivariate time series anomaly detection via frequency patching. In The Thirteenth International Conference on Learning Representations, 2025.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. wttp://///gabbbbbb.b./... 11361
CoRR, abs/1911. 1361, 2019.
Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:4085- 4098, 2022a.
Zhiwei Xu, Bin Zhang, Yuan Zhan, Yunpeng Baiia, Guoliang Fan, et al. Mingling foresight with imagination: Model-based cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 35:1 13377113111300 2022b.
Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent rein- forcement learning with skill discovery. In Proceedings of the 19th International Conference Autonomous Agents and MultiAgent Systems, AAMAS '20, pp. 1566-1574 2020a. ISBN 9781450375184.
Mingyu Yang. Yaodong Yang, Zhenbo Lu, Wengang Zhou, and Houqiang Li. Hierarchical multi- agent skill discovery. In Advances in Neural Information Processing Systems, volume 36, pp.
61759-61777, 2023.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint urXiv:2002.039333 2020b.
Yaodong Yang. Guangyong Chen, Weixun Wang, Xiaotian Hao, Jianye Hao, and Pheng- Heng. Transformer based working memory for multiagent reinforcement learning with action parsing. Advances in Neural Information Processing Systems, 35:34874-34886, 2022.
14

---

Published as a conference paper at ICLR 2025
Yigin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems. 34:10299-10312, 2021.
Yucheng Yang, Tianyi Zhou, Qiang He, Lei Han, Mykola Pechenizkiy, and Meng Fang. Task adap- tation from skills: Information geometry, disentanglement, and new objectives for unsupervised reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Wu. The surprising effectiveness of ppo cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611-24622 2022.
Won Joon Yun, Soohyun Park, Joongheon Kim, MyungJae Shin, Soyi Jung, David A Mohaisen, and Jae-Hyun Kim. Cooperative multiagent deep reinforcement learning for reliable surveillance via autonomous multi-uav control. IEEE Transactions on Industrial Informatics, 18(10):7086-7099 2022.
Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang. Discov- ering generalizable multi-agent coordination skills from multi-task offline data. In The Eleventh International Conference on Learning Representations, 2023.
Kaiqing Zhang, Zhuoran Yang. Han Liu, Tong Zhang, and Tamer Başar. Finite-sample analysis for decentralized batch multiagent reinforcement learning with networked agents. IEEE Transactions on Automatic Control, 66(12):5925-59) 2021.
Mingliang Zhang, Sichang Su, Chengyang He, and Guillaume Sartoretti. Hybrid training for enhanced multi-task generalization in multi-agent reinforcement learning. arXiv preprint arXiv:2408. 13567, 2024.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning, pp. 17042-27052 PMLR, 2022.
Tianze Zhou, Fubiao Zhang, Kun Shao, Kai Li, Wenhan Huang, Jun Luo, Weixun Wang, Yaodong Yang. Hangyu Mao, Bin Wang, al. Cooperative multi-agent transfer learning with level- adaptive credit assignment. arXiv preprint arXiv:2106.0057, 2021.
15

---

Published as a conference paper at ICLR 2025
A ADDITIONAL DERIVATION
A.1 OBJECTIVE FOR HIGH-LEVEL PLANNER
In this section, formulate objective using probabilistic inference, following the theoretical analysis in Levine| 2018). We assume that the training goal for behavior cloning to minimize the distance between the optimal trajectory p(r) formulated in Eq. and the rollout trajectory f(r)- p(T) is required to predict the next global state s'+1 that is not far from the offline dataset. Therefore, could formulate ((T) as below,
T II
(()) ((s)) II 1( k a:K)
( (lc:
(12)
where g(.) is a predefined predictor, of is the learned common skill. A practical way to match this objective by adopting an optimization- approximate inference approach. In this article, we minimize the KL-divergence between the approximated trajectory and the optimal trajectory to achieve this objective,
DEL (p(T)ll((T) -E,~A(r) [lgg(r) - log ((T))
(13)
In this way, we derive Eq. 13 and obtain the planner's objective,
-Dk -Kk(((())))
EDdD ((r)(()) log ((7)
T log logp(s1) Σ (logp(s4+100 1:K d.k) r(s,, ot (::k))-
BNND
T
logp(s1)
Σ
(log (logp(s(+100) 2: ) + log ,..kk((kk))) c.:Ki))
(14)
ETND
Σ
r(s 2:1) - (5 lo ..K.)
1=1
Σ ENAD [r(s,, log bogg($ 10 c2:Kk)]
A.2 OBJECTIVE FOR LOW-LEVEL CONTROLLER
To formulate the objective for our fine-grained action controller, we perform the controller as generative model and introduce the variational inference. With the help of Jensen's inequality, formulate the ELBO as below,
log logp(a) log / p(a, 2)
((210) log Ex(oo)
ppaa 2)
((210)
{(((0)
(15)
[2(() [logp(a, =)] (1 [log (=[0))
Eg(=|o) [p(a|z) p(=)] E n) [log {(210)] ELBO,
where q(-) is predefined encoder, denotes the local observation of each agent, and denotes the task embedding. Practically, we introduce a task-guided controller πT, as p(alz) and a task discriminator Iu q(=]o). The prior distribution p(=) is unreachable each task during training and we use p(D*) to represent the prior distribution of skill z in each task. We also embed the common skill 6 and local observation o, into ne.- Thus, rewrite Eq. 15 and formulate the objective of our low-level controller in decentralized manner,
CController (fc, w) LELBO
(efieo)) [log Ta (aflzt, of, ct) + E (xiloo) loog p(D')
-EDIDD
gu (zilo))
(o [log ne (ailet of cf] - DKL (ka(e(ll)())))
-EEiiDD
(16)
where i denotes the current task and D indecates the offline ataset.
16

---

Published as a conference paper at ICLR 2025
A.3 BRIDGING KL-DIVERGENCE AND CONTRASTIVE LEARNING
In this section, we illustrate how to bridge the contrastive loss to approximate the KL-divergence in Eq. We first introduce lemma. Then we give proof Theorem 3.1
Lemma A.1. Given D D as the the current task's data distribution. Denote a as the local observations sampled from the offline data ~ D', Iw is the task-specific skill encoder, skill & (z)())) Then we have,
p(z|D') plz)
/ pp Di) p(z)
dr p(()ga(zl))) dr Ex p(z)
g (zlar) p(=)
(17)
Theorem 3.1. Denote set of N training tasks with their offline data DJ, D' is the data of the sampled task. Let random variables x be some observations sampled from D', skill ~ (=(=))) h(x,
[=(s/l) p(Di) is the prior distribution of current task i, then we have
((2)
1(2,z)
Epi se log
- DE [PKL(O(())ll((i]]
EDiEDT 1(2, x1)
where x² are observations sampled from task Dj and E ...
...
Proof. We introduce the mutual information I(2, D*) between the learned skill and the current
task's 's data distribution D and rewrite the inequality as below,
M2,3)
log
D') 2 -ELNDi [D((((()))lb((P)) (18)
EDTEDT h(2,x*)
For the left side:
log p(= D()
I(z;D') Epi,
(==)
In ( (2|2z)
EDI,
log Er
p(2)
log I (=(x)
Ep,,.
p(z)
p(2)
N ] log N
-ED s2
log (2(x))))
(2(2)) (N (N- ((--1)]] + log N
p(2)
2 -Epiz
log
p(2)
I [u(2|2²))
2 -Epi,e L
log
(2(x)) (N D.EDTI. p(z)
+ log N
(19)
p(2)
(=(2(=*))
EEDi jaa
log
log N
g0 D'EDTV p(2)
(=|x)
ED',,. log
p(z)
log N
(=(=)
Ed EDT
2=(s/2*) p(x)
p(z)
(2(2)
EDi,e
log
p(2)
EDTEDT
p(2)
h(2,0:)
EEi
log
ED-EDT 11(22x))
where N is the number of the task set DT_ Here, (a) is derived from Lemma A.1
17

---

Published as a conference paper at ICLR 2025
For the right side:
/ Iu (2, log [u(2[x) (z)
D')
dedr
ow / gw lo (212 - f gw (2) log Iw (z)dz
(6) / 9w (ogg - / (og
(20)
/ I g log
gw (2|
(Di) dzdæ
EENDD |f (wo(Ela)) log
In (=
da
(DDi)
p
= BENDi (o (1xr))))))))
The inequality at (b) is derived from (o (Ppp(())) 0.
B PSEUDOCODE FOR HISSD
Algorithm HiSSD for Offline Multi-Tast Task MARL
1: Inputs:
2: High-Levve Planner To,, Low-Level Controller Tot Value Net o and Vlot Forward Predictor fo, Task-
Specife Skill Encoder 9w. Training Steps T, Task Numbers N. Offline Multi- Task Dataset D Current Training Task its data D' D Agent Numbers {K*)T, Ji=1, Batch Size learning rate target update rate т.
3: Training:
4: for each timestep in 1..N do
5: D' D"
Sample one task data from the offline dataset. D
(s,, ac, 00,011
HK 1-
7:
di To, (o,
Infer common skills.
8:
Compute (ot (ott)) V. (ot()) and Vo (fo(et**) with
9:
8/11 fo(c
10: zi - (ot
1ik # Infer task-specifat fic skills. clik)
dlik + To, (2)
12:
Optimizing Eq,
with Via (ol**,, (ol((o and
13: Optimizing Eq.
with Vita (ol*), Vi (fa(c!*)) (), rs, and 6/41
14: Optimizing Eq.
with clsk 2atk of and samples from other tasks
15: end for
16: Execution:
17: for each timestep it in current environment do
18: d. To, (o)
Infer common skills. Infer task-specific skills.
19: zt (=
20: a, + ne, (2) Or (k)
21: end for
C BENCHMARKS AND DATASETS
C.I SMAC
The StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al. 2019) is a widely used cooperative multi-agent testbed that contains diverse StarCraft micromanagement scenarios. In this paper, we utilize three distinct SMAC task sets defined by Zhang al. (2023): Marine-Ilard, Marine-Easy Easy, and Stalker-Zealor to evaluate the capacity of transferring policy to unseen tasks. The Marine- Hard and Marine Easy task sets include various marine battle scenarios, and the trained multi-agent policy needs to control groups of allied marines to confront equivalent or superior numbers of built- Al enemy marines. The stalker-zealot task set includes several tasks with symmetric stalkers and
18

---

Published as a conference paper at ICLR 2025
Table 4: Descriptions of tasks in the Marine-Easy task set.
Task type
Task
Ally units
Enemy units
Properties
3m
Marines
Marines
homogeneous & symmetric
Source tasks
5m
5 Marines
Marines
homogeneous & symmetric
10m
Marines
10 Marines
homogeneous & symmetric
4m
Marines
Marines
homogeneous & symmetric
6m
6 Marines
6 Marines
homogeneous & symmetric
7m
7 Marines
Marines
homogeneous & symmetric
8m
Marines
Marines
homogeneous & symmetric
Unseen tasks
9m
9 Marines
Marines
homogeneous & symmetric
11m
11 Marines
11 Marines
homogeneous & symmetric
12m
12 Marines
12 Marines
homogeneous & symmetric
Table 5: Descriptions of tasks in the Marine-Hard task set.
Task type
Task
Ally units
Enemy units
Properties
3m
Marines
3 Marines
homogeneous & symmetric
Source tasks
5m vs ___6_m__m 6m
5 Marines
6 Marines
homogeneous & asymmetric
@m_vs_100n 10m
Marines
10 Marines
homogeneous asymmetric
4m
Marines
4 Marines
homogeneous & symmetric
5m
5 Marines
5 Marines
homogeneous & symmetric
10m
10 Marines
10 Marines
homogeneous & symmetric
12m
12 Marines
12 Marines
homogeneous & symmetric
Unseen tasks
7m_vs_8m
Marines
8 Marines
homogeneous & asymmetric homogeneous asymmetric
8m_ km_vs_90 9m
8 Marines
9 Marines
10m vs 11m
10 Marines
11 Marines
homogeneous & asymmetric
10m vs 12m
10 Marines
12 Marines
homogeneous asymmetric
13m__s vs 15mm
13 Marines
15 Marines
homogeneous & asymmetric
zealots on each side. To achieve generalization to unseen tasks with limited sources, we train on three selected tasks and reserve the remaining tasks for evaluation. Detailed attributes of these task sets are enumerated in Tables and
As stated in the experiments section, utilize the same offline multi-task dataset Zhang et al. 22023 to maintain a fair comparison. Definitions of these four qualities are listed below:
The expert dataset contains trajectory data collected by QMIX policy trained with 2, 000, 000 steps of environment interactions. The test win rate of the trained QMIX policy (as the expert policy) is recorded for constructing medium datasets.
The medium dataset contains trajectory data collected by QMIX policy (as the medium policy) whose test win rate half of the expert QMIX policy.
The medium-expert dataset mixes data from the expert and the medium dataset to acquire more diverse dataset.
The medium-replay dataset is the replay buffer of the medium policy, containing trajectory data with lower qualities.
The Properties offline datasets with different qualities are detailed in Table
C.2 MAMUJoCo
Multi-Agen MuJoCo (Pegg et al. 202T) benchmark developed for assessing and comparing the effectiveness of algorithms in continuous multi-agent robotic control, In MAMuJoCo, robotic
19

---

Published as a conference paper at ICLR 2025
Table 6: Descriptions of tasks in the Stalker-Zealot task
Task type
Task
Ally units
Enemy units
Properties
Stalkers,
Stalkers,
2s3z
heterogeneous & symmetric
Zealots
3 Zealots
Stalkers,
Stalkers,
Source tasks
2s4z
heterogeneous & symmetric
4 Zealots
4 Zealots
Stalkers,
Stalkers,
3s5z
heterogeneous & symmetric
Zealots
5 Zealots
Stalkers, Zealots
Stalkers,
1s3z
heterogeneous & symmetric
3 Zealots Stalkers, 4 Zealots
Stalkers,
1s4z
heterogeneous & symmetric
4
Zealots
Stalkers,
Stalkers, 5 Zealots Stalkers, 5 Zealots
1s5z
heterogeneous & symmetric
Zealots
Stalkers,
2s5z
heterogeneous & symmetric
Zealots
Stalkers,
Stalkers, 3 Zealots Stalkers, 4 Zealots
Unseen tasks
3s3z
heterogeneous & symmetric
Zealots
Stalkers,
384z
heterogeneous & symmetric
4 Zealots
Stalkers,
Stalkers, 3 Zealots Stalkers, Zealots
4s3z
heterogeneous & symmetric
Zealots
4 Stalkers,
4s4z
heterogeneous & symmetric
Zealots
4 Stalkers, Zealots
4 Stalkers, Zealots
4s5z
heterogeneous & symmetric
system is partitioned into independent agents, each tasked with controlling specific set of joints accomplish shared objectives. To conduct offline multi-task learning. we choose HalfCheetah- v2 as our base scenarios and partition the robotic system into six agents. Besides the complete HalfCheetah robot, we design six new tasks by disabling each agent. Therefore, our MAMuJoCo multi-task data comprises seven tasks. Each task's name corresponds the joint controlled by the disabled agent. We follow Wang et al.] (2023a) and generate offline data using the policy trained by HAPPO (Kuba et al. (022). The hyperparameter env args agent obsk (determines up to which connection distance agents will be able to form observations) set to We list the average return of our datasets in Table
D IMPLEMENTATION DETAILS
D. DETAILS OF HISSD SMAC
We follow prior works and decompose the observation to process the varying observation sizes among tasks. The local observation o, decomposed into several portions, the one including agent i's own information ".nh and the other contains other entities' information {o }. We employ a transformer model for parallel tensor processing embed these portions. Each portion is embedded by a separate fully connected layer with an output dimension of 64, and the transformer processes these embedding according to the attention mechanism.
Q = (1o own 0,,1
entity
.....). .....),K K = W (lorwn
01,1
entity
D), V W. W
0,11
entity
,....),
QKT
[efmn : ei ..... softmax
entity
V, Vdk
(21)
where (o 0,11
entity
Decompose(o:). dx = dim(K).
20

---

Published as a conference paper at ICLR 2025
Table Properties of offline datasets in SMAC with different qualities.
Tasks
Quality
Trajectories
Average Return
Average Win Rate
expert
2000
19.8929
0.9910
medium
2000
13.9869
0.5402
3m
medium-expert
4000
16.9399
0.7656
medium-replay
3603
N/A
N/A
expert
2000
19.9380
0.9937
medium
2000
17.3288
0.7411
5m
medium-expert
4000
18.6334
0.8674
medium-replay
711
N/A
N/A
expert
2000
19.9438
0.9922
medium
2000
16.6297
0.5413
10m
medium-expert
4000
18.2595
0.7626
medium-replay
571
N/A
N/A
expert
2000
17.3424
0.7185
medium
2000
12.6408
0,2751
5m_vs_ _n_
medium-expert
4000
14.9916
0.4968
medium-replay
32607
N/A
N/A
expert
2000
19.6140
0,9431
medium
2000
15.5049
0.4146
9m ss_vs__0000 10m
medium-expert
4000
17.5594
0.6789
medium-replay
13731
N/A
N/A
expert
2000
19.7655
0.9602
medium
2000
16.6279
0.4465
2s3z
medium-expert
4000
18.1967
0.7034
medium-replay
4505
N/A
N/A
expert
2000
19.7402
0.9509
medium
2000
16.8735
0.4965
2s4z
medium-expert
4000
18.3069
0.7237
medium-replay
6172
N/A
N/A
expert 2000
19.7850
0.9518
medium
2000
16.3126
0.3114
385z
medium-expert
4000
18.0488
0.6316
medium-replay
11528
N/A
N/A
Table 8: Properties of offline datasets on HalfCheetah-v2 MAMuJoCo.
Tasks
Trajectories
Average Return
complete
100
2881 1.61.63
back thigh
100
2764.65
back foot
100
2880.78
front thigh
100
201 1.00
front shin
100
3048.01
The common skill encoder, the task-specific skill encoder, the individual value network, and the action decoder use single-layer transformers (64-unit hidden layers) to process the decomposed ob- servation. The mixing value network implements attention block, following Zhang et al. (2023). The forward predictor comprises two single-layer transformers. To tackle the partial observability, append the history hidden state hi_, in each agent's common skill encoder, value net, and action decoder when applying self-attention and thus get the output of ht.
21

---

Published as a conference paper at ICLR 2025
Trajectory (n.of... off m..
(8)
Goordiaaion Polay sy(of)
Tesk-Specfiic Skils Encoder (o()
Polec m,(08)
(oE)
Common Skills of....
Sak-Specili Skills stl of.
Forward d Predictor fele? ")
Azt on Decoder mntolc)).ai
Action Decoder ((of.effx2))
(a) High-Level
Planner
Local Information Raa. R
(b) Low-Level Controller
Costrolllr
Ap
Value Network
Network
Predicred Globel State 4
of
Va Network
(c) Optimization
TD Res-ecc
(c)
Aogi-
Seeploi
Fphaneer
Peess Ore
Natwork
Stop
Figure 4: Overall framework of Hi-SSD in SMAC. HiSSD utilizes hierarchical framework that jointly learns common and spec-fi skills from offline multi-task data to improve multi-agent policy transfer. (a) The high-level planner with common skills. HiSSD) integrates cooperative tem- poral knowledge into common skills and enables an offline exploration. (b) The low-level controller with task-specific skills. The task-specific knowledge can guide the action execution adaptively among tasks. (c) HiSSD uses an implicit Q-learning objective to train the value network.
The common skills, extracted through the common skill encoder, are fed into the forward predictor for global state prediction and the action decoder for execution. Our framework employs a dual- transformer architecture for global state prediction: the first transformer integrates enemy informa- tion, while the second processes ally and preprocessed enemy information predict subsequent global states. For the action execution, we concatenate the decomposed information outputted by the action decoder with the task-specinf skills and feed them into MLP get the real actions. Figure Saaree/illlsse |illusttraes the overall framework of HiSSD in SMAC. The hyperparameters used in SMAC listed in Table
Table 9: Hyperparameters of HiSSD for offline multi-task SMAC.
Hyperparameter
Setting
Hidden layer dimension
64
Hidden units MLP
128
Attention dimension
64
Skill dimension per token
64
Discount factor
0.99
a
B
0.05
E Trajectories per batch
0.9
32
Training steps
30000
Optimizer
Adam
Learning rate
0.0001
Weight decay
0.001 in Stalker-Zealol
0.0001 in others
22

---

Published as a conference paper at ICLR 2025
D.2 DETAILS OF HISSD IN MAMUJoCo
The observation shapes are consistent among tasks and we do not apply the observation decompo- sition deployed in SMAC. The common skill encoder, the task-specific skill encoder, the individual value network, and the action decoder use single-layer transformers (64-unit hidden layers) to pro- cess the observation. The common skills, extracted through the common skill encoder, are fed into the forward predictor for global state prediction and the action decoder for execution. We concate- nate the decomposed information outputted by the action decoder with the task-specific skills and feed them into MLP to get the real actions. The hyperparameters used in MAMuJoCo listed in Table [0]
Table 10: Hyperparameters of HiSSD for offline multi-task MAMuloCo.
Hyperparameter
Setting
Hidden layer dimension
256
Hidden units in MLP
256
Attention dimension
256
Skill dimension per token
256
Discount factor y
0.99
Target update rate
0.005
@x
10.0
2.0
e
0.9
Batch size
128
Training steps
1000000
Optimizer
Adam 0.0005
Learning rate
D.3 TRAINING COSTS
The training process of HiSSD with an NVIDIA GeForce RTX 3090 GPU and a 32-core CPU costs 12- 14 hours typically. Our released implementation of HiSSD follows Apache License 2.0, the same as the PyMARL framework.
E ADDITIONAL RESULTS
E. RESULTS ON OTHER TASK SET IN SMAC.
We follow the multi-task learning settings in ODIS (Zhagg et al. 2023) and conduct additional ex- periments on two offline multi-tase task sets: Marine-Eane -asy and Stalker-Zeale The results are pre- sented in Tables 11 and [2] We also present the average win rates Stalker-Zeate task set in Table 13 to show the improvement of HiSSD on tasks with heterogeneous units clearly, For the Marine- Easy task set, HiSSD gains convincing performance in most source and unseen tasks compared to other basclines. For the Stalker- Zealor task set, HiSSD obtains competitive performance and sur- passes ODIS when the dataset generated by near-optimal policy (i.c., Expert and Medium- Mediim-Experr). Moreover, both Siill-earming-base methods fail to outperform BC-based methods on some tasks in Stalker-Zealot task set. We suspect this is due the different task properties between Marine and Stalker-Zealot task set as the controlled items in Marine are homogeneous and in Stalker-Zealor heterogeneous. Therefore, it is more difficult for the policy learn generalizable cooperative patterns from the limited dataset in the Stalker Zealot task set.
E.2 SENSITIVITY ANALYSIS OF EYPER-PARAMENT
To investigate the model's sensitivity analysis to in Eq. 5 in Eq. 7 and in Eq. I1 conduct experiments on Marine-Hare task set in SMAC with expert data qualities. We list the results in While framework incorporates additional architectural components and
Taalle 14 15
and 16
23

---

Published as a conference paper at ICLR 2025
Table 11: Average test win rates in the Marine- Easy task set with different data qualities. Results of BC-best stands for the best test win rates between BC-t and BC-r.
Tasks
BC: BC-hest
UPDeT-m
Expert
ODIS
Hi-SSD (Ours)
BCC-bee
UPDeT-m
Medium
ODIS
Hi-SSD (Ours)
Source Tasks
3m
4.6
83.6:::2.66
97.74
99.5= 8.1
67.2+
60.2=29.9
57.8=
74.7:114.6
Smn
94.41 7.6
74.8::22.9
95.31: 5.2
99.9
0.0
79.24 5.9
67.8 5.9
82.8 5.2
81.6:10088
16.2772
83.6119.2
88.311.0.3
95.2
48.8.1
Unseen Tisks
4m
91.24 1.6
53.0+:32.3
90.6+ 7.0
94.4
2,9
62.5 11.6
41.7=174
13.3=16.1
74.5=+15.5
75.34:22.6
79.7
75.8-22.7
7m
70.3+11.0
44.2+11.2
12.
99.1=
0.7
99.9 0.0
65.2=25.2
96.1 14
91.3+ 2.3
8m
74.74:16.5
51.7::26.2
80,9
99.8=
0.3
96.9
2.2
88.4=13.7
97.7= 2.6
93.8+ 5.2
9m
97.71 2.6
76.3.113.4
99.21
99.9
0.0
78.91
65.8
87.5
2.2
75.2:115.5
1m
13.3:::1.8
53.6:122..4
816
99.2
0.8
42.24 4.7
13.
64.7
3.1
62.0:12188
56.74:30.0
44.3:422.8
70.3+:30.2
99.7= L.1
29.7+23.4
13.5=117
55.5+225.7
Medium- Espert
Medium Replay
Source Tasks
3m
18.
48.44:36.8
89.8 19-:t0 9.7
90.9= 5.9
77.8+ 3.2
29.7=10.0
79,7= 4.7
87.7= 2.9
5m
74.0:: 2.9
64.1:11999
83.7.1.6.00
79.4 6.9
5.5.L: 5.6
6.2.1088
311 54
87.5.L: 2.9
10m
90.6: 3.1
68.8::23.8
93.8
60.2=21.1
0.0 0.0
0.0=
0.0: 0.0
84.2:::49
4m
35.24:38.0
43.7::25.00
57.8118888
70,9 9.1
67.2 4.7
25.0=222.6
25.0 5.4
71.6::: 4.1
6mn
42.211 1.6
47.7.1.30.0
76.0.
70.6 6.1
7.8 10.2
3.1 5.4
99.8
0.3
7m
65.6:::6.4
57.81:299
66.4
15.0-11.7
0.8 1.4
0.0= 0.0
0.0- 0.0
99.8:t:
0.3
40.3::42.6
49.6
43.8+:1.5
9.5
0.8
96.7+
0.3
9m 70.8:1:6.66
47.74:24.8
73.41:6.2
14.6
0.8 1.4
0.0: 0.0
0.0: 0.0
88.8:: 1.3
15.5+12.4
85.92:1422
68.8420.3
70,0= 5.9
0.0%
0.0=
45.6 4.5
12m
29.7::29..8
46.15:15.5
62.54 8.0
62.7= 7.8
0.02
0.0
0.0:
0.0:2 0.0
38.0:t 3.7
learning objectives compared to prior approaches, empirical evaluations demonstrate its consistent robustness across varying hyperparameter configurations.
E.3 COMPARED TO HYGEN
HyGen (Zhang al. 2024 recent work that focuses on integrating offline pertaining and online exploration to speed up multi-task task MARL for policy transfer. HyGen first pre-trains the skill space with an action decoder using the global information, then implements online exploration to maintain hybrid replay buffer using offline and online data, improving the high-level policy and refining the HyGen, our method requires no online exploration
action decoder simultaneously. Compared and pretraining steps. HyGen only conducts common skills learning while our method leverages task-specific skills to complement the common skills discovery. We compare our method to HyGen in the SMAC's Marine hard task set and propose the empirical results in Table I7] While HiSSD demonstrates superior performance over HyGen across most tasks under expert-level data quality conditions, HyGen capitalizes online exploration achieve policy refinement when provided with moderate-quality offline datasets.
E.4 EXTENDED ABLATION STUDY
To demonstrate the effectiveness of learning task specifc skills, we conduct variant of HiSSD named w/o Specific and present the empirical results in Table [8] We find that policy with only common skills significantly outperforms the BC-based method, learning both skills further improves the transfer capability of the policy.
24

---

Published as a conference paper at ICLR 2025
Table 12: Average test win rates the Stalker-Zealot task set with different data qualities. Bold represents the best score in each task.
Experi
Medium
Tasks
RC-hest
UPDeT-m
ODIS
Hi-SSD (Ours)
BC-hest
UPDeT-m
ODIS
Hi-SSD (Ours)
Source Tasks
93.1: 4.6
50.0::33.4
97.7:
95.2:
48.84
35.0++23.0
49.2: 8.4
32.34:177
264z
78. 1: 8.1
23.44:6.6
60.9+ 6.8
79.8+ 6.0
12.54 8.1
18,+
12.8+12.2
17.0 2.2
3652
9..5+4.2
17.24
87.5+
92.8+
24.4+1244
25.6::24.2
28.9: 6.8
24.4
Unseen Tasks
1s3z
8
1.6:: 1.6
76.64 3.5
81.6.1:5.22
21.9:1.7.66
3.8 5.0
41.4:118.88
44.2 9.9
60.0-+32.33
26.6+:19.3
1722
42.0:226.11
6.24
2.5
50.74 7.5
18.1
165z
45.6-t26.9
29.74:26.4
2.5 = 2.3
16.7=112.3
3.1 2.6
5.0+ 4.2
14.1+ 8,4
2.5 2.2
2552
75.6:::1199
23.4::22.2
27.34
79.7
14.4: 9.0
16.9114.1
32.01 4.6
11.34
3632
80.6.1. 9.1
20.311:0.9
89.1. 5.2
88.0.L
45.61114.
24.4.128.6
23.41 9.2
21.91.0.7
3642
92.5+ 5.1
12.5:t:9.9
96.9+ 2.2
88.1 9.0
40.0=19..0
28.8::11.6
50.8
17.2 4.5
4647
18.4
1.8:13.5
79.7 10.9
73.4: 5.2
10.0
1.2 1.5
12.54 7.0
13.24 6.5
10.6+19..1
5.5
86.74
65.6
14.4
5.6 8.5
7.0+
4s62
48.1422388
4.7%
88.34 8.4
68.4
3.8 3,6
199 2.5
1.6 1.6
094 0.9
Medium-Fspert
Rechuum-Replay
Source Tisks
2s32
$7.5:t25.11
$1.5+:27.1
18.62:5.5
68.1 8.1
3.14 2.6
14.4::3.2
15.61:8822
9.0 1.5
24.11:2.66
41.9110.22
9.7
3652
63.1::1333
35.0::23.5
41.4
17.8
3133 6.3
20.0=:1..6
18.8 3.1
17.54 2.0
Unseen Tasks
25.0:115.4
0.0.1
21.1
164z
25.04:30.7
11.99 9.8
24.5
32.3
2.1 2.9
7.5+10.0
6.24 7.7
24.8 9.1
19.4
3.8
42.24:31.4
9.4+
7.34
11.9 9.6
788
4.4
255z
26.9:120.22
37.5:::2.55
43.0::10.7
25.6. 7.8
12.5:15.5
20.0116.8
14.11 8.1
16.51 2.8
3632
18.6.1.8.0
33.8.1.5.00
50.011 13.3
25.6.125.6
15.4
17.5
25.0.120.1
9.61
3:47
16.4
43.1
52.31 9.5
71.71
20.8:
15.61112
19.5-16.6
22.5+:0.6
463z
69.8+t 7.8
23.8::21.00
17.2 7.2
60.5++1.11
17.7 5,3
11.2415.0
8.6214.9
11.0+:0.4
10.61113.8
20.34
37.34
5.64 9.8
4.7
465z
17.3+ 5.3
11.9+16.1
21.9 2.2
17.0= 4,1
1.0 1.5
10.6
0.8 14
088 0.8
4s62
13.84 3.2
5.04:
18.04
19.7
0.0:: 0.0
6.9-1-3.8
2.3
0,4
Table 13: Average test win rates in the Stalker-Zealon task set with different data qualities.
Data Qualities
BC-best
UPDeT-m
ODIS
HISSD
Source Tasks
Expert
87.94 5.6
30.2:::26.
82.0 6.3
89.3-L 4.0
Medium
28.6+:10.1
26.52:19.2
37.01: 9.1
24.6+ 7.3
Medion-Expeet
12.9::77.9
48.5+:25.1
47.11:139
55.9% 9.7
Medium Medium-Replay
13.2.1:
15.6.1.3.2
14.11 8.8
10.81 1.6
Target Tasks
Expert
60.9+119.0
13.8
62.8+ 7.5
69.2+ 8.7
Medium
14.8:114.1
10.11:11.8
24.7.1 8.4
16.611 7.4
Medium-Expert
17.4
1861144
38.2411999
40.3-1:1.88
Medium-Roplay
13.6:27.6
10.74:188
11.0:1088
13.6:: 4.8
25

---

Published as a conference paper at ICLR 2025
Table 14: Average test win rates on Marine-Hard task set SMAC with expert data qualities for the model's sensitivity analysis balances the imitation and exploration in offline value estimation.
Expert 0.5
Tasks
0.1
0.3
0.7
0.9 (Ours)
Source Tasks
Sm6m
73.54 5.1
31.31: 3.3
70.8 244
74,4 1.5
66.14 7.0
9m
98.34 0.3
97.1
99.0=
96.74
0.6
95.54
Unseen Tasks
4m
97.71 1.6
99.01 0.3
99.01 0.8
98.41 0.5
99.21 1.2
5m
99,8+ 0.3
99.8
99.2#
99.1
99.24
10m
99.24 0.8
98.54 1.6
99.2 0.3
99.2 0.3
98.41 0.8
12m
69.01133.7
67.9 17.0
71.01.260
87.5
7.6
75.51.179
7m8m
29.24 2.8
23.1
32.74
25.4 299
35.31
8m1m
16.0:=16.6
47.5 9.2
45
44.6
47.0 6.2
10m11m
86.3::14.6
10m12n 2mn
11.51 6.9
9.4. 5.9
8.1 2.2
7.3 5.5
14.51 9.1
13m15m
0.2 0.3
0.4:
0.4 0.6
Table 15: Average test win rates Marine-Hard task set in SMAC with expert data qualities for the model's sensitivity analysis to a. a weights the TD-residual Eq.
Export
Tasks
a
a
a 20
a (0rr))))
Source Tasks
3m
99.24 0.6
99.9-t 0.0
99.84 0.3
99.5: 0,3
Smfmmn
70.8.1 4.7
71.0.1
66.51
66.11 7.0
9m
97.11 1.8
97.9 2.1
93.41 1.6
95.5: 2.7
98.4+t
98.1+
99.0:
99.2:t 1.2
5m
99.8.1
0.3
99.81 0.3
98.81 1.8
99.2:1 1.2
10m
99.4:1 0.5
99:41 0.9
99.01 0.3
98.41 0.8
12m
59.24136.2
63.8::33.6
66.5+226.5
17.5
7m8m
34.5:t 5.0
34.8++.0.3
34.6 2.8
35.3+t 9.8
8m9m
46.3+
59.24-13.3
53.34
10ml Im
83.3:
8.2
73.9
77.34 6.8
86.3++1..6
10m12m
10.2:
8.34 83
11.9 6.7
14.5 9.1
13m15mm
2.5 3.5
0.2.1
1.3. 1.8
1.3. 2.5
Table 16: Average test win rates on Marine-Hard task set SMAC with expert data qualities for the model's sensitivity analysis B. controls the margin of regularization in Eq.
Export 0.1
Tasks
0.001
B
Source Tasks
99.8.1 0.3
99.9 0.0
96.8.1 4.5
99.51 0.2
99.51: 0.3
5m6m
65.34 1.1
64.74
64,55 4.9
65.9t 2.7
66.11 7.0
9mm110
95.4 2.6
95.3#
0.3
94.3 0.6
92.0+ 2.4
95.5t 2.7
Unseen Tasks
98.3.1 1.9
97.7.1 0.6
97.5.1 2.2
98.8.L 0.3
99.2 1.2
5m
99.0st 0.8
98.8:
0.3
99.0tt 0.9
99.14 0.8
99.2: 1.2
10m
1.4
98.6
96.74 3.4
97.3+ 1.3
0.8
12m
64.8
66.9-122.0
18.14430.3
75.24-20.3
75.5.
7m8m
34.0:t 5.5
354+1220
36.5+13.00
40.6+
35.3+ 9.8
8m19
50.8+13.33
40.2+ 28
58.11-13.0
41.34 7.2
47.0+ 6.2
10ml Im
79.4:1 2.3
79.6.1 7.4
80.6.117.4
76.7115.8
86.3
10m12m 2m
7.741 1.6
11.2:110.6
7.9.1 3.1
14.4: 8.5
14.5.: 9.1
13ml 15m
1.0tt 0.8
1.74 1.6
1.0t 1.1
2.14 1.8
1.3 2.5
26

---

Published as a conference paper at ICLR 2025
Table 17: Average test win rates of the best policies over five random seeds in the task set Marine- Hard with Expert and Medium data qualities. Results of BC-best stands for the best test win rates between BC- and BC-r.
Expert HyGen
Medium HyGen
Tasks
BC-best
(Ours)
HiSSD (Ours)
Source Tasks
3m
97.74 2.6
99.11 1.0
99.5
0.3
65.4:11.77
91.5:::11.0
62.7: 5.7
5m6m
50.4
61.24
66.1
21.9 3.4
31,6:
26.44 3.8
9m110m
95.34
96.42
3,0
95.5#
2.7
63.811099
79.24 4.0
73.9: 2.3
Unseen Tasks
92.14
95.84
99.24
48.8412.11
77.3++0.22
5m
87.1 10.5
99.5 1.0
99.2=
1.2
76.6+14.1
96.5 6.0
88.4+ 8.4
10m
90.5+
93.5 5.0
98.4
56.24:20.6
96.44
98.0: 0.3
12m
15.8.15.2
85.21 6.0
75.51.19 19.7
24.0.110.5
81.5.11.00
86.4 6.0
7m8m
18.8
28.9
35.3
1.6 1.6
24.3.
14.24:0.11
8m9m
15.8
25.7
47.0=
3.1 3.8
24.5+
15.3+ 2.8
10ml Im
45.3-1
57.2443.0
86.3 14.6
19.7 8.9
47.24113.0
43.6.L 4.6
10m12m
1.0+
13.84
14.5
0.0+ 0.0
5.2:
0.6
13m15m
0.0= 0.0
9,5# 5.0
1.3 2.5
0.6 1.3
9.34 6.0
1.4+ 2.4
Table 18: Additional ablation studies on HiSSD. We report average test win rates of the best policies over five random seeds in the task set Marine-Hare with different data qualities.
Data Qualities
w/o Spocific
BC-best
HiSSD
Source Tasks
Expert
85.9 16.4
81.11218
14.5
Medium
23.11:20.0
50.3::20.1
55.9::992
Medium Expert
63.51126.1 47.74:25.1
41.711885
23.5
Medium--Replay
46.5:226.
$1.91:2000
Turgel Tasks
Expert
55.0::37.4
46.8:236.88
61.2::37.4
Medium
37.6.134.2
25.6:1.6.8
48.6 40.1
Medium-
51.2:1:4.3
38.8113300
37.6
Medium-Replay
42.8:236.2
41.2+:3.7
48.7=139.0
27


Reviews:
[{"review_id": "C1nrF9OJDj", "reviewer": "Reviewer_yFSo", "summary": "This paper proposes a new method called Hierarchical and Separate Skill Discovering (HiSSD) for learning generalizable multi-agent policies from offline multi-task data. The key idea is to jointly learn common skills that capture cooperative behaviors across tasks, as well as task-specific skills for fine-grained action execution. The method uses a hierarchical framework with a high-level planner that learns common skills and a low-level controller that learns task-specific skills. Experiments on SMAC and Multi-Agent MuJoCo benchmarks show improved generalization to unseen tasks compared to baselines. Overall, this paper presents a novel and effective approach to an important problem in multi-agent RL. With some additional analysis and clarifications, it would make a good contribution to the field.", "questions": "1. There is a very recent work that tackles the same problem from a different methodology perspective found here : \"Variational Offline Multi-agent Skill Discovery\" by Chen et. al. I would like to know the authors' thoughts on how their methodology is different from this work, and if there are scenarios where one or the other might work well/worse.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel approach to learning generalizable multi-agent policies by jointly learning common and task-specific skills.\n\n2. Theoretical analysis and derivation of objectives for skill learning.\n\n3. Comprehensive experiments on multiple benchmarks showing improved generalization.\n\n4. Addresses an important problem in multi-agent RL of transferring to tasks with varying numbers of agents.\n\n5. The authors have shared comprehensive details of implementation along with codebase, which further bolsters the experimental efforts.", "weaknesses": "1. Ablation study can be further expanded to have a comprehensive analysis of different components. Comparison to some recent related works like HyGen is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f8P7FiQXmS", "reviewer": "Reviewer_niW9", "summary": "This paper proposes HiSSD to leverage a hierarchical framework to jointly learn the common and task-specific skills in offline MARL settings. Experiments in SMAC and multi-agent MuJoCo show its effectiveness and generalizability to different tasks.", "questions": "1.\tFrom Equation (3), does $q(s_{t+1}\\mid c_t^{1:K})$ has any meaning in real? How does HiSSD model the sequence $c_t^{1:K}$?\n2.\tIf the negative samples are drawn from limited seen tasks when doing task-specific skill learning, how can we ensure a correct skill when facing unseen task distribution?\n3.\tWhy are the two baselines, ITD3-CQL and ITD3-BC, not compared in the SMAC experiments?\n4.\tHow is the performance if we only adopt a common skill without using the task-specific skills?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe division into common and task-specific skills provides a balance between generalization and task adaptation, addressing the limitations of previous methods that only focused on common skills.\n2.\tThe proposed objective to learn common skills for exploration and prediction is new.\n3.\tHiSSD enables multi-agent policy transfer across varying tasks, outperforming existing multi-task MARL baselines on two benchmarks", "weaknesses": "1.\tHiSSD’s hierarchical design and multi-component training can lead to instability. I think the authors should reveal more about the training details, hyperparameters selection, and tuning methods. \n2.\tIt can be better to present an algorithm of the whole training process and the execution process for understanding. \n3.\tIn the ablation studies, the HiSSD variants (e.g., w/o Planning and Half-Negative) often show strong performance compared to HiSSD with minor differences or even surpassing HiSSD. \n4.\tThe visualization experiments do not show clear evidence about the latent skills. For example, the task-specific skills in the same task seem closer in Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LlP46XprtZ", "reviewer": "Reviewer_7LGd", "summary": "This paper proposes a hierarchical framework, HiSSD, for jointly learning common and task-specific skills from offline multi-task datasets. The common skills represent the general cooperation patterns, while task-specific skills achieve a task-guided fine-grained action execution. The algorithm is broken up into two phases. In the first phase, the high-level planner discovers common skills. In the second stage, the low-level controller learns task-specific skills.", "questions": "- In Table 3, I wonder why presents BC-best's results, rather than ODIS. ODIS seems to be the baseline with the best performances.\n- In Figure 3, Large-scale tasks (i.e., 10m and 12m) overlap with each other but small-scale tasks don't. The paper said this is because 10m and 12m are similar. However, 3m, 4m, and 5m are also similar. I would appreciate some further discussion about this.\n- In Equation 6, the weight $\\alpha$ serves as the trade-off between guiding to space with high-reward and space that the execution policy ought to have a correct imitation. I wonder why the paper didn't conduct a sensitive analysis for this parameter like $\\beta$ in Equation 11, as shown in Appendix E.2.\n- The task-specific skills learn task-specific knowledge of source tasks, why do these skills play an important role in unseen tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written, with thorough experiments, and a variety of tasks across two environments.\n2. In order to achieve better performances in cooperative multi-agent environments, the paper brings cooperative temporal knowledge into common skills, which is reasonable.", "weaknesses": "1. According to the paper, HiSSD is superior to previous skill-based methods like ODIS because it brings cooperative temporal knowledge into common skills and discovers task-specific skills that can distinguish each task’s specific knowledge. However, HiSSD fails to outperform ODIS in some tasks.\n2. In MAMuJoCo, the paper didn't use ODIS as a baseline, which has a good performance in SMAC.\n3. The process of learning skills is similar to imitation learning, so it failed to perform well in unseen tasks, indicating low-level generalization ability. For example, in Table 1, the win rates for 10m12m and 13m15m are lower than 10% with the medium-quality dataset.\n4. In the passage, some verbs sound a bit odd. For example, 'contain' in the sentence 'Equipping offline multi-task MARL with skill learning to improve policy transfer still contains issues.'\n5. One concern for the paper is to evaluate HiSSD in maps that are more difficult (like the Stalker-Zealot task set in ODIS), to validate its effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wrecboTf3w", "reviewer": "Reviewer_94BV", "summary": "This paper introduces Hierarchical and Separate Skill Discovering (HiSSD), a method for learning generalizable skills in offline multi-task multi-agent reinforcement learning. HiSSD's hierarchical framework simultaneously learns common skills (cooperative behaviors across tasks) and task-specific skills (unique task characteristics). It uses an objective function combining reward maximization and next-state prediction for common skills, and contrastive learning for task-specific skills. HiSSD trains on offline data and enables decentralized execution with local information during testing.", "questions": "* Can you explain how performing global state prediction and value estimation directly leads to learning cooperative knowledge?\n* HiSSD claims to simultaneously learn common skills and task-specific skills, but how did you verify how distinct these two types of skills actually are? Was there any overlap or interference between the two skills?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper introduces Hierarchical and Separate Skill Discovering (HiSSD), a method for learning generalizable skills in offline multi-task multi-agent reinforcement learning. HiSSD's key strength is its hierarchical framework that simultaneously learns common and task-specific skills, overcoming existing limitations. It enables fine-grained action execution and trains using only offline data. During testing, HiSSD allows decentralized execution using local information and learned skills without centralized control. Experiments on SMAC and MAMuJoCo benchmarks showed HiSSD's performance improvements, especially in unseen tasks.", "weaknesses": "While the paper claims to integrate cooperative temporal knowledge into the process of learning common skills, it fails to adequately explain the clear causal relationship or mechanism between these two concepts. There appears to be a logical leap in assuming that simply performing global state prediction and value estimation directly leads to the learning of cooperative knowledge. Furthermore, the paper lacks a clear definition of what exactly \"cooperative temporal knowledge\" means and how it is measured and evaluated. This ambiguity makes it difficult to verify whether this knowledge has been learned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jRT4SiMz1l", "reviewer": "Reviewer_sVE6", "summary": "This paper proposes an algorithm for multi-task multi-agent skill discovery from offline data, which is a quite meaningful and challenging research direction. The novelty (compared with ODIS) mainly lies in jointly learning common and task-specific skills.", "questions": "Please see the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "(a) This paper targets at a challenging problem setup, i.e., recovering generalizable multi-agent skills from multi-task offline data.\n\n(b) The evaluation results are sufficient and show consistent improvements over the SOTA algorithm -- ODIS.", "weaknesses": "(a) The algorithm framework, including both the multiple components to learn (Fig. 1) and the objective design (Eqs. 4, 7, 11), is quite convoluted compared to ODIS, which could potentially bring training difficulty.\n\n(b) The writing needs to be significantly improved, especially for methodology part in Section 3. \n\n(c) It's not clear why the common skills can integrate cooperative temporal knowledge and enable an offline exploration.\n\n(d) Since all tasks share the same space for the local skills z^{1:K}_t, why z can embed task-specific knowledge?\n\n(e) Multi-agent skills should provide abstractions at both temporal-level and subgroup-level. That is, each multi-agent skill should embed a (short) control sequence of a subgroup of agents. How is this aspect shown in the algorithm design?\n\n(f) Thanks for releasing the code package. But it seems that the authors only made minimal changes to the readme file provided by ODIS. Please provide a detailed instruction (in the readme file) on how to reproduce all the reported results in this paper. Also, the released yaml files do not include the ones for MAMuJoCo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)