Bi-Level Knowledge Transfer for Multi-Task Multi-Agent Reinforcement Learning
Authors: Junkai Zhang
Jinmin He
Yifan Zhang
Yifan Zang
Ning Xu
Jian Cheng
Venue: NeurIPS 2025 poster ()
Keywords: MARL; multi-task; policy generalization for zero-shot
Primary area: reinforcement_learning
TL;DR: A Multi-Task Transfer Learning method via tactic-level and individual skill knowledge.

Abstract: Multi-Agent Reinforcement Learning (MARL) has achieved remarkable success in various real-world scenarios, but its high cost of online training makes it impractical to learn each task from scratch. 
To enable effective policy reuse, we consider the problem of zero-shot generalization from offline data across multiple tasks. 
While prior work focuses on transferring individual skills of agents, we argue that the effective policy transfer across tasks should also capture the team-level coordination knowledge.
In this paper, we propose Bi-Level Knowledge Transfer (BiKT) for Multi-Task MARL, which performs knowledge transfer at both the individual and team levels. 
At the individual level, we extract transferable individual skill embeddings from offline MARL trajectories.
At the team level, we define tactics as coordinated patterns of skill combinations and capture them by leveraging the learned skill embeddings. 
We map skill combinations into compact tactic embeddings and then construct a tactic codebook.
To incorporate both skills and tactics into decision-making, we design a bi-level decision transformer that infers them in sequence.
Our BiKT leverages both the generalizability of individual skills and the diversity of tactics, enabling the learned policy to perform effectively across multiple tasks.
Extensive experiments on SMAC and MPE benchmarks demonstrate that BiKT achieves strong generalization to previously unseen tasks.

Bi-eevel Knowledge Transfer for Multi-Task Multi-Agent Reinforcement Learning
Junkai Zhang 12, Jinmin Hel Yifan Zhangl.23 Yifan Zang", Ning Xu', Jian Cheng 1.25
DL, Institute of Automation, Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences
University Chinese Academy of Sciences, Nanjing
*Beijing Institute of Astronautical Systems, *AiRiA Zzhangjunkai2021, hejinmin2021, zangyifan2019) @ @ia.ac.cn, (yfzhang, jeheng @nwpr.ia.ac.... nxu muu@njsst.dd.cc
Abstract
Multi-Agent Reinforcement Learning (MARL) has achieved remarkable success various real-world scenarios, but its high cost online training makes it impractical to learn each task from scratch. To enable effective policy reuse, we consider the problem of zero-shot generalization from offline data across multiple tasks. While prior work focuses on transferring individual skills of agents, we argue that the effective policy transfer across tasks should also capture the team-level coordination knowledge. In this paper, we propose Bi-Level Knowledge Transfer (BiKT) for Multi-Tass Task MARL, which performs knowledge transfer at both the individual and team levels. At the individual level, we extract transferable individual skill embeddings from offline MARL trajectories. At the team level, we define tactics coordinated patterns of skill combinations and capture them by leveraging the learned skill embeddings. We map skill combinations into compact tactic embeddings and then construct a tactic codebook. To incorporate both skills and tactics into decision-making, we design bi-level decision transformer that infers them in sequence. Our BiKT leverages both the generalizability of individual skills and the diversity tactics, enabling the learned policy to perform effectively across multiple tasks. Extensive experiments SMAC and MPE benchmarks demonstrate that BiKT achieves strong generalization to previously unseen tasks.
Introduction
Multi-Agent Reinforcement Learning (MARL) has shown great potential in wide range of real-world
applications. such as autonomous driving [4] 44], robotic collaboration [I] 29], and distributed control [24] 39). However, training MARL agents from scratch for each new task often prohibitively expensive due to the high cost of online interactions and the complexity of coordination among agents 45 In real-world scenarios, we usually have access offline MARL trajectories of some known tasks. Thus, distilling the multi-agent policy from such data and transferring it to unseen yet related tasks offers cost-effective solution, which reduces reliance on online interaction. Consequently, we consider to solve the problem where agents are trained on known multi-tasks using onf data and then directly evaluated on previously unseen tasks, aiming to achieve zero-shot generalization.
"Coresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Individual Skills
Keep firing on one anemies
Escape Health (EIP)
Advaance
Similur Tactic
Different Tactic
Tactic: agents simultancoudl
Tactic: agents simultaneoussy
Tactic: agent fire
Tactic: Each agent protects itself
focus fire on an enemy
focus fire on an enemy
to profect low HP agents
(a) 5m_ 6m
7m vs 8m
5m vs 6m
(d) 5m
Figure 1: Skill and tactic examples in SMAC tasks. In (a) and (b), two tasks utilize a similar tactic: All agents keep fire the same enemy, which forms the team tactic to quickly eliminate enemy to gain a numerical advantage. In (c) and (d), two tasks utilize two different tactics: In (c), due to unbalanced HP. Agent 2,3 adopt Skill A, agents 4,5 use Skill B, and agent I adopts Skill C to draw enemy fire to protect the low-HP teammates. In (d), with balanced HP, agent 1,2,3 adopt Skill A and agent 4,5 adopt skill B, which corresponds the tactic that each agent protects itself low HPs.
An alternative approach is to leverage offline data to train task-agnostic policy with unified structure capable of taking actions in multiple tasks 10). However, they are primarily designed for online training, and their generalization heavily relies on the neural network's inherent capacity, which often fails when facing variations in agent number and corresponding observation spaces. Besides, the efficient policy transfer can also be achieved by the use of skill, which abstracts the action pattern of agents into compact representations [32]. This has been an effective strategy in single-agent generalization across tasks, allowing agents reuse knowledge without learning from scratch. Inspired by it, recent studies have explored skill-basee policy transfer to MARL (0] 41]. where each agent learns to extract transferable skills and generates actions conditioned on them. We refer to these skills as Individual skills, which capture the policy of individual agents in MARL.
However, transferring individual skills alone is insufficient for effective policy transfer in MARL. While individual skills facilitate the knowledge transfer of individual action patterns, they fail to convey how these skills interact cooperatively within a team across tasks. Therefore, a comprehensive knowledge transfer in MARL should also capture the cooperative patterns of skills, which we define as tactics. As illustrated in Figure I. different tasks may share similar tactics, which suggests that tactics can serve as valuable transferable knowledge. Besides, even in similar states, distinct tasks may require different tactical choices, which indicates the need for policy's 's adaptive selection in unseen tasks. Therefore, effective policy transfer in Multi-Taskk MARL requires levels of knowledge: Skill transfer at the individual level and the tactic transfer at the team level.
In this paper, we propose our novel Bi-Level Knowledge Transfer (BiKT) for Multi-Task Multi-Agen Reinforcement Learning. Specifically, at the individual level, we perform skill transfer by extracting latent skill representations from offline trajectories. At the team level, leverage these learned skill embeddings to learn a compact tactic embedding, which captures the combinations of individual skills. We then discretize the tactic embeddings and organize them into a finite set, which we define as tactic codebook. To employ both the skills and tactics for decision maaiin,,,mm,,ng,..... we design a bi-level decision transformer as the policy model, which selects appropriate tactics from the learned codebook and utilizes them to infer individual skills. The actions are then decoded from the skills. Through the above design, our BiKT benefits from the diversity of tactic codebook and the generalizability of individual skills, which can guide the policy to generalize in unseen tasks with high performance.
Our method exhibits the following contributions: 1) Our framework facilitates the bi-level knowledge transfer in multi-task MARL, enabling both individual and team-level generalization. 2) Our proposed tactic codebook captures the diversity of team behaviors across source tasks, providing the flcxibility to generalize to wider range of unseen tasks. 3) Our learned tacties serve team-level guidance, enabling agents to select different skills under similar scenarios across tasks, which prevents the monotony of skill combinations. Extensive experiments in Marine Hard, Marine Easy, Stalker Zealot, and Cooperative Navigation in SMAC (31 and MPE (22 showcase our outcome performance.

---

2 Related Works
Offline MARL Offline reinforcement learning has emerged as a compelling paradigm for developing Eecision-making policies solely from static datasets [8]] Recent efforts have extended this paradigm to multi-agent settings, leading to the increasing prominence of offline MARL 43.. A major challenge the distributional shift between the training dataset and unseen state-action spaces, which results in inaccurate value predictions 7 38. To mitigate this, sehavior-constrained learning frame- works have been proposed to encourage conservatism in policy updates [T6] 17]. These conservative principles are often embedded within existing online MARL methods, either by extending multi- agent policy gradient methods [5] 22 by applying value decomposition techniques for joint value estimation [30, 33 (35]. Furthermore, recent developments have concentrated on formulating safe and robust offline MARL algorithms, ensuring strong performance guarantees under distributional shifts 8 26 40 However, excessive conservatism may hinder generalization, as policies trained on static data may overfit to the offline distribution. In parallel, diffusion-basse generative models have been introduced to enhance offline policy learning [13 [27] Despite their potential, integrating such models with the policy improvement remains a challenging problem [14 36 46).
Multi-Task MARL Multi-Task MARL aims to train a unified policy that can handle multiple tasks simultaneously and generalize directly to unseen tasks, which presents two primary challenges: the design of a universal policy and ensuring effective generalization across tasks. To enable a universal policy, flexible architectures empower universal policies process variable inputs m TO 12] 47 A
curriculum learning strategy was grounded in evolutionary principles to enable scalability concerning the number of agents (2T]. Methods such as randomized entity-wise factorization (REFIL) enhance policy adaptation by improving generalization [12), while transformer-basee approaches like UPDeT leverage population-invarte
networks to manage variable agent configurations [IO] 19 37. Despite
these advances, they usually rely on simultaneous learning or fine-tuning across tasks, which limits their generalization. To address this, the MATTER focuses on learning task representations to capture inter-agent relationships
[28]. Additionally, ODIS [41] and HiSSD [20] promote generalization to novel tasks by incorporating skills within unified training paradigms. However, the pursuit of training policies robust enough for deployment in new tasks remains a key challenge, highlighting the need for continued research to push multi-task MARL towards more versatile solutions.
3 Preliminary
3.1 Cooperative Multi-Agent Reinforcement Learning.
cooperative multi-agent task can be modeled as a Decentralized Partially Observable Markov Deci- sion Process Each agent only has access to a partial observation of the global state and selects actions individually. Specifically, A = {1 -1.-... N) denotes the set of agents. E S is the global state, o² is the obser- vation of agent i derived from 8, and a is the action of agent i. Notably, each observation o' can M). where M is the total number of entities (including both allies and enemies). The E [0, 1) is the discount factor. The transition function is P
(25). which is defined by the tuple (N, {or E fa' J P, P.R, 7).
be decomposed into entity-wics wise features {el c²
+ [0, 1]. where the joint action is defined as a {af Ji=1 The action-observation history is oi (ob, ob ..... of), which serves as input to the policy T' T'(((T)) + [0, 1]. At each timestep, the environment provides reward re via the reward function R s a + R.
3.2 Multi-Task MARL: Policy Transfer via Offline Data
Although MARL algorithms have achieved substantial progress, they still require training from scratch for each task, which is time-consuming and interaction heavy. Nevertheless, policies trained on one task often exhibit transferable behavioral patterns applicable to similar tasks. For example, coordination policies learned in the SMAC 3m task can be reused in the 5m setting. To measure how well such knowledge transfers, we adopt multi-task MARL. as a principled benchmark. We denote TAre = Nure and Tiat = (Th,
( J as the sets of source and unseen target tasks, respectively, where Nure and Nog represent the task numbers. Source tasks { Nor, are associated with an offline dataset Dene = { Nor, collected by a pre-trained policy. A trajectory from D is defined as (s) ab. 1. ro...., si where H is the length of trajectory, of and at are the joint state and action
3

---

at time t, and r = (s, a,)) ) is the reward. Under the multi-task generalization setting, our objective is learn general multi-agent policy that maximizes the expected discounted return across all tasks, shown in Eq. T After training, is evaluated on Tigs without further fine-tuning.
max (Tac T
Σ Bapon
y R" (s) a.)
3.3 Decision Transformer
The Decision Transformer (DT) 3 leverages the Transformer architecture to formulate the Markov decision process conditional sequence modeling problem, which attains high performance under offline training. Given the offline trajectories consisting of states $t, actions dg. and rewards ra. DT models the joint distribution over actions as: = f(Rt St-L.t, Lt---) R₃ = =-'= denotes the target return-to-go, L is the content length, and f is the DT model which is trained to maximize the likelihood of the observed action sequences conditioned on past states, actions, and returns, shown in Figure This formulation allows DT to learn directly from offline trajectories.
H
max E (R (RRRR x
log f(a: Ra-Las 81 Li- a Lo-L-t-1)
4 Method
In this section, we introduce our novel Bi-Level Knowledge Transfer (BiKT) for Multi-Task Multi- Agent Reinforcement Learning method. We argue that effective multi-task generalization in MARL necessitates knowledge transfer at two distinct levels: (1) The individual level, which involves learning and reusing agent-specific behaviors, referred to as individual skills and (2) The team level, which aims to transfer coordinated behavioral patterns that emerge from combinations of individual skills, referred to cooperative tactics BiKT enables each agent to first generate team-level tactic, and then infer an individual skill conditioned on it. Our objective is to learn both individual skill and team tactic embeddings from source tasks, utilize them to guide the decision making, and then effectively reuse them in unseen tasks.
Specifically, the BiKT framework for multi-task MARL is structured into three stages: (1) Extracting the individual skill embeddings for each agent that encapsulate transferable behaviors applicable across source tasks. (2) Building cooperative tactic set, which captures coordination patterns of individual skills and supports knowledge transfer at the team level. (3) Developing a decision- decision-makin making model that integrates both individual skills and team tacties to guide the policy execution. The following sections describe each stage in detail.
4.1 Knowledge Transfer through Individual Skills
In multi- multi-agent scenarios, the abstraction of agent i's policy at timestep is usually used to denote individual skill 2f. We aim to fulfill the skill learning process with two components: First, each agent can extract its individual skill embeddings from the state 8 and joint actions a, which captures common and reusable action patterns in source tasks. Second, when acquiring 2/, agent i can reconstruct actions by its history Tf to take decentralized execution. To achieve them, we model the individual skill learning process as the Individual Skill Encoder: Paan (- (s,aa,.))) - 2¹ E RRN. where N₂ is the dimension of skill embedding, and Action Decoder: qaer (- 2.) + â', illustrated Figure _ For skill training, adopt the Variational Auto-Encoderr (VAE) [15] framework. Our objective is to maximize the log-likelihood of reconstructing actions while regularizing the latent skill distribution, given by Equation where 8j (st, rf. at), i N,t E H, p(2() is the uniform prior over skills, D is the KL divergence function, and Φ1 is the parameters of Patil and Juct.
Lxan (01) = EEg-DA Peappua(11s [log Qact(-17 =t)] Dx [paan(-Ist, a, 01(((=)))] (3)
Individual skills learning offers a task-agnostic abstraction over actions, effectively mitigating the challenge of action semantic misalignment across tasks. Besides, they serve as a commonly used knowledge and can be transferred across tasks. Our focus then shifts to leveraging cooperative patterns among skills to facilitate effective policy transfer at the team level.

---

A
= Cga m. ]
Individual Skill Encoder Potall (-
2
N.
Action Decoder
a v. at |-
N
Jact ,',²')
A,
B
Taclic Codebook C
Cooperative Tactic Encoder
NNS
Skill Decoder Dokill (- Ce. r
-.,,,..,, 1
Pinc (- 8, ...
Tg Tt
Figure 2: (A) The Individual Skill learning process, where ef stands for the feature of entity i. (B)
illustrates the Cooperative Tactic codebook learning process. NNS stands for the Nearest Neighbor Search: Search the nearest neighbor from the codebook C, i.e., arg min || c The network details and feature processing details about encoders and decoders are shown in the Appendix C
4.2 Knowledge Transfer through Cooperative Tactics
While single-agent can often generalize across tasks through reusable individual skills, multi-task MARL poses an additional challenge: Agents need not only to transfer individual skills but also to learn to coordinate them to achieve effective team-level behaviors across diverse tasks. Intuitively, the coordination of skills can be represented as a skill combination, structured as (2' x 2² x... X M However, due to variations of the number of agents across tasks, the resulting skill combinations may differ even when the underlying skill combinations are the same. As shown in Figure the "All focus fire to enemy" tactic in 5m _______mmm and 7m_ ____y__m corresponds to different skill combinations, yet they semantically represent the same team tactic. To handle such variation, we propose mapping process from a single tactic to multiple task-specific skill combinations, enabling the reuse of the same tactic across diverse task settings with varying agent numbers and environmental dynamics.
Specifically, we propose map the skill combinations into compact tactic set, denoted by tactic codebook C {c* [ Tk=1, where the c² E N. represents a reused cooperative tactic embedding, No is the embedding length, and K is the tactic number C. Each tactic cf is intended to represent a class of similar skill combinations across tasks and each agent can infer its individual skill based on the given c²_ To this end, we construct a Cooperative Tactic Encoder (-1s {2 L Ji=i) + & E IR N, a
Skill Decoder (-ITT- c²) + 2,, a the tactic codebook = (c* L shown in Figure 2 The Tactic Encoder maps the global state 8, and joint individual skills [=] . = into a tactie representation
2, which is then discretized via the nearest neighbor search in the tactic codebook C. The Skill
Decoder then reconstructs each agent's skill from local history 7 and the tactic ck
To train the encoder, decoder, and tactic codebook, we adopt the Vector Quantized Variational Auto- Encoder (VQ-VAE)
34 framework to optimize two objectives: the reconstruction error of individual skills and the codebook commitment loss. The reconstruction loss minimizes the distance between the individual skills embedding 2" and 2t; The commitment loss encourages Ptac to output close to the selected c², minimizing the distance between é and c². To promote diversity among tactics, we also impose a regularization term that penalizes similarity between tactics in C. Our complete training objective is given by Equation 4 where C (8t, ri, = ..., af") i E N, E H, db2 is the parameter set of Prae Oskill, and de the parameters C. sg[:] the stop-gradient operator, ,9,, 32 balances the codebook commitment loss and distance penalty, e is small constant for numerical stability.
Limetic
1| Usan (cr sg(c - (e), -))++)) |sg(&) - 1
31 sg( B2
Σ
k k1
Through this design, each learned tactic represents meaningful and reusable coordination pattern, and the resulting tactic codebook captures diverse range of team behaviors across source tasks. As
5

---

1
ag
Tactic Cadeback C
Bi-Level Decision Transformer
Action
Ct-1
R
observation decoupling
0
Figure 3: Bi-Level Decision Transformer (BDT) of agent i. At each timestep it receives structured input sequence, including the index of the tactic codebook: I_e. the previous skill ='==> the predicted returntto-oo Rt Σ Tr, and the agent's decoupled entity-wise observation [e', ,e, The BDT first predicts the index of the tactic codebook Iř, which is then used to retrieve the corresponding tactic e₁. Conditioned on the tactic Ct. the BDT generates its individual skill zí. which is decoded into the action oj via the Action Decoder fact-
result, it enables flexible generalization to broader set of unseen tasks. Moreover, these tactics act as stable team-level guidance, assisting agents in selecting appropriate individual skills across varying scenarios.
4.3 Decision Making with Tacties and Skills
After acquiring the individual skills and the tactic codebook, the remaining challenge lies how to leverage them to guide policy transfer in unseen tasks. To this end, we propose our Bi-Level Decision Transformer (BDT) as the policy má for agent i, which models an autoregressive distribution over the tactic G, E C and individual skills 2/: [c(c) R2,, dze: 2<2)) and T (2f Ri o'zes dz 2 detailed in Figure To accommodate varying observation dimensions across tasks, observations are decomposed into structured entity representations to ensure unified input format. Each agent learns first generate cooperative tactic, and then let them utilize it to take their individual skills.
The policy of each agent Th is trained in supervised manner. We employ the pre-trained individual skill encoder Paau and the cooperative tactic encoder Pue to generate and c² as supervision signals. In practice, promote tactic-level consistency, we train agent i at timestep to predict the index of the shared tactic codebook I rather than directly learning the tactic embeddings. The output dimension of I/ corresponds to the number of tactics in C. The training objective is defined in Equation 5| where Ω = (f'S Th c2es 2'2x) and is the hyperparameter to balance the learning speed for tactic and individual skill.
fpllicy (0) Baj-D... [-a log (T) R'g 05,, d'a)) log mh(z) R's
(5)
is worth noting that the design of DTT-baeed policies for MARL has been previously explored, like MADT 23 However, it is limited to single-task settings, DT tends to learn fixed policy pattern that often corresponds to specific team combination. In contrast, our BDT design can utilize the team tactic the team guidance to promote diverse team tactics, which extends the DT-based policy MARL from single task to multiple tasks.
Overall Training Summary. We first train the individual skill using Lam(()) to acquire skill embeddings and the action decoder Jas.. Next, learn to construct the tactic codebook by minimizing Lieetic (02, dc). At last, we leverage the pre-trained skill and tactic codebook to train the BDT by minimizing Epolicy (l). The pseudocode is detailed in the Appendix [
Decentralized Execution. For each agent i, initialize its policy BDT T with a high returntoo-g and input its history o to predict the cooperative tactic codebook index TÇ. Then, the corresponding tactic retrieved from the codebook C and used input into BDT to generate the individual skill.
Finally, the individual skill is decoded into action of.

---

Experiment
We exploit different environments conduct large number of experiments, including StarCraft Micromanagement (SMAC) [31] and Multi-Agent Particle Environment (MPE) [2). We conduct
five random seeds for each algorithm and evaluate them with 32 environments.
5.1 Experiment Setup
StarCraft II Micromanagement The StarCraft Multt--gent Challenge (SMAC) 31 is popular MARL benchmark which provides standard platform for evaluating multi-task learning and policy transfer capabilities in MARL. Following the experimental protocol proposed by [11], we leverage the task and corresponding offline datasets they released: Marine-Hard Marine-Easy and Stalker- Zealot. Each task set consists of distinct source tasks and unseen tasks. Specific details for each task set are presented each task set: Expert, Medium, Medium-Expee levels used for offline data collection, as detailed Appendix [.] Each group consists multiple tasks that share the same unit type but differ in the number of units.
Tables 4 and [ To evaluate generalization, we construct four groups in representing different policy
and
Cooperative Navigation The Cooperative Navigation (CN) task 22 is a widely adopted bench- mark for evaluating MARL algorithms. In CN, agents should collaboratively occupy some target landmarks. The environment consists of N agents and some landmarks a two-dimensional continu- ous space. Agents must coordinate their physical actions to reach the landmarks. The objective is for each agent to cover a distinct landmark while avoiding collisions. To assess the generalization
Table 1: The performance of different methods in Task set Marine-Hard. To simplify notation, asymmetric tasks are abbreviated (e.g., "5m6m" represents "5m vs 13_6mm)).
Expert
Medium
Tasks
UPDeT
ODIS
HiSSD
BIKT
UPDeT
ODIS
HiSSD
BiKT
Source Tasks
3m
82.8.11..0
98.4
99.5.10.3
99.810.2
51.2.134
85,9 62.7.1.77
86.1
5m6m
17.2.12..0
53.915.1
78.514.55
80.3
6.314.9
22.7.7.11
26.4...88
36.9
9m10m
3.115..4
80.418.7
95.5.12.7
99.410.4
28.5
78.1138
73.912.3
72.5
Unseen Tasks
4m
33.0.1.7.1
95.3
99.2.11.2
99.3
14.1::5.2
61.7
77.3
91.
5m
33.6:140.2
89.1110.0
99.2.11.2
99.910.1
67.2-121.3
85.9 88.41..44
9..415544
10m
54.7:14.4
93.8+2.2
98.4=0.8
99.410.1
32.9+11.3
61.3 98.010.3
96.3
12m
17.24:28.0
58.6211.8
75.5
99.010.2
3.243.88
35.918.1
86.416.0
92.514.0
7m8m
0.0:10.0
25.0=15.1
35.319.8
68.019.9
0.0:1..0
28.11:22.0
14.2+10.1
4.3.2.5
8m9m
0.0+0.00
19.6+6.00
47.0+6.2
50.0+6.2
2.3+22.6
47.0+-277
15.3+2.8
17.0+11.9
10m1 Im
0.0+0..0
42,4+7.2
86.3+14.6
90.6+-111
4.043.44
29.7+
43.6+4,6
25.6+4.66
10m 12m
0.0+0.00
1.6+1.66
14.549.1
14.6+33.5
0.0+0..0
1.6+1.6
0.6+00.5
2.2+1.3
13m15m
0.0+10.0
2.3+2.6
1.342.5
4.2+2.11
0.0+00.0
1.6116
1.4+2.4
2.5+133
Medium- Medium-Expert
Medium Medium-Replay
Source Tasks
3m
85.2.117.9
73.6122.0
86.6.3..7
99.4.11.3
41.4.10.11
83.6.114.0 78.815..3 78.7.6.3
5m6m
1.6.1..6
9.4.1.22
41.9.19.7
49.4.16.0
0.8.114
25.3110.3 28.113. 34.428.0
16.6.4.77
9m10m
24.3+18.7
31.3+14.5
83.6#6.9
58.3+22.4
0.8+1.4
45.8+3.99 47.6=7.0
Unseen Tasks
4m
43.94139.0
82.8113.5
91.116.1
98.1-10.2
35.9 $5.6661....11577.1
77.34
94.4
5m
33.6
82.8+17.7
98.3+1.8
98.8:0.11
61.7+20.3
96.1
88.1+134
97.910.2
10m
32.8+38.1
8288116.8
96.412.1
96.5+112
11.0=1.88
84.4115.1
94.712.6
96.7+1.5
12m
9.418.66
81.3120.6
88.4+11.8
95.6:10.3
2.3:2.6
84.416.6
90.313.6
92.7-2.6
7m8m
2.3+4.1
15.6+4.44
30.5+10.4
40.6+44.2
1.6++2.7
9,4+222
21.7+4.7
19.3+6.4
8m9m
10.2+4.6
9.5+
10.9+4.77
22.41-3.6
11.5++399
0.8+1.4
11.7+
10.5+4.0
10m1 Im
11.8+8.1
33.6++8.9
54.7+66.8
42.3+:3.2
0.8+1.4
35.9+5.22
42.5+444
43.8+66.6
10m12m
0.0.1..0
1.6.1.1.6
2.5
2.311.3
0.0.1..0
2.41144
0.510.3
0.6.10.2

---

Table 2: The performance of different methods in Task set CN.
Expert
Medium
Source Tasks
Unseen Tasks
Source Tasks
Unseen Tasks
CN-2
CN-4
CN-3
CN-5
CN-2
CN-4
CN-3
CN-5
UPDeT
90.6166.8
15,619.2
17.9
2.1::2.9
35.4
4.2:22.9
14.611.9
0.0:10.0
ODIS
100.0:10.0
13.2
85.617.6
20.0:1..8
65.0:154
28.7:16.7
43.8:5.2
8.1
HiSSD
96.4-12.8
49.2:17.2
89.815.2
25.3:12.99
59.3::5.2
24.2:43.9
44.9-29
5.142.9
BIKT
100.0:10.00
62.5:1..0
93.8::3.2
28.2:1..3
68.1:16.8
28.3:14.3
4..8:1522
7.144.7
Tettic
Tetti: Usage
0.25
Tral Im
Ro% 23.1
2632
0.29
020
0.2
0.5
000
Dalk
(a) Skills Marine-Hard
(b) Skills in Stalker Zenlot
(c) Tactic Freq. in Marine- Hard (d) Tactic Freq. in Stalker Zealot
Figure 4: (a) and (b) represent discovered individual skill embeddings from source tasks. Each color represents a different individual skill type, which can be reused in different tasks. (c) and (d) represent the frequency of tactic usage per task from source tasks. Each tactic ID corresponds to a distinct team tactic. Representative skill and tactic examples are shown in Figure 5
performance, the task set is denoted by CN and each task is denoted by CN-N, where N denotes the number of agents. Our training uses offline datasets provided by [4
5.2 Performance Comparisons
Baselines We compare BiKT against representative baselines. 1. UPDeT, which extends the
transformer-based UPDeT architecture by introducing a transformer- based Q-mixing network to enable effective multi-task policy learning. 2.ODIS [41], which leverages coordination skills extracted from offline multi-task data and learns to differentiate agent behaviors. 3. HiSSD (20). which employs
hierarchical framework that jointly learns shared and task-specific skills across multiple tasks. All methods are trained from source tasks and evaluated directly on unseen tasks.
Overall, our method outperforms baselines. The results for task set Marine-Hard and CN are shown in Table For paper limits, the results of Stalker-Zealot and Marine are detailed in the Appendix 10 In task set Marine Hard. our method achieves superior performance compared baselines, especially on the Expert and Medium- Expert datasets. In contrast, UPDeT simply relies on network design to enable multi-task learning, without explicitly considering policy transfer. This limits its ability to generalize beyond simple tasks like 3m. While ODIS performs well on source tasks, its success is limited to similar tasks(e.g., 3m, 4m, 5m...). It fails to transfer performance from sin_v__6m_ to more distinct unseen tasks such as fm_vs__mm 8m and 8m_vs_ vs_ vs_ especially under low-quality offline datasets. HiSSD learns to distinguish common and specific skills across, which implicitly serve as task-specific guidance information. In contrast, our method explicitly learns cooperative tactics from skill combinations, resulting in superior performance.
Ablation Study We detail them in the Appendix D.3 due to the paper limits.
5.3 Strengths of BiKT
Our method can discover common individual skills. Figure shows the t-SNE visualization of the learned individual skill embeddings for the source tasks of Marine-Hard and Stalker- Zealot. does not differentiate the skills from different tasks, which we include in Appendix D.2 Trajectories are collected using the trained BiKT policy. eexpainss several semantics of these skills, and more explanations for each skills are provided in the Appendix D.4] From the visualizations, we can conclude that the skills exhibit a clear clustering trend, which reflects a shared and reusable skill set Besides, the embeddings of the same skill type may vary across tasks for different action adaptation.

---

Individual Skills in Figure 4 (a), (b)
A1. Advance
B, Strong Agent Weak Enemy
Ag. Escape low HP
--- B2. Strong Agent draw attack
Ag. Keep firing one of enemies
B3. Weak Agent vs. Weak Enemy
Source Tasks
Unseen Tasks
Tactic Transfer
C
Cj.
Tactic 5 in Vigure (c): High HPP marine protect low HP marine.
A2
Aa
(a) 5m vs 6m
(b) 7m vs 8m
(c) 9m vs 10m
Tactic Transfer
B3
B
Taclic Codebook C
C
B
B
Tactic 2 in Figure (d): Stalker draws strong fire for asymmetric attack.
B2
(d) 2s3z
(e) 3s3z
(l) 384z
Figure 5: Examples of tactic transfer in different task sets. In case (a), there exists unbalanced health point (HP) among the agents, the high HP agent takes the skill A to advance, the low HP agents take skill Ag escape and other agents take skill A2 keep firing. It corresponds the tactic that high HP agent advances to draw firepower and protect the team, which is effectively transferred to tasks 7m_vs_ 8m and pii_vs__0mm In case (d), stalker takes the skill B2 draw attack from stalker enemies, the other stalker takes the skill B to attack the weaker zealot enemy, and the rest zealots take the skill B3 to attack enemies. Considering that the stalkers are more powerful than zealots, this forms the key tactic for winning, which is been learned to transfer to task 3s3z and 3s4z.
Our method can discover diverse cooperative tacties. We collect 32 trajectories of trained policies in different tasks and track the frequency of discovered team tactics. Figure 4 presents the usage distribution each tactic under the Marine- Hard and Stalker-Zealot task sets. We conclude that certain team tactics consistently utilized across all tasks. However, some tactics vary significantly in frequency depending on the task, often serving as key factors for success. team focus of tactic preferred in scenarios with numerical disadvantage (c.g., 5m_vs vs_6m), helping to quickly eliminate threats, while local team fire of tactic 4 suits simpler tasks like 3m. These results indicate that our method selects tacties adaptively, depending task-specific requirements.
Explicable example: Our method performs policy transfer by bi-level knowledge. Figure shows two examples illustrating how our method enables policy transfer by reusing learned tactics. In both cases, the policy leverages specific tactics learned from source tasks and successfully gener- alizes them to unseen tasks, facilitating coordinated team behavior. Notably, the team cooperative information in these two examples cannot be transferred by skills from the individual level. This demonstrates that agents leverage the same tactic to infer similar skill combination across tasks.
Conclusion
In this paper, we aim to tackle the problem of multi-task MARL generalization via offline data. Though existing works utilize skill-based methods to achieve knowledge transfer, we argue that transferring skills alone is not efficient. We think the efficient knowledge transfer in multi-task MARL includes two levels: the individual level that transfers skills and the team level that transfers cooperative tactics. To fulfill it, we introduce our BiKT method, which first discovers latent skill representations from offline trajectories using a VAE, then constructs a discrete tactie codebook via VQ-VAE, and finally learns a policy with Bi-Level Decision Transformer that select tactic id and executes skills sequentially. Experimental results demonstrate that BiKT significantly improves the zero-shot generalization performance of mutli-task MARL.

---

7 Acknowledgements
This work was supported in part by the National Key R&D Program of China (No. 20252D0122000, NSFC 62273347, the Key Research and Development Program of Jiangsu Province (BE2023016).
References
[1] Akshat Agarwal, Sumit Kumar, and Katia Sycara. Learning transferable cooperative behavior in multi-agent teams. arXiv preprint urXiv:1906.01200 2019.
[2] Yongcan Cao. Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics, 9(1):427-438 2012.
[3] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:15084-15097, 2021.
[4] Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. Multi-agent deep reinforcement learning for large-scale traffic signal control. JEEE transactions on intelligent transportation systems, 21(3):1086-109) 2019.
[5] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
[6] Claude Formanek, Asad Jeewa, Jonathan Shock, and Armu Pretorius. Off-the-grid marl: Datasets with baselines for offline multi-agent reinforcement learning. arXiv preprint 20X0022302.00521, 2023.
[7] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[8] Cong Guan, Feng Chen, Lei Yuan, Zongzhang Zhang, and Yang Yu. Efficient communication via self-supervised information aggregation for online and offline multiagent reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[9] Xiaotian Hao, Hangyu Mao, Weixun Wang, Yaodong Yang. Dong Li, Yan Zheng, Zhen Wang, and Jianye Hao. Breaking the curse dimensionality in multiagent state space: A unified agent permutation framework. arXiv preprint arXiv:2203.0522 055285, 2022.
[10] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent rein- forcement learning via policy decoupling with transformers. arXiv preprint arXiv:2101 100.0000,, 2021.
[11] Maximilian Hüttennaach, Adrian Šošić, and Gerhard Neumann. Deep reinforcement learning for swarm systems. Journal of Machine Learning Research, 20(54):1-311 2019.
[12] Shariq Iqbal, Christian Schroeder Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4596-4606. PMLR, 2021.
[13] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.00 09991, 2022.
[14] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 06:67195-67211 2023.
[15] Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
10

---

[16] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.01169, 2021.
[17] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1 11:1 1191, 2020.
[18] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[19] Jie Liu, Yinmin Zhang, Chuming Li, Chao Yang, Yaodong Yang, Yu Liu, and Wanli Ouyang. Masked pretraining for multi-agent decision making. 2023.
[20] Sicong Liu, Yang Shu, Chenjuan Guo, and Bin Yang. Learning generalizable skills from offline multi-task data for multi-agent cooperation. arXiv preprini arXiv:2503.21200 2025.
[21] Qian Long, Zihan Zhou, Abhibav Gupta, Fei Fang, Wu, and Xiaolong Wang. Evolution- ary population curriculum for scaling multi-agent reinforcement learning. arXiv preprins arXiv:2003. 2020.
[22] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb. OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperatieeccomptttiy enviromments. Advances in neural information processing systems, 30, 2017.
[23] Linghui Meng, Muning Wen, Chenyang Le, Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Yaodong Yang. et al. Offline tre-trained multi-agent decision transformer. Machine Intelligence Research, 20(2):233- 248 2023.
[24] Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE transactions on cybernetics, 50(9):382--388) 2020.
[25] Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized pomdps, 2015.
[26] Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: multi-agent reinforcement learning with actor rectification. In International conference machine learming, pages 17221-17237. PMLR, 2022.
Offline
[27] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models, arXiv preprint arXiv:2301.1067. 2023.
[28] Rongjun Qin, Feng Chen, Tonghan Wang, Lei Yuan, Xiaoran Wu, Yipeng Kang, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. Multi-agent policy transfer via task relationship modeling. Science China Information Sciences, 67(8):18211) 2024.
[29] Jorge Pena Queralta, Jussi Taipalmaa, Bilge Can Pullinen, Victor Kathan Sarker, Tuan Nguyen Gia, Hannu Tenhunen, Moncef Gabbouj. Jenni Raitoharju, and Tomi Westerlund. Collaborative multi-robot search and rescue: Planning, coordination, perception, and active vision. leee Access, 1-191617--91149 2020.
[30] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar. Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 1117)) 2020.
[31] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nan- tas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: : 1902.04043. 2019.
[32] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. arXiv preprint wtX.iv:1907.1666 2019.
11

---

[33] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv: 1006 70290, 2017.
[34] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[35] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020.
[36] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06199 2022.
[37] Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 16:99-99-1652,,, 2022.
[38] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: I.1 1361,, 2019.
[39] Xu Xu, Youwei Jia, Yan Xu, Zhao Xu, Songjian Chai, and Chun Sing Lai. A multi-agent rein- forcement learning-based data-driven method for home energy management. IEEE Transactions on Smart Grid, 11(4):201-3221 2020.
[40] Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-10312 2021.
[41] Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang. Discovering generalizable multi-agent coordination skills from multi-task offline data. In The Eleventh International Conference on Learning Representations, 2022.
[42] Junkai Zhang, Yifan Zhang, Xi Sheryl Zhang, Yifan Zang, and Jian Cheng. Intrinsic action tendency consistency for cooperative multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17600-17608, 2024.
[43] Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Başar. Finite-sample anal- ysis for decentralized batch multiagent reinforcement learning with networked agents. IEEE Transactions Automatic Control, 66 (6(12):5925-59440 2021.
[44] Zhi Zhang, Jiachen Yang, and Hongyuan Zha. Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization. arXiv preprint 10065119,, 2019.
[45] Zijie Zhao, Honglei Guo, Shengqian Chen, Kaixuan Xu, Bo Jiang, Yuanheng Zhu, and Dongbin Zhao. Empowering multi-robot cooperation via sequential world models. arXiv preprint arXiv:2509.13000 /3095, 2025.
[46] Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. international conference muchine learning, pages 27042-27059, PMLR, 2022.
[47] Tianze Zhou, Fubiao Zhang, Kun Shao, Kai Li, Wenhan Huang, Jun Luo, Weixun Wang, Yaodong Yang, Hangyu Mao, Bin Wang, et al. Cooperative multi-agent transfer learning with level-adaptive credit assignment. arXiv preprint arXiv:2106.0051 2021.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes] Our abstract and introduction accurately reflect the contribution
Justification: Please refer to and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes] We discussed our limitations
Justification: We provide them the Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA] Our paper does not include experiments.
13

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes] Our paper disclose the information for the experiments.
Justification: We provide them in Appendix C and our code is in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
We provide code in the supplementary material.
Justification:
We provide code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.cc/pulli///uiii wwwticccguie//eeeeeeeeeii CoodeSubeiiieeeec ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes] we specify the training details.
Justification:
We provide the experimental details in the Appendix C and B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes] We provide error bars in experiments.
Justification: Please refer to Experiments 5 and Appendix
D
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] We provide the computing resources for the experiments.
Justification: Please refer the Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: :tstps://neuur htt///nurrpppp
Answer: [Yes] We obey the NeurIPS Code of Ethics.
Justification: We obey the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA] Our method is applied to policy learning and transfer in MARL, whose main experimental scenario is game engine and does not contain social information and personal privacy, with no social impacts
Justification: [NA] Our method is applied to policy learning and transfer in MARL, whose main experimental scenario is game engine and does not contain social information and personal privacy, with no social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA] Our paper does not pose such risks.
Justification: Our paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The used assets are properly credited in our paper.
Justification: We use the SMAC experiment under the Apache 2.0 license and the MPE environment, which is released under the MIT license. For the offline dataset, we use from ODIS with the Apache License v2.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
The code is well documented in the supplemental materials.
Justification: The code is well documented in the supplemental materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA] Our paper does not involve crowdsoureing nor research with human subjects.
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA] Our paper does not involve crowdsourcing nor research with human subjects.
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Our method does not involve LLMs components.
Justification: Our method does not involve LLMs as components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
19

---

A PSEUDOCODE of our method
The pseudocode of our method is detailed in
Algorithm BiKT for Multi-Tann MARL
I:
Inputs:
2:
Nex The offline dataset Dee {Die J== Individual Skill Encoder Paain, Action Decoder cooperative tactic
encoder Plac skill decoder gaan, tactic codebook embedddings transformer TT J the number of source tasks Nue, the number of agents N, the tactic codebook number
[cs, ===1= [c=]]= the sksel-based decision
K. the content length of bi-level decision transformer L, learning rates 42,12,43
Training:
4: for each timestep l in 1..H do
5: Sample 6 (St, 7í, ac) Duc
# Individual Skill Learning
Use óž to compute Cain (ai) Eq. 3.
7: Calculate gradients to update painl and (eet with learning rate 1
8: end for
9: for each timestep t in 1..H do
Sample ç (st, ri. 2i y 24 Dav # Cooperative Tactic Codebook Learning
11: Use Çi to compute Lucie (62) in Eq. 4.
12: Calculate gradients to update Pae and gam. with learning rate lz
13: end for
each timestep 1..H
15: Sample S2;
(T ='c,, Ri) Dm
Bi-level Decision Transformer Learning
16: Use Ri to compute Epolicy (0) in Eq.
Calculate gradients to update paan and qea, with learning rate ls
end
19: Execution:
20:
for each timestep / in source task Tak do
21:
Given return-to-o {R: (N
22:
{ct) Td (ct Rize Rignde dess 2lee) {zi] 6 Té(zī Riger d'e cees =k)))
Select team tactic
23:
Take Individual skills
24:
{at t qealth, Thim 26 mi
Take Individual actions
25: end for
B Experiment Setting Details
B.1 SMAC
Environment Overview SMAC derived from real-time strategy game StarCraft II, focusing on micromanagement. Unlike typical StarCraft II games that involve both macromanagement (strategic planning) and micromanagement (fine control of units), SMAC is structured to emphasize decentralized control by requiring each unit to be managed by independent agent based solely local, limited observations. This setup necessitates multi-agents learning sophisticated cooperative behaviors under the challenge of partial observability. SMAC consists of diverse micro scenarios designed to assess how well agents coordinate to solve complex tasks. Each scenario involves two opposing armies with variations in initial positioning, unit types, and terrain features.
Observations, Actions and Team Goal. each timestep, agents gain local observations within their field of view, providing information such as distance, health, shields, and unit type of visible units, as well as terrain features. During centralized training, the global state includes comprehensive data on all units, including energy levels and attack cooldowns. Agents have discrete action set including movement, attacks, healing by Medivacs with certain constraints ensuring decentralization. The shooting range of units is limited compared to their sight range, necessitating strategie maneuvering. The allied units are controlled by agents trained to maximize the win rate against enemy units governed by the game's Al using scripted strategies.
Multi- Multi-Task Settings. assess multi-task generalization, we follow the ODIS setting with three task sets: Marine-Hard, Marine-Ear Easy, and Stalker-Hard Each task set consists of distinct training
20

---

Ageet
agent
adversary
(a) SMAC Agent Local Observation
(b) The Map 5m of SMAC
(c) Cooperatiave Navigation
Table 3: Descriptions of tasks in the Marine- Hard task set.
Task type
Task
Ally units
Enemy units
Properties
3m
Marines
Marines
homogeneous & symmetric
Source tasks
5m_vs_ 6m
5 Marines
6 Marines
homogemeous & asymmetric
9m_vs. vi_vs_10m
Marines
10 Marines
homogeneous & asymmetric
4m
4 Marines
4 Marines
homogencous & symmetric
5m
Marines
Marines
homogeneous & symmetric
10m
10 Marines
10 Marines
homogeneous & symmetric
12m
Marines
12 Marines
homogeneous & symmetric
Unseen tasks
7m_vs_8m
7 Marines
8 Marines
homogeneous & asymmetric homogeneous asymmetric homogeneous & asymmetric homogeneous & asymmetric
8m_vs_9m 9m
Marines
Marines
10m_vs_ 1mm
10 Marines
11 Marines
10m Lom_vs__
vs_ 12m
Marines
12 Marines
13m_vs_15mm
13 Marines
15 Marines
homogeneous & asymmetric
source tasks and testing scenarios. Specific details for each task set are presented in the Tables [5d The Marine-Hare and Marine-Eas sets comprise different marine battle scenarios where the learned multi-agent strategy must guide allied marines against enemy marines controlled by the game's AI, matching or exceeding in number. The Stalker-Zealot set consists of several challenges involving equal numbers of stalkers and zealots on either side.
Offline Dataset. In our experiments, we use the same dataset collected by ODIS for the fair comparison. Within each task set, the offline data is collected by pre-trained QMIX policies with different levels of performance: Expert, Medium, Medium Medium-Expert and Medium-Rephe policy trained with 2,000,000 environment steps. The Medium policy is trained until achieves approximately 50The Medium-Expers dataset is mixture of trajectories from both the Expert and Mediuu policies. The Medium-Replay dataset is obtained from the replay buffer of the Medium policy, which contains larger proportion of lower-quality trajectories. T summarizes the full settings of the offline datasets.
The Expert
B.2 Cooperaitve Navigation
To further evaluate our method, we consider a task set based on the Cooperative Navigation (CN) scenario, a representative cooperative task from the Multi-Agent Particle Environment (MPE). The environment consists of N agents and L landmarks situated in two-dimensional continuous space with discrete time steps. Agents must coordinate their physical actions to reach the L landmarks. Each agent observes the relative positions of other agents and landmarks, and the team receives a shared reward based on the proximity of any agent to each landmark that is, the goal is for all landmarks to be 'covered' by the team. Agents occupy physical space are penalized for collisions with one another, encouraging coordinated but non-overlapping behaviors. In this setting, agents must infer which landmark to cover and navigate there while avoiding others. The agents can execute discrete actions of moving towards four directions and "none" operation.
Environment Overview
21

---

Table 4: Descriptions of tasks in the Marine-Easy task set.
Task type
Task
Ally units
Enemy units
Properties
3m
3 Marines
Marines
homogeneous & symmetric
Source tasks
5m
5 Marines
Marines
homogeneous & symmetric
10m
10 Marines
10 Marines
homogeneous & symmetric
4m
Marines
Marines
homogeneous & symmetric
6m
6 Marines
Marines
homogeneous & symmetric
7m
7 Marines
Marines
homogeneous & symmetric
8m
8 Marines
Marines
homogeneous & symmetric
Unseen tasks
9m
9 Marines
Marines
homogeneous & symmetric
11m
11 Marines
11 Marines
homogeneous & symmetric
12m
12 Marines
12 Marines
homogeneous & symmetric
Table 5: Descriptions of tasks in the Stalker-Zealot task
Task type
Task
Ally units
Enemy units
Properties
Stalkers,
Stalkers,
2s3z
heterogeneous & symmetric
Zealots
3 Zealots
Stalkers,
Stalkers,
Source tasks
2s4z
heterogeneous & symmetric
4 Zealots
4 Zealots
Stalkers,
Stalkers, 5 Zealots
3s5z
heterogeneous & symmetric
Zealots
Stalkers,
Stalkers,
1s3z
heterogeneous & symmetric
Zealots
3 Zealots Stalkers, 4 Zealots
Stalkers,
1s4z
heterogeneous & symmetric
Zealots
Stalkers,
Stalkers, 5 Zealots
1s5z
heterogeneous & symmetric
Zealots
Stalkers,
Stalkers,
2s5z
heterogeneous & symmetric
Zealots
5 Zealots
Stalkers,
Stalkers,
Unseen tasks
3s3z
heterogeneous & symmetric
Zealots
3 Zealots Stalkers,
Stalkers,
3s4z
heterogeneous & symmetric
Zealots
4 Zealots
Stalkers,
Stalkers,
4s3z
heterogeneous & symmetric
Zealots
3 Zealots Stalkers, 4 Zealots Stalkers,
Stalkers,
4s4z
heterogeneous & symmetric
4 Zealots
Stalkers,
4s5z
heterogeneous & symmetric
Zealots
5 Zealots
Multi-Task settings. The task set of CN consists of different numbers of agents. Specifically, CN-n denotes a CN map containing n agents. Offline datasets are collected using the QMIX algorithm. Detailed dataset settings are summarized in Table 7
B.3 Computing Resources
For computing resources, we utilize the Intel(R) Xeon(R) Gold 5220 CPU and NVIDIA TITAN RTX GPU in the experiments. Each experiment in per task set lasts on average for 8 hours.
C Implementation Details
In this section, we will provide the model structure, the hyperparameters, and other training details of ODIS. We present each part of BiKT in the following sections.
22

---

Table 6: Properties of offline datasets in SMAC with different qualities.
Tasks
Quality
Trajectories
Average Return
Average Win Rate
expert
2000
19.8929
0.9910
medium
2000
13,9869
0,5402
3m
medium-expert
4000
16. 16.9399
0.7656
medium-replay
3603
N/A
N/A
expert
2000
19,9380
0.9937
medium
2000
17.3288
0.7411
5m
medium-expert
4000
18.6334
0.8674
medium-replay
711
N/A
N/A
expert
2000
19.9438
0,9922
medium
2000
16.6297
0.5413
10m
medium-expert
4000
18.2595
0.7626
medium-replay
571
N/A
N/A
expert
2000
17.3424
0.7185
medium
2000
12.6408
0.2751
5m_vs_6m
medium-expert
4000
14.9916
0,4968
medium-replay
32607
N/A
N/A
expert
2000
19.6140
0.9431
medium
2000
15,5049
0,4146
9m_ vs_ vs_10m
medium-expert
4000
17.5594
0.6789
medium-replay
13731
N/A
N/A
expert
2000
19.7655
0.9602
medium
2000
16.6279
0,4465
2s3z
medium-expert
4000
18.1967
0.7034
medium-replay
4505
N/A
N/A
expert
2000
19.7402
0.9509
medium
2000
16.8735
0,4965
2s4z
medium-expert
4000
18.3069
0.7237
medium-replay
6172
N/A
N/A
expert
2000
19,7850
0,9518
medium
2000
16.3126
0.31 14
3s5z
medium-expert
4000
18.0488
0.6316 16
medium-replay
11528
N/A
N/A
Table 7: Properties of offline datasets Cooperative Navigation with different qualitics.
Tasks
Quality
Trajectories
Average Return
Average Win Rate
expert
2000
1.0000
1.0000
CN-2
medium
2000
0.6152
0.6152
expert
2000
0.7173
0.7173
CN-4
medium
2000
0.4273
0.4273
C.1 Multi-Head Attention
We utilize Multi-Head Attention (MHA) to represent the embeddings of skills and tactics. This mechanism enables the model jointly attend to different representation subspaces, making effective for modeling contextual dependencies. Given query, key, and value matrices Q E IR K Sxd and V E Sdd the scaled dot-product attention computed as in Equation 6 Multi- head attention applies this operation across h heads, where each head has its own projection matrices w?? W.K, and W.V. The result of each head is shown in Equation 7 and the final output is formed
T xd
23

---

MLPP
B
Cancate
MAU Hoad Fotenta
alale decouple
MLLP
MLP
MLLP
ML
dbscrvation deccupling
MLP
Figure The detailed model structure of our individual skill learning.
by concatenating all heads and applying linear projection, as shown in Equation g
QK
Attention(Q K, V) softmax
V
(6)
VTl
head, = AAtentionnQWW, KWW VWY) MHA(Q, V) Concat(head head,) W
(7)
(8)
C.2 Details of Individual Skill Learning
The detailed architecture for individual skill learning is illustrated in Figure 6 During the skill encoding phase, each agent employs the shared encoder Paan ( 8, a, i) to compute its latent skill embedding 2' RNN, Specifically, the encoder takes as input the global state the joint action a, and the agent index i.
To construct the encoder input, we first concatenate the actions and entity features of the N allied agents. These concatenated representations, along with the remaining M - N entity features (e.g., enemies neutral units), are mapped into unified set of M embeddings. These embeddings are then passed through the Multi-Head Attention module of the Transformer. Finally, we take the N attention outputs corresponding to the allied agents and feed them into an MLP to generate the individual skill embeddings.
To reconstruct actions, each agent is expected to infer its action based on its trajectory history and individual skill embedding. To achieve this, we first extract entity features from the observation o², and process the history 7ं and entity features through separate multilayer perceptrons (MLPs). The resulting representations are then fed into MHA module capture relevant contextual dependencies. The MHA output is subsequently concatenated with the individual skill embedding 2 and passed through another of MLPs produce the action logits. These logits define the action distribution Past over N discrete action dimensions. Finally, the action is sampled from this distribution for execution.
C.3 Details of Cooperative Tactic Learning
The detailed architecture for the cooperative tactic codebook is illustrated in Figure 7 We construct the Cooperative Tactic Encoder Pime |,, {=*) + é RRN. the Skill Decoder qsan( r ..). 2t- and the tactic codebook ] The tactic embeddings codebook C randomly initialized.
{cx
The Tactic Encoder maps the global state and the individual skill embeddings [.) = into a continuous tactic representation & Specifically, each individual skill of is concatenated with the corresponding ally's entity features, forming N enriched ally representations. These, along with the remaining entity features, are separately embedded into vectors and then fed into the MHA module. The outputs averaged to produce the final tactic embedding, This tactic embedding is then discretized by searching the nearest neighbor in the tactic codebook C, resulting in the selected tactic C₁ E C.
To decode the individual skills, the entity features extracted from observations and agent i's trajectory history T are fed into the skill decoder 9aan- These inputs are first embedded separately using MLPs and then passed through the MHA module. The output embedding corresponding to agent and the individual skill representation & are passed through MLP generate the final decoded skill.

---

Conccite
Wull-Hee Attietion
Mi Head Amentt
sare
MI
dissovatioe decouste
Figure 7: The detailed model structure of our cooperative tactic codebook.
Subsequently, the Skill Decoder reconstructs each agent's latent skill 2t from its local trajectory history r' and the selected tactic Cş. This process enables the integration global cooperative strategies with decentralized agent behavior during execution.
During the tactic encoding phase, each agent employs the encoder Puc a, i) to compute its latent skill embedding z' E R. Specifically, the encoder takes as input the global state the joint action a, and the agent index i.
Stop-Gradient Operator. The stop-gradient operator, denoted as sgl-]. is used to block gradients during backpropagation while preserving values during the forward pass. Formally, for variable æ, the stop-gradient operation behaves as:
i S
Forward: sg
x,,
Backward:
dr
That is, in the forward pass, sg[æ] evaluates to the same value as x, but during the backward pass, no gradients are propagated through sg(z].
C.4 Hyper-parameters of BiKT
The hyperparameters of our method are detailed in Table g
Table 8: Hyperparameters our method.
Hyperparameter
Value
Individual skill dimension N
4
Tactic embedding No
64
The number of tactics in C: K
16
Hidden layer dimension of BDT
64
The multi-head number of BDT
2
The content length of BDT
10
Optimizer
Adam
Training steps for Laill
15000
Training steps for T-actic Training steps for Lepolicy
8000
30000
Batch size
32
learning rate 4
0.0004
learning rate l2
0.0001
learning rate la
0,0002
31
1
B2
0.01
0.05
25

---

Table 9: The performance of different methods Task Stalker Zealot.
Expert
Medium
Tisks
UPDeT-m
ODIS
IID
BIKT
UPDeT-m
ODIS
IGD
BIKT
Source Thcks
243r.
10.0::13.4
2..7222.
1.
97.922.33
350223.
49.2484
52.3
51.623.
2642
23.4
0092668
79.8
93.2=5.11
1882:00.3
328+-122
170::22
25.0:7744
$7.5229.6
90.8:1..0
930:2:55
25.6224.2
28.9168
Unseon Rasks
1s3u
1.61::1.6
76.6:13.55
81.6:115.22
77.0:442
38::5.00
41441188
4421999
32.41044
1652
29.7:1:26.4
2.5
16.7:1 12.3
8.7-14.3
5.0:1:4.2
14.118.1
25::22
265a
23.4::22.2
21.3:16.0
79.7:12222
13.3:13.2
1699:114.
32014466
1131337
24.2
39.14-5.2
38.0-1-43
96.4-146
24.4++28.6
2344992
21::::107
14
1721:45
4632
6.24444
64.1 13.0
88.6:2-4.1
96.0-:2.3
18.24:8.0
1334-75
3199:2522
18.7
4542
7..+-1355
79.7 10.9
73.4-4-.2
75.3-25.3
1.24115
12:4-700
1322-6.5
16.7:114
4552
5.5.1.88
86.7 12.6
63.6.13.7
61..1..88
5.6.1.5
7.014.11
451:13
8.3.144
4.74644
Mediom-Esport
Medium-Rophay
Packa
57.5:t-27.1
06.1-8.1
S1.3-2744
14.4+132
1564-182
90::1
30.2-8.44
53.1:124.6
41.42788
11.9::10.2
7.
60212
3450
35.0:523355
41.42 18.5
57.8 10.7
39.1
20.0::16 16.6
188431
175522
19.337.11
Unseen Tiska
1s3a
4.4:18.88
72.2-1:2..2 12.2
73.0:110.22
75.9:19.1
0.0:00.0
21.14120.4
36.3
30.199.1
1s82
11.9:198
44.5:1200 20.3
32.3:1:30.5
37,9
7:.:1:1000
6217777
24881991
26.31:22
165t
3.8:1.66
42.25:1.4
9.4:.9.5
14.42 19.4
11.91996
78166
44122
12552.7
3636
13.8:115.00
50.0:1113.
56.6:1225.6
$7.918.2
17.51:12.3
25.0:120.11
96133
27.6::.5
3641
43.11220.7
52.1:19.5
71.7119.7
75.6 13.3
15.62::1.2
19:51166
22.5::00
19421111
5532
23.8:121.0
17.241722
60.5:11.11
28.8:19.4
11.24:15.0
86.1149
110::104
10.4...11
20.3:688
37.319.4
63322
4652
11.94-1011
21.94222
17.0-14.11
24.3-15.
10.6::19.7
034114
084-8
4,4--355
5552
5.0:18.5
18.0:15.1
19.7:159
148::33.
69:11338
2314.1
2314411
35129
Table 10: The performance of different methods Task set Marine Easy
Tasks
UPDeT-m
ODIS
Export
Hi-SSD
BIKT
UPDeT-m
ODIS
Modium
Hi-SSD
BIKT
Source Ticks
3m
83.6 12.6
97.7-6-2.6
99.5-1-8.1
99,4- 1.3
60.2-429..9
57.8-1922
14781144
87.2--4777
74.8:1222.9
93.3-15522
99.9-1..0
99,9:10.0
67.81.99
828.132
8166100
74.21::99
53.0:4322.3
90.6:2:0
04.4:4:2.
96.9=:
41.7
63.34::6.11
74.5::1555
75.5::8.6
37.9:1:.6
79.7::.7.5
09.7:10033
99.7:10.1
73.8:122.7
8981177
88011
83.02::77
7m
44.24:13.2
16.9
09. 09.1:207
99.7:10.1
65.2425.2
96 16.11114
07.31233
$9.91:000
8m
$1.742262
80.9 14.4
99.8-10.3
99. 99.1=0.11
13.4
97.742266
93.825222
98.9:
99.9.10.0
9..920.0
87.51:2.3
75.21155
11.91118
53.6:1-22.4
09.2:10.8
99.3:11.0
64.74:311
62.02:11
68.21:477
12m
44.3:122.88
70.3:130.2
99.7.11.1
99.6:11.0
13.
41.4160
58.5
19.71:
Medium- Medium-Expert
Medlum-Replny
Source Tasks
48.411.368
39.8:19.7
90.9:1:3.9
91.3:14888
29.7.1 10.0
79.7.14.7
87.71:29
78.81:3.22
5m
64.19:17.9
83.721 16.0
79.4:1699
$5.3:15.9
6.2 10.8
31455
87.512299
88.51:166
10m
68.8.1:23.8
9388144
60.2:121.11
83.6:13.11
0.100000
001000
84.21:49
85.2:1:2.5
Unsron Tacks
4m
43.7::25.0
$7.84 18.8
90.9-99.1
72.2
25.0-4-22.6
25.04:-.4
71.64-4.1
77.2
47.74430.00
7600260
70.6-6611
78:2::14
00:00000
31455
09.
57.84132.9
8m
40.6-1
11.
71.:199.5
68.1246
00:000
16111
06.7.0
87.611177
47.74424.88
73.42 16.2
10.11:4466
7082666
00:000
a0400
1.3
26.6222
85.9::143
06.0-:20.3 20.3
70.9-1..99
15.5::12.4
00:00000
00400
45.61-455
52.8-133.6
46.-1:1555
63.1:2.0
62.7.1788
90.7:99.8
00:00
3801337
41.522..3
26

---

10m
(a) Skills in Marine-Hard
(b) Skills in Stalker Zealot
Figure 8: The individual skill embeddings.
D Additional Results
D.1 The performance comparison of other task sets.
The results for task Stalker Zealot and Marine Easy are shown Table 1 The Stalker Zealot requires different tactics in different tasks, which brings big challenge in policy transfer. The results show that our BiKT overall outperforms other baselines in both task sets. However, in Marine Easy, the tactics required for each task are similar, resulting that all methods can achieve high performance. makes that Hi-SSD and BiKT can both obtain high performance in Expert setting.
D.2 Visualization of individual skills from source tasks
We additionally show the individual skill embeddings from the task labels, Figure
D.3
Ablation study
We conduct ablation experiments on task set Marine Hard to evaluate different variants of our method, and the results shown in [
MADT W_T___OOF W_OD: To evaluate the impact of tactic and skill learning, we remove them and let the decision transformer learns to take action directly, which naturally degrades into the MADT method with observation decoupling, denoted by MADT_w_Ol For fairness, we utilize the same hyperparameter in MADT w OD. For convenience we also provide the results of MADT.
L 5: We set the context length of skill-based Decision Transformer æ, 5.
Ck=32: During the team tactic learning process, we set the tactic number of codebook as 32.
w/o C: We overpass the learning process team tactic and directly let the skill-based decision transformer to learn the individual skills. It is achieved by removing the of embedding token in Figure individual skills and then takes its action.
this time, the SDT policy learns to directly output the
Con-Tac: Compared with continuous individual skill embeddings, we utilize fixed number of tactics. For that we employ a VAE to learn team-level tactics and use Continuous Tactic (Con-Tace embeddings instead of discrete tactic set.
Ablation study: The action based policy struggles generalize different tasks. The results of MADT w_ OD in Table IT indicate that our proposed tactic and skill learning components play crucial role in the overall performance. Using an action- based policy introduces the challenge that the agent must take different actions under similar observations across diverse tasks, which cannot be addressed effectively without additional guiding information. As a result, MADT performs well only in tasks with similar numbers of agents and comparable problem settings, such as 3m, 4m, and 5m. 3_D leads to performance improvements, not
Although the observation decoupling MADT_W OD the primary contributing factor to generalization.
27

---

Ablation Study: Individual Skills Alone Cannot Transfer Diverse Team-Level Knowledge The results of w/o C in SToo that without the guidance of team tactics, our method's performance drops. This is because the policy must execute different skills without any external team-level information. In this setting, agents tend to learn a fixed combination of skills that only adapts well to limited set of tasks, such as 3m, 4m, and 5m. However, this approach fails to capture diverse team tactics required for more complex tasks like im_vs_8m 8m and 8m vs 9m, leading to a noticeable decline in policy transfer performance.
Diserete Tactic Codebook Outperforms Continuous Tactic Embeddings The results for Con-Tac in Table TT indicate that using continuous tactic embeddings can improve performance on some tasks. However, it still falls short of the results achieved with a discrete tactic codebook (BikT). We argue that each tactic should correspond to a clear, meaningful, and reusable coordination pattern. Different tasks often share common tactics, which provide stable team-level guidance and assist agents in selecting appropriate individual skills across varied scenarios. Continuous tactic embeddings tend to blur this clarity, thereby weakening the effectiveness of team strategies.
Ablation Study:
Ablation Study: Impact of Content Length and Tactic Codebook Size The results with a content length L = 5 for the BDT model demonstrate that our method's success does not heavily depend on a complex Transformer architecture. Instead, the key factor is the way we incorporate bi-level knowledge transfer in multi-task MARL, which proves to be highly effective. Regarding the tactic codebook size, the results with C = 32 show that our tactic learning process converges to meaningful and useful tactic embeddings. This indicates that moderately sized codebook is sufficient to achieve both efficiency and performance.
Table 11: The results of ablation study in task set Marine Hard
Source Tasks
Unseen Tasks
3m
5m_vs_6m
Dm_vs_10m 10m
4m
5m
10m
MADT
88.54339
3.140.0
1.0=1.5
83.3
75.0:16.8
1.0::1.5
MADT_w_O W_OD
90.2+22.8
10.2+3.5
16.2+6.8
88.4+22.3
83.2:22.7
12.7+:4.2
L-5
100.0:10.0
78.9+:3.5
98.4+1..5
99.3:10.9
99.9110.8
97.2+:1.5
CR=32
100.0+:0.0
80.9+33.2
99.2++0.3
99.2-10.1
99.3++0.2
99.3+00.2
w/o C'
98.2110.2
68.2115.3
83.2+:2.5
90.2:1..9
88.61:2.7
86.2433.
Con-Tac
99.24-0.4
64.3++6.7
80.3+44.4
92.2+:2.4
90.6+-4.6
88.34-4.2
BiKT
100.0110.0
81.3.1..5
99.4.10.4
99.3.10.1
100.0110.0
99.4.10.1
Unseen Tasks
12m
7m_vs_ vs_
8m_vs_ vm
10m_vs vs _s_lim 11m
10m_vs sin_v___200 1_m
13m_vs s___v__1__0 15_15m
MADT
0.0:10.0
0.0+0..0
0.0=0.0
0.0=0.0
0.0:1.00
0.0+0.0
MADT W_O OD
8.2+3.2
8.41:3.9
0.0=0.0
0.0=0.0
0.0:0.0
0.0:0.00
I.
98.74-0.9
64.8+-8.2
48.2+8.3
90.2+-2.3
12.0+11.8
3.2+1.5
CK=32
99.0:10.2
70.248.0
46.217.4
90,211211
12,34222
3.3
w/o C
57.2144.2
23.3:15.5
20.3-19.22
40.7-19.2
1.611.6
1.611.3
Con-Tac Tac
60.5+33.8
18.247.4
17.8+55.2
38.7+28.3
0.9:0..
0.5+0.4
BiКT
99.0:10.2
68.0:19.9
50.016.2
90.6::1.1
14.6::1.5
4,242-11
D.4 Semantic of Individual skills and tactics
We provide more examples for our learned skills and tactics, as shown in Figure 9 10 and 12
E
Discissions
We adopt different strategies for learning tactic embeddings and skill embeddings, each with distinct focus. For tactic embeddings. which serve as a shared team-level guiding signal for all agents, we represent them as fixed or discrete values. This design enables all agents to access the same team guidance, facilitating coordinated cooperation. Additionally, employ VQ- IQ-VAE to learn
28

---

2
A3
A4
A1. Advance
Ag. Escape on low HP
Ag. Keep firing on one of enemies
Aq. Keep away from neighbor
A1
A4
(a) Individual skills
(b) Examples of 1,, A2, A3
(c) Examples of A4
Figure 9: The semantics of individual skills in Marine-Hare
B
Strong, Weak
By. Strong Agent
B₁. Weak Agent vs. Weak Enemy
De. Advance
Weak Strong Enemy
B2
B
(a) Individual skills
(b) Examples of B,, B2, B3
(c) Exumples of Bg
(c) Examples of By
Figure 10: The semantics of individual skills in Stalker-Zealot.
compact set of tactic embeddings across multiple MARL tasks, allowing the discovery of a limited set of reusable team strategies. contrast, using a standard VAE for tactic learning would result in unique team guidance vector for each trajectory in every task, introducing higher randomness and instability when generalizing to unseen tasks. For skill embeddings, we use VAE to preserve the diversity and distributional structure of the skill embedding space. The agents' action semantics vary across environments and agent numbers. Even when two agents different scenarios exhibit similar behaviors skills, still expect their skill embeddings to differ. Using VAE allows for these embeddings to be mapped to actions with different physical meanings in different environment maps.
F Limitations
While our proposed method demonstrates strong generalization across the evaluated tasks, it remains an open question whether can consistently maintain performance when scaled to highly diverse and large-scale task distributions. Exploring more expressive or adaptive embedding mechanisms could be a promising direction for future work.
29

---

Tactic 2 in Marine-Hann Harine Team focus Fire
Tactic 4 in Marine-Hard: Local Team Fire
A3
A3
(a) 5m vs 6m
(a) 5m
Figure 11: The semantics of some tactics. The tactic ids correspond to Figure 4, which are learned from the offline trajectories. (a), agents are learned to take all fire to single enemy target, which quickly eliminate enemy and make up for the disadvantage in agent numbers. This tactic is more aggressive, and the win rate is not stable. In (b), the agents are learned to attack their enemy targets locally, without considering the disadvantage in agent numbers. provides rate, but it falls in 5m vs vs_ 6m.
stable win
Tactic 2: The team spreads out to disperse enemy fire
Tactic 3: Outnumbered attack: Many Few
B
(a) 9m vs 10m
(b) 255z
The semantics of some tactics. The tactic ids correspond to Figure 4. In (a), agents are learned to take skill A, to keep away from neighbors, which forms the team tactic that the team spreads out to disperse enemy fire. In (b), the agents learn to take skills B, and Bg, allowing the stronger agent attack weaker enemy, while the weaker agent also targets weak enemy. This tactic leverages numerical superiority to quickly eliminate weaker opponents.
Figure 12:
30


Reviews:
[{"review_id": "zSW2r7jwuD", "reviewer": "Reviewer_9d35", "summary": "This paper introduces BiKT, a novel approach for knowledge transfer across tasks for MARL at both individual and team level. The algorithm results in better performances on SMAC and CN compared to baselines, indicating the effectiveness of the method. Visualization and ablation studies help with better understanding what the neural network is actually learning.", "questions": "See Weaknesses.", "limitations": "yes", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed approach shows improvements on well-established benchmarks, outperforming the baseline methods by a large margin on some tasks. Especially on unseen tasks.\n\n2. The visualization of the learned skills help with comprehensive understanding of the method.\n\n3. Extensive experimental results and analyses make the paper really solid.\n\nWeaknesses:\n\n1. If the learned skills can be clustered into a number of groups, can we also consider a codebook for learning it? i.e., use VQ-VAE as well for skill learning.\n\n2. The intuition behind using codebook for team-level tactics and no codebook for skills is unclear. Have the authors tried different combinations of network formulation as an ablation study?\n\n3. In Table 1, for some tasks like 10m11m in Medium-Expert, the proposed method significantly underperforms the baseline method, can the author provide some clarifications for this?", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns.", "rebuttal": "Thank you very much for your recognition of our paper's methodologies, empirical results, paper writing, and the valuable feedback you provided. In response to your comments, we would like to make the following clarifications and feedback. We hope our explanations and analyses can eliminate concerns and make you find our work stronger.\n\n> Weakness1: If the learned skills can be clustered into a number of groups, can we also consider a codebook for learning it? i.e., use VQ-VAE as well for skill learning. \n>\n> Weakness2: The intuition behind using codebook for team-level tactics and no codebook for skills is unclear. Have the authors tried different combinations of network formulation as an ablation study?\n\nWe adopt different strategies for learning tactic embeddings and skill embeddings, each with a distinct focus.\n\nFor tactic embeddings, which serve as a shared team-level guiding signal for all agents, we represent them as fixed or discrete values. This design enables all agents to access the same team guidance, facilitating coordinated cooperation. Additionally, we employ VQ-VAE to learn a compact set of tactic embeddings across multiple MARL tasks, allowing the discovery of a limited set of reusable team strategies. In contrast, using a standard VAE for tactic learning would result in a unique team guidance vector for each trajectory in every task, introducing higher randomness and instability when generalizing to unseen tasks.\n\nFor skill embeddings, we use a VAE to preserve the diversity and distributional structure of the skill embedding space. The agents' action semantics vary across environments and agent numbers. Even when two agents in different scenarios exhibit similar behaviors or skills, we still expect their skill embeddings to differ. Using VAE allows for these embeddings to be mapped to actions with different physical meanings in different environment maps.\n\n**In summary:**\n\n- **Why not use VQ-VAE for skill embeddings?**\n   While skill embeddings could be clustered, we avoid using VQ-VAE to retain their diversity and the rich distributional properties of the embedding space, which facilitates skill embeddings mapping to different actions to suit environments, thereby improving multi-task performance.\n- **Why not use VAE for tactic embeddings?**\n   To promote cooperation, we prefer agents to share a fixed team guidance. VQ-VAE helps extract a compact set of stable tactics, reducing variability and instability in the generalization process.\n\nIn addition, we conducted the following ablation studies for validation:\n\n* **A1**: Replace the VAE with a VQ-VAE for skill embedding learning, and learn a skill embedding codebook with size 32.\n\n* **A2**: Use a VAE instead of a discrete embedding for tactic learning, followed by clustering into 16 categories. **We have conducted it in the Appendix**, lines 879-881. And the results are shown in Table 11. We list them here for convenience.\n\n* **A1_A2**: Apply both A1 and A2 simultaneously.\n\n| Different RTG | source-task 3m  | source-task 5m6m | source-task 9m10m | unseen-task 4m | unseen-task 5m  | unseen-task 10m | unseen-task 12m | unseen-task 7m8m | unseen-task 8m9m | unseen-task 10m11m | unseen-task 10m12m | unseen-task 13m15m |\n| ------------- | --------------- | ---------------- | ----------------- | -------------- | --------------- | --------------- | --------------- | ---------------- | ---------------- | ------------------ | ------------------ | ------------------ |\n| A1            | $90.6 \\pm 4.4$  | $43.6 \\pm 15.6$  | $53.1 \\pm 15.3$   | $93.8 \\pm 2.2$ | $88.4 \\pm 8.1$  | $60.3 \\pm 10.3$ | $53.1 \\pm 9.7$  | $13.5 \\pm 12.8$  | $9.8 \\pm 6.5$    | $28.1 \\pm 7.2$     | $0 \\pm 0$          | $0 \\pm 0$          |\n| A2            | $99.2 \\pm 0.4$  | $64.3 \\pm 6.7$   | $80.3 \\pm 4.4$    | $92.2 \\pm 2.4$ | $90.6 \\pm 4.6$  | $88.3\\pm 4.2$   | $60.5 \\pm 3.8$  | $18.2 \\pm 7.4$   | $17.8 \\pm 5.2$   | $38.7 \\pm 8.3$     | $0.9 \\pm 0.5$      | $0.5 \\pm0.4$       |\n| A1_A2         | $87.5 \\pm 5.4$  | $33.8 \\pm 16.0$  | $51.9  \\pm 17.9$  | $90.6 \\pm 6.2$ | $84.4 \\pm 9.7$  | $59.4 \\pm 22.0$ | $43.8 \\pm 11.8$ | $6.8 \\pm 6.2$    | $6.25 \\pm 3.7$   | $25.0 \\pm 4.4$     | $0 \\pm 0$          | $0 \\pm 0$          |\n| BiKT          | $100.0 \\pm 0.0$ | $81.3 \\pm 4.5$   | $99.4 \\pm 0.4$    | $99.3 \\pm 0.1$ | $100.0 \\pm 0.0$ | $99.4 \\pm 0.1$  | $99.0 \\pm 0.2$  | $68.0 \\pm 9.9$   | $50.0 \\pm 6.2$   | $90.6 \\pm 1.1$     | $14.6 \\pm 1.5$     | $4.2 \\pm 2.1$      |\n\nFrom the above experimental results, we observe the following:\n\n* Learning skill embeddings as a codebook (A1) limits policy generalization, as it cannot adapt to action mappings with varying dimensions across different maps.\n\n* Using a VAE followed by clustering for tactic embeddings (A2) reduces stability during generalization.\n\n* When both A1 and A2 are applied simultaneously, overall performance degrades significantly.\n\nThank you for pointing this out to us.  We believe that discussing the separate learning of skills and tactics is important, as it helps clarify the details of our bi-level knowledge framework. We will include this discussion in the revision.\n\n> Q1: In Table 1, for some tasks like 10m11m in Medium-Expert, the proposed method significantly underperforms the baseline method, can the author provide some clarifications for this?\n\nThank you for pointing this out. We attribute the performance gap to a combination of two factors:\n\n* **Unbalanced quality of task trajectories.**\n  In non-expert datasets, different source tasks exhibit varying winning rates. For example, in the *Medium* offline dataset, the winning rate for source tasks *3m*, *5m_vs_6m*, and *9m_vs_10m* are 54.02%, 27.51% and 41.46%.  This imbalance introduces biases in the policy quality learned from different tasks, which in turn affects generalization performance on unseen tasks. The unbalanced quality of task trajectories is a common challenge faced by all methods. We consider it a shared issue— as the quality of the offline dataset declines, each method tends to develop biases toward certain tasks during learning, which in turn leads to increased instability in performance across tasks.\n\n* **Accumulated sub-optimality in the bi-level learning structure.**\n  The individual skill learning depends on the quality of the offline dataset, while tactic learning further relies on both the learned individual skills and the dataset quality. When the offline data quality drops from expert level, the individual skill learning captures suboptimal action mappings, which then propagate to the tactic learning stage, resulting in suboptimal team-level strategies. Such accumulation of sub-optimal strategies biases the learning process, preventing some tasks from achieving expert-level performance.\n\nHowever, experimental results of Table 1 show that when the quality of trajectories in the offline dataset degrades, our method maintains the highest level of stability, with the smallest performance drop. This indicates that despite the sub-optimal trajectories introduced by lower-quality offline data, our Bi-Level Knowledge strategy is able to preserve robust and stable policy learning.\n\nAll your comments have greatly contributed to the improvement of our paper. If you have any new comments, please feel free to provide them, and we will promptly address them. If you find that we have addressed your concerns, we hope you will reconsider your rating."}, {"review_id": "JKG9Y1o5aU", "reviewer": "Reviewer_Pgpu", "summary": "This paper provides a method for skill level and tactic level transfer in the Multi-agent Multi-task Reinforcement Learning setting. Skills are low-level agent-specific abilities that are combined in tactics which are employed by multiple agents in a cooperative setting. The proposed methods incorporates 3 levels of learning to learn the key components, which include the action decoder, tactic codebook, and (Bi-level) Decision Transformer. Experiments are carried out on StarCraft 2 Micromanagement and Cooperative Navigation environments against several relevant baselines.", "questions": "# Questions\n- Line 104: The subscript $t$ is missing from the policy definition. Is this on purpose? I.e., shouldn't it be $\\pi^i(a^i_t | \\tau^i_t)$?\n- Line 114: What is the policy pre-trained on and why is this policy pre-trained. Why not use a random policy?\n- Figure 2: What do you mean by \"feature of entity $i$\", what exactly is the feature?\n- Section 4.1: Am I correct in understanding that the individual skills that are learned are from trajectories that are generated from policies that already solve a particular task? If so, what is the reasoning for this?\n- What is the stop-gradient operator used for in Equation 4?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- I think that the idea is an interesting one. Furthermore, the diagram were good. \n- The experimentation seems good with a wide variety of baselines being compared to.\n- I appreciate the ablation studies. I understand that page limits caused them to be moved to the Appendix. Could a summary of what was carried out and the key outcomes be added to the main text however?\n- I also appreciate that computing resources used are listed in Appendix B.3. However, I would be interested in knowing how long (assuming that the same compute resources) each of the methods in Table 1 and 2 took. \n## Weaknesses\n- Please provide citations for SMAC and MPE from the beginning of the paper rather than starting from Section 5.\n- There are some grammatical and notation errors and inconsistencies. This generally made the paper difficult to read. Please see below. \n#### Notation and writing style\n- Section 3.1: I think the authors should be careful about what are spaces and elements of the spaces. E.g., in line 97, I think $s$ and $o$ are the state *space* and observation *space* respectively and so should be upper-case and defined as such (in line 98). \n- Line 97, line 115, line 163: Please be consistent with whether you use \"$\\langle\\rangle$\", \"$()$\" or $[]$ for defining a tuple. I would prefer using $()$. \n- Line 104: The notion for the policy doesn't follow the appropriate notation for a function signature. It is correctly written for the transition function and the reward function. Either use a similar notation, or writing something like $\\pi^i(a^i | \\tau^i) \\in [0, 1]$.\n- Line 131 \"... to as individual skills $z$; and (2) The ...\": I think replace the semi-colon with a comma and make \"The\" lowercase (to \"the\"). The same should be done on line 139 and line 140.\n- Line 137 \"In specific...\": I think remove \"In specific\" or replace with \"Specifically\".\n- Line 150 \"...$\\mathbb{R}^{N_s}$ is the dimension of the skill embedding\": Did you mean that \"$N_s$ is the dimension of the skill embedding\"?\n- Line 160 \"While single-agent...\": Did you mean \"While *a* single agent...\"?\n- Line 150: What is $\\hat{a}$ and $\\hat{z}$? \n- Equation 3, 4: The input parameters $\\phi_1$, etc don't appear on the right-hand sides of the equations.\n- Line 163: I think $[z^1 \\times z^2 \\times ... \\times z^N]$ should be $(z^1, z^2, ..., z^N) \\in \\mathbb{R}^{N_s} \\times \\mathbb{R}^{N_s} \\times ... \\times \\mathbb{R}^{N_s}$. As mentioned before, please use *spaces* and *elements of spaces* correctly. \n### Suggestions:\n- Section 3.1 and 3.2: I think it might be worth highlighting that the superscript $i$ refers to different agents whereas the superscript $n$ refers to different source training tasks.", "was_revised": true, "final_justification": "I think that the authors have addressed the questions and concerns that I had, and have also used the suggestions I had made for clarity, etc. Thus I have increased my score to 5.", "rebuttal": "Thank you sincerely for your recognition of our methodology, experimental results, and writing, as well as for your thoughtful and constructive feedback. In response to your comments, we would like to offer the following clarifications. We hope these explanations help address your concerns and further highlight the strengths of our work.\n\n## Main Questions\n\n> Strength 4: I also appreciate that computing resources used are listed in Appendix B.3. However, I would be interested in knowing how long (assuming that the same compute resources) each of the methods in Table 1 and 2 took.\n\n  Under our computing resource, the whole training time for each method is as follows:\n\n|                                | UPDeT | ODIS | HiSSD | BiKT |\n| ------------------------------ | ----- | ---- | ----- | ---- |\n| Training Time (hours) per task | 7     | 12   | 15    | 8    |\n\n> W9: What is $\\hat{a}$ and $\\hat{z}$?\n\n  We apologize for not clearly explaining this in the paper. $\\hat{a}$ refers to the output of the Action Decoder $q_{act}$, and $\\hat{z}$ refers to the output of the Skill Decoder $q_{skill}$.\n  We use $\\hat{a}$ and $\\hat{z}$ to distinguish the decoded outputs from the original action $a$ and the learned skill embedding $z$ in the trajectory.\n> Q2: Line 114: What is the policy pre-trained on and why is this policy pre-trained. Why not use a random policy?\n>\n> Q4: Section 4.1: Am I correct in understanding that the individual skills that are learned are from trajectories...\n\n* Why not use a random policy?\n\n  Offline reinforcement learning (Offline RL) refers to the paradigm where agents learn solely from a fixed dataset of past interactions without interacting with the environment [1]. \n\n  As noted in [2, 3], offline RL critically depends on datasets containing expert-level trajectories. In contrast, datasets collected using random policies are typically of low quality, often containing a large number of uninformative or low-reward actions. It leads to a fundamental challenge: the agent is trained on suboptimal behavior data but expected to perform expert-level decisions at test time, which often results in severe performance degradation [2].\n\n* Why is the policy pre-trained? Why use a pretrained policy to collect trajectories?\n\n    In fact, the trajectories in the offline dataset exhibit expert-level behavior, aiming for the agent to learn an optimal policy [2]. Such offline RL datasets can be collected either via human demonstrators,  rule-based policies, or through pre-trained methods [3, 4], like QMIX in MARL.\n\n  It is worth noting that QMIX requires separate online training and trajectory collection for each individual task. Due to the differences in state and action spaces across tasks, QMIX cannot be directly applied to multiple tasks, nor can it support offline learning.\n* What is the policy pre-trained on？\n\n  For a fair comparison, all methods are evaluated using the same offline datasets, which are provided by ODIS [5] and collected using a pre-trained QMIX policy. Specifically, each offline dataset of tasks contains trajectories of varying quality levels: *Expert*, *Medium*, *Medium-Expert*, and *Medium-Replay*, as detailed in Table 6 and Table 7. These datasets are collected by controlling the win rate thresholds of QMIX.\n\n  * The *Expert* policy is trained for 2,000,000 environment steps.\n  *  The *Medium* policy is trained until it reaches approximately a 50% win rate.\n  * The *Medium-Expert* dataset is a combination of trajectories from both *Expert* and *Medium* policies.\n  * The *Medium-Replay* dataset is collected from the replay buffer of the *Medium* policy, which contains a higher proportion of low-quality trajectories.\n\n  We have considered expert trajectories of varying quality across different datasets and conducted experiments to validate the effectiveness of our method.\n\n> Q3: Figure 2: What do you mean by \"feature of entity $i$ \"...\n\nThe entity feature is a concept distinct from observation, and can be seen as a subset of it. The feature of entity $i$ represents the information of agent $i$ (e.g., position, health), while the observation of agent $i$ refers to the information of multiple entities within its observation range. In contrast, the state contains the complete set of entity features for all entities.\n\nThe dimensionality of an agent's observation can vary across different tasks, but the dimensionality of entity features remains fixed. So we use entity features as the input to the encoder to better support multi-task learning.\n\n> Q5: What is the stop-gradient operator used for in Equation 4?\n\nWe use the default stop-gradient operator as the original VQ-VAE method [6], which is essential for enabling end-to-end training despite the non-differentiable quantization step [6]. The outputs of *Cooperative Tactic Encoder* $p_{tac}$ encoder $\\hat{c}_t$ are quantized to the nearest embedding vector $c^k$ from the Tactic codebook $C$. However, the quantization step is non-differentiable, so to enable end-to-end training, the stop gradient is set as in Equation 9. That is, in the forward pass, it evaluates to the same value, but during the backward pass, no gradients are propagated.\n\n## Notation and writing style\n As you mentioned, there is still room for improvement in terms of notation and writing.  We apologize for any inconvenience for reading and appreciate your advice.  Your comments are relevant and helpful. In the revised version, we will refine the logical flow and improve the clarity of mathematical notation. Below, we address your concerns one by one.\n> Strengths 3: I appreciate the ablation studies. ...  Could a summary of what was carried out and the key outcomes be added to the main text however?\n\nWe agree with this point. In the revision, we plan to reorganize the manuscript and add a summary of the ablation study around Section 5.2, line 260.\n> W1: Please provide citations for SMAC and MPE from the beginning of the paper...\n\nIn the revision, we will provide citations for SMAC and MPE at line 64.\n> W2:  Section 3.1: I think the authors should be careful about what are spaces and elements of the spaces; \n>\n> W10: I think$[ z^1 \\times z^2 \\times ... \\times z^N]$ should be... \n\nIn the revision, we will clarify the distinction between spaces and the elements of spaces. Specifically, we will change *s* and *o* to uppercase in line 97. In line 163, in fact, the skill combination is the Cartesian product of skills $(z^1 , z^2 , ...)$. For the sake of standardization, we will moidfy $[ z^1 \\times z^2 \\times ... \\times z^N]$ as $(z^1 , z^2 , ... , z^N) \\in \\mathbb{R}^{N_s} \\times \\mathbb{R}^{N_s} \\times ... \\times \\mathbb{R}^{N_s}$\n> W3: Line 97, line115,  line 163: Please be consistent with whether you use \"$<>$\", \"$()$\" or$[]$ for defining a tuple. \n\nIn the revision, we will uniformly use \"()\" for defining a tuple.\n\n> W4: The notion for the policy doesn't follow the appropriate notation for a function signature....; Q1：Line 104: The subscript is missing from the policy definition. Is this on purpose?..\n\nIn the revision, we will modify $\\pi^i(a^i | \\tau^i) \\rightarrow [0, 1]$ in line 104 as $\\pi^i(a^i_t | \\tau^i_t) \\in [0, 1]$.\n\n> W5:  ...I think replace the semi-colon with a comma and make \"The\" lowercase (to \"the\")...\n\nIn the revision, we will modify \"The\" to \"the\" in lines 139 and 140.\n\n> W6: Line 137 \"In specific...\": I think remove \"In specific\" or replace with \"Specifically\"\n\nWe agree that removing the “In specific...” enhances readability. In the revision, we will modify it.\n\n> W7: Line 150 , ... Did you mean that $N_s$ is the dimension of the skill embedding\"?\n\nYes, the $N_s$ is the dimension of the skill embedding; we will modify it in the revision.\n\n> W8: Line 160 \"While single-agent...\": Did you mean \"While *a* single agent...\"?\n\nYes, for standardization, we will modify it as \"a single agent\" in the revision.\n\n> Suggestion 1: Section 3.1 and 3.2: I think it might be worth highlighting that the superscript $i$ refers to different agents whereas the superscript $n$ refers to different source training tasks.\n\nThank you for pointing this out. In fact, we have also been considering how to help readers quickly distinguish between the symbols for agents and tasks. Placing the clarification here indeed makes the distinction clearer from the beginning. We will emphasize the difference in notation between agents and tasks at the end of Section 3.2 to improve the clarity of the paper.\n\n[1] Levine S, Kumar A, Tucker G, et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems[J]. arXiv preprint arXiv:2005.01643, 2020.\n\n[2] Fujimoto S, Meger D, Precup D. Off-policy deep reinforcement learning without exploration[C]//International conference on machine learning. PMLR, 2019: 2052-2062.\n\n[3] Agarwal R, Schwarzer M, Castro P S, et al. Deep reinforcement learning at the edge of the statistical precipice[J]. Advances in neural information processing systems, 2021, 34: 29304-29320.\n\n[4] Fu J, Kumar A, Nachum O, et al. D4rl: Datasets for deep data-driven reinforcement learning[J]. arXiv preprint arXiv:2004.07219, 2020.\n\n[5] Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang. Discovering generalizable multi-agent coordination skills from multi-task offline data. In The Eleventh International Conference on Learning Representations, 2022.\n\n[6] Van Den Oord A, Vinyals O. Neural discrete representation learning[J]. Advances in neural information processing systems, 2017, 30.\n\nThanks again for your conscientious review. Your comments have been immensely helpful in improving our paper. If you have any additional feedback, please don’t hesitate to share it, and we will address it promptly. If you feel that our responses have adequately resolved your concerns, we hope you will consider improving your rating."}, {"review_id": "rin8CawQKo", "reviewer": "Reviewer_KvJL", "summary": "This paper introduces a novel approach for enabling zero-shot generalization in multi-task MARL using offline data. The key insight is that effective policy transfer requires capturing knowledge at two levels: **individual agent skills** and **team-level cooperative tactics**. The authors consider scenarios where agents are trained on known tasks using offline data and must generalize to unseen tasks without further training. While prior work focuses solely on transferring individual agent skills, the authors argue this is insufficient for MARL. They introduce the concept of \"tactics\". **_Individual Skill Learning_** uses a VAE to extract transferable skill embeddings from offline trajectories that capture individual agent behaviors. **_Cooperative Tactic Learning_** employs a VQ-VAE to map skill combinations into a discrete tactic codebook. **_Bi-Level Decision Transformer_** is a policy model that first selects an appropriate tactic from the codebook, then infers individual skills conditioned on the tactic, and finally decodes actions.", "questions": "Please refer to the \"Weaknesses\" section for clarification questions.\n\n1. How does the method ensure discovered tactics are semantically meaningful rather than arbitrary clusters? The regularization term $\\beta_2 \\sum_{k_1 \\neq k_2} \\frac{1}{||c_{k_1} - c_{k_2}||_2^2 + \\epsilon}$ encourages diversity but not necessarily meaningfulness. What makes a skill combination constitute a \"tactic\" beyond being a cluster in embedding space? How do you ensure discovered tactics are semantically meaningful?\n2. The choice of $L=10$ for the Decision Transformer seems arbitrary. How does performance vary with trajectory length?\n3. How does BiKT perform when source and target tasks have fundamentally different objectives or constraints? The current experiments only vary team sizes within similar game mechanics.\n4. What are the computational costs of the three-stage training? How does the method scale with team size, and what is the largest team size you've tested?\n5. Why did you choose the original SMAC benchmark instead of SMACv2 [1], which was specifically designed to address multi-task generalization challenges in MARL? SMACv2 includes procedurally generated scenarios, randomized start positions, and more diverse unit compositions that would better test the generalization capabilities of BiKT.\n\n[1] Ellis, Benjamin, et al. \"SMACv2: An improved benchmark for cooperative multi-agent reinforcement learning.\" arXiv preprint arXiv:2212.07489 (2022).", "limitations": "The paper's evaluation relies on the original SMAC benchmark, which may not adequately test the claimed generalization capabilities. SMACv2 [1] provides a more rigorous testbed for transfer learning in MARL due to several key advantages: procedural variation, dynamic opponent policies, compositional diversity. Given that the paper's central contribution concerns zero-shot generalization to \"unseen tasks,\" the choice of SMAC potentially limits the strength of the empirical claims. The relatively constrained variation in SMAC scenarios may overestimate the method's ability to handle genuinely novel tactical situations that would arise in more diverse multi-agent settings.\n\n**Recommendation**: The authors should either (1) include SMACv2 experiments to strengthen their generalization claims, or (2) explicitly acknowledge this limitation and discuss how SMAC's structural constraints may impact the generalizability of their findings to more varied scenarios.\n\n[1] Ellis, Benjamin, et al. \"SMACv2: An improved benchmark for cooperative multi-agent reinforcement learning.\" arXiv preprint arXiv:2212.07489 (2022).", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**: The paper presents a well-formalized approach that properly grounds the bi-level knowledge transfer in established frameworks (VAE for skills, VQ-VAE for tactics, Decision Transformer for policy). The evaluation covers multiple task sets (Marine-Hard, Marine-Easy, Stalker-Zealot, Cooperative Navigation) with different data quality levels (Expert, Medium, Medium-Expert, Medium-Replay). The explicit separation of individual skills and team tactics is an original contribution that addresses a genuine gap in existing work. The bi-level knowledge decomposition provides a new lens for understanding multi-agent coordination that could influence future MARL research. The use of VQ-VAE to create a discrete tactic codebook elegantly handles the challenge of varying team sizes across tasks. The Bi-Level Decision Transformer that sequentially infers tactics then skills is a novel architectural contribution. The method successfully discovers meaningful skills and tactics that transfer across tasks, with visualizations showing clear clustering of learned representations. The paper makes a compelling case that multi-agent coordination knowledge exists at multiple levels of abstraction, and capturing both individual and team-level patterns is crucial for effective generalization in MARL.\n\n**Weaknesses**: The paper lacks theoretical guarantees or analysis of when and why the bi-level decomposition should work. What conditions ensure that tactics are transferable across tasks? While ablations show robustness to some choices, critical hyperparameters like skill dimension ($N_s=4$) and codebook size ($K=16$) appear to be set arbitrarily without systematic justification. The three-stage training process (skills → tactics → policy) seems computationally expensive and potentially unstable. No analysis of computational costs or training time is provided. The distinction between skills and tactics, while intuitive, lacks formal mathematical definition. What exactly makes a skill combination a \"tactic\"? Key details are relegated to appendix without clear pointers. For instance, how exactly are entity features extracted and processed? Experiments are confined to relatively homogeneous task sets (same unit types, similar objectives). How well would this transfer to truly diverse tasks? All experiments involve relatively small teams (≤15 agents). The approach's scalability to larger multi-agent systems remains unclear. The baselines UPDeT and ODIS have already been outperformed; a comparison with their successors (e.g. TransfQMix [1], TETQmix [2], LDSA [3], RODE [4]), would strengthen claims. While the combination is novel, individual components (VAE, VQ-VAE, Decision Transformer) are well-established; the technical novelty is primarily in their integration. The paper doesn't deeply explore what makes a tactic transferable or how to ensure discovered tactics are meaningful beyond empirical observation.\n\n[1] Gallici, M. et al. (2023) \"TransfQMix: Transformers for Leveraging the Graph Structure of Multi-Agent Reinforcement Learning Problems\", arxiv:2301.05334.\n\n[2] Zhu, Y. et al. (2025) \"Multi-task multi-agent reinforcement learning with task-entity transformers and value decomposition training\", doi:10.1109/tase.2024.3501580. \n\n[3] Yang, M. et al. (2022) \"LDSA: Learning dynamic Subtask assignment in cooperative Multi-Agent Reinforcement Learning\", arxiv:2205.02561.\n\n[4] Wang, T. et al. (2022) \"RODE: Learning Roles to Decompose Multi-Agent Tasks\", arxiv:2010.01523", "was_revised": true, "final_justification": "After carefully considering all the points raised in the authors' response, I've decided to raise my score accordingly. I believe this final score accurately reflects the current state of the paper and its contributions. I do however believe it remains weak in its current form. I encourage the authors to continue their work in this direction, as the research area remains important and promising.", "rebuttal": "We sincerely appreciate your recognition of our methodology, experimental results, and writing, along with the constructive feedback. Below, we offer detailed responses and clarifications. We hope these can resolve your concerns and enhance the strengths of our work.\n## Main Questions\n> W3: The three-stage training process seems computationally expensive...\n>\n> Q4: What are the computational costs of the three-stage training? \n\nAs reviewer Pgpu has kindly acknowledged as one of our strengths, **we have provided a detailed description of our computing resources in Appendix B.3.** Additionally, we report the training time for each method as follows:\n\n|  | UPDeT | ODIS | HiSSD | BiKT |\n| -| -| - | - | - |\n| Training Time (hours) per task |7|12|15|8|\n\nCompared to other methods, our three-stage training does not introduce additional computing resource consumption.\n\n> W4:  The distinction between skills and tactics lacks formal mathematical definition...\n\nWe would clarify the relationship among three key variables: individual skills $z$, skill combination $S_c$, and tactic $c$.\n\nThe skill combintaion $S_c$ is defined as the *Cartesian Product* of individual skills $z^1,...,z^n$, i.e., $S_c = z^1 \\times z^2 \\times...\\times z^n$. This skill combination can serve as a representation of the team's overall behavioral pattern. However, due to variations in the number of agents across different tasks, the dimension of the skill combination differs even if the underlying team behavior from different environment maps remains the same. To address this challenge, we define the tactic as a mapping from skill combination: $\\text{Mapping Func}(S_c=z^2 \\times...\\times z^n) \\rightarrow c$\n The $\\text{Mapping Func}$ refers to our *Cooperative Tactic Encoder* $p_{tac}$ and *NNS* process in Figure 2.\n\n> W5:  Key details are relegated to appendix without clear pointers. For instance, how exactly are entity features extracted...\n\nIn Figure 2, we mention that the details of the encoders and decoders are provided in Appendix C, where we also explain how our entity features are processed.  However, this may lead to misunderstanding as it does not explicitly clarify it. In the revision, we will modify it as \"The network details and feature processing details about the encoders and decoders are shown in Appendix C\". Besides, we will add some key feature process summary in sections 3.1 and 3.2.\n> W7: All experiments involve relatively small teams (≤15 agents)...\n>\n> Q4: How does the method scale ...\n\nFor fair comparison, we adopt the same task setting and offline dataset as ODIS [1]. Beyond that, we also attempt an extended experiment with a larger number of agents. Due to time constraints, we will conduct experiments on additional maps in the future.\n\n| | ODIS | HiSSD | BiKT  |\n|-|-|-|-|\n|27m_vs_28m|$18.75 \\pm 7.8$|$25.8 \\pm 6.9$|$34.4 \\pm 7.2$|\n> W8: ...a comparison with their successors (e.g. TransfQMix [1], TETQmix [2], LDSA [3], RODE [4]), would strengthen claims. \n\nThese methods are not directly comparable to ours, as their problem settings differ fundamentally. Our problem setting focuses on **learning policies for multiple tasks simultaneously** from an **offline dataset**, with the goal of enabling **policy transfer to unseen tasks**. However, they do not conform to this setting and thus cannot serve as fair baselines for comparison. \n\n|| Learning by online or offline dataset | Can learn multiple tasks simultaneously | Can policy transfer |\n|-|-|-|-|\n|TransfQMix|online|No| Yes|\n|TETQmix|online|Yes| Yes|\n|LDSA|online|No| No|\n|RODE|online|No|Yes|\n|BiKT|offline dataset|Yes| Yes|\n\n> W9:  While the combination is novel, individual components (VAE, VQ-VAE, Decision Transformer) are well-established; the technical novelty is primarily in their integration.\n\nOur primary purpose is to validate the effectiveness of our BikT paradigm by components. For further details, please refer to our Response 1 to *Reviewer cecZ*.\n\n> W10: The paper doesn't deeply explore what makes a tactic transferable...\n>\n>Q1: How does the method ensure discovered tactics are semantically meaningful...\n\n* **Difference between arbitrary clustering of skill combinations and learned tactic.**\n  A key limitation of clustering based on skill combinations is that their dimensionality depends on the number of agents, making it challenging to generalize across tasks with varying agent counts. Our tactic overcomes this by mapping skill combinations into a fixed-dimensional space.\n* **Meaningfulness of the tactics.**\n  Figures 4(c) and 4(d) illustrate the distribution of tactics across different tasks, revealing both distinctions and similarities. For instance, tasks like 3m and 5m6m exhibit distinct tactic distributions, whereas 2s3z and 2s4z show similar team-level tactical patterns. This indicates that the learned tactics offer task-specific team guidance, rather than forcing a single multi-agent policy to generalize without considering task-specific preferences.\n\n> W6: Experiments are confined to relatively homogeneous task sets...\n>\n> Q3: How does BiKT perform when source and target tasks have fundamentally different objectives or constraints?\n>\n> Q5: Why did you choose the original SMAC benchmark instead of SMACv2...\n\nWe choose SMAC as the benchmark to ensure a fair comparison, as both our dataset and task setup follow ODIS [1].\n However, we also believe it is meaningful to evaluate our method in smac v2. To this end, we train QMIX for 5 million steps on protoss_3_3 and protoss_5_5, collect 2,000 trajectories from the trained policies, and then conduct experiments using this offline data. we can conclude that BiKT still achieves high performance in more dynamic environment.\n| | (source) protoss_3_3| (source) protoss_5-5 | (unseen) protoss_4_4|(unseen) protoss_6_6|(unseen) protoss_7_7 |(unseen) protoss_8_8|\n|-|-|-|-|-|-|-|\n|mean winning rate of offline data | $63.5$| $54.4$ |-|-|-|-|\n|ODIS| $43.3 \\pm 4.7$| $21.8 \\pm 5.4$| $18.8 \\pm 7.7$| $20.8 \\pm 7.6$| $15.6 \\pm 8.1$|$3.1 \\pm 1.5$|\n|HiSSD| $47.6 \\pm 6.7$| $38.8 \\pm 6.1$| $38.6 \\pm 5.9$| $26.7 \\pm 6.3$ | $21.9 \\pm 7.8$| $5.2 \\pm 3.9$|\n|BiKT| $54.9 \\pm 5.4$|$41.4 \\pm 7.3$|$35.4 \\pm 9.8$|$41.5 \\pm 8.7$| $20.8 \\pm 6.9$|$15.7 \\pm 7.4$ |\n\n> Format: Incomplete sentence Appendix B.1...; Missing figure reference...\nWe will modify it as \"... it achieves approximately a 50 winning rate.\" And we will add the figure reference.\n\n## Ablation Study\n|Different settings|source 3m| source 5m6m| source 9m10m| unseen4m | unseen5m | unseen10m | unseen12m | unseen 7m8m | unseen 8m9m | unseen 10m11m|unseen10m12m | unseen13m15m |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| w/o Tactic Codebook $C$ | $98.2 \\pm 0.2$  | $68.2 \\pm 5.3$ | $83.2 \\pm 2.5$ | $90.2 \\pm 1.9$ | $88.6 \\pm 2.7$ | $86.2 \\pm 3.2$ | $57.2 \\pm 4.2$ | $23.3 \\pm 5.5$  | $20.3 \\pm 9.2$ | $40.7 \\pm 9.2$ | $1.6 \\pm 1.6$ | $1.6 \\pm 1.3$ |\n| $L=5$   | $100.0 \\pm 0.0$ | $78.9 \\pm 3.5$ | $98.4 \\pm 1.5$ | $99.3 \\pm 0.9$ | $99.9 \\pm 0.8$  | $97.2 \\pm 1.5$ | $98.7 \\pm 0.9$ | $64.8 \\pm 8.2$ | $48.2 \\pm 8.3$ | $90.2 \\pm 2.3$ | $12.0 \\pm 1.8$ | $3.2 \\pm1.5$ |\n| $N_s=10$ | $100.0 \\pm 0.0$ | $81.8 \\pm 3.6$ | $99.2 \\pm 1.5$ | $98.8 \\pm 0.2$ | $100.0 \\pm 0.0$ | $99.4 \\pm 0.1$ | $98.8 \\pm 1.2$| $70.3 \\pm 10.3$| $47.5 \\pm 8.4$ | $89.4 \\pm 4.0$ | $13.4 \\pm 3.8$ | $4.1 \\pm0.3$ |\n| $K=32$ | $100.0 \\pm 0.0$ | $80.9 \\pm 3.2$ | $99.2 \\pm 0.3$ | $99.2 \\pm 0.1$ | $99.3 \\pm 0.2$ | $99.3 \\pm 0.2$ | $99.0 \\pm 0.2$ | $70.2 \\pm 8.0$ | $46.2 \\pm 7.4$ | $90.2 \\pm 2.1$ | $12.3 \\pm 2.2$ | $3.7\\pm 1.6$ |\n| BiKT  | $100.0 \\pm 0.0$ | $81.3 \\pm 4.5$ | $99.4 \\pm 0.4$ | $99.3 \\pm 0.1$ | $100.0 \\pm 0.0$ | $99.4 \\pm 0.1$ | $99.0 \\pm 0.2$ | $68.0 \\pm 9.9$ | $50.0 \\pm 6.2$ | $90.6 \\pm 1.1$ | $14.6 \\pm 1.5$ |$4.2 \\pm 2.1$ |\n\n> W1: The paper lacks theoretical guarantees or analysis of when and why the bi-level decomposition should work...\n\nThis question is both insightful, and the answer to it directly highlights our fundamental advantages. In the paper, we have indeed considered this point and conducted ablation studies. We consider 2 settings:\n* Remove the tactic learning. **(We have conducted it in line 879 of the Appendix.)** We copy the results in the above table.\n* Remove the tactic learning and individual skill learning. **(We have conducted it in line 867 of the Appendix.)** \n\n> W2: Critical hyperparameters like skill dimension ($N_s=4$) and codebook size ($K=16$) appear to be set arbitrarily...\n\nWe consider the following setting:\n* codebook size $K=32$ (**We have conducted it  in line 873 of the Appendix, we copy the result in the above table.**)\n* Skill dimension $N_s=10$\n\n> Q2: The choice of $L=10$ for the DT seems arbitrary...\n\nWe consider the following setting:\n* Content length $L=5$ (**We have conducted it in line 905-909, we copy the results here.**)\n\n**Summary:**\n* **Bi-level transfer matters**: **Tactics provide team-level guidance across tasks.**  Without them, the policy struggles in multitask transfer, relying solely on observation changes of different tasks. **Individual skills help generalize to varying agent numbers by learning action mappings of different lengths.** Without both, the method only generalizes to simple tasks like 3m to 5m.\n* **The perturbation of skill dimension has little impact on BiKT’s performance.**\n* **Effective tactics in the codebook converge** regardless of the codebook size (see Lines 908–911).\n* **BDT doesn't rely on the model scale advantage brought by longer DT content length.** We didn't use arbitrary $L$ but follow the common practice in Decision Transformer ($L=5$). We double it to 10 and the performance remains consistent.\n\n[1] Fuxiang Zhang, et.al. Discovering generalizable multi-agent coordination skills from multi-task offline data. In ICLR 2022\n\nThank you for your helpful comments and they’ve significantly improved our paper. If you have further suggestions, we’d be happy to address them. If you feel your concerns have been resolved, we kindly hope you’ll improve your rating."}, {"review_id": "SfzBtMQ7Px", "reviewer": "Reviewer_cecZ", "summary": "This paper presents a novel framework for zero-shot generalization across multiple tasks in multi-agent reinforcement learning (MARL) using offline data. The core insight is that successful policy reuse in MARL should not only transfer individual agent skills but also team-level coordination patterns, referred to as cooperative tactics. To this end, the authors introduce a Bi-Level Knowledge Transfer (BLKT) algorithm that decouples and encodes these two forms of knowledge using VQ-VAE-based embeddings and a decision-transformer-style backbone.\n\nI find the proposed approach conceptually sound and cleverly simple. While the building blocks are composed of established methods, the design is well-motivated, clearly articulated, and effectively implemented. The experiments demonstrate strong performance and offer qualitative insights into the role of coordination in multi-agent generalization.", "questions": "I do not have additional major questions for the authors beyond the clarification regarding return-to-go initialization. The paper is well-executed, methodologically sound, and clearly presented. I am confident in my recommendation to accept and believe the contribution will be of interest to the MARL and offline RL communities.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- **Clear motivation and novel framing**: The paper highlights a key insight in MARL generalization, namely, that coordinated team-level behaviors should be learned and transferred explicitly. This two-level abstraction (skills vs. tactics) is both intuitive and well-executed.\n- **Strong engineering and thoughtful design**: Although the components used (e.g., VQ-VAE, nearest neighbor search, decision transformers) are not novel individually, their combination is technically solid and contributes to a well-functioning pipeline.\n- **Clarity and structure**: The writing is excellent, concise, easy to follow, and logically structured. Figures (especially Figure 1) are helpful and informative.\n- **Empirical performance**: The method shows strong empirical results, and the evaluation setup is well designed and clearly explained in Section 3.\n- **Qualitative analysis**: I particularly appreciated the additional insights offered through Figure 4 (skill/tactic visualization) and the qualitative assessments in the results section and Table 2. These help substantiate the benefits of the proposed decomposition.\n\n---\n\n## Weaknesses\n\n- **Lack of algorithmic novelty**: While the method is well-motivated and effective, it primarily integrates existing techniques without introducing fundamentally new algorithms. However, given the strength of the empirical results and the thoughtful architectural integration, this is not a significant drawback in this case.\n\n- **Initialization details could be clarified**: In Section 3.2, for each agent \\(i\\), the BDT is initialized with a \"high return-to-go.\" However, it’s unclear how this value is chosen. I guess this is environment-dependent and can be inferred from the offline dataset. I would appreciate if the authors could provide some additional details on this point as that would help clarify some assumptions.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our methodology, experimental results, and overall presentation, as well as the insightful feedback you've provided. Below, we offer detailed clarifications in response to your comments, and we hope our revisions help resolve any concerns and enhance the strength of our work.\n\n> Weakness1:  While the method is well-motivated and effective, it primarily integrates existing techniques without introducing fundamentally new algorithms. \n\nWe would like to clarify that our main contribution lies in the Bi-level Knowledge Transfer framework, which provides a novel perspective on policy transfer in multi-agent reinforcement learning (MARL).  Instead of treating policy transfer as a monolithic process, we explicitly decompose it into team-level tactic transfer and individual-level skill transfer. This design offers several advantages:\n\n-  **Bi-level Knowledge provides essential guidance for policy transfer of MARL.**  Policy transfer in  MARL fundamentally requires agents to execute appropriate team policies across a variety of unseen tasks. Given that all agents must share the same policy model while adapting to different tasks, a team-level guiding signal is needed to steer agents toward task-specific policy preferences. We define this as the learning of tactic-level knowledge. Meanwhile, from the perspective of each agent’s local policy, certain behaviors or decision patterns can be reused across tasks. We refer to this transferable component as individual skill knowledge. Together, these two types of knowledge form the important foundation for effective policy transfer in MARL.\n\n- **Structured Knowledge transfer mitigates learning challenges.**  Tactics focus on team-level strategy learning, while individual skills emphasize the abstraction of local actions.\nThis separation enables the tactic codebook to capture the diverse patterns of skill composition across different tasks, summarizing the variability of team behaviors. Meanwhile, individual skill learning extracts reusable local action patterns across tasks, highlighting their generality.  The bi-level knowledge transfer structure alleviates the challenges of multi-task learning by introducing an explicit organization of knowledge.\n\n- **Flexible team-level generalization.**  Through BDT, our method selects appropriate tactics from the tactic codebook in different environments and enables agents to execute the corresponding skills, thereby enabling coordinated decision-making across different tasks.\n\nWhile our current implementation of BiKT is built upon existing modules, it is primarily intended to **validate the effectiveness of the bi-level knowledge transfer paradigm**. We believe that each module (e.g., skill encoder, tactic encoder) is modular and can be replaced with alternative techniques to suit different environments and multi-task settings. Therefore, the novelty of our work lies in proposing and validating a generalizable and modular framework for hierarchical knowledge transfer in MARL, which can inspire future extensions and adaptations.\n\n> Weakness 2/ Question 1: In Section 3.2, for each agent (i), the BDT is initialized with a \"high return-to-go.\" However, it’s unclear how this value is chosen. I guess this is environment-dependent and can be inferred from the offline dataset. I would appreciate if the authors could provide some additional details on this point as that would help clarify some assumptions.\n\nYour suggestion to clarify the return-to-go (rtg) setting is very helpful.\n\nTo clarify, the setting of return-to-go is not our main contribution, and we adopt the default return-to-go setting as the previous works: Decision Transformer [1], [2], [3]. During evaluation, we set the target return based on the desired performance — for example, using the maximum possible return to encourage expert-like behavior.\n\nSpecifically, the rtg is environment-dependent.  In fact, in SMAC and MPE,  the maximum rtgs in different maps are the same, which are 20 and 1, respectively, corresponding to a win rate of 100%. So we uniformly set the maximum return across all maps, including source tasks and unseen tasks.\n\nIn addition, we conducted an ablation study on the Marine-Expert task, with several alternative rtg configurations: \n\n* The higher return-to-go: 25, denoted by C1\n* The average return of all trajectories, denoted by C2 (19.8929 for 3m; 17.3425 for 5m6m; 19.6140 for 9m10m)\n\nThe results are shown below.\n\n| Different rtg | source-task 3m  | source-task 5m6m | source-task 9m10m | unseen-task 4m | unseen-task 5m  | unseen-task 10m | unseen-task 12m | unseen-task 7m8m | unseen-task 8m9m | unseen-task 10m11m | unseen-task 10m12m | unseen-task 13m15m |\n| ------------- | --------------- | ---------------- | ----------------- | -------------- | --------------- | --------------- | --------------- | ---------------- | ---------------- | ------------------ | ------------------ | ------------------ |\n| C1            | $100.0 \\pm 0.0$ | $80.4 \\pm 5.6$   | $98.4 \\pm 1.1$    | $99.4 \\pm 0.3$ | $100.0 \\pm 0.0$ | $100.0 \\pm 0.0$ | $98.8 \\pm 0.4$  | $67.5 \\pm 12.8$  | $51.3 \\pm 6.8$   | $89.8 \\pm 0.1$     | $13.8 \\pm 3.6$     | $4.4 \\pm2.4$       |\n| C2            | $100.0 \\pm 0.0$ | $81.8 \\pm 4.6$   | $98.2 \\pm 1.5$    | $98.8 \\pm 0.2$ | $100.0 \\pm 0.0$ | $99.4 \\pm 0.1$  | $98.8 \\pm 0.4$  | $71.3 \\pm 12.9$  | $47.5 \\pm 8.4$   | $90.4 \\pm 4.0$     | $12.5 \\pm 4.9$     | $3.1 \\pm0.3$       |\n| BiKT          | $100.0 \\pm 0.0$ | $81.3 \\pm 4.5$   | $99.4 \\pm 0.4$    | $99.3 \\pm 0.1$ | $100.0 \\pm 0.0$ | $99.4 \\pm 0.1$  | $99.0 \\pm 0.2$  | $68.0 \\pm 9.9$   | $50.0 \\pm 6.2$   | $90.6 \\pm 1.1$     | $14.6 \\pm 1.5$     | $4.2 \\pm 2.1$      |\n\n**Conclusion**: The results show that increasing the  return-to-govalue or using the average from the offline dataset has little impact on the performance of multi-agent policy transfer. This demonstrates that our method is relatively stable and does not rely on precise tuning of return-to-go values.\n\n[1] Chen L, Lu K, Rajeswaran A, et al. Decision transformer: Reinforcement learning via sequence modeling[J]. Advances in neural information processing systems, 2021, 34: 15084-15097.\n\n[2] Xu M, Shen Y, Zhang S, et al. Prompting decision transformer for few-shot policy generalization[C]//international conference on machine learning. PMLR, 2022: 24631-24645.\n\n[3] Mao H, Zhao R, Chen H, et al. Transformer in transformer as backbone for deep reinforcement learning[J]. arXiv preprint arXiv:2212.14538, 2022.\n\n\nYour comments have greatly contributed to the improvement of our paper. If you have any new comments, please feel free to provide them, and we will promptly address them. If you find that we have addressed your concerns, we hope you will improve your rating."}]

Author rebuttal:

Meta-review: This paper proposes a bi-level knowledge transfer framework for multi-task MARL, enabling transfer at both the individual skill level and the inter-agent coordination level. The method builds upon offline reinforcement learning with expert-level trajectories, incorporating a hierarchical structure and discrete encodings for skills and tactics.

The strengths of the paper lie in its well-motivated problem setting, the bi-level transfer strategy, and extensive empirical validation on multiple benchmarks (MPE and SMAC). Reviewers acknowledged the methodological soundness and empirical performance, and raised only minor concerns.

During the rebuttal, the authors provided thorough responses, clarifying key design decisions such as the use of expert trajectories, policy pre-training, and entity features. They also addressed writing and notation issues, promising improvements in the revised version.

Given the sound methodology, strong experimental results, and the authors’ detailed and constructive responses to all raised concerns, the paper appears to be in a good shape for acceptance.
Decision: Accept (poster)