Discovering Generalizable Multi-agent Coordination Skills from Multi-task Offline Data
Authors: Fuxiang Zhang
Chengxing Jia
Yi-Chen Li
Lei Yuan
Yang Yu
Zongzhang Zhang
Venue: ICLR 2023 poster (2023)
Keywords: multi-agent reinforcement learning
multi-task reinforcement learning
skill discovery
offline reinforcement learning

Abstract: Cooperative multi-agent reinforcement learning (MARL) faces the challenge of adapting to multiple tasks with varying agents and targets. Previous multi-task MARL approaches require costly interactions to simultaneously learn or fine-tune policies in different tasks. However, the situation that an agent should generalize to multiple tasks with only offline data from limited tasks is more in line with the needs of real-world applications. Since offline multi-task data contains a variety of behaviors, an effective data-driven approach is to extract informative latent variables that can represent universal skills for realizing coordination across tasks. In this paper, we propose a novel Offline MARL algorithm to Discover coordInation Skills (ODIS) from multi-task data. ODIS first extracts task-invariant coordination skills from offline multi-task data and learns to delineate different agent behaviors with the discovered coordination skills. Then we train a coordination policy to choose optimal coordination skills with the centralized training and decentralized execution paradigm. We further demonstrate that the discovered coordination skills can assign effective coordinative behaviors, thus significantly enhancing generalization to unseen tasks. Empirical results in cooperative MARL benchmarks, including the StarCraft multi-agent challenge, show that ODIS obtains superior performance in a wide range of tasks only with offline data from limited sources.

Published as a conference paper at ICLR 2023
DISCOVERING GENERALIZABLE MULTI-AGENT COOR- DINATION SKILLS FROM MULTI-TASK OFFLINE DATA
Fuxiang Zhang Chengxing Jia 2, Yi-Chen Li', Lei Yuan' Yang Yu Zongzhang Zhang't National Key Laboratory for Novel Software Technology, Nanjing University
PPolixir Technologies
(zhangfx, jiacx, liye, yuanl] e llmdda... nju.. edu.
(yuy, zzzhang)
ABSTRACT
Cooperative multi-agent reinforcement learning (MARL) faces the challenge of adapting multiple tasks with varying agents and targets. Previous multi-task MARL approaches require costly interactions to simultaneously learn or fine-tune policies in different tasks. However, the situation that an agent should generalize to multiple tasks with only offline data from limited tasks is more in line with the needs of real-world applications. Since offline multi-task data contains a variety of behaviors, an effective data-driven approach is to extract informative latent variables that can represent universal skills for realizing coordination across tasks. In this paper, propose novel Offline MARL algorithm to Discover coordInation Skills (ODIS) from multi-task data. ODIS first extracts task invariant coordination skills from offline multi-task data and learns delineate different agent behaviors with the discovered coordination skills. Then we train a coordination policy to choose optimal coordination skills with the centralized training and decentralized execution paradigm. We further demonstrate that the discovered coordination skills can assign effective coordinative behaviors, thus significantly enhancing generalization unseen tasks. Empirical results in cooperative MARL benchmarks, including the StarCraft multi-agent challenge, show that ODIS obtains superior performance in a wide range of tasks only with offline data from limited sources.
INTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) has drawn broad attention in addressing
problems like video games, sensor networks, and autopilot (Peng et al., 2017 Cao et al.) 2013
Gronauer Diepold 2022] Yun et al. [022] Xue et al., 20226). Since recent MARL methods mainly
focus on learning policies in one single task with simulating environments (Sunehag et al.) 2018
Rashid et al. 2020 Lowe et al. 2017 Foerster et al. 2018), there exist two obstacles when applying them to real-world problems. One poor generalization when facing tasks with varying agents and targets, where the practical demand is to adapt to multiple tasks rather than learning every new task from scratch (Omidshatici al. 2017). The other is potentially high costs and risks caused by real-world interactions through an under-learning policy (Levine et al. 2020).
Multi-agent systems are expected perform flexibly among multiple general scenarios where the agents and targets may differ. Multi-task MARL is one promising way to realize such flexibility and generalizability. Previous related works mainly focus on training simultaneously in a pre-defined task set (Omidshafiei et al.] 2017 Iqbal et al. (021) or fine-tuning a pre-trained policy to target tasks (Hu) al. 2021; Zhou et al.) 2021 Qin et al.) 2022a) in an online manner. Although these approaches exhibit promising performance in some tasks, the expensive cost of online interactions hinders their
applications to a broader range of tasks. Offline RL (Levine et al. 2020). aiming at learning policies from static dataset, is anticipated to remove the need for interactions during training. However, most current offline RL methods conservatively regularize the learned policies towards datasets
(Wu et al. 2019 Kumar et al. 2019: Yang al., 202 Fujimoto et al. 2019). Albeit conservatism
"These authors contributed equally. Work was done during an internship at Polixir Technologies. 1 Zongzhang Zhang the corresponding author.

---

Published as a conference paper at ICLR 2023
Coordination Skills
Discover
Reuse
Focus Fire
Move Back
Offline RL
Select
Training
Coordination skills
Mooin task:
Coordination Policy
Source 5m, 8m,
Figure I: An illustration of coordination skill discovery from multi-task offline data. Offline data from marine battle source tasks like 5m and 8m of StarCraft multi-agent challenge contain generalizable coordination skills like focusing fire, moving back, etc. After discovering these coordination skills from source data, a coordination policy learns to appropriately choose coordination skills through an offline RL training process. When facing the unseen task 10m, the agents reuse the discovered coordination skills to achieve coordination and accomplish the task.
effectively mitigates the distribution shift issue of offline learning, it will restrict the learned policy to be similar to the behavior policy, leading to severe degradation of generalization when facing unseen data (Chen et al. 2021b). Therefore, leveraging multi agent offline data to train policy with adequate generalization ability across tasks is in demand.
This paper finds that the underlying generic skills can greatly help improve the policy's general- ization. Indeed, humans are good at summarizing skills from several tasks and reusing these skills in other similar tasks. Taking Figure] [ as an example, we try to learn a policy from some StarCraft multi-agent challenge (Samvelyan et al. (019) tasks, 5m and 8m, where we need to control five or eight marines separately to beat the same number of enemy marines. Moreover, we aim to directly deploy the learned policy without fine-tuning to unseen target task, 10m, task with ten marines on each side. One effective way to achieve this to extract skills from the source tasks, like focusing fire on the same enemy moving back low-health units, and then apply these skills to the target task. Although these tasks have different state spaces or action spaces, these skills can be applicable to broad range of tasks. We refer such task-invariant skills coordination skills since they beneficial to realize coordination in different tasks. In other words, extracting such coordination skills from known tasks facilitate generalization via reusing them in unseen tasks.
Towards learning and reusing coordination skills in a data-driven way, we propose a novel Offline MARL algorithm to Discover coordInation Skills (ODIS), where agents only access multi-task dataset to discover coordination skills and learn generalizable policies. ODIS first extracts task- invariant coordination skills that delineate agent behaviors from coordinative perspective. These shared coordination skills will help agents perform high-level decision-making without considering specific action spaces. ODIS then learns a coordination policy to select appropriate coordination skills to maximize the global return via the centralized training and decentralized execution (CTDE) paradigm (Olichoek et al. 2008). Finally, we deploy the coordination policy directly to unseen tasks. Our proposed coordination skill is noteworthy compared previous works in online multi-agen agent skill discovery (Yang et al.) 2020 He et al.) 2020). which utilize hierarchically learned skills to improve exploration and data efficiency. Empirical results show that ODIS can learn to choose proper coordination skills and generalize to a wide range of tasks only with data from limited sources. To the best of knowledge, it is the first attempt towards unseen task generalization in offline MARL.
RELATED WORK
Multi-task MARL. Multi-task RL and transfer learning in MARL can improve sample efficiency with knowledge reuse (da Silva & Costa 2021). The knowledge reuse across multiple tasks may be impeded by varying populations and input dimensions, asking for policy networks with flexible structures like graph neural networks (Agarwal et al.) 2020) or self-attention mechanisms (Hu) et al. 2021 Zhou et al. 202T). Recent works consider utilizing policy representations or agent representations to realize multi-task adaptations (Grover et al. 2018). EPL (Long al.) 2020) introduces an evolutionary-basee
curriculum learning approach to scale up the number agents. REFIL (Iqbal et al. 2021) adopts randomized entity-wise factorization for multi-task learning. UPDeT
2

---

Published as a conference paper at ICLR 2023
(Hu et al. 2021) utilizes transformer- bandd value networks to realize adaptations among alterable populations and imputs. However, these approaches only consider simultaneous learning or fine-tuning in different tasks. Training generalizable policies for deployment in unseen tasks remains challenge.
Offline MARL. Offline RL attracts tremendous attention for its data-driven training paradigm without
interactions with the environment (Levine et al. (020). Offline MARL is promising research direction ZZang et al.
[021]] Formanek et al. (203)] that trains policies from a static dataset. Previous
work (Fujimoto et
2019) discusses the distribution shift issue in offline learning and considers
learning behavior-constrrineed policies to relieve extrapolation error from unseen data estimations (Wu et al., 2019; Kumar et al., 2019). Existing offline MARL methods often try to adopt conservative constraints upon current online MARL methods, which either extend policy gradient algorithms to
multi-agent cases (Lowe et al.
2017 Foerster et al., 2018 Iqhal & Sha 2019 Wang et al.] 202
Xue et al. 2022a) or adopt Q Q-learning paradigms with value decomposition (Sunehag et al. 2018
Rashid et al. 2020 Wang et al.
2021a, Cao et al.) 202T; Yuan et al. 2022b a). Most offline MARL
methods consider training policies with sufficient conservatism (Yngg et al. 2021 Jiang & Lu 2021
Pan et al. 2022 Guan et al. 2023). These methods can help learn an effective policy from offine
data, but the conservative learning manner may significantly degrade the performance when facing unseen tasks, as learned policies will fail to generalize with out-of out-off-dsttrbbl distribbution inputs. Some data sharing methods in single-agent RL (Li et al. |020,] Yu et al. 2021; (202) consider properly using multi-task data for offline policy training. However, potential coordination skills from multi-task MARL data cannot be efficiently exploited by these single-agent data sharing methods.
Skill discovery with hierarchical structures. Hierarchical RL (Barto & Mahadevan] 2003 Tang al. 2019 Pateria et al.. 2021) provides an approach to realizing temporal abstraction and hierarchi- cal organization. Skill discovery methods adopt the hierarchical structure to discover unsupervised
high-level skills with state empowerment from information theory (Eysenbach et al.) 2019 Campos it al. 2020; Sharma et al.
2020). This diverse skill discovery methods from online RL can also be extended to MARL, HSD YYang et al. 2020) learns latent skill variables via centralized training and MASD (He et al. 2020) considers performing skill discovery in an adversarial way. However, these methods are usually tailored to address data efficiency in online RL, where the proposed skills are different from our multi-task reusable skills towards generalizable MARL. On the other hand, recent works in offline RL also exhibit that high-level latent actions help tackle extrapolation errors from nut-of-distributie actions and learn primitive behaviors (Zhou al.) 2020 Ajay al.] [202T). A data-driven approach to extracting and reusing high-level multi-agent behaviors (i.e., our proposed coordination skills) can be a practical and promising direction toward multi-task generalization.
BACKGROUND
3.1 COOPERATIVE MULTI-AGENT REINFORCEMENT LEARNING
A cooperative multi-agent task can be modeled as decentralized partially observable Markov decision process (Dec-POMDP) (Oliehoek & Amato 2016) T = (I, P,S, O, R,T), where I = {1, [L..... ........ n is the set of agents, S's the set of global states, A is the set of actions, and Ω is the set of observations. At each time step with state E S, each agent i I only acquires an observation 0; E 52 drawn from the observation function (ss, i). and then chooses its action a, E A. The joint action a = (a1...... an) leads to next state s' ~ P(s" s, a) and the corresponding global reward r = R(s, a) The target is to find a joint policy π <aa T) to maximize the discounted return (T, (T((T, a)
2o E = 7* R(s,, ax) 8o 8, ag a. T.. where 7 E [0, 1) is a discount factor that trades off between current and future rewards. Here T = =TI, ,,)) where т, denotes the trajectory aa,1,,,,0 ,f)) of agent i.
(of, of of-1
Most value- cooperative MARL allaalgooithi algorithms apply the CTDE paradigm (Sunehag al. 2018 Rashid et al., 2020 Wang et al. 2021a), where agents can learn decomposable global value function QToI(T, a) represented by mixing network in the training phase and use the decomposed value function Q, (7,, a.) for decentralized decision. The global value function parameterized by can be learned by minimizing the squared temporal difference (TD) error (Sutton & Barto 2018). using experience replay and a target network parameterized by 0 to stabilize training (Mnih et al. 2015):
min Fr F.ar (r+ max Qtot (r',a';0) - Qtor (T, ar;)a
27
3

---

Published as a conference paper at ICLR 2023
Mixing Network Qua(55.2 2]
Action Decoder P(An Ix
Action Decoder p(a T,,
Action Decoder p(a, Tn
Q learning
(2,, - An)
((t, 1,
Ri Alg
Action Decoder p(a, (T.24)
Coordination Policy ((+,.4,)
Coordination Policy ((t
Coordination Policy ((t..2.)
Coordination Policy Q(t
Unlabeled Multi-task Data f.ff..., ri. (ah)...h)
Multi-task Trajectory Data sf, ff, Th.
Multi-task Trajectory Input fi. t6
Coordination skill discovery
(2) Coordination policy learming
Multi- tasti-taskkae decentralized execution
The overall framework of ODIS. ODIS is trained with multi-task offline data from limited tasks and can be deployed to different tasks with varying agents and targets. (1) ODIS learns state encoder and an action decoder from unlabeled multi-task data (without reward information) to discover coordination skills across different tasks. (2) ODIS performs centralized training to learn coordination policies that can maximize the global return with the help of a mixing network. (3) In multi-task decentralized execution, ODIS individually chooses coordination skill according the coordination policy and decodes specific actions using the action decoder.
Figure 2:
3.2 MULTI-TASK MULTI-AGENT REINFORCEMENT LEARNING
Recent multi-task MARL works consider policy learning among two several cooperative multi- agent tasks (Hu et al. 202I Iqbal et al.) (02T). our settings, we focus on learning and deploying policies in static task set [7.). A major difficulty in multi-agent multi-task transfer is varying agent numbers and observation/action dimensions. To learn a transferable or universal policy across different tasks, previous methods (Hu et al. [022] develop population-invariant network structure by utilizing the transformer structure (Vaswani et al. 2017). For the individual Q-network, the observation o, of agent can be decomposed into self/environmenall information oren and other entities' information (oother). The network further generates embeddings of Q, K, V and calculate the attention output according to the attention mechanism:
Q MLP
othhr 1,
..... 1), K ([oP oother (olher.....). .....), V MLPv((o.")) other 00,1 ........
QK?
own [e
other 1,1
..... = softmax
dk = din(K).
ViT
The output representations e,tT 5,1 .... can be further utilized to derive Q-values, where eown can be used for actions without interactions with other entities and e eother can be used for interactive actions with entity j. To handle the partial observability, we can append an additional historical embedding h - from the last time step to the transformer input sequence and acquire corresponding hf. notable that this decomposition technique with transformer models can also be applied to process
state information mixing networks or other modules (Zhou et al. 2021: Qin et al. 2022al to effectively handle multi-task inputs with varying data shapes.
METHOD
In this section, we will describe our proposed Offline MARL algorithm to Discover coordInation Skills (ODIS) from static multi-task dataset D, which is fully data-driven approach to learning generalizable policies for multiple tasks. As shown in Figure 2 ODIS begins with unsupervised coordination skill discovery that leverages a state encoder q(2; 2; 5, a, i) to extract the coordination skill 2, ......, 2n for each agent using the unlabeled portion of without reward information. The discovered coordination skill represents decoded to task-related action with local information т; by an individual action decoder p(a; T 2;). Both the state encoder and the action decoder trained in unsupervised learning manner to
abstraction of coordinative behaviors, which can be
4

---

Published as a conference paper at ICLR 2023
discover effective and distinct coordination skills. After that, ODIS learns coordination policy to select uppropriate coordination skills with offline RL training in the entire dataset D. This offline MARL training process adopts the CTDE paradigm with a conservative learning manner. In the decentralized execution phase, the agent can use individual coordination policy to choose the best coordination skill and decode it to specific action with the pre-trained action decoder. The high-level decision-making of ODIS promotes generalization across different tasks, especially in unseen tasks with varying targets and controllable agents. To tackle the alterable data shapes from multi-task settings, we design flexible and powerful network structures based on the transformer model to process input data, whose details are illustrated in Appendix
4.1 COORDINATION SKILL DISCOVERY
Coordination skill discovery is expected to extract effective coordination skills from offline data. We assume the coordination skill 2; for agent is a discrete variable from a finite set Z, where |z] is a hyper-parameter varying among tasks. We leverage state encoder q(2; 8, a, using the sampled state s and joint action a to obtain a coordination skill 2; for agent i. By feeding states and joint actions as input,
ensure the state encoder outputs appropriate coordination skills from the global perspective. In practice, we implement the state encoder with a transformer model, which simultaneously outputs coordination skills (21,.... ............ ...) of all agents in a sequence. Moreover, we need to derive task-related actions from the coordination skill in decentralized executions, where acquiring global information impractical for individual agents. Thus we further introduce an action decoder that can predict task-related action à: p(- Ti. 2;) with an agent i's local information Ti and the chosen coordination skill 25. The training objective is to maximize the likelihood of the real action a, from data, along with the Kullback-Leibe divergence between g(2; 8, i) and an uniform prior p(z;) as a regularization, following 3-VAE (Higgins et al., 2017). The regularization with uniform distribution of coordination skills can prevent the state encoder from choosing similar coordination skills for all inputs, thereby helping to discover distinguished coordination skills. The objective for coordination skill learning is presented as follows:
n
(0, 22) = -Eie,, D)
E (e [logp(a; Ti, 2.]] - 3DxL. (a( |s, a, I1 ((-))
(1)
where 6, and O₂ denote the parameters of the state encoder and the action decoder, respectively, and 3 is the regularization coefficient. Our implementations of the state encoder and the action decoder utilize the transformer structure to handle alterable dimensions of states, observations, and actions in multi-task settings. The key technique Section 3.22 and project each portion to an embedding with a fixed dimension. In Appendix C we
decompose input into sequence data as illustrated in
provide detailed descriptions of the network structures.
4.2 COORDINATION POLICY LEARNING
After coordination skill discovery, we train an action decoder that can generate a tesk-related action using an agent's local information and input coordination skills. Therefore, we can further consider on discovered coordination skills. As our coordination skills
developing high-level decision-makinn are universal for a given task set, we can train a generalizable coordination policy to decide on appropriate coordination skills for different tasks.
We adopt a value-based MARL method with the CTDE paradigm to learn a global value function Quor 2) that can be decomposed into individual value functions Q (7, (1),)1)) 21),...,
Qu(Tn, 2,). The
global value function Quae (T, 2) is trained with the squared TD loss as follows:
LTD (0.) B....a. (r (++ max Quot. (7',2';07) Qto (z)
(2)
where we use 0,, to denote all parameters in value networks including individual and mixing networks, 0,7 to denote parameters of target networks. Note that in Equation 2, cannot directly acquire the joint coordination skill z from the offline dataset D. Therefore, the state encoder q(2; s, a) trained in the coordination discovery phase is reused to provide joint coordination skill (21 (21...... 2n) from the state and joint action. When estimating Q-targets, we choose the joint coordination skill x by selecting each coordination skill zf with maximal individual Q-value Qe(7{, ={) to avoid search in
5

---

Published as a conference paper at ICLR 2023
the large joint coordination skill space, following previous MARL methods (Sunehag al. 2018 Rashid et al.) (020). When the value decomposition method satisfies the aadviduallglobal max (IGM)
principle (Wang et al. 2021a), the action selection with individual value functions is correct. We adopt a QMIX-style mixing network to ensure that can satisfy this property.
Our design of the individual value network and mixing network also utilizes the powerful transformer structure. Practically, the individual value network calculates local Q-values by taking representations from an observation encoder as input. The observation encoder has the same network structure the aforementioned action decoder to process local trajectory information. The mixing network takes the global state information as imput to generate non non-negative weights for combining individual Q-values. We also provide detailed descriptions of these networks in Appendix [
As the coordination skills from the previous discovery phase are obtained with global information, it remains a challenge that the coordination policy may not explicitly learn to choose effective coordination skills only with local observations. We find that directly learning this from the TD objective will be extremely inefficient. Therefore, it will be necessary to introduce an auxiliary objective to learn better representations that can guide the coordination policy towards a coordinative perspective. When updating parameters of the observation encoder, we use the last layer of the previous state encoder to process output representations from the observation encoder and thus acquire coordination skill distribution q.(- Tz). We expect that the output distribution can be similar to the pre-trained state encoder g(- I a, i). We calculate the KL--divergence between them update the observation encoder as the consistent loss Le below:
Lo(O,) = Σ B(E.T.)) [DRL (#:l- (T) |l (- 8, a, i))],
i=1
where do denotes parameters of the observation encoder in individual value network.
To tackle the aut-of-distribur issue in offline RL, we also adopt the popular conservative Q-learning (CQL) method (Kumar al. 2020). To be concise, the total loss term in the coordination exploitation phase is presented as Lp (. Φo) = LTD(0,) + aLLQu (0,) + ALc(oo), where a and are two coefficients and Loqr is the loss term from CQL.
Decentralized Execution. When performing decentralized executions in a test task, we use local information to calculate Q-values for each coordination skill with individual value network Qi(Ti, 2;) and then choose the coordination skill with maximal Q-value. The action decoder further uses the coordination skill and local information provide a task-related action for the particular task.
EXPERIMENTS
In this section, we design experiments to evaluate the following properties of ODIS] (a) The ability of multi-task generalization, including zero-shot generalization in unseen tasks. We conduct experiments in specially designed task sets from the StarCraft multi-agent challenge (SMAC) (Samvelyan et al. 2019) using offline data with diverse qualities. (b) The semantics of discovered coordination skills. We analyze the coordination skill usage in several test episodes from different tasks to investigate how ODIS makes decisions with different coordination skills. (c) Effectiveness of the ODIS structure. We conduct ablation studies to find out how the components of ODIS affect performance.
5.1 PERFORMANCES ON MULTI-TASK GENERALIZATION
Baselines. We introduce comparable baselines and perform several adaptations since none of the existing multi-task MARL algorithms considers offline learning. UPDeT (Hu et al.) (02T) is a state-of-the-art multi-agent transfer learning baseline that adopts transformer-based individual network to tackle multi-agent transfer. However, the mixing network of UPDeT not designed for simultaneous multi-task learning. As its alternatives, we implement two variants of UPDeT by adopting the transformer-bust based mixing network of ODIS (UPDeT-m), and the linear decomposable network of VDN (Sunehag et al. 2018)) (-PDeT-I), respectively. We also keep the same transformer structures and adopt the same CQL loss to these UPDeT baselines. Therefore, UPDeTTmm can be
"oode available wwt.ttp:://// pectts://giiee com/ LAMDAA RI/ODIS

---

Published as a conference paper at ICLR 2023
Table 1: Average test win rates of the final policies in the task marine-hard with different data
qualities. The listed performance is averaged over five random seeds. We abbreviate asymmetric task names for simplicity. For example, the task name "5m6m" denotes the SMAC map "5m 6m". Results of BC-best stands for the best test win rates between BC-t and BC-r.
Expent
Medium
Task
BC-best
UPDeTH
UPDeT-m
ODIS (ours)
BC-hest
UPDeTH
UPDeT-m
ODIS (ours)
3m
97.7
2.6
71.0 16.6
82.8 16.0
98.4
2.7
63.4 = 14.7
56.6 +
14.2
51.2 =
3,4
85.9 =
10.5
5m66
50.4
2.3
12.1 12.6
17.2
28.0
53.9
21.9
5.6
6.3
22.7
9m10m
95.3 1
1.6
26.6 1: 12.0
3.1 #
5.4
80.4
8.7
63.8
10.9
34.4 + 13.9
28.5
10.2
78.1
3.8
Unseen Tasks
28.6
33.0
95.3
21.6
5m
87.1 1: 10.5
40.1
25.9
33.6
40.2
89.1
10.0
76.6
14.1
77.4
16.0
67.2
21.3
85.9
11.8
10m
90.5
3,8
33,9
25.2
54.7
44.4
938
562 = 20.6
36.8
20.7
32.9
11.3
61.3
11.3
12m
70.8
15.2
10.9
18.9
17.2
28.0
58.6
11.8
240
10.5
5.3
32
3.8
35,9
8.1
7m8m
18.8
+ 3.1
0.8
1.4
0.0
25.0
15.1
1.6 =
1.6
2.4
26
0.0
0.0
28.1
22.0
8m9m
3.3
1.6
19.6
2.3
10ml Im
45,3
11.1
1.4
0.0
0.0
42.2
7.2
19.7
8.9
24
1.4
4.0
3,4
29.7 = 15.4
10m122
13m1.5m
0.0 1
0.0
0.0
1
0.0
0.0
0.0
2.3
2.6
0.6
1.3
0.0
1
0.0
0.0
0.0
1.6 +
1.6
Medium Expert
Medium Repila
Source Tasks
3m
67.7 23.7
50.1 23.9
85.2
17.9
73.6
22.0
81.1 1:
8.8
27.3 1:
25.9
41.4 =
20.1
83.6 1: 14.0
6.3
2
25.0
9m10m
26.0 = 13.9
27.7 24.1
24.3 = 18.7
31.3 14.5
33.4
13.1
2.3 4.1
0.8
14
34.4
8.0
81.3
41.0 8.0
43.9
39.0
828
13.5
61.5
23.4 15.5
35. 12.6
14.5
5m
74.0
2.9
65.7 10.1
33.6
40.2
828
17.7
75.0
24.2
$4.7
23.5
20.3
4.1
10m
78.1
6.7
39.8 20.1
32.8
38.1
828
16.8
8244 =
8.2
866
8.7
1100
78
84.4
13.1
2.3
7m88m
13.3 +: 4.5
4.0 4.2
2.3
1
4.1
15.6
4.4
7.3 =
6.4
233 1
26
16
2.7
9.4 1
2.2
56
10ml Im
26.6
4,7
8.0 12.2
11.8
8.1
33.6
8,9
468
6.6
23
4.1
08
144
35.9
52
10m12m
0.0
0.0
0.0 0.0
0.0
16
0.0
00
0.0
23
13ml5m
0.8 1.4
0.0
0.0
1.6
0.0
24
seen an ODIS variant without coordination skill discovery but learning an RI. policy to directly select actions. We also implement behavior cloning baselines since they are popular for offline tasks. We introduce transformer-based behavior cloning baseline (BC-t) that has the same structure as ODIS. As decision transformer methods (Chen al. 2021a) prevail in recent offline literature, we also append return-to-go information to BC-t and thus get baseline BC-r. More details of our implementations and hyper-past pareee-prameeerss are reported in Appendix C We notice that a recent work, MADT (Meng et al. 2021), proposed to train multi agett decision transformer with offline training and online tuning. However, we find that MADT is generally not comparable in our settings and put related discussion in Appendix G
StarCraft multi-agent micromanagement tasks. Following previous multi-task MARL methods
(Hu al. 2021] Qin al. 2022a), extend the original SMAC maps and sort out three task sets. In each task set, agents will control some units like marines, stalkers, and zealots, but the numbers of controllable agents and target enemies differ from tasks. We refer
three task
as marine-easy,
marine-hard, and stalker-zealot. Detailed descriptions of these task sets can be found in Appendix We adopt the popular QMIX algorithm (Rashid al. 2020) collect four classes of offline datasets with different qualities. Following guidelines in single-agen D4RI. offline RI benchmarks (Fu et al. 2020 Qin al.) (0222)), the four different dataset qualities are labeled as expert, medium, medium- expert, and medium-replay. We report the detailed properties of these datasets in Appendix B
conduct experiments in three task sets with four different data qualities. We train all methods with offline data only from three source tasks and evaluate them in wide range of unseen tasks. are shown in Table I while results of the two
The average test win rates in the task set marine- other task sets are deferred to Appendix ! The tables report the best test win rates between BC-t and BC-r as besbest. We find that ODIS generally outperforms other baselines in both source tasks and unseen tasks. ODIS discover and exploit common coordination skills from multi-task data,
7

---

Published as a conference paper at ICLR 2023
Source task: 5m
Unseen task: 12m
Source task: 2x37
Unseen task: 3c40
003
100
100
100
Paraebige
40
20
23
Time
Time sep
Time stop
TIMe dtop
(i) Coordination skill usages in the marine-easy task set
(b) Coordination skill usages in the stalker-zealot task set
skill (focus fire)
(advanice)
(ciaperso)
(c) Visualization of different coordination skills 5m (left) and 12m (right)
(absorb damago)
skit
skil (attack for)
t=7
t- 21
(advanco)
(d) Visualization of different coordination skills in 2s3z (left) and 3s42 (right)
Figure 3: An illustration of the semantics discovered coordination skills. We plot the percentages of used coordination skills in two maps from (a) the marine-easy task set and (b) the stalker-zealot task We anticipate the semantics of each coordination skill and visualize several test frames from corresponding time steps in (c) 5m and 12m maps and (d) 2s3z and 3s4z maps. Agents in colored circle choose the corresponding coordination skill with the same color in legends of (a) and (b).
resulting in superior and stable performance compared with UPDeT-1 and UPDeT-m, which cannot generalize well among different levels of tasks. We notice that behavior cloning methods present comparable performance, especially in expert datasets, indicating that our proposed transformer structure also enhances generalization across tasks. However, these behavior cloning baselines cannot effectively exploit multi-task data to further improve its performance, although add additional global return-to-go information in BC-r.
To evaluate the validity of ODIS, also compare ODIS with other single-task MARL approaches that can perform offline training, including ICQ (Yang et al. (021) and QPLEX (Wang et al. 202 1a). These works can adopt offline learning manners but cannot directly learn from multi-task data. We find that ODIS can exhibit comparable performance with these single-task offline MARL methods even in the unseen task of ODIS. These experiments reveal that the multi-task offline training manner of ODIS can indeed promote generalization to different tasks. We defer the experimental results and detailed discussion to Appendix D
Cooperative navigation tasks. To further validate the effectiveness of ODIS, we also conduct experiments in several cooperative navigation tasks from the multi multi-agent agent partiele environment (Lowe et al. 2017), where varying numbers of agents learn to reach corresponding landmarks. We notice that ODIS still shows superior performance both source and unseen tasks. The results and detailed descriptions of these tasks can be found in Appendix F
5.2 SEMANTICS OF DISCOVERED COORDINATION SKILLS
To investigate how the discovered coordination skill helps decision decision-making, we deploy ODIS agents to different tasks and record chosen coordination skills in test episodes. shown in Figure we exhibit the percentage of each coordination skill usage from policies learned in marine-easy and stalker-zealot task sets with expert datasets. For the marine-easy task set in Figure3Ja) we adopt coordination skill number of three and find that the curves of each coordination skill are pretty similar between source task 5m and an unseen task 12m despite different episode lengths. Further visualization shows that the three coordination skills represent different coordinative behaviors. The agents learn to utilize skill to disperse in different directions and then use skill 2 to advance. When enemies in the attack range, most agents choose skill I to focus fire specific enemies, and few
8

---

Published as a conference paper at ICLR 2023
3m-mmddium
9m vs_ 10m-medium
tasks
Unseen tasks
100
Mull- Task Discovery
consistent
-+
Single Task Discovery
wio consistant oSS
40
40
20
1.0
1e4
P2, 10m
7r
15m
ODIS with multi-tusk/simgb--uusk courdination skill discovery
(b) Performance of ODIS with/without consistent loss
Figure 4: (a) average test win rates of ODIS when performing multé-task/kinglettask coordination skill discovery. (b) average test win rates of ODIS with/without consistent loss.
agents choose skill 2 and skill 3 to pursue enemies or run away from fire. For the stalker- zelket task set in Figure3(b) we adopt coordination skill number four as the task is little more complicated. We exhibit the coordination skill usages in a source task 2x3z and an unseen task 3s4z, whose curves are also similar. The visualization indicates that the chosen coordination skill may present different semantics from masy. In the beginning, agents use skill 4 to advance. When approaching the enemies, some agents choose to attack nearby enemies with skill 2, and others learn to absorb damage forwardly with skill 1. We find that the stalker, melee unit, is more willing to draw enemy fire than the ranging unit stalker. After that, a few agents will use skill 3 to attack enemies far away, and more agents tend to choose skill 3 when nearby enemies fall.
5.3 ABLATION STUDIES
In ablation studies, we investigate the effectiveness of each component in our proposed ODIS structure. First, we try to find whether discovering coordination skills from multiple tasks can yield better performance in particular task. We perform ODIS offline training separately with multi-task and single-task coordination skill discovery. ODIS with multi-task coordination skill discovery can utilize data from the two other tasks to discover potentially more effective coordination skills, while ODIS with single-task task coordination skill discovery can only access data from the corresponding task. In coordination policy learning. both two approaches are learned from single-task data. Results in Figure 4 a) show that ODIS with multi-task coordination skill discovery acquires better performance, indicating that shared coordination skills across tasks can benefit other tasks.
We also conduct experiments to investigate our proposed consistent loss, which helps learn effective local representations for choosing coordination skills from coordinative perspective as the state encoder does. We run experiments with two variants of ODIS with and without consistent loss in the marine-hard task set and present average test win rates in three source tasks and four unseen tasks. As shown in Figure b).. the performance of ODIS with consistent loss is significantly better than ODIS without consistent loss, which shows the proposed loss can maintain the consistency of coordination skills chosen between coordination skill discovery and coordination policy learning.
Furthermore, the number of coordination skills is a key hyper-parameter of ODIS, so we conduct ex- periments to compare performances with different coordination skill numbers in Table 7 We find the performance ODIS not sensitive with coordination skills numbers ranging from 3 to 16. However, adopting less skill numbers or random selection will significantly fail, as discussed in Appendix F
6 CONCLUSION
We propose ODIS, an offline MARL algorithm applying coordination skill discovery from multi- task data, realizing multi-task generalization a fully data-driven manner. ODIS extracts and utilizes coordination skills shared among different tasks and thus acquires superior performance in both source and unseen tasks. The effectiveness of ODIS indicates that underlying coordination skills from multi-task data can be crucial for generalization in cooperative MARL. As the discrete coordination skill might be limited when facing dissimilar tasks, developing general representations among dissimilar tasks from multi-tast addition, research on offline MARL algorithms and benchmarks in various domains will also be helpful to real-world applications of MARI..
many-task data is a promising direction in the future. In

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported by National Key Research and Development Program of China (2020AAA0107200,, the National Science Foundation of China (61921006, 62276126), and the Natural Science Foundation of Jiangsu (BK20221442), We thank Feng Chen, Xinyu Yang, and the anonymous reviewers for their support and helpful discussions on improving the paper.
REFERENCES
Akshat Agarwal, Sumit Kumar, Katia P. Sycara, and Michael Lewis. Learning transferable cooperative behavior in multi-agent teamss. International Conference on Autonomous Agents and Multiagent Systems, pp. 1741-1743, 2020.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal. Sergey Levine, and Ofir Nachum. OPAL: offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2021.
Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning.
Discrete Event Dynamic Systems: Theory and Applications, 13(1-2):41-77, 2003.
Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i-Nieto, and Jordi Tor- res. Explore, discover and learn: Unsupervised discovery of state-covering skills. International Conference on Machine Learning, pp. 1317-1327, 2020.
Jiahan Cao, Lei Yuan, Jianhao Wang, Shaowei Zhang, Chongjie Zhang, Yang Yu, and De-Chuan Zhan. LINDA: multi-agent local information decomposition for awareness of teammates. arXiv preprint arXiv:2109 12508, 2021.
Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial Informatics, 9(1): 427-438 2013.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances Neural Information Processing Systems, pp. 15084-15097, 2021a.
Xiong-Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei (Tony) Qin, Wenjie Shang, and Jieping Ye. Offline model-based adaptable policy learning. In Advances in Neural Information Processing Systems, pp. 8432 8443, 2021b.
Felipe Leno da Silva and Anna Helena Reali Costa. Transfer Learning for Multiagent Reinforcement Learning Systems. Synthesis Lectures Artificial Intelligence and Machine Learning, Morgan & Claypool Publishers, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference Artificial Intelligence, pp. 2974-2982, 2018.
Claude Formanck, Asad Jeewa, Jonathan Shock. and Arnu Pretorius. Off-the- Grid MARL: A framework for dataset generation with baselines for cooperative offline multi-agent reinforcement learning. arXiv preprint arXiv:2302.0052/1 2023.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 1052-2062, 2019.
10

---

Published as a conference paper at ICLR 2023
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: survey. Artificial Intelligence Review, 55(2):895 02055-99333220 2022.
Aditya Grover, Maruan Al-Shedivat, Jayesh K. Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International Conference on Machine Learning, pp. 1797-1806, 2018.
Cong Guan, Feng Chen, Lei Yuan, Zongzhang Zhang, and Yang Yu. Efficient communication via self-supervised information aggregation for online and offline multi-agent reinforcement learning. arXiv preprint arXiv:2302.00600) 2023.
Shuncheng He, Jianzhun Shao, and Xiangyang Ji. Skill discovery of coordination in multi-agent reinforcement learning. arXiv preprint arXiv:2006.0400 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. UPDeT: Universal multi-agent reinforce- ment learning via policy decoupling with transformers. In International Conference on Learning Representations, 2021.
Shariq Iqbal and Fei Sha. Actor-attention-cree critic for multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 2961-2970, 2019.
Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 4596-4606, 2021.
Jiechuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2108. 01832, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pp- 1 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1179-1191, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0/64 2020.
Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Henrik Christensen, and Hao Su. Multi-task batch reinforcement learning with metric learning. In Advances in Neural Information Processing Systems, 2020.
Qian Long, Zihan Zhou, Abhinav Gupta, Fei Fang, Wu, and Xiaolong Wang. Evolutionary population curriculum for scaling multi-agent reinforcement learning. In International Conference on Learning Representations, 2020.
Ryan Lowe, Yi Wu, Aviy Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiie environments. In Advances in Neural Information Processing Systems, pp. 6379-6390, 2017.
Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu. Offline pre-trained multi-agent decision transformer: One big sequence model tackles all SMAC tasks. arXiv preprint arXiv:2 12.00284,, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
11

---

Published as a conference paper at ICLR 2023
Frans A Oliehoek and Christopher Amato. A Concise Introduction Decentralized POMDPs. Springer, 2016.
Frans A. Olichock, Matthijs T. J. Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pondps. Journal of Artificial Intelligence Research, 32:289-333 -53, 2008.
Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decen- tralized multi-task multi-agent reinforcement learning under partial observability. In International Conference on Machine Learning, pp. 2681-2699, 2017.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi- agent reinforcement learning with actor rectification. In International Conference Machine Learning, pp. 17221-17237, 2022.
Shubham Pateria, Budhitama Subagdja, Ah-Hwee Tan, and Chai Quek. Hierarchical reinforcement learning: A comprehensive survey. ACM Computing Surveys, 109:)):00:1110000 2021.
Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent iitirrectionallycooddiiie to play StarCraft combat games. arXiv preprint 1703. 10069, 2017.
nets: Emergence of human-level coordination in learning
Rong-Jun Qin, Feng Chen, Tonghan Wang, Lei Yuan, Xiaoran Wu, Yipeng Kang, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. Multi-agent policy transfer via task relationship modeling. In Deep RL Workshop on Neural Information Processing Systems (NeurIPS), 2022a.
Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hun Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. NeoRL: A near real-world benchmark for offline reinforcement learning. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2022b.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21 21(17)) 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In International Conference on Autonomous Agents and MultiAgent Systems, pp. 21 2186-2188, 88, 2019.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-avee aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo. Karl Tuyls, and Thore Graepel. Value-deccomposition networks for cooperative multi-agent learning based on team reward. In International conference on Autonomous Agents and Multiagent Systems, pp. 2085-2087, 2018.
Richard S Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT Press, 2018
Hongyao Tang, Jianye Hao, Tangjie Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Zhaopeng Meng, Changjie Fan, and Wang. Hierarchical deep multiagent reinforcement learning with temporal abstraction. arXiv preprint arXiv:1809 30. 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Jianhao Wang, Zhizhou Ren, Temy Liu, Yang Yu, and Chongjie Zhang. QPLEX: Duplex dueling multi-agent Q-learning In International Conference on Learning Representations, 2021a.
Yihan Wang, Beining Han, Tonghan Wang. Heng Dong, and Chongjie Zhang. DOP: off-policy multi-
agent decomposed policy gradients. In International Conference on Learning Representations, 2021b.
12

---

Published as a conference paper at ICLR 2023
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911119911.11611,, 2019.
Di Xue, Lei Yuan, Zongzhang Zhang, and Yang Yu. Efficient multi-agent communication via shapley message value. In International Joint Conference on Artificial Intelligence, pp. 578-584, 2022a.
Ke Xue, Jiacheng Xu, Lei Yuan, Miqing Li, Chao Qian, Zongzhang Zhang, and Yang Yu. Multi-agent dynamic algorithm configuration. In Advances in Neural Information Processing Systems, 2022b.
Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforce- ment learning with skill discovery. International conference on Autonomous Agents and Multiagent Systems, pp. 1566-1574, 2020.
Yiqin Yang. Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 10299-10312. 2021.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. In Advances in Neural Information Processing Systems, pp. 11501-11516, 2021.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine How to leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning, pp. 2561 1 2-6633,, 2022.
Lei Yuan, Chenghe Wang, Jianhao Wang, Fuxiang Zhang, Feng Chen, Cong Guan, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. Multi-agent concentrative coordination with decentralized task representation. In International Joint Conference on Artificial Intelligence, pp. 599-605, 2022a.
Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang. Multi-agent incentive communication via decentralized teammate modeling. In AAAI Conference on Artificial Intelligence, pp. 9466 1964,,44, 2022b.
Won Joon Yun, Soohyun Park, Joongheon Kim, MyungJae Shin, Soyi Jung, David A Mohaisen, and Jae-Hyun Kim. Cooperative multiagent deep reinforcement learning for reliable surveillance via autonomous multi-uav control. IEEE Transactions on Industrial Informatics, 18110)7788--796, 2022.
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Finite-sample analysis for decentralized batch multiagent reinforcement learning with networked agents. IEEE Transactions on Automatic Control, 66(12):5925 5925-990,, 2021.
Tianze Zhou, Fubiao Zhang, Kun Shao, Kai Li, Wenhan Huang, Jun Luo, Weixun Wang, Yaodong Yang, Hangyu Mao, Bin Wang, et al. Cooperative multi-agent transfer learning with level-adaptive credit assignment. arXiv preprint wwXiv:2166.005// 2021.
Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: latent action space for offline reinforce- ment learning. In Conference on Robot Learning, pp. 1719-1735, 2020.
13

---

Published as a conference paper at ICLR 2023
A DESCRIPTIONS OF DEFINED TASK SETS IN SMAC
The StarCraft multi-agent challenge (SMAC) Samvelyan et al. 2019) is a widely used cooperative multi-agent environment containing different types of StarCraft micromanagement tasks. In this paper, we sort out three task sets and call them marine-easy, marine-hard, and stalker-zealot, respectively. The marine-easy and marine-har hard task sets include several marine battle tasks where several ally marines need to beat the same number or a larger number of enemy marines. In the marine-easy task set, the number of enemy marines equals the number of ally marines, while tasks from the marine-hard task set contain both equal numbers and larger numbers of enemy marines. The stalker-zealot task set includes several tasks with symmetric stalkers and zealots on each side. We exhibit illustrations of these tasks in Figure (aa) and Figure (tb). For our goal of generalization to unseen tasks with limited sources, we select three tasks from the task set as training tasks, and the other tasks are only for evaluation. The detailed properties of these task sets can be seen in Tables and respectively.
Agent
Agent
(a) 5m (SMAC)
(b) 2s3z (SMAC)
(c) Cooperative navigation agents)
Figure 5: Illustrations of different kinds of tasks in the experiments section. (a) The marine battle task 5m vs "s_6m of SMAC, where agents control 5 controllable marines to beat built-in-Al marines. (b) The heterogeneous task 2s3z of SMAC, where agents control 2 stalkers and 3 zealots to beat the same number of built-in-Al units. (c) The cooperative navigation task with 3 agents from the multi-agent particle environment, where 3 agents need to reach corresponding landmarks.
Table 2: Descriptions of tasks in marine-easy task set.
Task type
Task
Ally units
Enemy units
Properties
3m
3 Marines
Marines homogeneous & symmetric
Source tasks
5m
5 Marines
Marines
homogeneous & symmetric
10m
10 Marines
10 Marines
homogeneous & symmetric
4m
Marines
Marines
homogeneous & symmetric
6m
6 Marines
6 Marines
homogeneous & symmetric
7m
7 Marines
Marines
homogeneous & symmetric
Unseen tasks
8m
8 Marines
Marines
homogencous & symmetric homogeneous & symmetric homogeneous & symmetric homogeneous & symmetric
9m
9 Marines
Marines
11m
11 Marines
11 Marines
12m
12 Marines
12 Marines
B PROPERTIES OF OFFLINE DATASETS
As stated in the experiments section, we construct offline datasets based on the PyMARL implemen- tatiorFJ of the MARL algorithm QMIX (Rashid et al., 2020). Following the popular single-agent offline reinforcement learning benchmark D4RL (Fu et al. 2020), we collect data with four types of qualities called expert, medium, medium-expert, and medhum-replay, respectively. Definitions of these four qualities are listed below:
The expert dataset contains trajectory data collected by QMIX policy trained with 2, 000, 000 steps of environment interactions. We also record the test win rate of the trained QMIX policy (as the expert policy) for constructing medium datasets.
thw.. httptttc...oooo/////////omommmmm
14

---

Published as a conference paper at ICLR 2023
Table 3: Descriptions of tasks in marine-hard task set.
Task type
Task
Ally units
Enemy units
Properties
3m
3 Marines
Marines
homogeneous & symmetric
Source tasks
5m_vs_6n v__s_6m
5 Marines
Marines
homogeneous & asymmetric
9m 9m_vs_ Dm_vs_1One 10m
9 Marines
10 Marines
homogeneous & asymmetric
4m
Marines
Marines
homogeneous & symmetric
5m
5 Marines
Marines
homogeneous & symmetric
10m
Marines
10 Marines
homogeneous & symmetric
12m
12 Marines
12 Marines
homogeneous & symmetric
Unseen tasks
7m_ vs 8m
Marines
Marines
homogeneous & asymmetric
8m_ 9m
8 Marines
Marines
homogeneous & asymmetric
10m_vs_ v__vs__ Ilm
10 Marines
11 Marines
homogeneous & asymmetric
10m_ vs 12m
10 Marines
12 Marines
homogeneous & asymmetric
13m_ vs_ vs_mmm
13 Marines
15 Marines
homogeneous & asymmetric
Table 4: Descriptions of tasks in stalker-zealot task set.
Task type
Task
Ally units
Enemy units
Properties
2 Stalkers,
2 Stalkers,
2s3z
heterogeneous & symmetric
Zealots
3 Zealots
2 Stalkers,
2 Stalkers,
Source tasks
2s4z
heterogencous & symmetric
Zealots
4 Zealots
Stalkers,
3 Stalkers,
3s5z
heterogeneous & symmetric
Zealots
5 Zealots
Stalkers, Zealots
Stalkers,
1s3z
heterogeneous & symmetric
3 Zealots Stalkers, Zealots
Stalkers,
1s4z
heterogeneous & symmetric
Zealots
Stalkers,
Stalkers,
1s5z
heterogencous & symmetric
Zealots
Zealots
Stalkers,
2 Stalkers, 5 Zealots
2s5z
heterogeneous & symmetric
Zealots
Unseen tasks
Stalkers,
Stalkers,
3s3z
heterogeneous & symmetric
Zealots
3 Zealots
Stalkers,
Stalkers, 4 Zealots
3s4z
heterogeneous & symmetric
Zealots
Stalkers,
Stalkers,
4s3z
heterogeneous & symmetric
Zealots
3 Zealots Stalkers,
Stalkers,
4s4z
heterogeneous & symmetric
Zealots
4 Zealots Stalkers,
Stalkers,
4s5z
heterogeneous & symmetric
Zealots
5 Zealots
The medium dataset contains trajectory data collected by a QMIX policy (as the medium policy) whose test win rate is half of the expert QMIX policy.
The medium-expert dataset mixes data from the expert dataset and the medium dataset to acquire a more diverse dataset.
The medium-replas dataset is the replay buffer of the medium policy, containing trajectory data with lower qualities.
As we consider generalization to unseen tasks, we only require offline datasets in the source tasks of the three task sets mentioned above. For the expert and medium datasets, collect 2, 000 trajectories to construct each dataset, and thus the medium-expert dataset contains 4, 000 trajectories as a mixture of the expert and medium data. The trajectory number of the medium-replay dataset depends
15

---

Published as a conference paper at ICLR 2023
Table 5: Properties of offline datasets with different qualities.
Task
Quality
# Trajectories
Average return
Average win rate
expert
2000
19.8929
0,9910
medium
2000
13.9869
0.5402
3m
medium-expert
4000
16,9399
0.7656
medium-replay
3630
N/A
N/A
expert
2000
19.9380
0.9937
medium
2000
17.3288
0.7411
5m
medium-expert
4000
18.6334
0.8674
medium-replay
771
N/A
N/A
expert
2000
19.9438
0,9922
medium
2000
16.6297
0.5413
10m
medium-expert
4000
18.2595
0.7626
medium-replay
571
N/A
N/A
expert
2000
17.3424
0.7185
medium
2000
12.6408
0.2751
5m_vs_6m
medium-expert
4000
14.9916
0.4968
medium-replay
32607
N/A
N/A
expert
2000
19.6140
0.9431
medium
2000
15.5049
0.4146
9m vs 10m
medium-expert
4000
17.5594
0.6789
medium-replay
13731
N/A
N/A
expert
2000
19,7655
0,9602
medium
2000
16.6279
0.4465
2s3z
medium-expert
4000
18.1967
0.7034
medium-replay
4505
N/A
N/A
expert
2000
19,7402
0.9509
medium
2000
16,8735
0.4965
2s4z
medium-expert
4000
18.3069
0.7237
medium-replay
6172
N/A
N/A
expert
2000
19.7850
0,9518
medium
2000
16.3126
0.3114
3s5z
medium-expert
4000
18.0488
0.6316
medium-replay
11528
N/A
N/A
the number of sampling trajectories before the medium policy stops training. When performing multi-task offline training in our experiments, we select up to 2, 000 trajectories of data from each source task. When the medium-replay dataset contains less than 2, 000 trajectories of data, select all trajectories. Data from different tasks are merged into a multi-takk dataset ensure that the policy trained with multi-task data simultaneously.
C STRUCTURES, HYPER-PARAMETERS AND TRAINING DETAILS OF ODIS
In this section, we will exhibit the network structure, pppem-er choices, and other training details of ODIS. As illustrated in Figure 6 the network structure of ODIS mainly contains four components, the action decoder, state encoder, observation encoder, and mixing network, respectively. These four components apply the attention mechanism process alterable state and observation spaces. The observation encoder and the action decoder take the observation as input, and the state encoder and the mixing network take the global state input.
As presented in the preliminaries, we decompose the observation information o, of an agent i into several portions including its own/environmental information ofwa and other entities" information (oother). Each kind of portion is fed into separate fully connected layer acquire embedding
16

---

Published as a conference paper at ICLR 2023
Coordination Skill Discovery
(II) Coordination Policy Learning
Qca(t..2)
Roconstruction Loss
II powe otherr
Self-Artention
TD COL Loss
f
Mixing
MLP
MLP Srr
Qut(F.))
m
Q+
Self Attention
Mixing Network
1i-1
(a) Action Docoder
Coordination
Coordination fe
(.(T.)))
2s, An
Softmax
From
From Og Og
Individual Value Network (MPP)
(c)
Mixing Network
owe other 4e
other M
Self Attention
Last Layer Classifier from State Encoder
Parameters
Alll
State Encoder
5) otther
Observation Encoder
other
0ag ... Gq
Og Gg
Observation 0g 00
(b) State
Encoder
Figure 6: The network structure of ODIS. In coordination skill discovery, ODIS leverages (a) local action decoder and (b) a global state encoder to discover coordination skills from multi-task offline data. In coordination policy learning, ODIS performs MARL with the CTDE paradigm to train indi- vidual coordination policies with the assistance of (c) mixing network. The individual coordination policy contains (d) an observation encoder to extract representations from local information.
Table 6: Hyper-parameteed of ODIS.
Hyper-pane paremerrrreeter
Value
hidden layer dimension
64
attention dimension
64
5.0
B
0.001
5.0
coordination skill number
(marine-easy); (marine-harde istalker-zealot
steps of coordination skill discovery
15000
optimizer
Adam
learning rate
0.0005
with the dimension of 64. For the self-attention network, we generate Q, K. V with fully connected layers from the embedding sequence and then perform self-attention (Vaswani et al., (017)) with the input and output dimensions of 64. The output sequence can be formalized as the representation of the agent i's own information e!nn and the representation to other entities {either) } in agent i's view. To handle the partial observability, we append the input sequence with historical embedding h when applying self-attention and thus acquire the output of h following the UPDeT structure (Hu al., 2021). For the observation encoder, only select the own information representation eq eowh to calculate Q-values and feed into the individual network or the coordination skill classifier. For the action decoder, we embed the chosen skill to vector of 64 dimension and concatenate it with the output sequence. When predicting actions, we divide available actions into interactive actions and non-interactive actions, where the interactive action means action that needs to interact with an entity and non-interactive action means an action that is only relevant to an agent itself. We output the values of non-interactive actions from e and the values of interactive actions with entity from corresponding e
and further concatenate them select the action with the maximal value.
17

---

Published as a conference paper at ICLR 2023
5m-medium
10m-medium
8m-medium
100
100
100
Average Test Win Rate
80
80
80
60
60
40
ODIS
40
40
ODIS (umseen)
ICQ
ICQ OPLEX
ICQ
OFLEX
GPLEX EX
0.0
1.0
2.0
3.0
te4
0.0
1.0
2.0
3.0
tel
0.0
1.0
2.0
1e4
Training Steps
Training Steps
Training Steps
5m-expert
10m-expert
8m-expert
Average Test Win Rate v
100
100
100
60
60
60
40
ODIS
CQ
IC0
ICQ
20
QPLEX
20
OPLEX
20
OPLEX
0
3.0
1.0
Training Steps
Training Steps
Training Steps
5m-medium-expert
10m-medlum-expert
8n-medlum-experrt
Average Test Win Rate %
100
ODIS (unseen)
80
80
80
QPLEX
60
60
ODIS
40
OD:5
40
CQ QPLEX
ICQ
20
20
QPLEX
20
0.0
1.0
3.0
1.0
2.0
3.0
0.0
1.0
2.0
3.0
Training Stops
Training Steps
Training Steps
Figure 7: Average test win rates of ODIS, ICQ, and QPLEX in three maps of the marine-easy easy task set with medium datasets (top), medium datasets (middle), and medium-expert datasets (bottom). Here ODIS is trained with offline data from three source tasks, 3m, 5m, and 10m, as shown in Table 2 while ICQ and QPLEX are trained with corresponding single-task data. Note that 8m an unseen task for ODIS, where ODIS can still acquire comparable performance with the two other offline MARL algorithms using offline data of 8m for training.
We also decompose the state information into several portions including the environment information gay, ally information {,,lly), J. and other entities' information {sgtherj. J. Like the process of the observation, we feed them into a separate fully connected layer to acquire embeddings with the dimension of 64 and further calculate Q, K, V and perform self-attention. The dimensions of Q and K are set to to reduce computation as the state may contain numerous entities, while V remains dimension of 64. The output sequence consists of the representation of each portion. For the mixing network, we utilize all ally information to generate non-negative weights by calculating the absolute value of MLP outputs. For the state encoder, we additionally append the action a; to the corresponding ally information s sllly perform self-attention and use ally representations from the output sequence as the input of the coordination skill classifier. The coordination skill classifier contains fully connected layer along with the softmax function to compute the probability of each coordination skill. The individual value network also contains fully connected layer to transform local representations into Qaluus..
Besides the above network structure, ODIS needs two training phases to perform coordination skill discovery and coordination policy learning separately. We implement ODIS with the aforementioned PyMARL framework to ensure that the mechanics irrelevant to the algorithm
the same previous methods. Other specific hyper-pane super-parameters are listed in Table 6 All the tabular results show the performance of ODIS with 50, 000 optimization steps, and the steps of the coordination policy
18

---

Published as a conference paper at ICLR 2023
Table 7: Average test win rates of the final policies trained with different coordination skill numbers (abbreviated as "skill num.") in the task set marine-hard with medium data quality. The listed performance is averaged over five random seeds. We abbreviate asymmetric task names for simplicity. For example, the task name "5m6m" denotes the SMAC map "5m_ __'6mm.. The column name "rand.
skill" stands for random coordination skill selection.
Task
skill num.
skill num.
skill num. 5 skill num.
skill num. 16
rand. skill
Source tasks
3m
67.8 + 12.1
72.7 = 10.2
85.9 + 10.5
85.2 + 8,9
80.5 + 5.6
0.0 0.0
5m6m
21.4 5.5
22.7 = 10.5
22.7 + 7.1
14.8
11.8
18.8 +
6.2
0.0 0.0
9m10m
66.1 + 9.2
47.7
= 8.1
78.1 +
3.8
45.3 + 12.2
65.6 - 12.7
0.0 0.0
Unseen Tasks
4m
44.3 12.5
78.1 9.4
61.7 + 17.7
65.6 5.8
64.1 10.9
0.0
5m
73.2 + 12.4
57.0
85.9 + 11.8
78.1
4.9
100.0 0.0
0.0
10m
37.1
23.4
30.7
61.31113
53.9 =
7.1
6
0.0
12m
22.7 + 26.8
37.8
35.9
8.1
15.6
13.1
30.5
25.4
0.0
7m8m
1.6 + 1.6
3.1 5.4
28.1 22.0
1.6 + 1.6
4.7 + 5.2
0.0
8m9m
3.1 + 3.8
1.6 +
1.6
4.7
2.7
1.6
1.6
3.1 1
3.1
0.0 0.0
10m11m
21.2 - 6.7
21.1
13.5
29.7 + 15.4
3.1 -
5.4
10.2 = 8.1
0.0
10m12m
0.0
0.0
0.0
0.0
1.6 1.6
0.0
0.0
0.0
0.0
0.0
13m15m
0.0
0.0
0.0
0.0
1.6 +
1.6
0.0
0.0
0.0 + 0.0
0.0 - 0.0
Table 8: Properties of offline datasets with different qualities in cooperative navigation tasks.
Task
Quality
# Trajectories
Average return
Average success rate
expert
2000
1,0000
1.0000
CN-2
medium
2000
0.6152
0.6152
expert
2000
0.7173
0.7173
CN-4
medium
2000
0.4273
0.4273
learning phase are the subtraction of the coordination skill discovery steps from the total steps. The training process of ODIS with an NVIDIA GeForce RTX 2080Ti GPÚ and a 32-core CPU costs 12- 14 hours typically. Our released implementation of ODIS, along with the provided offline datasets, follows Apache License 2.0, the same as the PyMARI framework.
D COMPARISONS WITH SINGLE-TASK OFFLINE MARL METHODS
Although ODIS aims at the multi-tase task offline MARL domain, we also compare ODIS with offline MARL methods trained in single-task in the marine-east casy task set, where ODIS utilizes three source tasks for training. We select two baselines, ICQ and QPLEX, for comparison. ICQ (Yang et al. 2021) is conservative-style offline MARL algorithm that can handle the severe distribution shift issue in offline MARL. QPLEX (Wang et al. 2021a) claims the IGM-complete value decomposition benefit offline training. As the baselines cannot leverage multi-task offline data, we feed the corresponding task dataset into these algorithms to train policies independently. We evaluate the performance three tasks, including 5m, 10m, and 8m, where 8m is an unseen task for ODIS. To train the two baselines in task 8m, we also collect data with QMIX policy like previous statements, and the properties of the data are exhibited in Table The data 8m only used for the two baselines but remains unseen to ODIS. We conduct experiments with medium, expert, and medium-exper data qualities. As shown in Figure ODIS outperforms other baselines in both source tasks 5m and 8m and the unseen task 8m, indicating that with learning through multi-task data, ODIS can not only perform better than other single-task offline MARL methods most tasks but present tremendous zero-shot task generalization to unseen tasks. ICQ can generally acquire good performance in expert data, but its conservative paradigm may limit the performance. QPLEX struggles in these datasets without particular tuning, and speculate that it is because datasets cannot provide
19

---

Published as a conference paper at ICLR 2023
Table 9: Average test success rates in the cooperative navigation task set with different data qualities.
Expert
Task
ODIS (ours)
BC-best
UPDeT-m
UPDeT-I
QMIX (online)
Source tasks
CN-2
100.0 = 0.0
99.0 + 1.5
90.6 + 6.8
91.7 = 9.7
100.0
CN-4
46.2 + 13.6
43.8 + 7.7
15.6 + 9.2
5.2 + 5.3
71.7
Unseen tasks
CN-3
85.6 = 7.6
72.9 + 2.9
47.9 + 10.3
30.2 + 14.5
83.4
CN-5
20.0 = 7.8
12.5 + 6.8
2.1 + 2.9
0.0 = 0.0
0.0
Medium
Task
ODIS (ours)
BC-t
UPDeT-m
UPDeT-1
QMIX (online)
Source tasks
CN-2
65.0 = 5.4
50,0 + 5.1
35,4 + 12.1
47.9 + 9.7
100.0
CN-4
28.7 = 6.7
24.0 + 3,9
4.2 + 2.9
8.3 + 2.9
71.7
Unseen tasks
CN-3
43.8 + 5.2
43.8 + 2.6
14.6 + 3.9
26.0 +
5.9
83.4
CN-5
8.1 +
2.5
7.3 + 3,9
0.0 + 0.0
1.0 +
1.5
0.0
diverse data for QPLEX perform policy exploitation and lead large extrapolation errors. The empirical results show that discovering and sharing coordination skills can be efficient and powerful across different tasks.
E EXPERIMENTS WITH DIFFERENT COORDINATION SKILL NUMBERS
The size of the coordination skill set is a key hyper-paramene parameter in ODIS, which represents the number of actions that be chosen in the coordination policy. We conduct experiments in the marine-hard task-set with medium quality to investigate whether decision-making in the coordination skill space works. We select the coordination skill number of the default setting for our main experiments. As shown in Table 7 we can find that the choices of coordination skill numbers 3, 5, 8, and 16 exhibit comparable performances in sophisticated tuning at the coordination skill number. We finally choose the coordination skill number of 5 in this task set because it obtains generally better performance. On the other hand, we can find that coordination skill number of 1 cannot generalize to most unseen tasks, as its performance entirely depends on the reconstructive ability of the action decoder with only 1 coordination skill that can be selected. We also evaluate the performance of ODIS with randomly choosing coordination skill when the total coordination skill number is 5, where the coordination policy is not trained and only rely the action decoder for decision-makee making. The results of ODIS with random skills exhibit all average test win rates of 0, indicating that the discovered coordination skill can be sufficiently utilized by the action decoder to generate task-relevant actions and cannot be simply disentangled from the framework.
unseen tasks, indicating that ODIS does need
F RESULTS ON COOPERATIVE NAVIGATION TASKS
To further evaluate the effectiveness of ODIS, we design and conduct experiments in task set from the cooperative navigation environment. Cooperative navigation is series of tasks from the multi-agent particle environment (MPE) raised by MADDPG (Lowe et al. (007)). In this environment, a number of agents try to reach corresponding landmarks and only acquire a reward of when they all successfully reach the landmarks. A visualization of this task can be found in Figure[cc) We add discrete control support to the original cooperative navigation tasks, where agents can execute actions of moving towards four directions and a "none" operation. We design a task set in the cooperative
20

---

Published as a conference paper at ICLR 2023
Table 10: Average test win rates of ODIS and MADT in the marine-hard task set with expert and medium data qualities.
Expert
Medium
Task
ODIS (ours)
MADT
Task
ODIS (ours)
MADT
Source tasks
3m
98.4 + 2.7
88.5 3.9
3m
85.9 + 10.5
4.2 +
1.5
5m_vs_oo vs_6m
53,9 +
5.1
3.1
+ 0.0
5m_vs_ jn_
22.7 + 7.1
21.9
2.6
9m_ 10m
80.4 + 8.7
1.0
1.5
9m v0m
78.1 + 3.8
14.6
7.4
Unseen tasks
4m.
95.3 + 3.5
83.3 + 5.3
4m
61.7 +: 17.7
31.2 +
15,9
5m
89.1 + 10.0
75.0
6.8
5m
85.9 + 11.8
63.5
12.6
10m
93.8 + 2.2
1.0 + 1.5
10m
61.3 + 11.3
33.3
14.1
12m
58.6 + 11.8
0.0
12m
35.9 8.1
0.0 0.0
7m_vs vs 8m
25.0 + 15.1
1.0
7m_vs_8n
28.1 22.0
1.0 -
1.5
8m vs
19.6 + 6.0
0.0
8m_vs vs_9m
4.7 2.7
0.0 0.0
10m vs 11m
42.2 + 7.2
0.0
10m_vs_ vs_ 11m
29.7 + 15.4
0.0 0.0
10m_ vi__vs__2mm 12m
1.6 + 1.6
0.0
10m_vs_12n
1.6 + 1.6
0.0 0.0
2.3 +
2.6
0.0 0.0
Lim_vs_15m
1.6 + 1.6
0.0 1 0.0
Table 11: Average test win rates of ODIS and MADT in small task set (including source tasks of 3m and 5m, and unseen tasks of 4m and 6m) with medium data qualities.
Source Task
ODIS (ours)
MADT
Unseen Task
ODIS (ours)
MADT
3m
85.9 + 10.5
60.4 + 10.3
4m
95.3 + 3.5
51.0 + 5.3
5m
85.9 + 11.8
74.0 + 3.9
6m
78.0 + 9.0
10.4 + 7.4
navigation environment by setting different numbers of agents and naming the cooperative navigation task with agents CN-n. Thus we set up four tasks named CN-2, CN-3, CN-4, and CN-5, respectively. We collect different qualities of data using the QMIX algorithm. The detailed dataset properties are shown in Table K
We compare ODIS with the aforementioned baselines, including BC-t, BC-r, UPDeT-m, and U-D and evaluate in the expert and medium datasets, respectively. The results are exhibited in Table 9 ODIS outperforms other baselines in both expert data and medium data. As reference, we also results online QMIX algorithms in the table. We find that in two unseen tasks, CN-3 and CN-5, ODIS trained with expert data can acquire better performances than leam-from-seee algorithm. A notable observation is that ODIS can generalize to CN-5, while an online QMIX will fail to learn valid policy in this task.
QMIX
G EVALUATIONS OF ODIS AND MADT
MADT (Meng al. 2021 recent approach to training multi-agent decision transformer with offline training and optional online tuning. However, MADT not a baseline originally designed for simultaneous multi task training. We compare ODIS with MADT in the marine-hard task set with expert and medium data qualities, and exhibit the results in Table IO We find MADT cannot generally leamn valid policy in multi-task settings, and we provide the following three explanations:
1. MADT utilizes feature encoding and action masking techniques deal with different input shapes. All inputs are encoded into the same shape with zero padding, and the action space has to be aligned to the maximal action space with unavailable actions masked. These techniques may induce poor generalization when the input shape changes dramatically (e.g.. the observation size of task 3m is 42 while for 10m 1 Im is 132).
21

---

Published as a conference paper at ICLR 2023
2. The multi-task generalization setting proposed by MADT adopts task set with similar input shapes (i.e., the observation size range is 25-48), which makes the above issue less significant.
3. Our implemented baselines (including ODIS, BC-t, and raised UPDeT variants) decompose input with particularly designed observation and state encoders to handle the varying shape issue better. It means our baseline design is generally fair and comparable. The benefits of our encoding method can be seen from the comparison between MADT and BC-t.
To validate the observed phenomena, we additionally conduct a simple experiment where the input shapes of the task set are restricted to small range. The results are shown in Table I1 MADT generalizes better in this small task set, while ODIS can still outperform MADT.
Table 12:
Average test win rates of ODIS in the marine-hard task with expert data of 2000 trajectories (ODIS-2000) and 1000 trajectories (ODIS- 1000).
Task
ODIS-2000
ODIS-1000
3m
98.4 = 2.7
100.0 + 0.0
Source tasks
5m vs 6m
53.9 = 5.1
60.0 = 3.6
9m_vs_ 10_vs_mmm
80.4 8.7
80.0 + 21.8
4m
95.3 3.5
92.5 + 4.7
5m
89.1 10.0
90.0 + 11.1
10m
93.8 2.2
74.4 + 27.8
12m
58.6 11.8
56.9 25.1
7m_vs_ vs_ 8m
25.0 15.1
33.8 + 17.7
8m_ 9m
19.6 6.0
32.5 + 10.8
Unseen tasks
10m_ vs_ 11m
42.2
7.2
36.2 - 25.8
10m_vs_ 12m
1.6
1.6
3.8 + 5.0
13m_vs_15m
2.3 2.6
0.0 + 0.0
H PERFORMANCE ON DIFFERENT SIZES OF DATASETS
When constructing the offline datasets, we choose the number of trajectories mainly based on recent offline MARL works (Yang et al., 202T) and the single-agem offline RL benchmark D4RL (Fu ct al. 2020). datasets D4RL. are transition-bar -based, we estimate the corresponding trajectory numbers based on the average episode length in our evaluated benchmarks. We finally chose trajectory number of 2000 for each task to compose multi-task offline datasets. We conduct additional experiments in the expert dataset of the marine-hare hard task set evaluate ODIS with datasets of 1000 trajectory numbers. The results are exhibited in Table 12 We find that ODIS-100U can also present a good performance, indicating that ODIS is not very sensitive to the data size.
Table 13: Test win rates of online QMIX algorithms in all test tasks. We abbreviate asymmetric tasknames for simplicity. For example, the task name "5m6m" denotes the SMAC map "5m_vs. mo__s_66m"
Task QMIX win rate %
3m
4m
5m
6m
7m
8m
9m
99.1
98.2
99.4
99.9
99.3
99.2
99.4
Task
10m
11m
12m
5m6m 71.9
6m7m 74.3
7m8m
8m9m
QMIX win rate %
99.2
99.4
99.6
88.8
93.1
Task QMIX win rate %
9m10m 94.3
10m1 1m 97.3
10m12m 42.7
13m15m 64.6
1s3z
1s4z
1s5z
98.5
97.5
95.3
Task QMIX win rate %
2s3z 96.0
2s4z
2s5z
3s3z
3s4z
3s5z
4s3z
95.1
94.4
95.1
87.0
95.1
94.6
Task QMIX win rate %
4s4z
4s5z
84.1
91.6
22

---

Published as a conference paper at ICLR 2023
I ADDITIONAL RESULTS ON MULTI-TASK OFFLINE LEARNING IN SMAC
As stated in the experiments section, we conduct experiments three designed task sets called marine-easy, marine-hard, and stalker-zealot, respectively. As we have presented the empirical results in the marine-hard task set in Table we here exhibit the results in the marine-easy and stalker-zealot task sets. The empirical results in the marine-easy task set are shown in Table [4 and the empirical results in the stalker- stalker-zealot zealot task set are shown in Table [5] As reference, we also exhibit the online QMIX performances all tasks Table 13 We find that ODIS generally outperforms other baselines in most source and unseen tasks. A specific case is that the performance of ODIS is not so promising in medium-replay data because the medium-replat data of marine-easy and stalker-zealot has very low qualities and thus hinders ODIS from discovering effective coordination skills. Despite all that, ODIS can still reach a similar performance compared to other baselines, indicating the validity of the ODIS algorithm. In addition, ODIS can achieve comparable performances to online QMIX in some unseen tasks with zero-shot generalization, showing the effectiveness of our proposed method.
23

---

Published as a conference paper at ICLR 2023
Table 14: Average test win rates in the marine-casy task set with different data qualities. Results of BC-best stand for the best test win rates between BC-t and BC-r (the same to Table 15
Expert
Medium
Task
BC-hest
UPD-T-
UPDeT-m
ODIS
BC-hest
UPDeT-
UPDeT-m
ODIS
Source tasks
97.7 +
19.7
60.2
5m
94.4 = 7.6
34.8 31.2
74.8 22.9
95.3 5.2
79.2 +
5.9
70.2 = 933
67.8 +
5.9
82.8
5.2
10m
86.1 = 22.7
39.6 48.2
83.6
19.2
88.3
20.3
63.1
35.9
10.5
48.8 +
7.9
71.9
6.6
Unseen Tisks
4m
91.2 - 1,6
18.8
+
4.1
53.0
32.3
90.6 + 7.0
62.5
11.6
26.6
= 23,4
41.7 + 17.4
63.3 + 16.1
75.3 1: 22.6
9.1
6.7
37.9
8.6
79.7
17.5
86.0
4.7
40.2
13.2
75.8
22.7
89.8
17.6
6.4
72.7
16,9
65.2
8m
74.7
+: 16.5
31.0
11.4
51.7
26.2
80.9
14.4
96.9
2.2
18.2
7.8
88.4
13.7
97.7 1
2.6
2.6
14.7
11m
83.3
+ 11.8
20.3
4.7
53.6
22.4
83.6 + 12.4
42.2
4.7
26.6
7.
23.4
11.
64.7
3.1
56.7
30.0
16.3
6.4
44.3
22.8
70.3
= 30.2
29.7
23.4
5.5
13.5
11.7
41.4
6.0
Medium-Export
Medium-Roplay
Seen Tasks
18.8
43.7
48.4 36.8
89.8
77.8
13.5
29.7 + 10.0
79.7 -
5m
74.0 +
2.9
45.6 27.1
64.1 17.9
83.7 16.0
5.5 +
5.6
16.1 = 27.9
6.2 : 10.8
3.1 3.1:5. 5.4
10m
90.6 1:
3.1
42.3 1: 26.0
23.8 68.8
93.8
0.0 1:
0.0 1: 0.0
0.0 d: 0.0
Unseen Tasks
4m
35.2 +: 38.0
53.5
+: 31.2
43.7
25.0
57.8
18.8
67.2
25.5
18.2
25.0 1: 22.6
25.0 15.0.155
6m
42.2
1.6
57.0
25.1
47.7
30.0
76.0
6.0
7.8
10.2
12.1
21.0
0.0 : 0.0
1:4
65.6
16.4
32.0
30.3
57.8
32.9
66.4
14.6
0.0
0.0 0.0
0.0
8m
40.3
1: 42.6
35.9
1 19.6
40.6
1: 19.3
43.8
11.5
0.8 1:
1.4
0.8 1
1.4
0.0 1 0.0
1.6 1.6
75.0
9.4
47.7
0.0
0.0
11m
55.5
12.4
51.6
35.9
85,9
14.2
68.8 + 20.3
0.0 -
0.0
0.0 = 0.0
0.0 = 0.0
0.0 0.0
12m
29.7
+: 29.8
19.7
+ 20.0
46.1
15.5
62.5
0.0 =
0.0
0.0 +: 0.0
0.0
Table Average test win rates in the stalker-zealot task set with different data qualities.
Expert
Medium
Task
BC-best
UPDeT-I
UPDeT-m
ODIS
BC-best
UPDeT-1
UPDeT-m
Source tasks
283z
93.1 + 4.6
53.1 39.1
50.0 + 33.4
97.7
2.6
48.8
9.8
30.6 12.7
35.0 15.0+20 23.0
49.2 = 8.4
2042
78.1
48.4 24.3
23.4 + 26.6
60.9
6.8
12.5 +
8.1
28.8
4.1
18.8
10.3
32.8
12,2
3652
92.5
4.2
40.6 11.5
17.2 + 19.8
87.5
9.6
24.4
+ 12.4
15.0
10.3
24.2 25.6
28.9
6.8
1832
45.6
23.8
26.6 25.0
1.6 +
1.6
76.6
3,5
21.9
+: 37.6
33.1
18.0
3.8
5.0
41.4
18.8
1s4z
60.0
32.3
37.5 31.9
26.6
1 19.3
17.2
10.5
6.2 1
7.7
35.0
7.2
2.5
3.6
50.7
7.5
1s52
45.6
+ 26.9
1488 = 13.9
29.7
+ 26.4
2.5
2.3
3.1 +
2.6
13.1
11.4
5.0
1
4.2
14.1
8.4
27.3
3337
80.6 + 9.1
49.2 25.8
20.3 + 10.9
89.1
5.2
45.6 +
14.6
23.8
6.7
24.4
28.6
23.4
9.2
364z
92.5
5944 164
12.5
2.2
40.0
17.5
28.8
18.8:00
31.6
50.8
4s32
67.5
+ 19.8
50.8
24.8
6.2
4
4.4
64.1
13.0
28.8
26.4
3.1
4.0
11.2
18.0
13.3
=
7.5
AwAz
53.1
18.4
41.4
16.0
7,8
13.5
79.7
10.9
20.0
12.0
191
2.5
1.2
1.5
12.5
7.0
4657
40.6
19.1
17.0
5.5
1:
7.8
86.7
12.6
14.4 1
8.5
5.0
5.4
5,6
8.5
As62
48.1
+ 23.8
10.9
7.2
4.7
1
6.4
88.3
8.4
3.8 1
3.6
2.5
2.3
1.9
1
2.5
1.6
1.6
Mediom-Exper
Medium-Replay
263r
57.5 4: 25.1
39.4 2 20.8
$7.5 + 27.1
58.6 = 15.5
3.1 2.6
6.9 10.7
14.4 13.2
15.6
18.2
204z
37.5
15.3
32.0
+ 24.6
7.8
6.2
12.5
3652
63.1 + 13.3
18.0 = 11.8
35.0 4: 23.5
41.4 = 18.5
31.3 11.3-:200 6.3
5.0 +
7.0
20.0
16.6
18.8 3.1
Unseen Tisks
1832
55.6
37.7
+:
184z
25.0
+:50+000
30.7
35.2 = 18.0
11.9 +
9.8
44.5
20.3
2.1
2.9
6.9
13.8
7.5
10.0
6,2 = 7.7
1:52
14.4
19.4
16.4
11.6
3.8 +
4.6
42.2
31.4
7,3
6.4
19
3.8
11.9
9.6
2552
26.9 1: 20.2
5.5
7.8
37.5
22.5
43,0
10.7
12.5
1:
15.5
44
88
20.0
168
14.1
367
35.6
1: 18.0
7.0 +
344
33.
15.0
50.0
13.3
35.4
12.1
56
7.0
17.5
12.3
25.0
20.1
3647
74.4
16.3
3.1
3.1
43.1
20.7
523
9.5
20.8
+
9.0
50
8.5
15.6
19.5
16.6
4632
69.8
+ 7.8
0.0 =
0.0
23.8
210
172
7.2
17.7
53
112
13.9
11.2
15.0
8.6
14,9
4647
41.9
311
20.3
68
38
5.6
465z
17.3
+ 5,3
08 =
14
11.9
16.1
21.9
2.2
1.0
15
3.1
4.0
10.6
19.7
08 1.4
4662
13.8
08
14
5.0
8.5
18.0
5.1
0.0 +
0.0
25
5.0
13.
24


Reviews:
[{"review_id": "oSSesUqus6", "reviewer": "Reviewer_1rgQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents ODIS, a new approach to learning policies for multi-agent, multi-task RL problems via skill-discovery and conditioning. Leveraging transformer models, the proposed approach can learn state-action-space invariant policies from offline data, and through additional loss functions the policies are encouraged to learn distinctive \"coordination skills\" that can be used to aid in action generation. Experimental evaluations show ODIS often outperforms simple BC baselines as well as other MARL approaches using comparable optimization steps.", "summary_of_the_review": "The paper presents an interesting approach that is clearly described and shows strong empirical performance relative to baselines that are given in the paper. The main weakness of the paper is that many of the relevant details of the method (network architecture, dataset/training details, experimental design/results, comparison to contemporary work, etc.) are relegated to an appendix. However, the main body of the paper does not often suffer for moving so much information to the appendix, and overall the submission is strong.", "strength_and_weaknesses": "Strengths:\n* ODIS is shown to perform better than baselines in multi-agent problems.\n* ODIS generalizes to new state-action space configurations, such as adding more Marines in the StarCraft II micro-control domains.\n* The appendix has a wealth of detail on the experimental design and results, additional results and experiments, and network architecture and training details.\n* The approach is relatively clear and the loss functions are well-introduced and explained.\n* Contributions are evaluated with ablation studies across new objectives, showing the importance of the consistent loss that is introduced as part of the method.\n\nWeaknesses:\n* The experimental setup and results are jumbled together in a somewhat confusing way. Section 5.1 introduces the baselines and describes how they relate to ODIS, and at the same time it is delivering results and discussion. There is a lot of information for the reader to make sense of all at once, without having a full-picture of the results or baselines.\n* The experiments and introduction mention an additional navigation task, but it is not present in the main body of the paper.\n* The main body of the paper references and discusses a result that is only present in the appendix (Figure 7).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "P6d4EBxK2n", "reviewer": "Reviewer_e7iL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to discover generalizable multi-agent coordination skills from multi-task offline data (multi-task marl with offline data). The insight is that we can extract universal skills for coordination from offline multi-agent multi-task data. The authors propose the ODIS algorithm which consists of two stages. Firstly, task-invariant coordination skills are extracted from the offline multi-task data, and it learns to delineate different agent behaviors with the discovered coordination skills. Secondly, a coordination policy is trained to choose optimal c coordination skills under CTDE. The authors conduct extensive experiments based on several maps from SMAC to validate the effectiveness of ODIS.", "summary_of_the_review": "In all, the paper studies an intersting and new setting, which aims to improve generalization in MARL via offline multi-task multi-agent data. The proposed method is also reasonable, with significant improvements over baselines.", "strength_and_weaknesses": "### Strengths\nThe paper tackles an important problem in MARL - generalization to tasks with varying agents and targets based on the recent success of offline MARL algorithms that leverage the offline dataset (compared to a fully online way). The setting is novel and new. The proposed method is also reasonable and clearly written, which is motivated well. Figure 1 does a good job in motivating the problem and the proposed method, where ODIS is responsible for generalizing to the unseen 10m task given offline data from source tasks like 5m and 8m via discovering coordination skills and learning a coordination policy. The paper is clearly written and easy to follow.\n\n### Weaknesses\nOne minor concern for the paper is to evaluate ODIS in maps that is more difficult (like corridor and 2c_vs_64zg), as the experiments are mainly focused on easy maps.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OO33oyPe0u5", "reviewer": "Reviewer_E6Nd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel algorithm, ODIS, for learning multiple multi-agent tasks from an unlabeled offline dataset. The algorithm is broken up into two phases. In the first phase, a skill extractor is learned from global states, and the encoded skills are fed to per-agent action decoders. In the second stage, per-agent observation encoders ", "summary_of_the_review": "Overall, this paper proposes a novel idea with promising results in an important field, and it is well written and well supported. While I have some minor questions, I would recommend this paper for acceptance. ", "strength_and_weaknesses": "Strengths: \n\n1) This paper addresses the intersection of three important areas of research - multi-agent, multi-task, and offline RL. It is the first paper to directly address these areas and notably outperforms single-task offline MARL methods. \n\n2) The paper is well written, with thorough experiments, appropriate baselines, and a variety of tasks across two distinct environments.\n\nWeaknesses:\n\n1) The method requires access to global state to learn the skill extractor. It's not clear to me why learning observation encoders and a mixing network, as is done in the coordination policy, is not sufficient, and some intuition would be helpful. \n\n2) The authors state that the distribution of skills is prevented from collapsing by maximizing KLD with a uniform skill distribution. While Figure 3 seems to support this, learning diverse skills automatically has historically been a difficult problem [1, 2], and I would appreciate some further discussion or ablations around this. \n\n3) In most tasks, the results are only slightly better than behavioral cloning, even on the lower quality datasets. \n\n4) The transformer-based architecture is closely related to [3], and I would recommend it be added to the literature review. \n\nQuestions:\n\n1) How many evaluation trajectories are run for each result?\n\n2) Is any parameter sharing used in the observation encoder or action decoder?\n\n3) Are there any results comparing IQL or QPLEX on other datasets besides medium? \n\n4) Does freezing the last layer of the skill encoder affect the performance of the skill classifier? \n\n5) Is there a reason to not standardize the size of the medium-replay dataset, using the same construction described? This would make the results more comparable to the other datasets.\n\n[1] Eysenbach, B., Gupta, A., Ibarz, J., & Levine, S. (2018, September). Diversity is All You Need: Learning Skills without a Reward Function. In International Conference on Learning Representations.\n\n[2] Haarnoja, T., Hartikainen, K., Abbeel, P., & Levine, S. (2018, July). Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning (pp. 1851-1860). PMLR.\n\n[3] Iqbal, S., & Sha, F. (2019, May). Actor-attention-critic for multi-agent reinforcement learning. In International conference on machine learning (pp. 2961-2970). PMLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "NI9Vo6YTDvN", "reviewer": "Reviewer_PuN1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Although there have been studies that extended MARL into offline learning concepts and multi-tasking learning, no studies have attempted both simultaneously. Thus, the current research is meaningful in that it tries to tackle both offline and multi-task learning simultaneously in MARL. The principal methodology that the current study uses is inferring the latent vector that can possibly represent the essential skills for inducing coordination among agents and using these inferred skills in independent decision-making.", "summary_of_the_review": "Although it is a very challenging study that combines offline learning and multi-task learning with MARL, the methodological novelty is not great. In addition, too much overstatement of the latent variable z.", "strength_and_weaknesses": "To conduct multi-task learning, the current study utilizes a kind of high-level action called a skill that co-exists in multiple tasks. In addition, to conduct offline learning, the current study uses imitation learning with a conservative Q-value estimation technique. Each technique is already an existing methodology, and it seems this study used these two techniques simultaneously to achieve two purposes. \n\nHowever, this study did not formalize or discuss additional issues that may arise when both cases occur simultaneously. Both problems have a common issue of distribution shift, but no means to overcome this problem is specifically suggested.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster