Multi-Agent Policy Transfer via Task Relationship Modeling
Authors: Rong-Jun Qin
Feng Chen
Tonghan Wang
Lei Yuan
Xiaoran Wu
Yipeng Kang
Zongzhang Zhang
Chongjie Zhang
Yang Yu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-agent reinforcement learning
cooperative transfer learning

Abstract: Team adaptation to new cooperative tasks is a hallmark of human intelligence, which has yet to be fully realized in learning agents. Previous works on multi-agent transfer learning accommodate teams of different sizes, but heavily rely on the generalization ability of neural networks for adapting to unseen tasks. We posit that the relationship among tasks provides the key information for policy adaptation. To utilize such relationship for efficient transfer, we try to discover and exploit the knowledge among tasks from different teams, propose to learn effect-based task representations as a common latent space among tasks, and use it to build an alternatively fixed training scheme. We demonstrate that the task representation can capture the relationship among teams and generalize to unseen tasks. As a result, the proposed method can help transfer learned cooperation knowledge to new tasks after training on a few source tasks, and the learned transferred policies can also help solve tasks that are hard to learn from scratch.

Under review as a conference paper at ICLR 2023
MULTI-AGENT POLICY TRANSFER VIA TASK RELATIONSHIP MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
Team adaptation new cooperative tasks is hallmark of human intelligence, which has yet to be fully realized in learning agents. Previous works on multi-agent transfer learning accommodate teams of different sizes, but heavily rely on the generalization ability of neural networks for adapting to unseen tasks. We posit that the relationship among tasks provides the key information for policy adaptation. To utilize such relationship for efficient transfer, we try to discover and exploit the knowledge among tasks from different teams, propose to learn effect- based task representations as common latent space among tasks, and use it to build an alternatively fixed training scheme. We demonstrate that the task representation can capture the relationship among teams and generalize to unseen tasks. As result, the proposed method can help transfer learned cooperation knowledge to new tasks after training few source tasks, and the learned transferred policies can also help solve tasks that are hard to learn from scratch.
INTRODUCTION
Cooperation in human groups is characterized by resiliency to unexpected changes and purposeful adaptation to new tasks (Tjosvold) 1984). This flexibility and transfcrability of cooperation is hall- mark of human intelligence. Computationally, multi aulti- reinforcement learning (MARL) (Zhang et al. 2021a) provides an important means for machines to imitate human cooperation. Although recent MARL research has made prominent progress in many aspects of cooperation, such as policy decentralization (Lowe et al., 2017, Rashid et al. 2018 Wang et al. 202 a Cao et al. (021).
communication (Foerster al. 2016 Jiang & Lu 2018), and organization (Jiang et al. 2019] Wang
al. 2020a; 2021b), how to realize the ability of group knowledge transfer is still an open question.
Compared to single-agent knowledge reuse (Zhu et al. 2020), a unique challenge faced by multi- agent transfer learning is the varying size of agent groups. The number of agents and the length of observation inputs in unseen tasks may differ from those in source tasks. To solve this problem, existing multi-agent transfer learning approaches build population- -attttiiiioniinnnnntt (Long et al. 2019) and
input-length-invarian (Wang et al. 2020c) learning structures using graph neural networks (Agarwal al. (020)) and attention mechanisms like transformers (Hu et al.) 2021 Zhou et al. (02T). Although these methods handle varying populations and input lengths well, their knowledge transfer to unseen tasks mainly depends on the inherent generalization ability of neural networks. The relationship among tasks in MARL is not fully exploited for more efficient transfer.
Towards making up for this shortage. we study the discovery and utilization of common structures in multi-agent tasks and propose Multi-Agent Transfer reinforcement learning via modeling TAsk Relationship (MATTAR). In this learning framework, we capture the common structure of tasks by modeling the similarity among transition and reward functions of different tasks. Specifically, we train a forward model for all source tasks to predict the observation, state, and reward at the next timestep given the current observation, state, and actions. The challenge is how to embody the similarity and the difference among tasks in this forward model, we specifically introduce difference by giving each source task a unique representation and model the similarity by generating the parameters of the forward model via a shared hypernetwork, which we call the representation explainer.
To learn well-formed representation space that encodes task relationship, alternative-fixe training method is proposed to learn the task representation and representation explainer. During training, representations of source tasks are pre-defined and fixed mutual orthogonal vectors,

---

Under review as a conference paper at ICLR 2023
Predittonn
TD
Prodidion loss
Envionment.
Predictons v/,
Qut
Prediations of e/,0.r
Popu atton lina lant Polky
Forward
Expairer
Invoriant
(foed)
5, a
0
5.0, a,.0
Training
Task resrescrtaation
Phase
Police
Shep repremarzation learnigg
Phase
Step Policy eoccution
Figure 1: Transfer learning scheme of our method. The black arrows indicate the direction of data flow and the red ones indicate the direction of gradient flow. The dashed arrows indicate the flow between the hypernetwork and the generated network.
and the representation explainer learned by optimizing the forward model prediction loss on all source tasks. When facing an unseen task, we fix the representation explainer and backpropagate gradients through the fixed forward model to learn the representation of the new task by few samples. Furthermore, design population-invariant policy network conditioned on the learned task representation. During policy training, the representations for source tasks are fixed, and the policy is updated to maximize the expected return over all source tasks. On unseen task, obtain the transferred policy by simply inserting the new task representation into the learned policy network.
On the SMAC (Samvelyan al. 2019) and MPE (Lowe al. 2017) benchmarks, show that the learned knowledge from source tasks can be transferred to a series of unseen tasks with great success rates. We also pinpoint several other advantages brought by method. First, fine- tuning the transferred policy on unseen tasks achieves better performance than learning from scratch, indicating that the task representation and pre-trained policy network provide a good initialization point. Second, training on multiple source tasks gets better performance compared to training on them individually and other multi-task learning methods, showing that MATTAR also provides method for multi-age agent multi-task learning. Finally, although not designed for this goal, our structure enables comparable performance against single-task learning algorithms when trained on single tasks.
empirically
2 METHOD
In this paper, focus knowledge transfer among fully cooperative multi-agent tasks that can be modeled as a Dec-POMDP (Oliehoek & Amato 2016) consisting of a tuple G=(T, A, P. R, O,n,7) 7). where / is the finite set of agents, ronment, and y E [0, 1) is the discount factor. At each timestep, each agent receives an observation o E Ω drawn according actions form a joint action A", which leads to a next state s' according to the transition function P(s'ls, a), and reward
S the state of the envi-
the observation function ( i) and selects an action a, E Individual
R(s, a) shared by all agents. Each agent has local action action-observatiio history T ( A)* x Ω. Agents learn to collectively maximize the global action-value as) = 8, ag al (a little notation abuse: the subscript here for and indicates the timestep while the subscript for observation and action elsewhere in this paper indicates the index of the agent).
function Qtor (s, a) = Eso.s no..0 7
Overall. framework first trains several source tasks {S.} and then transfers the learned cooperative knowledge to unseen tasks [T,}. As shown in Fig. our learning framework achieves this by designing modules for (1) task representation learning and (2) policy learning. In the following sections, we first introduce how we design the representation learning module and its learning scheme in different phases. Then, we describe the details of policy learning, including the population-invarian structure for dealing with inputs and outputs of varying sizes.
2.
TASK REPRESENTATION LEARNING
Our main idea in achieving knowledge transfer among multi ageent tasks is to capture and exploit both the common structure and the unique characteristics tasks by learning task representation. A task distinguishes itself from other tasks by its transition and reward functions. Therefore, incorporate
2

---

Under review as a conference paper at ICLR 2023
Qro
Qtor(T,
Q:
Pt
Dot Product
Population-
MLP
Invariant Mixing Layers
Population-Invatt Invarinnt Mixing
Softmax
Task
MLP
Scaled Dot Product
Qi QL
MLP
Task Rep.
Input-Length-Inwwn Innaaraan Q-Network
one onwn other
Attention
MLP.P
MI.Py
ghr
endly
Self-Attention
Of
dew Di
of
of
Population Invariinnttnattttttt Mixing
(b) Overall Framework
Input-Length- Invariant Q-Network (d) Attention
Figure 2: Population- invarriann network structure for policy learning.
task representation learning component into the learning of forward model that predicts the next state, local observations, and reward given the current state, local observations, and actions.
We associate each task i with representation 2; IR and expect it to reflect the relationship of the transition dynamics of tasks. For modeling task similarity, all source and unseen tasks share a representation explainer, which takes as input the task representations and outputs the parameters of the forward model. We then train the representation explainer on all source tasks. Concretely, for a source task S, parameters of the forward model are generated 7/s fo(2z), where 0 denotes the parameters of the representation explainer. The forward model contains three predictors: J f and fr Given the current state s, agent j's observation oj. and action aj. these predictors estimate the next state s', the next observation of. and the global reward r, respectively.
A possible method for training on source tasks is to backpropagate the forward model's prediction error to update both the representation explainer and task representations. However, in practice, this scheme leads to representations with very small norms, making it difficult to get a meaningful representation space. To solve this problem, we propose pre-determining the task representation for each source task and learning the representation explainer by backpropagating the prediction error. Such method can help form an informative task representation space and build a mapping from task representation space to the space of forwarding model parameters. In practice, we initialize source task representations as mutually orthogonal vectors. Specifically, we first randomly generate vectors in R for source tasks, and then the Schimidt orthogonalization (Björck [1994) these vectors to obtain source task representations. With the pre-defined task representations, the representation explainer is optimized to minimize the following loss function:
(()) = -
Nore Js, (0).
where Nar is the number of source tasks, and
Js. (9) ED Σ, [ll,e (s, (s, 0z) s'12 A 11,
+2112+>e(f, (s, (s)))))))))) r]
is the per-task prediction loss. Here, D is the replay buffer, and λ1, A2 are scaling factors.
We fix the source task representations and learn the representation explainer during the training phase. In the current implementation, the training data here is collected with the uniform random policy; the data the same for task representation learning in the transfer phase. We find it simple and effective in the experiment, while more efficient approaches may be adopted here. When it comes to the transfer phase, we aim to find good task representation that can reflect the similarity of the new task to source tasks. To achieve this goal, we fix the trained representation explainer and learn the task representation by minimizing the prediction loss of the forward model on the new task. Specifically, we randomly initialize a task representation keep 0 fixed, and get the forward model parameterized by 1 fo(=). Then the task representation z updated by backpropagating the prediction loss for transition and reward functions through the fixed fn-
To keep the new task representation the well-formed space of source task representations, we learn the new task representation linear combination of source task representations:
ENer pizz i4 2 0. ENe p4
3

---

Under review as a conference paper at ICLR 2023
In this way, what are learning is the weight vector μ. To make the learning more stable, additionally optimize an entropy regularization term H((u). The final loss function for learning & is:
JT (x)==A((p)) + Σ, (1,f(((s,,,,))-5((1111 -)) [1 The detailed architectures for task representation learning are described in App.
2.2 TASK POLICY LEARNING
After the task representation is learned by modeling the transition and reward functions, it will be used to learn and transfer the policy on the source and unseen tasks.
A difficulty faced by multi-agent transfer learning is that the dimensions of inputs and outputs vary across tasks. We use a population-invariant network (PIN, Fig. ( to solve this problem. The idea is not novel, and our contribution here is light-weigh structure that achieves comparable better performance to other complex PIN modules (Igbal et al. 2021). The single-task experiments in App. demonstrate its advantage as the performance gains mainly come from our PIN design. Our PIN uses the value decomposition framework and consists of two main components: an individual Q-network shared by the agents, and a monotonic mixing network (Rashid al. 2018 that learns the global Q-value as combination of local Q-values
For the individual Q-network, like in previous work (Iqbal al. 202 we decompose the ob- servation o; into parts relating to the environment o agent itself orwn, and other entities of oother other s }. We adopt attention mechanism get a fixeddimeesionas embedding h: {o;
MLP ((0, SnV
own
K MLP ((o otheer
other; .....),
V = MLP ([ogtherr
other,
D), h sof softmax( de),
where [ is the vector concatenation operation, dg is the dimension of the query vector, and bold symbols are matrices. Embedding h is then fed into the subsequent network together with the task representation 2 for estimating action values. For the mixing network, we decompose the state s into parts relating to different entities gentity and the environment gemv We apply similar self- attention module to integrate information from these parts of the state and obtain a fixed-dimensional embedding vector /tttate Astate, the task representation 2, and gentity are used to generate the parameters of the mixing network. In some multi-agent tasks, the number of actions also varies in different tasks. We use mechanism similar to other popular population-invarian networks (Wang et al. 20206 Hu et al. 2021) to deal with this issue, which is discussed detail in App. G
During the whole process of policy learning, fix the task representation Compared to policy learning, which typically lasts for 2M timesteps, the training of task representation costs few samples. In practice, we collect 50K samples for learning task representations and the representation explainer. When transferring to new tasks, we use the individual Q-network and the representation explainer trained on source tasks. We learn the task representation for 50K timesteps and insert into the individual Q-network. Agents execute in decentralized manner according to this Q-network.
CASE STUDY
To offer an intuitive understanding of the task representation learning, we design simple case study to show how MATTAR works, which refers to two-player navigation problem. In this experiment setting, each task corresponds to goal position and two agents are expected to cooperatively navigate to the specific goal from an initial region. We design eight source tasks with different goals and one unseen target task. Detailed deseriptions of the experiment setting can be found in App. B
In the training phase, we learn the representation explainer and the agent policy on eight source tasks. Afterward, we optimize the task representation on the unseen task and transfer the learned policy. In Fig. 3(a) we study the learned representation for the unseen target task. We can find source tasks 4 and 3 occupy the largest two coefficients, which are about 70% and 30%, respectively. Actually, relevant, their goal positions closer to that of the target unseen task compared to other source tasks, and our mechanism for task representation learning captures the similarity between their reward functions. To confirm hypothesis, embed the learned task
these two source tasks
4

---

Under review as a conference paper at ICLR 2023
Leurned coeflicians
Mudel pelliction
0.8
Model prediction error
Learned coefficients
0.6
0.0
task-4 tosk-5 talk-6
Source tasks
unseen lask goal
initial region for agents
(a) The Iearned task representations.
Visualization of the task trajectorics.
Figure 3: The results of the case study on the two-player navigation problem. (a) The task repre- sentation learned by MATTAR. (b) We draw roll-outed trajectories of agent policy on source tasks (represented by circular symbols) by arrows, with different colors for different tasks. For simplicity, we only selectively show trajectories for five source tasks. Besides, we test MATTAR on one unseen task and draw the trajectory with continuous red dashed arrow. More details can be found in App. B
representation into the representation explainer and test the prediction error the obtained forward model in each source task. The results exhibit that lower prediction errors obtained in tasks 4 and 3, implying that the learned representation can reflect different tasks (e.g., ., the rewards in these tasks).
Furthermore, visualize the task trajectories roll-outed in Fig. 3(b) Capturing the similarity between the unseen target task and source tasks by task representation, the transferred policy reuses the policy in relevant source tasks, When embedding different source task representations, the agent policy exhibits different behavioral strategies to reach the corresponding goal. When encountered an unseen new task, the learned task representation guides the agent to move up first, which typical behavior in task 4, and further to the target position when the goal is within sight, demonstrating MATTAR can indeed extract common latent representation among tasks to ease solving unseen tasks.
4 EXPERIMENTS
In this section, design experiments to evaluate the following properties of the proposed method: (1) Generalizability to unseen tasks. Our learning framework can extract knowledge from multiple source tasks and transfer the cooperation knowledge unseen tasks, and task representations play an indispensable role in transfer (Sec. 4.J). (2) A good initialization for policy fine-tuning. Fine-tuning the transferred policy can succeed in super hard tasks, which can be solved when learning from scratch (Sec. (4.)). (3) Benefits of multi-task training. Our multi-task learning paradigm helps the model better leverage knowledge of different source tasks boost the learning performance compared to training on source tasks individually (Sec. (.3)) (4) Performance advantages on single tasks. Although we did not design our framework for single-task learning, our network performs better against the underlying algorithm (App. H).
We evaluate MATTAR on the SMAC (Samvelyan et al., 2019) and MPE (Lowe et al. 2017) benchmarks. To better fit the multi-task training setting, we extend the original SMAC maps and sort out three task series, which involve various numbers of Marines, Stalkers/Zealots, and Marines/Maneuvers//Mciivaaa
respectively. For the MPE tasks, test a discrete version of Spread and Gather, with different numbers of agents trying to cover all or one landmarks. The detailed environmental settings described in App. C To ensure fair evaluation, we carry out all the experiments with five random seeds, and the results are shown with a 95% confidence interval in all figures. For a more comprehensive description of experimental details, please refer to App. E
4.1 GENERALIZABILITY TO UNSEEN TASKS
As the major desiderata of our method, we expect that MATTAR has better transfer performance on unseen tasks. We note that there are few related works applicable tasks of varying sizes, and
5

---

Under review as a conference paper at ICLR 2023
Table Transfer performance (mean win rates with variance) on the first series of SMAC maps.
Source Tasks
Unseen Tasks
2633
345z
3657, 3062
Is8z
1s9z
248z
2:9z
7s3z
MATIAR
1.00::0.00
0.99::0.01
0.48:20.13
0.79:10.00
0.60:20.112
0.93-20.09
0.84 20.04
0.16:10.11
rep.
0.99-10.01
0.96:10.02
0,20:10.08
0.12-10.13
0.07::0.12
0.47:10.18
0.25=0.20
0.15:100.19
O uukk rep.
0,23:1:.112
0.08
0.00
0.02-1.0.33
0.00:100000
0.01110.01
0.02-00.02
0.00:1:0.00
UpDeT-b
0.9440.04
0.86::0.13
0.09::0008
0.16::0.11
0.11:0.10
0.29::0.22
0.15=0.13
0.02:40.04
L REFIL.
0.60.10.11
0.47.10.15
0.03.10.03
0,08.10.06
0.04
0.14:10.12
0.06
0.01110.01
0.75:10.09
0.43:0.133
0.01
0.08:10.04
0.03:20.01
0.08
0.05-0004
0.06:0.004
Table 2: Transfer performance (mean win rates with variance) the second series of SMAC maps.
Source Tasks
Unseen Tasks
MMM2
MMM4
MMMU
MMM1
MMM3
MMM5
MMM6
MATTAR
1.00:10.000
0,92+0,20
0.93-00122
0.98-0002
0.97:20.04
0.86=0.10
0.47=0.15
0.09-0002
w/o task rep.
0.94::0.05
0.23::039
0.33=00.25
0.81
0.37=0..66
0.07=0.05
0.22=0.30
0.09
O tusk rep.
0.61.10.00
0.071.0.6
0.21
0.28 0..1
0.11.10.13
0.08 0.08_0.10
0.08
0.02.10.04
UPD:T-b
1.00110.00
0.782:004
0.41
0.73:10.11
0.8410.07
0.57=0.15
0.00=0000
0.00
UPDeT-m 0.48.1.0.03
0.15
0.20:10.07
0.3010.16
0.27.10.13
0.28 0.08
0.00=00.00
0.00
REFI.
0.97::0.01
0.0410002
0.06-10.03
0.93-10.22
0.38
0.12
0.00=0.00
0.00
Table 3: Transfer performance (mean win rates with variance) on the third series of SMAC maps.
Source Tasks
Unseen Tasks
9m
5m
6m
MATTAR
1.00:10.000
0,72+0.05
0.83=0005
0..09
0.94-00.27
0.97=0.02
0.04=0.05
1.00=0.00
w/o task rep.
0.97-10.01
0.01
0.01=00011
00003
0.86:00.33
0.88=0.04
0.00=0.00
0.95
O tusk rep.
0.78.1..339
0.16110.1
0.3011.22
0.40:10.28
0.00.10.00
0.21 0.21
0.01
0.67:10.47
UPD:T-b
1.00-10.000
0.93-00.5
0.1100.19
0.94=0.04
0.81:00088
0.95=0.06
0.29=0.17
1.00=00.00
UPDeT-m
0.77::0.09
0.32+0.03
0.35=0005
0.43=00.22
0.36:20.04
0.57=0.03
0.10=0.06
0.91=00099
REFI.
0.73::0.03
0.00
0.01
0.03
0.68
0.74=0.02
0.00=0.00
0.71
Unseen Tasks
6m
7m
7m,8m
8m
9m
9tn, 10m
10m
10m_12m
MATTAR
0.74::0.15
1.00:0.000
0.83=0.04
1.00=0.000
1.00:00.00
0.84=0.09
1.00=0.00
0.07=0001
w/o task rep.
0.03:1.0.02
0,9410003
0.08
0.93:10.04
0.86.10.05
0.04=0002
0.52-20.22
0.00110.00
0 task rep.
0.311:0.22
0.67:0047
0.49=00.5
0.67:00.7
0.66:10.46
0.32=0.24
0.65=0.46
0.00=:0.00
UPDeT-b
0.78:20.05
0.9920.01
0.73=0.11
0.99:20.22
0.99:20.01
0.80=0.16
0.99-0.01
0.07=0.04
UPDeT-m
0.35:t0.10
0.92-00.03
0.38=0.05
0.83=0005
0.66:0.111
0.33=0.09
0.17=0.08
0.03=00.22
REFIL
0.01:100000
0.66:10.03
0.01
0.63:20.05
0.55:10.05
0.01=0000
0.46::0.02
0.00::0.00
compare our method against the state-of-the-art multi-agent transfer method UPDeT (Hu ct al. 202 and REFIL (Igbal et al.) 2021). UPDeT transfers knowledge from a single source task. For fair comparison, we transfer from each source task to every unseen task and calculate the best (UPDeT-b) and mean (UPDeT-m) performance on each unseen task. For the test phase, we conduct transfer experiments on both source tasks and unseen tasks. Results on the SMAC benchmark are shown in Tab. and those the MPE are shown in Tab. and 5
We find that MATTAR shows superior transfer performance on unseen tasks, significantly outper- forming UPDeT-b and REFIL, especially in complex settings requiring sophisticated coordination like the MMM series. MATTAR is effective even for unseen tasks that are very different from source tasks, For example, the MATTAR policy learned on 2s3z, 3s5z, and 3s52_ve vss 3s6z can win 79% of games on 1s82. On MPE tasks, MATTAR also shows good generalization performance on unseen tasks with different numbers of agents and overall outperforms the baselines and ablations. For example, On Spread tasks, MATTAR exhibits significant advantage over UPDeT on all unseen tasks and outperforms REFIL with a margin of 13% on the unseen task with 6 agents.
To explain the performance of our method, we first check the learned representations on unseen tasks and then carry out ablation studies.
What representations are learned for unseen tasks? When encountering unseen task, we first learn its representation as a linear combination of the representations for source tasks. Specifically, we directly update the coefficients of this linear combination by backpropagating the prediction error of the forward model. For deeper understanding of how our method transfers the learned knowledge, here investigate the learned coefficients of the linear combination.

---

Under review as a conference paper at ICLR 2023
Table Evaluation on MPE: Transfer performance (mean success rates with variance) on Spread with different numbers of agents.
Source Tasks
Unseen Tasks
9
3
8
MATTAR
1.00::0.00
0.97=00.33
0.17=0.16
0.12+0005
0,98:20.01
0.75=0.12
0.19=0.10
0.09=00099
wio task rep.
1.00=10.00
0.55==0.30
0.02==0.01
0.00=0.00
0.83
0.30=0.13
0.03=0.04
0.00=00.00
0 task rep.
0.98::0.01
0,96=0.03
0.13=0.18
0.09=0004
0.94=0.07
0.70=0.17
0.09=0.09
0.09+0.07
UPDeT-b
0.73+:0.23
0.09:0000
0.02=00022
0.00+0000
0.41=0.16
0.05=0.02
0.00=0.00
0.02=00.22
UPDeT-m REFI.
0.39:10.12
0.06110.00
0.001..000
0.00110.00
0.18:10.08
0.02-0001
0.00
0.01 1.001
1.00:10.000
0.9320.04
0.07
0.00
0.96:0.03
0.73-0.06
0.06=0.05
0.05
Table Evaluation on MPE: Transfer performance (mean success rates with variance) on Gather with different numbers of agents.
Source Tasks
Unsenn Tasks
15
MATTAR
1.00.0.000
1.00:0.00
0.82:10.11
0.84
100100.00
1.00:0.00
0.63-0.12
0.55 0.222
wio task rep.
1.00-10.00
1,00:0000
0.88-=0.09
0.84-1.177
1.00+:0.00
0.99=0.01
0.62=0.13
0.53-00.18 18
0 task rep.
1.00::0.00
1.00=0.000
0.73-0.26
0.75=0.18
1.00:00000
0.99=0.01
0.54=0.26
0.50=0.29
UPDeTh
0.66-10.34
0.3811038
0.25=0.12
0.09
0.6110033
0.36=0.27
0.03-0.03
0.06::0.06
UPDeT-m 0.32::0.16
0.15
0.07=00.02
0.04
0.20:20.10
0.11=0.07
0.01=0.01
0.02=00.22
REFIL
1.00:10.00
1.00:00000
0.99=0002
0.99
1.00=1.00
1.00=0.00
0.73=0.14
0.51
In Tab. 6 we show the coefficients of two unseen tasks for each series of maps. We observe that the largest coefficient typically corresponds the source task that is the most similar the unseen task. For example, 5m is the closest source task to the unseen task 4m, and the coefficient of 5m is 61%. There are also some exceptions. For example, for the unseen task 10m vs 12m, the coefficients of two source tasks, 5m and 10m vs 11m, are equal, and they together take up 86% of all the coefficients. While 1 10m_vs_11 1m is very similar to 10m_vs_12m as the team composition similar, the policy for solving 5m is different from that for 1 10m_vs_ _2mm we find that agents first form some groups to set up an attack curve quickly in 10m_vs_12m. Therefore, on a local battlefield, there are around five allies fighting against similar number of enemies. In this case, the policy learned from 5m can be used locally to promote coordination in 10m.vs.1222
We conclude that, in MATTAR, task representation learning captures the similarity of task dynamics and thereby discovers opportumities of policy reuse. These representations help unseen tasks effec- tively leverage the knowledge from the most similar source tasks reuse the knowledge from mixing of source tasks, which is also consistent with the results in case study (Sec.).
Ablations. To further investigate the role of task representations in our method, we design two ablations. (1) 0 task rep. uses the trained MATTAR models but feed a zero-valued task represen- tation into the policy network when transferring.
(2) w/o task rep. denotes completely remove components related to task representation. including the forward model, the representation learning process, and the task representation in the policy network. Train and transfer the appulation-nvvariann policy. This ablation can reveal the generalization ability of the policy network itself.
We can see that these ablations bring about large drop in the performance of MATTAR on nearly all unseen tasks. For example, after trained on 2s3z, 3s5z, and 3s5z_vs 11.52vv__330 3s62,, MATTAR achieves a win rate of 0.79 on 1s8z, while w/o task rep. obtains win rate of 0.12 and 0 task rep. does not win any games. We can thus conclude that task representations play an indispensable role in policy transfer.
Results are shown in Tab. I 3
4.2 A GOOD INITIALIZATION FOR POLICY FINE-TUNING
When evaluating the performance of MATTAR on unseen tasks, we only train the task representations and reuse other parts of the policy. In this section, we investigate the performance of MATTAR after fine-tuning the policy. Specifically, train the task representation 2 for 50K timesteps. Then we fix the learned task representation and fine- tune the policy for 2M timesteps. We compare against learning from scratch in Fig. Learning from scratch means the policy is randomly initialized and trained for 2M timesteps. We consider two ver- sions for learning from scratch. The first version (Learn from scratch w/o repr.) totally removes
unseen task, we first fix the representation explainer and
7

---

Under review as a conference paper at ICLR 2023
Table 6: Our method models task relationship and exploits it for knowledge transfer. Task representations of unseen tasks are learned as linear combination of source tasks' representations, whose coefficients are shown in this table. These weights reveal the encoded task relationship.
Source
Unseen Tasks
Source
Unscen Tasks
Source
Unseen Tasks
4m
10m 12m
3s4z
3s57_3s7z
MMM0
MMM6
5m
0.61
0.43
2s3z
0.21
0.18
MMM
0.44
0.30
5m_6m
0.13
0,07
3s5z
0.59
0.21
MMM2
0.15
0.14
8m-9m
0.14
0.08
3s52_3s6z
0.21
0.61
MMM4
0.40
0.56
10m_1 11m
0.12
0.43
Modian Text Win Raae 5/
MATTAR fine-tminng
80
Malian Teat Win Raae
30
30
Lean from seatch
Loan from amatch (wclo repr.
80
80
60
QMIX
QPLEX
40
0.0 0.4M
0.3M
1.2M 1.M
2.0M
80M
0.4M
0SM
1.2M 1.0M
2.0M
0.0
944
0,8M
1.2M
1.0M
2.0
Timestops
Timestops
1 Liinestogs
(a) 10m_vs__me 12m
(b) MMM6
(c) 3s52_v8_3s77
Figure 4: On unseen tasks: task representations provide a good initialization. Fine-tuning the policy can learn cooperation policies effectively than learning from scratch.
task representations from the policy, while the second version (Learn from scratch w/ repr.) inserts the same task representation as in the fine-tuning experiments.
We observe that the task representation and the reused policy provide good initialization. For example, on 10m_ve som_vs__uu saa_vs_12m, 12m, after fine-tuning for 2M timesteps, MATTAR ainettnnig converges to win rate of around 0.86, while both versions of learning from scratch can only achieve a win rate of about 0.4. Furthermore, empirically only MATTAR 3s52.vs 3s72, a task harder than the super hard map 3s52.v8.3s62.
achieves non-zero winning rates on
4.3 BENEFITS MULTI-TASK LEARNING
MATTAR adopts scheme where multiple sources are learned simultaneously. Our aim is to leverage knowledge from more tasks and to be able to generalize the learned knowledge to a larger set of unseen tasks. Empirically, we find that this multi-task training setting helps not only unseen tasks but also the source tasks themselves.
In Fig. [ we present the performance of MATTAR on source tasks when training with multiple and single source task. The experiments are carried out on three tasks from three different series. We that training multiple tasks significantly boosts learning performance. For example, $s52.vs. 3s6z, after 2M training samples, MATTAR with multiple tasks converges to the win rate of around 0.6, while training solely on this task can only achieve win rate of about 0.05. Furthermore, we can also observe that MATTAR significantly outperforms the state-of-the-art deep multi-agent multi-task method (REFIL (Iqbal al., 202T)). These results reveal that MATTAR also provides good learning framework for multi-agent multi-task learning. It can leverage experience on other tasks to improve performance on similar task.
RELATED WORK
Multi-task learning with metadata (You et al. 2016 Zheng et al. 2019) approaches have been explored to exploit richer context information in the form of learned task embeddings, with focus on task relation discovery. (Sohhani et al. (021)) use context information high-level, under-spe and unstructured natural language for multi-task learning. Compared the settings studied in this paper, multi palti- problems pose additional challenges. Transition dynamics and reward functions in multi-agent tasks are influenced by the interaction among all agents the relation between tasks has
8

---

Under review as a conference paper at ICLR 2023
Molita Win Roie- %
Median Tox Win Rate %/
100
muliiple tasks
Mediun Teat Win Rate 5
100
Leann in single oak (Durs)
30
multiple uasis (REFIL)
80
60
Leaen in single oak (REFIL)
60
40
40
40
20
20
1.2M Timestops
04M
2.0
Timestegs
Timestops
(a) 5m vs 6m
(b) 3s5% vs. 3s6z
(c) MMM2
Figure 5: On source tasks: MATTAR provides a framework for multi-agent multi-task learning. Training on multiple tasks helps improve performance than learning on single task.
to be conditioned on the interaction pattern of multiple agents. Different from previous contextual multi-task RL methods, we include agent interaction modeling in the task representation learning module to deal with this problem. Nevertheless, we think incorporating natural language as prior high-level metadata of tasks can further improve the performance of our method in further work.
Related to our work, context-based meta-RL also reuses experience from multiple tasks and infers how to solve a new task from collected small amounts of experience (Zintgraf et al., 202T). For example, PEARL (Rakelly et al. (019) performs online task inference by probabilistic filtering of latent task variables. Other meta-R RL methods exploit the dynamics recurrent networks for implicit task inference and fast adaptation (Wang et al. [017]] Duan et al. 2016). These methods tend to form dynamic variables or embeddings, while the context information is utilized to obtain fixed task representations in approach. Besides, although their ability of task adaptation has been proved, context-based meta-RL. has not been extended to multi-agent cases and cannot deal with varying numbers of actions and different lengths of observation inputs. For future work, it is important to discuss whether the inspiration of context-based meta-RL can further improve the performance of our multi-agent multi-task learning framework.
Learning task or skill embeddings for multi-task transfer reinforcement learning has also been extensively explored. When testing, |Hausman al. (2018) learn new skill embedding by inter- polating learned skill embeddings on source tasks by approximate variational inference. Arnekvist] Amreeiissl al. 2019) also focus on learning skill embeddings but conditioned on optimal Q-functions for different S Co-Reyes et al. (2018) adopt a hierarchical framework and learns high-level policy to control low-level skill latent space. Similar to ours, Co-s,eee et al. (2018) learn the low-level skill space by encoding experience trajectories and decoding states and actions. Another work that is similar to ours is Zhang et al. (2018), which conditions the policy on latent representations learned by dynamics and reward module. Lan et al. (2019) learn task embeddings, fixes the policy, and updates the encoder at the test time. Compared to these methods, our task representation learning method differs by (1) it is specially designed for multi-agent settings and considers varying input sizes and intra-agent interaction; and (2) to the best of our knowledge, the alternatively fixed learning scheme is proposed for the first time. More contents include multi-task reinforcement learning, multi-agent transfer learning, multi-agent representation learning, and modular RL are shown in App. A
6 CONCLUSION
In this paper, we study cooperative multi-agent transfer reinforcement learning by learning task representations that model and exploit task relationship. Previous work on multi-agent transfer learning mainly deals with the varying population and input lengths, relying on the generalization ability of neural networks for cooperation knowledge transfer, ignoring the task relationship among tasks. Our method improves the transfer performance by learning task representations that capture the difference and similarities among tasks. When facing a new task, our approach only needs to obtain a new representation before transferring the learned knowledge. Taking advantage of task relationship mining, our proposed method MATTAR achieves the best transfer performance and other bonuses compared with multiple baselines. An important direction in the future is the transfer among tasks from different task distributions, and whether a linear combination of source tasks' representations can fully represent unseen tasks is also valuable topic.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Akshat Agarwal, Sumit Kumar, Katia Sycara, and Michael Lewis. Learning transferable cooperative behavior in multi-agent teams. In International Conference Autonomous Agents and MultiAgent Systems, pp. 1741-1743, 2020.
Isac Amekvist, Danica Kragic, and Johannes A. Stork. VPE: variational policy embedding for transfer reinforcement learning. In International Conference on Robotics and Automation, pp- 36-42, 2019.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pp. 501- 510, 2018.
Åke Björck. Numerics of gram-schmidt orthogonalization. Linear Algebra and Its Applications, 197: 297-316, 1994.
Jiahan Cao, Lei Yuan, Jianhao Wang, Shaowei Zhang, Chongjic Zhang, Yang Yu, and De-Chuan Zhan. Linda: Multi-agent local information decomposition for awareness of teammates. arXiv preprint arXiv:2109. /2508, 2021.
John D. Co-Reyes, Yuxuan Liu, Abhishek Gupta, Benjamin Eysenbach, Picter Abbeel, and Sergey Levine. Self-consistent trajectory autocncoder: Hierarchical reinforcement learning with trajectory embeddings. In International Conference on Machine Learning, pp. 1008-1017, 2018.
Bruno Castro da Silva, George Dimitri Konidaris, and Andrew G. Barto. Learning parameterized skills. In Internationnl Conference Machine Learning, pp. 1443-1450, 2012.
Marc Peter Deisenroth, Peter Englert, Jan Peters, and Dieter Fox. Multi-task policy search for robotics. In International Conference on Robotics and Automation, pp. 3876-3881, 2014.
Yunshu Du, Wojeiech M. Czarnecki, Siddhant M. Jayakumar, Razvan Pascanu, and Balaji Lakshmi- narayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv: /tl::8/22.022 182.00224, 2018.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL3: Fast reinforcement learning via slow reinforcement learning. arXiv preprint wrXiv:161..0277 2016.
Jakob Foerster, Ioannis Alexandros Assacl, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-21444 2016.
Dibya Ghosh, Avi Singh. Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning. International Conference on Learning Representations, 2018.
Aditya Grover, Maruan Al-Shedivat, Jayesh K. Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International Conference on Machine Learning, pp. 1797-1806, 2018.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin A. Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018.
Yann-Michaël De Hauwere, Peter Vranex, and Ann Nowé. Learning multi-agent state space represen- tations. In International Conference on Autonomous Agents and MultiAgent Systems, pp. 715-722, 2010.
Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. UPDeT: Universal multi-agent reinforce- ment learning via policy decoupling with transformers. In International Conference on Learning Representations, 2021.
Wenlong Huang. Igor Mordatch, and Deepak Pathak. One policy control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pp. 4455 4464, 2020.
10

---

Under review as a conference paper at ICLR 2023
Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference Machine Learning, pp. 4596-4606, 2021.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi multi-agent cooperation. In Advances in Neural Information Processing Systems, pp. 7254-7264, 2018.
Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. In International Conference Learning Representations, 2019.
Jens Kober, Andreas Wilhelm, Erhan Öztop, and Jan Peters. Reinforcement learning to adjust parametrized motor primitives
situations. Autonomous Robots, 3344)331--779 2012.
Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is a cage: The role of morphology in graph-based incompatible control. In International Conference on Learning Representations, 2021.
Lin Lan, Zhenguo Li, Xiaohong Guan, and Pinghui Wang. Meta reinforcement learning with task embedding and shared policy. In International Joint Conference on Artificial Intelligence, pp. 2794-2800, 2019.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17:1334-1773, 2016.
Qian Long, Zihan Zhou, Abhinav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Exolutionary population curriculum for scaling multi-agent reinforcement learning. In International Conference on Learning Representations, 2019.
Ryan Lowe, Yi Wu. Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. In Advances in Neural Information Processing Systems, pp. 1379-6390, 2017.
Frans A Olichoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs.
Springer, 2016.
Shayegan Omidshafici, D Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, and Jonathan How. Learning to teach in cooperative multiagent reinforcement learning. In AAAI Conference on Artificial Intelligence, pp. 22366,66, 2019.
Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexei Efros. Learning to control self-assembling morphologies: A study of generalization via modularity. In Advances in Neural Information Processing Systems, pp. 2292-2302, 2019.
Janith C Petangoda, Sergio Pascual-Diaz Vincent Adam, Peter Vrancx, and Jordi Grau-Moya. Disentangled skill embeddings for reinforcement learning. arXiv preprint arXiv: 2019.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- off-policy meta-reinfforeement learning via probabilistic context variables. In International Conference on Machine Learning, pp.
2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 4292-4301, 2018
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The Star- Craft multi-agen aagttt challenge. In International Conference Autonomous Agents and MultiAgent Systems, pp. 2186-21888 2019.
Felipe Leno da Silva and Anna Helena Reali Costa. Transfer learning for multiagent reinforcement learning systems. Synthesis Lectures on Artificial Intelligence and Machine Learning, 15(3):1-129, 2021.
11

---

Under review as a conference paper at ICLR 2023
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. In International Conference on Machine Learning, pp. 9767-9779, 2021.
Yanchao Sun, Ruijie Zheng, Xiyao Wang, Andrew E Cohen, and Furong Huang. Transfer RL
across observation feature spaces via model-based regularization. In International Conference Learning Representations, 2022.
Mihai Suteu and Yike Guo. Regularizing deep multi-task networks using orthogonal gradients. arXiv preprint arXiv:/912.000
2019.
Yunzhe Tao, Sahika Genc, Jonathan Chung, Tao Sun, and Sunil Mallya. Repaint: Knowledge transfer in deep reinforcement learning. International Conference on Machine Learning, pp- 10141-10152, 2021.
Dean Tjosvold. Cooperation theory and organizations. Human relations, 37(9):7337767, 1984.
Samir Wadhwania, Dong-Ki Kim, Shayegan Omidshafiei, and Jonathan How. Policy distillation and value matching in multiagent reinforcement learning. In (EEE/RSJ International Conference on Intelligent Robots and Systems, pp. 8193-8200, 8200, 2019.
Jane X Wang. Zeb Nurth-Nelson, Hubert Soyer, Joel Z Leibo, Dhruva Tirumala, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matt M Botvinick. Learning to reinforcement learn. In CogSci, 2017.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: Duplex dueling multi-agent Q-learning In International Conference Learning Representations, 2021a.
Tingwu Wang. Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International Conference on Learning Representations, 2018.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. ROMA: Multi-agent reinforcement learning with emergent roles. In International Conference on Machine Learning, pp. 9876-9886, 2020a.
Tonghan Wang. Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. RODE: Learning roles to decompose multi-agent tasks. In International Conference on Learning Representations, 2021b.
Weixun Wang, Tianpei Yang Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. Action semantics network: Considering the effects of actions in multiagent systems. In International Conference on Learning Representations, 2020b.
Weixun Wang, Tianpei Yang. Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few to more: Large-scale dynamic multiagent curriculum learning. AAAI Conference on Artificial Intelligence, pp. 7293-7300, 2020c.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. DOP: Off-policy multi-
agent decomposed policy gradients. In International Conference on Learning Representations, 2021c.
Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent repre- sentations to influence multi-agent interaction. In Conference on Robot Learning, pp. 575-588, 2020.
Zhiyuan Xu, Kun Wu, Zhengping Che, Jian Tang, and Jieping Ye. Knowledge transfer in multi-task deep reinforcement learning for continuous control. In Advances in Neural Information Processing Systems, pp. 15146-15155, 2020.
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems, 2020.
Tianpei Yang, Weixun Wang, Hongyao Tang, Jianye Hao, Zhaopeng Meng. Hangyu Mao, Dong Li, Wulong Liu, Yingfeng Chen, Yujing Hu, Changjie Fan, and Chengwei Zhang. An efficient transfer learning framework for multiagent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 17037-17048, 2021.
12

---

Under review as a conference paper at ICLR 2023
Qiang You, Ou Wu, Guan Luo, and Weiming Hu. thread mining in web communities. In Machine Learning and Data Mining in Pattern Recognition, pp. 421-4344444 2016.
clustered multi-task learning for
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, pp. 5824-5825558 5836, 2020.
Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learning. In International Conference on Learning Representations, 2018.
Kaiqing Zhang, Zhuoran Yang. and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pp- 321-384, 2021a.
Shenao Zhang, Li Shen, and Lei Han. Learning meta representations for agents in multi-agent reinforcement learning. arXiv preprint arXiv:2108, 12988, 2021b.
Zimu Zheng, Yuqi Wang. Quanyu Dai, Huadi Zheng, and Dan Wang. Metadata-driven task relation discovery for multi-task learning. In International Joint Conference on Artificial Intelligence, pp- 4426 4436,4432,200 2019.
Tianze Zhou, Fubiao Zhang, Kun Shao, Kai Li, Wenhan Huang, Jun Luo, Weixun Wang, Yaodong Yang, Hangyu Mao, Bin Wang, Dong Li, Wulong Liu, and Jianye Hao. Cooperative multi-agent transfer learning with level-adaptive credit assignment. arXiv preprint 104i:::206.000511 2021.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning deep reinforcement learningg survey. arXiv preprint arXiv:2009.077 07888, 2020.
Luisa M. Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD: Variational Bayes-adaptive deep RL meta- learnie....ng. Journal of Machine Learning Research, 22(289):1-39, 2021.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
A MORE RELATED WORK
Transfer learning Transfer learning holds the promise of improving the sample efficiency of reinforcement learning (Zhu et al. 2020). which learns knowledge from source tasks to accelerate the learning efficiency in the unseen task. Previous work uses successor features, decouples the transition dynamics and reward function, and learns faster in simulated navigation and robotic arm settings (Barreto al.) 2018). DSE (Petangoda et al. 2019) models the transfer process as variational inference and further learns a latent space to transfer skills across different dynamics. |Sun et al. (2022) apply model-based regularizer learn task-level transfer across various observation spaces. Related to our work, REPAINT (Tao et al.) (021) combines task representations with on-policy learning and uses an advantage- based experience selection approach to transfer useful samples. Our method differs from these works by (1) its mechanism for alternately updating the task representation and the representation explainer and (2) its mechanism for handling different input/output sizes.
The basic idea behind multi-agent transfer learning (Silva & Costa, 2021) is to reuse knowledge from other tasks or other learning agents, corresponding to intra-agent transfer and inter-agent transfer, respectively. It is expected that the knowledge reuse can accelerate coordination compared to learning from scratch. The inter-agent transfer paradigm aims at reusing knowledge from other agents with different sensors or (possibly) internal representations via communication. DVM (Wadhwania et al., 2019) treats the multi-agent problem as a multi-task problem to combine knowledge from multiple tasks and then distills the knowledge by a value matching mechanism. LeCTR (Omidshafiei et al. 2019) learns to teach in a multi-agent environment and learns to advise others in a peer-to-peer manner. MAPTF (Yang al. 2021) takes a further step by proposing an option-based policy transfer for multi-agent cooperation, and it significantly boosts the performance of existing methods in both discrete and continuous state spaces. On the other hand, intra-agent transfer refers to reusing knowledge from previous tasks, focusing on transferring knowledge across multi-agent tasks. The varying populations and input lengths impede the transfer among agents, with which the graph neural networks and the transformer play promising roles. DyMA-CI. (Wang et al.] (020c) designs various transfer mechanisms across curricula accelerate the learning process based on a dynamic agent-number network. EPC (Long al. 2019) proposes curriculum learning paradigm via an evolutionary approach to scale up the population number of agents. UPDeT (Hu et al. 202I) and PIT (Zhou et al. (02T) make use of the generalization ability of the transformer to accomplish the multi-agent cooperation and transfer between tasks. Although these methods can accelerate the learning efficiency of MARL algorithms, they do not exploit task similarity for better transfer performance. By contrast, our method explicitly models task relationship by learning a hidden space in which tasks with similar dynamics have similar representations.
Multi-task reinforcement learning is another relevant research topic that enables an RI. agent to leverage experience from multiple tasks improve sample efficiency and avoid learning from scratch on every single task. Various approaches have been proposed to achieve multi-task learning, such as distilling separate tasks' knowledge into shared policy (Levine et al. 2016,] Ghosh et al. 2018 Xu et al. 20200). conditioning policies on tasks (Deisenroth et al., 2014). mapping tasks to parameters of a policy (da Silva et al.) 2012 Kober et 2012 Yang et al. 2020), and solving the problem of Yu et al. 2020) meaning that gradients of
negative interference (Du et al. 2018 Suteu & Guo 2019
different tasks may conflict.
Multi-agent representation learning Learning effective representation in MARL is receiving significant attention for its effectiveness in solving many important problems. CQ-Learning (Hauwere et al. 2010) learns to adapt the state representation for multi-agent systems to coordinate with other agents. Grover al. (2018) learn useful policy representations to model agent's 's behavior in multi- agent system. LILI Xie et al.) (020)) learns latent representations to capture the relationship between ego-agent's 'ss behavior and the other agent's future strategy. RODE (Wang et al.) 2021b) uses an action encoder to learn action representations and applies clustering methods to decompose the joint action space into restricted role action spaces to reduce the policy search space of aueti-agent cooperation. MAR (Zhang et al. 2021b) learns meta representation for multi-agent generalization. Our approach differs from these works by learning representations for tasks and using a representation explainer for efficient policy transfer.
14

---

Under review as a conference paper at ICLR 2023
Another line of research single-agent multi-task learning that has the potential to be effective in multi-agent settings is modular RL. Modular RL decentralizes the control of multi-joint robots by learning policies for each actuator and thus holds the promise to deal with input and output with varying lengths. Each joint has its controlling policy, and they coordinate with each other via various message passing schemes. To do so,, Wang al. (2018) Pathak al. (2019) represent the robot's morphology as a graph and use GNNs as policy and message passing networks. [Huang et al. (2020) use both bottom-up and top-down message passing schemes through the links between joints for coordinating, All of these GNN-like works show the benefits of modular policies over a monolithic policy tasks tackling different morphologies. However, recently, Kurin al. (202T) validated hypothesis that any benefit GNNs can extract from morphological structures is outweighed by the difficulty of message passing across multiple hops. They further propose transformer-basee method, AMORPHEUS, that utilizes self-attention mechanisms as a message passing approach. Although modular RL can deal with varying action numbers of different tasks and can implicitly model the interaction between agents through GNN, Transformer, message passing, it still cannot cope with varying observation lengths and does not incorporate task-level context information compared to our method. In summary, work distinguishes itself from previous multi-task work by (1) its flexibility of handling varying lengths of observation and varying numbers of actions; (2) its utilization of agent-interaction modeling in capturing task relation; and (3) its alternatively fixed task representation learning scheme.
B DETAILS ABOUT CASE STUDY
Problem definition. In case study (Sec. 3 design two-player navigation problem, which essentially is a grid-world problem. The whole map is a grid world with size [15, 15), and in each episode agents are randomly initialized in the centering initial region which is 3 x sub-area. The agents can observe local information within sight range of and are expected to reach specific goal position for the current task. Each task corresponds a specific goal position and there exist eight source tasks as marked in Fig.
For the reward function, we define the reward ry at each timestep as the opposite of the L, distance from the agents to the target position:
T1. Jabs (agent pos. [0] goal pocelppsspp (0) abs aaseage pocesnlpposs]]1 [1] goal fooe_pos[]] pos
Application of MATTAR. Note that the transition func- tions are all the same in different tasks. To better capture the similarity and distinctions between tasks, we only op- timize the reward prediction loss for forward model in the task representation learning step. The training phase is conducted the eight source tasks, where the represen- tation explainer and the agent policy are trained. When transferring to the unseen task, we learn the task repre- sentation by optimizing the reward prediction loss on the new task and insert the learned task representation into the trained policy network to obtain the transferred policy.
Two results in the main text. For Fig. 3(a) we conduct transfer phase for multiple times independently. Each time, we record the learned coefficients for the target task, and show the results in the form of a bar chart where each bar corresponds to a specific source task. Besides, each time we obtain target task representation, we insert it into the representation explainer and obtain a forward model. We test the prediction loss of this forward model on eight source tasks and present the results in the form of a plot chart. Besides, note that the samples for computing model prediction error collected with random policies. For Fig, 3(b) first insert source task representations to the trained agent policy and roll out several trajectories. For simplicity of presentation, we only visualize the trajectories of agent 1 and selectively show trajectories for
source Itsk gools
unsecn task goal
initial region for agents
Figure 6: Tow-player Navigation prob- lem for Case Study.
15

---

Under review as a conference paper at ICLR 2023
five source tasks 21, 4,5 and 8). We visualize trajectories for two independent runs. To avoid overlapping of arrows, we place an arrow at a random position within a grid to represent that the agent once passed this grid and the direction of arrow indicate the moving direction of agent. For the trajectory of the unseen target task, we visualize one trajectory and represent it with a dashed line.
C DETAILS ABOUT THE BENCHMARKS
(a) SMAC: 2s3z
(b) SMAC: MMM2
(c) MPE: Spread (d) MPE: Gather
Figure 7: Snapshots of the experimental environments used in this paper.
SMAC (Fig. [b))(b)) StarCraft II Micromanagement Benchmark (Samvelyan et al. 2019) contains combat scenarios StarCraft II unit micromanagement tasks and is a popular benchmark for multi-agent reinforcement learning. We consider partially observable setting, where agent can only see circular area around it with a radius equal to its sight range, which default to We train ally units with MATTAR fight against enemy units controlled by the built-in AL At the beginning of each episode, allies and enemies spawn in pre-defined regions on the map. Every agent takes actions from discrete action space including oo-op, move direction]] attack/enemyide, d]. and stop. Under the control of these actions, agents can move and attack on a continuous map. Agents will get a shared reward equal the total damage dealt to enemy units each timestep. Killing each enemy unit and winning the combat (killing all the enemies) will bring additional bonuses of 10 and 200, respectively. We consider three series of SMAC tasks, each including various maps. The detailed descriptions are shown in Tab. 7
MPE ( (d) Multi-Agent Particle Environment (Lowe et al. 2017) is multi-agent particle world containing several navigation and communication tasks. In our experiments, we consider a discrete version of MPE and use two tasks, Spread and Gather, to evaluate our method. In both tasks, there are nLagent agents on a field with size [field.size, fieldssize tasked to reach landmarks. The agents When a landmark is within a reach rangex reach range sub-field around an agent, we say the agent has reached the landmark. In Spread, require each agent reach landmark that is not occupied by any other agents, while in Gather, the agents share common landmark. both of these tasks, only when all agents reach the landmark, collective reward of given. For both Spread and Gather, we test several tasks with different numbers of agents. The detailed settings of these tasks are listed in Tab. 10 and
observe objects around it within distance of sight_range
D NETWORK ARCHITECTURE AND HYPERPARAMETERS
Our implementation of MATTAR is based on PyMARL with StarCraft 2.4.6.2.69232 and uses its default hyperparameter settings. We apply the default e-greedy action selection algorithm to every algorithm, decays from to 0.05 in 50K timesteps. We also adopt typical Q-learning training tricks like the target network and double Q-learning. MATTAR has hyperparameters λ₁ X2, and the scaling factors of the observation prediction loss, the reward prediction loss, and the entropy regularization term, respectively. We set them to 1, 10, and 0.1 across all experiments. For other hyper-parameters, we the default settings of QMIX presented in the PyMARL framework. For RODE [Wang et al. (2221b. ASN (Wang et al. 20206). QPLEX (Wang et al. 2021a). QMIX (Rashid) al. 2018), and UPDeT (Hu et al. 2021). we use the codes provided by the authors with the default
We use PyMARL with 69234.6...6923. Performance is always comparable among versions.
16

---

Under review as a conference paper at ICLR 2023
Table 7: Settings of tasks in the MMM series. The bolded names indicate the source tasks.
Map Name
Ally Units
Enemy Units
Type
Difficulty
Medivac, Marauders,
Medivac, Marauders, Marines
MMM0
Asymmetric Heterogeneous
Easy
5 Marines
Medivac,
Medivac,
MMM
2 Marauders,
2 Marauders,
Asymmetric & Heterogeneous
Easy
Marines
Marines
Medivac,
Medivac,
MMMI
Marauders,
2 Marauders,
Asymmetric & Heterogeneous
Hard
Marines
Marines
Medivac,
Medivac,
MMM2
Marauders,
Marauders, 8 Marines
Asymmetric Heterogeneous
Super Hard
7 Marines
Medivac,
Medivac,
МMM3
Marauders,
3 Marauders,
Asymmetric Heterogeneous
Super Hard
8 Marines
9 Marines
Medivac,
Medivac,
MMM4
3 Marauders,
4 Marauders, Marines
Asymmetric & Heterogeneous
Super Hard
Marines
Medivac, Marauders,
Medivac, Marauders, 10 Marines
MMM5
Asymmetric & Heterogencous
Super Hard
Marines
Medivac, Medivac,
MMM6
Marauders, Marauders, Asymmetric Heterogeneous
Super Hard
8 Marines
11 Marines
Table 8: Settings of tasks in the SZ series. The bolded names indicate the source tasks.
Map Name
Ally Units
Enemy Units
Type
Difficulty
Stalkers,
Stalkers,
Is8z
Symmetric & Heterogeneous
Easy
Zealots
Zealots
Stalkers,
Stalkers,
1s9z
Symmetric & Heterogeneous
Easy
9 Zealots
Zealots
Stalkers,
Stalkers, Zealots
2s3z
Symmetric & Heterogeneous
Easy
3 Zealots
Stalkers,
Stalkers, Zealots
2s8z
Symmetric & Heterogeneous
Easy
8 Zealots
Stalkers,
Stalkers, Zealots
2s9z
Symmetric & Heterogeneous
Easy
9 Zealots
3 Stalkers,
3 Stalkers,
3s5z
Symmetric & Heterogeneous
Easy
Zealots
Zealots
3 Stalkers,
Stalkers, Zealots
3s52.vs_3s6z
Symmetric & Heterogeneous
Super Hard
5 Zealots
Stalkers,
Stalkers,
7s3z
Symmetric & Heterogeneous
Easy
3 Zealots
Zealots
hyperparameters settings. We describe our network structure Tab. used for all experiments in the paper.
This network architecture
E EXPERIMENTAL DETAILS
Our experiments were performed on 2 NVIDIA GTX 2080 Ti GPUs. For all the performance curves in our paper, pause training every 10K timesteps and evaluate for 32 episodes with decentralized greedy action selection. We present the percentage of episodes which the agents defeat all enemies within the time limit. We now provide details about each part of experiments.
17

---

Under review as a conference paper at ICLR 2023
Table 9: Settings of tasks the M series. The bolded names indicate the source tasks.
Map Name
Ally Units
Enemy Units
Type
Difficulty
3m
3 Marines
5 Marines
Symmetric & Homogeneous
Easy
4m
4 Marines
5 Marines
Symmetric & Homogeneous
Easy
4m_vs_5m
4 Marines
5 Marines
Asymmetric & Homogeneous
Hard
5m
5 Marines
5 Marines
Symmetric & Homogeneous
Easy
5m vs 6m
5 Marines
Marines
Asymmetric & Homogeneous
Hard
6m
6 Marines
6 Marines
Symmetric & Homogeneous
Easy
6m vs. 7m
6 Marines
7 Marines
Asymmetric & Homogeneous
Hard
7m
7 Marines
Marines
Symmetric & Homogeneous
Easy
fin_v__mm
7 Marines
8 Marines
Asymmetric & Homogeneous
Hard
8m
8 Marines
8 Marines
Symmetric & Homogeneous
Easy
8m 9m
8 Marines
Marines
Asymmetric & Homogeneous
Easy
9m
9 Marines
9 Marines
Symmetric & Homogeneous
Easy
9m vs 10m
9 Marines
10 Marines
Asymmetric & Homogeneous
Fasy
10m
10 Marines
10 Marines
Symmetric & Homogeneous
Easy
10m_vs 11m
10 Marines
11 Marines
Asymmetric & Homogeneous
Easy
10m_vs, v2m
10 Marines
12 Marines
Asymmetric & Homogeneous
Super Hard
Table 10: Settings of the Spread tasks. The bolded identities indicate the source tasks.
Task Identity
# of Agents
# of Landmarks
Field Size
Sight Range
Reach Range
2
2
2
6
5
2
3
8
2
10
2
5
5
5
10
2
6
6
15
14
2
15
2
8
8
15
14
2
9
9
9
15
15
2
Generalization to unseen tasks For baselines and ablations, we carried out experiments with different random seeds. In each experiment, we evaluate the trained model for 32 episodes on each unseen task. The results recorded in Tab. 3 are the mean and variance of these 5 random seeds.
Fine-tuning For the performance of fine-tuning MATTAR, we trained 2 source models with different random seeds for each unseen map and used 2 random seeds for each source model for fine-tuning For learning from scratch, we carried out experiments with 4 different random seeds for each map.
Multi-task learning We carried out experiments with 5 different random seeds for both multi-task learning learning on single task. For the experiments of multi-task learning on three tasks shown in the paper, the training sets are (5m, 5m_vs-6m, 8m-vs-9m, 10m_vs 11m), (2=3z, 3s52, 3s52.vs_3s 62). and (MMM, MMM2, MMM4), respectively.
Single-task learning For this experiment, we tested each baseline and ablation algorithm with random seeds.
F FORWARD MODEL FOR TASK REPRESENTATION LEARNING
In our method, we utilize dynamics modelling to learn task representations which can capture the similarity between different tasks. We hypernetwork the representation explainer generate the parameters of the forward model. In practical implementation, the forward model consists two components, encoder and decoder (Fig. ((a)).
For the encoder network, we first use the population-invann invaariiannnniiaitttt embedding layer to get a fixed- dimensional embedding vector and feed it into fully-connected layer whose parameters generated
18

---

Under review as a conference paper at ICLR 2023
Table 11: Settings of the Gather tasks. The bolded identities indicate the source tasks.
Task Identity
# of Agents
# of Landmarks
Field Size
Sight Range
Reach Range
2
6
2
3
8
2
4
4
10
2
10
2
15
2
9
9
15
15
2
10
10
20
19
2
15
15
20
2
Table 12: Hyperparameters about the network structure in our experiments.
name
value
The hidden dimension for mixing network
32
The number of layers for the hypernet in mixing network
2
The hidden dimension for the hypernet
64
The length the encoding vector of agent ID
The dimension of task representations
32
The output dimension of the encoder in the forward model
32
The output dimension of the attention module The hidden dimension for the query and key attention module
64
8
Predictions of s' of.r hidden state
Cnon-interaction
interaction
Hyper- network
Generated Layer
Decoder Module
Non interaction layer
Interaction laper
Encoder Module
Rep.
o
embedding layer
Task Rep.
Oj
Qj
S
0j
aj
(a) Network architecture for the forward model
(b) Q-network for interaction actions
Figure 8: The architecture of our forward model and the Q-network for interaction actions. In (b). ontity, denotes the observation component corresponding the influence of the i-th interactive action, Qnon-interaction denotes the Q-values of non-interactive actions, and Qinteraction denotes the Q-values of interactive actions.
by the representation explainer. The output hidden variables are fed into the decoder predict the next state, the next observation, and the global reward. The encoder module and the representation explainer are shared among tasks and are fixed when learning representations for unseen tasks. The decoder module is task-specific, and allow the decoder to be optimized together with task representations when adapting to unseen tasks.
For the population-invaria embedding layer in the encoder module, like in the policy, decom- pose the input state and observation into several entity-specifis components, pass them through an embedding layer, and do pooling operation for output vectors. We also deal with the case of the varying number actions in the input by incorporating actions into to observation o,, We concatenate non-interaction actions with agent i's own observation component own and interaction actions with
19

---

Under review as a conference paper at ICLR 2023
Modian Win Rate %
MATTAR
Median Tox Win Bate 5%
100
Mollian Win 5/
QMIX
80
RODE
60
40
UPPeT
40
80
REIFIL
Timestops
Timostops
Timesteps
(a) MMM2
(b) 5m vs 6m
(c) 3s5z
Figure 9: A bonus: when learning from scratch on single tasks, MATTAR architecture exhibits good
performance. For performance more SMAC benchmark, please refer to Fig. T0
100
1.00
9/
80
Median Tex Win Rate
80
Toxt
Test
20
20
20
0.0M
0.4M
08M
1.2M
1.8M
20M
9.0M
0.4M
0&M
1.2M
L.M
20M
0.0M
0.4M
0.5M
1.2M
LoM
2.0
Timestops
Tinestops
Timesteps
(a) 2s3z
(b) 3s.vs..2
(c) 3s52. 3s6z
Mediun Text Win Rame %.
Modian Text Win 3/
Molion Text Win Ratts
80
80
80
80
60
60
0.0M
0.4M4
08M
1.2M
1.6M
20M
9.04
0.4M
08M
L.M LAM
20M
0.004
0.4M
0.534
1.2M LAM
2.0M
Timesiqps
Timestaps
Timestepe
(d) corridor
(c) 10m_vs_11m
(f) 6h_vs_8z
100
50
Mealan Tost Win Rate
80
MATTAR
QMIX
Moltan Toat
QPI
20
20
0.0
0.43
03M
1.2M Timestops
1.0M
20M
9.0
0.4M
08M
L.M
LAM
20M
Timestops
(g) 2c_vs_64zg
(h) 1s_
Figure 10: More results for the performance of MATTAR on the SMAC benchmark when learning from scratch on single tasks.
the observation components corresponding each entity. We also note that other population invvaiiat structures can also be applied to our approach.
G VARYING NUMBERS OF ACTIONS
In some multi-agent environments, there are interaction actions that have semantics relating to other opponents in the environment. In this case, the action dimension is related to the number of opponents, preventing flexible transfer to unseen tasks. To deal with this problem, we adopt the structure shown in Fig. ((b) for the estimation of Q-values for these interaction actions. In this structure, Q-values
20

---

Under review as a conference paper at ICLR 2023
for interaction and inniiteeaaction actions estimated separately. For oon-interaction actions, use a cunnecconneeted network whose imput is the concatenation observation encoding h and task network that takes as imput the concatenation of
representation For interaction action, h, task representation 2, and the observation component relating to the corresponding opponent.
H BONUS: PERFORMANCE ON SINGLE-TASK TRAINING
Although not designed for this goal, we find that MATTAR can outperform state-of-the-ard MARL algorithms when trained some single tasks. Specifically, remove the task representation module and train MATTAR from scratch. We compare our method with two state-of-the-art value- MARL baselines (QMIX (Rashid et al. (2188) and QPLEX (Wang al. 2021a)). a role-hased learning algorithm (RODE) (Wang et al. 2021b), and one underlying algorithm of MATTAR which considers the Q-values of interaction actions separately (ASN) (Wang al. 2020b). For the representative tasks of the three series in the main text (MMM2, 5m_vs_6n and 3s5z), we additionally compare with two methods with similar attentional mechanisms (UPDeT (Hu al. 202T and REFIL (Iqbal
et al. )02T
Fig. 9 shows the learning curves of different methods. We find that our population-inse invariiat network structure achieves comparable performance in all tasks. It is worth noting that this structure even significantly outperforms other algorithms on the super hard map MMM2. Fig. T0 we show the comparison more SMAC maps, on which MATTAR also has comparable performance. Given that our underlying algorithm QMIX, this is an inspiring result. We hypothesize that our self-attention scheme increases the representational capacity by learning to attend to appropriate entities in the environment.
1 TESTING OF ONE ALTERNATIVE FOR SOURCE TASK DEFINITION
100
Median Text Win Rtle
80
80
Median Test Win Rote.
80
Mediun mmm
-00
60
40
20
0.0M
0.4M
0.8M
1.2M
1.6
2.0M
0.0M
0.404
0.5M
1.2M Timesteps
L.M
2,004
0.084
0.4M
0.8M
1.2M
1.0M
2.0M
Timesteps
Timestops
2x3z
(b) 3s5z
(6) 355z 356z
Figure 11: Testing of one alternative for source task definition. In this experiment, we apply normalization layer on top of the joint learned vectors with the representation explainer, and use it as the task representations for source tasks.
One possible method for determining the source task representations is to joint learn representation explainer and task representations together. However, this practice often brings representations with very small norms. A remedy for this problem is to additionally apply normalization layer top of the jointly learned vectors. We conduct experiments to validate this approach, and the results of learning performance learning curves on source tasks begin to drop after about 1M samples for all these three tasks. These results show that it is hard to get meaningful representation space when learning together with the representation explainer. We hypothesize the reason behind the phenomenon is that there are limited signals that can guarantce information about task relationship is encoded in the representation space.
source tasks for the SZ series are shown in Fig. T1 It is interesting that the
J DISCUSSIONS WHEN TASKS UNDER GREAT DIFFERENCES
Our approach shows great advantages over ablations and baselines in the experimental results reported in the main text. To further explore the ability of our approach, we additionally conduct two experiments, and the results are reported in Tab. 13
21

---

Under review as a conference paper at ICLR 2023
Table 13: Two additional experiments where the unseen tasks quite different.
Source Tasks
Unseen Tasks
1s2z
1s32
2x37
3652
3652. 3sb2
467z
4s7z 4sk2.
MATTAR
0.94++0.04
0.97=00.03
0.91=:0.07
0.68+t0.12
0.01=00.01
0.39:t0.19
0,00=t0000
MMM
MMM2
MMM4
1s8z
2630
36fz
7s3x
MATTAR
0.99:00001
0.85:20.01
0.89::0.03
0.00::0.00
0.00:t0000
0.00::0000
0.05:+0000
In the first experiment, design source tasks and target tasks with huge gap in terms of the number of agents. Specifically, We train MATTAR on three source tasks: 1s2z, 1s3z, and 2s3z, each of which contains small number of agents and has a difficulty level of "easy", and we test it tasks with more agents and higher difficulty level. As we can see, the transfer performance drops when the number agents has huge increase, and the win rate on 4s7z 4s8z.
In the second experiment, we train MATTAR on three tasks from the MMM series and test it some tasks of the SZ. series. We can see that when the source tasks are not diverse enough to cover unseen tasks, the transfer performance is close to O.
From the results, we can see that the transfer performance faces a relatively large drop when our approach transfers from tasks with few agents to those with more agents, and a failure will appear when testing on those hard tasks as the decision skills required in these tasks cannot be acquired by training those easy tasks. Besides, when try transfer across different series of tasks, our approach struggles even when testing on the 2s3z task, which is a quite easy task. This phenomenon indicates that policy transfer across quite different tasks is still quite hard problem that is worth exploring, and our approach may struggle when the target task can not be well covered by the source tasks. How overcome this challenge and achieve more general policy transfer in multi-agent reinforcement learning remains an open problem.
22


Reviews:
[{"review_id": "rsbGe0Ccz87", "reviewer": "Reviewer_AfTn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies transfer learning in multi-agent setting, where existing works based on graph neural networks or attention mechanisms achieves generalization across tasks implicitly through the generalization of neural network function approximations. The authors propose to explicitly model the task relationships by learning a task embedding and the resulting policies will depend on the task representations, which can be inferred for new tasks given a few trajectories. Experimental results demonstrates the effectiveness of the proposed framework.", "summary_of_the_review": "Overall I think this is a well-written paper with a reasonable framework to capture the task relationships for multi-agent transfer learning. The paper could be improved with some theoretical characterizations of the problem and the proposed framework.", "strength_and_weaknesses": "Strength\n- The proposed framework seems novel and reasonable to the problem considered.\n- Nice presentation with many useful figures and intuitive discussions so that the paper is easy to follow.\n- Promising experimental results.\n\nWeaknesses/Questions\n- Can you provide some theoretical characterization of the proposed framework? Transfer learning for multi-agent setting is very complicated and challenging, due to varying population size, input dimension, and even different action spaces. For example suppose source domain action space is a continuous interval [0, 1] while target domain action space is finite {0, 1}, then it is guaranteed that the two tasks are not transferable in some scenarios. For example, in single-agent case, which is simpler than multi-agent, [1] studies the alignment of two MDPs.\n- When do you expect the transfer to unseen tasks will succeed or fail? How many samples are needed to reliably learn a task representation for unseen task. It seems to achieve this kind of generalization across tasks, you may need datasets from a lot of different tasks, since the latent space for task representation is quite large and many values can be undefined.\n- Does the proposed framework also apply to competitive/mixed multi-agent system?\n\n[1] Domain Adaptive Imitation Learning", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pE422aNdpr", "reviewer": "Reviewer_oPef", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for policy transfer in a multi-agent reinforcement learning setting. The method focuses on transferring to new population size. It learns task representations that capture relationships among source tasks. On an unseen task, a new representation is learned as a linear combination of the learned representations. The unseen task's representation is then plugged into the Q-learning population-invariant policy.\n\nThe method is evaluated on two multi-agent benchmarks. It outperforms baseline methods on unseen tasks. The task representation enables higher performance after fine-tuning the policy. It also effectively leverages the multi-task learning setting.", "summary_of_the_review": "Overall, I like the proposed method as it is very general. But some of results are not convincing and the authors make false claims instead of explaining the observed phenomenon. The paper is borderline and I am learning towards rejection. \n\n=======After rebuttal=======\n\nI thank the authors for giving very detailed and comprehensive responses. I decided to raise the score to 6, hoping that the authors will incorporate their responses into the final version, especially the explanation about the ineffectiveness of the learned task representation on MPE. ", "strength_and_weaknesses": "**Strengths**\n* The paper is clearly written. The method is well-described. Experiment hypotheses are thoroughly constructed. \n* To my best knowledge, the method seems to be novel, although I am not too familiar with the multi-agent RL literature. It is also technically sound. \n\n**Weaknesses**\n\nMethod\n* The major weakness is that learning a new task representation requires collecting diverse trajectories on the new task. The paper uses the policy trained on source tasks to collect trajectories on a new task. The policy thus needs to generalize decently to the unseen tasks to generate interesting trajectories. I think this point needs to be emphasized in the paper. \n* Initializing the representations of the source tasks as orthogonal vectors ignores the relationships between them. \n* The learned forward model may be used for planning but only the task representation is used. \n* This is not really a weakness. But the method seems to solve more general problem than generalizing to new population size. Based on the current formulation, I believe it can also be applied to a single-agent setting. Is there any particular reason for formulating and showcasing it on an multi-agent setting? \n\nExperiments\n* Results on the MPE tasks are not convincing. The task representation makes very little difference (first 3 rows of tables 4 and 5). The authors incorrectly claim \"MATTAR...  significantly outperforms the baselines and ablations\". ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ezZXAB6d_B", "reviewer": "Reviewer_dhhQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper puts forth a fully-cooperative multi-agent/task algorithm, MATTAR, that can be used to transfer knowledge learned from source tasks to improve learning unseen tasks at test time. MATTAR learns a high-level mapping from task representations to state, observation, and reward function parameters and combines this with a policy learning approach that allows for inputs and outputs of varying sizes. The paper evaluates MATTAR on multiple benchmarks against multiple baselines and shows that MATTAR generally attains superior performance in terms of win rate, sample efficiency, and feasibility (i.e., winning at all).", "summary_of_the_review": "The paper clearly motivates and addresses an important AI problem. It proposes the algorithm MATTAR and shows that it generally outperforms what it believes to be competitive baselines on two game/simulation benchmarks extended to multi-task transfer learning settings.\n\nHowever, I believe that this paper falls far short of acceptance due to the lack of novelty, limited experimentation, poor experimental design, and lack of analysis of results.", "strength_and_weaknesses": "Strengths\nThe paper motivates and addresses an important AI problem: fully-cooperative multi-agent learning, which can be and has often been denoted as just multi-task learning for several years (Parisotto et al., 2015). \n\nMATTAR is simple, easy-to-implement, and parametrizes a model that simultaneously outputs the next state, next observation, and global reward. This algorithm can handle varying input and output sizes, which is often the case across differing tasks, using a population-invariant network (PIN). However, the use of a high-level model that maps task representations to model parameters is not new nor is the use of PINs new, as the paper itself claims on page 4. In fact, it is unclear what the paper's contribution regarding PINs is, so the paper would benefit by clearly explaining and justifying this. Perhaps this is in Appendix G, but regardless, I apologize if this is clearly stated in the paper and I missed it.\n\nThe paper graciously provides a case study on a toy problem to help readers gain an intuitive understanding of how MATTAR works, but this case study doesn't include visualization of trajectories to task 3, even though it corresponds to one of the two non-zero learned coefficient components, and it omits the fact that from Figure 3(b), it is also typical behavior in task 4 for the agent to move left when justifying why the agent moves up first in moving towards the unseen task goal. I assume that the agent moves up instead of left first actually because unseen goal is further up than left from the initial agent region or because the trajectory from only one rollout or one policy is being shown. \n\nThe paper is easy to read, clear, and the Figures, Tables, etc. are great at capturing and explaining salient points the paper makes.  It does a great job of providing enough information on the experimental design to reproduce the experiments.\n\nThe paper evaluates MATTAR on multi-task settings on two benchmarks, SMAC and MPE, which have varying degrees of difficulty. Experiments are carried out with five random seeds, which in some cases is sufficient, but in certain cases here is insufficient. The paper claims that \"... the super hard map 3s5z_vs_3s6z and [sic] cannot be solved by learning from scratch,\" which is too strong of a claim generally, but especially if only 5 seeds were run.\n\nThe paper shows that MATTAR generally outperforms multiple baselines on multiple multi-task transfer learning setups. However, I'm unconvinced that the baselines are competitive, as UPDeT doesn't even aim to address the same problem setting as MATTAR does (single source task vs. multi-source task transfer) though the paper does aim to provide a fair comparison by introducing UPDeT-b. Regardless, there exist a plethora of multi-source task transfer learning algorithms (often i.e., fully-cooperative multi-agent learning algorithms) that may be better suited to compare against. The paper would also benefit by evaluating MATTAR on a set of more diverse benchmarks than just closed, game or toy-type settings, such as Starcraft (SMAC) and MPE, especially one or two real-world settings so that the research community has a better idea of how MATTAR performs generally.\n\n---\n\nParisotto, Emilio, Jimmy Lei Ba, and Ruslan Salakhutdinov. \"Actor-mimic: Deep multitask and transfer reinforcement learning.\" arXiv preprint arXiv:1511.06342 (2015).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "00Dt6bCJTyg", "reviewer": "Reviewer_U9UT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents MATTAR, which learns the common structure of tasks and transfers this knowledge to unseen tasks. Specifically, the framework first learns the representation explainer based on the pre-defined and mutual orthogonal task representation vectors. Then, during the transfer phase, a task representation vector for a new task is learned for adaptation. MATTAR also includes the task policy learning component that leverages PIN for addressing the varying number of agents across tasks. Experiments in the SMAC and MPE domains demonstrate the effectiveness of MATTAR's transfer capability. ", "summary_of_the_review": "Overall, I have a positive evaluation of this paper. While the novelty can be a concern, the experimental results are solid and detailed, and the paper conveys the main findings clearly. Therefore, I initially vote for 6 (marginally above the acceptance threshold) and will make a final decision on the recommendation after the authors' response.\n\n", "strength_and_weaknesses": "**Strength:**\n1. The paper is generally well-written and conveys the main methods well. \n2. Experiments in Sections 4.1-4.3 show the benefits of MATTAR against competitive baselines. The ablation studies and related empirical analyses improve the understanding of MATTAR. \n\n**Weaknesses:**\n1. Novelty can be limited because 1) the idea of learning the small-size task parameters (i.e., the weight vector $\\mu$) for adaptation is similar to Zintgraf et al. (2021) and 2) the task policy learning component is based on PIN. \n2. I agree that the idea of pre-defining the task representation vectors as mutual orthogonal vectors and learning linear weight vectors is new and interesting. Because this idea is the main contribution, this paper can benefit from having a comparison and discussion (i.e., pros and cons) w.r.t. this idea. For instance, compared to Zintgraf et al. (2021), linear weight learning has advantages, including intuitive understanding and well-formed representation space. However, this linear weight learning can have disadvantages as mentioned in the conclusion (future work). Not necessary, but I wonder whether there can be a small experiment comparing the ideas between the linear weight learning and non-confined learning (similar to Zintgraf et al. (2021)). \n3. Related to the weakness #2, if all source tasks are set up to be identical, can the mutual orthogonal vector idea hurt the performance?\n4. MATTAR applies fine-tuning for unseen tasks. I wonder it would be more fair for baselines in Table 1-5 to apply fine-tuning. \n\n**Minor:**\n1. In Section 2, for completeness, source tasks $\\{S_i\\}$ and unseen tasks $\\{T_j\\}$ can be mathematically defined using $G$ (Dec-POMDP) with different transitions and reward functions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject