One by One, Continual Coordinating with Humans via Hyper-Teammate Identification
Authors: Cong Guan
Feng Chen
Ke Xue
Chunpeng Fan
Lichao Zhang
Ziqian Zhang
Pengyao Zhao
Zongzhang Zhang
Chao Qian
Lei Yuan
Yang Yu
Venue: Accepted by TMLR ()

Abstract: One of the primary objectives in modern artificial intelligence researches is to empower  agents to effectively coordinate with diverse teammates, particularly human teammates. Previous studies focused on training agents either with a fixed population of pre-generated teammates or through the co-evolution of distinct populations of agents and teammates. However, it is challenging to enumerate all possible teammates in advance, and it is costly, or even impractical to maintain such a sufficiently diverse population and repeatedly interact with previously encountered teammates. Additional design considerations, such as prioritized sampling, are also required to ensure efficient training. To address these challenges and obtain an efficient human-AI coordination paradigm, we propose a novel approach called \textbf{Concord}. Considering that human participants tend to occur in a sequential manner, we model the training process with different teammates as a continual learning framework, akin to how humans learn and adapt in the real world.  We propose a mechanism based on hyper-teammate identification to prevent catastrophic forgetting while promoting forward knowledge transfer.  Concretely, we introduce a teammate recognition module that captures the identification of corresponding teammates. Leveraging the identification, a well-coordinated AI policy can be generated via the hyper-network. The entire framework is trained in a decomposed policy gradient manner,  allowing for effective credit assignment among agents. This approach enables us to train agents with each generated teammate or humans one by one, ensuring that agents can coordinate effectively with concurrent teammates without forgetting previous knowledge. Our approach outperforms multiple baselines in various multi-agent benchmarks, either with generated human proxies or real human participants.

Published in Transactions on Machine Learning Research (10/2024)
One by One, Continual Coordinating with Humans via Hyper-Teammate Identification
Cong Guan', Feng Chen', Ke Xue', Chunpeng Fan?, Lichao Zhang? Ziqian Zhang', Pengyao Zhao?, Zongzhang Zhang', Chao Qian', Lei Yuan! Yang Yu
National Key Laboratory for Novel Software Technology, Nanjing University School of Artificial Intelligence, Nanjinn University
Poluir Technologies
(chumpeng.fors lichao. zhang) @polizin ai, shangzaBlanda nju.. edu.cn, pengyao. pengyao.shao www.@@hoiiiii.cccommmmmm ai, (zzzhang, qionej @nju. guant@lamda. njuu. edu, en, yuy@njj.edu.co
(guanc, chenf, zuck) Glomda.
Reviewed on OpenReview: https: // openreview. net/ forum? id= HVwwmpoowolooooooo WBM
Abstract
One of the primary objectives in modern artificial intelligence researches is to empower agents to effectively coordinate with diverse teammates, particularly human tcammates. Previous studies focused on training agents either with fixed population of pre-generated teammates through the co-evolution of distinet populations of agents and teammates. However, it is challenging to enumerate all possible teammates in advance, and it is costly, or even impractical to maintain such a sufficiently diverse population and repeatedly interact with previously encounteree leammates. Additional design considerations, such as priori- tized sampling, are also required to ensure efficient training. To address these challenges and obtain an efficient human--Al coordination paradigm, we propose novel approach called Concord. Considering that human participants tend to occur in a sequential manner, we continual learning framework,
model the training process with different teammates akin to how humans learn and adapt in the real world. We propose mechanism based on hyper-teammate identification to prevent catastrophic forgetting while promoting for- ward knowledge transfer. Concretely, we introduce teammate recognition module that captures the identification of corresponding tcammates. Leveraging the identification, a well-coordinated AI policy can be generated via the hyper-network. The entire framework trained in decomposed policy gradient manner, allowing for effective credit assignment train agents cooperate with teammates one
among agents. This approach enables by one, ensuring that agents coordinate effectively with concurrent teammates with- out forgetting previous knowledge. Our approach outperforms multiple baselines in various multi-agent benchmarks, either with generated human proxies or real human participants.
1 Introduction
Cooperative Aglti-Agent Reinforcement Learning (MARL) has made significant progress recent years, enabling multiple agents to work together towards shared goal in diverse domaïns such as active voltage control [Wang al. (021aa) and dynamic algorithm configuration (Xue et al. 2022b). However, building
AI agents that can effectively coordinate with unseen teammates, espocially human teammates, remains
significant challenge (Klien et. al. 2004 Mutlu et al. 2013 Strouse et al. 2021 Koster et al. 2022 Yuan
et al. 2023c Previous approaches focused on building effective behavior models from human data, which
* *Corresponding Author

---

Published in Transactions on Machine Learning Research (10/2024)
Tedious Training Proces
Execation
Emergent New
Teumunne
Emergent Teammate
Fuil
Fixed Pre-gamirated Temmmnii Population
Hard to cnumerate all potential leammates advance
Difficalt to merosentally trai with usseen Icermmate
Sequential training with encountered teammates
(a) Typical scenarios classic Human- coordination methods
Scenario of continual coordination learning
This figure provides a visual comparison between our proposed setting of continual coordination learning and typical scenarios of classic Human- AI coordination methods.
Figure 1:
is inefficient for complex problems (Kidd & Breazeal 2008 and may raise privacy concerns (Pan et al. 2019). Recent approaches (Treutlein et al.) 2021 Mirsky et al. 2022; Fosong et al.) 2022) show remarkable coordination abilities a wide range of tasks such as Overcooked (Carroll al. 2019). but stilll suffer from ce al.
over-fitting to their training teammates and struggle to coordinate effectively with unseen agents (Mahajan
2022 Thus, the challenge of building Al agents capable of generalizing to unseen teammates in the
open- world Zhou 2022 remains.
Training with different teammates is a promising approach to tackling the mentioned issue, involving the generation of diverse leammates and an efficient training paradigm. To achieve the former, one approach is to use hand-crafted policies [Xie et al. 2021 Papoudakis et al. 2021), special object regularizer (Derek &
Isola
2021 Lupu et al., 20211) ; or Population-
Training (PBT) (Strouse al. 2021 Xue al. 2022a 2023). Regarding the (raining paradigm, naive way is sell-play (Tesaurro 1994 Silver el al.
Zhao el al.
2018 where the agent iteratively improves via playing against itself. Fictitious Co-Play (FCP) (Heinrich)
et al. 2015 Strouse et al. 2021) trains agents to be the best response to both the fully-trained agents and their checkpoints. These approaches have demonstrated significant progress in benchmarks such as Overcooked (Carroll et al. 2019 and Hanabi (Tupu et al. 2021) offering promising prospects for human-AT coordination and cooperation. However, the aforementioned methods pre-generate various teammates and necessitate access to all of them during training, which might be unfeasible in the real world. On the one hand, the task of enumerating all potential teammates advance poses significant challenges. Maintaining a sufficiently diverse population has already placod a substantial demand on computing and storage resources, incurring considerable costs. Furthermore, if agents are required to cooperate with an unseen teammate, learning from scratch with all previous seen teammates (especially human participants) would be even more wasteful, considering the limitations of global time differences and economic costs. On the other hand, learning a generalized agent via interactions with a population of teammates requires meticulous design to ensure efficient training, such as prioritized sampling (Zhao el. al. 2023).
To address the aforementioned challenges, we draw inspiration from the manner in which humans learn to coordinate with diverse teammates. Human participants, with whom agents trained to cooperate, typically emerge sequent Instead of learning to cooperate with all agents simultaneously. humans continually adapt to new teammates while retaining knowledge from previous interactions (Hadsell et al. 2020). In accordance with this idea, we propose a new coordination learning setting and formulate it as a Multi-Agent Continual Markov Decision Process (MACMDP), where the controlled agents are trained to coordinate effectively with different teammates that appear sequentially. Fig. | offers a visual comparison between this novel problem setting and the conventional scenarios found in previous works. We observe that naively applying single-agent continual learning methods to MARL is inefficient, and typical MARL approaches are inadequate for continual learning scenarios (RQ1 in Sec. 4.2 Therefore, propose general framework called Concord (abbreviation for Continual coordination) to enable efficient continual training of AT agents and solve the MACMDP. Concretely, we construct. a leammate representation space where the teammates are represented as latent embeddings, and we introduce a recognition module that enables recognizing teammates through few-shot interaction. We then utilize the learned teammate emboddings based on a hyper-network (Ha al.) [0177] Oswald et al. 2020) generate the policy parameters of each controlled agent. Finally, we use value decomposition to facilitate credit assignment among agents, enabling
2

---

Published in Transactions on Machine Learning Research (10/2024)
Human Policy xf
Human Policy n2
Human Policy ng
Human Policy mz
Teammate Policy Stream
Collaborative training
Al Agent Policy (())
Agent Policy (2))
Agent Policy (())
Agent Policy nt))
Continual Learning Timeline
Figure 2: Workflow diagram of Multi-Agent Continual Markov Decision Process.
framework to handle multiple controlled agents rather than only one. To evaluate the effectiveness of Concord, we conduct extensive experiments various cooperative multi-agent benchmarks including Over- cooked (Carroll et al., 2019) and StarCraft Multi-Agent Challenge benchmark (SMAC) (Samvelyan et al., 2019). We compare Concord against multiple methods in different scenarios, and the results demonst that Concord significantly outperforms these methods, achieving superiority across a range of evaluation metrics. Our main contributions arc:
To the best of our knowledge, this is the first time that human-Al coordination challenge is explored in a continual manner via our formulated MACMDP.
We propose contimual MARL paradigm based on hyper-teammate identification, which is capable of effectively coordinating with diverse leammates and striking balance between forward transferring and avoiding forgetting.
Empirical studies on various benchmarks demonstrate the effectiveness of Concord in continuously coordinating with diverse teammates, including real human tcammates.
2 Problem Formalization
The purpose of human-Al coordination is for Al agents to cooperate well with diverse human teammates, Previous works rely on a fixed policy population, involving extensive training of AI agents, but this approach is computationally expensive and does not align with real-world scenarios where humans are encountered sequentially. Currently, there limited research on training AT agents to work well with humans in the continual setting. To address this gap, we introduce problem formulation for continual multi-agent coor- dination learning, that agents can learn to cooperate with unseen human teammates continually.
We formalize the continual multi-agent coordination learning problem a Multi-Agent Continual Markov Decision Process (MACMDP) consisting of tuple M == S, A, P, R, 7, where / is the set of Al agents, S and A are the state and action space, respectively. P the transition function, R the reward function, 7 E [0, 1) is the discount factor and T is the continual learning length. Besides, θ the teammate type space, and ş an indicator function which means that μ({) E O(1 < S T) denotes the tcammate policy encountered at the Ç-th stage. For the sake of brevity, denote the policy determined by μ in the (-th stage T?!
These tcammates are encountered sequentially, as depicted in Fig. 2 In human-Al coordination tasks, cooperating with different humans can be defined as different. tasks, and each human leammale policy T can be represented by a task indicator Ç. In MACMDP, AI agents can not interact with the previous human teammates, which corresponds to the assumption that the agent can not interact with previous environments in the common continual reinforcement learning setting (Khetarpal et al., 2202)) but are expected to remember how to cooperate with all previous human teammates. When handling the task Ç.
3

---

Published in Transactions on Machine Learning Research (10/2024)
the AI policy πA and human policy =" select actions a a" according to the current state s, respectively. The next environment state s' is obtained according to the environment transition function P(s" ss, a a and a global reward is returned by the global reward function R(s, a aat.. We use the discounted return Je(TA, ((²) Eg [E,>R(ss, a a²)]] as the objective on task Ç. A special note is that the AT agents may need to collaborate with multiple human teammates currently, not just one. Thus, the human policy nu here can refer to several teammates, and we actually conduct continual coordination learning with human teammates one (team) by one (team). Here, the term "team" may consist of one single teammate like in Overcooked multiple tcammates in sccnarios of more players.
A workflow diagram illustrating MACMDP is provided in Fig. 2 for better understanding. As depicted in this diagram, the Al agents are expected to be collaboratively trained with the teammate policies one by one, and interacting with the previous teammates is prohibited. Under this problem formalization, our goal is to find the AT agent policy F that can maximize the expected return with current. human learmmates after continual process, without forgetting the previous cooperation patterns, which can be formally represented Je( as 4 Je m )
3 Method
In this section, we introduce the proposed Concord framework. We first introduce the teammate-adaptive Al framework in Sec. 3.1 which includes three main components, ie,, hunan-aware actor critic, teammate representation and recognition, and forgetorggg mechanism. Then, we give the overview of our algorithm in Sec. 3.2. which includes the pseudo-code of the two phases and a brief diagram illustration.
3.1 Teammate- Teammate-adaptivee Al Framework
Al agents are anticipated to identify different teammates and adapt accordingly, enabling them to acquire the ability to coordinate continually. Additionally, the agents should be able to remember previous teammates while learning to coordinate with new ones. We here propose a comprehensive pipeline (depicted in Fig. 3 and App. A) that involves designing a human-aware actor-critic structure for policies training, building teammate representations for teammates capturing, and implementing an anti-forgetting mechanism to address the forgetting problem during continual learning.
Human-Aware Actor-Critic In a human-AI coordination scenario, the decision-making process of the overall multi-agent system jointly determined by both the Al agents and the humans. To enable AI agents to identify different teammates, we introduce teammate-adaptive structure, an extension and variant of hyper-network (Ha et al.) 2017 Oswald et al. 2020). This structure is specifically designed for continual human-AI coordination tasks and can generate actor networks that can effectively coordinate with specific leammates by leveraging consistent Leammate representations. Concretely, the parameters of the Al actor networks are obtained through 0,*c (xyper(=<), where phyper represents the hyper-network parameterized by w. In essence, we incorporate the ability to adapt to different teammates into the hyper-network, so that it can generate appropriate actor networks for cooperating with the human teammates based on the input of different teammate representations zς. More details are reported in App. A.1
To better assess the contribution of AI the overall performance, we here design a linearly decomposed centralized critic like DOP (Wang et al. 2021b). Therefore, we can create individual critics for the un- controllable human agents, and model the global Q function by combining the individual Q values of both human and AI agents. This approach enables the global Q function to more accurately rellect the joint decision- decision-making process of human-Al coordination, and facilitate credit assignment between Al and hu- mans, allowing us to isolate the contribution of the AI agents and provide more precise learning signals for updating TA. The critic and actor networks are updated through minimizing the critic loss Ceritic ETaat" (Q, ---)²² " and actor loss Lactor E E kz( log nn (aAlmAs00) Q (T (T)
respectively, where k2(T) represents the coofficients of value decomposition generated by an additional hyper- network when inputting the historical information t, the global value Qi (T, a HH is obtained through
4

---

Published in Transactions on Machine Learning Research (10/2024)
Trainable Networkss
G Generated Parameters
Stot geittic
4. Cooperate with human
Mixing Network
Q"
te
Trajectory z#
clor
3. Generalee Adaplive
Critic
Hyper- Network
Human Critic
Recognizer
gactor
F
Teammate Rep.
Crecon
AI
R Recognition Process
Human ctor
Teammate Rep.
Recognizer
Hyper-Nett Netw
1. Rew-shot Interaction
Training P Phase
Test Phase
Human
Policies
72
Past
Future
Overall Continual Learning Process
Figure 3: The overall framework of Concord. During the training phase, a human-aware actor-critic is trained for AI to coordinate with humans effectively, where the AI actors are generated via hyper- network. Teammate representations are trained via backpropagation, while generated by the recognizer in the test phase, When testing, the Al first interacts with human to collect few trajectories, then obtains the corresponding tcammates representation for decision-making:
(T, aA, =, (T, att) (T() (T, att) + b(T), where hi (T) and ((T) are generated by learnable networks whose inputs are global state, and (T( is restricted be non-negative ensure monotonicity, and y denotes the target value y = r + Baaa aa" Quol(T'' aat2 a(A,attt) Note that we refer to the parameters of the critic network as o, and those of the target network as Φ'.
Teammate Representation Learning To enable our method to adapt to different teammates, as men- tioned above, we use one hyper-network that can generate adaptive actor networks by inputting different leammate representations. Here, the leammnte representations serve the crucial role of recognizing the behavior patterns of different human teammates. To adopt this teammate representation scheme, we have necessary prerequisites: 1) the teammate representations can help the hyper-network generate Al actor that can cooperates well with human; 2) the teammate representations should be obtainable with minimal amount of interaction with humans during the test phase.
To fulfill the first prerequisite, we design an end-to-end update scheme for teammate representations. In particular, we maintain a separate teammate representation for each human team encountered during the continual learning process This teammate representation 2ę is then fed into the hyper-network obtain the parameters the AI actor network gåc which expected coordinate effectively with n² Thus, we directly update the hyper-network and teammate representations with the gradients back-propagated from optimizing the actor loss Lactor, However, hold assumption that the identity of the teammates is unknown during testing. Thus, although we have learned representations for each team, we do not know which one to utilize during testing, which leads to the second prerequisite above.
In terms of the second prerequisite, we additionally design a recognizer network that can reconstruct learned teammate representations from limited interaction data. This recognizer network typically implemented as trajectory-encont that takes the trajectories of human teammates as input and outputs the prediction of the corresponding teammate representations. For training the recognizer network, we encode trajectories
5

---

Published in Transactions on Machine Learning Research (10/2024)
of varying lengths and minimize one reconstruction loss denoted as 1 = -E== > =c1 This loss function encourages the encoder to effectively reconstruct teammate representations using trajectories of different lengths. During the test phase, we roll out some trajectories (typically 2 in our experiments) with the teammate. Each entire trajectory undergocs encoding by the encoder, yielding a reconstructed representation for each trajectory. Then, they are averaged to yield the final recognition outcome, thereby achieving few-shot recognition of human teammates.
Anti-forgetting Mechanism One of the key challenges in continual multi-agent coordination learning is to avoid calastrophic lorgetting of previous human leammates as the Al policy learns and adapts to new ones. Specifically, within our framework, we aim to combat the forgetting problem in both the recognizer network fn and hyper-network & phyper simultaneously, which correspond to the ability to recognize and coordinate with the previous human teammates, respectively. To tackle this challenge, we employ an anti- forgetting mechanism to enforce regularization on both fque and payper networks during the continual learning process. For the recognizer network, we utilize a small replay buffer store the teammate representations and few of interaction trajectories of the previous human teammates. During cooperation learning with the (-th human team, the actual loss of the recognizer network is Creron training the hyper-network at the Ç-th cooperation task, we additionally include one regularization loss L - - - II
Creson Similarly, when
phyper [I (z)- 1 f (z:)ll in addition to the actor loss to avoid f plyper's i forgetting the
previous cooperation knowledge, where flyper denotes the hyper-network obtained after the (S 1) task. In summary, we propose an muti-forgetting mechanism that effectively safeguards against the forgetting of past knowledge in both the updated hyper-network from the previous round and small memory pool containing few-shot interactions and teammate representations of previous human teammates.
and the recognizer network through the preservation of the
3.2 Overall Algorithm
Finally, we provide the overall description of the procedure of our approach under a continual coordination learning setting. The main parts of our approach Concord can be decomposed into the training phase and the test phase. The pseudo-codes for these two phases are respectively shown in Alg. 1 Tand Alg. 2 A brief diagram illustrating the main idea of our approach can also be found in App. A.2
During the training phase, we conduct cooperation learning with human teammates one (team) by one (team) from H to nÿ where we learn the hyper-network and the recognizer network together. As detailed from lines 5 to 13 of Alg. [ during the collaborative training with the Ç-th human team, we firstly optimize the human- aware actor-critic architecture by minimizing the geritic and Lactor losses, where the teammate representation 2ς is concurrently updated in an end-to-end manner with the back-propagated gradient V Cactor While for the hyper-nctwork f plyper, we update it with the actor loss Lactor plus one regularization loss L which means that the actual loss function for the hyper-network is
greg
Chyper = Lactor + GreR _ Cactor
I (x)) -
dupper (2,)
(1)
5-1 1=1
where grog is the coefficient balancing these two loss terms. We optimize the critic network, hyper-network and teammate representation in lines 11, 12 and 13 of Alg. D respectively. Following the training process of hyper-network and teammate representation, in line 15 we update & with the latest hyper-network, and in line 17 we retain a certain amount of trajectories to buffer B. Subsequently, from lines 18 to 21, we further optimize the recognizer network via reconstructing the representations of all previously encountered human tcammates, which means that we update the fe via minimizing the reconstruction loss ΣS= 2r where Crecon indicates the reconstruction loss for the leammate representation of the i-th human leam. After completing these steps, we proceed to the collaborative training with the next human team. In brief, the collaborative training with each group of human tcammates is decomposed of two scparate stages, where we train the hyper-networt network and teammate representations at the first stage and update the recognizer network at the second stage.
6

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm Training phase
Require: Parameters 1, 6, (z4)-m Small buffer B
Explanation:
f Recognizer network parameterized by y
{TC J(=1- Teammate policy stream
=H Few interaction data obtained by interacting with ot zc: Learned leammate representation for leammate policy <²
J phyper: Hyper-network parameterized by
B: small buffer that stores few of human trajectories and teammate representations for all previous
seen teammates.
Ensure: Update the parameters v. 0, {zc
Set v None
for TH in the teammate policy stream do
// Training critic network, hyper-network and teammate representation
for episode in RL training period do
5:
Compute the critic loss Ceritic, actor loss Lactor
6:
if vp is None then
Let Chyper : Luctor
7:
clse
Let Chyper == Cactor + greg
10:
end if
11:
Update & via gradient descend method with gradient V Cariti
Update zi via gradient descend method with gradient V Chyper
12:
13:
Update via gradient descend method with gradient Cactor
14:
end for
15:
Let : : // Store the hyper-network parameters the last round
16:
// Training recognizer network
17:
Select multiple human trajectories r from the RL training process, and store each (T =x) into the
bulfer B.
18:
for iteration in Recognizer training period do
Compute Crevon via sampling data from B 20: Update y via gradient descend method with gradient V, Creoon
19:
21:
end for
22:
end for
When it comes to the test phase, in line 1 of Alg. we first allow few interaction with the humans. The, as detailed in lines 2 and 3 respectively, the sampled human trajectories are fed into the recognizer network fene to help allain the predicted leammate representation, which then inputted into the hyper-network hhyper obtain the actor network for the Al policy. The final actor network here is employed to coordinate fo
with the humans.
4 Experiments
In this section, we first conduct experiments on the widely-used Overcooked (Carroll al., 2019) verify whether Concord can effectively coordinate with human teammates under the continual setting. Then, we evaluate Concord's ability to coordinate with multiple teammates through experiments on SMAC (Samvelyan let al. 2019). The human teammates involved in these experiments consist of both human-like models and real human participants.
To comprchensively analyze the performance of various algorithms the context of continual human-Al coordination, we introduce the Single Coordination Performance Pt(t), where 21.2...1 and Pt(t) represents the episode reward when coordinating with the i-th human team after completing training with

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 2 Test phase
Require: Parameters 77, v
Explanation:
reee Recognizer network parameterized by
Collaborative policy of the teammates
2
#: Few interaction data obtained by interacting with the teammates
2c: Predicted teammate representation obtained by inputting Te to fque
f
phyper; Hyper-network parameterized by sb
Actor: AI actor network generated by inputting & to f Ensure: Deployed Al policy in coordination with leammates
phyper
Interact with teammates using π. obtain 7!
2: 2 + (T(
AI Actor + fo phyper (&c)
return Deploy AI Actor as AI policy in coordination with the teammates
the t-th team. For principled assessment of the continual learning ability, introduce followings metrics inspired by the concepts applied in (Wolozyk et al. 2021 Lopez-Paz & Ranzato 2017):
Average coordination performance. P= N Σ P((T), which is the average episode reward when coordinating with all the human teams after the training process.
Forgetting. N Enn 1- P.(T)- P.()), which is used measure the average performance loss with seen human teams (i..., forget) after training with the new team.
Forward transfer. N Eim P.(i 1), which is used to measure the coordination ability with unseen human teams.
We compare the proposed Concord with the multiple baselines. All experimental results are averaged over five random seeds, accompanied by standard deviation. More details including network architecture and are provided in App. A.1 and B.1
4.1 Baselines and Environments
To completely evaluate the performance of Concord, we introduce the following baselines for comparison. First of all, adapt EWC (Kirkpatrick et al.) 2017) and CLEAR (Rolnick et al.) 2018), which two classic continual learning algorithms,, to our experiment setting. Then, we additionally design two other baselines: Vanilla and Oracle. Vanilla does not do any extra design for the continual learning process, while Oracle is always allowed for collaborative training with all human teammates. Here, to address the potential unfamiliarity some readers with the field of continual learning, we have provided additional descriptions for the EWC and CLEAR algorithms to enhance the readers' understanding of the specific processes of EWC and CLEAR, as well as how we adapt them in our context.
EWC Elastic Weight Consolidation (EWC) regularizatiin--aseed technique that selectively reduces the plasticity of weights to alloviate catastrophic forgetting. The technique Fisher information matrix evaluates the importance of all weights and uses that for gradient updales. There are two variants of EWC: Offline EWC keeps a model with a Fisher matrix for every previous task, which causes costly storage. Online EWC uses only one Fisher matrix that computed based on all the previous tasks. Here, we choose Online EWC as our beuchmark.
CLEAR When training, CLEAR samples mini-batch from the current task and another replay mini- batch. The samples in the replay mini-batch are selected with the same probability as all previous tasks. CLEAR leverages behavioral cloning from replay mini-batch to enhance stability. Specifically, two additional
8

---

Published in Transactions on Machine Learning Research (10/2024)
20
10
10
201
Player
Player
Time
Reward
(a) Coordination Ring (b) Asymmetric Advantages
(c) Many Orders
(d) Open Asymmetric Advantages
Figure 4: Four layouts of the Overcooked environment, with each one emphasizing different coordination strategies.
loss terms are added to induce behavioral cloning between the network and its past self. (1) the KL divergence between the historical policy distribution and the present policy distribution, (2) the l norm of the difference between the historical and present value functions. Unlike the original paper, we use Cross Entropy loss instead of KT. loss to guide the agent to choose the same action its past self under the same state because we find it more effective. Additionally, we don't add any constraints to criticism for easier learning.
Oracle Here, we would like to explain why the multi-task learning approach can be seen as Oracle. 1) Multi-task learning (Oracle) can access all the previous models in the training process. However, Concord (and other continual learning methods) can only be trained with the previous leammates once in the process. 2) Multi-task learning (Oracle) knows which human team it is coordinating with in the test phase. However, Concord (and other continual learning methods) has to identify the teammates by some specific mechanisms.
In the preceding discussion, we introduced the baseline algorithms compared in our study. In the following part, we will proceed to discuss the environments utilized in our experiments. Specifically, our experiments are conducted in two widely recognized multi-agent cooperation environments: Overcooked (Carroll et al. 2019) and SMAC (Samvelyan et al. 2019).
Overcooked Overcooked is an environment proposed for coordination challenges. In this environment, two players control one kitchen chef to cook and serve soup. Specifically, they are required to put specific ingredients in one pot, make soup and deliver it, for example, in Coord. Ring layout, the players need to put three onions in pot, collect an onion soup from the pot after 20-time steps, and deliver the dish counter. The agents will receive points for each dish served, and the goal is serve as many dishes possible within the allotted time. Moreover, we select. 4 different layouts in our experiments, which are shown in Fig.
StarCraft Multi-Agent Challenge (SMAC) SMAC is a popular benchmark for cooperative multi- agent reinforcement learning, which contains multiple combat scenarios. In this environment, the AI agents and human teammates control specific ally units and are expected to coordinate defeat the enemy units controlled by the built-in AI. Specifically, at the beginning of each episode, the ally units are spawned on the fixed regions on the map. At each timestep, the ally agents select actions from the action set (no-op, move [right, left, up, down], stop, attack [enemy_id]]] and then global reward will be given according to the damage caused to enemy units. Extra rewards of 10 and 200 will be given if one enemy unit killed or the ally units win the combat. We conduct experiments on three SMAC maps in this paper as shown Fig. 5 which are 3m, 2z1s, and 4m, respectively. On all these thrce maps, there exist the same number and type of units on both ally and enemy sides, where 3m and 4m signify the presence of 3 marines and 4 marines, respectively, and 2x1s indicates 2 zealots and 1 stalkers.

---

Published in Transactions on Machine Learning Research (10/2024)
OBAS
30008880
3m
(b) 2z1s
(c) 4m
Figure 5: Three SMAC maps that are selected in our experiments.
100
Oracle
Average reward percentage
L
Concord
80
EWC
CLEAR
Vanilla
60
40
Coord. Ring
Asy. Advantages
Many Orders
Open Asy. Advantages
Figure 6: The overall performance of distinct algorithms, where Concord achieves comparable coordination ability with Oracle and oulperforms other baselines.
4.2 Continual Coordination with Human-like teammates
First, we evaluate continual coordination ability with human-like teammates on Overcooked (Carroll et al. 2019), where different layouts present a unique challenge that can be overcome through effective coordination. The players are required put three onions in pot, collect an onion soup from the pot after 20 time steps and deliver the dish to counter. The agents will receive 20 points for each dish served, and the goal is to serve many dishes as possible within the allotted time. We run each algorithm for random seeds and each seed was tested 32 times. The final results were obtained after averaging these results. We investigate the following research questions (RQs).
How does Concord perform compared to other methods? As can be seen from Fig. 6 Vanilla achieves the most inferior coordination ability in almost all scenarios, indicating a specific consid- cration for the tasks where teammates appear sequentially. Other successful approaches for single agent continual learning, like FWC, and CLEAR, also suffer from performance degradation the involved bench- marks, demonstrating the necessity of specific design for MARL. The Oracle method, where we train all the lasks simultaneously, can be seen an upper bound of performance on the related benchmarks, acquiring superiority over all baselines. Our approach Concord, obtains comparable performance to Oracle, indicating the efficiency of all the designed modules. Besides, according to Wilcoxon rank-sum test with significance level 0.05, Concord is significantly better than EWC, CLEAR and Vanilla, except for CLEAR on Many Orders and EWC on Open Asy. Advantages, which further strengthen our results.
RQ1:
Furthermore, we also show the comparisons different methods on forgetting and forward transfer in Tab. [ Concord achieves the minimum forgetting on all the four layouts, validating the effectiveness of the proposed anti-forgetting mechanism. Besides, the forward transfer values of Concord on all the layouts best.
10

---

Published in Transactions on Machine Learning Research (10/2024)
104
114
34
16
I17
125
156
161
167
140
128
179
196
2
95
113
19L
90
42
49
72
139
46
95
15
138
300
27
133
177
53
131
123
15L
145
155
3
13
2
855
9
14
71
80
14
70
146
181
145
175
133
74
91
68
171
167
174
121
156
91
130
122
45
130
109
144
155
145
196
188
178
134
100
120
105
135
190
126
153
16
107
144
135
166
136
159
147
158
144
124
139
163
151
121
10
140
135
113
135
125
146
10
152
180
146
201
167
190
117
136
138
167
165
137
V5
70
154
117
45
66
84
145
80
144
148
157
14%
197
143
174
115
176
183
145
167
10
10
13
26
196
11
19
15
104
168
162
186
193
174
178
106
128
141
174
75
175
135
115
21
140
107
146
144
171
147
161
162
2
5
10
11
12
2
9
11
Partner Index
Partner Index
Figure 7: Single coordination performance of Vanilla (left) and Concord (right) on the Asym. Adv. layout. The value at i-th row and j-th in the matrix is Pz(j).
Table I: Comparisons of distinct algorithms in all layouts on forgetting and forward transfer metrics. The best and second best values each layout are colored in red and blue, respectively. The symbols '+, -- and 's'' indicate that the result is significantly superior to, inferior to, and almost equivalent to Concord, respectively, according to the Wilcoxon rank-sum test with significance level 0.05.
Algorithm
Courd, Ring
Asy. Advuntages
Many Orders
Open Asy. Advantages
Forg.
F. Transfer
Foog.
F. Transfer
Forg.
F. Transfer
Forg.
F. Transfer
Concard
-1.31
4.23
103.09 13.02
-16.11 6.88
119.59 9.41
-1.27
6.69
100.85 + 3.14
-6.83 4.59
162.29 6,94
EWC
-16.47
14118
92.03 1 8.61
-54.14 = 13.95
50.98 8.001
-20.52
8285
-7.69 1: 3.38 se
153.85 :: 3.68 is
CLEAR
-10.79
2.62
80.51 2.46
-120.61
72.65 1.78
-2.38 3.86
99.58
-61.144655.7
117.14+ 14.92
Vanilla
46.92-11.55
80:94 9.19
.109.80
= 20.48
00.8247.333
-68.36 10.14
67.61 4.16
-123.80 4 17.91
100:94 32.31
indicating that Concord can coordinate well with unseen leammates. Vanilla is worst, demonstrating the necessity of mechanism for continual learning. Specifically, we compare the single coordination performance of Concord and Vanilla. the Asym. Adv. layout of Overcooked environment in Fig. 7 As expected, Vanilla exhibits a considerable amount of forgetting, as indicated by the lower triangle of the matrix, and has poor forward transfer ability, as reflected by the upper triangle of the matrix. While Concord performs significantly better than Vanilla, More results on other layouts can be found in Fig. 14 of App. C.1
RQ2: How does the teammate reprosentation mechanism work? As an important aspect of Con- cord, the learning of leammate representations helps AT agents prevent catastrophic forgetting and effey adapt different human teammates. The human recognizer encodes a representation embedding for each human team based their behavior patterns. Using this embedding, generated and utilized to effectively cooperate with that human team. We demonstrate the differences in behavior patterns among distinct human-like models and the corresponding relationship between embedding and behavior pattern in Fig. g Two main observations can be made: 1) The behavior patterns of human- like models are diverse and can be characterized through four key steps. Visualization results indicate that these models exhibit. varying behavior patterns. The output representations from the recognizer network also differ significantly, forming distinct clusters. 2) Similar behavior patterns have similar embedding represen- tations, demonstrating the effectiveness of our representation mechanism in identifying leammates' behavior patterns. For instance, the behavior patterns of models 3 and 4 are identical, and their embedding represen- tations are similarly aligned. This pattern of similarity also holds for models 6 and 11, as well as 8 and 12. Additionally, models 5, 6, 8, 11, and 12, which all utilize the right pot for cooking, show embeddings that are clustered on the same side of a dividing line, in contrast to other models.
appropriate actor network is
To further investigate the impact of tcammate representations on AI agent behavior, we also present the behavior patterns displayed by AI actors generated by the hyper-network in response to different teammate representations. The findings indicate that the AI agent. can adapt its behavior to different teammates.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Key steps
Directions
counlerclockwise (CCW)
placing an onion inlo the pot
2. picking up dish from the dish dispenser .dclivering soup the serving
clockwise (CW)
3.picking up ready soup from pot
bi-direction (Bi-D)
Behavior Patterns
CCW
Bi-D
CW
CCW
CCW
Sougs CCW
Soup: CCW
Soup: UW
Soup: CW
Soup: CW
Soup: CW
Soupe CW
Soup: CW
Deliver: CW
Deliver: CCW
Deliver: CCW
Deliver: CWV
Deliver: CW
Deliver: CWV
Deliver:
Deliver: CW
lask
fask
1,4k3,4
Lask
lask 6,
lask 7, 10
task
Lask
Representation Embeddings
task
task
fask
task
fask
2
lusk
task
$..
task
task
Pap
rask
task
fask S
Task
lask 10
task
task
top pot
right pol
Ego Agent Behavior Patterns
Onion: CCW
Onion: CCW
Onion: CCW
Onion: Bi-D
Onion: CW
Onion: CW
Onion: Bi-D
Onion: Bi-D
Onion: CCW
Onion: CCW
Onion: CCW
Onion: CW
Dish: CCW
Dish:
Tish: CW
Dish:
Dish: CW
Dish:
Soup: CCW
Soup:
Songe CCW
Soup: CCW
Soup: CW
Soup: CW
Soup:
Soup: Soup:
Soup: CCW
Soup
Soup: CCW
Deliver: CCW
Deliver CW
Deliver: CUW
Deliver COW
Deliver: CW
Deliver: CW
Deliver: COW Deliver: Deliver. CW
Deliver: CCW
Deliver: CW
Deliver CCW
task
task 2
fask
task
task 5
task 6
task 7
fask 8
task 9
task 10
task 11
task 12
Figure 8: The I-SNE (Van der Maaten & Hinton 2008 projection different behavior patterns. Completing delivery task requires several key steps, including placing an onion into the pot, taking a dish, taking a ready soup, and delivering the soup, and each step has three directions for human-like models: always clockwise (CW), always counterclockwise (CCW), and bi-direction (Bi-D). Then, the behavior pattern of teammate can be defined as the dircctions used in the key steps.
1.0
Average reward per spisode
160
Hoxa Orioo
Crasel Hace Crte cow Tocks
(hieag oee Tokas
0.8
140
0.6
Concord
120
0.4
BWC CLEAR
100
0.2
Vanilla
2000
Memory Decay
0
8
10
0.0
P&P
P&M
P&E
MRM
M&E
E&E
(a) Study on value decomposition
(b) Experimental results on SMAC
Figure 9: (a) Performance comparison between Concord with and without value decomposition. (b) Fx- perimental results on the map 4m. The gray shaded bar is used to denote the degree of forgetting, which equals to the highest test win rate during training minus the final test win rate. The x-axis represents the combination two teammates, e.g., P&P means both teammates are poor (P) models.
demonstrating varied targeted actions accordingly. Notably, the agent exhibits similar behaviors when interacting with teammates who have comparable behavior patterns. For instance, when comparing the behaviors with the 3-rd and 4-th teammates, only minor differences are observed in how the AI agent places the onion into the pot.
12

---

Published in Transactions on Machine Learning Research (10/2024)
RQ3: Is the value decomposition useful? One component of our approach design is the human-aware actor-critic architecture, which decomposes the global Qiot into the individual Q values of teammates and Al agents. We here design an ablation that drops the value decomposition practice in Coord. Ring layout of Overcooked environment, which means treating the teammates as part of the environment and only learning the Q value of the AT agent. This reduces to applying single agent reinforcement. learning algorithm like previous works (Heinrich et al., 2015 Strouse et al., 2021).
As shown in Fig. ((a) where we report the testing performance all 12 tasks after the whole continual train- ing, the red curve consistently surpasses the orange one. This indicates that the complete Concord algorithm generally shows better anti-forgetting performance over the baseline w/o value decomposition, demonstrating the effectiveness of the human-aware actor-critic architecture to facilitate collaborating learning.
Other Ablation Studies and Further Analysis Due to space limitations, further analysis, such as the number of episodes for recognition, comparison between few-shot and zero-shot recognition, comparison with given teammate ID, ablation studics and scnsitivity analysis of the recognition mechanism, as well as sensitivily analysis of the agent work can be found in Appendix C.2 Furthermore, we also include the discussions with ZSC methods, discussions with meta-RL methods, experiments with more continual teams, as well as experiments conducted on the SMAC environment in the rest of Appondix C
4.3 Continual Coordination between Multiple Human-like teammates and Multiple Al Agents
In the previous experiments, there was only one agent and one teammate in the environment. Multi- player human-AI coordination, i.e. the coordination between multiple agents and multiple teammates, an understudied yet important setting in practical applications. In this section, to test the generalization ability of Concord, we use three maps in the SMAC environment (Samvelyan et al. 2019) to assess the coordination ability between multiple human-like teammates and multiple AI agents. We use DOP (Wang et al., 2021b) to train the human-like teammates. Three checkpoints with low, medium, high win rates during the training process were selected to simulate poor (P), medium (M) and expert (E) human players, respectively. Then, we train the agent via different algorithums to continually coordinate with single human-like teammate (maps 3m and 2s1z) or multiple human-like teammates (map 4m). Fig. [b]] presents the results on map 4m, from which we can see that the gray shaded area of Concord is smallest, indicating that Concord generally has better anti-forgetting performance over other baselines. Results maps 3m and 2s1z are reported in
App. C.6
4.4 Continual Coordination with Humans
Finally, we conduct experiments on the Open Asy. Advantages layout to investigate the ability of coordinat- ing with real humans. We first collect around 160 human-human trajectories (for a total of 64k environment timesteps) and partition them into two subsets, splitting each trajectory into two single-agent trajectories. Based on the collected human data, we construct 8 sets of human proxy models through behavior cloning. Then we train different methods by using these human proxy models as the teammates that appear sequen- tially. Note that the Al agents obtained by different methods will play with the real human participants in the test phase, not the human proxy models. More details about the construction of human proxy models and ethical statement concerning this experiment are provided in App. D.1 and D.2 respectively.
Main results. We show the single coordination performance after training in Fig. 10(a) Among these algorithms, Concord is the closest to oracle in terms of performance, and outperforms CLEAR and EWC
baselines with most teammates, which is consistent with the results in Section 4.2 The Vanilla method performs the worst and shows catastrophic forgetting, which again illustrates the importance of the anti- forgetting in continual human-Al coordination.
Human preference. To check the preference of Al agents obtained by different algorithms of human players, we additionally introduce a popular subjective metric, i.c., human preference (Strouse et al. 2021 Yu et al. 2023), to evaluate different methods, with detailed explanation in App. D.3. A.p.D.3. As shown in Fig. 10(b) Concord
is better than the three baselines and comparable with Oracle. For instance, the percentage of volunteers who
13

---

Published in Transactions on Machine Learning Research (10/2024)
Homan3
Oracle
0.125
0,375
0.5
0.875
Uhmaz4
[limm2
Concord
-0. .0.125
0.375
0.625
0.875
EWC
-0.375
-0.375
0.25
0.75
Hunand
Hamael
CLEAR
-0.5
-0.625
-0.25
0.625
Goooel
Hunazó
Humot
Vanilla
-0.875
-0.875
-0.75
-0.625
Vadaa
Oracle
Concord
EWC
CLEAR
Vanilla
Main results
(b) Human preference
Figure 10: (a) Single coordination performance results with humans. (b) Human preference results for method over column method. The value at i-th row and j-th column is the difference in the percentage of humans who prefer i-th method compared to those who prefer j-th method more.
prefer Concord method is 62.5% higher than those who prefer CLEAR. This shows that Concord is relatively more preferred by humans, demonstrating the value of Concord for practical human-Al coordination.
5 Related Work
Many real-world problems consist of multiple interactive agents, which often be represented as Multi- Agent Reinforcement Learning (MARL) problem (Busoniu et al. 2008 Zhang et al. 2021). Additionally, when these agents share a common objective, it falls under the category of cooperative MARL (Oroo- ilooy & Hajinezhad 2022), which has exhibited significant advancements finding (Sartoretti et al. 2019), active voltage control (Wang et al. 2021a). and dynamic algorithm con- figuration (Xue et al. 2022b). Numerous approaches have been proposed facilitate agent coordination, including policy- based methods such MADDPG (Lwwe al.) 2017) and MAPPO (Yu al., 2022), value- based techniques like VDN (Sunchag et al. 2018) and QMIX (Rashid et al. 2018), and other approaches like the transformer architecture (Wen et al. (022). These methods have demonstrated remarkable coor- dination abilities across wide range of tasks, including SMAC, Hanabi, and GRF Yuu et al. 2022). In
diverse domains such as path
addition to the mentioned approaches and their variants, numerous other methods have been proposed to investigate cooperative MARL, such as efficient communication strategics
(Zhu et al. 2022), offline policy deployment (Zhang et al. 2023), model learning MARL (Wang el al.] 2022a) policy robustness in the presence of perturbations (Guo al. 2022), and training paradigms like CTDE (centralized training with
dccentralized exccution)
[Lyu et al. 2021), and more.
Building Als that can cooperate and coordinate with different leammates or human remains fundamental requirement and challenge for Al (Dafoe et al.) 2021). This ability is essential for numerous applications, in- cluding human-machine communication (Guzman & Lewis 2020), cooperative autonomous vehicles (Toghi et al. 2021,, and assistive robot control (Losey et al.) 2022). typical approach utilizes techniques like modelling (Albrecht: & Stone. 2018) to capture others' intentions or behavior or (repcatedly) build an ef- fective behavior model over human data and plan with the human model (Sheridan, 2016). These methods require expensive and time-consuming data-collection process. Benefiting from the success of cooper- ative MARL (Wong et al., 2022), many related approaches like ad-hoc teamwork (AHT) (Mirsky et al. 2022), zero-shot coordination (ZSC) (Treutlein et al. 2021), few-shot teamwork (FST) (oosong et al. 2022) have been developed recently. AHT is the research problem of designing agents that can coordinate with new teammates without prior coordination (Stone et al., 2010). including teammates type inference (Bar- rett Stone [015 Chen al. (2020), changing point detection (Ravula al. 2019), partial observation solving (Gu et al. 2021). adversarial training (Fujimoto et al. (2022), robustness training (Rahman et al. 2022), etc. The goal of 7SC is to train the agent(s) that can coordinate effectively with wide range of
14

---

Published in Transactions on Machine Learning Research (10/2024)
unseen teammates (Hu et al.) 2020 Treutlein et al. 2021
Many works emerging to improve the ability of multi-agent system from different. aspects, including diversity measurement (Tupu et al.
2021 Zhao et al. 2023), training paradigm design (Hu et al. 2020; Strouse et al. 2021; Xue et al. 2022a). human bias based approaches (Yu et al. 2023), diverse teammates generation [Charakorn et al.) 2023 3, ctc. The agents in AHT aim to coordinate with leammales from a target population, while agents in ZSC need aim to coordinate with arbitrary unseen tcammates without any prior coordination (Mirsky et al., 2022 Lauffer et al. 2023) Few-shot adaptation plays a crucial role in single agent reinforcement learning (Kumar al,, 2020; Osa et al. 2022 Gaya et al. 2022b), which samples K episodes by any policy to obtain latent variable for downstream tasks. In FST setting (Fosong et al. 20222, skilled agents trained in a team to complete one task are combined with skilled agents from different tasks and together must learn to adapt to an unseen but related lask.
Another topic that is related to our work is Continual Learning, which involves sequential representations of tasks or samples (Parisi et al., 2019 Masana et al.) 2020 Kudithipudi et al.) 2022). Continual learning, or to say incremental learning or lifelong learning, is considered in many different fields, while in the field of continual reinforcement learning (Khetarpal et al. 2022). , we expect the agents to sequentially learn preision-making across different tasks. Among the previous works, EWC (Kirkpatrick et al., 2017 conducts l distance-basee weight regularization with the weights learned in the previous tasks. This algorithm requires additional supervision information to select a specific Q-function head and task-specific exploration schedule for different tasks. Different from EWC, CLEAR (Rolnick et al.] 2018) is a task-agnostic method without need for task information during the continual learning process. stores a large experience replay buffer and addresses the forgetting problem by sampling data of previous tasks from the buffer. Several other approaches, such HyperCRL (Huang et al.) 2021) and (Kessler et al. 2022a), utilize learned world model to enhance the efficiency of continual learning. To address scalability issues in scenarios with a large number of tasks, CN-DPM ŞLee et al.) 2020) and LLIRL (Wang et al.) 2022b) decompose the task space into subsets of data (tasks) and employ techniques like Dirichlet Process Mixture or Chinese Restaurant Process to expand the neural network for efficient continual supervised learning and reinforcement learning tasks, respectively. OWT. (Kessler et al. (022b)) is a recently proposed approach that utillzes a multi-head architecture to achieve high learning efficiency. CSP (Gaya et al., 2022a) incrementally builds a subspace of policies for training reinforcement lcarning agent sequence of tasks. Despite the various approaches that have been proposed to solve the continual learning problem, they mainly focus on the field of single- agent reinforcement learning. There is still limited research considering continual learning in multi-agent problem setting, let alone the problem of multi-agent continual coordination learning. MACPro (Yuan et al., 2023b) firstly formulates the multi-agent coordination problem, and develops an efficient algorithm for the setting. Macop (Yuan et al. 2202aa) further endow the controllable agents a multi-agent system with continual learning ability for a high generalization coordination policy.
Even the mentioned methods could endow the trained agents with the ability to coordinate more efficiently with different teammates. They need to train with all teammates at all times, which requires extensive computing and storage resources and extra design, such as prioritized sampling. In this work, we show these approaches are of low learning efficiency and take a further step for this problem by modeling it as a continual learning problem.
6 Conclusion and Future Work
Recognizing the importance and practicality of human-AT coordination, this study takes a significant stride towards addressing this challenge in continual manner. We begin by formulating the problem as a MACMDP, where teammates are encountered sequentially. Subsequently, mechanism based on hyper- teammate identification is proposed to avoid catastrophic forgetting while promote forward knowledge trans- fer for multi-agent coordination. To the best of our knowledge, our proposed method, Concord, is the first to address the human-Al coordination problem via continual training. Experiments on various multi-agent benchmarks validate the effectiveness of Concord, demonstrating its strong ability to continually coordinate with generated human-like models or real human participants across a range of evaluation metrics. Currently, Concord has the following two main limitations. Although Concord alleviates the issue of storage overhead compared to previous methods, it still necessitates a certain amount of storage to prevent forgetting. Be-
15

---

Published in Transactions on Machine Learning Research (10/2024)
sides, addressing large-scale problems poses challenge for current human-AI coordination methods. Further research is required to delve deeper into these two topics. Expanding Concord to decentralized cooperative decision-making settings beyond human- Al coordination is also an interesting future direction.
Acknowledgment
This work is supported by the Natural Science Foundation of Jiangsu (BK20243039, BK20241199).
16

---

Published in Transactions on Machine Learning Research (10/2024)
References
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66 95, 2018.
Samuel Barrett and Peter Stone. Cooperating with unknown teammates in complex domains: A robot soccer case study of ad hoc teamwork. In Proceedings of the Twenby-Ninth AAAI Conference on Artificial Intelligence, pp. 2010-2016 2015.
Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learnin. IEEE Transactions Systems, Man, and Cybernetics, Part C (Applications and Revieus), 38 (2):156-172) 172, 2008.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-AT coordination. In Advances in Neural Information Processing Systems 32, pp. 5175 5186, 2019.
Francisco M Castro, Manuel J Marin-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End- to-end incremental learning. In Proceedings of the European conference computer vision (ECCV), pp. 233 248, 2018.
Rujikorn Charakorn, Poramate Manooupong, and Nat Dilokthanakul Generating diverse cooperative agents by learning incompatible policies. In The 1 11th International Conference Learning Representations, 2023.
Shuo Chen, Ewa Andrejczuk, Zhiguang Cao, and Jie Zhang. Aateam: Achieving the ad hoc teamwork by employing the attention mechanism. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, pp. 7095 7102, 2020.
Allan Dafoe, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, and Thore Graepel. Cooperative ai: machines must learn to find common ground. Nature, 593(7857):33 36, 2021.
Kenneth Derek and Phillip Isola. Adaptable agent populations via a generative model of policies. In Advances in Neural Information Processing Systems 34, pp. 3902-3913, 2021.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer vision-ECCV 2020: 16th European conference, Glasgow, UK, August 23-28, 2020, proceedings, part XX 16, pp. 80-102. Springer, 2020.
Elliot Fosong, Arrasy Rahman, Ignacio Carlucho, and Stefano V Albrecht. Few-shot teamwork. arXiv preprint arXiv:2207.09300 2022.
Ted Fujimoto, Samrat Chatterjee, and Auroop Ganguly. Ad hoc teamwork in the presence of adversaries. arXiv preprint Xrc::22208.00000 05071, 2022.
Jean-Baptiste Gaya, Thang Doan, Lucas Caccia, Laure Soulier, Ludovic Denoyer, and Roberta Raileanu. Building a subspace of policies for scalable continual learning. arXio preprint arXi::2211.1044,, 2022a.
Cean-Baptiste Gaya, Laure Soulier, and Ludovic Denoyer. Learning a subspace of policies for online adapta- tion in reinforcement learning. In The 10th International Conference on Learning Representations, 2022b.
Pengjie Gu, Mengchen Zhao, Jianye Hao, and Bo An. Online ad hoc teamwork under partial observability. In The 10th International Conference on Learning Representations, 2021.
Jun Guo, Yonghong Chen, Yihang Hao, Zixin Yin, Yin Yu, and Simin Li. Towards comprehensive testing on the robustness of cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2204.07992 2022.
Andrea L Guzman and Seth C Lewis. Artificial intelligence and commuiiction: A human-machine com-
munication research agenda. New Media & Sociely, 22(1):70- 86, 2020.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. In The 5th International Conference on Learning Representations, 2017.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cogniline sciences, 24(12):1088-1444 2020.
Johannes Heinrich, Mare Lanctot, and David Silver. Fictitious self-play in oxtensive-form games. In Pro- ceedings of the 32nd International Conference on Machine Learning, pp. 805-813, 2015.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Infor- mation Processing Systems 29, pp. 4563-4573, 2016.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In Proceedings of the 37th International Conference Machine Learning, pp. 4399 4410, 2020.
Yizhou Huang, Kevin Xie, Homanga Bharadhwaj, and Florian Shkurti. Continual model-based reinforcement learning with hypernetworks. In IEEE International Conference on Robolics and Aulomation, pp. 799 805, 2021.
Samuel Kessler, Piotr Milos, Jack Parker-Holder, and Stephen J Roberts. The surprising effectiveness of latent world models for continual reinforcement learning. arXic preprint orX1v:2211.1594, 2022a.
Samuel Kessler, Jack Parker-Holder, Philip J. Ball, Stefan Zohren, and Stephen J. Roberts. Same state, different task: Continual reinforcement learning without interference. In Thirty-Sirth AAAI Conference on Artificial Intelligence, pp. 7143-7151 31511,1, 2022b.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learn- ing: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401-1476, 2022.
Cory D Kidd and Cynthia Breazeal. Robots at home: Understanding long-term hurnan-robot interaction. In IEEE/RS) International Conference on Intelligent Robots and Systems, pp. 3230 3235, 2008
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barbbikkaaariikkaa,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3528, 2017.
Glen Klien, David D Woods, Jeffrey M Bradshaw, Robert R Hoffman, and Paul J Feltovich. Ten challenges for making automation a "team player" in joint human-agent activity. IEEE Intelligent Systems, 19(6): 91 95, 2004.
Raphael Koster, Jan Balaguer, Andrea Tacchetti, Ari Weinstein, Tina Zhu, Oliver Hauser, Duncan Williams, Lucy Campbell-Gilinghaan Phoebe Thacker, Matthew Botvinick, et al. Human-centred mechanism design with democratic AL. Nature Human Behaviowr, (110):1398-1444 2022.
Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, et al. Biological underpin- nings for lifelong learning machines. Nature Machine Intelligence, 4(3):196-25) 210, 2022.
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent RL. In Advances in Neural Information Processing Systems 33, pp. 1198-82101 2020.
Niklas Lauffer, Ameesh Shah, Micah Carroll, Michael D. Dennis, and Stuart Russell. Who needs to know? Minimal knowledge for optimal coordination. In Proceedings of the 40th International Conference Machine Learning, pp. 18599-18613, 2023.
Soochan Lec, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. In The 8th International Conference on Learning Representations, 2020.
David Lopez-Par Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems 30, pp- 8467-6476, 2017.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Dylan P Losey, Hong Jun Jeon, Mengxi Li, Krishnan Srinivasan, Ajay Mandlekar, Animesh Garg, Jeannette Bohg, and Dorsa Sadigh. Learning latent actions to control assistive robots. Aulonomous robols, 46(1): 115 147, 2022.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-copetiiive environments. In Advances in Newral Information Processing Systems 30, pp. 6379-63999 2017.
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordina- tion. In Proceedings of the 38th International Conference Machine Learning, pp. 7204-7213, 2021.
Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting centralized and decentralized critics in multi-agent reinforcement learning. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 844-852, 2021.
Anuj Mahajan, Mikayel Samvelyan, Tarun Gupta, Benjamin Ellis, Mingfei Sun, Tim Rocktäschel, and Shimon Whiteson. Gencralization cooperative multi-agent systems. orXiv preprint 22i:2202.0004,, 2022.
Marc Masana, Xialci Liu, Bartlomicj Twardowski, Mikel Menta, Andrew Bagdanov, and Joost van de Weijer. Class-incremental learning: survey and performance evaluation on image classification. arXiv preprint Xrxi::2010. 15277, 2020.
Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V Albrecht. A survey ad hoc teamwork research. In Europeon Conference on Mulli-Agent Systems, pp. 275 293, 2022.
Bilge Mutlu, Allison Terrell, and Chien-Ming Huang. Coordination mechanisms in human-robot collabora- tion. In Proceedings of the Workshop on Collaborative Manipulation, pp. 1 6, 2013.
Afshin Oroojlooy and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning. Ayplied Intelligence, pp. 1 1-46, 2022.
Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Discovering diverse solutions in deep reinforcement learning by maximizing state-acce iitee-action-basee mutual information. Neural Networks, 152:90-104, 2022.
Johannes Von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. In The 8th International Conference on Learning Representations, 2020.
Xinlei Pan, Weiyao Wang, Xiaoshuai Zhang, Bo Li, Jinfeng Yi, and Dawn Song. How you act tells a lot: Privacy-leaking attack on deep reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 368 376, 2019.
Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial observ- ability for deep reinforcement learning. In Advances in Neural Information Processing Systems 34, pp. 19210, 2021.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and Stefano V Albrecht. Towards robust ad hoc teamwork agents by creating diverse training teammates. arXiv preprint 0rXic:2207.44338 2022.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the 35th International Conference Machine Learning, pp. 4295-4304, 2018.
Manish Ravula, Shani Alkoby, and Peter Stone. Ad hoc teamwork with behavior switching agents. In Proceedings of the Twenty-Eine Fighth International Joint Conference on Artificial Intelligence, pp. 550-556, 2019.
19

---

Published in Transactions on Machine Learning Research (10/2024)
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne. Experience replay for continual learning. In Advances in Newral Information Processing Systems 32, pp. 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Focrster, and Shimon Whiteson. The Starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Aulonomous Agents and MultiAgent Systems, pp. 2186 2188, 2019.
Guillaume Sartoretti, Justin Kerr, Yunfei Shi, Glenn Wagner, TK Satish Kumar, Sven Koenig, and Howic Choset. Primal: Pathfinding via reinforcement and imitation multi-agent learning. IEEE Robolics and Automation Letters, 4(3):2378 -2358-23855 2019.
Thomas B Sheridan. Human-robot interaction: status and challenges. Human factors, 58(4):525--33 532, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanclot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140- 1144, 2018.
Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. Ad hoc autonomous agent teams: Col- laboration without pre-coordination. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, pp.
2010.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. In Advances in Newral Information Processing Systems 34, pp. 14502 14515, 2021.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, and Thore Graepel. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2085 2087, 2018.
Gerald Tesauro. Td-gammon, self- self-teaching backgammon program, achieves master-level play, Neural computation, 6(2):215 219, 1994.
Behrad Toghi, Rodolfo Valiente, Dorsa Sadigh, Ramtin Pedarsani, and Yaser P Fallah. Cooperative au- tonomous vehicles that sympathize with human drivers. In IEEE/RS. International Conference on Intel- ligent Robots and Systems, pp. 4517-4524 2021.
Johannes Treutlein, Michael Dennis, Caspar Oesterheld, and Jakob Foerster. A new formalism, method and open issues for zero-shot coordination. In Proceedings of the 38th International Conference on Machine Learning, pp. 10413-10422 2021.
L. Van der Maaten and G. E. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9 (11):2579-2605, 2008.
Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C. Green. Multi-agent reinforcement learning for active voltage control on power distribution networks. In Advances in Newral Information Processing Systems 34, pp. 3271-3284, 2021a.
Xihuai Wang, Zhicheng Zhang, and Weinan Zhang. Model-based multi-agent reinforcement learning: Recent progress and prospects. arXie preprint orXi::2203. 2022a.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. DOP: Off-policy multi-agent decomposed policy gradients. In The 9th International Conference on Learning Representations, 2021b.
Zhi Wang, Chunlin Chen, and Daoyi Dong Lifelong incremental reinforcement learning with online bayesian inference. IEFE Transactions on Neural Networks and Learning Systems, 33(8):4003-400 -4006, 2022b.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. In Advonces in Neural Information Processing Systems 35, pp. 16509 16521, 2022.
Maciej Wolczyk, Michal Zajkac, Razvan Pascanu, Lukasz Kucifiski, and Piotr Miloš. Continual world: A robotic benchmark for continual reinforcement learning. In Advances in Neural Information Processing Systems 34, pp. 28496 28510, 2021.
Annic Wong, Thomas Bäck, Anna V Kononova, and Aske Plaat. Deep multiagent reinforcement learning: Challenges and directions. Arlificial Intelligence Review, pp. 1 1-34, 2022.
Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to influence multi-agent interaction. In 4th Conference on Robot Learning, pp. 575-588, 2021.
Ke Xue, Yutong Wang, Cong Guan, Lei Yuan, Fu Haobo, Fu Qiang, Chao Qian, and Yang Yu. Heterogeneous multi-agent zero-shot coordination by coevolution. arXiv preprint arXiv:2208. 04957, 2022a.
Ke Xue, Jiacheng Xu, Lei Yuan, Miqing Li, Chao Qian, Zongzhang Zhang, and Yang Yu. Multi-agent dynamic algorithm configuration. In Advances in Neural Information Processing Systems 35, pp. 20147- 20161, 2022b.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Advances in Neural Information Processing Systems 35, pp.
2022.
Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learning zero-shot cooperation with humans, assuming humans are biased. In The 11th International Conference on Learning Representations, 2023.
Lei Yuan, Lihe Li, Ziqian Zhang, Feng Chen, Tianyi Zhang, Cong Guan, Yang Yu, and Zhi-Hua Zhou. Learning to coordinate with anyone. preprint arXiv:2309.1900 2023a.
Lei Yuan, Lihe Li, Ziqian Zhang, Fuxiang Zhang, Cong Guan, and Yang Yu. Multi-agent continual coordi- nation via progressive task contextualization. preprint or orXiv:2305. 13937, 2023b.
Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. survey of progress on cooperative multi-agent reinforcement learning in open environment. arXiv preprint ar arXi:23312.01058, 2023c.
Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang. Discovering gener- alizable multi-agent coordination skills from multi-task offline data. In The 11th International Conference on Learning Representations, 2023.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pp. 321-384, 2021.
R. Zhao, J. Song, H. Hu, Y. Gao, Y. Wu, Z. Sun, and Y. Wei. Maximum entropy population based training for zero-shot human-ai coordination. In Proceedings of the 37th AA Conference on Artificial Intelligence. (AAA!), pp. 6145 6153, 2023.
Zhi-Hue Zhou. enviroennnnnmmnt machine learning. National Science Review, 9(8,, 2022.
Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent reinforcement learning with communication. Xiv preprint Xrxi:22003 08975, 2022.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Appendix
A
More Details about the Proposed Method
A.1 Network Architecture
Predicted Teummate Rep.
Teummate Rep.
GRU O
St
FC
FC
FC
Actor Netwock
PC
FC
PC
its
fiy
lss
GRU
hg
by, RNN Hidden
20x20
FC
S/ us!
Recognizer
Na. ag
Sg. 02
32x21
FC322200
FC32x20
Recognizer
Trajectory
Reconstruction Process
Human Trajectory
Hlper-Natwokk
(a)
(b)
Figure 11: (a) Network architecture of the hyper-network and actor network. (b) Network architecture of the recognizer network. T, in the figure is short for T.g, indicating the horizon of the human trajectory T!!
Hyper-Network The hyper-network in our method is used to generate the parameters of the AI actor by inputting the tcammate reprosentation to it. Thus, the network architecture determines the size of parameters generated by the hyper-nelwork. We model the actor network with three layers shown in Fig. [I(a) including fully-connected layer, followed by 64 bit GRU, and further followed by another fully-connected layer that outputs a probability distribution over local actions. In our design, the paramoters of this actor network completely generated by the hyper-network. We model the pyper-network as three multi-layer perceptions (MLPss, each generating the parameters of one layer of the actor network, which is also shown in Fig. [1) We set each MLP with two hidden layers of 20 neurons and ELU non-linearity function, and the imput dimension of each MLP network equals to the dimension of tcammate representation, which is set 32 throughout all our experiments.
The recognizer network (see Fig. 11(b) is utilized to reconstruct the teammate representation from a few of human interaction trajectories. Specifically, the recognizer network is built as RNN-based trajectory encoder, which takes the human trajectory as input. For each human trajectory T! the recognizer will output a predicted teammate representation for each timestep t, which we denote as $t, with the trajectory information before t. Then we update the recognizer network via minimizing the following reconstruction loss: Creoon Σ
Recognizer Network
T.-1
T. 1T-11-1122 112 =2.122-
(2)
which is a weighted sum of the & distances between every &f and zc, where we utilize one hyper-parameter to control the weights. On one hand, by minimizing the l distance between each at and &ç; we ensure the recognizer's ability reconstruct the representation &ς from variable-length trajectory information. On the other hand, we assign larger weights to the at loss terms with larger t. The reason for this practice is that during the final test phaso, only the predicted teammate representation at the last timestep 27--1 is utilized the input of the subsequent networks, which means that hope that & with larger should be more accurate. From a different perspective, the reconstruction losses for &t where t is smaller than T, are utilized auxiliary losses to facilitate the recognizer network training. In practice, the ppper-parameter À is set 0.994 and 0.92 in Overcooked and SMAC, respectively. Above, have explained how the recognizer
22

---

Published in Transactions on Machine Learning Research (10/2024)
Teammate Policies
Al Agent Policies
n (1
T
Hyper-network
hyper
F
payper
hyper
hyper
hyper
Teammate Representations
2
Ø1
222
T Z2
33
23
87 "
2
Recognizer
fg
cene
?
H
Teammate Datasets
Replay Buffer,
{(sf". at) ) (s at ).... (s u
T(C)
Continual Learning
Unknown Teammate
Figure 12: A high-level workflow showing the main idea of approach. We use & to denote the initialized representation and &ς for the learned leammnae represenlation. During the recognizer training, each task stores a small amount of data in the Replay Buffer, where N is the number of trajectories per teammate, and supervised learning is performed to align the behavioral trajectories and teammate representations.
network generates the reconstructed teammate representation with human trajectory. However, we aim to utilize multiple trajectories to recognize teammates in our algorithm. In practice, firstly generate representation with each human trajectory and then compute the mean vector of these representations.
A.2 Overall Workflow Concord
Following the workflow of the multi-agent continual coordination learning in Fig. 2 the main idea of our approach is depicted in Fig. 12 In more details, Concord can be decomposed into two phases, the training phase and the test phase. During the training phase, we conduct the continual coordination learning, and during the test phase, we recognize the involved human teammates and let the AI agents cooperate with them. The overall flow of these two phases are respectively shown in Alg, and Alg. 2 in the maintext.
A.3 Storage Requirement During Continual Learning
To address the potential forgetting problem of the hyper-network and recognizer network, Concord needs to store some variables when learning continually. Specifically, algorithm requires storing the following two types of variables:
The parameters the hyper-network the last round, denoted 3, are used compute Lew, This variable always occupies static storage space, which is not relevant to the continual learning length T. Therefore, the storage cost for 15 is considered acceptable.
b. A small buffer B is utilized to store a few of human trajectorics and tcammate representations for previous leammates. For each encountered teammate policy = we store 36 human trajectories and one representation vector with dimension of 32. The final storage cost for these variables grows linearly with the continual learning length T. In practice, we find that the storage cost for these variables are relatively acceptable because we only store few of human trajectories and the storage cost for teammate representations small.
23

---

Published in Transactions on Machine Learning Research (10/2024)
B Experiment Setup
B.1 Syper-parameen Selection
In Tab. 2 we list the selection apper-parameters that are critical to our algorithm. We adopted this set of syper-paramcters throughout all the experiments this paper. However, we utilized different training periods per task on different environment scenarios, of which the details are listed in Tab.
Table 2: Hyper-parameters selected for our proposed approach.
Hyper-parameter
Value
Discount factor 7
0.99
RMSprop learning rate for actor / critic
0.0005 / 0.0001
Adam learning rate for hyper and recognizer network
0.001
Batch size for actor learning
32
Buffer size for critic learning
5000
Max/Min value for e-greedy exploration
0.5 / 0.05
Step number of exploration decay period
500k
Embedding dimension for teammate representation (Demb)
32
Regularization loss weight (6**)
2.0
Episode number for teammate recognition
2
Episode number for training / validating recognizer network
32 4
Table Training period per task for different environment scenarios. Environment
Scenario
Step nunber
Coord. Ring
5M
Asy. Advantages
10M
Overcooked
Many Orders
7M
Open Asy. Advantages
7M
3m
3M
SMAC
2x1s ls
1.5M
4m
2M
B.2 Construction of Human-like Teammates
To evaluate the performance of coordination with human-like teammates, we should first construct a set of teammates. Building AIs that can cooperate with humans remains a fundamental challenge (Dafoc let al. )021), as requires human-like teammates under the property of high behavior diversity. We here use Maximum Entropy Population-based training (MEP) (Zhao et al., 2023) and Hidden-utilies utility Self-Play (HSP) (Yu al.) 2023) algorithms to model diversity of human-like behavioral policies, for their proven diverse agent ability in the Overcook environment. MEP trains partners with population entropy bonus to promote the pairwise diversity between partners and the individual diversity of partners themselves. HSP explicitly models human biases as hidden reward functions in the self-play objective thus can generate an augmented policy pool with biased policies. We use the code released by the HSP algorithn[] to train MEP and HSP policies, and the number our human-like models defined as 12 (among them, MEP policies and 4 HSP policies), to simulate real human behavior diversity and preference. As shown in Fig. 13 the diverse teammates allow the agent to learn more universal coordination behaviors, and therefore have stronger continual human-Al coordination performance.
The code used HSP https: wtt.://...conmaaaaaao0//////////
24

---

Published in Transactions on Machine Learning Research (10/2024)
Human-like Models:
12
(a) Coord. Ring
(b) Asymm. Adv.
(c) Many Orders
(d) Open Asymm. Ado.
Figure 13: Ilustratioon of diversity of leamintts.. We plot the distribution of teaminttes' actions Tp {- F by t-SNE (Van der Maaten Hinton 2008) projection, where each point represents the t-SNE results of teammates action distribution in each episode, and different teammates have different action distributions, thus the obtained teammates can be seen diverse.
Additional Experimental Results
C.1 Additional Results Overcooked Environment
In Sec. 4.2 of the manuscript, we compare the detailed single coordination performance of Concord and Vanilla on the Asym. Adv. layout of Overcooked environment. To further analyze the situations on several other layouts, present. additional experimental results on the other three layouts here. Fig. 14 16 demonstrate that Concord continues to suffer less forgetting and achieves better forward transfer performance.
96
130
45
85
142
99
120
72
83
112
106
115
42
72
1G5
115
95
62
72
56
37
49
130
90
131
105
27
123
108
112
181
131
120
115
118
145
133
118
94
118
93
130
112
1:7
37
56
36
147
56
84
114
106
115
39
152
126
99
44
66
156
151
176
50
74
145
113
105
64
114
116
106
137
162
136
175
1m
120
110
61
53
76
155
128
39
109
113
100
121
125
115
129
155
134
132
129
2
10
11
12
2
7
11
12
Partner Index
Partner Index
Figure 14: Single coordination performance of Vanilla (left) and Concord (right) on the Coord. Ring layout. The value at i-th row and j-th the matrix is P((j).
C.2 Additional Analysis
One important part of Concord is to utilize the recognizer network recognize the teammates through few of interaction trajectories during the test phase. To analyze whether such few-shot mechanism is necessary and examine how the number of episodes used for teammate recognition influences the algorithm performance, we conduct ablation studics on the layout Asym. Adv. of Overcooked environment. We first provide the experimental results under different numbers of episodes for recognizing leammales, and then we further investigate the zero-shot coordination ability of different methods. Moreover, we also provide further analysis, including comparison with given teammate ID, ablation studies and sensitivity analysis of the recognition mechanism, as well as sensitivity analysis of the agent actor network architecture in this section.
25

---

Published in Transactions on Machine Learning Research (10/2024)
24
10
115
17
2
195
59
15
203
25
36
35
51
223
102
92
162
146
200
61
130
94
3
18
37
11
59
134
66
13
25
51
13
331
165
139
168
101
704
14
31
116
67
111
64
46
671
65
34
122
73
223
159
115
114
210
12
126
L25
150
133
131
118
125
116
66
91
143
11
169
85
234
154
119
173
112
99
59
118
107
45
41
130
V7
64
62
46
152
118
76
224
141
115
194
140
319
96
83
141
162
116
64
10
1S4
47
135
162
37
57
115
131
67
222
155
113
161
106
218
160
134
125
113
65
50
16
137
114
114
142
2
11
11
12
Index
Partner Index
Figure 15: Single coordination performance of Vanilla (left) and Concord (right) on the Many Orders layout. The value at i-th row and j-th in the matrix is P.(j).
119
I53
151
151
335
I71
150
152
90
136
33
150
158
44
187
23
2
146
179
156
111
63
6
144
195
105
231
15
196
159
35
185
10
3
175
1S
142
137
196
300
145
190
150
196
190
166
192
172
122
172
165
112
159
116
130
173
190
8
47
1:7
311
127
65
131
112
60
62
131
195
195
175
200
195
175
145
179
195
1877
180
10
1.5
204
195
149
189
50
195
189
Partner Index
Partner Index
Single coordination performance of Vanilla (left) and Concord (right) the Open Asy. Adv. layout. The value at i-th row and j-th in the matrix is P.(j).
Figure 16:
Number of episodes for recognition. The selection of the number of episodes for human recognition is a critical hyper-parameter in Concord. As depicted in Fig. 17 (a), utilizing 2 episodes yields notable improvement compared to using only 1 episode. Although employing more episodes necessitates additional data, the impact is not substantial. Consequently, the number of episodes for human recognition has been set to in our experiments.
Few-shot and zero-shot recognition. We ablate the few-shot mechanism for different methods, shown in Fig. 17 (b). For the zero-shot version of our approach, we refrain from pre-sampling certain trajectories to recognize teammates. Instoad, we utilize in-episode trajectory information to estimate the tcammate representation, which is subsequently inputted inlo the hyper-nelwork. Additionally, to better differentiate between the methods, we evaluate their generalization ability this ablation study by conducting tests with an additionally generated teammate policy. As we can see from the results, Concord achieves better performance than other methods under zero-shot setting, while only Concord with few-shot mechanism achieves comparable performance to the Oracle baseline. This result demonstrates the necessity of the few-shot mechanism.
Comparison with given teammates ID. To investigate the influence known teammates, we ad- ditionally test the performance of Concord (ID), which can be seen as Concord equipped with an Oracle identification mechanism. As show in Tab. we can see that Concord (ID) shows a slight advantage over Concord, but has significant differences. This means that the identification accuracy of our identification
26

---

Published in Transactions on Machine Learning Research (10/2024)
Average reward per episode
Average reward per episode
150
160
100
140
50
120
100
5
Oracle EWC CLEAR Naive zeroshot
Concord fewshot
Number of episodes for recognizing human
Figure 17: (a) The performance comparison with different episode numbers for the recognizer. (b) The performance of different algorithms in zero-shot testing and the performance of Concord algorithm in few- shot testing.
Table 4: Comparisons of distinct algorithms in all 4 layouts on average coordination performance with 5 random seeds. Concord (ID) is that the agents are given identities, which can be seen as Concord equipped with an Oracle identification mechanism. The best mean value on each problem except for Oracle is high- lighted in bold. The symbols +, -, and s indicate that the result is significantly superior to, inferior to, and almost equivalent to Concord, respoctively, according to the Wilcoxon rank-sum test with a significance level of 0.05.
Enviromment
Oracle
EWC
CLEAR
Vanilla
Concord (ID)
Concord
Coord. Ring
127.05 3.23
89.38 = 11.87
102.54 3.25
13.55:12.50
126.34 +: 2.86
123.28 3.00
Asy. Advantages
191.91 2:42
5148866881
30.22:3.228
47.28 9.76
169.27 1 6.39
160.08 =: 13:45
Many Orders
140.81 + 4,22
98.03
22.81
180.45
6.01
79.85 + 9.82
137.22 5.48
134.76 4.01
Open Asy. Advantages
197.83 4.87
177.63 +: 3.58
106.95 19.78
06.03 :: 3.24
191.06 6.84
189.68 5.71
+/-/20
2/0/2
0/4/0
0/3/1
0/4/0
0/0/4
mechanism is effective. Besides, the performance comparison between Concord (ID) and Oracle validates the effectiveness of other parts of Concord besides our identification module.
Ablation study for anti-forgetting mechanism. The anti-forgetting mechanism also an important component of our method. To verify the effectiveness of this module, design additional ablation experi- ments concerning this part. Specifically, we remove this anti-forgetting mechanism and conduct experiments on Overcooked environment, with the final results presented in Tab. 5 The results show that when the anti-forgetting mechanism removed, our method suffers a significant performance drop, demonstrating the effectiveness of the anti-forgetting mechanism.
Sensitivity analysis of replay buffer size. To analyze the sensitivity of the replay buffer size in terms of addressing the forgetting issue, we add a sensitivity analysis in Tab. G It can be observed that Con- cord achieves relatively good performance across the ranges of buffer size for each tcammate from 8 to 64. Note that Concord distinguishes itself from conventional replay-based methods to prevent any potential misunderstanding. In our approach, the replay buffer is solely employed to prevent the forgetting of team- mate recognition rather than to help learn solving previous tasks. An example of conventional replay-based methods in our experiments is CLEAR.
Additionally, to further illustrate the rolc of teammate recognition, we conducted an experiment using random DDs. Unlike Concord, which identifies leamintees by inputting trajectories inlo a recognizer net work, Random ID randomly selects one representation from all stored teammate representations during the testing phase. We can observe that the performance of Concord given the random ID worse than standard Concord. This demonstrates that our teammate recognition module can accurately identify teammates and leverage their information to achieve better collaboration.
27

---

Published in Transactions on Machine Learning Research (10/2024)
Table 5: Experimental results of ablating the anti-forgetting mechanism. The "Ablation" in the table indicates Concord without anti-forgetting mechanism, and the better results bold.
Courd Ring
Asy. Advantages
Many Orders
Open Asy. Adv.
Method
Concoed
Ablation
Conrord
Concord
Concard
Ablution
Episode Reward
123.28 3.01
85.25 +: 10.23
160.08
13.45
50.12 à: 8.76
134.76 4 4.01
81.13
189,68 5.71
90.99 10.87
Forg.
-1.31 + 4.23
-20.15 + 2.52
-16.11 6.86
-9045 + 18.96
-1.27 + 6.69
-60.28 2.71
-6.83 + 4.59
-100.08 + 20.31
Transfer
13.02
78.84 11.79
39.89
68.87
162.29
17.52
Table 6: The sensitivity analysis of replay Buffer. The best mean results for Episode Reward are in bold. Here, Forg. represents the metric of Forgetting, F. Transfer represents the metric of Forward transfer, Random Rep. represents the identification mechanism with randomly generated representations, Random ID represents the identification mechanism that randomly selects one teammate representation from all proviously stored tcammate representations, and Oracle ID represents Concord with an oracle identification mechanism.
Coord. Ring
Asy. Advantages
Size
Episode Reward
Forg.
F. Transfer
Episode Reward
Forg.
F. Transfer
4
109.97 + 0.98
-3.67 1.67=5.60
99.69 = 1.31
81.97 7.32
644.62
108.57 + 14.17
117.72 + 1.97
-7.78 = 1.85
113.63 2.72
128.61 + 13.87
1
8.31
16
122.16 1.86
-2.01 1.13
114.81 1: 1.72
136.19 13.61
15.336
143.57 1 8.46
24
126.51 + 1.94
4.21
117.09
165.94 3.64
-..73
136.42 + 9.83
32
124.91 + 1.96
-1.93
113.57 + 2.56
2.56
166.41 016.4114.00 +.47
-12.46
131.01 5.04
48
126.27
00.27
113.94 2.03
167.75 + 13.96
16.49
148.96 4.92
64
111.94 0.48
1.84
104.21 2.08
166.8611.777
3.7712
123.81 +: 16.62
Random Rep.
36.92 0 6.94
-1.58+21.68
18.45+5.660
33.27 + 8.15
-0.36 14.83
39.36 + 14.17
Random ID
87.63 + 9.58
-1.27 1.14
85.32 + 11.92
110.61 + 9.98
-5.42 + 0.93
92.75 + 7.34
Oracle
126.34 + 2.86
1.76
88.45 0.25
169.27 + 6.39
-3.36 + 3.12
109.79 + 2.02
Sensitivity analysis of the size of 8Ac, The default configuration of the agent's network 3-layer the agent's performance
Multilayer Perceptron (MLP) with hidden size of 64. We aim function of the size of 5 To do this, we modified the network structure to produce agents with varying 0A sizes and then compared their performances. "Concord (32)" denotes agent with hidden layer size of 32, whereas "Concord (4 layers)" refers to 4-yyyer MLP with each hidden layer sized at 64. Our findings indicate that the parameters generated by hyper-network are adequately expressive to accomplish the task. A slight reduction in the number of parameters does not significantly degrade performance. In contrast, as we continue to increase the size of the network parameters, performance slightly declines. This may be due to the larger parameter size requiring more data for training.
Table 7: The sensitivity analysis of 8 Concord (32) represents a hidden layer size of 32, while Concord (4 layers) represents a 4-layer MLP with hidden layer size of 64.
Coord. Ring
Average coordination performance.
Forgetting
Forward Transfer
# Parameters
Concord (64)
123.28 + 3.01
-1.31 + 4.23
103.09 + 13.02
631398
Concord (32)
119.83 = 10.92
-4.18 + 1.82
90.765 + 9.95
188550
Concord (48)
120.83 + 9.05
-4.05 + 2.87
79.27 + 16.07
377718
Concord (96)
126.55 + 3.77
-2.15 + 5.21
87.69 + 10.03
1332294
Concord (128)
118.66 8.24
-3.93 3.68
84.96 3.05
2291238
Concord (256)
108.83 = 7.99
-7.21 4.28
76.99 8.01
8707494
Concord (4 layers)
121.83 + 2.21
-3.53 + 1.37
79.02 0.61
719838
28

---

Published in Transactions on Machine Learning Research (10/2024)
Mony Orders
Open Any. Advantages
ZSC (QoiSero)
ZSC Bascnia Feg.
115
ZSC (Baomiia E. Tanskr
Concert thog
120
135
Avcigereva
100
25C (Lrnfem)) Fres
25C (Uafeen)))) E. Tanni
30
(Biaznial)
Cuscod Frog.
Casead F. Tuzel:
Figure The performance comparison between Concord and 7SC (i.e., multi-task learning) on Many Orders and Open Asy. Advantages.
C.3 Discussions with Zero-Shot Coordination (ZSC) Methods
In human-Al collaboration, the classic approach (i.e., Zero-Shot Coordination, ZSC (Strosse et al. 2021; Zhao et al.) 2023 Yu et al.) 2023)) is to pre-generate all the diverse teammates and allow the Al agent to simultancously train with them. In many real-world applications, humans do not collaborate with robots this way. We do not want an AI to stop updating after collaborating with humans. Instead, as more and more people collaborate with the agent, the robot. should continue to learn and improve its collaboration skills. This is the desired capability for agent in human-Al collaboration scenarios. Thus, the training approach of ZSC has several drawbacks, such as necossitating access to all of them during training is unfeasible and enumerating all potential leammates in advance is very hard, as we discussed in our paper.
In the aforementioned realistic scenarios, we hope that after the agent collaborates with a human to complete task, it will retain the ability collaborate with them and improve its ability cooperate with other teammates in the future. These two requirements correspond the two important indicators in continual learning: anti-forgetting and forward transfer. We frame this challenge within the context of continual learning and seek solutions accordingly.
The essence of ZSC is multi-task learning, which allows the AI agent to train with multiple teammates simultancously. This differs from our continual scenario where we believe that teammates arrive in sequence and the agent can only train with the current leam, not past ones. Besides, the focus of ZSC methods is to provide a set of good partners for teammate training, while our work proposes a new problem formulation. We can also use ZSC techniques, as we did when constructing human-like teammates in our experiments.
Here, we conduct additional experiments to enhance the comparison between our setup and ZSC, which is akin to multi-task learning. In our experimental design, teammates are introduced sequentially, and both Concord and ZSC are allocated the same training budget. However, unlike Concord, ZSC is permitted to train with past teammates. When a new teammate (here we consider 2-player scenarios for simplicity) arrives, there should be a strategy to allocate the training timesteps between the new teammate and old ones. We propose two strategics, respectively "Uniform" " and "Binomial" strategics. The "Uniform" strategy implies that all leammates are allocated an equal number of training timesteps during the current training session. The "Binomial" strategy indicates that the new teammate and all old teammates each receive half of the timesteps. It is worth noting that if the new teammate uses all the timesteps without training with the old teammates, the ZSC method is indeed the "Vanilla" method we discussed in our paper. This comparison more clearly differentiates our setup from ZSC. Note that this setup is not fair to us, as we can not retrain with past teammates.
The experimental results can be found in Fig. 18 Concord significantly outperforms ZSC using different strategies, demonstrating the significant advantages of our algorithm in this practical scenario.
29

---

Published in Transactions on Machine Learning Research (10/2024)
Table 8: Comparison with two Meta RL baselines.
The best results are bold.
Coord. Ring
Asy. Advantages
Method
Episode Reward
Torg.
F. Transfer
Episode Reward
Forg.
F. Transfor
Concord
123.28 3.01
-1.31 + 4.23
103.09 13.02
160.08 + 13.45
-16.11 + 6.85
119.59 + 9.41
PEARL
87.19 + 1.31
-43.43 + 1.28
87.28 + 6.14
122.25 + 5.44
-108.25 + 5.2493
68.15 + 3.30
PEARL w/ Auti-Forg
88.50 1.66
-32.15 3.09
81.85 1.70
133.50 8.88
-90.50 4.08
79.75 + 4.55
C.4
Discussions with Meta RL Methods
Our paper proposes new formulation MACMDP, where the agents learn to cooperate with different team- mates continually. Another domain investigating the agent's adaptability is meta RL. However, although continual RL and meta RL involves the changes of task distribution, but they are two completely different problem formulations. Continual RL emphasizes maintaining performance on both old and new tasks in a changing environment, while meta RL Icarns how to learn and focuses on improving the lcarning efficiency on one new task. We strees that Concord follows continual RL setting, and our evaluation phase involves only testing with teammates that have already been encountered, rather than involving coordination with unseen tcammates like in meta RI.
Though our Concord follows continual RI, a setting different from meta RL, it shares certain similarities in practice with context-basee based meta RL method, as both involve learning task representations. However, EWC and CLEAR do not have such recognization mechanism. To strengthen baseline, we introduce PEARL, a classic context- based meta RL method, to our problem setting.
We design two variants to adapt to our setting, PEARL and PEARL w/ Anti-Forg. PEARL, compared Vanilla, adds an encoder that processes trajectory data model teammnte representations. This encoder is optimized using reinforcement learning losses and a standard normal distribution regularization, described in the original PEARL study. PEARL w/ Anti-Forg mitigates the challenge of forgetting in continual learning by utilizing small buffer to store trajectories from previous training stages like us. When training with the current teammate, it regularize the out puts of the current Al policy and encoder networks to be close to those of the last-round networks on the old data. This is a typical practice in regularization based methods (Castro ct al. 2018 Douillard et al. 2020).
From the results, we can find that PEARL performs poor in our experiments, which reveals the significant differences between meta RL and our problem setting, where meta RL trains the policy on set of tasks during the meta-train phase, while PEARL needs to face the tasks one-by-one in our problem setting, Besides, though PEARL w/ Anti-Forg performs slightly better than PEARL, our Concord still exhibits significant superiority over it. The reason why PEARL w/ Anti-Forg does not achieve ideal results might be that to constrain the output of the policy network to be close to the old network often requires storing more old data. Thus, when both storing 64 trajectories, our method shows better performance, reflecting the advantage of employing the hyper-network.
C.5 Experiments with More Continual Teams
An interesting questions is to investigate the performance when have more teams to be contimually coordinate. In this section, we double the teams to 24 to show that. For better presentation, compared Vanilla and Concord and plotted the entire single coordination performance, which can be used to obtain all other metrics. The results can be found in Fig. 19. We found that, even though having more teammates brings significant challenges to continual learning, our Concord method still performs well and significantly outperforms Vanilla.
C.6 Complete Value Decomposition Study Results
In Sec. 4.1 of the manuscript, we study the influence of value decomposition in the Coord. Ring layout of Overcooked environment with line charts, of which the concrete statisties are presented in Tab. To
30

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 19: Single coordination performance of Vanilla (left) and Concord (right) on the Asym. Adv. layout with more partners. The value at i-th row and j-th in the matrix is P:(j).
08
EWC
0.8
CLEAR
0.6
0.6
Memory Decay
0.4
0.4
02
Malium
Figure 20: Experimental results on SMAC maps 3m and 241s. The gray shaded bar is used to denote the degree of forgetting, which equals to the highest test win rate during training minus the final test win rate.
further validate the effectiveness of value decomposition when there exist multiple AI agents cooperating with multiple human-like teammates, we further conduct value decomposition study on different. SMAC Similar to the results in Overcooked, Concord with value decomposition also achieves better performance different SMAC maps, which further significantly remonsttrats the effectiveness of our value decomposition.
maps, of which the results are reported in Tab. 10 12
Table 9: Performance comparisons on eleven tasks between Concord with and without value decompasition in the Coord. Ring layout of Overcooked environment.
Algoritha
Task1
Tuok2
Twok
Twok
Taoks
Taskii
Task7
Task8
Tasky
Taskid
Task11
Tuok1
Averaae
Concord
113
103
121
128
113
129
158
134
132
110
129
123
124
Concord w/o Human Critie
100
100
92
101
89
148
162
117
107
104
118
134
114
C.7 Additional Results SMAC Environment
In Sec. 4.2 of the manuscript, we compare Concord with other methods the SMAC map 4m. Here, further provide the experimental results on the other two SMAC maps in Fig, 20 to demonstrate the consistency of the conclusion across different maps. From the results, we can observe that the Concord method still exhibits the smallest gray shaded region in these two maps, indicating that generality of the anti-forgetting ability our approach.
Detailed Information for Human Participation
D.1 Experimental Settings for Continual Coordination with Humans
To assess the performance of Concord in coordination with humans, we recruited 8 volunteers (25% female, 75% male, aged between 18 and 35). The volunteers were divided into four groups and given a comprehensive
31

---

Published in Transactions on Machine Learning Research (10/2024)
Table 10: Performance comparisons on three tasks between Concord with and without value decomposition the SMAC map 3m.
Algorithm
Task1
Task2
Task3
Average
Concord
0.98
0.98
0.99
0.9833
Concord w/o Human Critic
0.97
0.97
0.99
0.9766
Table 11: Performance comparisons three tasks between Concord with and without value decomposition the SMAC map 271s.
Algorithm
Taskl
Task2
Task3
Average
Concord
0.91
0.99
0.98
0.96
Concord w/o Human Critic
0.92
0.93
0.97
0.94
Table 12:
Performance comparisons on three tasks between Concord with and without value decomposition
on the SMAC map 4m.
Algorithm
Task1
Task2
Task3
Task4
Task5
Task6
Average
Concord
0.81
0.82
0.91
0.91
0.92
0.99
0.8933
Concord w/o Human Critic
0.75
0.75
0.89
0.89
0.95
0.98
0.8683
introduction to the basic gameplay and experimental procedures. All volunteers were fully informed of their rights, and the experiments were approved by the relevant department. The Overcooked game remotely deployed on server accessible through the volunteers' browsers. Using this sctup, we were able to collect trajectories of human players interacting with each other, as shown in Fig. 21] We collect around 160 buman-huuman trajectories (for a total 64k environment timesteps) and partition the these into two subsets, splitting cach trajectory into two single-agent trajectories. Based on the collected human data, we construct sets of human proxy models through behavior cloning. Then train different methods by using these human proxy models the teammates that appear sequentially. Note that the AT agents obtained by different methods will play with the real human participants, not the human proxy models. In preliminary experiments, we observed that human models trained through behavior cloning outperformed those trained with Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016). Hence, we decided to employ the former throughout our experiments.
D.2 Ethical Statement
In terms of the experiments that has human participation, we claim that we have employed effective practice avoid possible hical issues.
Test introduction Before the experiments, we informed the volunteers of the specific details of the entire experiments, including that they would be arranged to play the game with Al agents, and that their playing data during test and the final test. results would be utilized for the academic research of this work. All the collected data is only for experiment purposes and we only make the testing results public in this paper. The volunteers voluntarily choose whether to participate the experiment, and all participating voluntcers were informed of and agreed to our practice.
Possible Risk for Humans and Broader Impact The main risks for the volunteers are twofold: the potential leakage of their personal information and the time cost. Regarding the former, we only invited the voluntcers to participate in our experiments without the requirement for any unnecessary personal information. Additionally, we will keep the volunteers' identity information completely confidential. As for the latter, we first adjusted the frequency of the AT decision-making in the experiment to make the
32

---

Published in Transactions on Machine Learning Research (10/2024)
Open Asymmetric Advantages. The layout of the two rooms, where all players are initialized in the right room, and players need to coordinate to finish cooking, delivery cooking and delivery at the same time, well as ingredients (tomatoes onions). The example trajectories represent different. Human- AI coordination policies.
Figure 21:
coordination process between humans and AI more comfortable. Besides, we provided appropriate material compensation for the volunteer's participation in the experiments.
Furthermore, the experimental results are encouraging in the sense that we demonstrate Concord is a promis- ing method for dealing with Human-AI coordination in multi-agent systems via continual learning. It is not yet. at the application stage and does not have broader impact. However, this work learns one by one coordination via contimual lcarning, making Concord more practical in rcal-world applications.
D.3 Definition of the Human Preference Metric
Sec. 4.3 of the main paper, we validate our approach to conduct continual coordination with real hu- mans. To provide a more comprehensive comparison of our approach with other baselines, we present the experimental results in two different dimensions. Among them, the main results are in the form of the single coordination performance, which is objective metric. To further consider the humans' subjective feelings, additionally include one metric refer to as "Human preference" in the main paper (Strouse et al. Yu et al. 2023). To calculate the human preference, some human participants have tasks as judges. Note the real human participants in our experiments we recruited have two tasks:
2021
Players. We first collect some trajectories of them, then construct human proxy models based these trajectories. Then we train different methods by using these human proxy models the partners that appear sequentially. When testing, the obtained agents by different methods will play with the real human participants, not the human proxy modes. The final single coordination performance is shown in Fig. 10(a)
Judges. We assign a different participant to each player a judge, which will judge the preferences in the game between trained agents and the corresponding human player.
In terms of the formal definition of this metrie, the human preference for method A method can be caleulated follows. Let N be the total number of human players participating in the experiment, NA be the number of human players who rank A over B, and NB be the number of those who rank B over A. Then, Human preference for method A over method B is computed as N4 Ng The human players will first play with the trained agents by different algorithms. The game replays are recorded for the following judging process, where each algorithm evaluated against with a different algorithm. There is 5 different algorithms (i.c., Oracle, Concord, EWC, CLEAR, and Vanilla), thus there total 10 rounds of evaluations. In cach evaluation, human judges are asked to watch the replays of the two algorithms and vote for the algorithm that is preferred. For example, let N be the number of human judges, A and B are the two methods compared in round, NA 3 be the number of human players who rank A over B, and NB 5 be the number of those who rank B over A. Then, Human preference for method A over method B is computed as -0255. Similarly, human preference for method over method is 0.25. The final human preference is shown in Fig. [0(b)
N
No Ng N
33


Reviews:
[{"review_id": "Q622b9HKbI", "reviewer": "Reviewer_MVea", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths: \n\n- The method proposed is a reasonable architecture for this problem setting. \n\n- The authors include a variety of ablations to disentangle performance of various components (although I think more is required). \n\n- The specific research questions proposed are valuable. \n\n## Weaknesses: \n\n- There is a lot of overlap between the introduction and the related work section, that makes the beginning parts of the paper unnecessarily tedious to read. Indeed, the introduction is very dense with related work. I would recommend shortening the introduction to just be motivation and minimal technical background. It may also be beneficial to move the related work to later in the paper since it is primarily a discussion of related methods and seems to be relatively light on technical prerequisites for the proposed method, but I will leave that organizational choice to the authors. \n\n- The presentation of the method is hard to follow. I realize that the method is relatively complex, but it was hard to follow the various components and the various training phases. I appreciate the figures that the authors have included, but the method is still hard to follow. The authors should consider, at the beginning of Section 4, first including a more substantial overview of the algorithm in terms of generic components, before the details of each of these components is presented. \n\n- Relatedly, the writing is clunky throughout. I would recommend the authors try to improve the flow of the writing all around (and generally simplify word choice). In particular, the authors use the phrase \"in specific\" many times throughout the paper, and I don't think this is common in English. Replace with \"in particular\" or \"specifically\", but better yet probably just remove completely. \n\n- The baselines presented (EWC and CLEAR) seem quite weak. While I am familiar with these methods for continual learning, it is not obvious to me that they are a fair comparison. As far as I understand, Concord has a pre-training phase (or at least retains data from the previous episodic interactions---this is unclear to me), and an evaluation phase where the continual learning elements (the recognizer network and the hypernetwork) are relevant. This, really, corresponds to a form of black-box meta-continual learning or meta-reinforcement learning. It would be more fair to compare to existing methods within this literature. Generally, the authors need to both strengthen baselines and more solidly explain the data availability throughout the phases of training. \n\n- The framing of this work, as being centrally-focused on interaction with humans, feel potentially limited. This method, as far as I can tell, is applicable to decentralized cooperative decision-making settings, whether or not the agents are humans or not. Indeed, fundamentally what the recognition network is doing is some form of intent inference, which is applicable to all decentralized multi-agent settings. \n\n- Answering the question of \"how does the teammate representation mechanism work\" with a t-SNE plot doesn't really sufficiently answer the research question. There are other possible analyses possible---in particular looking at how the outputs of the representation mechanism impact agent behavior, among others---that would strengthen this section. \n\n- The description of the single coordination performance metric is unclear and needs to be explained better. It is phrased as \"after completing training with the t'th teammate\". Doe this correspond to episodes? Is there always guaranteed to only be a single teammate? Is the method extensible to multiple teammates? \n\n- It isn't clear from where the target for the recognizer network is obtained, and the reconstruction loss for the recognition network is confusing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Concord. This method aims to improve human-AI agent collaboration, within a multi-agent RL problem setting. They exploit a continual learning approach, and show better knowledge transfer is achieved. They experiment on Starcraft and Overcooked benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Revise the introduction as described previously. \n- Polish the writing; in particular, improve presentation of the method and the problem setting. \n- Strengthen baselines.", "broader_impact_concerns": "No major broader impact concerns, although the authors may consider adding a discussion on this since it is a paper on human-AI interaction.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kBB8ZwheVM", "reviewer": "Reviewer_Xue1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-The approach uses a simple but clever approach for adapting to new humans by using a modular or compositional structure for the value function with a linear combination of value networks for the human and the AI agent.\n-The paper clearly states some assumptions, such as \"To adopt this teammate representation scheme, we have two necessary prerequisites: 1) ... 2) ... test phase.\" These assumptions or \"pre-requisistes\" are then explained in the next paragraphs.\n-The paper proposes a reasonable anti-forgetting mechanism by continuing to train (regularize) with data collected from previous cooperation tasks.\n-Pseudocode is included. It could be better to use line numbers to guide the reader.\n-The paper states and uses clear, good metrics from prior work.\n-Figure 8 is impressive, but only two sentences exist talking about it. I recommend spending more time to dive into the details here.\n-The paper achieves positive results (primarily shown by Figure 6 and Table 1). \n-The paper does a good job of providing a relatively thorough exploration of the data, visualizing performance in numerous ways.\n-The paper benchmarks across multiple domains (Overcooked, SMAC)\n\nWeaknesses\n-The second pre-requisite (representations should be obtainable with minimal amount of interaction) is fair, but it isn't clear that anything about the approach guarantees or works to \"minimize\" the need for the data. Learning a good encoder should help, but there appears not to be any discussion about the minimal amount of data needed relative to the diversity or distinguishability of player types.\n-There are editing issues. For example: \"artificial intelligence researches\" should be \"artificial intelligence research\"\n-Figure 2 is missing a legend for the colors other than green and blue. What are purple, orange, yellow, etc.?\n-The hyper-network approach requires the generation of the set of parameters for the AI actor. This would seem to limit the size of the network and the sophistication of the AI agent's behaviors. It would be helpful to acknowledge this point and do some kind of sensitivity analysis as to the performance of the agent (or complexity of behavior) as a function of the size of \\theta_i^{A_\\zeta}.\n-The paper could have used numerous other baselines, such as those cited by Zhao et al. (2023), including Fictitious Co-Play (Strouse et al., 2021). FCP is mentioned in this paper, but it isn't clear why that and other papers didn't serve as baselines.\n\nZhao, R., Song, J., Yuan, Y., Hu, H., Gao, Y., Wu, Y., Sun, Z. and Yang, W., 2023, June. Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 5, pp. 6145-6153).\n\nStrouse, D.J., McKee, K., Botvinick, M., Hughes, E. and Everett, R., 2021. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34, pp.14502-14515.\n\n-Figure 9a is a bit difficult to understand the trends as there is a significant amount of variance. Also, the horizontal axis is missing a label. A major selling point of this paper is that it is continual, and there isn't too much analysis of its ability to learn continually. I would recommend extending the x-axis on Figure 9a much further to the left. How would it do after a hundred cooperation tasks?\n\n-Statistical testing should be applied to Figure 6 and Table 1 and even Figure 7 (e.g., a X^2 test)\n\n-The paper describes that the approaches are baselined using \"human-like\" agents; however, details are omitted. The paper would be improved by including precise details and not rely upon references to prior work to fill in most of the details.\n\n-The paper uses an oz-of-wizard evaluation approach with synthetic human agents. This is a clear weakness. It is true that many papers are published without an actual human-subjects evaluation, but many papers are published with these evaluations. It is unclear why such an evaluation was deemed out of scope or unnecessary given the paper's clear motivation. As it stands, the claims of enabling agents to work with humans is not supported by the results.\n\n-A limitations section is missing.\n\n-Submitting code along with the synthetic data used in the experiments with human-like agents would be important.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a system for learning agents that can collaborate with human-like agents such that the system continues to allow new human-like agents to incorporated in a continual or lifelong fashion. The approach is comprised of three basic components: (1) a compositional or mixture-based value function to describe the team's value, (2) a trajectory encoder to distinguish among human-like agents, and (3) a mechanism to prevent catastrophic forgetting for continual learning. All three mechanisms are derivatives of prior work. The collection of results enables a novel system that achieves positive empirical results in a study with human-like agents as opposed to actual human subjects.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the stated weaknesses. In particular, the points about need a human-subject experiment to support the claims, adding statistical testing, adding a limitations section, adding baselines, adding a more exhaustive evaluation of the lifelong/continual learning properties of the algorithm, and adding details about how to create the human-like agents, would be important to address.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GzqiwWEgiQ", "reviewer": "Reviewer_QTFP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Clear writing, easy to understand the main idea\n- Combination of straightforward ideas that led to overall better performance than baselines\n\nWeaknesses\n- Not enough ablation studies and analysis that verify the effectiveness & usefulness of the proposed methodologies\n- Formulating the problem as a continual learning setup is the most important part of this paper, but intuition of \"Human participants, with whom agents are trained to cooperate, typically emerge sequentially\" is not sufficiently supported in the paper. While the effectiveness of the proposed ideas in this setup is supported in the paper, why we should formulate the problem in this specific way is lacking in the paper (e.g., what would happen if you compare this approach to the best available baseline in this benchmark with the same budget? How is the budget critical? Would there be large difference if we assume infinite amount of resources for building teammate models, ... there are so many things to consider to compare the benefits of different setups)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Concord, which is a framework that can be used to solve human-AI coordination problem by formulating the problem as a continual learning setup. The reason for formulating the problem as a continual learning is that previous approaches have to pre-define all teammate models or are unable to adapt to new test setup of unseen teammate is available. For methodology, this paper (i) introduces a teammate representation which is learned end-to-end for updating the actor function, (ii) teammate recognizer that can extract teammate representations from a sequence of transitions from the teammate, (iii) value decomposition of human and AI agents, and (iv) anti-forgetting mechanism. The experiments are conducted on Overcooked and SMAC benchmarks, with both human-like or real human teammates (test-time only) setups, where the proposed approach is validated to be outperforming the baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please provide more experiments with regard to ablation and analysis of the proposed components. There is only one ablation on value decomposition\n- Please provide more experimental results and supports that shows why we should formulate this problem as a continual learning setup\n\nMinor ones\n- It's difficult to parse/understand the exact setup of Section 5.3\n- ` Additionally, we don’t add any constraints to criticism for easier learning.` -- What does criticism mean here?\n- `In particular, we maintain a separate teammate representation for each group of human teammates encountered during the continual learning process.` -- Does this mean that we assume which teammates are of which type? It could be nice to give more explanation here\n- Reference formats in Section 2 (Related Work) is a bit broken; please fix it", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper could be strengthened by including baselines from the existing literature on their target environments, but it is nevertheless above the threshold for publication as is.
Decision: Accept (poster)