Collaboration with Dynamic Open Ad Hoc Team via Team State Modelling
Authors: Jing Sun
Cong Zhang
Zhiguang Cao
Venue: Accepted by TMLR ()

Abstract: Open ad hoc teamwork presents the challenging problem of designing an autonomous agent that can rapidly adapt to collaborate with teammates without prior coordination in an open environment. Existing methods primarily rely on fixed, predefined teammate types, overlooking the fact that teammates may change dynamically. To address this limitation, we propose a novel reinforcement learning approach, the Open Online Teammate Adaptation Framework (Open-OTAF), which enables a controlled agent to collaborate with dynamic teammates in open ad hoc environments. To achieve this, the controlled agent employs a dual teamwork situation inference model to capture the current teamwork state, facilitating decision-making under partial observability. To handle the dynamic nature of teammate types, we first introduce a Chinese Restaurant Process-based model to categorize diverse teammate policies into distinct clusters, improving the efficiency of identifying teamwork situations. Next, to model heterogeneous agent relationships and accommodate a variable number of teammates, we represent the team as a heterogeneous graph and leverage heterogeneous graph attention neural networks to learn the representation of the teamwork situation. Extensive experiments across four challenging multi-agent benchmark tasks—Level-Based Foraging, Wolf-Pack, Cooperative Navigation, and FortAttack—demonstrate that our method successfully enables dynamic teamwork in open ad hoc settings. Open-OTAF outperforms state-of-the-art methods, achieving superior performance with faster convergence.

Published in Transactions on Machine Learning Research (06/2025)
Collaboration with Dynamic Open Ad Hoc Team via Team State Modelling
Jing Sun
brystalsunn@estgg.eddu edw.
Faculty of Data Science City University of Macau
Cong Zhang*
cong.ztang220@@miil.com
College of Computing and Data Science Nanyong Technological University
Zhiguang Cao
School of Computing and Information Systems Singapore Management University
Reviewed on OpenReview: https: openreview.. net/ forum9 BukMU42P3G
Abstract
Open ad hoc teamwork requires autonomous agents to achieve rapid, adaptive collaboration with teammates without prior coordination in open environment. Existing methods pri- marily rely fixed, predefined learmate types, overlooking the fact that leammates may change dynamically. To address this limitation, we propose novel reinforcement learning approach, the Open Online Teammate Adaptation Framework (Open-OTAF), which enables a controlled agent to collaborate with dynamic Leammates in open hoc environments. To achieve this, the controlled agent employs dual teamwork situation inference model to capture the current teamwork state, facilitating decision-making under partial observability. To handle the dynamic nature of teammate types, first introduce Chinese Restaurant Process-based model to categorize diverse teammate policies into distinct clusters, improv- ing the efficiency of identifying teamwork situations. Next, to model heterogeneous agent relationships and accommodate a variable number of teammates, we represent the team as a heterogeneous graph and leverage heterogeneous graph attention neural networks to learn the representation of the tcamwork situation. Extensive experiments across four challenging multi-agent benchmark tasks Level-Based Foraging, Wolf-Pack, Cooperative Navigation, and FortAttack demonstrate that method successfully enables dynamic teamwork in open ad hoc settings. Open-OTAF outperforms state-of-the the--arttt methods, achieving supe- rior performance with faster convergence. These results validate Open-OTAR capacity to adapl to unknown leammates while maintaining computational efficiency.
1 Introduction
Recently, multi-agent reinforcement learning (MARL) has garnered growing interest for addressing complex multi-agent tasks, such as navigation human-shared environments and real-time strategy games (Roesch) et al.) 2020: Nguyen al. 2020 Boldrer al.) 2022 Zhou al., 2021 Zhang nent framework in this domain is centralized training with decentralized execution (CTDE), which enables agents to train with global information in centralized manner while making decisions based only local observations during execution in decentralized manner. Numerous MARL methods have been developed within the CTDE framework, primarily categorized into policy-based and based approaches (Lowe
2018h. A promi-
Corresponding Author.

---

Published in Transactions on Machine Learning Research (06/2025)
Sunchag et al. 2017: Yu et al. 2022 Policy-based methods, such as MAD- Yu et al. 2022], leverage observations and actions of controlled agents to train policy networks. In contrast, value-based methods, exemplified by VDN (Sunchag et al. 2017) and QMIX (Rashid et al. 2018), construct a joint value function to optimize team performance. However, these MARL approaches assume a fixed team configuration, where factors such as team size, formation, and goals remain unchanged, and teammate types are known and static.
et al. 2017 Rashid et al.
2018:
DPG Lowe et al. 2017), COMA Foerster et al. [001 8) and MAPPO
In many roal-world applications, agents must collaborate with unknown and diverse teammates in real-time, giving rise to the ad hoc teamwork (AHT) problem (Stone et al., 2010). Existing research AHT of- ten assumes that all teammates' behaviours are predefined out of types, each corresponding to a specific coordination stralegy (Albrecht & Stone, 2018 Barrett et al.) 2017). However, in dynamic environments, teammates' strategies may evolve umpredictably, leading to severe miscoordination if assumed to be static and fixed (Chen et al. 2020). A common approach to adapt to dynamic teammates is to leverage Bayesian posteriors of teammate types and incorporate them into reinforcement learning when learning policies (Bar- rett et. al., 2017; Ravula 2019 Chen et. al. 2020). These methods require access to leammates' observations and actions to compute the posteriors, which is infeasible in partially observable environments. Recent ef- forts have attempted to address partial obscrvability, but they primarily focus on closed environments where the number of agents remains fixed [Gu et al. 2021 Ribeiro et al. 2022 Zhang et al. 2023 Rahman et al.,
2024 Li et al. 2024 Chen et al. 2025 However, many real-world scenarios require adaptation to a dynam- ically changing number of agents. For example, an autonomous vehicle must adjust its driving behaviour based on the number of surrounding vehicles, which could be driven by humans or produced by different manufacturers, each with distinct driving styles (Barrett & Stone 2015 Albrechtt et al., 2021 Achiev- ing online adaptation with dynamic teammates in open AHT remains open challenge, especially under partial observability where agents must infer teammate behaviors and intentions from limited observations. While communication and message-passing methods (Foerster et al., 2016; Das et al. 2019) have shown promise in improving belief-sharing and coordination, they often introduce additional computational costs, especially when agents cooperate with dynamic teammates (Albrecht & Ramamoorthy 2016 Rahman et al. 2021a). This trade-off between policy adaptability and computational scalability highlights the critical need for efficient collaboration frameworks in open environment.
To address the issues mentioned above, we propose the Open Online Teammate Adaptation Framework (Open-OTAF), a novel reinforcement learning framework that explicitly models the teamwork situations, which indicates the influence on the environmental dynamics caused by other teammates, to enhance coordi- nation in open environments. Concretely, Open-OTAF employs a dual-structured model to learn a fixed-size teamwork situation embedding, capturing both agent-specific and leam-level coordination patterns. To avoid redundant encodings of similar teammates, we leverage a Chinese Restaurant Process (CRP) (Blei & Frazier, 2011) to probabilisticall cluster teammates by behavioral similarity, generating an ad hoc belief over team composition. This clustering reduces the context search space and improves representation efficiency. To address dynamic team compositions and heterogeneous interactions, we represent the team a heteroge- ncous graph and process it through a Heterogencous Graph Attention Network (HAN) to generate role-aware and relationally structured (eamwork situation embeddings. For policy training, we sample represent ilive teammates during centralized training, encoding their identities into distinct contexts to enrich joint policy learning. During decentralized exccution, agents approximate the global context using local obsorvations. This iterative process yields a robust policy capable of adapting unseen leammates and dynamic team configurations. To summarize, we highligh the following contributions:
To the best of our knowledge, this is the first work that considers the online adaptation with varying number of heterogeneous teammates in open ad hoc teamwork;
We propose accommodating the heterogeneity of teammates in an open AHT setting and introduging the heterogeneous graph attention network to learn tcamwork situations facilitate tcamwork effectively.
We present a formal derivation, grounded in variational inference, that makes the local teamwork sit- uation embodding informatively consistent with global context, bolstering the theoretical foundation of our Open-OTAF designs.
2

---

Published in Transactions on Machine Learning Research (06/2025)
We show that the proposed method achieves superior performance on range of challenging multi- agent tasks, including the Tevel Based Foraging (LBF), Wolfpack, Penalized Cooperative Navigation (PCN), and the Fortattack (Rahman et al. 2022). The extensive experiments confirm that Open- OTAF achieves dynamic teamwork for open ad hoc teams and delivers superior performance against slate-of-the-art methods with a faster convergence speed.
The remainder of this paper is organized as follows. Section 2 provides a brief review of the works in MARL, ad hoc teamwork, agent modelling, and graph neural networks. Section 3 introduces the preliminaries of the proposed method. Section elaborates on the detailed designs of the Open-OTAF framework. Section provides the evaluation experiments and ablation studies. Finally, Section b concludes the paper and presents future works.
2 Related work
Cooperative MARL. Recently, the developments in MARL, have led to remarkable progress in creating intelligent agents that can efficiently cooperate to solve complex tasks (Hernandez-Lcal et al., 2019 Zhang et al., 2021; Gronauer & Diepold, 2022). MARL algorithms use RL techniques to train a fleet of agents in multi-agent system. There are two main frameworks of MARL methods: centralized and decentralized learning. Centralized methods (Claus & Boutilier 1998) learn single policy to predict the joint actions of all agents directly, while decentralized learning (Littman 1994) entails each agent independently. The framework of CTDE bridges the gap between the two, which permits information sharing during training, while policies are only consuming the agents' local observations, enabling decentralized execution (Lowe) ct al. 2017). In cooperative MARL, the policy gradient methods are sub-class of CTDE, wherein cach agent consists of receetr actor and a centralized critic, which is jointly optimized with the shared information of the controlled agents (Lowe et al.) 2017 Foerster et al.] 2018). The value decomposition methods represent the joint Q-function as a function of agents' local Q-functions (Sunehag et al., 2017 Rashid
ct al. 2018 Son et al., 2019 Wang et al. 2020). However, these studies presume the team configuration
(e.g.. team size, teammate types, team formation, and goals) stays unchanged.
Ad hoc teamwork. In many real-world settings, agents are required to collaborate in diverse teams without previous joint training, which comes up with ad hoc teamwork (AHT) problems. There are three main assumptions that characterise AHT (Mirsky et al., 2022). The primary assumption is the lack of prior coordination between agents, which means that the learner should be able to cooperate with the team on-the- fly, without the opportunily to rely on previously agreed collaboration strategies. The second assumplion is that the learner lacks control over its teammates. Lastly, teammates are assumed to be collaborative, indicating that all agents in the team have a common goal and can take actions that will benefit the team. However, leammates might have additional objectives that may vary per leammate and even have different rewards. Farly works in ad hoc teamwork operated under the assumption that the teammate's behaviour was known to the learning agent (Stone & Kraus 2010 Agmon & Stone (012)). Other approaches have relaxed this assumption and assumed teammate behaviour to be unknown. For instance, (Barrett et al., 2017) developed a hod that infers teammates' policies from their observed behavior and subsequently incorporates these estimates into the decision decision-making process. Such methods typically utilise the concept of types, which encapsulates the important information determining an agent's 's behaviour. Existing bype- methods assume that the number of teammates is fixed and their behaviours are categorized into several known types (Barrett& & Stone, 2015 Mirsky et al., 2020 Albrecht & Stone. 2019 Chen et al., 2020 Ravula 2019). However, it is quite often that teammates with unknown types may enter and leave the environment without prior notification. The drawback of these approaches is that finite and fixed types might not cover all possible situations in complex environments (Chandrasekaran et al. 2016 Rahman et al. 2021b 2022 Another strong assumption in these works is that the agent could access other teammates' obscrvations and actions, which is unrealistic in partially observable environments. Though some previous arts have directed their attention toward addressing ad hoc teamwork under partial observability, they only considered closed environments in which the number of agents is fixed (Gu al.] 2021 Ribeiro et al.; 2022; /chhferr et al., 2022 Zhang et al. 2023 Rahman et al. 2023). However, quite often that agents may join and leave the team anytime, making learning in such open AHT environments still open research challenge. Existing
3

---

Published in Transactions on Machine Learning Research (06/2025)
approaches neglect the heterogeneity (i.e., the varying types) among teammates and the varying team size. To bridge this gap, we propose novel application of HAN, which simultaneously handles the potential team composition while generating semantically embeddings that capture varying team size and compositions.
Agent modelling. Agent modelling is a research topic that emerged alongside game theory (Brown 1951
With the powerful representation capabilities of recent deep neural networks, the controlled agents can reason about other agents' unknown goals and behaviours (Albrecht & Stone, 2018 He et al. 2016 Albrecht &
Stone 2019). For example, the recursive reasoning method, McLIBA, conditions the ad hoc agent's s policy belief over teammates, which is updated following the Bayesian rule (Zintgraf et al.) 2021). While these works assume that the tcammates choose their actions independently. As result, the Relational Forward Model (RFM) architecture was proposed, which can be used to model tearmatess potential effects on each other [Tacchetti et al.) 2019). However, the aforementioned works commonly assume access to other agents' information. Recent work proposes to use VAE to learn teammate information under partial observability teamwork (Gu et al. 2021). Building upon these developments, we propose an agent modcling framework for open ad hoc leamwork that predicts teammate behaviors by inlegrating joint action-value estimales with action predictions from an agent model. This approach enables optimal action selection despite uncertainty in teammate strategics.
Graph neural network. Graph neural networks (GNNs) are a type of powerful neural architect that can work with graph-structured data (Wu et al. 2020 which have been exploited to learn factored action values to simplify the computation of optimal joint actions using coordination graphs (CGs) (Böhmer et al., 2020; Naderializadeh et al.] 2020). Unlike these methods that use CGs model joint action values for fully cooperative setups, some works them in open systems to model the impact of other agents' actions towards the learning agent's 's returns (Rahman et al. 2021b Jiang et al., 2019) to handle dynamic input sizes. These works have overlooked the different relationships and importance among the agents introduced by the open environment. We first introduce the heterogeneous graph to compute the ultimate teamwork situation embedding to facilitate docision-making by considcring the potential effect other agents. Together with previous works, we show the great potential of HAN, which could inspire further research.
3 Problem formulation
We aim to train single autonomous agent, the ad hoc agent, that can co- operate with various teammates with- out pre-coordination in partially ob- servable open environment. The prob- lem can be formulated by extending the Tramework of partially observable MDP (POMDP) (Lowe et al. (2177) an Open POMDP, which is defined by tuple M, A,T,T, R.R,R,), 7). Specifically, M denotes the dynamic set of agents in- volved in the task. S is a set of states describing the possible configuration of agents and the external environment, where the number and type of agents vary. Each agent has its own observation space O+ O with a corresponding observation function O + O* where Ok E O. The agent types can be determined by their policies, which belong to a global policy set r.
Sample
Partial observation of
Unknow teammates
Controlled agent
Shared reward
R
Teammate set I
Environment
Figure 1: Visualization of the Dec-POMDP with teammate set.
Figure shows the detailed schematics of this problem. We denote the policy of the ad hoc agent as x and the joint policy of all other agents as -- Each agent k selects an action a E 4 from its own action space A E A, giving rise to a joint action [a',... ... .a"]= E A x A2 x ... x A with n being the variable number of agent. The state transits by following a transition function transition, agent l receives a new observation and obtains a scalar reward according to its reward function b R S * R. The initial state is determined by some prior distribution p : S [0,
x A x .... A + S. After each
4

---

Published in Transactions on Machine Learning Research (06/2025)
The ad hoc agent's optimal policy x² is defined by maximizing the joint action value Q (s, a"), which indicates the expected accumulative team reward over different ad hoc teamwork:
+00
(o, a a)= E ai ixx aoitooo ~
we 100 = 8, uo - a, Pl
(1)
1=U
(s, a @ (s,d',a =
"), VT s,""""""
(2)
Traditional MARL often optimizes team utility directly via Q" (s, aa",, but this fails to attribute individ- ual contributions ad hoc teams (Gu et al. 2021). Marginal Utility is defined to measure the contribution of an ad hoc agent to the whole team utility [Genter et al. 2011). It represents the increase (or decrease) in a team's utility when an ad hoc agent is added to the team. Given teammates' actions a=, the marginal util- (s.a', "e-.1Q"(ss,a's
ity can be calculated as u' (s, a a=²) ")= E,--1Q" action (no-op or current policy output). The relationship between the marginal utility and the team utility (denoted by the joint action value) can be calculated as: arg max,: u'(ss, aa arg maxg: Q (s, as,a', a a)), where u² (s, a², a denotes the marginal utility when the ad hoc agent chooses the action a' under the state s. The agent chooses the action that maximizes the marginal utility to ensure the maximal team utility,
-((=) Suull a a"),, where au is a bascline
To enable adaptive perision-making the ad hoc agent's policy must respond to teammate behaviors. We address the complexity of unknown leammates by encoding their interactions into a teamwork situation a compact latent representation of the team's strategic state. This representation captures how teammates collectively influence environmental dynamics, allowing the agent to optimize its bohavior for the current cooperative context. For example, in our truck-drone delivery system, the teamwork situation embedding encodes the state of task phases, interaction patterns and on. To model the concept clearly, we formally define the teamwork situation as:
Definition 1. (Teamwork situation) At each time step t, the ad hoc agent is in the teamwork situation C; E C, which is the current underlying teamwork state yielded by the encironment state St and other teammates' actions - It reflects the high-level semantics about the teammates' behaviours.
Though different teammates generate diverse state-action trajectories, they may lead to similar teamwork situations at certain moments, and the ad hoc agent's action would affect their transitions (Gu et al., 2021 By accurately identifying the current. teamwork situation, the ad hoc agent can choose the aclion accordingly to ensure online adaptation. In this work, the dynamic teammate mainly refers to the different combinations and numbers of teammates. At the beginning of each episode, the tcammates' types are randomly sampled and kept fixed in that episode. However, the leamnate combination and number are different across episodes. In addition, the set of behaviour types used for sampling is different for training and evaluation.
4 Methodology
4.1 The overall framework
Open-OTAAL aims to estimate the ad hoc agent's marginal utility for choosing corresponding actions to maximize the team utility. To enable adaptation to unknown teammates, we model the marginal utility as a conditional function on the inferred latent variable, which implicitly represents the current teamwork situation. Open-OTAF jointly optimizes the marginal utility function and the latent variable by two lcarning objectives in an end-to-end fashion. Figure 2 depicts the overall framework of our Open-OTAF algorithm. It splits the team into two parts: teammates and the ad hoc agent.
We regard other teammates as part of the environmental dynamics perceived by the ad hoc agent. Different combinations of teammates lead to diverse and complex dynamics. We expect to learn leamwork situations to describe the core information of teammates' behaviours implicitly. As similar teammates might possess similaritics in their identifications, learning a specific context for each tcammate but ignoring the relationships among them could lead to trivial encodings. We thus assign them to different clusters via the Chinese Restaurant Process (CRP) to shrink the context. search space (Section 4.2 and Figure a))- Then, we
5

---

Published in Transactions on Machine Learning Research (06/2025)
Q Loss
Global situation inference model
Cluster Loss
Integrating network G
C
Decoder
random
84,
CRP-based teammate cluster model
choose
<'(r)))
(ng)
Agent model
Local teamwork situation inference model
model
Marginal utility network
T
T
Controlled agent
Figure 2: Overall framework of Open-OTAF. In training phase, we introduce a global teamwork situation inference model to obtain C₁ Then, an integrating network G and a teamwork situation decoder are jointly proposed to regularize the information embedded in the learned variable. For controlled agent, we introduce local teamwork situation inference model to infer proxy representation Zi of Ci from local observation.
introduce a global teamwork situation inference model that takes the controlled agent's state, the actions of its teammates, and the estimation of the team's composition as input. This inference model processes these inputs to infer and output the current teamwork situation C; enabling a more informed decision-making process. However, the size of the obtained embedding changeable due to the variable team size. To handle the dynamic team size and explicitly model the complex relationships among agents, we introduce HAN effective function approximation model to obtain the fixed-length embedding Cj which be optimized by the loss function (Section 4.3 and Figure 3(c))
For the controlled agent, we expect to condition its policy on the current teamwork situation Cj. As the partial observability impedes the direct access to C₁, we introduce local teamwork situation inference model to infer a proxy representation Z; of C₁ from local observation. We force Z; to be informationally consistent with C by an information-based loss function (MT loss) to maximize the conditional mutual information. To select optimal actions from the joint action value model, we propose action-value computation method which implements the joint action value model fully connected Coordination Graphs (CG) with action predictions learned using agent model. Then, train marginal utility network to estimate the ad hoc agent's 's conditional marginal utility (Section 4.4) Similar to the CTDE scenario, we relax the partial observability in the training phase. Open-OTAF is granted access to the global state and other teammates' actions during training. For the controlled coordinating policy training, we sample representative teammates coordinate with by capturing their identifications into distinguishing contexts augment. the joint policy during the contralized training phase. Each agent then utilizes local information to approximate the global context information. The mentioned processes proceed alternately, and can finally obtain robust policy to adapt to any teammates gradually during the decentralized execution phase. We show the meaning of the notations in Table A.1
4.2 CRP-based teammate cluster model
In this section, introduce the CRP-based teammate cluster model in detail. When identifying the team- work situation, it is irrational and inefficient to treat each new teammate novel type ignoring their similarities. Accordingly, we aim to acquire clearly distinguishable boundaries of tcammates' behaviours by applying a behaviour detection module to assign teammates with similar behaviours to the same cluster.
As shown in Figure aa) we have a batch of training teammates with the corresponding trajectories [(r'), fr2)....
(7)) 1 .... sampled from the interaction between the 4th tcammate and the environment. The trajectory set k is a set of trajectories the 4th teammate and the environment, and T denotes the time horizon. Given the high dinensionality
(So, ag..... .T) sampled from the interactions between

---

Published in Transactions on Machine Learning Research (06/2025)
MfAl Clustering
Node-level ngsregation
Seraaiie-
aggregation
plot) p(r, (
Meta-path
Node-level
C
CRP
Decoder
Lntent varibale
attention network
warged mode
Semuntic-leve attention netwock
Deco
inference network
Eneader
Team
Toum up with
Node-level attention network
City
Tourerates
Controlled agent
Meta-pohh
Mode
CRP-based teammate clustering
Latent variable Inference
(c) Teamwork situation embeddinng
Figure 3: Global teamwork situation inference model: we first design a teammate cluster model to acquire clearly distinguishable boundaries of teammates' behaviours and formulate the distribution of dynamic teamn mates. Then, we introduce a variational auto-encoder to get the leamwork situation by taking the global state S and teammate actions a- as input. The current teamwork situation can be encoded by variable-size latent variables e¹. Then, introduce the HAN to learn fixed-size embedding of the teamwork situation.
of the trajectories, we use a trajectory encoder Ew, to extract fcatures from the trajectory т and represent it as Bu, (T); where wi is the parameter of the trajectory encoder E. Concretely, partition the trajectory into Ts (so, $1."" ST-1, ST) and (ao. .r--)). For the t teammates generate so far, o² = E [Bwn (T*)] will be used to represent its behavioural type.
If M clusters are instantiated so far, the cluster that the htth teammate belongs to will be inferred by P(r (7*) P(ut,1 TT), E {1,... .M, ",, The ut is variable indicating whether the h teammate belongs to the m cluster based on its representation vt, Based on the Bayesian rules (Blei & Frazier 2011).
the posterior distribution ( = can be decomposed as P(rMI= 1::,",T T x P(c*) P T 7.)) The prior P(v))) ) is given by CRP, representing the probability of the wtth tcammate belonging to m cluster without the leammale strategy information. M clusters are formed so far, the prior P(w = =nm/(k - + X) when m S M, else P = A/(k- 1 A) when m M + 1, where Nn denotes the number of teammate belonging to the m cluster, ΣM Amm = k and X > 0 is a concentration hyperparameter that controls the probability of the formation of a new cluster.
To estimate the conditional probability P(T ) we use a parametric network Dwy that take 7h ut input to estimate the likelihood of taking action Th i..., P(r=lum, Dw, (r Ir um) Then wh can be update by wh = The Fm to when m I M, else uh = R if m +1, where om is the mcan value of the m 41
cluster. Combining the estimated prior distribution and predictive likelihood, we are able to decide which cluster the tth teammate belongs to and their corresponding probability, that is m+ = arg max, P(o (T)). If m* M, we will assign the kth teammate the mhth cluster. Then, cluser center Um. will be updated by Umx = Em. tv. when m* S M, else üM+1 = x* if m = M + 1. To force the learned representation v to capture the behavioural information of each teammate and estimate the predictive likelihood more precisely, the E,, and Dwy can be optimized by:
#m=+1
Lo(w) E U. DE log( wz (Ta Ew, (r)),
(3)
where K is the number of teammates and w the weight of the encoder and decoder networks.
4.3 Learning to represent teamwork situation
After clustering leammates into behaviorally distinct groups, we train a robust policy capable of adapting to dynamic team compositions by conditioning the controllable agent's actions on inferred teammate behaviors. Despite the diversity and complexity that unknown tcammates' behavior exhibits, the CRP formalized before helps acquire clearly distinguishable boundaries based on tcammates' behavioral types with regard high- level semantics. Furthermore, motivated by the ODTIS (Gu et. al. 2021), which aims to guide the context

---

Published in Transactions on Machine Learning Research (06/2025)
encoder to identify the adaptive behaviors, we expect to utilize a global context encoder to embed the teammates' information into a compact but descriptive representation.
As shown in figure b,, we encode the teamwork situation in a stochastic embedding space via a latent variable inference network, which takes trajectory 7."h = (s : ....................... ... ,{") and their cluster information (k. pl as input, which emerges from the interactions between paired joint, policy (H, R*) and the environment, and - is the joint policy of uncontrollable teammates belonging to the m" cluster. For clarity, we omit time step t in all formulas below. To solve this issue, the current teamwork situation can be encoded by latent probabilistic variables et drawn from multivariate Gaussian distribution N Mi (et). To enable the dependency revealed in the dcfinition, we use a trainable neural network f to learn the parameters of the Gaussian distribution of c' by (Me,, parameters of f. Note that the size of c' corresponds to the mumber of teammates, which can change in the open ad hoc teamwork. Thus, we introduce the heterogeneous attention graph neural network, using the inherent ability of the graph neural network to transform the dynamic-size teammate's information into fixed-size output. In heterogeneous graphs, mela-paths are predefined composite relations that encode high-order semantic relationships between nodes through sequences of edge types. The HAN leverages these meta-paths in two aspects: (1) it computes relationship-speeifee attention weights for cach meta-path, enabling the model to dynamically prioritize relevant interactions, and (2) it aggregates node embeddings along each meta-path, producing a final representation as a weighted combination of these semantically distinct pathways. This dual-process architecture enables HAN to model complex relational patterns while preserving interpretability through teta--ppthh-ttitttee attention distributions.
s(7";, {k, p(u [= Or), ~ (Met, oet), where Of are the
We denote the heterogeneous graph H (X,E), where X (X,Vii E [1.....n]} is a set of nodes representing the team agents, and & is the set of symmetric edges between every node pair, We use HAN with various types of edges to model open ad hoc teamwork. The reason is that the agent number is variable and the relationships among agents are different in open ad hoc team- work. As shown in Figure 4 the controlled agents may not construct a cooperation relationship with the leammates planning to leave or just entering the enviroument, and the influence of teammates on the controlled agent may be different. We model the interaction relationship between the con- trolled agent and different teammates via different meta-paths, i..,, + {cooperate, no cooperate). As shown in Figure B(c), each meta-path has its own attention network and learns the attention weights ag for the neighbor- ing agents of the corresponding relationship. For instance, the "cooperate" meta-path only takes the node embedding of the noighbor- ing agents that cooperate with the controlled agent input. Thus, different meta-paths have diverse semantic information. The net- work is implemented as two-layer MLP with: e = MLP(WW, [cf [ef]]e"]), where Wa, is learnable weight matrix for meta-path 2p. ef and é are node embedding and || denotes concalenation. After that, we normalize them if to get the weight. coefficient 0% softmar, (e%,). Then, the learned embedding of node i for the meta-path 4p can be calculated as C²2, =[EENN aij .d), where Nā. (p E {1,2)) denotes the meta-path p based neighbors of node i, and g denote the activation function. Then, the final teamwork situation embedding is aggregated by all semantic embeddings, where the semantic level attention network is used to learn the weight Bb, of different meta-paths can be calculated by 36, exp(MLP(CS,, ))/ eaP )).
Contolledd agent
Teammates
Cooperate (Meta-path 1)
Leave
cooperate (Meta-path 2)
Enter
Figure 4: Heterogeneous graph representation of open ad hoc teamwork.
We can fuse these semmntic-specific embeddings to obtain the final embedding as C E 36 Cb,, Our model can get the optimal combination of neighbors and multiple meta-paths in hierarchical manner, which enables C to better capture the complex structure and rich semantic information.
8

---

Published in Transactions on Machine Learning Research (06/2025)
We then use marginal utility network to estimate the ad hoc agent's marginal utility as a'(r', C*) u'(s,a', a-)), representing its incremental contribution to the team's expected return, conditioned on the current tearwwrk context. This quantifies how much a' improves team performance, given teammates' behaviors encoded in C After that, a loss function (Q loss), an integrating network G and a tcamwork situation decoder g (as shown in Figure 2 are jointly proposed to regularize the information embedded in the learned variable Ct. The integrating network generates the joint action value's estimation as G(w,, C*) ~ Q C¹, the increase of the ad hoc agent's 's marginal utility results in the improved joint action value. We adopt modified asynchronous Q-learning loss function (Q-loss) (Mnih et al. 2016) as the optimization objective:
(s,a', a', a ")), where G maps the ad hoc agent's utility wf into the value estimation. Given any
Lq = -ug +o max G(ui41, C/-1) - G(u,, Ci)]
(4)
where G is periodically updated target network. The expectation is estimated with uniform samples from the replay buffer DD
4.4 Learning marginal utility function under partial observability
For ad hoc agents, we aim to model the marginal utility as a conditional function on the inferred team- work situations. However, partial observability impedes the agent's access to C* encoded from the global state-action trajectory. Thus, we introduce a local inference model f to derive proxy representations of the teamwork situation et from local observations B; (of, rt-, -,, We assume that B; can partly reflect the current teamwork situation since the transition implicitly indicates the underlying dynamics, which is primarily influenced by other teammates' behaviors. We denote the estimation of et as 2 Similar to c', we encode z' into a stochastic embedding space as (Uz), = (((B:O,,),= ~ "N(u,,, Oy- are parameters of f*. Then, with the predicted teammate type information, we obtain the final team- work situation embedding Z' via the HAN. To make Z' informatively consistent with C" we introduce an information-based loss function LMr to maximize the conditional mutual information 1(2'; CB) between Zt and C². Due to the difficulty and feasibility of estimating the mutual information, variational dis- tribution (Z B²) used to approximate the conditional distribution pe pg(2"C" IC", B*). Inspired by the information bottlencek (Alemi et al. 2016), we derive tractable lower bound for the MI as follows.
where
Theorem 4.1. Let I(Z;; B*) be the conditional mutual information between the local teamwork siluation embedding Z of agent i and global embedding Cs.. Then, the lower bound is given by:
[Zz'Ci/B) 2 Exios, [lo (<(Z'(C", ,B')
(5)
p(7:
The lower bound can be rewritten as a loss function: Cas Dx where D is the replay buffer.
Notice that the entropy of our labels is independent of our optimization procedure and can be ignored. The proxy encoder is conditioned on the transition data. Given the transitions, the distribution of the proxy representations p( ) is independent of the local histories. Thus, we have:
I('','i)B") 2 -Ecs (CETp((CCC BZ (C B')]dZ'
In practice, we sample data from the replay buffer, and the lower bound can be rewritten as loss function to be minimized by:
LMi Rz (Px.. 1 CZ" B']]]
To achieve a robust policy, we use the inferred Z to estimate the agent's marginal utility yu(Z), .a). Consider- ing the inffuence of other teammates on the controlled agent, we adopt the coordination graphs (CGs) (Rah- man et al. 2021b) to estimate the ad hoc agent's marginal utility. Given the embedding Z the agent
9

---

Published in Transactions on Machine Learning Research (06/2025)
modelling module computes the likelihood of the agent's actions p'a²|Z"). Then, the marginal utility of agent i can be calculated as:
a('',a')= Q'aa + ((Q'(a Q ((M)(')))pp(aa(z'))
JEN.jii
(7)
Ca a ( Z") Z')p(a'
J.K.NsJA
Q' ('(a21 Z") represents j's contribution towards the learner's returns by executing a², while Qi at IZ) denotes and k's contribution towards the learner's returns by jointly choosing a' and a². We imple- ment (a Z') and Q** (a², Iz') as multilayer perceptrons (MLPs) parameterized by f and ó to enable generalization ion across different numbers of the agents. Afterward, the marginal utility can be used to ap- proximate the joint action-value function. Therefore, the overall optimization objective can thus be derived as Lio = L + CMI. During the training phase, the controlled agent interacts with different teammates collect Iransition data into the replay bulfer. Then, samples from D are fed into the framework for updating all parameters induced by the overall loss. During execution, the controlled agent conditions its behaviour the inferred teamwork situations by choosing actions to maximize the utility function. We sunmarize our training procedure in Algorithm in the Appendix A.1
5 Experiment
To thoroughly assess the Open-OTAF algorithm, we evaluate our model in four widely-used partial observ- able multi-agent environments for various settings, including the Level-Based Foraging (LBF), Wolfpack, the Penalized Cooperative Navigation (PCN), and the Fortattack (Rahman et al. 2022). From the experiments, we aim to answer the following questions: 1) Can Open-OTAF yield superior performance than SOTA basc- line methods (Figure 5)? 2): Are the main components, i.e., the CRP, MI, and AM, necessary and effective 3) Does Open-OTAF have generalization capacity with the varying teammale size and type (Figure A.6 and Table ? 4): How do the key hyperparameters, cluster numbers, affect the final
(Figure 6 Figure A.5
?
results (Table A.4
5.1 Experiment Setup and baselines
MAVIC
MUSICC VILIR
VLLER. Vsero
We select four multi-agent tasks our enviromments, as shown in Figure A.2 in the Appendix A.2.1. Among them, Level Based Foraging (LBF), Wollpack and Fortattack are three scenarios com- ing from (Rahman al.) 2021b).. LBF is cooperative grid world game with agents that are rewarded if they con- currently navigate to the food and col- lect it. In Wolfpack, multiple agents (predators) need to chase and encounter the adversary agent (prey) win the game. The FortAttack environment de- fines bounded two-ddimensional space where agents are constrained within spe- cific coordinate ranges. We also evaluate our method in the penalized cooperative navigation, coming from the MPE envi- ronment (Lowe al. 2017), where mul- tiple agents are trained to move towards landmarks while avoiding collisions with
Open Cite
500550452000
3te0ch.220)
(a) LBF.
(b) Wolfpack.
ALZOPC
DETO
Avuragg 14
Steps2e2200
(c) PCN.
(d) Fortattack.
Figure 5: Performance comparison across various scenarios. For LBF, Wolfpack, PCN and Fortattack, the training steps are 640, 0000 and we evaluate all the methods with 100 test episodes after every 3200 training steps.
10

---

Published in Transactions on Machine Learning Research (06/2025)
each other. We simulate open environments by creating open process that determines how agents enter and leave during episodes, for both training and testing. In the LBF, Wolfpack, and PCN environments, the number of timesteps an agent can exist in the environment is determined by uniformly sampling from a certain range of integers. In FortAttack, agents are removed once they are shot by opponents. After being removed, agents can reenter the environment after specific period of waiting time. The type of an agent entering an environment is uniformly sampled from all available types. More details about the environments and algorithm configuration are provided in the Appendix A.2.1 and A.3
To illustrate the advancement of our method, compare the proposed method with two type- based base- lines, GPL and VAE-GPI, which can be used to solve open ad hoc teamwork under full and partial observ- ability separately. Although GPL assumes full observability of the state, we apply GPL in our experiments by treating the learner's observations as input. ODITS (Gu et al.) 2021) applies a centralized "teamwork sit- uation encoder" for end-to-end learning to adapt to arbitrary teammates closed environments. As ODITS considers only single ad hoc agent in the closed environment, we further extend the ODTIS to open envi- ronments by rntrodgg graph neural network. Lastly, to demonstrate the advantages of HAN, also where GNNs are introduced to produce the representations of
compare the algorithm teamwork situations. Table A.2 (in Appendix A.2.4) provides summary of the different algorithms.
5.2
Experiment results
5.2.1 Analysis of the average return
We first compare the overall performance of Open-OTAF with baselines. We report the performance in terms of average returns (solid line) and the standard deviation (shaded areas) with random seeds, as summarized in Figureßand Table From the curves, we observe that the returns Open-OTAF consistently outperform the other four baselines with a faster convergence speed in all benchmark games while possessing lower variance, which basically shows the effectiveness and high efficiency of the proposed method. In particular, the comparison with Open- ODen-OTAF(GNT) suggests the effectiveness of using HAN to produce representations of the teamwork situation. The utilization of HAN can boost the performance of open ad hoc tcamwork than conventional GNN. Otherwise, the comparison between Open-OTAF and ODTIS shows that using the coordination graphs and agent model for optimal marginal utility estimation can boost performance as well. enables our mothod to estimate the marginal utility by considering the effect of other agents, which leads significantly better training. Also, ODITS shows superior performance than GPL and VAE- GPL, manifesting the necessity of utilizing global states to improve training efficiency. Furthermore, GPT, performs poorly than other methods because the sequence of observations perceived by the learner contains the least useful information compared to other methods. These results show that our method can design an autonomous agent capable of robust leamwork under dynamie teammates without pre-coordination.
5.2.2 Ablation studies
To determine the importance of each component of Open-OTAF, conduct ablation study for the different components to see how it affects the train- ing, and the results over 5 random seeds are shown in Figure 6 We take the LBF as an example. More ablation studies are provided the Appendix A First, to illustrate the importance of the CRP-based teammate cluster, we derive the method of w/o [ CRP by removing the CRP process and taking each new teammate a new cluster. Furthermore, we pick w/o MI to investigate how maximizing mu- tual information between global and local informa- tion accelerates learning efficiency. Finally, w/o AM is introduced illustrate the impact of other team-
3.5
Open DTAI
3.0
wo.
Average retum
2.0
0.5
0
250
500
750
1k 1.25
1.5k
1.75
2
Steps (x3200)
Figure 6: Ablations for different components.
We the w/o place of without conciseness.
11

---

Published in Transactions on Machine Learning Research (06/2025)
mates in the calculation of the marginal utility. We only utilize network to calculate the marginal utility without considering the effect of other agents.
In Figure the comparison results show that w/o MI suffers the most severe performance degradation in all games, which demonstrates the necessity of utilizing global information to facilitate the learning of the local teamwork situation inference model. Furthermore, w/o CRP achieves lower performance than ApAN-ATAF, which manifests that the CRP-based teammate model helps the controlled agent adapts to dynamic teammates rapidly. We also find agent modelling helps learn more accurate marginal utility by considering the effect of other teammates, which can bring about a slight coordination improvement. It can also demonstrate that the cluster model can help to learn robust ad hoc agent. The testing results are also provided in Figure A.55 where we provide the average return (bars) and the standard deviation (error bars) over different seeds. Experimental results demonstrate that our CRP-based teammate classification model, when augmented with global information during training, enables agents to infer significantly more accurate teamwork situation representations compared to bascline methods. This improvement stems from two key mechanisms, the first one is that. the CRP's ability to probabilistically cluster leammates by behavioral similarity, which reduces uncertainty in type estimation, and the integration of global state information during centralized training, which provides necessary context for disanbiguating observationally similar but strategically distinct teammate behaviors.
5.2.3
The generalization capacity
To assess the generalization ability of Open-OTAF, we performed experiments involving different teammate numbers and types. First, we evaluate the generalization ability by changing the teammate types during testing. Similar to the training procedure of Open-OTAF, we control one agent and test it with teammates randomly sampled from a pool consisting of 10 different policies, which were not seen during training. The generalization performance of Open-OTAF and the baselines is provided in Table [ We show the average and 95% confidence bounds during testing utilizing 5 seeds. The data gathered by averaging the returns at the checkpoint which achieved the highest average performance during training. We highlight in bold the algorithm with the highest average returns. The results show that our method significantly outperforms other baselines during testing. Unsurprisingly, methods that achieve high returns during training, will also achieve better performance when generalizing to different leammate lypes. Open-OTAF's performance in this diverse teammate-type setting was significant, outclassing the baseline performances by 5.1% 49.8%.
Table 1: Results of generalization capacity under different teammate types.
Env
GPD. VAE GPLODDTS OpennOOTFF()Openn-OTT
LBF
0.73(0.62)
(.85
2.14(..47)
(.17
(.39
(0.25)
Wolf
(1.5))
(6.9(.
(9.3(1.39)
21.((148)
(1.3(1.42)
(1.2(1.26)
(..1
(.4)
(.
(.69
(.77) 0.86
Fortattack
(.12(0.26)
(.44().22)
(.49)
0.58 0.54
0.62(
Moreover, to assess Open-OTAF's generalization ability further, we conduct an expanded experiment involv- ing different teammate sizes. Open-OTAF was initially trained on 5 teammates with 10 different training policies and then tested on 3, 5, 6, and 10 teammates with 10 different testing teammate policies. Results, summarised in Figure A.6 (in Appendix A.4). indicate consistently superior performance of Open-OTAF. Experimental results show that, with desirable generalization to various teammate sizes, our method out- performs the toate-oo-thheartt method. In summary, when encountering unknown teammates, the CRP-based module first classifies them into of the 10 clusters (we set the maximum of cluster number 10). Then, the attention network uses the clustering results as the semantic information to generate weights for better teamwork situation estimation. The leamwork situation is the input of the controlled agent's policy and affects its action selection. We admit that we may not train a best-response policy for unknown teammates. It is also impossible to train a best-response policy for every possible unknown tcammate. However, the trained clustering ability of the CRP-based module and the generalization capability of the attention network allows the controlled agent to select. reasonably good actions. The ablation studies, as shown in Figures A.6.
12

---

Published in Transactions on Machine Learning Research (06/2025)
and Table demonstrate promising outcomes. It can be noted that the CRP-based teammate cluster module and the heterogeneous attention graph neural network (HAN) can improve generalization performance.
6 Conclusion
This work proposes novel online adaptation reinforcement learning algorithm address the challenging open ad hoc teamwork problem. In Open-OTAF, the CRP-based leammale cluster model aims to predict the probability distributions of the teammate's types, which can be considered as the agent's beliefs about the current status of its team. The inferred teammate types with environment state are fed into representation learning model to encode the teamwork situation representations as latent probabilistic variables. With the assistance of the heterogeneous attention graph, we manage to infer the teamwork situation and adapt to the new teammate effectively. We also propose an information- based proxy encoder to infer the learned variables from local observations overcome partial obscrvability. With the guide of the global tcamwork situation, the ad hoc agent adapts to new leammates' behaviours dynamically and quickly by conditioning its policy the inferred variables. Extensive experiments demonstrate that Open-OTAF not only obtains higher average return but also achieves a faster convergence speed. Our method can be seen as a primary attempt for online teammate adaptation with dynamic teammates in open ad-hoc teamwork, we sincerely hope it can be a solid foothold for applying RL to practical applications. For future work, we will relax the CTDE assumption to enhance the applicability of our approach, investigating a robust ad hoc agent within fully decentralized framework, and validating the method in weal-world applications.
References
Noa Agmon and Peter Stone. Leading ad hoc agents in joint action settings with multiple teammates. In AAMAS, pp. 341-348, 2012.
Stefano V Albrecht and Subramanian Ramamoorthy. A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems. arXiv preprint arXiv:1506.00 01170, 2015.
Stefano V. Albrecht and Subramanian Ramamoorthy. A game-theoretie model and best-response learning method for ad hoe coordination in multiagent systems. Autonomous Agents and Multi-Agent Systems, 30 7077779-778, 2016. doi: 10.1007/s10458--015-9000
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-95, 00666655,220 2018.
Stefano Albrecht and Peter Stone. Reasoning about hypothetical agent behaviours and their parameters. arXiv preprint 1rXi::1906.11000 11064, 2019.
Stefano V Albrecht, Cillian Brewitt, John Wilhelm, Balint Gyevnar, Francisco Eiras, Mihai Dobre, and Subramanian Ramamoorthy. Interpretable gosdd prediction and planning for autonomous driving. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1043-1049. IEEE, 2021.
Alexander Alemi, lan Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottle- nock. arXi preprint arXin: 161.: 004 10, 2016.
Samuel Barrett and Peter Stone. Cooperating with unknown teammates in complex domains: A robot soccer study of ad hoc teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Samuel Barrett, Peter Stone, and Sarit Kraus. Empirical evaluation of ad hoc teamwork in the pursuit domain. In The 10th International Conference on Autonomous Agents and Multiagent Systems- Voluue 2, pp. 567 574, 2011.
Samuel Barrett, Avi Rosenfeld, Sarit Kraus, and Peter Stone. Making friends on the fly: Cooperating with new teammates. Artificial Intelligence, 242:132-177 171, 2017.
13

---

Published in Transactions on Machine Learning Research (06/2025)
David M Blei and Peter Frazier. Distance dependent chinese restaurant processes. Journal of Machine Learning Research, 12(8), 2011.
Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In International Con- ference on Machine Learning, pp. 980-99 991. PMLR, 2020.
Manuel Boldrer, Alessandro Antonucci, Paolo Bevilacqua, Luigi Palopoli, and Daniele Fontanelli. Multi-agent navigation in human-shared environments: A safc and socially-aware approach. Robotics and Autonomous Systems, 149:103979, 2022.
George W Brown. Iterative solution of games by fictitions play. Activity analysis of production and allocation, 13(1):374-376, 1951.
Muthukumaran Chandrasekarau, Adam Eck, Prashant Doshi, and Leenkiat Soh. Individual planning in open and typed agent systems. Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, pp. 82 91, 2016.
Feng Chen, Xinwei Chen, Rongiun Qin, Cong Guan, Lei Yuan, Zongzhang Zhang, and Yang Yu. Efficient multi-agent cooperation learning through team-mate lookahead. Journal of Transactions on Machine Learning Research, 2025.
Shuo Chen, Ewa Andrejezuk, Zhiguang Cao, and Jie Zhang. Aateam: Achieving the ad hoc teamwork by employing the attention mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7095-7102, 2020.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. AA ATTT/AAT, (9977746-552) 1998.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pincau. TarMAC: Targeted multi-agent communication. In Proceedings of the 36th International Confer- ence Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp. 1538 1546. PMLR, 2019.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Coun- terfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Jakob N. Foerster, Yannis M. Assael Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 29, pp.
2016.
Katie Genter, Noa Agmon, and Peter Stone. Role- based ad hoc teamwork. In Proceedings of the AAAI Conference Artificial Intelligence, volume 25, pp. 1782 1783, 2011.
Prem Gopalan, Jake M. Hofman, and David M. Blei. Bayesian nonparametric nonhomogeneous pois- son processes with applications to us gun violence. NewrIPS, pp. 402 410, 2014. URL https: pdf
7/proceedings. wwwaaiinnnnn.ureee /ttp://///iii..ooppppp/////aaapp
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a survey. Arlificial Intelligence
Review, pp. 1-49, 2022.
Pengjie Gu, Mengehen Zhao, Jianye Hao, and Bo An. Online ad hoc teamwork under partial observability. In International Conference on Learning Representations, 2021.
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforcement learning. In International conference on machine learning, pp. 1804 1813. PMLR, 2016.
Pablo Hernandez- Bilal Kartal, and Matthew Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Mulli-Agend Systems, 33(6):750-799) 2019.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. In International Conference on Learning Represenlations, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster); 2015.
Yang Li, Shao Zhang, Jichen Sun, Wenhao Zhang, Yali Du, Ying Wen, Xinbing Wang, and Wei Pan. Tackling coopcrative incompatibility for zero-shot human-ai coordination. Journal of Artificial Intelligence Research, 80:1139 1185, 2024.
Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learnöng proceedings 1994, pp. 157-163. Elsevier, 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAT Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. Advances in newral information processing systems, 30, 2017.
Reuth Mirsky, William Macke, Andy Wang, Harel Yedidsion, and Peter Stone. A penny for your thoughts: The value of communication in ad hoc teamwork. Good Systems- Published Research, 2020.
Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V Albrecht. A survey of ad hoc teamwork research. In Mulli-Agent Systems: 19th European Conference, EUMAS 2022, Düsseldorf, Germany, September 14-16, 2022, Proceedings, pp. 275- 293. Springer, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nalure, 518 518((540):529 ---3, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Inter- national conference on machine learning, pp. 1928 1937. PMLR, 2016.
Navid Naderializadeh, Fan H Hung, Sean Soleyman, and Deepak Khosla. Graph convolutional value decom- position in multi-agent reinforcement learning. arXiv preprint arXix:2010.04 744004 2020.
Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE transactions on cybernetics, 50(9): 6826-3839, 2020.
Arrasy Rahman, Ignacio Carlucho, Niklas Höpner, and Stefano V Albrecht. A general learning framework for open ad hoc leamwork using graph-based policy learning. arXiv preprint ar arXiv:2210.05,4 0..4448, 2022.
Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and Stefano V Albrecht. Generating teammates for training robust ad hoc teamwork agents via best-response diversity. Transactions on Machine Learning Research, 2023.
Mohammad A. Rahman, Dany Fagan, Alessandro Sordoni, and Neil Rabinowitz. Learning to share in multi-agent reinforcement learning. In Proceedings of the 35th A AAT Conference on Artificial Intelligence (AAAI), volume 35, pp. 11472 11480. AAAI Press, 2021a. URL https://ojs aaai www///nnee./...oooo..o..ee..e.ee php/ AAAT/article/vien/IIe
Muhammad Rahman, Jiaxun Cui, and Peter Stone. Minimum coverage sets for training robust ad hoc teamwork agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17523- 17530, 2024.
Muhammad A Rahman, Niklas Hopner, Filippos Christianos, and Stefano V Albrocht. Towards open ad hoc teamwork using graph-based policy learning. In International Conference on Machine Learning, pp. 8776 8786. PMLR, 2021b.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. Tn International Conference on Machine Learning, pp. 4295 4304. PMLR, 2018.
Manish Chandra Reddy Ravula. Ad-hoe leamwoork with behavior-swilchiing agents. PhD thesis, 2019.
João G Ribeiro, Cassandro Martinho, Alberto Sardinha, and Francisco S Melo. Assisting unknown teammates in unknown tasks: Ad hoc teamwork under partial observability. arXiv preprint orXie:2201.08533 2022.
Martin Roesch, Christian Linder, Roland Zimmermann, Andreas Rudolf, Andrea Hohmann, and Gunther Reinhart. Smart grid for industry using multi-agent reinforcement learning, Applied Sciences, 10(19),6900, 2020.
Lukas Schäfer, Filippos Christianos, Amos Storkey, and Stefano Albrecht. Learning task embeddings for leamwork adaptation in multi-agent reinforcement learning. arXiv preprinl ar Xiv:2200 02249, 2022.
Kyunghwan Son, Dacwoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference machine learning, pp. 5887-5896. PMI.R 2019.
Peter Stone and Sarit Krans. To teach or not to teach?: decision making under uncertainty in ad hoc teams. In AAMAS, pp. 117-124. Citeseer, 2010.
Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. Ad hoc autonomous agent teams: Col- laboration without pre-coordimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp. 1504-1509, 2010.
Peler Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jader- berg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arxi 001.7766059990 2017.
A Tacchetti, H Francis Song, PAM Mediano, V Zambaldi, J Kramár, NC Rabinowitz, T Graepel, M Botvinick, and PW Battaglia. Relational forward models for multi-agent learning. In 7th Interna- tional Conference on Learning Representations, ICLR 2019. ICLR, 2019.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning, arXie preprint Nk::2008.0106: 2020.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph noural networks. IEEE transactions on newral networks and learning systems, 32(1):4-24, 24, 2020.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances Neural Information Processing Systems, 15:24411-24624, 2022.
Jilic Zhang, Zhanshan Wang, and Hongwei Zhang. Data-based optimal control of multiagent systems: A reinforcement learning design approach. IEEE transactions cybernetics, 49(12):4441-444 2018.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of thcories and algorithms. Handbook of Reinforcement Learning and Control, pp. 321-384, 2021.
Ziqian Zhang, Lei Yuan, Lihe Li, Ke Xue, Chengxing Jia, Cong Guan, Chao Qian, and Yang Yu. Fast teammate adaptation in the presence of sudden policy change. orXiv preprint ar Xxxx:2305.05911 2023.
Weigui Jair Zhou, Budhitama Subagdja, Ah-Hwee Tan, and Darren Wee-Sze Ong. Hierarchical control of multi-agent reinforcement learning team in real-time strategy (rts) games. Expert Systems with Applica- lions, 186:115700 2021.
Luisa Zintgraf, Sam Devlin, Kamil Ciosck, Shimon Whiteson, and Katja Hofmann. Deep interactive bayesian reinforeement learning via meta-learning. In Proceedings of the 20th International Conference on Au- tonomous Agents and MulliAgent Systems, pp. 1712 1714, 2021.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Table A.1: Notations of algorithm
Notation
Description
Notation
Description
Concentration parameter in CRP cluster model.
G
Integrating network.
O
Observation space.
T
Transition function.
State space.
Eun
Trajectory encoder.
A
Action space.
Dw2
Trajectory decoder. Mean value of the mt cluster. Set of nodes.
Policy of the controlled agent.
Om
m
Joint policy of all other agents. Action of controlled agent. Joint action of other agents. Local observation of agent i. Marginal utility of agent i. Batch of local transition data.
X
a
E
Set of edges.
The global policy set. Weight for node pair i, Weight for meta-path. + Set of Meta-path. Time horizon
o'
u'
Bap
Bi
M
Number of clusters
T
Trajectory which sampled from the interaction between 4d teammate and environment.
The learned embedding represents the behavioural type of 7 Variable of the 8th teammate belongs to the m cluster based on its representation et.
mm
number of leammates belonging the m² cluster. Teamwork situation inferred by global information. Importance of meta-path based node pair i,j- Teamwork situation inferred by local information.
To
Ci
Cop
Zi
M
Dynamic of agents involved in the task.
A Appendix
A.1 Details about Derivation
Details about CRP and derivation of cluster assignment Chinese restaurant process (CRP) Blei &
2011 is discrete-time stochastic process that defines prior distribution over the cluster structures. It can described simply as follows: as cach customer enters restaurant with an infinite number of tables, he chooses to sit down alone at new table with probability proportional to a concentration parameter sits with other customers with probability proportional to the number of customers sitting the occupied table. Customers sitting at the same table will be assigned to the same cluster. Motivated by the Gopalan ct al. 22014), the cluster assignment of the hhth teammate P(wiiizh -h) can be decomposed as:
Frazier
Pl =d)))
Plu Tb Th)
P(T 1T73,70 T2)
P(T Tal rkluth P((
(8)
P(T3 rb)
P( lut T P( le
P( k
P(T
x P( T, P(T P(u
As Tá is set of states that is not determined by the behavioural type of the teammates if neglecting the correlation in time dimensionality, |( is constant. Accordingly, we can obtain that P(u 1rk + x P
Variational Bound of teammates context approximation To make context vector Z' generated by local trajectory encoder informatively consistent with global context C¹, we propose to maximize the mutual information between Z' and C conditioned on the agent i's local trajectory B¹. We draw the idea from variational inference Alemi et al. 2016) and derive lower bound of this mutual information term.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Theorem A.1. Let (B)) be the conditional mutual information between the locol teamwork situation embedding Z of agent i and global embedding C Then, the lower bound is given by:
1("',1B²) 2 E2 log (z(Z' B¹)
(9)
p(Z) Bc)
The lower bound can be rewritten loss function:
LMI
(Z B')ll.
(10)
where D is the replay buffer.
Proof.
1 = B2 log ( B¹)
(Z BB)
Ezi,oi [og
p (Bt)
(11)
Ec B [DrLp( Z' (Z B\)]
2 E2i,c. log (Z* B³)
BB)
where the last inequality holds because of the non-tegatyvity of the KL divergence. To prevent degenerate solutions, we employ the information bottleneck regularization, that is we add a small penalty H(Z* to maximize the entropy of Z', preventing deterministic collapse. Then it follows that:
E2,00BB log (E(2'(C B³) B()
Ezico qe (Z' |C" - (ogg((
(12)
(or,or,::logee CZ + Ex) B¹)]
= Ec [f plZ B (o)g B dzz + +H(Z'B)
To illustrate the flow of Open-OTAF, show the training procodure in Algorithm D A set of teammates can be generated via heuristic and RL algorithms, and store the small batch of trajectories into replay bulfer D. The encoder and decoder are trained to force the learned representation to precisely capture the behavioral information and precisely estimate the predictive probability (Line 6). Afterward, the CRP prior and predictive likelihood are calculated to determine the assignment the teammates m (Linc 7-9). Then, we update the existing cluster or instantiate a new cluster based on the assignment (Line 11-14). In the training, we first sample a teammate from the cluster and it this episode. The teammates pair with the controllable age they make decisions together. To train the agent policy networks and the context encoders, the moving average values of context vectors are updated, and the optimization objectives are calculated (Tine 24-29).
A.2 Details about Environment setup and Baselines
A.2.1 Environment setup
Setup for LBF: In LBF, the learner retrieves objects that are positioned in an x 8 grid world, as shown in Figure A.2 In the Open Dec-POMDP setting, one agent is controllable and will stay the environment for the whole episode. The maximum number of teammates is 5, where the teammates can enter and leave the environment. freely and their policy will change well. Fach agent, including the learner and their
18

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm 1 Training Procedure
Require: Batch training teammates' behavioural policies {ny ")== learning rate a; concentration param A, initial clustering
number M = 1.
1: Initialize the replay buffer
2: while env not done do Generate data
3: for k ..,..., K do
4: Sample data D = {(s,, aj, rre)))) using o' and x
5:
Sample small batch of trajectories 7f.
6:
Update Eus Duz by minimizing Lc. Extract the feature of the trajectory 7 by Eun (r").
7:
8:
Calculate the CRP prior probability P(u Calculate the predictive likelihood P(r ( o
9:
10:
arg max,. P(u P(r.) ( if m= M then
11:
Assign the kt teammate the m² cluster.
13:
Update the cluster center Vin Pm Update the number cluster m: Amm = mm + 1.
14:
15:
else if
16:
thenInitialize the (M + 1)th cluster with the hthh teammate. Initialize the cluster center 5M+1 = =
17:
18:
Update M = M + and BM4+1 =
19:
end if
20:
& k+1
21:
end for
22:
Add {K, Pl }. D into D.
23:
for steps in training steps do
24:
Sample trajectory D D
for
T do
Compute (n.g oc) = f(er, a) and sample ci-
27:
Compute (1.7i102)) = (B() and sample zt.
28:
Compute C₁2 23 using HAN.
29:
Computer LQ. Lui and Lc.
30:
Update G and global model by minimizing Lo-
31:
Update local model by minimizing Catt-
32:
end for
33:
end for
34:
end while
leammates, is assigned numerical level, as are the objects themselves. The level of agents and the foods are randomly chosen from {1,2,3] 3} at the beginming of an episode. In the partially observable version, agents can only observe entitics in a 5 x 5 grid surrounding them, including the positions and levels of all the entitios. Any agent has six actions, i.e., [up, down, left, right, load, no-op), where the first four actions mean the agent moves towards the corresponding direction, load means loading food next to it, and a no-op action indicates doing- nothing during an entire episode. The agents' objective is collect all the objects and the collection is only successful if the sum of the involved agents' levels is equal to greater than the item level. Every agent that collects object is given reward equal to the level of the object. An episode terminates if all available objects are collected or after 50 timesteps.
Setup for Wolfpack: In Figure A.2 b), the multiple hunters attempt to catch randomly prey in a grid world of size a 10 x 10, where the "catching" means the prey in the cardinal direction of at least two hunters. Partial obscrvability is induced by limiting the learner's observation entities within a Manhattan distance of 3 from itself. The observation of each agent is the coordinates of its location, its ID, and the coordinates of the prey & if observed. Agents in this environment have five actions, i.e. [up, down, left, right, no-op]. Every hunter in pack that captured prey is given reward of two times the size of the capturing pack. We penalize agents by -0.5 for positioning themselves next to prey without teammates positioned in other adjacent grids from the prey.
Sctup for Penalized Cooperative Navigation (PCN): PCN, multiple players must cooperate through physical actions to reach a set of landmarks as shown in Figure A.2 The agents need to learn to infer the landmark they must cover, and move there while avoiding other agents. The action space of each agent
19

---

Published in Transactions on Machine Learning Research (06/2025)
(a) Level-based foraging.
(b) Wolfpack.
agent
agent 2
agent 3
Penalized cooperative navigation.
(d) Fortattack.
Figure A.2: State visualization of benchmark environments.
contains five discrete movement actions, i.e. [up, down, left, right, no-op]. Their objective is to simulaneouslyy cover two destination grids to get a reward of 1. However, a penalty of -0.2 is imposed on the learner if it reaches a destination without the presence teammates simultancously covering the other destination. Similar LBF, the learner can only see the destination grids or teammates they are inside x region surrounding the learner.
The Fort environment defines bounded two-dimensional space where agents are constrained within specific coordinate ranges. In Figure A.2 (d), the horizontal axis is restricted to values between -0.8 and 0.8, while the vertical axis spans from - to 1. The fort is modeled as semicircle with a center at (0.0, 1.0) and radius of 0.3. During initialization, the al Lackesss are positioned with their vertical coordinates set between -1 and -0.8, ensuring they begin at a distance from the fort. This positioning allows the attackers to start from a location far enough to stratogize their approach. Converscly, the defender's initial position constrained to the vertical coordinate range between 0.8 and 1, placing them closer to the fort, thus providing advantageous starting position for defense.
Setup for Fortattack:
20

---

Published in Transactions on Machine Learning Research (06/2025)
A.2.2 Teammate Types
In open ad hoc teamwork, the primary challenge lies in the fact that the controlled agent lacks prior knowledge of its prospective teammate to interact with. Hence, we randomly choose multiple teammate types to contemplate all potentialities in simulating an open ad hoc teamwork environment where an agent may need to cooperate with unfamiliar counterparts. Therefore, in all environments, each type of tcammate policy is implemented either via different heuristics or reinforcement learning- based policies. We vary the leammate's policies in terms of their efficiency in executing a task and their roles in team. To create a diverse set of teammates for open ad hoc teamwork, we used the following mixture of heuristics along with RT-bbaed models to control tcammates during the training process. Our method aims to Iearn teamwork situations to cooperate with previously unknown leammates' behaviours. To investigate the policy transfer ability and generalization of the proposed methods, we evaluate the method by comparing the performance under different tcammate types.
For each environment, we design 20 different. policies, and we have one of 10 policies as the training set and the other 10 policies as the testing set. Specifically, in the Wolfpack environment, we used the 9 heuristic policies proposed by (Barrett et al. 2011) along with teammates. Similar to Wolfpack, we also create a diverse set of teammate types for LBF which requires agents to adapt their policies towards their teammates to achieve optimal performance. With LBF, we use 8 heuristics (Albrecht Stone, 2019 Albrechtt & Ramamoorthy 2015), an A2C algorithm (Mnih et al. 2013) and DQN (Mnih et al.] 2016) as our teammate policies. During the testing, for Wolfpack and LBF, we use 10 RL-based policies, which were implemented by different MARL algorithms to develop several teams of agents. To ensure diversity, the corresponding models at 2 different checkpoints. On the other hand, for PCN, we utilize 5 different MARL algorithms, to develop 15 different policies showing distinct policy representations from all developed models. In the training, we adopt 5 heuristics (Rahman et al. 2022) and random sample 5 different policies to control the teammates. The other 10 teammate policies as the testing set. In Fortattack environments, the policies governing each teammate type are implemented using either 10 distinct behavioral heuristics or MARL-based approaches.
A2C algorithm (Mnih et al. 2015) to control
use different random sceds for cach RL-based algorithm and save
A.2.3 Environment Openness
We create open process that determines how agents enter and leave during episodes. The number of timesteps an agent can exist within the enviromment is determined by sampling from certain range of integers. After staying for the predetermined number of timesteps, agents are removed from the environment. After that, agents can reenter the environment after a specific period of waiting time. The type of an agent entering an environment is uniformly sampled from all available lypes.
In all environments, a teammate only exists in the environment for a certain number of timesteps. If a tcammate has surpassed its designated temporal lifespan, it is immediately removed from the environment. A teammate that has been removed is allocated waiting period, which is the duration before it is pushed into reentry queue. Given a non-empty reentry quene, agents within the queue are reintroduced to the environment if the number of agents does not exceed the aforementioned upper limit. It is important to note that the reentry queue is randomized, thus inducing an alcatory team composition during learning. The maximum number of players for training is 3 and the maximum number of players for training is 5. For Wolfpack, tcammates' lifetime is sampled uniformly between 25 and 35 timesteps while the waiting period is sampled uniformly between 15 and 25 timesteps. By contrast, in the LBF, PCN environment, the teammates" lifetime is sampled uniformly between 15 to 25 timesteps while the waiting period is sampled uniformly between 10 and 20 timesteps.
Unlike LBF and Wolfpack, where agent counts remain fixed, FortAttack dynamically adjusts the number of agents based on in-game actions: an agent is removed only when shot by an opponent. The respawn delay is determined by the distance to the shooter at the time of climination, following lincar ciaroolation-ggenn shot the closest possible distance (minimum separation) incur an 80-timestep penalty, while those hit at longer distances experience proportionally shorter delays. This design ensures that tactical positioning
21

---

Published in Transactions on Machine Learning Research (06/2025)
Table A.2: The comparison between different algorithms.
Method
GNN VAE AM
MI CRP
HAN
MADDPG
GPL,
VAE-GPI.
ODTIS
Open-OTAF(GNN))
Open-OTAF
Table A.3: Hyperparameters in the experiments
Notation
Meaning
T
640,0000
1
0.99
Optimizer
Adam
Batch size
128
Learning rate
le 4
Evaluation interval
3200
Action selector start
e greedy
1
finish
0.05
Replay memory size
5000
influences respawn timing, adding strategic depth. At episode initialization, agents are evenly distributed between attacking and defending teams, with counts set to the predefined maximum.
A.2.4 Baselines
MADDPG is benchmark of MARL algorithm for closed environments. We employ MAPPDG to control the ad hoc agents, using the learner's observations as input to the policy. GPL and VAE-GPL are two type-based methods designed for addressing open ad hoc teamwork problems. Specifically, GPL is suited for scenarios with full obscrvability, while VAE-GPL is applicable in scttings with partial observability. Although GPL assurnes full observability of the state, we apply GPL in our experiments by treating the learner's observations as input. ODITS improves zero-shot coordination performance in an end-to-end fashion. Two variational encoders are adopted to improve the coordination capability, As ODITS considers only a single ad hoc agent in the closed environment, we further extend the ODTIS open environments by introducing a graph neural network. Moreover, to demonst.ate the advantages of HAN, also include comparison with the Open- OTAF algorithm, where GNNs employed to produce the representations of teamwork situations.
A.3 Algorithm configuration
For the LBF, wolfpack, PCN and Fortattack, the training steps T are 640, 0000 and evaluate all the methods with 100 test episodes after every 3200 training steps. The parameters of the networks updated by Adam optimizer (Kingma & Ba. 2015) with learning rate le - 4 for all environments. The discounting factor is 7 = 0.99, learning rate and the concentration hyperparameter is À = 1. Since they induce the best performance compared with other values. For exploration, we use e greedy from 1.0 to 0.05. Batches of 128 episodes are sampled from the replay buffer, and all components in the framework are trained togother in an end-to-end fashion. More details of the are carried out in machine with Intel Core 10-10X40X CPU and single Nvidia GeForce 2080Ti GPU.
are provided in Table A.3 All experiments
22

---

Published in Transactions on Machine Learning Research (06/2025)
In the CRP-based teammate cluster model, we introduce encoder Eus to model the tearmate trajectory into a latent space as their behaviour type, where Fu, is implemented by a long short-term memory (LSTM) network which takes as imputs and outputs 16-dimensional behavioural embeddings v. To predict the likelihood of the action, we use an RNN-based decoder Dwn, that consists of a GRU cell whose hidden dimension 16, takes of and v as input and reconstrucis the action at. we set the maximum of cluster number as 10 in all environments.
In the global teamwork situation inference model, the latent variable inference network is implemented as 2-layer MLP and LSTM which receives the global trajectory as input. More details provided in Figure
A.2 It subsequently produces the mcan and covariance matrix for the variational parametric distribution. Assuming that he and ez are the hidden and cell states of the LSTM at timestep 4, the LSTM updates the
vectors following:
Hot = MLP (e)) od MPa
(13)
where 6g, h4 = LSTMa (-1,, and H₂ = [st, [s,,a,,, {ki, [mx)) The latent variable o vector will be sampled from the distribution C ~ (Nee )..). Then we use the inherent ability of graph neural network to transform the dynamic-size team's information into a fixed-size output. As for the local teamwork situation inference model, it receives the ad hoc agent's preprocessed observation as input and outputs 24.
To capture the heterogeneity among teammates and handle dynamic team sizes, model the team heterogencous graph and introduce a HAN to compute the final teamwork situation embedding. The global teamwork situation embedding cf and predicted teammate cluster information will be concatenated as the node input of the HAN to calculate the fixed embedding C₁- To estimate the joint value function, we first map C into the parameters of G by a hyper-network. Then, maps the ad hoc agent's 's utility uy into the value estimation. This alternative design changes the procodure of information integration.
For the ad hoc agent, we utilize coordination graphs and agent modelling to calculate the marginal utility by considering the effect of other agents as shown in equation 7 (in the main paper). More details are provided in Figure A.4 The single utility of Q' (a(a/ (a/ and pairwise utility as multilayer perceptions (MLPs) parameterised by 3 and ā respectively. Concretely, given Zi, the agent modelling module computes the likelihood of observed agents' actions g by multilayer perceptrons (MLPs), The agent marginal utility can be calculated by equation 7 After that, the marginal utility ja will be fed into the mixing network to calculate the joint action function finally. To maximize the mutual information between local and global teamwork situations embedding conditioned on the agent i's local trajectory, variational distribution network, which is similar to the global inference network, is used to approximate the conditional distribution.
are implemented
A.4 Experiment Results
To determine the importance of each component of Open-OTAF, we also conducted an ablation study of the different components under different scenarios. The comparison results under Wolfpack, PCN and Fortattack environment are provided in Figure A.4 The results show that the w/o MI variant exhibits the most significant, performance degradation across all games, highlighting the importance of incorporating global information to support the learning of the local teamwork situation inference model. Additionally, w/o CRP performs worse than Open-ATAF, indicating that the CRP-based teammate model enables the controlled agent to adapt more effectively to dynamic teammates. Moreover, we observe that agent modeling contributes to learning more accurate marginal utilities by accounting for the influence of other teammates, leading to modest improvements in coordination.
The testing results are also provided in Figure A.5 where provide the average return (bars) and the standard deviation (error bars) over different seeds. The results show that w/o CRP and w/o MI suffer the most severe performance degradation in all environments when compared with Open-OTAF. It shows that the introduction of the CRP model for teammate types classification and the guidance of global information help agents to estimate more accurate teamwork situations.
The cluster number m may influence the algorithm's performance. We conduct an ablation study for different values of cluster number under different scenarios to further reveal the influence of the cluster number during
23

---

Published in Transactions on Machine Learning Research (06/2025)
ReLU
LSTM, 100 units
FC. |A] units
FC, K units
FC, 100 units
FC, 60 units
FC, 60 units
ReLU
ReLU
ReLU
FC, 100 units
FC, 70 units
FC, 70 units
Input
Input
Input
(a) LSTMa
(b) MLP3
(c) MLP;
FC, 70 units
FC, 30 units
FC, |A| units
ReL.U
ReI.U
ReI.U
FC, 30 units
FC, 70 units
FC, 20 units
Input
Input
Input
(d) Edge processing (H)N
(e) Node processing (H)N)
(1) MLP, and MLP,
Figure A.2: Architecture details of teamwork situation inference model: (a) the architecture used in teamwork situation inference network; (b) singular utility computation; (c) pairwise utility computation; (d) edge embedding computation in the agent and auxiliary agent model; (e) node embedding computation in the agent and teammates; (f) the MLP used by the agent and teammates process the resulting GNN node embeddings. FC denotes a fully connected layer, LSTM denotes an LSTM layer, and the accompanying number denotes the size of the layer. Labels on the arrows indicate the linelinear functions used between the layers while no labels indicate no non-lincar functions being applicd to the resulting output vectors.
training. We vary the cluster number to 5, 8, 10, 12, and 15, respectively, while fixing other configurations. Here, we present the evaluation results of our method under different scenarios (Table A.4). We can see that the performance of m 2 8 is consistently better than that of m = 5. This observation aligns with the intuition that well-separated clusters enhance the inference of teamwork situations by more effectively capturing similarities among teammates, thereby improving overall performance. Experimental results confirm that performance increases with the number of clusters. However, it is noteworthy that the improvement exhibits only marginal gains when the cluster count exceeds 10, suggesting a saturation effect in the benefits of further granularity. To balance compulational overhead and performance, we set 10 throughout the experiments.
Table A.4: The Evaluation Results for our methods on various cluster numbers m.
Eny
m=
m=8
m=10
m=12
m=15
LBF
2.56
2.62
2.77
2.78
2.77
Wolf
21.56
26.98
27.19
27.21
27.26
PCN
0.81
0.83
0.91
0.93
0.95
Fortattack
0.57
0.59
0.68
0.69
0.69
24

---

Published in Transactions on Machine Learning Research (06/2025)
Que Ci)
Parameterizing
FC, units
ReLUU
Weight and Biases
FC, 100 units
Absolutization
FC, 100 units
C
(a) Integrating Network
p(a.) 22)
p(aî) z2)
Qaat
p(allz;)
(aa²)
a²
eia²)
Q(a²,a")
aaat.a")
MLP,
Q(a*)
CG Model
Zin
L2
Agent model
z
(b) Marginal Utility Network
Figure A.4: Architecture details of interacting network and marginal utility network.
16
Openortyl
Opevo
Open-osu Pc.4
22
Avenge retum
Aveergge returr
6.4
10
0.2
250
520
750
1.25
1.5k
1.70
250
750
1/
1.25
1.5
1.55
3k
350
5:0
750
1.5
3.75
Stops (x3200)
Stops (X3200)
Stopo0063200)
(a) Ablation study in Wolfpack.
(b) Ablation study of PCN.
(c) Ablation study of Fortattack.
Figure A.4: Ablations for different components under Wolfpack, PCN and Fortattack environment.
Intuitively speaking, adapting with dynamic teammates with local information seems more complicated and may need more data. To investigate whether the HAN, CRP, and AM can increase the computation complex- ity in our method, we provide the computational cost between different algorithms. We gather the number of floating-point operations (FLOPS) of single inference and the number of paramcters for each method. A.5 the computation complexity of different algorithms. It shows that our method has a compa- rable model volume and computational complexity baselines. This similarity arises because all methods
25

---

Published in Transactions on Machine Learning Research (06/2025)
1.50
14
mean
1.25
Test return mean
12
1.00
8-
0.75
6-
4-
0.25
2
0.00
WO_CRP
WO_MI
Open-OTAF
wo_M
Open-OTAF
(a) LBF.
(b) Wolfpack.
1.0
0.8
Test return mean
2.0
Test return
3.5
0.4
1.-
0.0
Wo, WR_CRP
wo_MI MI
Wo_AM
Open-OTAF
Wo, WO_CRP
wo_MI MI
wo_AM
Open-OTAF
(c) PCN.
(d) Fortattack.
Figure A.5: Testing performance comparison across various scenarios for different components.
WA
Open 1200010010000
VXC
Teat peturm menn
Test retur mean
Teammate size
Teammate size
Teaminaee size
(a) LBF.
(b) Wolfpack.
(c) PCN.
Figure A.6: Generalization study with varying teammate size.
share the same CTDE framework, whose primary bottleneck is centralized training. CTDE's cost stems from processing high-dimensional joint observation-action spaces and scales poorly with team size (curse of dimensionality). It requires agents to learn coordination strategies while simulating teammates' behaviors, alongside iterative gradient synchronization and policy updates. While CTDE enhances collaboration, these demands slow training, especially in partially observable environments.
26

---

Published in Transactions on Machine Learning Research (06/2025)
Table A.5: Computation cost under different methods
LBF PM..PS(M Params (M)) (M MM. (Mrans)))))))))))))
Wolf
Fortattack
PCN
Methods
MADDPG
7.98
0.100
8.09
0.102
8.03
0.101
7.89
0.102
GPL
8.01
0.102
8.11
0.104
8.07
0.104
7.96
0.106
ODTIS
7.99
0.101
8.10
0.102
8.05
0.103
7.95
0.105
Open-OTAF
8.02
0.104
8.13
0.105
8.08
0.106
7.98
0.107
27


Reviews:
[{"review_id": "aRex1Cr29t", "reviewer": "Reviewer_nBpW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Apologies for the delay in submitting this review. Before accepting to review this paper, I had requested the AE for additional time to review based on my other commitments. I only accepted to review this paper after the additional time was granted by the AE. I assure the authors and the AE that I have written my review independently without looking at the other submitted reviews. \n\nStrengths: \n\n1. The paper studies a very challenging and timely problem. \n\n2. Experiments appear to be very comprehensive. \n\n3. The final insights that were presented in the second paragraph of Page 12 (paragraph just before Conclusion) are very clear and comprehensive. But, I feel that this is coming too late in the paper. This summary of Open-OTAF, its expected strengths, and limitations, should have been discussed earlier in the paper to develop a clear intuition. \n\nWeaknesses: \n\nThere are several clarity issues in the text that needs to be addressed. I was confused about important details in the paper, even after taking multiple passes through it. I am listing my major concerns here: \n\n1. It is not clear if the paper is restricted to environments with a shared team reward or not. The description in Section 3 mentions that each agent has its own reward function (reward function being parameterized by the agent index), while the Figure 1 mentions that the reward is shared. \n\n2. The term Marginal Utility is very confusing. This function has not been formally defined anywhere and is suddenly used at the end of Page 4 by using a relation with the Q function. A formal definition is required for this term, seeing how it is central to the paper. \n\n3. Similarly, the term \"teamwork situation\" is very confusing to me. As example illustrating this is required. Also, the associated assumption in Page 4, \"we assume that they may lead to similar teamwork situations at certain moments\", needs justification in the paper. \n\n4. There are several design choices introduced in Section 4 that are quite hard to follow. A clear intuition for each of these design choices along with descriptions of what they are expected to achieve needs to be presented. I particularly found the cluster model descriptions lacking in clarity. Firstly, how is it assumed that trajectories of past behaviours are a good proxy for current agent strategies (since clustering is done based on these trajectories). I would assume that past behaviours, especially ones that are several time steps into the past, are quite irrelevant to the current strategies of agents. Secondly, if the environment is partially observable, how are these trajectories for all agents obtained in the first place. I would assume that it is quite common to have situations where agents move in and out of view, and their action histories cannot be accurately observed by another representative agent. Thirdly, how are the number of clusters determined and misclassification errors handled. The experiments use 10 clusters, but this seems like an arbitrary choice. In the experiments, a study on the effect of varying the number of clusters needs to be performed. \n\n5. Several results in the experiments are close and a statistical significance test is required. For example, in Table 2, the performance of Open-OTAF seems quite close to the performance of VAE_GPL. \n\n6. The citations in the entire text need to be fixed. Citations are used as nouns in sentences where they are not supposed to be nouns (see \\citet and \\citep commands in latex and use them appropriately). \n\n\nMinor Comments: \n \nPg 2: briefly review -> brief review \n\nPg 6: Then, we can then calculate -> We can then calculate \n\nPg 11: controlled agent adapts to -> controlled agent adapt to", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper contributes Open Online Teammate Adaptation Framework (Open-OTAF), which is designed for online adaptation in open ad-hoc teams (environments with unknown teammates), where teammates can have dynamically evolving numbers and strategies. The framework is presented for fully cooperative games where all agents are assumed to share a common goal. The agents in Open-OTAF aim to compute a \"teammate situation\" (an embedding) using Graph Attention Networks. This embedding forms the basis for learning the best responses for each agent. The framework uses the centralized training and decentralized execution (CTDE) paradigm, where the centralized training uses global context with sampled representative teammates and the decentralized execution simply relies on the local observations.  The functionality and superiority of Open-OTAF is empirically demonstrated on multiple MARL test beds.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see \"Weaknesses\" above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "I8N2U5QR9z", "reviewer": "Reviewer_B9hn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper tackles open ad hoc teamwork with dynamic teammates, which is a more realistic setting than prior fixed teammate cooperation tasks.\n2. The overall algorithmic framework is reasonably designed, with each module serving a specific purpose, and the effectiveness is supported by ablation studies.\n3. The experiments across the four environments demonstrate consistently strong performance.\n\n**Weaknesses**\n\n1. The experimental environments are quite simplistic, confined to 2D worlds and discrete action spaces. There are concerns about whether the proposed algorithm can be effectively extended to more complex interactions.\n2. The proposed components of the algorithm, such as CRP Clustering and HAN, introduce noticeable computational overhead, which may pose challenges for larger-scale experiments.\n3. It is unclear how the maximum number of clusters influences the experimental results. From my understanding, the CRP Clustering is performed based on the rollout trajectories of teammate agents. However, during the course of interaction, do teammate policies remain fixed, or are they updated? Even if the teammates follow fixed policies, their behaviors may still change in response to updates in the controlled agent’s policy. Would this not lead to a proliferation of possible teammate types, potentially making clustering more difficult?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Open-OTAF, a novel MARL framework that enables adaptation to dynamic teammates in open ad hoc environments by combining CRP-based teammate clustering, heterogeneous graph attention networks, and mutual information-guided local-global teamwork situation inference.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pbaVU1n6ol", "reviewer": "Reviewer_pDn3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* **Flexible**: Can handle dynamic team sizes and allows for the discovery of novel behavior types during training. \n* **Comprehensive Evaluation & Ablation studies that further isolate the contribution of each component**\n\nWeaknesses: \n1. The combination of CRP, VAE, HAN, CG, and MI loss makes the **framework conceptually dense**. The practical feasibility of deploying such a system at scale should be discussed in more detail. The paper lacks discussion on training stability, sensitivity to hyperparameters, or engineering challenges. \n2. **Limited Discussion on Scalability and Efficiency**: There is no analysis of training time, runtime cost, or memory footprint, especially compared to simpler baselines. Given the added architectural complexity, such discussion would be important for real-world adoption. \n3. **Sparse Justification for Design Choices**: While the paper ablates some components, it doesn't completely offer qualitative insights into why certain design choices and the readers are left to figure it out for themselves.\n\nQuestions:\n\n1. **Section 4.2 – Cluster Mean Update:** The paper defines the candidate cluster embedding as $v_k^m = \\frac{n_m \\bar{v}_m}{n_m + 1}$, but this expression appears to omit the embedding of the current teammate $v_k$. Shouldn't the correct update be $v_k^m = \\frac{n_m \\bar{v}_m + v_k}{n_m + 1}$ to reflect the posterior mean after incorporating $v_k$?\n\n2. What is the computational cost (training time, memory) of Open-OTAF relative to baselines like MADDPG or VAE-GPL?\n\n3. Can you elaborate more on what meta-paths are and its overall role in the performance?\n\n4. What prevents Equation 5 from collapsing into a degenerate solution where $\\mathcal{Z}_t$ collapses to something uninformative like a constant? Aside from empirical results, of course.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Open-OTAF (Open Online Teammate Adaptation Framework), a novel framework designed to enable an autonomous agent to collaborate with unknown and dynamic teammates in partially observable and open environments -- settings where team composition can change over time. Specifically, the paper tackles *open ad hoc teamwork*, an important variant of multi-agent collaboration where both the number and type of teammates vary dynamically. The authors go beyond standard assumptions like fixed number of teammates and types, or full observability.\n\nLet me just start by saying that by no means is this framework simple -- and indeed has a lot of moving parts. I had to revisit previous sections on multiple occasions to be able to keep track of the underlying story.  That said, here’s how I understand its main contributions:\n\n* A Chinese Restaurant Process (CRP) for online clustering of teammate behaviors. This is required so as not to model every single teammate individually, but by behavior types. It also allows for incoming individuals to either form new clusters or join a previous one. This gives a clustered representation for each teammate. Since the team size is dynamic, some modeling is required to obtain a single representation of the entire teamwork. Therefore, the authors propose their next contribution, HAN.\n\n* Heterogeneous Graph Attention Network (HAN) is then employed to map these teammate clusters into a fixed-size embedding of the current teamwork situation, accounting for variable team sizes and heterogeneous relationships between agents.\n\n* Mutual Information (MI) Loss is used to align the agent’s local belief (formed from partial observations) with the global teamwork situation inferred during training, helping the agent generalize at test time when global information is unavailable.\n\nOverall, the empirical results show that Open-OTAF outperforms several baselines like MADDPG and GPL, both in performance and convergence speed, across a suite of benchmark multi-agent tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Aside from addressing the questions raised above, I encourage the authors to provide more intuitive explanations throughout Section 4 -- about how each component is essential to the story. Posing questions every once in a while would help as you move through subsections within Sec 4.0.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lvgA6jiUDC", "reviewer": "Reviewer_69MY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The manuscript presents several notable technical contributions:\n- The paper establishes a rigorous mathematical relationship between marginal utility and team utility, demonstrating that $arg max_{a^i} u^i(s, a^i, \\textbf{a}^{-i}) = arg max_{a^i} Q^{π_i}(s, a^i, \\textbf{a}^{-i})$. This theoretical foundation justifies the optimization of marginal utility as a means to ensure optimal team performance, providing a solid basis for the proposed framework.\n- The teammate clustering approach combines a Chinese Restaurant Process (CRP) for prior estimation of teammate behaviors with neural network-based likelihood estimation. This probabilistic framework enables effective categorization of diverse teammate policies into distinct clusters, thereby reducing the complexity of the context search space and facilitating more efficient representation learning.\n- A significant contribution is the employment of a Heterogeneous Graph Attention Network (HAN) as an alternative to conventional graph neural networks. This design choice enables the transformation of dynamic-size team information into fixed-size output embeddings, effectively addressing the variable team composition challenge intrinsic to open teamwork and aiming to capture the heterogeneous relationships among agents.\n- A rigorous theoretical foundation is provided to justify the Centralized Training with Decentralized Execution (CTDE) paradigm through formal derivations. By ensuring that local teamwork situation embeddings maintain informative consistency with global context, the framework enables an ad hoc agent to derive proxy representations of the teamwork situation solely from local observations. This theoretical basis establishes the validity of the decentralized execution mechanism.\n\nThe manuscript exhibits significant deficiencies in its presentation quality that substantially impair comprehension and verification of the technical contributions. The inconsistent mathematical notation throughout the paper creates ambiguity regarding the precise mechanisms being proposed. Furthermore, the nomenclature used to describe key components varies across sections introducing confusion. Critical procedural details of the framework implementation are inadequately specified by lack of clarity or defined in the supplementary material and not addressed in the main text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript introduces Open-OTAF (Open Online Teammate Adaptation Framework), which addresses the problem of open ad hoc teamwork under partial observability and dynamic settings. Collaboration here includes interactions with possibly unknown, heterogeneous teammates whose number can dynamically change over time of execution. Differently from previous research, the presented framework does not make limiting assumptions on team composition and allows decentralized execution.\n\nThe presented framework addresses online adaptation with varying numbers of heterogeneous teammates in open ad hoc teamwork setting by proposing 3 main components:\n- A Chinese Restaurant Process (CRP)-based clustering approach to accommodate different teammate behaviors, categorizing diverse teammate policies into distinct clusters to reduce the complexity of the context search space.\n- A heterogeneous graph attention network (HAN) adopted to learn effective teamwork situation representations that accommodate variable team sizes and diverse inter-agent relationships.\n- A conditional policy that enables the ad hoc agent to adapt dynamically to new teammates' behaviors by conditioning its policy on the locally inferred teamwork situation.\n\nExperimental validation across four multi-agent benchmarks (Level-Based Foraging, Wolfpack, Penalized Cooperative Navigation, and FortAttack) demonstrates superior performance compared to baselines (MADDPG, GPL, VAE-GPL, ODTIS), with comprehensive ablation studies confirming the necessity and effectiveness of each key component. The empirical results indicate not only higher average returns but also significantly faster convergence speed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical points:\n\nThe notation adopted across the manuscript and specified in Table 1 lacks consistency, which significantly impedes comprehension. Some points that must be addressed includes:\n- Inconsistent formatting style for mathematical entities e.g., bold font adopted for agent sets appears inconsistently, and where curly braces are used to represent sets clear index bounds should be defined\n- Switching between superscript and subscript index position in occurrences of the same variables e.g., $o$, $a$, $\\beta^{\\Phi^i}$, $C^i_t$ in Sec. 3, 4, A, Algorithm 1 and Fig.1, 3\n- Inconsistent meaning of sub/superscripts throughout the paper. A systematic approach should be adopted where, for example, superscripts consistently denote agent indices while subscripts indicate temporal indices.\n- Variable definitions changing between main text and supplementary material e.g., concentration parameter symbol differs between Section 4.2 and Algorithm 1/A.3\n\nAditionally, the notation results excessively complex and specific, leading to multiple symbol overlaps and definitional ambiguities. I recommend simplifying the notation through several approaches (not mandatory suggestions):\n - $\\omega = ${$ \\omega_1, \\omega_2$} parameter set could easily be simplified to a single larger parameter set $\\omega$\n - Remove variables that are introduced but never utilized\n - Omit time subscripts in explanatory sections where temporal indexing is not essential (with appropriate notation in the text)\n - Consider standardizing agent indexing to use either $i$ or $k$ consistently\n\nSpecific examples of problematic notation from Table 1 include:\n- $T_k$ indicates $k^{th}$ teammate and $T$ the time horizon\n- $D_{\\omega_2}$ represents the trajectory decoder while $D$ denotes sampled data from replay buffer\n- P (number of meta-paths) is introduced but never used\n- $\\mathcal{N}$represents both the dynamic set of agents and the normal distribution\n\n---\nMajor points (important clarifications needed and strongly suggested changes):\n\n- Sec.2: The statement \"The policy gradient methods are a sub-class of CTDE\" is erroneous unless restricted to Cooperative MARL contexts. Please rephrase.\n- Fig.1 and Fig.3: Font size and element dimensions results too small for comfortable reading.\n- Sec.3: The variable $Q$ is used without formal introduction as joint action value function.\n- Sec.4.2: The explanation of variable $v^m_k$ requires substantial clarification. The text states \"$v^m_k$ represents that the $k^{th}$ teammate belongs to the $m^{th}$ cluster\" suggesting a boolean assignment variable, which aligns with the prior distribution definition. However, in the predictive likelihood equation, computed by the decoder, $P(\\tau^{\\alpha}_{k} | v^m_k; \\tau^s_k) = D(\\tau^{\\alpha}_k | \\tau^s_k ; v^m_k)$, following A.3 (in the network input description) and Eq.1, the Decoder computing the predictive likelihood appears to be conditioned on the Encoder's output features i.e. $v_k$ set of features rather than the $v^m_k$ boolean variable. Additionally, the cluster assignment process \"we are able to decide which cluster the $k^{th}$ teammate belongs to\" lacks formal definition in the main text (I assume $m = argmax_m P(v^m_k|\\tau_k)$ from Algorithm 1); $\\bar{v}^m$ has not been formally defined and neither its update following teammate cluster assignment $\\bar{v}^m = \\frac{n^m \\bar{v}^m + v_k}{n^m+1}$ when $m \\leq M$, else $\\bar{v}^m = v_k$ if $m = M +1$ (from Algorithm 1).\n- Eq.1: Please clarify whether the negative log likelihood loss is computed on the set of trajectory actions, as per equation definition, or only on the last action $a_t$ as described in A.3.\n- Sec.4.3: The sentence \"Note that the size of {$c^t_k$} corresponds to the number of teammates, which can change across episodes in the open ad hoc teamwork\" raises questions about whether {$c^k_t$} and {$k$, $p(v^m_k|\\tau_k)$} have sizes equal to or upper-bounded by the number of teammates (changing across the episode). Additionally, the section's opening statement \"After the teammates are divided into different clusters, we randomly choose multiple teammates to interact with the controlled agent\" suggests that the teammates are clustered and then a number of those (fixed number?) is randomly sampled and considered in the global teamwork situation computation. Please clarify this point and also if the sampling takes in consideration the clustering subdivision.\n- Sec.4.3: Is it correct to state that the two meta-path graphs are complete graphs that are learned in the sense that the node-level attention weights the connections between adjacent nodes? I have some doubts regarding this characterization. Alternatively, do the graphs representing coordination and no-coordination reflect fixed relationships that are part of the adopted environment? Additionally, please clarify the definition of neighbors in the graph: does the notation $\\mathcal{N}$ refer to a specific subset of the entire dynamic set of agents involved in the task? Lastly, how are different meta-paths able to learn distinct relationships, and where is evidence of this presented in the work?\n- Sec.4.3: More details about $att_{node}$ and the semantic level attention network are required as their structure and descriptions are not provided. No further detail can be inferred from section A.3 where HAN network is cited (also Fig.A.2 d and e) but without explicit description of such components.\n- Sec.4.3: The rationale for adopting the hypernetwork $G$ is inadequately discussed.\n- Sec.4.4: Local trajectory $B$ has been mentioned without formal definition of what it represents and how $\\mu_{\\phi_i(B^i_t)}$ and $\\sigma_{\\phi_i(B^i_t)}$ have been obtained and used in place of $\\mu_{c_t}$ and $\\sigma_{c_t}$.\n- Sec.5.2.1: The text states \"These results show that our method can design an autonomous agent capable of robust teamwork under dynamic teammates without pre-coordination\" however the dynamic aspects involved in the experimental setup are not explained in the main text, but reported only in the supplementary material.\n- Fig.6: The ablation graph does not specify which environment was used for testing.\n- A.2.1: The \"Setup for Fortattack\" paragraph lacks a description of the game dynamics.\n- A.2.2 and A.2.3: Details about the Fortattack environment are missing.\n- Fig.A.2: Please clarify what is meant by \"auxiliary agent model\".\n\n----\nMinor points (e.g. typo, personal suggestions):\n\n- Sec.1: Consider adding citations to communication/message passing methods literature to strengthen the claim \"achieving online adaptation with dynamic teammates in open AHT environments remains an open challenge\". In such research branch, heterogeneous collaboration in a POMDP setting is a topic with relevant works but faced with additional cost of information sharing to develop beliefs of other agents.\n- Sec.1: \"Section 2 provides a briefly review\" -> \"brief\"\n- Sec.2: \"These works [Barrett et al.(2017)]\" is poorly phrased\n- Sec.2: \"Agent modellingg\" -> \"Agent modelling\"\n- Sec.4.1: \"notifications\" -> \"notations\"\n- Sec.4.1: \"[...]. Then, we can then\" is redundant\n- Sec.4.2: \"demotes\" --> \"denotes\"\n- Sec.4.2: Add reference to derivation in A.1 when presenting the conditional probability proportional relation\n- Sec 4.3: \"g denote the activate function\" -> \"activation\" ?\n- Sec 5.1: \"the environments and aldorithm configuration are provide in\" -> \"algorithm\", \"provided\"\n- Sec 5.2.2: \"The results that the\" is poorly phrased\n- Sec 5.2.3: \"methods that achieve low returns\" -> \"high\" ?\n- A.2.2: \"[...]. Specificity, in the Wolfpack environment\" -> \"Specifically\"\n- A.2.2: \"5 different MARL-based to develop\" -> \"MARL algorithms\" ?", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)