Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
Authors: Xinzhi Zhang
Hohei Chan
Deheng Ye
Yi Cai
Mengchen Zhao
Venue: ICML 2025 poster (2025)
Keywords: Ad Hoc Teamwork
Offline Reinforcement Learning
Primary area: reinforcement_learning->multiagent
TL;DR: This paper frames offline ad hoc teamwork as a sequence modeling problem and proposes goal-based Decision Transformers to train the ego agent for effective collaboration with unknown teammates.

Abstract: The ability of agents to collaborate with previously unknown teammates on the fly, known as ad hoc teamwork (AHT), is crucial in many real-world applications. Existing approaches to AHT require online interactions with the environment and some carefully designed teammates. However, these prerequisites can be infeasible in practice. In this work, we extend the AHT problem to the offline setting, where the policy of the ego agent is directly learned from a multi-agent interaction dataset. We propose a hierarchical sequence modeling framework called TAGET that addresses critical challenges in the offline setting, including limited data, partial observability and online adaptation. The core idea of TAGET is to dynamically predict teammate-aware rewards-to-go and sub-goals, so that the ego agent can adapt to the changes of teammates’ behaviors in real time. Extensive experimental results show that TAGET significantly outperforms existing solutions to AHT in the offline setting.

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
Xinzhi Zhang Hohei Chan 1
Deheng Ye Yi Cai
Mengehen Zhao
Abstract
Previous research on AHT primarily focuses on online rein- forcement learning (RL) methods (Barrett & Stone, 2015; Durugkar et al., 2020; Mirsky et al., 2020; Ye et al., 2020), which typically require the access to the environment in- volving various pre-trained teammates. However, these approaches face significant challenges in practice. First, the tcammates' joint policy space could be extremely large, and it impossible simulate all combinations of teammates' policies during training. Existing works (Gu et al., 2021) usually train the ego agent's policy against subset of team- mates' policies, which greatly reduces the generalization ability of the learned policy. Second, environmental sim- ulators are often unavailable or expensive, especially for real-world environments, without which the online RL ap- proaches cannot be applied. Fortunately, in many cases, of- fline data collection provides a safer and more cost-effective alternative. For instance, numerous traffic data is recorded by cameras and road sensors every day. This motivates us to study the problem of AHT in an offline setting.
The ability of agents to collaborate with previ- ously unknown teammates on the fly, known as ad hoc teamwork (AHT), is crucial in many real- world applications. Existing approaches to AHT require online interactions with the environment and some carefully designed teammates. How- ever, these prerequisites can be infcasible in prac- tice. In this work, we extend the AHT problem to the offline setting, where the policy of the ego agent is directly learned from multi-agent in- teraction dataset. We propose a hierarchical se- quence modeling framework called TAGET that addresses critical challenges in the offline setting, including limited data, partial observability and online adaptation. The core idea of TAGET is to dynamically predict teammate-ane aware rewards-to to- go and sub-goals, so that the ego agent can adapt to the changes of teammates" behaviors in real time. Extensive experimental results show that TAGET significantly outperforms existing solu- tions to AHT in the offline setting.
However, addressing AHT problems in the offline setting remains challenging: (1) Offline RL heavily relies on the pre- collected datasets, which can hardly cover diverse teammate behaviors and environmental dynamics, leading to a poor generalization of the learned policy; (2) In partially observ- able environments, incomplete information makes it more challenging to accurate teammates modeling. This issue is aggravated in the offline setting because we cannot compen- sate for missing information through active exploration; (3) Existing sequence modeling methods for decision-making (e.g., Decision Transformer (Chen et al., 2021)) primarily rely on the return-guided action generation. However, a sim- plc return signal fails to capture enough information about the frequently changing teammates' intentions and policies in the setting of AHT.
1. Introduction
Ad hoc teamwork (AHT) refers to enabling autonomous agents to collaborate with unknown teammates without prior coordination (Stone et al., 2010). This capability is crucial in dynamic and uncertain environments where agents must quickly form teams, adapt to new teammates and achieve common goals efficiently (Barrett et al., 2017). ). For instance, rescue robots should form an ad hoc team in real time to survey affected areas, locate survivors, and deliver supplies. Each robot must adapt to the changes in teammates' policies caused by some unpredictable events. Another typical appli- cation of AHT is autonomous driving, where each car must collaborate with random nearby to accomplish tasks such cross-passing and overtaking (Teng et al., 2023).
To address the issues mentioned above, we introduce a novel framework called TAGET, which stands for Teammate- Aware Goal driven hiErarchical Decision Transformers. Specifically, TAGET adopts hierarchical framework con- sisting of a high-level module for teammate- aware sub-goal prediction and a low-level module for action generation. The key innovations of TAGET are as follows. (1) We adopt a trajectory mirroring strategy to improve data efficiency, where each agent plays as the ego agent in turn, so that one piece of multi-agent trajectory can be used multiple times
Schooo of Software Engineering, South China University of Technology, Guangzhou, China TTencent, Shenzhen, China. Cor- respondence to: Mengehen Zhao <zzme @ww.eeussuuuuu....
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
for training; (2) We employ two separate state encoders for the ego agent and all the agents respectively. By regular- izing the output of the two encoders, the ego agent is able to infer the teamwork information based only on its local observations; (3) We extend the conventional return-to-go (RTG) to lcammate-awar return-to-oo (TA-RTG), which accurately reflects the team return-to-go conditioned on the current global state. Based on the TA-RTG, a teammate- aware sub-goal (TA-Goal) is decoded and used to guide the low-level Decision Transformer to generate corresponding actions. By dynamically predicting the TA-RTG and the TA-Goal at every step, the ego agent is able to adapt to the changes of teammates in real time. The main contributions of our work are summarized as follows.
(BCQ) (Fujimoto et al., 2019) and Conservative Q-learning (CQL) (Kumar et al., 2020), focus on addressing the out-of- distribution (OOD) problem by constraining policy learning, Decision Transformer (DT) (Chen et al., 2021; Li et al.; Xie et al., 2023) and Trajectory Transformer (TT) (Janner et al., 2021; Xu et al., 2021) reframe offline RL as a sequence modeling problem, achieving performance comparable to state-of-the- the-art offline RL methods while providing a novel perspective policy learning (Zheng et al., 2022; Xu et al., 2022b; Reed et al., 2022; Badrinath et al., 2024; Ma et al., 2023). Furthermore, recent works (Yang et al., 2021; Meng et al., 2021; Tseng et al., 2022; Jiang & Lu, 2023) have ex- tended offline RL to multi-agent settings and tried to learn an optimal joint policy from the offline datasets. However, these works do not apply to the ad hoc setting, where the teammates' policies may not follow the optimal joint policy, By contrast, we try to learn teammate- can adapt to the changes to teammates' policies in real time.
1. We extend the AHT problem to the offline setting, where an ego agent needs to learn a policy that adapts to various teammates from the offline dataset.
policy that
2. We propose a hierarchical sequence modeling frame- work called TAGET, which successfully addresses the offline AHT problem using commatte-wware goal- conditioned Decision Transformer.
Teammate modeling, Teammate modeling (Barrett et al., 2012) greatly facilitates multi-agent learning by helping the agents to understand each other. Traditional AHT methods primarily rely on accurate teammate modeling to adapt to unknown teammates, which refines teammate models in real time through online interaction using approaches such as meta-learning (Al-Shedivat et al., 2017; Kim et al., 2021), I), Bayesian inference (Zintgraf et al., 2021), and value decom- position (Zhang ct al., 2023; Gu et al., 2021). While these methods perform well in online settings, they struggle in offline settings due to the inability to constantly improve the teammate models by interacting with the environment (Ma et al., 2024). Although some offline teammate modeling approaches leverage methods like in-context learning (Jing et al., 2023) to compensate for the inability to learn from online interactions, they still struggle with low accuracy. In our work, instead of predicting each teammate's policy, we directly predict the future global states, which are more informative to the ego agent as they effectively summarize the influence of teammates' policies on the environment.
3. We empirically validate TAGET in three classic envi- ronments. The results show that TAGET achieves the best performance against strong baselines, with an av- erage improvement of 37.83% across all environments.
2. Related Work
Ad hoc teamwork. Existing works on AHT predominantly rely on online interaction with the environment and associ- ated teammates. Early approaches assume fully observable environments with a fixed small number of teammates, and train an agent to collaborate with them through plenty of online interactions (Barrett et al., 2017; Chen et al., 2020; Rahman et al., 2021b; Huang et al., 2021). Recent works consider more realistic settings. For example, ODITS (Gu et al., 2021) and LIAM (Papoudakis et al., 2021) improve the adaptability of the ego agent in partially observable en- vironments. GPL (Rahman al., 2021a) and OAHT (Wang al., 2024) learn to address the open ad hoc problem, where the number of teammates can change. TEAMSTER lever- ages masel RL methods to approximate the poten- tially complex environments and teammates (Ribeiro et al., 2023). However, these online approaches perform poorly in the offline setting, because they heavily rely on online explorations to estimate the environments and teammates.
3. Problem Statement
Our objective is to train an ego agent to collaborate with unknown teammates in partially observable environments from fixed dataset. To this end, we formulate the problem as a decentralized partially observable Markov decision pro- cess EDec-POMDP with additional joint policy space of teammates, which can be formally described as tuple (N, S, O, P, R, O,T), where N = (1, 2,...,n n) rep- resents the set of agents involved in the task and S denotes the global state space. We consider a partially observable set- ting in which each agent receives an individual observation of O via an observation function O S x A + (OO)). The joint action space is defined as A = A' x A² x X N.
Offline reinforcement learning. Offline RL (Levine al., 2020) aims to learn policies from static datasets without on- line exploration, effectively reducing data costs and safety risks (Xu et al., 2022a), with notable success in single-agent settings (Qu et al., 2023; Wei et al., 2021). Traditional of- fline RL. algorithms, such as
Q-learning
2

---

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
4.1. Overview
Online AHT
Rolloct data
TAGET aims train an ego agent from an offline dataset so that it can maximize the expected team reward given any teammates' policies. Figure 2 illustrates the framework of TAGET, which comprises three main modules. First, the offline data pre-processing module mitigates the issue of limited data through a trajectory mirroring strategy, where each agent alternates as the ego agent to make the most use of multi-agent trajectory data. Second, the high-level teammate-awar sub-goal prediction module learns la- tent variable from local observations to capture the core information of teamwork situations. Specifically, it pre- dicts teammate-awand return-to-ge and then combines it with the latent variable to predict teammate-awan sub-goal, which guides decision making and improves the adapta- tion to dynamic teammate behaviors. Finally, the low-level coal-conditioned action generation module utilizes the De- cision Transformer predict actions conditioned the high-level TA-Goal, ensuring flexibility and robustness in -tatstationary environments.
Environment
Joint action
!
Team reward
Butfer B
Ego agent
Teammates
Learn
(0p oi, Tei P111 done)
Sample
Teammate modeling
Training
Deployment
Ego agent
Teammates
Leen
Offinee dataset D
Joint action
Team reward
Enviromment
(b) Offline AHT
Figure 1. Comparison of Online and Offline Ad Hoc Teamwork Framoworks. (a) shows the online AHT framework, where the ego agent learns through real-time interaction, collecting data and updating its policy continuously. (b) illustrates the offline AHT framework, where the agent trains a pre-collected dataset.
4.2. Offline data collection and pre-processing
As illustrated in Figure 2, enable policy learning without online interaction, we first collect a comprehensive offline dataset of multi-agent interaction trajectories, denoted as D = { T2, TM }. Each trajectory тj represents a com- plete episode of interaction and is structured as follows:
r' TE== (OxA)* represents the observation-action trajec- tory of agent i. The transition function P : S x AA + A(S) determines the probability distribution over the next state, given the current state and joint action. In cooperative scenarios, all agents share the common reward function R S A + (RR. We denote by I the joint policy space of teammates. In practice, I can be approximated by sampling from pre-collected diverse policy pool.
T = ( 01 a,...,, Rr.....,,,,r., or.
Here, o, denotes the joint observations, a, denotes the joint actions, and RS Σi Tr represents the return-to-go at time step t. Such offline datasets are relatively easy to collect in real-word woaldworld scenarios, as they can be obtained by recording interactions during task execution.
Figure| presents the comparison between online AHT and offline AHT. Unlike online methods, where agents interact with the environment during training, our approach uses an offline dataset D train an ego agent policy x (af lo²; D). The trained ego agent is then deployed with unknown team- mates, whose policies are randomly sampled from the joint policy space r. The goal for the ego agent is to achieve the maximum expected return:
Note that, the raw multi-agent interaction trajectories treat all agents equally without differentiating the ego agent and the teammates. If any agent is arbitrarily selected as the ego agent, it would ignore the decision-making perspective of other agents the ego agent. Therefore, pre-processing technique called trajectory mirroring strat- egy, which reorders the positions of states and observations within a trajectory so that each agent alternates as the ego agent. This strategy effectively utilizes the dataset to extract information from multiple decision-making perspectives and mitigates the challenge of limited data.
Do
propose data
max E
P.D.
Σ
rz
dg
4. Method
The offline AHT problem faces many practical challenges, including limited data, inaccurate teammate modeling in partially observable environments, and online adaptation to previously unknown teammates. In this section, we intro- duce a hierarchical goal-based Decision Transformer frame- work called TAGET, which addresses the above challenges by a unified sequence modeling approach.
Formally, for trajectory т with N agents, the trajectory mirroring strategy generates N trajectories by re-ordering the positions of the states and observations. For each mir- rored trajectory Tj.s, agent is treated as the ego agent, while the remaining agents are considered as teammates.
3

---

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
Offline data collection and pre-processing
Pre-collected datasct
Play as ego agent in turn
Ego agent
Trajectory
D
Tu (... oiiii oa aatt...)
Smireer
Taa f.. R2 of ot aa?.....)
Sample
tu. Re o2....
Trajectories
High-level: Teammate-aware goal prediction
Lor (+)
Ego agent
Teummates
LyToa (d)
TA-RTG Predictor (R(Re
Causal Transformer
TA-Goal Decoder
(o(
M
Ri
Proxy Encoder
so(s{lof)
Tenm Context Encoder fo (Aililo)
Ap
GrGGilkf,R Rt)
G
o
Liu (9, s)
LYa (Qa
Low-level: Goal-conditioned action generation
Figure 2. Training overview the proposed TAGET. First, the (1) Offline data collection and pre-processing module collects trajectories and then generates mirrored trajectories by treating each agent the ego agent. These processed trajectories then learned by the hierarchical decision framework. At each time step I, the (2) High-level: Teammate-awan goal prediction module extracts a latent representation 2f from local observation oi, approximating the team context hū by regularization. The module predicts TA-RTG R. based on by and then utilizes RS and h; to predict TA-Goal G,- Finally, the (3) Low-level: Goal-conditioned decision-making module employs Causal Transformer condition the predicted goal Ge, generating the next action for the ego agent.
The transformation is defined as:
the teammate-aware RTG (TA-RTG) as inputs. The team context encodes the current global state, while the TA-RTG estimates the expected return. Taking these two as inputs, the TA-Goal prediction captures both the current state and future return, serving the high-level objective to guide the ego agent action generation.
Ty fmiror (Ty, (Re, of or ai a, '1 It-
(3)
where of and at are the observation and action of the ego agent, while o, and a, represent the joint observations and actions of all the other agents. For example, as shown in Figure 2, if trajectory involves three agents, the trajectory mirroring strategy generates three trajectories: T,,1 Ty.2, and Tj.3- These trajectories are distinct in downstream modules because the information of the ego agent and the teammates is processed in different ways.
First, we learn to represent team context under partial ob- servability. In multi-agent scenarios, each agent typically has access only to local observations, which hinders under- standing of teammates' intentions. We propose two state encoders: team context encoder for all agents and proxy encoder for the ego agent. With the regularization between the outputs of the two encoders, the ego agent is able to infer the team context based only on its local observations.
4.3. High-level: feammate-aware goal prediction
For decision making, vanilla DT relies on scalar return- to-go (RTG) signal to guide policies. However, teammates may exhibit highly dynamic and unpredictable behaviors in AHT, so simple RTG can hardly capture uncertain- ties in teammates and the environment. To this end, we introduce new concept called teammate-awen sub-goal (TA-Goal), the predicted global state at future step that serves as a high-level objective. TA-Goal captures the dy- namics of teammate behaviors, providing more robust and adaptive guidance than RTG in complex multi-agen set- tings. To predict the TA-Goal we use the team context and
Team context encoder: The team context is high-level se- mantic representation derived from the current observations of all agents. To model this, we represent the team context in stochastic embedding space H. where is encoded as latent probabilistic variable h drawn from multivariate Gaussian distribution. We utilize function / to learn the parameters of the multivariate Gaussian distribution:
(Mis, ow) (o(o), 1¹ ~ ~N(p), (4)

---

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
Proxy encoder: Due to the partial observability during execution, directly applying f to capture the team context is infeasible. To overcome this limitation, we introduce Proxy Encoder f that approximates the team context from only local observations of, assuming that of can partially capture the cooperative context as it inherently reflects team- mate interactions and behavioral features. Similarly, z', as an approximation of h', is encoded into stochastic embed- latent probabilistic
enables more accurate and adaptable estimate of future performance and informs the TA-Goal generation process.
Teammate-aware return-to-go (TA-RTG): The TA-RTG R₁ represents the predicted return-to-ge -o- based on the team context hi, which quantifies the contribution of teamwork situation to long-term task impact. It provides critical dy- namic optimization feedback for generating TA-Goal. To predict the TA-RTG, we design an RTG predictor Rg that takes h; as input: (R.l))). The training objective of the predictor is to minimize the discrepancy between the predicted RTG R& and the ground truth Rs. Specifically, we use the following MSE loss function:
ding space, where it is represented variable drawn from multivariate Gaussian distribution. The Proxy Encoder I is used to learn the parameters of this Gaussian distribution:
(2)) folo)), 2 ~ (
(5)
(RO(M)) - Rd) )
2
LRTG = ETTD
Here, 2² serves latent representation of teammate dy- namics, robustly guiding decision masiing when global ob- servations are unavailable.
Although the TA-RTG provides more accurate information than conventional RTG by considering teammates' behav- iors, it is still less informative for downstream action genera- tion, especially in environments with sparse rewards. There- fore, we further predict the TA-Goal, which represented by the future global state and provides richer contextual in- formation than the TA-RT RTG. Based on the team context and the TA-RTG, we predict the TA-Goal to guide the decision of the ego agent. The TA-Goal enables the ego agent to coordinate its actions with dynamic teammate behaviors, thereby enhancing robust collaboration in complex multi- agent environments.
Latent variable regularization: To ensure zj to effectively represent the team context under partial observability, we aim for 2j to remain consistent with hi. Inspired by pre- vious works (Ajay et al., 2020; Pertsch et al., 2021; Xie et al., 2023), we use the following objective to minimize the discrepancy between the two representations and stabilize the high-level representation:
T DEL (fe(1 or (T
LReg(e (6,4p) ETTD
Σ
T
Teammate-aware goal (TA-Goal): The TA-Goal G. repre- sents the predicted global state at time t t k, which provides high-level objective to guide low-level decision-making It is predicted based on the team context hū and the pre- dicted TA-RTG RG To predict the TA-Goal, we designed the TA-Goal decoder G5 that takes hi and Re as inputs: GG,,, Re). The TA-Goal decoder is trained by min- imizing the difference between the predicted G. and the ground-truth G, in the dataset. G, is constructed by con- catenating the discretized and one-hot encoded observations of all agents at time step t + k, representing the global state at that future moment. The loss function is defined as:
+ BLETDD
DEL (Ifo(- of, of')) f11 of) Σ
where DKL denotes the Kullback-Leibe (KL) divergence, [-] indicates the gropggradeent operator, and 3 is hyperpa- rameter balancing the two regularization terms. The first Kl.-divergence term DRa (f of ") (0, T)) is de- signed as a reverse KL to enforce a compact latent space. Following the beta-VAE framework (Higgins et al., 2017), reverse KI. encourages the learned latent distribution to concentrate around the modes of the prior standard normal distribution, effectively preventing overfitting to noisy obser- vations and improving generalization to unseen teammates. In contrast, the second term employs forward KI,, which aligns the proxy encoder's output with the team context encoder by minimizing the divergence in expectation. This hybrid regularization ensures the latent space regularity and the consistency between local and global representations under partial observability.
LTA-Gool EEND
Σ
Gt, log Gr.i
(8)
+ (1 - GG,) log(1 (og(1-GGa))
where Gt,₁ = St+h Concat
In total, the overall loss function Ltiih-tevel for the high-level network is defined as:
Then, we learn to predict TA-RTG, which is conditioned on the team context. Unlike traditional RTG in single-agent settings, TA-RTG captures the influence of dynamic team- mates, making it better suited for AHT scenarios. This
LHigh-level = AReg Log + ARTGLRTG + ATA-Goa (9)
where AMI ARTO, ATA-Gool are weighting parameters used to balance the contributions of different loss terms.
5

---

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
4.4. Low-level: Goal-conditioned action generation
adaptability to diverse teammate strategies, we adopt the Value Diversity (SVD) method proposed in CSP (Ding et al., 2023) to collect data. This method encourages the in- clusion of a wide range of behavioral policies in the dataset. We trained four distinct populations of multi-agent rein- forcement learning (MARL) policies for each environment. From these, one population was randomly sampled as the testing teammate set, while the remaining three were used to collect interaction trajectories for the offline dataset. Each training population consists of three unique checkpoints, while each testing population includes eight checkpoints. Each checkpoint represents a distinct joint strategy, captur- ing a diverse range of behaviors and interaction patterns. By separating training and testing populations, this approach ensures a rigorous evaluation of the model's ability adapt to unseen teammate policies.
We use a goal-based Decision Transformer as the low-level policy network and solve the offline AHT problem through conditioned sequence modeling. The context for the low- level module is composed of two inputs: the TA-Goals predicted by the high-level network and the pre-processed trajectory segments. Specifically, the inputs to the low-level DT are constructed as follows:
Ti. = (Ge,or de 1 de---=KK+1)
(10)
where K' denotes the context length, i.e., the number of past time steps included in the sequence. The causal Transformer processes this sequence and outputs the predicted action of for the current time step. The training of the low-level mod- ule aims to minimize the deviation between the predicted action and the ground-truth action from the dataset. The loss function for training is defined as:
To enhance the coverage of offline datasets, we employed trajectory mirroring strategy, as we suggested above. For each trajectory in the dataset, this method treats every agent as the ego agent in turn, generating additional trajectories from different perspectives. The resulting dataset enables more robust policy learning in dynamic and partially observ- able environments, as demonstrated in our experiments.
LOT (xb)
E TwD Ge-GG
[- log Ps (at 1 oz G.)]
(11)
We replace RTG with TA-Goal as the prompt to overcome the limitations of DT in AHT problems. Compared RTG, TA-Goal empowers the model take actions based on the future state adapted to dynamic teammate behaviors. This allows the model to effectively collaborate with unknown teammates in dynamic MARL environments.
5.2. Baselines
We compare our method against several state-of-the-an baselines commonly used in offline reinforcement learn- ing and ad hoc teamwork tasks: Decision Transformer (DT), Prompt-DT, ODITS-off and LIAM-off. These base- lines are representative as they address the AHT problem from different perspectives.
5. Experiments
5.1. Environments and Data Collection
To evaluate the effectiveness of our proposed method, we conducted experiments in three widely used multi-agent reinforcement learning (MARL) environments: Predator- prey (PP), Level-Based Foraging (LBF) and Overcooked. These environments allow manipulation of teammates' poli- cies, thus are suitable for evaluating AHT methods. The Predator-Prey environment is a classic MARL benchmark where a team of predators must collaborate to capture prey in partially observable grid world(L et al., 2017). Suc- cessful teamwork requires coordination, as the prey can evade capture without joint effort. In the LBF environ- ment, agents are tasked with collecting food items scattered randomly in a grid-world. Each agent and food item is as- signed a level and a group of agents can collect an item only if the sum of their levels meets or exceeds the level of the item (Papoudakis et al.. 2020). The Overcooked environment requires agents to prepare and deliver dishes efficiently in complex kitchen layouts (Carroll et al., 2019). Detailed information about the environments is provided in the Appendix A for further reference.
Decision Transformer (DT) (Chen et al., 2021) refor- mulates reinforcement learning as a sequence modeling task, where the policy is learned by predicting the next action given a sequence of observations, actions and return-togo DT has demonstrated strong performance in offline reinforcement learning settings by leverag- ing the transformer architecture to model long-term dependencies in sequential data.
Prompt-DT (Xu et al., 2022b) is an extension of DT that incorporates additional prompts to guide the decision- mreision-making process to achieve a few-shot adap- tation. This method has been shown enhance gen- eralization and adaptability, making it an appropriate baseline for our approach, which also deals with di- verse and dynamic teammate behaviors.
ODITS-off is a variant of the ODITS framework (Gu et al., 2021) which uses mutual information regulariza- tion to infer teammates' behaviors from local obser- vation. We treat the trajectories in the offline dataset as if they were obtained through online interactions,
To train our model in an offline setting, we utilize pre- collected interaction trajectories. To ensure the model's
6

---

Ad Hoc Teamwork via Offline Boal-Based Decision Transformers
PP-4
LBF-4
Overcooked-1
0.15
0.1
Return
0.5
50
0.05
40
0
50
100
50
100
0
50
100
PP-8
LBF-8
Overcooked-8
0.08
60
0.06
Return
50
0.04
0.02
40
50
100
50
100
Epochs
Epochs
Epochs
TAGET
DT
Prompt-DT
ODITS-off
LIAM-off
CQL
MADT
Figure 3. Average evaluation returns curves and confidence interval of five evaluated methods during training
3
8
Ego Agent
Teammates
TA-Goal
Prey
Figure 4. The visualization of one episode of the PP game, illustrates how the TA-Goal guides the ego agent (Red) to collaborate with its teammates in capturing the prey. At each step. instead of directly leading the ego agent the prey, the TA-Goal leads the ego agent towards the optimal path with consideration and prediction of teammates' moves. The TA-Goal continuously adjusts as the agents move, ensuring that the ego agent responds its tcammates optimally.
replacing online interaction and buffer-based learning with direct learning from the offline dataset.
vative value functions. It penalizes unseen actions in the Q-function optimization, thus ensuring safer policy learning from static datasets. We adapt CQL to the AHT settings by training the ego agent in the offline data collected under diverse teammate behaviors.
LIAM-off (Papoudakis et al., 2021) is another offline- variant of online AHT method. To adapt LIAM into LIAM-off, we modify it to learn how to reconstruct global information from local observations directly from an offline dataset instead abling the extraction of strong representational latent variables that guide the controlled agent's actions with- out requiring online interactions.
Multi-Agent Decision Transformer (MADT) (Meng al., 2023) extends the Decision Transformer frame- work to multi-agent scenarios. It models the joint tra- jectory of all agents and leverages inter-agent depen- dencies using umified transformer-basee policy. We adapt it to the AHT settings, providing strong base- line for evaluating offline AHT performance in settings where coordination and adaptation are essential.
replay buffer, en-
Conservative Q-Learning (CQL) (Kumar et al., 2020) is a widely used offline RL algorithm that effectively addresses the overestimation issue by learning conser-

---

Ad Hoc Teamwork via Offline Roal-Based Decision Transformers
LBP
Overcooked
0.15
0.1
Return
0.05
50
100
0
50
100
50
100
Epochs
Epochs
Epochs
TAGET w/o TeadmateEncoder
w/o RTG Predictor
w/o Goal Decoder
w/o Data Preprocessing
Figure 5. Impact of Model Components on Performance
5.3. Comparisons with Baselines
method introduces sub-goals as tokens that dynamically guide the agent's 's actions. These TA-Goals, predicted based on dynamic team context and TA-RTG, enable the ego agent to adapt to real-time changes teammate behaviors effec- tively. Figure shows a real trajectory sampled from the PP game environment, where the yellow pentagram represents the TA-Goal. It demonstrates how the TA-Goal guides the ego agent in cooperating with teammates and capturing the prey effectively. ODITS and LIAM occasionally achieve moderate returns but are characterized by higher variabil- ity and instability, particularly in larger test teammate sets (e.g., PP-8 and Overeooked-8) LIAM exhibits notable fluctuations, likely due to overfitting specific teammate behaviors during training.
Results. We evaluate trained model by interacting with teammates in the test teammates sets. These teammates are not involved the collection of the offline training dataset, making this evaluation an effective measure of the model's generalization ability. Performance across different sizes of test teammate sets evaluated for three environments. In the case of PP-8, one teammate set was randomly selected from eight distinct teammate sets to test our method in the PP environment, one teammate set represents a group of agents that interact with our ego agent in one episode, and similarly for other cases. The returns from each evaluation are averaged over fifty episodes interaction. The shaded region represents the 95% confidence interval. The results, summarized in Appendix B, show that our method achieves superior performance across all testing environments.
In summary, TAGET outperforms all baselines in terms of return, stability, and generalization across a wide range of multi-agent environments. This indicates that TAGET better equipped to handle the complexities of cooperative tasks with diverse and unseen teammates, addressing key limitations observed in the baseline methods.
Analysis. The results in Figure 3 provide a comprehen- sive comparison of five evaluated methods: TAGET, DT, Prompt-DT, ODITS, and LIAM. Returns are averaged over trials, with shaded regions representing 95% confi- dence intervals using the standard normal distribution for- mula: 11.9 - where is the sample standard 7
5.4. Ablation Study
deviation and n 50. TAGET consistently achieves the highest returns and demonstrates superior stability across all environments and test teammate set sizes, underscoring its strong generalization ability unseen teammates, evidenced by its higher curves. The consistent gap between TAGET and baselines such as DT and Prompt-DT demon- strates the ability of TAGET to effectively model diverse teammate behaviors, a critical challenge AHT. DT and Prompt-DT show limited performance, with returns plateau- ing lower levels. They lack explicit mechanisms to adapt to dynamically changing teammates in ad hoc scenarios. Prompt-DT, in particular, shows limited performance in en- vironments where teammate behaviors change rapidly, as relies on static prompts for guidance. In contrast, our
To evaluate the contribution of each component in our pro- posed method, we conducted an ablation study by systemat- ically removing individual components and observing the impact performance across three benchmark environ- ments. The results are summarized in Figure 5.
Across all environments, the full method (TAGET) consis- tently outperformed its ablated variants, demonstrating the importance of each component. Notably, the removal the TA-Goal decoder led to the most significant performance drop, especially in the overcooked environment, where goal- driven actions are crucial due to poor teammate strategy.
The absence of data proceesccsing also led to consistent per- formance degradation in all three environments, underscor- ing its fundamental contribution to the model's generaliza-
8

---

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
tion ability. The impact was particularly evident in the LBF environment. By generating diverse interaction trajectories, our trajectory mirroring strategy effectively mitigates overfit- ting, enhancing the adaptability to unknown teammates and the effectiveness offline learning. These findings strongly validate the design choices in our method, demonstrating that each component of our method contributes synergisti- cally its overall performance. This collective contribution significantly improves its adaptability in AHT scenarios.
datch, I., and Abbeel, P. Continuous adaptation via meta- learning in nonstationary and competitive environments. 2017.
arXiv preprint
Badrinath, A., Flet-Berliac, Y., Nie, A., and Brunskill, E. Waypoint transformer: Reinforcement learning via su- pervised learning with intermediate targets. Advances in Neural Information Processing Systems, 36, 2024.
Barrett, S. and Stone, P. Cooperating with unknown team- mates in complex domains: A robot soccer study of ad hoc teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
6. Conclusion
In this work, we introduced TAGET, a novel hierarchical framework that effectively addresses the challenging prob- lem of offline ad hoc teamwork. Through extensive exper- iments, we demonstrated that TAGET successfully over- comes three eritical challenges: limited offline data through our mirrored data pe-prccessing strategy, partial observ- ability via regularization of two state encoders, and online adaptation through leammate-aware return-to-go prediction. The strong empirical results across diverse cooperative en- vironments validate the effectiveness of our hierarchical approach combining teammate-aware goal prediction with goal-conditioned decision making.
Barrett, S., Stone, P., Kraus, S., and Rosenfeld, A. Learn- ing teammate models for ad hoc teamwork. In AAMAS Adaptive Learning Agents (ALA) Workshop, pp. 57-63. Citeseer, 2012.
Barrett, S., Rosenfeld, A., Kraus, S., and Stone, P. Mak- ing friends on the fly: Cooperating with new teammates. Artificial Intelligence, 142:132-171,22 2017.
Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S., Abbeel, P., and Dragan, A. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
Acknowledgements
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
This research is supported by Guangdong Basic and Applied Basic Research Foundation 15025A1515012244 and the Fundamental Research Funds for the Central Universities (2024ZYGX7R069).
Impact Statement
Chen, S., Andrejezuk, E., Cao, Z., and Zhang, J. Aateam: Achieving the ad hoc teamwork by employing the at- tention mechanism. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 34, pp. 7095-7102, 2020.
Our work seeks to advance the field of multi-agent reinforce- ment learning by addressing the challenges of offline ad hoc teamwork with the proposed TAGET framework. We open up exciting opportunities for generative decision-making under the uncertainty of teammates, particularly in scenar- ios where agents must adapt to dynamic and unpredictable changes of teammates. Future research may consider more uncertainties from both the teammates and the environments, making the decision models more robust. We believe that our work facilitates the application of generative decision models in real-world uncertain scenarios.
Ding, H., Jia, C., Guan, C., Chen, F., Yuan, L.. Zhang, Z., and Yu, Y. Coordination scheme probing for generalizable multi- agent reinforcement learning. 2023.
Durugkar, I., Liebman, E., and Stone, P. Balancing indi- vidual preferences and shared objectives in multiagent reinforcement learning. International Joint Conference on Artificial Intelligence, 2020.
References
Fujimoto, S., Meger, D., and Precup, Off-policy deep reinforeement learning without exploration. In Interna- tional conference machine learning, pp. 2052-2062. PMLR, 2019.
Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. Opal: Offline primitive discovery for acceler- ating offline reinforcement learning. arXiv preprint arXiv:2010.13611,200 2020.
Gu, P., Zhao, M., Hao, J., and An, B. Online ad hoc team- work under partial observability. In International confer- ence on learning representations, 2021.
A--hhviivat, M., Bansal, T., Burda, Y., Sutskever, I., Mor-

---

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta- vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
Meng, L., Wen, M., Yang, Y., Le, C.,, Li, X., Zhang, W., Wen, Y., Zhang, H., Wang, J., and Xu, B. Offline pre- trained multi-agent decision transformer: One big se- quence model tackles all smac tasks. arXiv preprint 1r.0:22112.02845, 2021.
Meng, L., Wen, M., Le, Li. X., Xing, D., Zhang, W. Wen, Y., Zhang, H., Wang, J., Yang, Y,, et al. Offline pre-trained multi-agent decision transformer. Machine Intelligence Research, 20(2):233-26) 2023.
Huang, S., Chen, W., Zhang, L., Xu, S., Li, Z., Zhu, F., Ye, D., Chen, T., and Zhu, J. Tikick: Towards playing multi-agent football full games from single-agent demon- strations. arXiv preprint arXiv:2110.045550000 2021.
Mirsky, R., Macke, W., Wang, A., Yedidsion, H., and Stone, P. penny for your thoughts: The value of communica- tion in ad hoc teamwork. International Joint Conference Artificial Intelligence, 2020.
Janner, M., Li, Q., and Levine, S. Offline reinforcement learning as one big sequence modeling problem. Ad- vances in neural information processing systems, 34: 1273-1286, 2021.
Papoudakis, G., Christianos, F., Schäfer, L.., and Albrecht, S. V. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint arXiv:2006.0786) 2020.
Jiang, J. and Lu, Z. Offline decentralized multi-agent rein- forcement learning. In ECAI, pp. 1148-1155, 2023.
Jing, Y., Li. K., Liu, B., Zang, Y., Fu, H., FU, Q., Xing, J., and Cheng, J. Towards offline opponent modeling with in-context learning. The Twelfth International Conference on Learning Representations, 2023.
Papoudakis, G., Christianos, F., and Albrecht, S. Agent mod- elling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 14:19220-19222, 2021.
Kim, D. K., Liu, M., Riemer, M. D., Sun, C., Abdulhai, M., Habibi, G., Lopez-Cot, S., Tesauro, G., and How, J. A policy gradient algorithm for learning to learn in multia- gent reinforcement learning. In International Conference on Machine Learning. pp. 5541-5550 PMLR, 2021.
Pertsch, K., Lee, Y., and Lim, J. Accelerating reinforcement learning with learned skill priors. Conference on robot learning, pp. 188-204, PMLR, 2021.
Qu, Y., Wang, B., Shao, J. Jiang, Y., Chen, C., Ye, Z.,, Line, L., Feng, Y., Lai, L.., Qin, H., al. Hokoff: Real game dataset from honor of kings and its offline reinforcement learning benchmarks. Advances in Neural Information Processing Systems, 36:22166 62-22190, 2023.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33: 1179-1191, 2020.
Rahman, M. A., Hopner, N., Christianos, F,, and Albrecht, S. V. Towards open ad hoc teamwork using graph-based policy learning. In International conference on machine learning, pp. 8776-8786, 2021a.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint ar arXiv:2005.0/643 2020.
Rahman, M. A., Hopner, N., Christianos, F., and Albrecht, S. V. Towards open ad hoc teamwork using graph-based policy learning. In International conference on machine learning, pp. 8776-8786. PMLR, 2021b.
Li, W., Luo, H. Lin, Z., Zhang, C., Lu, Z., and Ye, D. A survey transformers in reinforcement learning. Trans- actions on Machine Learning Research.
Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, Multi-agent actor-critic for mixed compettive environments. Advances in neu- ral information processing systems, 30, 2017.
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., et al. A generalist agent. arXiv preprint arXiv:2205.0677 2022.
Ribeiro, J. G., Rodrigues, G., Sardinha, A., and Melo., F. S.
Ma, L., Wang, Y., Zhong. F., Zhu, S.-C., and Wang, Y. Fast peer adaptation with context- aware exploration. arXiv preprint ar 402226.2.024688. 2024.
Teamster: Model-based reinforcement learning for ad hoc teamwork. Artificial Intelligence, 324:104013, 2023.
Stone, P., Kaminka, G., Kraus, S., and Rosenschein, J. Ad hoc autonomous agent teams: Collaboration without pre- coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp. 1504-1500 2010.
Ma, Y., Xiao, C., Liang, H., and Hao, J. Rethinking deci- sion transformer via hicrarchical reinforcement learning. arXiv preprint arXiv:231 1.0067,, 22226,,0020 2023.
10

---

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
Teng, S., Hu, X., Deng, P., Li, B., Li, Y., Ai, Y., Yang. D., Li, I.., Xuanyuan, Z., Zhu, F., al. Motion planning for autonomous driving: The state of the art and future perspectives. IEEE Transactions on Intelligent Vehicles, 8(6):3692-3711, 2023.
Zheng, Q. Zhang, A., and Grover, A. Online decision trans- former. In international conference machine learning, pp. 27042-27059, PMLR, 2022.
Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., and Hof- mann, K. Deep interactive bayesian reinforcement learn- ing via meta-learning. arXiv preprini arXiv: 100.03864 2021.
Tseng, W.-C., Wang, T.-H. J., Lin, Y.C.,, and Isola, P. Of- fline multi- agent reinforcement learning with knowledge distillation. Advances in Neural Information Processing Systems, 35:226-237, 2022.
Wang, J. Li, Y., Zhang. Y., Pan, W., and Kaski, S. Open ad hoc teamwork with cooperative game theory. arXiv preprint arXiv:2402.15259 2024.
Wei, H., Ye, D., Liu, Z., Wu, H., Yuan, B., Fu, Q., Yang, W., and Li, Z. Boosting offline reinforcement learning with residual generative modeling. In 30th Interaational Joint Conference on Artificial Intelligence, IJCAI 2021, pp. 3574-3580. International Joint Conferences on Artificial Intelligence, 2021.
Xie, Z.,, Lin, Z., Ye, D., Fu, Q., Wei, Y., and Li, S. Future- conditioned unsupervised pretraining for decision trans- former. In International Conference on Machine Learn- ing, pp. 38187-38203. PMLR, 2023.
Xu, H., Zhan, X., and Zhu, X. Constraints penalized q- learning for safe offline reinforcement learning. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 8753-8760, 2022a.
Xu, K., Zhang, Y., Ye, D., Zhao, P. and Tan, M. Relation- aware transformer for portfolio policy learning. In Pro- ceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence,
pp. 4647 4653, 2021.
Xu, M., Shen, Y., Zhang, S., Lu, Y., Zhao, D., Tenenbaum, J., and Gan, C. Prompting decision transformer for few- shot policy generalization. In international conference machine learning, pp. 24631 1-445.. PMLR, 2022b.
Yang, Y., Ma, X., Li, C., Zheng, Z., Zhang, Q., Huang, G., Yang, J., and Zhao, Q. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 1429999-11332. 2021.
Ye, D., Chen, G., Zhang, W., Chen, S., Yuan, B., Liu, B., Chen, J., Liu, Z. Qiu, F, Yu, H., al. Towards play- ing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems, 33: 621-632, 2020.
Zhang, Z., Yuan, L., Li, L., Xue, K., Jia, C., Guan, C., Qian, C., and Yu, Y. Fast teammate adaptation in the presence of sudden policy change. In Uncertainty in Artificial Intelligence, pp. 2465 2476. PMLR, 2023.

---

Ad Hoc Teamwork via Offline Koal-Based Decision Transformers
A. Details of Environments.
(a) Predator-Prey
(b) Level-Based Foraging
(c) Overcooked
Figure Impact different goal steps
Level-based foraging. In the Level-based Foraging (LBF) environment, two agents forage within 20 20 grid world containing four food items. Each agent perceives the environment through a local observation space defined by 5 window centered its current position. The action space for each agent includes five discrete actions: four directional movements (up, down, left, right) and a "collect" action. The agents share team reward, which is proportional to the level of the collected food and normalized to 1. An episode ends either when all food items are collected or after 50 timesteps. To enhance the challenge, a strict coordination rule is imposed: food can only be collected when both agents are adjacent to the same food item and simultancously perform the "collect" action. This rule forces agents to synchronize both spatially and temporally, making cooperation essential. During evaluation, TAGET or baseline methods control one agent, while the other agent's behavior is governed by teammates sampled from predefined policy sets. The variability in coordination strategies reflected in the sequence in which agents collect all food items.
Predatory prey. The Predator-Pres (PP) environment is more complex variant of the LBF. In this scenario, two predator agents operate within 10 x 10 grid world, each with a local 5 x 5 observation space centered on its current position. The objective is for the predators to capture four prey, which move randomly across the grid throughout the episode. The action space for each predator consists of four directional movement actions (up, down, left, and right). An episode ends either when all prey have been captured or after 200 timesteps. The random movement of the prey increases the difficulty of the task, predators must maintain continuous coordination effectively chase and surround them. To simplify the interaction. explicit "capture" actions are removed, and prey considered captured once they are fully surrounded by both predators. This eliminates the risk of failed captures but still requires precise spatiotemporal coordination. The coordination strategies adopted by the predators are reflected in how they organize their pursuit and divide roles, as well as in the sequence in which prey are captured. This design highlights the importance of cooperative behaviors in multi-agent environments.
Overcooked. In the Overcooked environment, two agents must collaborate virtual kitchen to complete cooking tasks. Preparing a dish requires a series of actions and includes a waiting period. Once the dish cooked, an agent must pick up, transport it the designated delivery point, and place to complete an order. In this environment, both agents share action space consisting of six discrete actions: moving in four directions, interacting with the object in front, and remaining idle. The objective is to complete many orders possible within 400 timesteps through efficient collaboration. Specifically, in our environment, we adopted the "Surrounding" layout, as shown in Fig. In this layout, only the green agent can operate the cookware, while only the blue agent can reach the delivery point. Since a single agent cannot independently complete the entire task, the two agents must cooperate closely to fulfill the orders. In this layout, coordination is mandatory, and agents must adapt to each other's behavior and preferences to complete the tasks.
12

---

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
B. Experiment Results
Methods
PP-4
LBF-4
Overcooked-4
PP-8
LBF-8
Overcooked-8
DT
54.6 0.6
0.098 + 0.008
0.16 + 0.19
55,4 + 1.6
0.055 + 0.009
0.08 = 0.03
Prompt-DT
56.3 0.7
0.102
0,40 + 0.19
54.4 1.2
0.045
0.008
0.64 = 0.15
ODITS-off
5 1.6
0.065 + 0.036
0
54.7 1.4
0.000 0.000
0.06 0.03
LIAM-off
58.2
0.005
0.42 0.26
55.3 1.8
0.010
0.24+
CQL
60.6 1.0
0.065 +
0.46 0.08
59.0 0.8
0.005
0.009
0.36 0.08
MADT
1.1
0.010
0
62.4 0.6
0.025
0.008
0.12
63.2
0.140 0.080 0.080
0.98 + 0.15
62.9 + 1.3
0.080 0.010
0.77 0.15
TAGET
+2.93%
+37.25%
+113.04%
+8.01%
+45.45%
+20.31%
Table 1. Average Return Comparison with Baselines
C. Impact of Goal Steps
The pyper-parameter goal_steps plays erucial role in our algorithm, determines how far into the future the teammate observations are considered when generating sub-goals. Specifically, we define goal_steps as the number of steps ahead whose teammate observations concatenated form the sub-goal. These sub-goals are then used for supervised learning, guiding the agent to align its actions with future team objectives.
We evaluate the impact of different goal_steps by testing the values of 1, 2, 3, and 6 across three environments, the results are shown in Table 2. We observe that the optimal value of goal sleps depends on the complexity and dynamics of the environment. Predator-Prey. the environment's simplicity and need for long-term planning result in the best performance with goal _steps A larger goal_steps enables the agent to make farsighted decisions, coordinating with teammates over extended horizons. Level-Based Foraging, _oal_steps 2 achieves optimal results. In Overcooked, highly complex environment, goal steps performs best. Larger goal steps such as 6 lead to reduced accuracy in sub-goal prediction, the rapidly changing environment introduces noise and mismatches between predicted and actual teammate behaviors.
PP
LBF
Overcooked
Step=1
61.4 + 0.8
0.145 + 0,009
0.42 + 0.04
Step=2
60.1 + 0.7
0.153 0.010
0.90 + 0.07
Step=3
59.6 0.7
0.137 0.009
0.98 + 0.15
Step=6
63.2 + 1.1
0.123 0.008
0.30 + 0.03
Table 2, Average Return Comparison between different goal steps
D. Pseudocode of Algorithm
Algorithm 1 demonstrates trajectory mirroring strategy for pre-processing the offline dataset. This process iterates through all trajectories, generating new trajectories and adding them to the new dataset.
Algorithm 1 Offline Data Pre-processing
Input: Pre-collected dataset D (71, 72.... ...,, }
Output: Offline dataset '
Initialize D' x-0 (
for j = 1 to m do
for = 1 to n do
Apply trajectory mirroring strategy to generate trajectory Ty:
Tj. (T,
Add T to D': D' D' (Ta))
end for end for
13

---

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
Algorithm demonstrates the offline training process of TAGET. It involves sampling trajectory batches from the offline dataset, computing team context and TA-RTG, generating TA-Goal and action predictions, and optimizing both high-level and low-level objectives using gradient descent. The training is repeated until the high-level and low-level losses converge.
Algorithm 2 Offline training of TAGET
Require: Offline dataset D repeat
Sample a batch of trajectories B _ D for each trajectory
do
Randomly select sub-sequence of length 3K Leigh 0, LpT f- 0
for each time step t in sub-sequence do
Compute (/z,, 22 = fö (o/). Sample 2 ~ ((H2, Tz,)
Compute (Uh, 0h.,) = fo(oi, 0, Sample hi ~ N Nunn Oh.)
Compute TA-RTG: Rg = Ry(h))
Compute TA-Goal: G, = Ge R,)
Predict action aj using the low-level network
Compute I. Reg- LATG, LT A-A-Go Lor using Eq.6, Eq.7, Eq.8, 10. Accumulate high-level loss HHghh-ovd using Eq. 9
Accumulate low-level loss Lpr using Eq. 10 end for Update parameters vb, sp 0, 0, using gradient descent end for until convergence of hige-level loss Leigh-eeel and low-level loss LIT
Algorithm 3 illustrates the online testing process of TAGET. It updates the sequence iteratively using TA-Goal, observations, and actions. Based on the updated sequence, the low-level network generates action decisions, enabling adaptive and context-aware decision-making throughout the testing phase.
Algorithm 3 Online testing of TAGET
Require: parameters v), $ Initialize sequence $ +- for I = l..... T do
Keep the length of the sequence S as 3K
Compute (Hz,, o2,) = fä (of) Sample Team Context 2f ((u, xz.)
Compute TA-RTG: R, (2))
Compute TA-Goal: G. Gg Ge R.) Add (G., }f} into sequence
Generate the action at through low-level network using S
Add { into sequence S
end for
E. Network Architecture and Implementation Details
We illustrate the overall network structure of our method in Figure 7. Our model adopts the Decision Transformer (DT) backbone, with the following configurations: embedding dimension of 64, context window length K 30, 2 transformer layers with I attention head each, ReLU activation, and dropout rate of 0.3. The network is optimized using Adam with learning rate of 0.01, batch size of 2048, and a weight decay of 0.0001. There are several task-specific coefficients to balance different learning objectives in our training loss. Specifically, we set the weighting parameters as follows: a = 0.0001, 100, 7 100, and a 0.001
14

---

Ad Hoc Teamwork via Offline Goal-Based Decision Transformers
Le
LeakyReLU
LeakyReL.U
LeakyReLU
FC, X Izl Units
FC, 2 X Izl Units
FC, 1 Units
FC, IGI Units
LeakyRel 1
LeakyReLUU
LayerNorm
BatchNorm
BatchNorm
hidden=64,
BatchNorm
LeakyReLU
head=4
FC, 512 Units
FC, 256 Units
Multi-head Cross-Attention
FC, 512 Units
LeakyRoLU
BatchNorm
BatchNorm
LayerNorm
BatchNorm
LeakyReL.U
FC, 512 Units
FC, 64 Units
FC, 512 Units
FC, 512 Units
(of) Proxy Eneoder
(of, o
(h+)
(he, Rs)
Team Context Encoer
TA-RTG Predictor
TA-Goal Predictor
Figure 7. Architectural details of TAGET
15


Reviews:
[{"review_id": "rjjSKyoGl9", "reviewer": "Reviewer_cnPV", "summary": "This paper addresses ad hoc teamwork in the offline setting. It proposes a method called TAGET, which is based off the decision-transformer architecture and learns from a dataset of offline cooperative multi-agent interactions. It has a couple of main ideas. First, trajectory mirroring: where one agent is sampled to be the ego agent, and the remainder are teammates to be modeled. Second, agent modeling - where the teammate returns to go and a latent variable corresponding to teammate subgoals is modeled. TAGET is evaluated on Predator Prey, Level-Based-Foraging, and Overcooked, and compared against the Decision-Transformer, Prompt-DT, offline ODITS, and offline LIAM. Overall, it improves against baselines on all tasks, in terms of generalization to unseen teammates.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind suggestions and helpful feedback!\n\n**Q1: Concerns about teammate diversity.**\n\nA1.  We apologize for the confusion about the teammate generation. First and foremost, Soft Value Diversity (SVD) is not a MARL algorithm but a framework specifically designed to generate diverse policies. SVD maximizes the differences in value estimates across observation-action pairs between different teams [1]. This design ensures diversity between training and testing policy sets. As a result, we did not conduct separate experiments to demonstrate the diversity of teammate sets, since this diversity is inherent to the SVD framework. \nTo assuage your concerns, we've added the **cross-play experiments** between different populations, demonstrating significantly lower cooperation efficiency between populations compared to within-population cooperation, empirically validating that the policies represent truly distinct strategies. In cross-population cooperation experiments, each population sequentially designates one agent as the ego agent to collaborate with members from other populations. The matrix notation (Row1, Col3) specifically denotes an experimental configuration where Population 3's designated ego agent interacts with teammates from Population 1, establishing a systematic evaluation framework for inter-population coordination capabilities. As shown in the cross-play matrix(https://anonymous.4open.science/r/materials-E410/cross-play-matrix.md), in each row of the matrix, the diagonal positions behave best, which means that **strategies between populations don't work well in coordinating**, validating the **diversity** of our test teammate sets.\n\n**Q2. Concerns about the baseline and oracle upper bounds.**\n\nA2. Thanks for your raising the issue about baseline selection. As there are no existing methods specifically designed for offline AHT, our paper addresses this gap by proposing the first dedicated solution. To provide a reasonable comparison, we adapted methods from offline RL and online AHT.  We’ve now added MADT [2] as a baseline, as you suggested. Our results show that **TAGET outperforms MADT**(https://anonymous.4open.science/api/repo/materials-E410/file/comparison.pdf). Due to time constraints, we only tested on the teammate test set of size 4, and we will fill in the experiments subsequently. It's important to note that standard MARL methods, including MADT, don’t fully align with the offline AHT setting, as they typically assume teammates of the same type. AHT, however, involves an ego agent adapting to diverse teammates. Our new experiments with MADT confirm this mismatch, highlighting the key strength of TAGET in handling teammate uncertainty.\n\n**Oracle Upper Bounds**: We have included oracle upper bounds represented by the average self-play scores of test teammates, which are shown in the cross-play matrix. **TAGET achieves 86.0%, 77.2%, and 95.1% of the oracle upper bounds** in PP, LBF, and Overcooked environments respectively, which demonstrates strong performance considering the challenging nature of AHT with unknown teammates.\n\n**Q3. Questions about experimental analysis.**\n\nA3.Our offline interaction dataset includes ​2,300,000 transitions in PP, 1,430,000 transitions in LBF, and 5,500,000 transitions in Overcooked. While we have not yet conducted a systematic ablation on dataset size variation, our experiments with ​data augmentation ablations have revealed that our method still maintains reasonable performance under reduced data diversity. We will add related ablation experiments in the future. It's worth mentioning that our trajectory mirroring strategy mitigates the negative effect when the diversity of offline datasets is low (shown in Fig. 5).\n\n* [1] Ding, Hao, et al. \"Coordination Scheme Probing for Generalizable Multi-Agent Reinforcement Learning.\" (2023).\n* [2] Meng, Linghui, et al. \"Offline pre-trained multi-agent decision transformer.\" Machine Intelligence Research 20.2 (2023): 233-248.", "claims_and_evidence": "The train and test set of teammates were generated using the same MARL algorithm (Soft Value Diversity). The authors claim, “Each checkpoint represents a distinct joint strategy, capturing a diverse range of behaviors and interaction patterns.” (L282) However, no empirical evidence is presented to back up this claim about a crucial aspect of the experimental design. I would like to see cross-play matrices demonstrating how well different checkpoints can coordinate with each other, focusing especially on whether training teammates can coordinate with testing teammates.\n\nTypically, AHT papers will also generate test-sets using hand-coded heuristic policies, which should be easy to code up / get from other papers for LBF, Overcooked, and Predator-Prey. The analysis would also be stronger if the authors tested against heuristic policies that cover a wide range of teammate behavior.", "methods_and_evaluation_criteria": "Evaluation method follows standard protocol in AHT.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I have concerns about the strength of the baselines. While TAGET outperforms baselines in all tasks (Fig. 3), many of the proposed baselines are simply not designed for offline ad hoc teamwork, and look like they are entirely failing to learn in most tasks.\n\n- Decision Transformer - offline single-agent RL\n    \n- Prompt DT - offline single-agent RL\n    \n- ODITS - online AHT\n    \n- LIAM - online AHT\n    \nI do not think it’s surprising that LIAM and ODITS fails to learn in the offline setting, and that the DT and Prompt DT fail to generalize to the multi-agent setting/unseen teammates (a characteristic of the AHT problem).\n\nWhile I understand that the authors have proposed the first (to my knowledge) offline AHT method, and therefore there is no other offline AHT baseline, I wonder if they can compare to a method that is closer to the intended setting? For example, why not compare against MADT (Meng et al. 2021), which is an offline multi-agent decision transformer?\n\nI would also be better able to understand how well TAGET is performing if the authors could add some oracle upper bounds to the results plots. For example, perhaps the average self-play score of the test teammates?", "relation_to_broader_scientific_literature": "The paper is mostly well-situated in the literature, and cites relevant papers that i am aware of. However, I did notice that the authors cited Durugkar et al 2020 as an AHT paper, when it is not. Durugkar et al. 2020 is closer to a decentralized MARL paper.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "- Strengths\n    \n    - This paper fills a hole in the AHT literature\n        \n    - The paper has been well-contextualized w.r.t. related work.\n        \n    - The ablation study shows that each component of TAGET contributes positively to the final performance\n        \n- Weaknesses\n    \n    - Hard to understand how strong the method is due to lack of good baselines. The baselines fail because they were not designed for the problem setting, and no oracle upper bounds are presented to contextualize perforamnce.\n        \n    - Lack of information about the diversity of the traning and test teammates\n        \n    - Overall, the experimental analysis is a little sparse. Some additional empirical questions could be considered, which I have put in the “Questions” section.", "other_comments_or_suggestions": "Missing word at Line 33, right column", "questions_for_authors": "- How many transitions are included in the offline interaction dataset for each task?\n- How does TAGET’s performance change as the size of the offline dataset changes?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sKqXIAdXtA", "reviewer": "Reviewer_TMEP", "summary": "This paper introduces **TAGET (Teammate-Aware Goal driven hiErarchical Decision Transformers)**, a novel framework for offline ad hoc teamwork (AHT). The AHT problem requires an agent (ego agent) to collaborate with unknown teammates without prior coordination. Unlike existing approaches that rely on online reinforcement learning (RL) and direct environmental interactions, TAGET enables offline learning from pre-collected multi-agent datasets, addressing critical challenges such as limited data availability, partial observability, and dynamic teammate adaptation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your constructive feedback.\n\n**Q1. Concerns about the trajectory mirroring strategy.**\n\nA1.  Our trajectory mirroring strategy is a data preprocessing method specifically designed for the AHT, not a data augmentation method. It works by sequentially designating different agents as the ego agent in a given trajectory. As a result, the combinations of the remaining teammates change, increasing the diversity of teammate sets. This approach not only enhances data efficiency but also provides a wider variety of teammate combinations for training. Through ablation experiments, we have demonstrated that this preprocessing method improves algorithm performance without incurring additional computational overhead. If you believe that other augmentation methods could provide meaningful comparisons, we are open to considering further experiments, but our current focus is to showcase the effectiveness of this method in AHT.\n\n**Q2. Concerns about the teammate's diversity and teammate heterogeneity.**\n\nA2. Thank you for your insightful feedback. To clarify, the robustness of TAGET is focused on its generalization to unseen teammates, especially in terms of adapting to diverse and unknown behaviors. In our experiments (Fig. 3), we evaluate performance using diverse sets of previously unseen teammates to test this generalization. We have added the cross-play experiments between different populations, the cross-play matrix can be seen in https://anonymous.4open.science/r/materials-E410/cross-play-matrix.md.\n\n\n**Q3. Concerns about baseline**\n\nA3. Thanks for this insightful feedback! We agree that including classic offline RL methods like CQL [1] and QT [2] would provide a more comprehensive benchmark suite. We’ve now added CQL as a baseline. **TAGET outperfoms CQL** across all environments, which can be found in https://anonymous.4open.science/api/repo/materials-E410/file/comparison.pdf. \n\n**Q4. Concerns about evaluation metrics.**\n\nA4. There is currently no specific metric for measuring teammate adaptability.  However, achieving a higher return generally indicates better cooperation with teammates in a game. To address this concern, we could consider comparing our agent’s adaptability with the Oracle Upper Bounds suggested by Reviewer cnPV, which would provide a meaningful comparison of our approach in terms of teammate adaptation.\n\n**Q5. Concerns about the novelty.**\n\nA5. Thank you for your comment. While we build on existing research such as Decision Transformer and return-to-go estimation, previous methods are not directly applicable to the novel problem of offline AHT. To address this, we introduce TA-RTG and TA-Goal to handle the unique challenges of offline AHT, where agents must infer and adapt to the behaviors of unseen teammates. This teammate-aware approach enables better generalization in AHT settings, differentiating our method from prior works. We believe this extension of offline RL into the AHT domain is a key innovation.\n\n**Q6. Concerns about the direct ablation for TA-RTG.**\n\nA6. Thank you for your suggestion. We have added direct ablation experiments for TA-RTG, and the results can be found in https://anonymous.4open.science/api/repo/materials-E410/file/ablation_new.pdf. The experimental results show that **TA-RTG outperforms traditional RTG**, as it is able to capture changes in teammates' strategies and adapt to unknown teammates.\n\n**Q7. Confusing figures and lack of detailed descriptions**\n\nA7. Thank you for your valuable feedback. We acknowledge that some figures, such as Figure 3, could benefit from clearer explanations. We will make sure to correct these issues in future revisions.\n\n**Q8. Concerns about the supplementary material.**\n\nA8. Thanks for your useful suggestion.  We commit to including these additional experiment details, such as detailed hyperparameter settings, neural network architecture, the method for computing standard deviation and the computational time of each algorithm, in the Supplementary Material of the revised version. Furthermore, we will also open-source our code.\n\n* [1] Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33, 1179-1191.\n* [2] Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., ... & Levine, S. (2018, October). Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning (pp. 651-673). PMLR.", "claims_and_evidence": "1. The trajectory mirroring strategy is claimed to enhance data efficiency, and ablation results (Figure 5) support this. However, no alternative data augmentation methods are tested, making it unclear if mirroring is the best approach. Testing other augmentation techniques (e.g., data mixing or adversarial augmentation) would provide stronger evidence.\n\n\n2. The paper assumes TAGET is robust to diverse teammate behaviors, but no systematic study on the impact of teammate diversity is provided. Evaluating performance across different levels of teammate heterogeneity would make the generalization claim more convincing.", "methods_and_evaluation_criteria": "1. The hierarchical TAGET framework aligns well with the offline ad hoc teamwork (AHT) problem, effectively addressing data scarcity, partial observability, and dynamic teammate adaptation. The trajectory mirroring strategy is a reasonable approach to improve dataset efficiency.\n\n2. The evaluation on Predator-Prey, Level-Based Foraging, and Overcooked is appropriate for testing multi-agent coordination and adaptability. \n\n3. Baseline comparisons include DT, Prompt-DT, ODITS-off, and LIAM-off, which are relevant, but classic offline RL methods like CQL [1] or QT [2] should be included for a more comprehensive benchmark suite.\n\n4. Evaluation metrics focus on return scores, which effectively measure task performance. However, no explicit measure of adaptability to new teammates over time is provided. A metric tracking adaptation efficiency would strengthen the evaluation.\n\n5. Authors did not explicitly claim how to compute standard errors with repeat experiments, which makes it hard to tell whether the improvements are meaningful. Computational time of each algorithm should also be listed for fair comparison. \n\nOverall, the methods are well-designed, but the evaluation could be strengthened with real-world benchmarks, additional baselines, and statistical validation.\n\n- [1] Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33, 1179-1191.\n\n- [2] Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., ... & Levine, S. (2018, October). Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning (pp. 651-673). PMLR.", "theoretical_claims": "There is no novel theoretical claim in this paper. All mathematical formulations to the problem are well-defined.", "experimental_designs_or_analyses": "The detailed experimental designs are not clarified, such as parameter selection, model architecture, encoder/decoder selection. Authors should include the main algorithm in the main paper, while putting all the substantial tricks into the appendix.", "relation_to_broader_scientific_literature": "This paper introduces TAGET, addressing key limitations of prior ad hoc teamwork (AHT) methods in multi-agent reinforcement learning (MARL), particularly in offline settings. Previous online AHT methods rely heavily on environment interactions, limiting generalization and adaptability to unknown teammates. TAGET overcomes these issues by learning teammate-aware goal-driven policies from offline datasets.\n\nTraditional online AHT approaches [1][2] require continuous adaptation to teammates through reinforcement learning (RL), which is infeasible without environmental access. TAGET builds on offline RL advancements (e.g., Decision Transformer [3], Conservative Q-Learning [4]), reformulating AHT as a sequence modeling problem while introducing TA-Goal and TA-RTG for better teammate modeling.\nUnlike prior offline MARL works [5][6] that learn joint optimal policies, TAGET focuses on real-time adaptability to changing teammates. The trajectory mirroring strategy further differentiates TAGET by enhancing dataset efficiency, addressing data limitations in offline settings.\nAdditionally, prior teammate/opponent modeling methods [7][8] rely on explicit policy inference, which struggles in offline settings due to missing exploration opportunities. TAGET circumvents this by predicting future global states (TA-Goal) rather than teammates' policies, making it more robust to unseen agents.\n\nBy integrating ideas from offline RL, sequence modeling, and multi-agent adaptation, TAGET advances offline AHT research and sets a foundation for applying decision transformers in multi-agent collaboration without online interactions.\n\n- [1] Barrett, S., & Stone, P. (2015, February). Cooperating with unknown teammates in complex domains: A robot soccer case study of ad hoc teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 29, No. 1).\n\n- [2] Rahman, M. A., Hopner, N., Christianos, F., & Albrecht, S. V. (2021, July). Towards open ad hoc teamwork using graph-based policy learning. In International conference on machine learning (pp. 8776-8786). PMLR.\n\n- [3] Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., ... & Mordatch, I. (2021). Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 15084-15097.\n\n- [4] Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33, 1179-1191.\n\n- [5] Tseng, W. C., Wang, T. H. J., Lin, Y. C., & Isola, P. (2022). Offline multi-agent reinforcement learning with knowledge distillation. Advances in Neural Information Processing Systems, 35, 226-237.\n\n- [6] Jiang, J., & Lu, Z. (2023). Offline Decentralized Multi-Agent Reinforcement Learning. In ECAI (pp. 1148-1155).\n\n- [7] Gu, P., Zhao, M., Hao, J., & An, B. (2021). Online ad hoc teamwork under partial observability. In International conference on learning representations.\n\n- [8] Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., & Hofmann, K. (2021). Deep Interactive Bayesian Reinforcement Learning via Meta-Learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (pp. 1712-1714).", "essential_references_not_discussed": "The references in this paper are sufficient.", "other_strengths_and_weaknesses": "## Strengths: \n1. Novel Offline Ad Hoc Teamwork Framework – The paper introduces TAGET, a hierarchical decision-transformer-based approach to offline AHT, addressing critical challenges such as limited data, partial observability, and teammate adaptation.\n\n2. Effective Use of Goal-Based Adaptation – The introduction of Teammate-Aware Goal (TA-Goal) and Teammate-Aware Return-to-Go (TA-RTG) is a significant innovation that helps the ego agent dynamically adapt to unseen teammates without relying on explicit online interactions.\n\n3. Data Efficiency via Trajectory Mirroring – The trajectory mirroring strategy improves the use of offline datasets by allowing each agent to be treated as an ego agent in different contexts, effectively increasing the amount of usable training data.\n\n4. Addresses a Practical Limitation of Online AHT – Most prior ad hoc teamwork approaches require online interactions, which are impractical in real-world applications. TAGET’s offline approach makes AHT more feasible for real-world deployment.\n\n\n## Weaknesses:\n\n1. Limited Novelty in Offline RL Techniques – While TAGET innovates in offline AHT, its use of hierarchical decision transformers and return-to-go estimation builds on existing offline RL techniques (e.g., Decision Transformer, CQL) rather than introducing fundamentally new RL methods. A clearer differentiation from prior offline RL works would strengthen its contribution.\n\n2. No Direct Ablation for TA-RTG Effectiveness – The paper claims that Teammate-Aware Return-to-Go (TA-RTG) is superior to standard RTG, but no explicit ablation study compares TAGET with standard return-to-go instead of TA-RTG. This omission makes it unclear whether TA-RTG is necessary for TAGET’s success.\n\n3. Lack of Analysis on Teammate Diversity Impact – The generalization of TAGET to diverse teammates is assumed, but the paper does not systematically analyze how teammate heterogeneity affects performance. Testing against teammates with drastically different strategies or levels of expertise would provide a stronger understanding of its adaptability.\n\n4. Confusing Figures and Lack of Detailed Descriptions – Some figures lack clear explanations (e.g., Figure 3 performance curves do not explicitly mention how the 95% confidence intervals are computed, and no verbal explanations of those curves are provided). The captions should give more precise descriptions of experimental setups and key observations.", "other_comments_or_suggestions": "Please refer to the “Other Strengths And Weaknesses” section.", "questions_for_authors": "Please refer to the “Other Strengths And Weaknesses” section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KXEEoIjfDU", "reviewer": "Reviewer_85vr", "summary": "The paper presents TAGET, a novel hierarchical framework designed to address the challenges of ad hoc teamwork in settings where only offline multi-agent interaction data is available. Traditional ad hoc teamwork approaches rely on real-time interactions, but TAGET circumvents the need for costly online simulations by learning robust policies from pre-collected datasets. At its core, TAGET integrates a high-level module that dynamically predicts a teammate-aware sub-goal, leveraging a new teammate-aware return-to-go signal and dual state encoders to capture the evolving team contex, with a low-level, goal-conditioned Decision Transformer that generates adaptive actions. The framework also incorporates a trajectory mirroring strategy that repurposes each multi-agent trajectory by cyclically treating each agent as the ego agent, thereby enhancing data efficiency and capturing diverse decision-making perspectives. Extensive experiments conducted in environments such as Predator-Prey, Level-Based Foraging, and Overcooked demonstrate that TAGET significantly outperforms existing baselines in terms of overall return, stability, and generalization to unseen teammate behaviors, marking a substantial advancement in offline multi-agent reinforcement learning.\n\n## Update After Rebuttal\nThe authors have addressed part of my concerns. However, I still have some concerns about its potential effect: I have a different view of the feasibility of transformer based method is the route towards the final goal of juggling realistic deployment and offline training. The features of decentralized execution and parallel decision making is crucial to my best experience.\n\nFor this reason, I raised my score from 1 to 2.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions.\n\n**Q1. Concerns on the difficulty of collecting offline data and the applications of methods**\n\nA1.We apologize for the misunderstanding. Our claim that offline data is \"relatively easy to collect\" refers to its cost and safety advantages over online training, not to ignore practical challenges. We use simulator-collected data to avoid exploration risks (e.g., collisions in autonomous driving) and improve data efficiency with trajectory mirroring. For real-world data, we address challenges like long-horizon rewards using multi-sensor fusion [1] and hierarchical reward decomposition [2]. While data collection is challenging, offline RL remains valuable for providing a safe, low-cost baseline and enabling future online refinement.\n\n**Q2. Concerns on empirical validation and intuitive explanation of TA-Goal's advantages over RTG**\n\nA2.Towards empirical validation, our experiments (Fig. 3) show that TAGET (TA-Goal-based) outperforms DT (RTG-based). Intuitively,  RTG is a long-term scalar target that doesn't adapt to changing teammate behaviors (e.g., obstacle avoidance). TA-Goal, on the other hand, dynamically adjusts based on teammate intent, enabling adaptive actions (e.g., flanking maneuvers) as shown in Fig. 4. https://anonymous.4open.science/api/repo/materials-E410/file/ablation_new.pdf\n\n**Q3. Why is the team context modeled as a multivariate random variable and selected Gaussian distribution**\n\nA3. The team context is modeled as a multivariate random variable to capture uncertainties in teammates' policies, enhancing decision robustness. The Gaussian distribution is chosen due to: (1) mathematical and computational advantages; (2) uncertainty modeling capability (3) research alignment—Gaussians are standard in deep probabilistic models (e.g., VAEs [3][4] ). Many existing works [5] adopt this approach, and we will clarify this in our paper more clearly.\n\n**Q4. The evidence to support this modeling strategy.**\n\nA4.  In partially observable settings, the ego agent infers the global team context $h_i$ from local observations $o_i$. KL-divergence regularization (Eq. 6) aligns $z_i$ with $h_i$, enabling latent team dynamics capture without global info. This method has been validated in previous works [5][6] for improved generalization. If additional evidence is needed, we will include it in the revised manuscript.\n\n**Q5. Eq.(6). The first KL-divergence is a reverse KL-divergence.**\n\nA5. The first term in Eq. (6) is a reverse KL-divergence, following the beta-VAE approach [4], which encourages the latent representation to match a standard normal prior, preventing overfitting and improving generalization. The second term uses forward KL to align the proxy and team context representations.\n\n**Q6. Eq.(8) explanation about why the loss is formulated as binary cross entropy**\n\nA6. Thanks for the useful feedback. The groundtruth goal $\\hat{G}$ is constructed by concatenating discretized observations (e.g., grid-world positions, task states) and one-hot encoding the global state $s_{t+k}$, where each dimension represents a binary indicator. We use BCE loss per dimension to handle binary classification, which aligns with the discrete nature of our environments. Using MSE for continuous regression would be suboptimal, as the goal representation is categorical and requires probabilistic modeling of binary outcomes. We will also fix the inconsistency between $\\hat{G}$ and $G$.\n\n**Q7. Concerns about the applicability of methods in Methods And Evaluation Criteria.**\n\nA7. While collecting offline datasets is challenging, it is manageable in AHT. The core challenge is enabling the ego agent to handle unknown teammates, which previous works have addressed in two ways: achieving generalization through algorithms and constructing diverse sets of teammates. Even with algorithmic generalization, a diverse offline dataset is still needed for better performance, which doesn’t mean it’s not worth exploring. The offline method is not meant to replace online interactions but is motivated by safety and cost considerations, and can be further optimized when combined with online methods.\n\n**Q8. The dataset generation in Experimental Designs Or Analyses.**\n\nA8. Thank you for your feedback. For more details, please refer to our response to Reviewer cnPV, A1.\n\n* [1] Li, et al. \"Dynamic Semantic SLAM Based on Panoramic Camera and LiDAR Fusion for Autonomous Driving.\" IEEE TAITS 2025.\n* [2] Matsuda, et al. \"Hierarchical Reward Model of Deep Reinforcement Learning for Enhancing Cooperative Behavior in Automated Driving.\" JACII 2024.\n* [3] Kingma, et al. \"Auto-encoding variational bayes.\" 20 Dec. 2013.\n* [4] Higgins, et al. \"beta-vae: Learning basic visual concepts with a constrained variational framework.\" ICLR. 2017.\n* [5] Gu, et al. \"Online ad hoc teamwork under partial observability.\" ICLR. 2021.\n* [6] Xie, et al. \"Future-conditioned unsupervised pretraining for decision transformer.\" ICML, 2023.", "claims_and_evidence": "### Supported Claims:\n1. The claims of better performance than other offline methods are supported by strong experimental evidence.\n\n### Problematic Claims:\n1. The authors claim that \"TA-Goal captures the dynamics of teammate behaviors, providing more robust and adaptive guidance than RTG in complex multi-agent environments.\" However, I didn't see the evidence to support this claim and it is non-trivial to understand this claim by common sense.\n2. The authors claim that \"To model this, we represent the team context in a stochastic embedding space $\\mathcal{H}$, where it is encoded as a latent probabilistic variable $h_{i}$ drawn from a multivariate Gaussian distribution.\" However, I didn't see any evidence to support this viewpoint. Why is the team context modeled as a multivariate random variable? Furthermore, why is it Gaussian distribution?\n3. The authors claim that \"Similarly, $z_{i}$, as an approximation of $h_{i}$, is encoded into a stochastic embedding space, where it is represented as a latent probabilistic variable drawn from a multivariate Gaussian distribution.\" Can the authors provide any evidence to support this modeling strategy?\n4. It seems like in Eq. (6), the first KL-divergence is a reverse KL-divergence. Can the authors provide more insight into this design?\n5. In Eq. (8), the groundtruth goal is constructed by the concatenation of team observations. As a result, it is not a binary variable. Can the authors give an explanation about why the loss in Eq. (8) is formulated as binary cross entropy?", "methods_and_evaluation_criteria": "### Evaluation Criteria:\nThe evaluation criateria used in this paper is acceptable to the AHT problem.\n\n### Methods:\nThe authors claim that the offline data is easy to collect and propose to use decision transformer as an offline RL method to address AHT. However, I have some concerns on the difficulty of collecting offline data, which may affect the reasonableness of the proposed method. First, if we consider collecting offline samples from simulators, why not train the ego agent through interaction with environment? On the other hand, if we consider collecting data from real-world scenarios, what is the way to capture long-horizon rewards? It is not easy to track a team of agents constantly in a long episode, especially with accurate records of data. For example, the traffic data captured by CCTV cameras is with lower frequency than the control rate by an agent. Let's say that the distance between two cameras is 1 km, and there may happen multiple decision making steps. On the other hand, if each agent is able to collect data by the camera installed on its body, the alignment between timestamps could extremely difficult. In conclusion, offline data collection is not as easy as what the authors claimed.", "theoretical_claims": "This paper has no theoretical claims.", "experimental_designs_or_analyses": "In general, the experimental design is following the conventional experimental setup of AHT.\n\nHowever, how the teammate sets are generated may miss some information. The authors have mentioned that they have trained four distinct populations of MARL policies. What are the MARL algorithms used? How are the four distinct populations formed?", "relation_to_broader_scientific_literature": "In prior work, the ego agent (learner) was not trained by the offline dataset. As I mentioned above, this could be due to the validity and difficulty of collecting offline data. In turn, this could lead to a question that if offline RL paradigm does fit in the aim of the AHT problem.\n\nThe major issue of previous methods is how to construct diverse teammate sets (models). With the same techniques of establishing diverse teammate sets, I suspect if the offline method proposed in this paper is necessary. The direct evidence is that in experiments it seems like the proposed offline method still requires diverse teammate sets, like what is required in AHT with online interaction.\n\nIf the problem of offline data collection is dismissed, the major contribution of this paper is proposing a training paradigm which does not need online interactions. The remaining questions is that if the offline collected data without interactions can actually have good quality as the data collected with interactions.", "essential_references_not_discussed": "Another recently published paper in ICML 2024 [1] which studies online interactions for AHT could be cited in related works to enhance the understanding of the main contribution of this paper.\n\n[1] Wang J, Li Y, Zhang Y, Pan W, Kaski S. Open Ad Hoc Teamwork with Cooperative Game Theory. In International Conference on Machine Learning 2024 Jul 8 (pp. 50902-50930). PMLR.", "other_strengths_and_weaknesses": "1. The methods are well described, however, the reason behind part of the procedure is not explained.\n2. The proposed algorithm is novel with an idea to use offline collected data, though the applicability to real-world problems is still under concern.", "other_comments_or_suggestions": "In Eq. (8), $\\hat{G}$ and $G$ interchangebly represent the groundtruth in multiple places. I believe this could be a typo.", "questions_for_authors": "The authors are required to make clarification for the problematic claims listed in **Claims And Evidence**. In addition, the authors are requested to answer the concerns about the applicability of methods in **Methods And Evaluation Criteria** and the dataset generation in **Experimental Designs Or Analyses**.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "7zbCN4Qjzs", "reviewer": "Reviewer_nbn1", "summary": "The paper introduces TAGET, a hierarchical framework for offline ad hoc teamwork (AHT) that leverages teammate-aware goal-conditioned Decision Transformers. The core idea is to dynamically predict teammate-aware return-to-go (TA-RTG) and sub-goals (TA-Goals) to enable real-time adaptation to unknown teammates without online interactions. Key innovations include a trajectory mirroring strategy for data efficiency, dual state encoders with regularization for partial observability, and a hierarchical structure combining high-level goal prediction with low-level action generation. Experiments in three environments demonstrate TAGET’s superiority over baselines, with an average performance improvement of 48.08%. Ablation studies validate the contributions of individual components, such as TA-Goal decoding and data preprocessing.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.  Questions in Claims And Evidence.**\n\nA1.  We apologize for any confusion. Firstly, we perform implicit modeling of teammates to capture their changes and update our TA-goal, so there is no explicit metric to measure the accuracy of teammate modeling. As for the generalization of the policy, we demonstrate this in our experiments. Specifically, we test the policy with teammates that have never been seen during training. The fact that our method achieves higher scores when cooperating with these unknown teammates, compared to other baseline methods, serves as evidence of the generalization ability of our proposed approach.\n\n**Q2. Questions in Methods And Evaluation Criteria.**\n\nA2. Regarding your concerns about data augmentation, more details can be found in our response to Reviewer TMEP, Response A1. Regarding the accuracy of TA-Goal, during training, we use the real state after $k$ steps as the ground truth, which is a common practice in many works. During testing, since there are no trajectories for reference, we cannot directly measure the accuracy of TA-goal's prediction. Our goal during testing is not to achieve the absolute accuracy of the TA-Goal but to assess its ability to generalize to unknown teammates in AHT.\n\n**Q3. Questions in Experimental Designs Or Analyses.**\n\nA3. The evaluation of teammate modeling accuracy and policy generalization is answered in A1. Since TA-RTG is a predicted value rather than a fixed input, it cannot be ablated in the traditional sense. Besides, we agree that further testing in more complex environments, such as SMAC, is necessary, and we plan to conduct such tests.\n\n**Q4. Questions in Essential References Not Discussed.**\n\nA4. Thank you for pointing out these relevant works. We acknowledge the importance of the two and will include a discussion of them in the revised manuscript to better situate our contributions within the existing literature.\n\n**Q5. Concern about innovation.**\n\nA5. Thank you for the comment. Our method is not designed to simply combine existing techniques, but is driven by a novel problem setting - offline AHT. This new setting raises unique challenges, which motivate our methodological choices. Rather than applying existing tools directly, we analyze why previous approaches like DT are insufficient and propose tailored solutions. \n\nOur trajectory mirroring strategy is a data preprocessing method specifically designed for the AHT setting. It improves data efficiency and teammate diversity without introducing extra computational cost. After preprocessing, we learn policies from offline data. While we adopt the Decision Transformer (DT) architecture, we find that standard DT is not well-suited for our problem, as RTG cannot capture dynamic teammate behavior. To address this, we propose TA-RTG, a novel formulation that incorporates teammate-awareness. Furthermore, given the dynamic nature of the environment, we propose the novel concept of TA-Goal, a sub-goal representation predicted from TA-RTG, to guide the transformer’s decision-making in a hierarchical manner.\n\n**Q6. Concern about the method’s robustness when applied to low-diversity datasets.**\n\nA6. Thank you for raising this important point. While our method benefits from diverse offline data, we also aim to improve robustness through model design. Specifically, the proposed TA-RTG and TA-Goal mechanisms help the model infer and adapt to teammate behavior even under limited diversity. In our ablation studies, the variant without the trajectory mirroring strategy can be seen as a setting with reduced data diversity. We observe that the method still maintains reasonable performance under this condition, demonstrating a degree of robustness to limited diversity. Nonetheless, we acknowledge this as a valuable direction and plan to further explore robustness under low-diversity settings in future work.\n\n**Q7. Difference between TA-RTG and RTG.**\n\nA7.  The main difference between the two is that TA-RTG considers teammate behavior in a team-shared reward setting. TA-RTG incorporates how teammates' actions influence the collective reward, adapting the agent's strategy based on the team context. TA-RTG shifts from relying solely on the agent’s observation to considering the team context, allowing it to adapt to changes in teammate behavior (TA-RTG and RTG ablation: https://anonymous.4open.science/api/repo/materials-E410/file/ablation_new.pdf). \n\n**Q8. Concern about poor generalization.**\n\nA8. Thank you for highlighting this important point. Our method addresses the generalization issue through both data-level and model-level designs. On the data side, the trajectory mirroring strategy increases teammate diversity. On the model side, the proposed TA-RTG and TA-Goal explicitly encode teammate behavior and adapt decision-making accordingly. These components together improve the model’s ability to generalize to unseen teammates. We will clarify this contribution further in the revised manuscript.", "claims_and_evidence": "The authors raise issues such as poor generalization of the learned policy and challenges in accurately modeling teammates. However, the subsequent experimental section does not clearly demonstrate whether the proposed method effectively addresses these problems. For instance, how accurate is the modeling of teammates, and how well does the policy generalize?", "methods_and_evaluation_criteria": "The method and validation don’t quite make sense. In my past experience with offline reinforcement learning, data augmentation doesn’t provide significant improvements in offline scenarios. Moreover, the accuracy of Teammate-aware goal prediction is hard to guarantee, and no experimental validation is provided.", "theoretical_claims": "No, there are no proofs in this paper.", "experimental_designs_or_analyses": "The evaluation of teammate modeling accuracy, policy generalization, and the ablation of return in the return-to-go function is missing. Additionally, experiments involving humans as teammates are not included. The experimental scenarios are relatively simple, and further testing in more complex environments, such as SMAC [1], is needed.\n\n[1] Wang, Caroline, et al. \"N-Agent Ad Hoc Teamwork.\" arXiv preprint arXiv:2404.10740 (2024).", "relation_to_broader_scientific_literature": "This paper clearly situates TAGET within offline RL (e.g., Decision Transformer) and AHT (e.g., ODITS, LIAM), and highlights gaps in prior AHT methods (online dependency) and offline RL (lack of teammate adaptability).", "essential_references_not_discussed": "[1] **multi-agent decision transformer**: Meng, Linghui, et al. \"Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks.\" arXiv preprint arXiv:2112.02845 (2021).\n\n[2] **offline teammate modeling**: Zhu, Zhengbang, et al. \"Madiff: Offline multi-agent learning with diffusion models.\" arXiv preprint arXiv:2305.17330 (2023).", "other_strengths_and_weaknesses": "**Strengths**:\n\n1. The offline AHT problem addressed is novel.\n\n**Weaknesses**:\n\n1. The method lacks innovation. Techniques such as dataset augmentation, local observation-based global state prediction, and decision transformers are not new. The authors need to further clarify how these techniques are organically combined.\n\n2. The experimental section is insufficient, lacking evaluations of teammate modeling accuracy, policy generalization, and ablation studies on the return in the return-to-go function.\n\n3. The method still relies on algorithms that collect data driven by diversity. I’m curious about the method’s robustness when applied to low-diversity datasets.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. What is the difference between teammate-aware return-to-go and conventional return-to-go? (Has it shifted from observation to global state?)\n\n2. How does the method address the issue of “poor generalization of the learned policy” mentioned in the paper? If this isn’t addressed, it remains a point of concern.\n\n3. Can the trajectory mirroring strategy be understood as a form of data augmentation? How is it different from traditional data augmentation? How does this strategy work when the dataset’s diversity is low?\n\n4. What is the impact of dataset quality on performance? Does a dataset with low diversity greatly affect the performance of the method?\n\n5. Can you further clarify how the various components of the method are integrated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for learning an ad hoc teamwork policy offline. It does so by making modifications to the decision transformer approach as well as additional pre-processing of the offline data to enable easier identification of teammate goals and capabilities.
The proposed approach seems to be the first to show a viable offline approach for ad hoc teamwork. But I would still advise authors to ensure that they do not use maximalist language such as `DT is uniquely suited for offline AHT` unless the paper is proving this statement as part of its claims.
Decision: Accept (poster)