Towards Offline Opponent Modeling with In-context Learning
Authors: Yuheng Jing
Kai Li
Bingyun Liu
Yifan Zang
Haobo Fu
QIANG FU
Junliang Xing
Jian Cheng
Venue: ICLR 2024 poster (2024)
Keywords: Opponent Modeling
Offline
Transformer
Primary area: reinforcement learning

Abstract: Opponent modeling aims at learning the opponent's behaviors, goals, or beliefs to reduce the uncertainty of the competitive environment and assist decision-making. Existing work has mostly focused on learning opponent models online, which is impractical and inefficient in practical scenarios. To this end, we formalize an Offline Opponent Modeling (OOM) problem with the objective of utilizing pre-collected offline datasets to learn opponent models that characterize the opponent from the viewpoint of the controlled agent, which aids in adapting to the unknown fixed policies of the opponent. Drawing on the promises of the Transformers for decision-making, we introduce a general approach, Transformer Against Opponent (TAO), for OOM. Essentially, TAO tackles the problem by harnessing the full potential of the supervised pre-trained Transformers' in-context learning capabilities. The foundation of TAO lies in three stages: an innovative offline policy embedding learning stage, an offline opponent-aware response policy training stage, and a deployment stage for opponent adaptation with in-context learning. Theoretical analysis establishes TAO's equivalence to Bayesian posterior sampling in opponent modeling and guarantees TAO's convergence in opponent policy recognition. Extensive experiments and ablation studies on competitive environments with sparse and dense rewards demonstrate the impressive performance of TAO. Our approach manifests remarkable prowess for fast adaptation, especially in the face of unseen opponent policies, confirming its in-context learning potency.

Published as a conference paper at ICLR 2024
TOWARDS OFFLINE OPPONENT MODELING WITH IN-CONTEXT LEARNING
Yuheng Jing Kai Li!.2. Bingyun Liu Yifan Zang
Haobo Fu Qiang Fu Junliang Xing Jian Chengl.3.4.
f denotes corresponding authors
Institute of Automation, Chinese Academy of Sciences
2 School of Artificial Intelligence. University of Chinese Academy of Sciences
3 School of Future Technology, University of Chinese Academy of Sciences
AiRiA 5 *Tsinghua University Tencent Al Lab
(jingyuheng2022, kai. 1i, Liubingyun2022. zangyifan2019, ) @ia.ac.cm en, (haobofu, 1eenee]8tencent. com, jlxingetsinghua. edu, on
ABSTRACT
Opponent modeling aims at learning the opponent's behaviors, goals, or beliefs to reduce the uncertainty the competitive environment and assist decision-making. Existing work has mostly focused on learning opponent models online, which is impractical and inefficient in practical scenarios. To this end, we formalize an Offline Opponent Modeling (OOM) problem with the objective of utilizing pre- collected offfine datasets to learn opponent models that characterize the opponent from the viewpoint of the controlled agent, which aids in adapting to the known fixed policies of the opponent. Drawing on the promises the Transform- introduce general approach, Transformer gainst
ers for decision-make making, Opponent (TAO), for OOM. Essentially, TAO tackles the problem by harnessing the full potential of the supervised pre-trained Transformers' in-context learning capabilities. The foundation of TAO lies in three stages: an innovative offline pol- icy embedding learning stage, an offline opponent- aware response policy training stage, and deployment stage for opponent adaptation with in-context learning. Theoretical analysis establishes TAO's equivalence to Bayesian posterior sam- pling in opponent modeling and guarantees TAO's convergence in opponent policy recognition. Extensive experiments and ablation studies on competitive environ- ments with sparse and dense rewards demonstrate the impressive performance of TAO. Our approach manifests remarkable prowess for fast adaptation, especially in the face of unseen opponent policies, confirming its in-context learning potency.
INTRODUCTION
general autonomous agent must possess the ability to model complex behaviors, goals, or beliefs of other agents in the environment refine its decision-making, long-standing problem often referred to as opponent modeling in the adversarial domains (Albrecht & Stone 2018 Nashed & Zilberstein 2022; Yu et al., 2022). Nevertheless, most existing work adopts the setting of learning an opponent model online, which is prohibitively impractical and inefficient. In practical situations, is not always feasible engage with the competitive environment or opponent policies execute the online learning process. For instance, e-commerce platforms are constrained to modeling the behavior of specific users offline by passively accumulating their data (Li & Zhao 2021). Even online learning is available, the opponent may have switched the policy across episodes, possibly long before the current opponent's model learned. Then a multitude rollouts are necessary to acquire opponent model, which is exorbitantly inefficient (Albrecht & Stone 2018).
Given the multifaceted issues in learning the opponent model online, we contemplate extending the opponent modeling problem into the offline setting, concept refer as Offline Opponent Modeling (OOM), to make learning more accessible and enhance the sample efficiency of opponent modeling. The underlying insight is that acquiring replay data historical plays of certain quality

---

Published as a conference paper at ICLR 2024
Embed to Latent Space
Learn to Respond
Adapt with In-context Learning
unseen
seen
emb.
traj.
by Offline Traj.
action
traj.
by Interacting with Env.
action
Opponent Policy Encoder
Opponent
emb.
In-context Control Decoder
Opponent
cmb.
In-context Control Decoder
Policy Encoder
Policy Encoder
traj-
traj frag.
traj. frag.
traj. 1,+
Offline
Oflline
Opponent
traj.
Dataset
Dataset
Context Window
(a)
(b)
(c)
Figure 1: Procedures of TAO. (a) Offline stage 1: Embed the trajectories of different opponent poli- cies into a latent space through representation learning. (b) Offline stage 2: Learn to respond to different opponent policies based consecutive fragments sampled from their trajectories. (c) De- ployment stage: Deploy in new environment and adapt to unknown opponent policies through in-context learning. Superscript denotes the terms of the controlled agent, -1 denotes the terms of the opponent, and * denotes terms generated by the approximate best response policy.
generally straightforward (Levine et al. 2020). The acquired offline data can be used to learn the response policy against the fixed opponent policy found in offline data rather than undertaking thou- sands of rollouts learn through trial and error. Numerous real- applications echo this principle. For instance, mastering strategies to handle certain scenarios in Go games through practicing high- quality Go exercises; understanding the basketball playing styles and patterns of specific opponents by analyzing replay videos of their NBA performances (Daly-Grafsted & Bornn 2020)
Recent studies have shown that the Transformer (Vaswani al. 2017). supervised pre-trained for decision-making, possesses robust in-context learning ability in meta Reit--Reinforeemeet Learning (RL) (Lee et al. 2023). This ability is exactly what is demanded in opponent modeling. Such insight stems from the fact that when we deploy the opponent model into new environment, we expect to adapt to unknown opponent policies as swiftly as possible, ideally without gradient updates. We provide the related work opponent modeling, offline meta-RL, etc. in Appendix A
Embracing this idea, we introduce Transformer Against Opponents (TAO), general Transformer- based approach dedicated to resolving the defined OOM problem. The procedures of TAA unfolds in three stages (see Fig. I): 1) We generatively and discriminatively embed opponent trajectories using an Opponent Policy Encoder (OPE) to acquire opponent policy representation that is beneficial for the downstream learning: 2) Upon the characterization of opponent policies offered by OPE, adopt in-context learning-bused supervised pre-training for In-context Control Decoder (ICD) to learn to respond reasonably according to opponent trajectories; 3) We deploy in new environment inhabited by unknown opponent policies and adapt with in-context learning with the help of an Opponent Context Window (OCW), which continuously collects opponent trajectories.
Theoretically, we prove that TAO equivalent Bayesian posterior sampling in opponent mod- eling. Further analysis demonstrates the convergence of TAO for opponent policy recognition. We conduct experiments in both sparse reward and dense reward environment, comparing TAO with series of baselines for opponent modeling and offline meta-RI. Our approach demonstrates out- standing performance under different kinds of test settings. Concurrently, ablation studies on TAO's components underscore their vital roles in the process. Surprisingly, when we deploy TAO into new environment inhabited by unknown opponent policies, it exhibits a faster and superior adap- tation than other approaches, particularly in confronting previously unseen opponent policies. This underlines the formidable in-context learning capability inherent our approach.
2 PROBLEM FORMULATION
We the Aurtiall--bseevaable Stochastic Game (POSG) (Yang & Wang 2020) for basic formal- ization of the competitive environment. POSG defined by a tuple (I, S, tor hee A, T, (R') ) (2¹ Jee), where {1, 2, ..., N) is the set of agents. S is the state space. O' the observation
2

---

Published as a conference paper at ICLR 2024
space of agent i. A A AP ... A is the joint action space. T 8 x A denotes the transition dynamics, which defines the probability distribution on the next state given the previous state and the joint action. R' S x A x S - R denotes the reward function of agent Q S x A x O + [0, 1] denotes the agent i's observation function, which defines the probability distribution over its possible next observation given the previous state and the joint action.
[0.
In this study, we utilize to denote the controlled agent and -1 to denote the opponent and focus on modeling one opponent. We assume that the opponent's policy originates from a of fixed policies II { 1.k (a (a-1o= [ )=,,,,,,,... K, which are obtained by the scripts or RL algorithms pre-training.
2.1 OFFLINE OPPONENT MODELING
Assuming that the interaction between the controlled agent and the opponent policy π 1,k gener- 1.k
ates their respective trajectories, denoted as 71.k (og 1 1.k 1.k:
1.k 1,K ,....) T ag .0o 01 0j ,71
and -1.1 (o 1,k:
00 .r -11 ..... T-1.k, The resultant dataset is thus denoted as D (TLk T-1x). Within the context of offline learning, we presume the availability of the dataset D = {D* )==1, Specifically, T is acquired through interactions with π -1, while employing its approximate best response policy π _1,,,,, usually with certain noise.
Og 1,k 1,k
To ,0 -I, -1,
The objective of OOM is to use Delt to pre-train opponent-aware adapiive controlled agent policy Ma(a" lo'; D) and deploy Mo into a new environment with an unknown test opponent policy set 11 such that the controlled agent achieves the maximum expected return (i.e., cumulative reward):
max EE, I110 D TTA
Σ
R dj Ma: π
D is the opponent's 's information data, which is sampled from D during offline pre-training and is required to be collected during deployment. See Appendix B for detailed assumptions of OOM.
2.2 RATIONALE BEHIND OOM
Why not learning online? On the one hand. online learning struggles to satisfy real-world scenarios, given that interactions with opponent policies are not consistently accessible. On the other hand, online learning of opponent models bears substantial complexity and inefficiency. As with many existing algorithms, they gather copious data and undertake intricate online updating processes.
Relationship with offline meta-RL. The core objective of offline meta-RL lies in addressing task adaptation problems using offline datasets within single-agent environment, In contrast, OOM mainly focuses on opponent adaptation using offline datasets within a multi-agent setting. In a broader sense, OOM can be perceived as specialized extension of offline meta-RL RL. Specifically, the heightened challenge of OOM compared to offline meta-RL stems from the increased complexity of environment dynamics arising due to opponent policies and the involvement of multiple agents.
Main Challenges. As implied by Eq. [T. the main challenges of OOM arise from several dimen- sions: 1) OOM requires effective learning of high-level semanties from different opponent policies to aid controlled agent's decision decision-making:: 2) OOM necessitates extraction of robust response policies against corresponding opponent policies from Doff, 3) OOM emphasizes generalization, a crucial aspect for transferring the garnered high-level knowledge to adapt to unseen opponent policies.
3 TRANSFORMER AGAINST OPPONENT
We propose a three-stage Transformer-ber based approach, TAO, to address the three challenges of OOM mentioned above respectively. Firstly, we argue that opponent policies, compared to a naive task, involve more complex semantics, necessitating the learning of good representation. To achieve this, we propose Policy Embedding Learning (PEL) to induce opponent policy embed- dings conducive to downstream learning in Section 3.1 Secondly, to recognize and respond to the current opponent policy using only a subset of opponent trajectories, we introduce in-context learning- based supervised pre-training in Section 3.2 This pre-training enables predicting near- optimal actions conditioned on D; Thirdly, we aim to generalize the knowledge acquired from the offline dataset to respond unknown opponent policies during deployment. Therefore, we collect
3

---

Published as a conference paper at ICLR 2024
opponent trajectories during deployment, leveraging the model's in-context learning ability to adjust responses to continually adapt to the current opponent policy, as described in Section 3.3 Lastly, we theoretically analyze TAO's convergence opponent policy recognition in Section 3.4
3.1
OFFLINE STAGE 1: POLICY EMBEDDING LEARNING
In anticipation attaining an opponent policy representation favorable to downstream learn- ing, we propose PEL charac- terize the distinct behavioral pat- terns and high-level semantics of different opponent policies.
Bay11
ICD: In-context Control Decoder
Cross
OPE: Opponent Policy Encoder
We aim for the resulting policy embedding to be both genera- tive and discriminative: 1) Gen- erative: the policy embedding should have the ability to emu- late different opponent policies; 2) Discriminative: the policy embedding should be adept at distinguishing between different oppo- nent policies. Intuitively, generative helps policy embedding implicitly recognize opponent policies, as different policies provide different soft labels. Discriminative helps policy embedding distinguish the opponent policies that hard to distinguish.
OCW: Opponent Context Window
Rusion
lusion
Offinne Dataset
Figure 2: The overall architecture and flow of TAO.
Specifically, using an OPE Mo. T + Z parameterized by 0c. we conduct PEL with opponent
trajectories stemming from Doff where T-1 (T-1-*) = = (=o 21 : ....) E E denotes a trajectory embedding consisting of per-timestep embedding tokens (i.e., 2, ). As shown in Fig. we fuse the embedding tokens of a- a-t,r r1 10- and employ Average Pooling, denoted as function AP, to aggregate 2 to acquire an average trajectory embedding AP = Σ EEEEER Z for subsequent loss calculations. The implementation of OPE based on GPT2 (Radford al., 2019).
Generative Loss. As we aim at acquiring the ability to emulate different opponent policies, we pro- pose conditional imitation learning (Hussein et al. 2017) using OPE Mo. and an ancillary decoder Tha O-1 x Z A-!, parameterized by Φd. For given opponent policy - we predict its actions 1,k conditioned on its observations -1,
11, along with the average trajectory embeddings z 1,k obtained from its other trajectory. The corresponding generative loss is:
K
Lgen
Σ (o,a))
log T&s (a (clo,Ap (Mo. (T (T))m))
(2)
K
1=1
Hence, by leveraging auxiliary task that imitates the opponent's policies, indirectly guide the trajectory embeddings of various opponent policies toward generative. Such generative imbues the policy embedding with certain recognition capabilities, the actions of different opponent policies serve as soft labels for their respective policy types.
Discriminative Loss. As expect the policy embedding to be adept at distinguishing between different opponent policies, we use the labels of opponent policy type (i.e., index in offline oppo- nent policy set [Toff) corresponding employ dot product to measure the similarity between two distinct average trajectory embeddings. Specifically, we maximize this similarity from the same opponent policy while minimizing this sim- ilarity from different opponent policies. The corresponding discriminative loss is:
the trajectories for adequate distinction between them. We
B
exp (Z.
2, 271/p)
Cais = EE
log
(3)
Eie exp (z, -
jEE..f
Here, 2, AP (Mo, (7, where 7, T- B is the batch size of sampling- p denotes the temperature factor. We define KC: T- [K] as an indicator function the opponent policy type, then ε = {j E [B] (C(r--)) K)r Such a method drives trajectory embeddings of the same opponent policy towards proximity, while those from different opponent policies are pushed towards
4

---

Published as a conference paper at ICLR 2024
dissimilarity. Intuitively, this could help recognize the different opponent policies, even if the two opponent policies are similar, their policy embeddings will be relatively separate.
The comprehensive loss function for offline stage is:
Lomb
Lgen + A Cdis-
(4)
Within, a and À are weighting coefficients for generative and discriminative losses.
3.2 OFFLINE STAGE 2: IPPONENT-AWAR RESPONSE POLICY TRAINING
Given that goal is to handle multiple opponent policies using a single model, necessitate a mechanism to recognize the current opponent's policy and subsequently produce the appropriate response actions. To this end, leveraging the characterization of the opponent policies offered by the trained OPE, we propose in-context sarniingbbsee supervised pre-training to learn an ICD Me, : Z x T + A', parameterized by 0d, with the same D Formally, ICD autoregressively predicts near-optimal actions at conditioned on trajectory 71.^ and opponent's 's trajectory embedding =
Given the opponent in-context data (i.e., D in Section 2.1 encode it using OPE to obtain good representation 2 Conditioned on z ICD is designed to recognize the current opponent policy and generate an appropriate response. Intuitively, such supervised pre-training induces an opponent- aware adaptive response policy capable of recognizing and adapting to changes in opponent polices.
Response Loss. Postulating 0
{02, 0, in-context learning- barning-based based supervised pre-training loss is:
Les
1 K
E ETIANTIL
Σ
log Ma. (at 1y: Me. (GetOfED (T (T-
(5)
Here, V/2 (Go, Og. ao. .... G., G== Er= Ry denotes Reward-To-Gi (RTG). We use fun- tion GetOffD to sample opponent in-context data D. GetOffD begins by sampling C' trajectories -1,k
from T-1,k, following which it samples H consecutive fragments {(a PT-11 0 from each trajectory and ultimately stitches them together. Such sampling method context data stems from the intuitions that given opponent policy, its play style might be more pronounced over continuous timesteps (so we sample consecutive fragments), and it can showcase different playing behaviors across various episodes (so we sample from multiple trajectories).
-1. ()) --1-1 J==h opponent in-
Fig. shows the flow of offline stage 2, and we use the same causal Transformer as in Chen et al. T for the implementation of ICD. Intrinsically, we apply cross-attention to discover the associa- tion between the opponent's 's policy and the corresponding response policy. Thus, we have established an implicit mechanism for discerning opponent policies and generating respective responses.
3.3 DEPLOYMENT STAGE: IN-CONTEXT OPPONENT ADAPTING
We expect to generalize the knowledge acquired from Deft to respond to unknown opponent policies during deployment. Therefore, we introduce an OCW W to collect opponent trajectories to leverage Mg's 's in- in-context learning ability to continually adapt to the current opponent policy.
W retains the nearest trajectories of the opponent (T 1 the last episode. The OCW and the flow of deployment stage shown in Fig. While interacting with the new environment, actions of the controlled agent are sampled using following formula:
1), where the index of
at Maa (: vl Mo. (GetOnD (W))).
The function GetOnD samples H fragments from each trajectory of W and then stitches them together. Thus, by freezing 6, we can adapt to diverse opponent policies by accumulating their trajectory data d The key impetus behind our approach is: If the current opponent policy is similar or identical to that prior one from 11 we aim to re-engage the suitable responses quickly. Conversely, if unfamiliar, strive exploit it by extrapolating from the existing knowledge.
See the pseudocode of TAQ in Appendix
5

---

Published as a conference paper at ICLR 2024
3.4 THEORETICAL ANALYSIS
We conduct theoretical analysis from the following two aspects. 1) Theorem 3.1 We prove that under some reasonable assumptions, TAO equivallnnt Posterior Sampling in Opponent Modeling (PSOM), an algorithm built upon Posterior Sampling (PS), where PS guarantees convergence and proven be sample-efficient RL algorithm (Osband et al. 2013). The detailed introduction of PSOM is in Appendix [DI] 2) Theorem 3.2) We prove the convergence of the PSOM algorithm for opponent policy recognition and demonstrate that PSOM converges to the optimal solution if PS converges to the optimal solution. Based on Theorem 3.1 TAO has the same guarantees as PSOM.
Theorem 3.1 (Equivalence of TAO and PSOM). Assume that the learned model Mg consistent and the sampling of o from Tpm is independent of the opponent policies, then given म and its W, we have P [W, & ;
PWW म = Ma) for all possible En-
Given any -- Il Tpre (;; denotes the probability distribution on the trajectories set T-1 of π -1, Eir denotes history consisting of (o, a") with length H, where a² is the near-optimal action. म denotes the true opponent policy interacted with during deployment and W consists of (WC
--'s in-context data. P( [W, #, PSOM) and P म Maa denotes the same form of
probability of generating En conditioned given W and म under PSOM and TAO, respectively. We provide a detailed explanation and a complete proof of Theorem 3.1 in Appendix D.2 D.2
Theorem 3.2. When the opponent policy - originates from TI the PSOM algorithm using (o-1,a=3) _a= tuples as in-context data guarantees to converge, and it converges to the optimal solution if the original PS using (o" a.,''''' o", r²) tuples as in-context data converges to the optimal solution.
Founded on Theorem 3.1 TAO shares the same properties. See the full proof of Theorem 3.2 in Ap- pendix D.3 Note that the proof is built upon using (o (o-1,a-1) tuples as in-context data, while TAO 1-1 in implementation, providing more information and assuring the guarantees.
uses (o
Intuitively, when the current true opponent policy -- belongs to II if PS correctly recognizes the #- then PSOM is guaranteed to recognize it correctly. Even if PS fails to recognize the お correctly, PSOM still has a possibility of making a correct recognition. Based on Theorem 3.1 TAO shares the same properties the PSOM algorithm mentioned above.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Environments. We consider two emblematic competitive environmental benchmarks: 1) Markov Soccer (MS) (Lanctot al. 2 A sparse-reward discrete soccer environment that exemplifies Adversary (PA) (Lowe et al.) 2017): A
a two-player zero-sum game; 2) dense-reward continuous multi-particle environment that exemplifies a three-player non-zero-sum game. See Appendix E for the environmental specifics.
Baselines. Given that most opponent modeling approaches employ online learning settings, we draw comparisons with Embedding braseedding-beeeddd Opponent Modeling approaches these are comparatively straightforward to adapt to the OOM setting. To ensure an equitable comparison, we mandate all ap- proaches to use the same neural architecture as ours (see Appendix F). The baselines include: 1) DRON-concat (He et al.) 2016a): Encode the opponent's 's hand-crafted features with an auxiliary network; 2) DRON-MoE (He et al., (016 Encode the opponent's 's hand-crafted features using Mixture-of- Expert (MoE) structure: 3) LIAM (Papoudakis et al. (02T): Encode opponent informa- tion with AutoEncoders (AE); 4) MeLIBA (Zintgraf et al.) 20221; Encode opponent information with variational AE; 5) Prompt-DT (Xu et al. (022): A offline meta-RL approach based on DT and prompt learning with the potential to tackle OOM; 6) TAO without PEL (abbreviated as TAO w/o PEL): A variant of TAO without pre-training OPE via Cemb- See Appendix G
for details of baselines.
Opponent policy and offline dataset. We provide the details of the opponent policies in Ap- pendix H and the offline datasets construction procedure in Appendix []]
Test settings. We focus testing against non-stationary unknown opponent. Non-stationary refers to the opponent switching its policy every E episodes, which is concealed from all approaches. Meanwhile, unknown signifies that the type (i.e., index in πTet) of the opponent's policy remains

---

Published as a conference paper at ICLR 2024
LoW
PA
Ph-ma
155
54-
-1re
(b)
Figure 3: (a) Average performance under different test settings. The evaluation metrics for MS and PA are return. (b) Overall performance under the mix setting, where each point indicates the average result of E episodes against an unknown opponent policy that the opponent switches to.
concat
DRON-NcE
Prompt--0
w/o (oL((uur)
(ours)
seen
MS unseen
MS mix
1.0
0.5
Return
-0.5
02
05
04
63
88
07
06
c8
06
07
05
03
67
o1
oB
Opponent Palicyy Type
Opponent Policy Type
Opponent Policy Type
PA seen
PA unseen
PA mix
550
Retur
550
01
02
04
02 03
68
07
a6 o8 o7
06
64
06
o8
03
Opponent Policy Type
Opponent Policy Type
Opponent Policy Type
Figure 4: Detailed performance under three settings. We use ol.... o5 to denote all opponent policy types in seen and o6....,o8 o8 to denote all those in unseen. The y-axis tick interval of PA 50.
perpetually concealed. We adopt three kinds of test settings for I[ where seen is equivalent the offline opponent policy set II unseen contains strictly previously unseen opponent policies, and mix a mixture of the former two. For all testing experiments, we adopt the same specifications as follows: 1) E is set to 50; 2) We evaluate all approaches against the non-stationary unknown opponent over 2500 episodes; 3) We adopt the checkpoints 2000 training steps of offline training to perform testing for all approaches; 4) We report the average and standard error of the mean results over 5 random seeds. See hyperparameter settings in Appendix[1
4.2 EMPIRICAL. ANALYSIS
To comprehensively analyze TAO's effect and underlying mechanisms, we ask the following ques- tions and design corresponding experiments answer them:
Question 1. Can TAO achieve competitive results against other approaches for the OOM problem?
In Fig. 3(a) we show the average total performance under three different online test settings. We also provide Fig. (3b) to show the smoothed per-opponent policy performance curve under the mix setting. Across the three settings in MS and PA. TAO demonstrates competitive results compared to other baselines, particularly in the unseen setting. This implies that TAO can respond reasonably unknown opponent policies. It's worth noting that even without PEL, TAO still achieves results generally not worse than other baselines, emphasizing the critical role of TAO's design choice of in-context learning-based training.
Under the three settings, LIAM and MeLIBA generally show better results than DRON-MoE and DRON-coneat. This could be due to the fact that reconstructing the opponent's information is more effective in uncovering the underlying relationship between the opponent's 's and controlled agent's s policies than direct encoding. DRON-MoE surpasses DRON-concat in general, probably because the MoE network implicitly selects more appropriate responses than concatenating the hidden states. Under the seen setting, Prompt-DT's performance closely approaches that of TAO w/o PEL, suggest- ing that return-ondditioed expert demonstrations may also indicate the current opponent's policy
7

---

Published as a conference paper at ICLR 2024
Policy Embedding
Polkc Embedding
Oppumertt Cortart Sega-enc
Healmap
Healmap
8
2.00
15
8
3.4
230
-50
18
1.2
-120
-150
-50
0
50
150
-100
-50
50
150
62
01 65
3.03
63
67
63
01
05
2.00
Opponent Policy Type
Osponent Policy Typc
(a)
(b)
Figure 5: (a) Visualization for the dimensionality embeddings of different opponent poli- cies with i-SNE. (b) The attention heatmap of TAO against different opponent policies. For each opponent policy type, we visualize the attention weights over the final 20 timesteps in an episode.
0.40
M5
0.5
mi
500
mx
0.6-
0.25
0:30
9.00
50.20
Unsoon
nix
205n
unscon
mix
20
20
50
25
22
Seiting
Tex: Seling
Number Opponent Polacy Swietts
Number Opponert Policy Soltchee
(a)
(b)
Figure 6: (a) and (b) are the ablation experiments of PEL under the same plotting settings as in Fig.
certain extent. Whereas under the unseen and mix settings, Prompt-DT performs poorly, high- lighting the difficulty of achieving effective generalization to unseen opponent policies when relying solely on the controlled agent's information without incorporating the opponent's This suggests that the original offline meta-RL approaches might not be enough for tackling the OOM problem.
Fig. 4 iillustrates the smoothed per-episode performance curve. Given that the Il settings we em- ploy encompass opponent policies with varying strengths, the game results exhibit substantial dispar- ities accordingly. TAO delivers stable and favorable performance MS and PA. We notice that TAO shows some adaptability. trying to stabilize its exploitation of the same opponent's policy through- out games and adjusting when the opponent's policy switches. In the unseen and mix settings, this adaptability of TAO becomes more pronounced in general. In contrast, other approaches always fail to respond to unseen opponent policies reasonably, and, certain instances, they are susceptible to greater exploitation by the opponent.
Question 2. Does TAO learn a good policy embedding (representation) of opponent policies?
We use the OPE trained by offline stage to embed all opponent traiectories in Dell and visualize the dimensionallt--reducee embeddings (denoted as Fredncea) in Fig. 5(a) The colors indicate the respec-
tive opponent policy type to which 3 belongs. It can be observed that "reduced corresponding each opponent policy is roughly clustered into the same cluster. Intuitively, TAO learns policy embedding that distinguishes different opponent policies well, showing. to a certain extent, that our loss design for PEL has achieved the effect we expected. This can help TAO's downstream learning. which is verified by the overall better results of TAO compared to TAO w/o PEL. The clustering ef- fect in PA is more pronounced than in MS. This could be attributed the greater divergence among opponent policies in PA, making them easier to differentiate.
In Fig. 5 we show the attention heatmap of TAO. This heatmap visualizes TAO's ' cross aoos-anttnttion weight across different opponent policies. The x-axis represents different opponent policy types. The y-axis represents the position in the opponent context sequence obtained by GetOnn(W)) and each position has weight indicated by the depth of color. In MS and PA. TAO exhibits distinctive atten- tion pattern for different opponent policies, with a propensity to concentrate on specific timesteps. Meanwhile, TAO maintains relatively consistent attention distribution for given opponent policy across various timesteps within the same episode. This suggests that TAO has learned a representa- tion that is helpful with discerning the different opponent policies. Moreover, it has discovered some unique characteristics of each opponent policy's 's action sequence.
Question 3. What the contribution of each component in PEL to tackling the 00M problem?
8

---

Published as a conference paper at ICLR 2024
Table
Average performance on different opponent policy configurations.
Oppoment Policy Configuration
Opponent Policy Configuration
seen
unsven
mix
Seen
untven
mix
'TAO (ours)
0.51 +: 0.01
0.55 + 0.01
0.43 +: 0.01
0.43 : 0.01
0.65 + 0.02
0.40 + 0.01
(burs)
1:
0.41 0.00
0.02
1
M (Avg. Return 1*)
DRON-concat
0.15 0.02
-0.02 + 0.05
-0.05 = 0.03
-0.05 + 0.01
-0.11 + 0.01
-0.17 2 0.01
DRON-MoE
0.15
0.38 =
0.15 0.02
0.09 0001
0.31 = 0.04
0.11 0.11 0.01
LIAM
0.01
0.47 0.01
0.32 0.01
0.23 0:01
0.47 1: 0.01
0.241:0.0
McLIBA
0.38
041 0.01
0.30 0.00
0.29 0.01
0,42 + 0.01
0,25 0.01
PromptDDT
0:48
0.33 +
0.39 0.02
0.20 +.01
TAO (ours)
105.6 + 3.0
109.5
101.6 + 3.4
0.5
39.2 0.9
35.3
TAO w/o PEL (ours)
2.7
81.5 7.9
67.4 4.4
2.7
312 1.1
29.4 + 0.8
PA
DRON-come concacl
21.4 114::.1 3.3
40.0 10.4
12.:449
0.8 28.0
21.5 0.8
26.0 0.4
(Avg. Return
DRON MoE
94.2
20.5 1: 0.6
60.9 100110
33.1 1.6
18.5
04
27.4 1: 0.5
LIAM
83.6 5.9
60.5 3.
4.3
13.5 0.6
17.1 0.4
27.6 0.6
McLIBA
5.6
48.5 3.
482
3.78
1933
27.1
Prompt-DY
66.9 - 8.2
-10.1
+: 4.0
2388
9.
22.5
0.8
25.0 0.3
25.1 + 0.2
In Fig. we show the results for ablations PEL. TAO without Cgen sets a to O, while TAO without Lais sets À to O. TAO w/o PEL performs the worst under all three settings, while in Question 1, we found that TAO w/o PEL showed good results and verified the effectiveness of in-context learning- based supervised pre-training. The improvement in results suggests that PEL further benefits the TAO's subsequent in-context learning-basee training process as we intended. TAO without Leen is almost uniformly superior to TAO without Ldis, while consistently worse compared to TAO. im- plying that discriminative learning makes a more critical contribution to the improvement of TAO's ability. One possible explanation that discerning different opponent policies holds greater signifi- cance for the controlled agent's ss decision-making than simulating the opponent's 's policy.
Question 4. Does TAO exhibis robustness to different opponent policy configurations?
Since II is a subset of the I under the mix setting, we can choose different opponent poli- cies to put into If and set up the three test settings accordingly to evaluate the robustness of all approaches to different opponent policy configurations. Specifically, we establish two configura- tions: For the first one, seen contains o0,02, o2, o3, 04, 06, unseen contains ol, 05, 07; for the second one, seen contains o0, o2, o3, 05, 07, unseen contains ol, 04, o6. In different configurations, dispar- ities are evident in the numerical outcomes across the board because of the varying strength of the opponent's 's policy in TToff greatly impacting the subsequent testing stage. Therefore, we focus on the relative performance between different approaches.
Tshos the primary outcomes, where the results in bold denote the best among all approaches. TAO maintains its competitive performance in both environments and across different opponent pol- icy configurations. In contrast, other approaches even fail under the new configuration. For example, Prompt-DT performs poorly in P A under configuration 1, probably because the policies of seen and unseen opponent are pretty different. Thus, Prompt-DTT struggles to generalize knowledge from opponent policies to unseen opponent policies using only controlled agent trajectory prompts. This indicates TAO is relatively more robust approach compared to other baselines under OOM setting.
Additional empirical analysis. We provide experimental analysis on performance the offline training, adaptation speed in the testing, and ablation studies of the OCW's size C' in Appendix
CONCLUDING REMARKS
In this paper, we formalize a problem, OOM, aiming to mitigate the impracticality and inefficiency commonly associated with existing opponent modeling research. To tackle the OOM problem, we propose general Transformer-base
approach TAO, whose procedures consist of 3 stages: 1) Of- fline stage 1: Employ PEL to generatively and discriminatively embed opponent trajectories into la- tent spaces; 2) Offline stage 2: Adopt supervised pre-training to generate approximate best responses to different opponent policies; 3) Deployment stage: Accumulate information in new environments and adapt to unknown opponent policies with in-context learning. We provide theoretical analyses on TAO's s equivalence to PSOM and guarantee TAO's convergence in opponent policy recognition. Extensive empirical analysis validates TAO's efficacy in addressing the OOM problems. Our sults indicate that TAO responds reasonably to opponent policies even unseen before, showcasing its in-context learning capabilities. We discuss limitations and future work in Appendix L

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is supported in part by the National Science and Technology Major Project (2022Z0011640);; the Natural Science Foundation of China under Grant 62076238, Grant 62222606, and Grant 61902402; the Jiangsu Key Research and Development Plan (No. BE2023016); and the China Computer Federation (CCF)-Tencen)) Open Fund.
REFERENCES
Abien Fred Agarap. Deep learning using rectified linear units (ReLU), arXiv preprint Nao::880333333 2018.
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilaa Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In In- ternational Conference on Learning Representations, 2018.
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-99 95, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. Stat, 1050:21, 2016.
Nolan Bard, Michael Johanson, Neil Burch, and Michael Bowling. Online implicit agent modelling. In International Conference on Autonomous Agents and MultiAgent Systems, pp. 255-262, 2013.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return- conuincondiio supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems, pp. 1542-1553, 2022.
John Bridle. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. Advances in Neural Information Processing Sys- tems, pp. 211-217, 1989.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in Neural Information Processing Systems, pp. 15084-15097, 2021.
Djórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.0728) 2015.
Qiwen Cui and Simon S Du. Provably efficient offline multi-agent reinforcement learning via strategy-wise bonus. Advances in Neural Information Processing Systems, pp. 11739-11751, 2022.
Zhongxiang Dai. Yizhou Chen, Bryan Kian Hsiang Low, Patrick Jaillet, and Teck-Hua Ho. R2-b2: Recursive rassoning-based bayesian optimization for no-regret learning in games. In International Conference on Machine Learning, pp. 2291-2301, 2020.
Daniel Daly-Grafsteinn and Luke Bornn. Using in-game shot trajectories to better understand defen- sive impact in the nba. Journal of Sports Analytics, 6(4):235-242. 2020.
Authony DiGiovanni and Ambuj Tewari. Thompson sampling for markov games with piecewise stationary opponent policies. In Uncertainty in Artificial Intelligence, pp. 738-748, 2021.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement -innning-identffaablltty challenges and effective data collection strategies. Advances in Neural Information Processing Systems, pp. 4607-4618, 2021.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:611..02799, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- neta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1126-1133 1135, 2017.
10

---

Published as a conference paper at ICLR 2024
Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learingg awareness. In International Conference on Au- tonomous Agents and Multtigentt Systems, pp. 122-130, 2018a.
Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivar, Tim Rocktäschel, Eric Xing, and Shimon Whiteson, DiCE: The infinitely differentiable Monte Carlo estimator. In International Conference on Machine Learning, pp. 1524-1533, 2018b.
Claude Formanek, Asad Jeewa, Jonathan Shock, and Amu Pretorius. Off-the-grin marl: a framework for dataset generation with baselines for cooperative offline multi-agent reinforcement learning. arXiv preprint arXiv:2332.00032 2023.
Haobo Fu, Ye Tian, Hongxiang Yu, Weiming Liu, Shuang Wu, Jiechao Xiong, Ying Wen, Kai Li, Junliang Xing, Qiang Fu, et al. Greedy when sure and conservative when uncertain about the opponents. In International Conference Machine Learning, pp. 6829-6848, 2022.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. In International Conference on Learning Representations, 2022.
Hossein Gholamalinezhad and Hossein Khosravi. Pooling methods in deep neural networks, a re- view. arXiv preprini arXiv:2009.00 07445,, 2020.
Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International Conference on Machine Learning, pp. 1802-1811, 2018.
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep rein- forcement learning. In International Conference Machine Learning, pp. 1804 1813, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016b.
Pablo Hernandez-Leal, Matthew E Taylor. Benjamin S Rosman, Luis Enrique Sucar, and E Munoz de Cote. Identifying and tracking switching, non-stationary opponents: A bayesian approach. In AAAI Conference on Artificial Intelligence Workshop on Multiagent Interaction without Prior Coordination, pp. 560-566, 2016.
Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. Advances in Neural Information Processing Systems, pp. 3-10, 1993.
Zhang-Wa Wei Hong, Shih- Shin-Yang Su, Tzu- Yun Shann, Yi-Hsiang Chang, and Chun-Yi Lee. A deep policy inference q-network for multi-agent systems. In International Conference on Autonomous Agents and MultiAgent Systems, pp. 1388-122 1396. 2018.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: survey of learning methods. ACM Computing Surveys, 50(2):1-35,200 2017.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1::2, 2020.
Jicchuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforeement learning. arXiv preprint arXiv:2108. 01832, 2021.
Jiechuan Jiang and Zongqing Lu. Online tuning for offline decentralized multi agent reinforcement learning. AAAI Conference on Artificial Intelligence, pp. 8050-8059, 2023.
Dong Ki Kim, Miao Liu, Matthew D Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan How. A policy gradient algorithm for learning to learn in multiagent reinforcement learning. In International Conference on Machine Learning, pp. 5541-5550. 2021.
Diederik Kingma and Max Welling. 13Xiv: 2013.
Auto-encoding variational bayes.
arXiv preprint
11

---

Published as a conference paper at ICLR 2024
Mare Lanctot, Edward Lockhart, Jean-Baphin Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. OpenSpiel: A framework for reinforcement learning in games. arXiv preprint arXiv:1908.0.. 09453, 2019.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, maxime gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In International Conference on Learning Representations, 2023.
Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. arXiv preprini arXiv:2306. 14892, 2023.
Kuang-Hun Hueii Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadar- rama, lan Fischer, Winnie Xu, Eric Jang. Henryk Michalewski, et al. Multi-game decision trans- formers. Advances in Neural Information Processing Systems, pp. 27921-27936, 2022.
Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. International Conference on Learning Representa- tions, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164 2020.
Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Henrik Christensen, and Hao Su. Multi-task batch reinforcement learning with metric learning. Advances in Neural Information Processing Systems, pp. 6197-6210 2020.
Sheng Li and Handong Zhao A survey on representation learning for user modeling. In Interna- tional Joint Conferences on Artificial Intelligence, pp. 4997-5003, 2021.
Wenzhe Li, Hao Luo, Zichuan Lin, Chongjie Zhang, Zongqing Lu, and Deheng Ye. survey on transformers in reinforcement learning. arXiv preprini arXi::2301.03044, 2023.
Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In Machine Learning Proceedings, pp. 157-163. 1994.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Computing Surveys, 55(9):1-35,2000
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-oompetivve environments. Advances in Neural Information Process- ing Systems, pp. 1-12, 2017.
Yongliang Lv, Yan Zheng, et al. Limited information opponent modeling. JCML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023.
Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: literature survey, Artificial Intelli- gence Review, 42:275-293, 2014.
Luckeciano Melo. Transformers are meta-reinforcement learners. In International Conference on Machine Learning, pp. 15340-15359, 2022.
Linghui Meng, Muning Wen, Chenyang Le, Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Yaodong Yang, et al. Offline pre-trained multi-agent decision trans- former. Machine Intelligence Research, 20(2):233-248 2023.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In International Conference Learning Representations, 2018
12

---

Published as a conference paper at ICLR 2024
Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In International Conference on Machine Learning. pp. 7780-7791, 2021.
Samer Nashed and Shlomo Zilberstein. A survey of opponent modeling in adversarial domains. Journal of Artificial Intelligence Research, 73:277-327, 2022.
lan Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, pp. 3003-3011, 2013.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi- agent reinforcement learning with actor rectification. In International Conference Machine Learning, pp. 17221-17237, 2022.
Georgios Papoudakis and Stefano V Albrecht. Variational autoencoders for opponent modeling in multi-agent systems. arXiv preprint urXiv:2001.10822 2020.
Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial ob- servability for deep reinforcement learning. Advances in Neural Information Processing Systems, pp. 19210-19222, 2021.
Keiran Paster, Sheila Mcllraith, and Jimmy Ba. You can't count on luck: Why decision transformers and rvs fail in stochastic environments. Advances Neural Information Processing Systems, pp. 38966-38979, 2022.
Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline meta- reinforcement learning with online self-supervision. In International Conference on Machine Learning, pp. 17811-17829, 2022.
Manish Prajapat, Kamyar Azizzadenesheli, Alexander Liniger, Yisong Yue, and Anima Anandku- mar. Competitive policy optimization. Uncertainty in Artificial Intelligence, pp. (4-74, 2021.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International Conference on Machine Learning, pp- 4218 4227,
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International Conference Machine Learning, pp- 4257-4266, 2018
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-py meta-reinforrcemett learning via probabilistic context variables. In International Conference on Machine Learning, pp. 5331-5340, 2019.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Ec- cles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions Machine Learning Research, 2022.
Benjamin Rosman, Majd Hawasly, and Subramanian Ramamoorthy, Bayesian policy reuse. Ma- chine Learning. 104:99-127. 127, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process opti- mization in the bandit setting: no regret and experimental design. In International Conference on Machine Learning, pp. 1015-1022, 2010.
13

---

Published as a conference paper at ICLR 2024
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, ((L):1122-1958, 2014.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-22 -29424,,, 1933.
Qi Tian, Kun Kuang, Furui Liu, and Baoxiang Wang. Learning from good trajectories in offline multi-agent reinforcement learning. In AAA! Conference Artificial Intelligence, pp. 11672- 11680, 2023.
Wei-Cheng Tseng, Tsun Huun- Johnson Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent reinforcement learning with knowledge distillation. Advances in Neural Information Processing Systems, pp. 226-237, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems, 2017.
Adam R Villaflor, Zhe Huang, Swapnil Pande, John M Dolan, and Jeff Schneider. Addressing optimism bias in sequence modeling for reinforcement learning. International Conference on Machine Learning, pp. 22270-222833 2022.
Friedrich Burkhard Von Der Osten, Michael Kirley, and Tim Miller. The minds of many: Opponent modeling in stochastic game. In International Joint Conference Artificial Intelligence, pp.
3845-3851, 2017.
Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint /rXiv:/61..057000 2016.
Ying Wen, Yaodong Yang. Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In International Conference Learning Representations, 2019.
Ying Wen, Yaodong Yang, and Jun Wang. Modelling bounded rationality in multi-agent interactions by generalized recursive reasoning. In International Joins Conferences on Artificial Intelligence, pp. 414 421, 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Transformers: State-of-the-an natural language processing. In Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pp. 38-45, 2020.
Young Wu, Jeremy McMahan, Xiaojin Zhu, and Qiaomin Xie. Reward poisoning attacks on offline multi-agent reinforcement learning. In AAAI Conference Artificial Intelligence, pp. 10426- 10434, 2023.
Zhe Wu, Kai Li, Hang Xu, Yifan Zang, Bo An, and Junliang Xing. L2e: Learning exploit your opponent. In International Joint Conference on Neural Networks, pp. 1-8, 2022.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In International Con- ference Machine Leurning, pp. 24631-24645 2022.
Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04119, 2023a.
Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Dichotomy of control: Separating what you can control from what you cannot. In International Conference on Learning Represen- fations, 2023b.
14

---

Published as a conference paper at ICLR 2024
Tianpei Yang, Jianye Hao, Zhaopeng Meng, Chongjie Zhang, Yan Zheng, and Ze Zheng. Towards efficient detection and optimal response against sophisticated opponents. In International Joint Conference on Artificial Intelligence, pp. 623-629, 2019.
Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theo- retical perspective. arXiv preprint arXiv:201 1.00883, 2020.
Xiaopeng Yu, Jiechuan Jiang, Wanpeng Zhang, Haobin Jiang, and Zongqing Lu. Model-based op- ponent modeling. Advances in Neural Information Processing Systems, pp. 22208-28221, 2022.
Luyao Yuan, Zipeng Fu, Jingyue Shen, Lu Xu, Junhong Shen, and Song-Chun Chun Zhu. Emer- gence of pragmatics from referential game between theory of mind agents. arXiv preprint arXiv:2001.07752, 2020.
Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjie Fan. A deep bayesian policy reuse approach against non-stationary agents. Advances in Neural Informa- tion Processing Systems, pp. 962-972, 2018.
Luisa Zintgraf, Sam Devlin, Kamil Ciosek, Shimon Whiteson, and Katja Hofmann. Deep interactive bayesian reinforcement learning via meta-learning. International Conference on Autonomous Agents and MultiAgent Systems, pp. 1712-1714, 2021.
15

---

Published as a conference paper at ICLR 2024
A RELATED WORK
Oppoment modeling, Recent studies on opponent modeling based on machine learning techniques can be broadly divided into the following categories:
1) Embedding- based Opponent Modeling: Embed the opponent's 's policy into a latent space through representation learning methods to assist the decision-makee of the controlled agent. Grover et al. 2018)) employed imitation learning (Hussein et al. 2017) and contrastive learning (Jaiswal et al.) 2020j to generate policy embeddings of opponent trajectories. They then combined these embed- dings with RL for policy optimization. Similar efforts, such as He et al. (2016a) and Hong et al. (0188, utilized auxiliary networks for encoding hand-crafted opponent features, predicting oppo- nent actions, and tuning the policy network for better performance. Paapoudakis et al. (2021) pro- posed using an autoencoder to utilize the observations and actions of the controlled agent to recon- struct the observations and actions of the opponent to learn embeddings to assist decision-making. In contrast, Papoudakis & Albrecht and Zintgraf et al. (2021) adopted Variational AutoEncoders (VAE) Kingma & Welling 2013) learn the high-dimensional distribution of opponent policies.
2) Opponent Modeling with Online Reasoning: Detect or infer the opponent's 's policy online through methods such as Bayesian and then yield responses accordingly. Bard et al. (2013) first trained the mixture-offexperr counter-strategies against a known set of fixed opponent policies. Subsequently, they employed a bandit algorithm during testing to select the most appropriate counter-strategy. Based on BPR+ (Rosman et al., 2016)] Hernandez-LLel et al., 2016). Zheng et al. (2018) proposed a rectified belief model to more accurately detect opponent policies. Additionally, they introduced a distillation policy network to achieve improved outcomes, DiGiovanni & Tewari (2021 I) employed
Thompson sampling [Thompson] 1933) and change detection to tackle opponent switching between stationary policies. Fu et al. (2022) proposed using bandit algorithm to choose greedy or conserva- tive policies to play against non -on-stationary opponent. The greedy policy was trained by VAE along with conditional RI. and updated online through variational inference. 'he conservative policy was fixed, robust policy. [Lv et al. (2023) proposed a similar approach to exploit non-stationary opponent.
3) Opponent Modeling with Evolving Opponents: Estimate the opponent's policy and corresponding impacts considering the opponent's learning (i.e., gradient update). Foerster et al. (2018a) modeled the update of the opponent's policy and estimated the learning gradient of the opponent's 's policy. Fo- erster et al. (2018b) introduced an Differentiable Monte-Carlo Estimator operation to consider how to shape the learning dynamics of other agents based on |Foerster et al. (2018a)'s approach. Letcher
ee al. (2019) further combined stability guarantees from Ahoa with opponent-shaping abilities from Foerster et al. (2018a) to improve theoretically and experimentally. (Al-Shddivat et al. (2018) and Kim et al. (2021) ) utilized meta-learning to adapt to evolving opponent. Kim et al. (2021) also introduced a term that closely relates to shaping the learning dynamics of other agents' policies, thereby considering the impacts of agent's current policy on future opponent policies.
4) Opponent Adapting with Meta-learning: Based on meta-learning methods, train against a set of known opponent policies to quickly adapt to unknown opponent policies at test time. While most meta-RL methods assume the tasks in training and testing share a similar distribution, this category explores the potential application of meta-RL. competing with unknown non-stationary opponent. [Al-Shedivat et al. (2018) presented a gradient-based meta-learning algorithm for contin- uous adaptation in non-stationary and competitive environments, demonstrating its effectiveness in improving adaptation efficiency. Kim et al. (2021) proposed a novel meta multi-agent policy gradi- ent algorithm that effectively addresses the non-stationary policy dynamics inherent in multi-agent RL. Zintgraf et al. (202T) introduced a meta-learning approach for deep interactive Bayesian RL in multi-agent settings, which leverages approximate belief inference and policy adaptation for im- proved opponent adapting- Wu et al. (2022) proposed meta-learning framework, named Learning to Exploit (L.2F), for implicit opponent modeling, enabling agents to quickly adapt and exploit op- ponent with diverse styles through a few interactions during training.
5) Theory of Mind (ToM))based Opponent Modeling: Reason about opponent's mental states and in- tentions, to predict and adapt to opponent behavior by modeling their beliefs, goals, and actions. Von Der Osten et al. (2017) proposed the multi-agent ToM model to predict opponents' actions and in- fer strategy sophistication in stochastic games. Upon Zheng et al. (2018)*s 's Bayesian online oppo- nent modeling approach, Yang et al.] (2019) introduced the ToM method, which uses higher-level decision-making to play against the opponent who also performs opponent modeling.
16

---

Published as a conference paper at ICLR 2024
Rabinowitz al. (2018) and Raileanu et al. (2018) also explored methodologies for modeling an opponent's 's mental state. Rabinowitz et al. (2018) employed three networks to reason about agent actions and goals, simulating a buman-like ToM. Raileanu et al. (2018) proposed using its own policy along with the opponent's observations and actions to learn the opponent's goals.
Predict opponent's actions by simulating multi-level nested beliefs, en- abling anticipation of opponent behavior based on their expectations of the controlled agent's rea- soning process. |Wen al. (2019) and Wen et al. (2021) proposed performing recursive reasoning by modeling hypothetical nested beliefs via the agent's joint Q-function. They showed improved adaptation in stochastic games. Dai et al. (2020) employed recursive reasoning by assuming agents select actions based on GP-UCB acquisition functions (Srinivas et al.) 2010). It achieves faster regret convergence in repeated games. Yuan et al. (2020) proposed an algorithm using ToM-based recur- sive reasoning and adaptive RI to enable agents to develop pragmatic communication protocol, allowing them to infer hidden meanings from context and improve cooperative multi-agent commu- nication. Yu et al. (2022) introduced a model-based opponent modeling approach that uses recursive imagination in an environment model and Bayesian mixing to adapt to diverse opponents.
6) Recursive Reasoning:
Most existing work assumes that the environment and opponent policies are consistently available, whereas our work focuses on opponent modeling under challenging scenarios where they are not.
Offline meta-RL. Offline meta-RI. aims at using offline datasets to develop the agent's ' understand- ing of tasks within the training distribution and extrapolate this knowledge novel tasks. (Li et al.) 2020 Mitchell et al. 2021| Dorfman et al., |021 Pong et al. 2022). The key principles are meta- learning, where the model learns to quickly adapt based on limited samples of new task (Finn al., 2017). and task information inference (Rakelly et al. 2019). Algorithmically, in-context learning stands within the domain of meta-learning and can be considered as taking a more agnostic ap- proach by learning the learning algorithm itself (Duan et al., 2016; Wang et al.) 2016) Mishra et al., 2018 Laskin al. 2023). Lee et al.) (2023) introduced supervised pre-training to empirically and theoretically showcase robust in-context learning abilities in decision-making Seen through another lens, OOM can be approximately regarded as an offline meta-RL problem. When we consider the opponent (teammate) policy in the field of opponent modeling as task, we can roughly regard O0M as an offline me problem. In precise terms, OOM poses more complexity than offline meta-RL, and we will discuss the relationship between the two in Section 2.2
Transformers for decision-making, There has been growing interest in applying the Transformers to decision-making by formulating the problem as sequence modeling (Yang et al.] 2023a Li et al.) 2023 Chen et al.] (202I) proposed Decision Transformer (DT), which models action sequences conditioned on returns with a causal Transformer trained on offline data. Follow-up works studied enhancements like improved conditioning (Furuta et al. 2022] Paster et al. 2022) and architectural
innovations (Villaflor et al., 2022). Another exciting direction leverages the generality and scalabil- ity of Transformers for multi-task learning (Lee et al., 2022; Reed et al. 2022). Transformers for decision-making also exhibited meta-learning capabilities (Melo (022). Recently, [ee et al. (2023) proposed a Transformer- based in-context learning approach that empirically and theoretically im- proves over behaviors seen in the dataset in terms of regret where DT cannot (Brandfonbrener al., 2022 Yang et al., 20236). Building upon these insights, we design a Transformer-basee approach to fully leverage its in-context learning abilities to tackle the OOM problem.
Offline multi-agent RL (MARL). Offline MARL has emerged as an important subfield of MARL in recent years (Cui & Du 2022 Tian et al. 2023 Wu et al.) 2023 Jiang & Lu 2023 Formanek al. 2023). This line of research aims to learn effective policies purely from pre-collecter collected datasets without any further multi-agen, agent environmental interactions. The offline setting avoids costly or dan- gerous online exploration, making MARL more practical for real-world applications.
A key challenge in offline MARL is handling the distribution shift between the offline dataset and the true environment dynamies at test time (Jiang & Lu 202T). When opponents employ policies not seen in the offline data, the performance of policies trained purely offline can degrade significantly. Recent works have proposed various techniques to address this issue. For example, Pan et al. (2022) introduced Offline MARL with Actor Rectification (OMAR) that uses zeroth-order optimization to update the actor based on the offline critic conservatively. Tseng et al. (2022) proposed distilling knowledge from teacher Transformer trained with global information into decentralized student
17

---

Published as a conference paper at ICLR 2024
policies to enable effective decentralized execution test time. The Transformer architecture has also been increasingly adopted for offline MARL. For instance, the Multi-Agent Decision Trans- former (MADT) (Meng et al. 2023) treats MARL sequence modeling problem and predicts actions in an autoregressive manner.
Offline MARL and OOM aligned in their pursuit of extracting effective policies from offline datasets in multi-agent settings and utilizing them in testing environments to boost efficiency and bridge the gap with real-world scenarios. Nevertheless, most offline MARL approaches focus on cooperative settings, while in this work, we concentrate on OOM in competitive environments. Fur- thermore, whether explicitly or implicitly. OOM must introduce modeling of other agents in the environment for adaptability, while offline MARL does not emphasize such requirement.
B DETAILED ASSUMPTIONS OF OOM
Environment types to which OOM is applicable. While our paper focuses on OOM within com- petitive settings, it's worth mentioning that the extension of OOM to cooperative and mixed envi- ronments where competition and cooperation are intertwined is straightforward. In cooperative and mixed environments, the corresponding OOM problem can be formalized by instantiating the op- ponent other agents that are desired to be modeled. Their optimization objectives have the same form as Eq. T to learn an modeled agent-aware adaptive controlled agent policy.
Number of controlled agent and opponent. According to our formalization of the OOM problem, there is no limit on the number of controlled agent and opponent in the environment. Yet, some assumptions of the OOM problem need to be emphasized again: 1) The trajectories of all controlled agents in the offline dataset are generated by an approximately best response policy against a fixed opponent policy; 2) All opponent agents adopt the same fixed opponent policy.
Availability of the environment. During offline training, we hypothesize no interaction with the environment and the opponent. In another word, opponent policies in the offline opponent policy sei IT cannot be called explicitly for learning. It is only possible to sample the trajectories T-1,k produced by each opponent policy from D Yet, for comparing the learning efficacy across all approaches, we permit a small number of interactions per set interval of checkpoints for evalua- tion. During testing (i.e., deployment), interaction with the new environment is allowed, where the opponent adopts policies from the test opponent policy set I[test
Specification of opponent policy. All opponent policies in II are prohibited from learning, and there is no policy switching at the granularity of timestep. In our experiments, we employ three distinct test settings for the opponent policy set I[est seen, equivalent to ITotI; unseen, encompass- ing previously unseen opponent policies; and mix, which combines elements of both. Our study includes diverse opponent policies that are independent of one another and possess varying degrees of strength. See the details of the opponent policies in Appendix H
Accessibility of opponent information. During offline training stages, only limited opponent infor- mation be gleaned from the provided Doff, and no extra information can be sourced. However, during testing, collection of the opponent's 's trajectory r after each episode is permissible, enabling all approaches to adapt to unknown opponent policies based on their own mechanism.
18

---

Published as a conference paper at ICLR 2024
C PSEUDOCODE OF TAO
Algorithm Transformer Against Opponent (TAO)
1: x Offline Stage Policy Embedding Learning x/
2: Initialize model Ma, with parameters de, and set the batch size B.
3: while not converged
4: Initialize empty batches Byen 0 Bais 0-
5: for & in [K] do
-1.k) 6: Sample B trajectory tuples (T 1.k T 1.kk ~ T-1.k, 1,1, and add these tuples into B gen-
7: end for
8: Sample B trajectories - T to form Bais.
9: Calculate Leen in Eq. (2 using gen. and calculate Lais in Eq. using Bais.
Calculate Lemb Eq. 14. and backpropagate update A..
11: end while
12: /* Offline Stage 2: Opponent-awin aware Response Policy Training x
13: Initialize model Ma, with parameters 0a, and set the OCW size C.
14: while not converged do
15: Initialize empty batches Bies (--
16: for & in |K] do
17:
Sample B trajectory tuples (rlik, {T-11, Je=,,.)) where 7l.k Tlik, (To Jel.... T--II, and add these tuples into Bies-
18: end for
Calculate Lies in Eq. (5) using Bres and backpropagate update Og.
20: end while
/* Deployment Stage: In-context Opponent Adapting 22: Initialize empty OCW W = {). and set the opponent policy switching interval E. do 24: Sample unknown opponent policy π IT
for op max
25: for ep in [E] do
26: Deploy Ma by sampling of via Eq. 66 at each timestep / in this ep, where 0 - (0e, Oz). -1 27: Add --1 = (o 00 :o ..... into W, and clip W to contain the nearest C trajectories.
28: end for
29: end for
19

---

Published as a conference paper at ICLR 2024
D DETAILED THEORETICAL ANALYSIS
D.1 ALGORITHM OF POSTERIOR SAMPLING IN OPPONENT MODELING
We instantiate Bayesian posterior sampling algorithm in the context of opponent modeling, ferred to as Posterior Sampling in Opponent Modeling (PSOM). In the PSOM algorithm, we use opponent trajectory consisting of consecutive (o- a tuples as in-context data and add them to the OCW W. Following up, we describe the PSOM algorithm in a most general way (Osband et al., Lee al. )023).
2013
Given the initial distribution of opponent policies IIg < Hpre where Hm is the probability distri- bution on II and an empty OCW W = =0, for E [C]:
1. Sample the opponent policy म5 by 1I, and compute the approximate best response policy x for the controlled agent;
2. Interact with the true opponent policy म using T/= and add the opponent trajectory to the W.
3. Update the posterior distribution 11.(T ()= P( W)).
D.2 PROOF OF THEOREM 3.1
Theorem 4.1 (Equivalence of TAO and PSOM). Assume that the learned model Mg consistent and the sampling of o- from T is independent of the opponent policies, then given --1 and its W. we have P(S) W =I: = P(Sh - MM) ) for all possible Ehr-
Proof. Differing from the main text, in this section, we use π to denote opponent policies poste- riorly sampled from pre. where ppe is the probability distribution on II and use x- to denote the true opponent policy interacted with during the deployment stage of TAO. For clarity and ease of understanding, all trajectory sequences are indexed starting from in this section (originally start- ing from 0 in the main text). Define 5 = (o),a!" ".............. :..., .
1.*) as controlled agent's 's history,
where H denotes the maximum length of this history and a is sampled from the approximate best response policy NI.- against the posteriorly sampled π 1:: (:;x=1 denotes the probability distribution the trajectories set T of -1 during TAO's offline stage 2.
Let π ~ Unr and OCW W contain sampled trajectory fragments of π and let Querry e O², O',ali it E - E (0) x - and h E [o, H 1] be arbitrary, the full joint probabil- ity distribution during TAO's offline stage 2 can be denoted as:
P W. W. qquery a².²) = Ippe (TT ( (W:
CII π (""(looo x Unif H - l1l query (o (Qquery) VE[M]
1, (aa'"""aaa (a".") 1
1o ग 1)).
(7)
Herein, DH A((O)))), which is independent of the opponent policies. Qquery is set the uniform over O Moreover, do small modifications the original TAO by sampling h Unif(O, H 1] and truncating c) from -
We define the random sequences and subsequences of 1) the controlled agent's observed trajectory and 2) the controlled agent's 's true trajectory under PSOM algorithm, respectively, as follows:
1) = (hg W) = (0,
C.J.PPON. PSOM AA I.PSO
O PSPOM A 1.PSMM):
2) EPSOM (h; W) = (.SPSOM A L.POOM
S
PSOM
PSOM
h
These two types of trajectories are generated in the following manner:
TPSOM ~ P( - IW), 0 I.PSOM ~ P(O PSSM |s SPSOM ), SPSOM P.
-1
A IPOSM n m (-10) PSOM
TPOMM ) A, PPSOM - 1-10, -.PSMM ), );;21.
0 1+
LPSOM ~ P.-ISPSOM ), ~ T(-1 A. 1.PSOM A, -.PSOM )
20

---

Published as a conference paper at ICLR 2024
Within, P denotes the initial distribution S. Analogously, define the random sequences and subsequences of 1) the controlled agent's observed trajectory and 2) the controlled agent's 's me trajectory under TAO algorithm, respectively, follows:
1) pe (h; W) (O).pre A ..... O A )
2) Epr (h; W) = (S A A Spm
).
These two types of trajectories generated in the following manner:
- (STt) ST Ps 0 pre
P(O)
A...ie Pppe(-1, W, W, 1, (i 1;W)), A, 1. # (-10,
1,p ), 1,
0. pre 1+1 P(-1 S
T A! A.A 1.re ) 2.
To simplify matters, we will refrain from explicitly referencing W for E and in our notations, except when required to avoid confusion. Next, we introduce a common assumption to ensure the neural network fits the pre-training data distribution.
Assumption D.1 (Learned model is consistent). Let M3 denote the pre-trained model derived by TAO. For any given (O: W, Ep 1;W)), Ppre (A..pre O..ete W, W., 1; Ma(A. pre 0 pre W, W, - 1; W)) for all possible A!
Upon Assumption D.I
we will limit attention Prre for the rest of the proof.
To prove that VEhr P(S (,, - " PPSMM) P(E W, क - Mo)) (i.e., Theorem 3.1 it is equiva- lent to prove that
P(EpsOM (H) = E = P(= (H) = Eh-
(8)
We will prove that Vh [ff],
P(EPSOM(h)) = <% = P(=pee (h) 5%
(9)
using Mathematical Induction and then introduce two lemmas for the evidence of Eq. (8) Here, 5% is controlled agent's true history, and it can be obtained by replacing all the in c), to s.
First, we propose lemma to assist in proving Eq. (9) for the base case when h = 1.
Lemma D.2. If the sampling of a- from Tore independent of the opponent policy, then Ppre(T ()) = P(TpsoM = π WW).
Proof. Assuming the sampling of o from T independent of the opponent policy, we have:
P(
(W) Ipre P(W)
(10a)
x Ipre (
II प (a7
(10b)
jE((W]
x Ipre
II jE[[W]]
-- 10, (TT=(((77)
(10c)
-Ippe 1-1 (W:
(10d)
α PTre(T -I
(10e)
x denotes that the two sides are equal up multiplicative factors independent of Eq. 10a
is derived through the Bayesian posterior probability formula. Eq. (10b) uses the fact that in
posterior sampling independent of the opponent policies. Eq. (T0cp holds because of the assump- tion that the sampling of o from Tpe is independent of the opponent policies. Eq. {10d) uses the
definition of T-= Eq. (10el is derived through the Bayesian posterior probability formula. n
21

---

Published as a conference paper at ICLR 2024
Now, we prove that Eq. (9) holds when h 1:
P(EPsOM(1)
( P(S A1 P.P.
AI.MMM a²))
(Ila)
= p(si) /
P(A1 P(AA.PSOM P.OMMM 0 PSOM = { s1)
(11b)
o)
=((s1)
P(A = at, PPSOM x 100 P() dof
(11c)
p((1) m-1,0)
T"a0(aHoot:T 1) PPPOM (TssOM(TSSOM ", ...MMMM of) P(o dof
(1ld)
= p((1)) /
Tt T "PsoM =
n "= "p))pppoosssss do
(11e)
T-1
= p(s1)
(11f)
=p(s1) / Ppo(atto) W) P() of PP(Er (1) = )
(11g)
(11h)
Eqs. (Ia) lld). tional probability formula based on Eq. 7 Eq. (ele) holds because the sampling of of is indepen-
and I1h are derived using Bayesian Law of Total Probability and condi-
dent of -! Eq. 11 is derived through Lemma D.2
Now, we start proving Eq. (9) for the other cases when h # 1. Before starting, we introduce another lemma to help finish this proving process as well as the proof Eq. (8)
Lemma D.3. Given any en, if P(Ersou = 4 = P(Epre (1) {(²), then P(Epsons()
(i) = =
P( pre (1) = 4.
Proof.
P(EpsOM = =
P(EpsoM( = P(E) )dg!
(12a)
Jez
P(Epro (1)= = P Ef dde
(12b)
Je
= =P P(E (i) =E!)-
(12c)
Based on Eq. 11 we have P(E (1) = <1) = P(= (1) = E.).
Now, we utilize the inductive hypothesis to demonstrate the validity of the entire statement. Suppose that P(EPsoM(AA (h-1)) = 1 = P(E (h- 1) e/1 1 according to Lemma D.3 we can conclude
that P(Epson
- P( (h- 1) 4
Since
P(EPsOM(()) = 5 =
P(EPsOM - P 8h, 1 1.PSOM = af EPSOM (h- 1
-1
and
P(Epc = E =
P( (h- 1) = 1 = 3/, A a 1) = 1h-1),
to prove that P(EPsoM(1) (h) = </2 (h)rr(()) en ), it is equivalent to prove:
P(SpSOM S/1, 1.PSM af. EpSoM (h- 1) 1
(13)
-PSSTTe Sh, A 1.pr = d E (h- 1) = =hh1)-
22

---

Published as a conference paper at ICLR 2024
By expanding the formula, get:
PlS Sh. A A..SSOM a), EpSOM (h 1) 1h-1)
T(s 8h-1, 11: 1.PMM = = a,1.s%5000 S/SSMM = 3/: EpsoM - 1) = Eh-1)
(14a)
S ((3p/5h-1100 - an a- | 1a0-1107 P 1 1
on 1
PSOM o/e TPSOM
10,, PSOM 0/. =psOM (n 11 -hh-1)
P(ch-115h-11 P (
Ploglsa do
(14b)
In Eq. [1b). the first integral term is independent of the algorithm choice, while the terms in the
second integral term satisfies:
P( h
1.POM = ai,, TTSOM ग. 10) = o =psom = 1
(15)
n x 1 = n T o E (h 1) 4
Based on Eq. and Eq. 15 to prove that Eq. (13) holds, it equivalent to demonstrate:
P PSTMM
P.OM
o ElsoM 1) ch- Pree (T 1o W, W,Ch.1).
(16)
We prove that Eq. 16 holds through the following analysis:
P P(TiSOM=
1 PSOM o/, =psom (h 1) 2h-11
P(O) 1.O.MM
oh EpsoM 1) = - = π 11 P(TpSOM = x 11)
(17a)
P( PSOM = ohh EpsoM = E
h
x Ppre(t (
II T क - (a
(17b)
iE A-
α Ppr((T (TTTT
II ग 1
(a 1
(17c)
EE 1-1
α Ppre ( query (o), -
II ) sE[b-1]
(17d)
x Prcc (T= W. 1)
(17c)
Eq. (17a) is derived through the Bayesian posterior probability formula. In Eq. 17b). we decompose the probability of observing that sequence of observations o² and actions a' based on P(E (h
1) = 1 = P(Epee (h - 1) = 1
). Eqs. (17c) and (17d) use the fact that the sampling of of is
only related to the true opponent policy --1 and is independent of π Eq. 17c is derived by the definition of (o- W, 1)
Therefore, we finish the proof of P(EPsoM(hi) = ( P(Epe(h) = 5% where
P(EpsoM(h)) = E%l
= P(Epre(h 11) 1 (8
Tt. ('"'ooo n P ( 10 W, W.E. 1
(18a)
= =P(Epo(hh - 1) = 11)
f Ppolan W, DFP
(18b)
(h) =
(18c)
Thus, based Mathematical Induction, Eq. (9) holds for any h [H). According to Lemma D.3
we can conclude Yh e [H], P(EpsoM()) = (n = P(=) pre (h) = (. Hence, Eq. (8) is satished. This concludes the proof.
23

---

Published as a conference paper at ICLR 2024
D.3 PROOF OF THEOREM 3.2
Theorem 4.2. When the true opponent policy #- originates from 11 the PSOM algorithm using _o''aa") aa tuples as in-context data guarantees to converge, and it converges to the optimal solution r!') tuples as in-context data converges to the optimal solution.
if the original PS using (o' aa' ,,',r")
Proof. In this section, denote the OCW consisting of (o- aa-' tuples to contain in-context data as W, and the OCW consisting of (o' a', o' r'') tuples to contain in-context data as W,, where of is the next observation of transitioned to. Note that we use (o-1, aa_ r-1 tuples as in-context data in the original TAO, which provides more information than using (o a aa tuples. Thus, the guarantees analyzed here apply to the original TAO.
To begin, we propose lemma and its corollary to demonstrate the convergence guarantee of the PSOM algorithm and analyze the properties of the opponent policy converges to.
Lemma D.4. Let To
arg a f( ;:)) arg min -T fo-1.--1
Pw a 1a-11)1 log P(a Poa-110-1 x - then (f(x-1:W) #
11T (
P(r 10 ) P O. P(m. (o--,a-"))a mm))
Proof. Here, To denotes the equivalent class of opponent policies to which the posterior probability
converges with non-zero probability. Pw the distribution of (o-1, aa 1 samples in W. n is the
length of the consecutive (o-1, sequences. To prove that P(r 1(0 la )mm) P. 0 under the given conditions, it is equivalent to prove:
P(m 11o-1,0-110
-g- log P(T 100 1, 1(m) P. 400.
(19)
P( 1001 -__m)
By expanding the formula, we can get:
T - -log P(T 1( a )=m) log P( log
P(a 1 1-
P(To 1(01, a 1mm)
P(TT
P(a- To
According to the definition of To and the condition f(x ((=-:WW) # f f W), we have
P(a | 1TT10 P(a noo ,0-1) f(T -1 W) f f( ; W) = C> 0.
E B(o-)_a-'>>Pw - logi
Here, C is positive constant. Therefore, based on Law of Large Numbers, we have lim,-
P(] 1--i-_ Cl > e) = 0, where e is any positive number. Hence, Eq. (19) is satisfied and the proof ends.
O
Corollary D.5 (Corollary of Lemma D.4p. When Pw(o 1 ()(o- 1 1,, -1), which means -1 is originated from o then - ap
Proof. Since Pw(lot,a--) =
-1) holds, we have
To arg
min LefTaan
-1 Pwlo-1 a -0-1) log a1 200 1 -da
(20a)
/
arg min Elllm
Bwlo-))P(a 100,-= log(P(a" P(-00aa-a 1/00-,1~1111 -))o0 -'aa da-
1,a
(20b)
arg
min
0-110-1 Pw(o-1))(c 10- 1,- न 11) log(P(a 10-1, d))o da
+
Pwlo (PP(a -0-11 # - log( - 10- - -a Id
(20c)
a-
24

---

Published as a conference paper at ICLR 2024
f 0-1,0-1
Pwlo PP - log P( Pog(P(a-100 1-
arg
min ETlaan
log ( 1
do
da
(20d)
min arg
(20c)
Ellmm
arg min Elllen
E. (o-L) (ar((Paa"lo-11 - 1000=,,"))011(11111111 ,-
(20f)
This corollary holds evidently due to the non-negativity of the KL divergence DxI(P(a >>> 1,,,,,,,,, ,. When W contains the in-context data of the op- ponent policy a-1 from I1T DxL PP=(P(a''oo-1 converges to the equivalent class of opponent policies that have the same action distribution as π i..,, Va P(a '1 To= P(a 1o
| --)) = O, and PSOM
By extension, it can be concluded that when W contains the in-context data of an opponent policy outside of II Lemma enners the convergence of PSOM algorithm to the opponent policy that minimizes (P
| -|o-, म 11 between actions of the converged opponent
policy and the true opponent policy.
Using similar proof methodology as Lemma D.4 can proved straightfor- arg min, f(T f(T==, ,,""
f-1 ward that: Let To
arg rigmin,--emee - - o0 (or "ot,at at o01, rd) log( o="lot,at aalnn x ")) da do then (f (,) # S(= - W")
P(a "Iro" a² o0 4 0.
P(rg "Iool,a Vot. (or,,)))))
Now, introduce novel lemma elucidate the properties of the opponent policies attained through the original PS algorithm employing W while also concluding the relative optimality of the solutions produced by the PSOM algorithm in comparison it.
if Va- P(a " 100- 1 = 1(--11-1,==1 holds, it can be deduced that Vo" .r, P(d" r r'oo', a 1 = P(o", ,r ,''lo', aa, a but the reverse is not true.
1_ --1 Lemma D.6. Given o at, To
11)),
Proof. For the forward deduction (i.e., =) we have:
Vo r1, P(o" 1 a², 1)
Σ P(o 10') " P((', ,1'10' at at,
(21a)
1,a
Σ P(o-olo))P(aa --))) r at
(21b)
0-1,a-1
P(o" "' m
(21c)
For the backward deduction (i.e., F), counterexamples exist. For example, when P(o r r- at, at, 001 -0-,a-11 takes equal values for some a1 Á-1 A-1, it is possible that Va AA-iAA- Po no P(a P(a-"1o-1, -1 and E. 1 P( ". no = Po -11
Ea--EA-1 P(a P(a-11o011.0 1 11), so that -1 E A-1 A-1,PH P(a
does not necessarily holds. This implies that using (o', a², ot, ,_r,r") r1) tuples as in-context data will lead to distributions opponent policies other than the equivalence class of #-1 with non-zero probability, resulting in suboptimality.
According to Lemma D.6 no c mb holds. According to Corollary D.5 we have #- E no Thus, we can conclude that --1 E .o c - Hence, the PSOM algorithm converges to fewer solutions with non-zero probabilities compared to the PS algorithm, thereby providing more optimal solutions. other words, the PS algorithm converges to the optimal solution, then the PSOM algorithm converges to the optimal solution. This concludes the proof.
25

---

Published as a conference paper at ICLR 2024
(a)
(b)
Figure 7: The two environmental benchmarks. (a) The MS environment. We visualize the entire map an initial timestep. (b) The PA environment. We visualize the clipped map at certain timestep.
E ENVIRONMENTAL SPECIFICS
Markov Soccer (MS) (Ltttman 1994 Lanctot et al. 2019): A sparse-reward discrete soccer environment that exemplifies two-player zero-sum game. The game is played on 4 x 7 grid in Fig. 7(a) The game begins with the controlled agent (blue) and the opponent (red) in random squares in the left and right half (except the goals), respectively, and the ball goes to one of them random square in the middle column. The players have five actions: move N, S, W, E. or stand still (i.e., NO-OP), An action is considered invalid if it leads the player to a shaded square (grey) or outside the border. If player has the ball, possession will be exchanged and transition will not take place when the two players move into the same square.
randomly none of them
Both the control agent's and the opponent's objectives are taking the ball into the other's goal (the controlled agent's 's goal is blue, and the opponent's ' goal is red). and the game ends when this hap- pens. The reward sparse, providing + for success, for failure, and for instances where the maximum timestep limit is exceeded, only at the end of the games.
For specific implementation of MS, we adopt the open-source code of OpenSpiel, which is avail- able at uttttps://ggthhui wttp://mmmeeemmi//////.oome
Particleworid Adversary (PA) (owwe al. 2017): dense-reward continuous multi- particle environment that exemplifies a three-playe zero-sum game. PA includes two controlled agents (purple), opponent (red), one normal landmark (black), and one target landmark (green) shown in Fig. [ All agents observe position of landmarks and other agents. There are no borders present the entire game map.
The controlled agents' objective is to minimize the distance between himself and the target landmark while keeping the opponent away from the target landmark. The opponent's objective is minimize the distance between himself and the target landmark which is unknown to him (i.e., the opponent does know which one of the landmarks is the target one). This necessitates that the controlled agents develop the skill of strategically distributing themselves to ensure coverage of all landmarks, effectively deceiving the opponent. The two controlled agents and one opponent are given dense rewards induced by their objectives and states using Euclidean distance as the metric.
For specific implementation of PA, we adopt the open-source code of Multi-Agent Particle Environment, which is available at htttps://qiths com opmna1/ multiagent-paat wwreeett--paitlleeeeeeee Additionally, we introduce a collision mechanism to the orig- inal environment to increase the randomness and dynamics of the environment.
NEURAL ARCHITECTURE SETTINGS
In MS and PA, we adopt the same neural architecture settings as follows:
OPE (Ma_): The backbone of the OPE architecture mainly implemented based on the GPT2 (Radford et al.) 2019) model of Hugging Face (Wolf et al. 2020). The backbone is a GPT2 encoder composed of self-attention blocks. Each self-attention block consists a single-head at- tention layer and feed-forwatt layer. Residual connections (He et al. 2016b) and LayerNorm (Ba) jet al., 2016) are utilized after each layer in the self-attention block. Within each attention layer, dropout (Srivastava et al. 2014) is added to the residual connection and attention weight. Actions
26

---

Published as a conference paper at ICLR 2024
a_ rewards r- observations o-! are fed into modality-peccific linear layers, and a positional episodic timestep encoding is added. We adopt the same timestep encoding as in Chen et al. (2021). Then, we use fusion linear layer to fuse the 0--10 o, embedding tokens at each timestep into fused embedding tokens. Finally, the sequences of fused embedding tokens are fed into the backhone, which autoregressively outputs the per-timestep embedding tokens correspond- ing to each [o-) ro-11, ") tuple using a causal self-attention mask.
In offline stage we use Average Pooling (Gholamalinezhad & Khosravi 2020 to aggregate all the per-timestep embedding tokens output by the backhone, i.e., 200 :2 ..... to obtain the average trajectory embedding x-1 for loss calculation. In offline stage 2 and deployment stage, we input all the per-timestep embedding tokens output by the backbone, i.., 2 (20 21 ....), as key and value into the cross-attention layers of ICD.
the backbone, except for the fully connected layer in the feed-forward layer (the ferd-forward layer consists of fully connected layer that increases the number of hidden layer nodes and projection layer that recovers the number of hidden layer nodes), which consists of 128 nodes with ReLU (Agarap 2018) activation functions, the other hidden layers are composed of 32 nodes without activation functions. The poddllty-speccifee linear layers for actions, rewards, and obser- vations are composed of 32 nodes with ELU (Clevert al. 2015) activation functions.
ICD (Ma,): The backbone of the ICD architecture uses the same causal Transformer as in Chen jt al. (202T), which is mainly implemented based on the GPT2 model of Hugging Face. The backbone is GPT2 decoder composed of three self-attention blocks. Each self-attention block consists of a single-head attention layer, a single-head cross-attention layer, and a feed-forward layer. Residual connections and LayerNorm are utilized after each layer in the self-attention block. Within each attention layer, dropout is added to the residual connection and attention weight. RTGs G¹ observations o², actions a² are fed into modality-speciffe linear layers, and positional episodic timestep encoding is added. The sequences consisting of embedding tokens of G of a are fed into the backbone, autoregressively predicts actions of using causal self-attention mask. Specifically, the obtained hidden states at the positions of of embedding tokens are fed to linear layer to output actions.
In offline stage 2 and deployment stage, we input all embedding tokens output by the back- bone as query into the atosstattention layers of all self-attention blocks. For the original in- put sequence wt (G), og. ag. .... G1, of) ). we truncate the latest If timesteps of it, ie.. HI1...., H G. of). as input.
yf² = (G:- - In the backbone, except for the fully connected layer in the feed-forward layer, which consists of 128 nodes with ReLU activation functions, the other hidden layers are composed of 32 nodes with- out activation functions. The modality-speciff linear layers for RTGs, observations, and actions are composed of 32 nodes without activation functions.
Ancillary decoder for PEL (TT.)): The model of Thy consists of 2 linear layers, with observation 01, " policy embedding -1 as input, and action a as output. Among them, o and = are concatenated and fed to the linear layers, and LayerNorm is added after the first linear layer. All hidden layers comprise 32 nodes: the first hidden layer includes a ReLU activation function.
DRON-concat: For a fair comparison, use the GPT2 decoder to implement the backbones of the DRON-concat architecture. The DRON-concat architecture contains 2 backbones, and the backbones' architectural design is identical to that of ICD, except that it does not include the cross-attention layer. One of the backbones is used to encode the sequence consisting of RTGs G observations o²,, and actions at, while the other backbone is used to encode the opponent's hand-crafted features (He et al.] 2016a). To be versatile in different environments, we consider using the opponent's previous action and the opponent's 's previous action frequency as the hand- crafted features. We concatenate the embedding tokens obtained by the two backbone encodings in the dimension of the hidden state and feed them to a fusion linear layer to get concatenated embedding tokens. Within, the hand-crafted feature embedding tokens are aligned with the posi- tions of a² embedding tokens, and other positions are filled with 0 and aligned with the positions of G1, a² embedding tokens. Finally, given the concatenated embedding tokens, the actions at are autoregressively predicted at the locations of observations of using the same manner as ICD. Both the fusion linear layer and the linear layer of hand-crafted features comprise 32 nodes with- out activation functions, while the number of other hidden layer nodes and the activation function settings are identical to those of ICD.
27

---

Published as a conference paper at ICLR 2024
DRON-MoE: For fair comparison, use the GPT2 decoder to implement the backbones of the DRON-MoF architecture. The DRON-Mo MoE architecture contains 2 backbones, and the back- bones' architectural design is identical to that of ICD, except that it does not include the cross- attention layer. One backbone is used to encode the sequence consisting of RTGs G², observations o², and actions at, while the other backhone is used to encode G o at sequence and the oppo- nent's 's hand-crafted features. For the first backbone: After encoding G], of, aj. the of embedding tokens are fed an expert linear layer and output a predefined number of expert of action prob- ability distributions. For the second backbone: Before inputting, the opponent's 's tures are concatenated with the corresponding observations o' according to the timestep, and they are fed to mixing linear layer; after encoding G., 0/, a and hand-crafted features, o' embedding tokens are fed to an expert linear layer, and a weight vector with a length of the number of experts is outputted. Finally, the obtained weight vector is used to perform weighted summation of the obtained action probability distributions, and the actions a; are predicted autoregressively. The mixing linear layer consists of 32 nodes without activation function, and the expert linear layer consists of 32 nodes with Softmax (Bridle, 1989) activation function. The number of other hidden layer nodes and the activation function settings are identical to ICD. The predefined number of experts is set to 5.
fea-
LIAM: For fair comparison, we use the GPT2 model to implement the backbones of the LIAM architecture. The architectural design of LIAM's backbone the same as that of ICD, except that it does not include cross-attention layer. In addition to feeding the sequence G of af into the backbone and predicting the action af in an autoregressive manner using a causal self- attention mask, we also employ an extra decoder to learn auxiliary task (Papoudakis et al.] 2021). This auxiliary task involves reconstructing the opponent's 's observations o, and actions a. using the observations of and actions -/_1 of the controlled agent. Specifically, we use the of token embeddings obtained through the backbone as input to the extra decoder to predict of and ar', as the of token embeddings contain all the information of of and a!_1- The extra decoder consists of two linear layers with 32 nodes and no activation functions. The number of other hidden layer nodes and the activation function settings are identical to ICD.
MeLIBA: For fair comparison, use the GPT2 model to implement the backbones of the McLIBA architecture. The architectural design of McLIBA's backbone is the same as that of ICD, except that it does not include cross-attention layer. In addition feeding the sequence Gl,of of, af into the backbone and predicting the action at in an autoregressive manner using a causal self-attention mask, we also employ extra encoding layers and an extra decoder learn an auxiliary task for the Variational AutoEncoders ( (VAE) (Zintgraf et al. (021). This auxiliary task aims maximize Evidence Lower BOund (ELBO) by reconstructing the opponent's 's future actions a -1 " using the opponent's future observations o, , 1 ... and by minimiz- ing a certain form of KI. divergence, conditioned on the past trajectory ob, ag. rb. ... .oj of the controlled agent. Specifically, we use additional two-level hierarchical linear layers to encode the controlled agent's 's past trajectory, where the two levels embed the controlled agent's past trajec- tory into the Gaussian distribution parameters μ and a of a permanent latent variable (also called agent character) and temporal latent variable (also called mental state), respectively. On the one hand, i and a of the permanent latent variable and the temporal latent variable output the con- trolled agent's actions to respond to the opponent reasonably. On the other hand, the permanent latent variable and the temporal latent variable, sampled from the Gaussian distribution param- eterized by i and o, are used as the input to the extra decoder, where this decoder conditions also on the opponent's 's future observations o ,04+, ... to predict the opponent's 's future actions a, .. The extra encoding layers consist of two levels of linear layers, where each level has two linear layers with 32 nodes and no activation functions. The extra decoder consists of two linear layers and recurrent layer, where these layers have 32 nodes and activation functions. The number of other hidden layer nodes and the activation function settings are identical to ICD.
Prompt-DT:
We implement Prompt-DT's architecture directly based on its open-source code, which is available at httpss: / htt///itttii hithtb..hhuugggggggggggg.... com/ mxu34 /////3////rpppp dt. We use high-quality data against opponent policies from offline datasets expert demonstrations (i.e., prompts). High- quality data refers to trajectories of the controlled agent that rank near the top in terms of return across all episodes in which it competes against specific opponent policy. Specifically, the top 20% for the high-quality data, sample 1 trajectory, and then sample consecutive fragments with length of from the trajectory as prompts. The linear layers for RTGs, observations, and ac-
select
28

---

Published as a conference paper at ICLR 2024
tions in the prompts consist of 32 nodes without activation functions. The number of other hidden layer nodes and the activation function settings are identical to ICD.
G DETAILS OF BASELINES
DRON-concat (He et al. 2016a): An opponent modeling approach that encodes the opponent's hand-crafted features with auxiliary network. It concatenates the resulting opponent's ' hidden states and the hidden states obtained by the controlled agent's 's observation to aid downstream policy optimization of the controlled agent.
DRON-MoE (He et al., 2016a): An opponent modeling approach that encodes the opponent's hand- crafted features using a Mixture-off of- off-reert (MoE) structure (Masoudnia & Ebrahimpour (2014). considers the hidden states obtained by the opponent's 's hand-crafted features and the controlled agent's s observations as a gating signal, applying a weighted summation with predefined number of expert of the hidden states obtained by the controlled agent's observations.
LIAM (Papoudakis et al. 2021): An opponent modeling approach that encodes opponent infor- mation with autoencoders (Hinton & Zemel 1993). It utilizes the observations and actions of the controlled agent to reconstruct those of the opponent, thereby embedding the opponent policy's s semantics into latent space. Then, it uses the obtained embeddings to assist downstream policy optimization of the controlled agent.
MeLIBA (Zintgraf et al., 202T): An opponent modeling approach that encodes opponent informa- tion with Variational AutoEncoders (VAE) (Kingma Welling 2013). It uses VAE to reconstruct the opponent's 's future actions to infer an approximate belief for the opponent. It aims to learn an approximate Bayesian optimal policy based on the embedding generated by the VAE.
Prompt-DT (Xu et al., 2022): An offline meta-RL approach based on the Decision Transformer (DT) (Chen et al.) 2021 and prompt learning (Liu et al. (023). Based on the DT, it samples expert task trajectories as prompts and obtains task adaptability through offline dataset pre- pre-training. This approach holds the potential to partially address the challenges posed by the OOM problem.
TAO without PEL: A variant of our approach. It performs concurrent optimization ICD and OPE utilizing only Lres, without pre-training OPE using Lemb. We introduce this baseline to conduct ablation studies for PEL and scrutinize the efficacy of TAO's architectural design.
29

---

Published as a conference paper at ICLR 2024
H DETAILS OF OPPONENT POLICIES
H.1 OPPONENT POLICIES IN MARKOV SOCCER
Opponent Policy Name
Source
Description
Snatch: Move towards the football along the shortest path when not possessing the ball.
SnatchattackPolicy
script
Attack: Move towards the goal of the trolled agent nearest the opponent while possession of the ball, following the shortest path.
Snatch: Move towards football along the shortest path when not possessing the ball.
SnatchEvadePolicy
script
Evade: When possessing the ball, move along the path farthest from the controlled agent.
This policy is obtained through self-play training using TRCoPO (Prajapat al. 202T) algorithm 30000 episodes.
The open-source code is avail- able at /tttp:://gtthnn thub.coom mrniish-rraircone
ALPolicyl
RL algorithm
We adopt original hyperparameter settings from the open-source code.
This policy is obtained through self-play training using the TRGDA (Prajapat et al. 2021) algorithm for 10000 episodes.
The open-source code is avail- able at https://githut com/ raaishh-ra/trcope
MuPolicy2
RL algorithm
We adopt the original hyperparameter settings from the open-source code.
This policy is obtained through self-play training using the PPO (Schulman et al. 2017) algorithm for 50000 episodes. The open-source code is avail- able berkeLeyydeprrcooure honework/tree http:/ttteeemateeeeeeeeeee httm///atteer//eeee
https::/github.co
ALPolicy3
RL algorithm
We adopt the original hyperparameter settings from the open-source code, and the other opponent policies trained using PPO also use the same settings.
Guard: When not possessing the ball, de- fend front the opponent's goal nearest controlled agent.
o6 GuardAttackPolicy
script
Attack: Move towards the goal of the con- trolled agent nearest the opponent while in possession of the ball, following the shortest path.
Guard: When not possessing the ball. de- fend in front of the opponent's goal nearest controlled agent.
07 GuardEvadaPolicy
script
Evade: When possessing the ball, move along the path farthest from the controlled agent.
This policy is obtained through self-play training using the PPO algorithm for 100000 episodes.
68
RLPolicy4
RL algorithm
30

---

Published as a conference paper at ICLR 2024
H.2
OPPONENT POLICIES IN PARTICLEWORLD ADVERSARY
Opponent Policy Name
Source
Description
At beginning each episode, ran- domly select one of two landmarks as the fixed target. Throughout this episode, al- ways move towards the fixed target along the shortest path.
ol
FixOnoPolicy
script
At the beginning of each episode. randomly select one of the two controlled agents the chasi ng target. Throughout this episode, always move towards the chas ng target along the shortest path.
02 ChaseOnePolicy
script
At beginning of each episode, the midpoint of the two landmarks as the fixed target. Throughout this episode, al- ways move towards the fixed target along the shortest path.
o3
MiddlePolicy
script
At the heginning of each episode, randomly select one of two landmarks the f xixed target. During this episode, begin by mov- ing towards shortest path. When the distance to the cur- rent fixed target is less than or equal to 0.096, switch the fixed target to the other landmark and then move towards the new fixed target along the shortest path. Repeat this process until the episode ends.
fixed target along the
o4
BouncePolicy
script
This policy is obtained through self-play training using the PPO algorithm for 5000 episodes.
o5
AlPolicyS
RL. algorithm
At the beginning cach episode, ran- domly choose one the two landmarks or the midpoint of the two landmarks as the f fixed target. Throughout this episode, al- ways towards the fixed target along the shortest path.
FixthreePolicy
script
At the beginning of each episode, randomly select one of the two controlled agents the chasing target. During this episode, be- gin by moving towards the chasing target along the shortest path. When the distance the current chasing target less than or equal 0.3, switch the chas ing target other controlled agent and then move towards the new chasing target along the shortest path. Repeat the process until the episode ends.
07 ChaseBounccPollcc
script
This policy is obtained through self-play training using the PPO algorithm for 10000 episodes.
08
RLPolicy6
RL algorithm
31

---

Published as a conference paper at ICLR 2024
I CONSTRUCTION OF OFFLINE DATASETS
The offline dataset, denoted as Deft (D* k=1,,,..... where is the predefined number of oppo- nent policies. For each E [K], D consists of totally A episodes' interacting trajectories, generated by the interactions between the opponent policy π 11,4 against it.
and the approximate best response policy
However, bring research in line with real-world scenarios where the offline datasets always noisy, we introduce certain modifications: We assume that D E [K] comprises a larger propor- tion, 7), of expert data and a smaller proportion, 1 1) of noisy data. The expert data is derived from the interactions between the opponent policy -- and its approximate best response policy π 1,8. The noisy data procured by interacting with -1 through any differing policy, e.g., play against o 1,1 with π 1.2..
In this work, we set the opponent's 's policy to -11 and utilize pre-training with an RI. algorithm to derive opponent policy л 111ss approximate best response policy Il.k,* Additionally, a rough but practical approach to attain æl.k,* involves considering the policies generated by top-ranking algorithms in particular competitive contests as approximate best response policies against 1,k
For the specific implementation
offline dataset Doll, we adopt the following settings:
In both MS and PA, K to 5, A 2000, and 1 to 0.9;
In both MS and PA, for each & E [K], fix the opponent policy - and use PPO Schulman et al. 2017) to pre-train its approximately best response policy π Tl,k;
both MS and PA, for each & [K], the noisy data generated by the interactions between 1,k and all possible m-...* where j E [K]A] Aj + k, each for 50 episodes (iee. accounted for
1- of the D
K-I
J HYPERPARAMETERS
J.1 HYPERPARAMETERS FOR OFFLINE STAGE
Hyperparameter Name
MB
PA
Dimensionality of the opponent's 's observations
12
Dimensionality of the opponent's actions
5
Whether normalize the opponent's observations
True
Trus
Maximum horizon length for each episode number truining steps Ratch (F3)
100
100
Number of updating epachs at each training step
10
Weighting coefficient for generative loss Cera (c)
1.0
1.0
Weighting coefficient for discriminative loss Cai (A)
1.0
1.0
Warm-up epochs (the learning rate multiplied amngpahi1 lincarly during initial warn-up opochs of training)
spedi allow it
10000
10000
Learning rale for AdamW [Loshchilov Hutter 2018 optimizer
x 10
x 10
Weight decay coefficient for AdamW optimizer
x 10
x 10
Maximum the gradients (clip excented))
0.5
0.5
Temperature factor discriminative Number tailed descriptions)
layer
encoder (see Appendix
32
32
Dropout factor for the GPT2 encoder of OPE Number of self attention blocks for the GPT2 encoder of OPE
0.1
0.1
Number attention head GPT2 enceder of OPE
Random seeds
{0, 1,2,3,4)
(0, 1,33,4
32

---

Published as a conference paper at ICLR 2024
J.2 HYPERPARAMETERS FOR OFFLINE STAGE 2
Hypergarameier Name
Dimensionality opponent's observations
12
Dinensionality of the opponent's ss actions
Whether to normalize the opponent's observations
True
True
Dimensionality of the cuntrolled ugent's 's observations
12
10
Dimensionality of the controlled agent's 's actions
5
Whether normalize the controlled agent's observations
Truc
Truc
Maximum horizon length for each episode
100
Length consecutive fragment each trajectory sampled offline dateset
D by OPE (H)
20
20
Number of trajectorics sampled OPE (C)
5
Length of the input sequence for ICD after iruncation (ff) (see Appendix for detailed
descriptions)
Reward scaling factor (all the rewards are multiplied by seoan wellng laoe. to reduce the variance of training)
Total number of training sleps
2000
2000
Batch size (B)
128
128
Number updating epochs each training step Warm-up epochs
10000
Learning rate for Adami optimizer
X 10
X 10
Weight decay coefficient for AdamW optimizer
10
10
Maximum
gradients
0.5
Number hidden layer nodes
encoder
Dropout lactor the GPT2 encoder of OPE
0.1
0.1
Number of sell atlention blocks for the GPT2 encoder OPE Number of attention head for the GPT2 enceder of OPE
3
1
Number of hidden layer nodes for the GPT2 decoder of ICD Dropout factor for the GPT2 docoder of ICD
32
32
0.1
0,1
Number of self-attenting blocks for the GPT2 decnder of ICT)
Number
GPT2 decoder of
Random seeds
{0,
2, 4]
1,2,,2,3,4)
J.3 HYPERPARAMETERS FOR DEPLOYMENT STAGE
Hyperparameter Name
MS
Dinensiomality of opponent's observations
12
Dimensionality opponent's
Whether normalize the oppoment's observations
True
Dimensionality controlled agent's observations
12
Dimemsiionlity of the controlldd ugent's 's actions
5
Whether to normalize the controlled agent's observations
True
True
Maximum horizon length for each episnde
100
100
Total number of episodes for testing
2500
2500
Opponent policy switching interval (E) (the opponent switches policy every F.
opisodes)
Length of the consecutive fragment for each trajectory sampled from the OCW by OPE (H
20
Size the OCW (C) (the OCW collects the latest opporeent trajectories)
5
Length of the inpul sequence for JCD after truncation (H)
20
Reward scaling factor
1
100
Number of hidden layer nodes for the GPT2 encoder of OPE
32
32
Dropout factor for the GPT2 encoder of OPE
0.1
0.1
Number of self self-attention blocks for the GPT2 encoder of OPE layer nodes
Number
GPT2 enceder of decoder ICD
Number
Dropout factor GPT2 decoder
Number of well atlention blocks for the GPT2 decoder ICD
3
Number of attcution head for the GPT2 decedor of ICD
1
Random seeds
1,
(,,,,4))
33

---

Published as a conference paper at ICLR 2024
Pamppor
coun))
MS seen
MS unseen
PA seen
PA unseen
200
200
Retum
0.4
0.4
100
100
1000
100
0.0
0.0
200
-0.
-0.2
300
-300
-0.4
-0.4
400-
400
Trining
Training Step
Training Step
Training Step
Figure 8: Performance for offline response policy training. All approaches are evaluated against each opponent policy in seen and wnseen for 100 episodes separately every 200 training steps. We report the average and standard error of the mean results against all opponent policies over 5 runs.
Table 2: Experimental results on adaptation speed.
XS (Avg. Min. Episode J)
FA (Avg. Min. Episode L)
aussee
mix
snom
mir
TAO (ouns)
2,74 1: 0.27
3,78
1 0.93
2,83 1: 0.32
1,9 + 0.2
3,7
0.5
2.8
0.2
PEL (ours)
DRON-coneat
14.53 + 1.78
29.27
1.44
19.60
2.10
5.4
54
72211
52 1.0
DRON MaE
13.99
0.97
6.62
1: 1.86
6.95
1: 1.50
3.5
0.3
6.7
17104 04
49
0.5
LIAM
6.90 +: 0.88
7.66
+ 0.35
5.78
+ 0.68
0.
10.4
6.5
McLIBA
6.12 0.74
12:51
+ 1.75
6,36 + 1.08
2.9
13.3
6.8
0.8
Prompt-DT
5.67
2.29
2.1
16.7
75
K ADDITIONAL EMPIRICAL ANALYSIS
To further examine TAO's effect and underlying mechanisms, we continue to ask the following questions and design corresponding experiments to answer them:
Question 5. How does TAO perform in the offline training compared with other baselines?
In Fig. S we present the average results of playing against opponent policies in the opponent policy sets separately (i.e., without switching opponent policies) during offline response policy training. The evaluation outcomes for TAO exhibit a consistent and incremental improvement under both seen and unseen settings as the training progresses. In contrast, other approaches like DRON-coneat display relatively unstable performance during training.
The evaluations during offline training and the subsequent testing results in Fig. demonstrate that the relative performance of all approaches remains constant. However, the performance of most approaches during testing has shown certain improvements compared to the last evaluation of the offline training. This observation can likely be attributed to the increased number of episodes con- ducted during testing, offering more chances to exploit the opponent policies with low strength (i.e., weak and exploitable opponent policies).
Question 6. Does TAO demonstrate faster adaptation speed compared with other baselines?
We measure the adaptation speed by the minimum number of episodes to achieve specified return. For all opponent policies in the seen test setting, the specified return is set to be the average return of all trajectories against this policy in the offline dataset; for all opponent policies in the unseen test setting, the specified return is set the average return of all approaches against this opponent policy under the unseen test setting.
Table 2 shows the average adaptation speed under the original opponent policy configuration. TAO showcases consistently faster adaptation in MS and PA than other approaches, underscoring its ca- pability to acquire knowledge through in-context learning swiftly. Furthermore, it's important to highlight that there isn't straightforward positive correlation between adaptation speed and per- formance. For instance, in scenarios involving unseen opponent policies, Prompt-DT performs ad- mirably but often displays sluggish adaptation speed.
Question 7. How does the opponent context window (OCW) size C' affect the efficacy of TAO?
We vary the OCW size C' to investigate how the context length affects TAO, and the relevant perimental results are shown in Fig. 9 In both environments, the average performance exhibits a similar trend pattern as C changes. worth noting that, C' increases, the average performance
34

---

Published as a conference paper at ICLR 2024
unseen
0.40
0.35
0.30
0.25
0.20
0.15
10
20
10
20
1
5
10
20
OCW Size
OCW Size (C)
OCW Size (C)
200
Avg. Retur
180
160
140
120
100
10
20
5
10
20
5
10
20
OCW Size (C)
DCW Size (C)
OCW Size (C)
Figure 9: Average performance for TAO on different sizes OCW,
does increase monotonically under the three test settings. Under the unseen and mix settings, continual decrease in performance is evident, beginning at C 5. One possible explanation is that when relatively large, due to the non-stationary opponent policy switching, TAO is prone be influenced by the context of the previous opponent, making it challenging to recognize the current opponent accurately. The quest to theoretically and empirically identify the most suitable value for C' is reserved for future research endeavors.
L LIMITATIONS AND FUTURE WORK
1. We only focus on exploiting opponent with fixed policies. TAO might face difficulties han- dling opponent who exhibit policy alterations within an episode or evolve (update) their policies as the games unfold. OOM with changing policy opponents represents formidable but crucial problem;
2. We do not consider recursive reasoning. i.e., scenarios where opponent also perform op- ponent modeling. A challenging future research direction lies in learning opponent models for recursive reasoning using offline datasets;
3. Our empirical analysis merely models one opponent agent in the environment. In situations involving a larger number of agents to be modeled, it becomes imperative to account for the interrelated behaviors and correlations among these agents to ensure the efficacy of modeling:
4. This paper's scope is limited to examining OOM problems exclusively within competi- tive settings. Our future research will focus on understanding and mitigating the distinct challenges that arise in cooperative and mixed environments.
35


Reviews:
[{"review_id": "YI5cGIXfjk", "reviewer": "Reviewer_fLaD", "summary": "This paper proposes an offline RL method specifically for domains with other agents.\nThe method consists of a transformer model that is used to encode the opponent trajectory (called opponent policy embedding, OPE) in some latent space and a transformer that, with this encoding as input, is trained to predict the best action through behavioral cloning (called in-context control decoder, ICD).\n\nOPE is trained with two (additive) losses of which one is based on the likelihood of the predictions over the opponent policy and the other on a discriminative loss over the trajectory index.\nICD is trained on, as far as I can tell, the typical behavioral cloning loss (likelihood of predicting the action take in the offline data set).", "questions": "In what way, during execution, do we assume that the agent has access to the opponent trajectory?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "This paper tackles a difficult problem, that of adapting quickly to (sometimes unseen) opponents.\nIt does so with an arguably scalable approach (e.g. with transformers) that can be deployed online is a real-time manner by pre-computing the heavy work that many algorithms do online instead (to infer their opponents policy).\n\nThe solution, additionally, proposes a significant amount of clever and proper uses of existing techniques to tackle the many obstacles that come with a complex problem like theirs.", "weaknesses": "My main concern is on the lack of clarity and theoretical rigor in this paper.\nAdditionally, while there was motivation for the problem setup and high-level design choices, most of the proposed method lacked description or reasoning.\n\nFor example, a common occurrence in the paper is the mathematical description of a loss used with barely any description of what it achieves (I *believe* equation 5 is typical behavioral cloning; supervised learning on the action in the data set, but I had to infer this), often including cryptic terms as \"GetOnD\" which \"samples fragments from each trajectory of W and then stitches them together\".\nIt was sometimes unclear what even the intended task was of a particular component in the model and necessary to reverse engineer it from the proposed solution.\n\nLack of rigor is especially visible in the theoretical analysis where the first theorem reads as \"assume that (...) M is (...), then <statement>\" but (I assume) the notation is such that <statement> does not even mention \"M\", making it unclear why that assumptions affects the statement at all to begin with.\nThe second theorem states their method has \"better optimally guarantees\" but does not mention what \"better\" means in this context.\n\nAs a result this paper reads as a recipe for training a particular model to solve their a problem setting of which the generality is difficult to judge.\nThis is to the detriment of the paper because, despite theirr focus on opponent modeling, I would not be surprised if (with some notation/generality) adjustments this could just as easily be applied to any (non-opponent) meta-learning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qSG4kc7uKo", "reviewer": "Reviewer_QAav", "summary": "This paper proposes a new setting for the opponent modeling problem (OOM) where environment and opponent policies are unavailable. Under this setting, the authors introduce a novel model, named Transformer Against Opponent (TAO). TAO adopts Transformer’s capabilities on sequence modeling and in-context learning to learn discriminative policy embedding and predict response actions. The authors provide a theoretical analysis that establishes TAO's equivalence to Bayesian posterior sampling in opponent modeling and guarantees TAO's convergence in opponent policy recognition. The paper also includes empirical analysis to demonstrate TAO's effectiveness in achieving competitive results against other approaches for the OOM problem.", "questions": "-\tDoes GPT2 stay frozen or get fine-tuned in policy embedding learning?\n\n-\tUnder the setting of OOM, no opponent policy should be available for learning except for trajectory sampling. Does the discriminative loss cause policy label exposure to the model and violate the setting?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is overall well-structured and easy to follow. The authors provide a comprehensive analysis of the problem they are addressing and the existing work in the field. \n\n2. The new setting of learning opponents offline seems reasonable to me. And the idea of modeling opponent policies as an in-context learning problem is interesting. \n\n3. Experiment results demonstrate the superior performance of TAO on the OOM problem.", "weaknesses": "My concerns mainly lie in the experiment section.\n\n-\tThe authors should give a more detailed description of the constructed datasets which are the key points of this paper.\n\n-\tFor the mix type, what is the proportion of the number of the opponent policies between seen and unseen? How do the models, including TAO and baselines, perform on different proportions?\n\n-\tWhat’s the meaning of “# of opponent policy” in Fig 3b and Fig 6b? Does it stand for the number of policy items or the number of policy types, for seen or unseen or total? From the results in Fig 3b and Fig 6b, in the PA dataset, the model's performance exhibits a noticeable trend of initially improving and then declining under different number of opponent policies. However, in the MS dataset, while the model's performance fluctuates, there is no clear trend. Why is there such a difference and what causes this performance variation?\n\n-\tThe authors should also provide a visualization of unseen policies in Fig 5a to make the demonstration more convincing.\n\nMinor issue:\nPara 2 in introduction, numerous real-life …**: For instance,** …", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0P5QmeWDWA", "reviewer": "Reviewer_Z2nF", "summary": "This work studies the offline opponent modeling problem. To tackle this problem, the authors first use a policy embedding learning process to learn the embedding of opponent policies. This process adopts a generative loss to imitate opponents and a discriminative loss to discern different opponent policies. Next, the authors introduce a decoder that conditions the opponent policy embedding and the controlled agent's trajectory and outputs the agent's action. Through the supervised training based on offline data, the decoder learns to respond to the opponent policy given the policy embedding. After deployment, the proposed method collects the latest trajectories of opponents, generates corresponding embeddings, and samples the actions that adapt to the current opponent policy.", "questions": "1. In section 4.2, the function GetOffD begins by sampling $C$ trajectories from $\\tau^{-1,k}$, following which it samples $H$ consecutive\nfragments from each trajectory and ultimately stitches them together. The reason for this design should be explained more clearly. Specifically, during deployment (section 4.3), the method collects the latest $C$ trajectories which are reasonable. However, during the training, the sampled $C$ trajectories are often from different periods which may not be very related to $y_t$ and $a_t$. Why this would help with the training?\n\n2. In experiments, the authors mention \"previously unseen opponent policies\". How to define \"unseen\"? Is it possible that an unseen policy is similar to a certain seen policy?\n\n3. Why the win rate can have a negative value?\n\n4. Some compared works are out-dated, e.g., He et al., 2016a. Why not compare with Zintgraf et al., 2021 mentioned in the related work section. In addition, there exist some related works which could be compared with. For example,\n\n-- Rahman, Muhammad A., et al. \"Towards open ad hoc teamwork using graph-based policy learning.\" International Conference on Machine Learning. PMLR, 2021.\n\n-- Zhang, Ziqian, et al. \"Fast Teammate Adaptation in the Presence of Sudden Policy Change.\" UAI 2023.\n\n-- Zhu, Zhengbang, et al. \"MADiff: Offline Multi-agent Learning with Diffusion Models.\" arXiv preprint arXiv:2305.17330 (2023).\n\nAll these works model agents' policy and they claim their method can adapt to different agent policies. I understand that their scenarios are cooperative systems. However, their methods are comparatively straightforward to adapt to the setting of this paper.\n\n5. The environments used for evaluation are very simple. Is it possible to conduct experiments in more complex domains?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is organized well and written clearly.\n\n2. The authors perform extensive experiments and use various settings to support their claims.", "weaknesses": "1. The technical contribution of this paper is not very impressive. The main working flow of the proposed method is an integration of existing works and does not provide new insights.\n\n2. The authors miss some works that should be compared with.\n\n3. The environments used in the experiments are quite simple.\n\nPlease see more discussions in the following Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LPOcbbtMXx", "reviewer": "Reviewer_F2KT", "summary": "This work investigates the problem of rapid adaptation to an unknown opponent policy. The proposed approach is to learn a latent opponent model and play the response to the currently predicted opponent. They developed this under the namesake TAO (Transformers Against Opponent), a sequential application of three existing methods (learning policy embeddings, computing responses to known opponents, and few-shot adaptation). TAO is evaluated in the Markov Soccer and Multiagent Particle games and tends to perform at least as well as an existing method.", "questions": "- Sec 3.1, \"Specifically Tk1 ... usually with certain noise.\" The notation is a touch confusing here wrt your datasets. Does this mean that you only consider datasets where each player plays the same policy? If so, why?\n- Sec 4.2, how do you control for the ego agent's influence on the trajectory? \n  - It's not clear to me what these \"fragments\" are, why you're taking them, and why it's sensible to stitch together what appears to be sub-trajectories from differing episodes. \n  - I would have liked to see baselines/ablations related to their opponent-modeling proposal.\n- Sec 4.4, why are the differences in optimality guarantees not directly stated in the paper?\n  - What is it about TAO that enables the improvement in results? \n  - It's not clear to me that TAO should beget any improvements, so is it proof of a stronger bound?\n- Sec 5.1, \"We adopt three kinds of test settings for Π test, where seen is equivalent to the offline opponent policy set Π off, unseen contains some previously unseen opponent policies, and mix is a mixture of the former two.\" It appears _unseen_ and _mix_ are redundant and contain only different partitions of held-in and held-out policies. \n  - Why not also test generalizations to strictly held-out policies? \n  - How were the ordering of policies selected for the adaptation experiments? \n- Figure 4 has a non-standard y scale. Why are all the BR performances so bad against the subset of opponents that necessitated expanding the low-performance bracket? This smells like there is an issue in the quality/diversity of the opponent population because there are only extremely high and low-scoring performances.\n- Sec 5.2, Question 1\n  - \"TAO consistently outperforms other baselines ... this implies that TAO can respond effectively to unseen ...\" No evidence supporting this claim, and it is exceptionally strong. At best, the authors can claim that it performs better than baseline methods. \n  - Suggest also computing best responses directly to each opponent independently. This allows some measurement of what BR performance actually is, so we can have a better estimate of how much of true BR performance is recovered and how quickly. \n  - In Figure~4, while it is true that TAO looks like it's performing well, I don't think there's any clear narrative here. It's losing to other methods sometimes, often draws with many other methods, and the variance is exceptionally large. Indicates the need for additional study as to when each method is effective. \n  - How are you measuring \"strong adaptability ... swiftly adjusting\"?\n  - Fig 3 (b) is not explained ever and I'm not sure what it's trying to say. \n- Sec 5.2, Question 2\n  - It's not clear to me how a tSNE embedding is sufficient to show \"ability to recognize\". I would expect a clear quantitive result here.\n  - One such way to do this would be to look at the similarity of the embeddings of known opponent trajectories compared to circumstances where the agent has varying degrees of information inaccuracy. For example, it's previous episodic context could be from a different opponent. Another option is to measure the clustering of the embeddings. Or one could consider training a classifier on the frozen embeddings to see how well it could classify their originating opponent policy.\n  - Moreover, the authors need to supply similarity comparisons of their opponent policies.\n    - From investigating their description in the appendix I'm actually surprised they're not overlaping in tSNE space. Some are very similar! \n    - An explanation for this could be that the game is just very easy and each opponent policy defines nearly unique state spaces (after the initial few steps). Any performance differences could be attributed to that method/BR being poorly trained in that specific game state.\n- Sec 5.2, Question 3\n  - \"This suggests that PEL .... robust in-context learning abilities\" No evidence offered to support that PEL offers exactly \"more robust in-context learning abilities\".\n  - Should mention whether these results match previous work and discuss their (ag/disag)reement. This experiment appears to me to be largely a reproducibility experiment of a known method for policy representation learning.\n- I would have preferred if the authors had ablated the various three components of TAO. Did the authors investigate these?\n- For Figure 3 and 6 could you account for the performance difference between the subplots and why it appears that in 3(b), 4, and 6(b) the error is often much higher than seen in any bar charts of tables.\n- Perhaps the most interesting result, to me, is that of Question 7 in the appendix. Where you see evidence that larger context lengths can further confuse TAO as their is further uncertainty as to what evidence should be considered. I was wondering if the authors could speak more on this and how they think this could be addressed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- I view the main effort of this work as addressing the question \"Can transformers solve the few-shot opponent adaptation problem?\" To which, it appears the answer is yes, at least as well as existing approaches.\n- The related work is extensive, especially when it considers different methods of opponent modeling. It is also worth noting that ad-hoc teamwork is a relevant literature area despite focusing on common-interest/team games. \n- The implementation details of both their methods and their versions of the baseline methods are extensively detailed in the appendix.", "weaknesses": "- The major weakness of this work is the lack of precision in the discussion of their empirical results. In each experiment, the authors skip straight to making a concluding remark after describing their experimental method. A discussion of the results, beyond \"see Fig X\" is required and why the provided evidence leads to the resulting conclusion. In general, I also found the claims to be too strong for the provided figures. \n- This work redefines the notion of \"offline opponent modeling\" and expands its problem description to also include adaptation. I think this redefinition confuses more than aids in clarifying the problem space. I would strongly recommend against this redefinition. I think this also causes this paper to feel like it's trying to do too much and then only do a cursory treatment of each problem (modeling and adaptation). I think the paper would have been stronger if the authors focused on these problems independently and then together.\n- It's not clear to me what is added to our understanding of this problem from using transformers and \"in-context learning\" --- a vogue rebranding of concepts like correlation devices. I was hoping the authors could comment on their vision of this research, beyond that transformers are a neural network architecture that previous methods cannot readily apply.\n\n\n**Minor (Not Effecting Score)**\n- The language in the paper at times feels very unnatural (uncanny ChatGPT-esque), and I personally found it hard to read.\n- In the figures \"# Opponent Policy\" --> \"# Opponent Policies\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)