Opponent Modeling with In-context Search
Authors: Yuheng Jing
Bingyun Liu
Kai Li
Yifan Zang
Haobo Fu
QIANG FU
Junliang Xing
Jian Cheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Opponent Modeling
In-context Learning
Search
Primary area: reinforcement_learning
TL;DR: We propose OMIS, a novel opponent modeling approach based on in-context-learning-based pretraining and decision-time search, ensuring effectiveness and stability in adapting to unknown opponent policies, as proven theoretically and empirically.

Abstract: Opponent modeling is a longstanding research topic aimed at enhancing decision-making by modeling information about opponents in multi-agent environments. However, existing approaches often face challenges such as having difficulty generalizing to unknown opponent policies and conducting unstable performance. To tackle these challenges, we propose a novel approach based on in-context learning and decision-time search named Opponent Modeling with In-context Search (OMIS). OMIS leverages in-context learning-based pretraining to train a Transformer model for decision-making. It consists of three in-context components: an actor learning best responses to opponent policies, an opponent imitator mimicking opponent actions, and a critic estimating state values. When testing in an environment that features unknown non-stationary opponent agents, OMIS uses pretrained in-context components for decision-time search to refine the actor's policy. Theoretically, we prove that under reasonable assumptions, OMIS without search converges in opponent policy recognition and has good generalization properties; with search, OMIS provides improvement guarantees, exhibiting performance stability. Empirically, in competitive, cooperative, and mixed environments, OMIS demonstrates more effective and stable adaptation to opponents than other approaches. See our project website at https://sites.google.com/view/nips2024-omis.

Opponent Modeling with In-context Search
Yuheng Jing .2 Bingyun Liu Kai Li Yifan Zang 1,2 Haobo Fu Qiang Fu Junliang Xing² Jian Cheng 1.3.41
denotes corresponding authors Institute of Automation, Chinese Academy of Sciences 2 School of Artificial Intelligence, University of Chinese Academy of Sciences
3 School of Future Technology, University of Chinese Academy of Sciences
AiRii Tsinghua University
Tencent Al Lab
{ bitpieengg222.iiiiiiiuuuuuuu/ kaai @ia.ac.cn, {haobofu, leonfu) com, jlxing@tsinghua. edu. en
kaa. w zangy1fan2019, j iaan. cheng)
Abstract
Opponent modeling is a longstanding research topic aimed enhancing decision- making by modeling information about opponents multi- environments. However, existing approaches often face challenges such having difficulty gener- alizing to unknown opponent policies and conducting unstable performance. To tackle these challenges, we propose novel approach based on in-context learn- ing and decision-time. search named Opponent Modeling with In-context Search (OMIS). OMIS leverages in-context saannnng-based pretraining train Trans- former model for decision-making. It consists of three in-context components: an actor learning best responses opponent policies, opponent imitator mimicking opponent actions, and a critic estimating state values. When testing in an environ- ment that features unknown non-stationary opponent agents, OMIS uses pretrained in- n-context components for decision time search to refine the actor's 's policy. Theoret- ically, we prove that under reasonable assumptions, OMIS without search converges in opponent policy recognition and has good generalization properties; with search, OMIS provides improvement guarantees, exhibiting performance stability. Empiri- cally, in competitive, cooperative, and mixed environments, OMIS demonstrates more effective and stable adaptation to opponents than other approaches. See our project website https::p... //sites/.teee/eeeeee/e google com hwwm/view/nnps2024---1
Introduction
Opponent Modeling (OM) is pivotal topic in artificial intelligence research, aiming to develop au-
tonomous agents capable modeling the behaviors, goals, beliefs, or other propertics adversaries teammates (collectively termed opponents). Such modelings are used to reduce uncertainty
multi-agent environments and enhance decision-making
59 1o1 28 107
53
92 [05] [7]
102 66, 68). Despite the methodologies and insights proposed by existing OM approaches, their
processes generally boil down to two stages: (1) Pretraining: pretrain model with designed OM methodology training set of opponent policies; (2) Testing: deploying the pretrained model in certain way on a testing set of opponent policies to benchmark adaptability to unknown opponents.
For these two processes, different OM approaches usually have their respective focuses: (1) Pretraining-Focuun Focusee Approach (PFA) 229 28 63 focuses on acquiring knowledge responding to various opponents during pretraining and generalizing it to the testing stage; (2) Testing -Focuse Approach (TFA) 3. [01 focuses on updating the pretrained model during testing to reason and respond to unknown opponents effectively. However, existing PFAs and TEAs have their respective common and noteworthy drawbacks. For PFAs, they have limited generalization abilities, as the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

generalization of their pretrained models often lacks theoretical guarantees. Moreover, PFAS typically involve minimal additional operations during the testing stage, making them practically challenging to handle unknown opponents. For TFAs, they have performance instability issues. The finetuning (Le.,, update the pretrained model) of TFAs during testing can be tricky, as involves several gradient updates using only few samples to adjust the policy. Without careful manual hyperparameter tuning. TFAs always perform unstablely when facing unknown opponents.
To overcome the inherent issues of PFAS and TFAs, we propose a novel approach named Opponent Modeling with In-context Search (OMIS). The core motivation behind OMIS is 'think before you acr': when facing an opponent with an unknown policy during testing, we first guess about his current policy based on historical context. We then conduct Decision- Time Search (DTS) for few steps using this imagined opponent policy, estimate the returns of each legal action, and choose the best one to act on. Such process intuitively helps derive a policy more optimal than making a direct decision regarding only the current state. This approach is often reflected in real-life situations, such as the 'deep thinking' strategy employed by professional players in Go, chess, and other board games.
To enable such a DTS, we build three components: an actor, to respond appropriately to the current opponent during the DTS; an opponent imitator, who imitates the actions of the current opponent, enabling the generation of transitions in an imagined environment during the DTS; a critic, who estimates the value of the final search states, as we do not search until the end of the game. We argue that all three components should be adaptive, meaning they dynamically adjust based on changes in opponent information. Therefore, we adopt In- In-Context Learning (ICL)-based pretraining to learn three in-context components, as ICI. can endow them with the needed adaptability.
In summary, the methodology design of OMIS is as follows: (1) For Pretraining, we train a Trans- former [84] model for decision-making based on ICL 120] 88, 57 43). We build our model with three components: an actor, who learns the best responses to various opponent policies; an opponent imitator, who imitates opponent actions; and a critic, who estimates state values; (2) For Testing, we use the pretrained three in-context components for DTS [80 8T [ to refine the actor's original policy. Based on predicting opponent actions and estimating state values, this DTS performs rollouts for each legal action, promptly evaluating and selecting the most advantageous action.
Theoretically. OMIS can provably alleviate the issues present in PFAS and TFAs. For limited gen- eralization ability of PFAs, OMIS's pretrained model is proven converge opponent policy recognition and to have good generalization properties: OMIS's pretrained model can accurately recognize seen opponents and recognize unseen opponents the most familiar seen ones some extent. For performance instability issues of TFAs, OMIS's DTS avoids any gradient updates and theoretically provides improvement guarantees.
Empirically, extensive comparative experiments and ablation analyses in competitive, cooperative, and mixed environments verify the effectiveness of OMIS in adapting unknown non-stationary opponent agents. Statistically, OMIS demonstrates better performance and lower variance during testing than other approaches, reflecting the generalizability and stability of opponent adaptation.
2 Related Work
Opponent modeling. In recent years, OM has seen the rise of various new approaches based on different methods, including those based on representation learning (2 28
107
63
40
Bayesian
learning [106.] 19 22 (4), meta-learning 3 41 107 [4], shaping opponents' learning 22 23 47).
and recursive reasoning 990 101). All approaches can be broadly categorized into PFAS and TFAs.
OM based on representation learning and meta-gradient-free meta-learning methods such as Duan et al. (20 typically fall into PFAs. PFAs' generalization on unknown opponents often lacks any theoretical analysis or guarantees. This also leads to PFAS not always performing well empirically, Our work utilizes ICL pretraining to provide good theoretical properties regarding generalization.
OM based on Bayesian learning and geeaa-raddeentbrarg based meta-learning such as Finn et al. [2]] typically belong to TFAs. The finetuning of TFAs makes them unstable, the opponent may continuously change policy during testing, making it challenging to adapt with a small number of samples for updating. Our work employs DTS to avoid finetuning and has improvement guarantees.
In-context learning. Algorithmically, ICL can be considered as taking more agnostic approach by learning the learning algorithm itself 200 88 57 43 Recent work investigates why and how

---

pretrained Transformers perform ICL [27 49 103
1 69 Xie et al. [95] introduces Bayesian proves Transformers can implement
framework explaining how ICL works. Some work 88 ICI. algorithms via in-context gradient descent. Lee et al. 44 proposes supervised pretraining to empirically and theoretically demonstrate ICI. abilities in decision-making. Unlike existing decision- related work focusing on single-agent settings, our work explores the theoretical properties and empirical effects of using a Transformer pretrained based on ICL under the setting of OM.
2
Decision- time search. DTS involves searching in simulated environment before each real action, aiming to obtain a more 'prescient" policy than no search [8) 81 13 50]. One of the most represen- tative works is the AlphaGo series (74 76, 72). which achieves remarkable results in games like Go and Atari based on a DTS algorithm called Monte Carlo Tree Search (MCTS) and self-play. Our work explores how to make DTS work in the context of OM. The DTS of the AlphaGo series assumes that opponent adopts the same strong policy as the agent we control. In contrast, the DTS in our work dynamically models the opponents' actions, focusing on better adapting to the current opponents.
See App. A for an overview of OM and related work on Transformers for decision-making.
3 Preliminaries
We formalize the multi-agent environment using an n-agent stochastic game (8, {A S is the state space, A' is the action space of agent E [m). A I7= A is the joint action space of all agents, P S x A x S + [0, 1J is the transition dynamics, Rs 8 A - R is the reward function for agent i, 7 is the discount factor, and T is the horizon for each episode.
P,
Following the tradition in OM, we mark the agent under our control, i.e., the self-agent, with the superscript 1, and consider the other script -1. The joint policy of opponents is denoted as π la ()) [[i T (a>l)s) where a-1 is the joint actions of opponents. Let the trajectory timestep in the current episode be y (cur) = {sp, ag. 90 To: ro ...... $ s.). The historical trajectories (cur-1) 1) (cur)
1 agents as opponents, marked with the super-
1/ ) is always available to the self-agent. During the pretraining stage, opponent policies are sampled from a training set of opponent policies I = (=-1.k) 1.k the testing stage, opponent policies
H4 = (VT (0) 1/T
k== During sampled from testing set of opponent policies 1etest which includes an unknown number of unknown opponent policies.
In OM, the self-agent's policy can be generally denoted as (' D) (abbreviated as T), which dynamically adjusts based on the opponent information data D (referred to as in-context data in this paper). D can be directly composed of some part of the data from Hy, or it can be obtained by learning representation from H,, Building upon the pretraining, the objective of the self-agent is to maximize its expected return (ie., cumulative discounted reward) during testing:
E
Pe ),r5" " No (-1se), a I-11 -a](( (.t.11..))) aneemiinnnn
Σ (=0
rt R (s). (s..0t.077) (z))
4
Methodology
In Sec. 4.1 we present how we build the in-context actor, opponent imitator, and critic for OMIS with ICL-based pretraining: in Sec. 4.2 describe method of using pretrained in-context components for DTS: in Sec. (4.3] we provide a theoretical analysis of both the ICL and DTS components of OMIS.
We provide an overview of OMIS in Fig. and the pseudocode of OMIS in App. B
4.1 In- Hi--Conexxt-.earninge based Pretraining
To ensure that the actor learns high-quality knowledge of responding to various opponents, we first solve for the Best Responses (BR) against different opponent policies. For each opponent policy π 11, in I|tain (where k E [K], we keep the opponent policy fixed as π 11, and sufficiently train
the PPO algorithm (73 to obtain the BR against x 1,k denoted as BR(=-1AA 1.k) == π I. (als).
To generate training data for pretraining the three components, we continually sample opponent policies from Iaain and use their corresponding BR to play against them. For each episode, we sample n -1, from tran as opponents and use its BR n 1. self-agent to play against it.
3

---

Pretraining
Testing
Search
'(te)
kealactees R
Tate
prodictons
Ag
X
Bd
Ve
causal transformer
Roed
Resa
Q (6)).
argueas
~M
11
Figure 1: Left: The pretraining procedure and architecture of OMIS. The pretraining steps are as follows: (1) Train BRs against all policies in II (2) Continuously sample opponent policy from IT and collect training data by playing against using its BR. (3) Train a Transformer model using ICL-based supervised learning, where the model consists of three components:
actor no. an opponent imitator μg, and a critic V.- Right: The testing procedure of OMIS. During testing, OMIS refines सa through DTS each timestep. The DTS steps are follows: (1) Do multiple L-step rollouts for each legal action, where Ty and ság are used to simulate actions for the self-agent and opponent, respectively. Vu is used to estimate the value of final search states. (2) Estimate value Q for all legal actions, and the search policy Tseach selects the legal action with the maximum Q. (3) Use mixing technique to trade-off between Teeach and He to choose the real action to be executed.
The procedure of generating training data is as follows: for each timestep we construct in- context data D == (Depik D D.topk) about -1, which is used provide information about -11k for self-agent to recognize -1,k Depik = {(A,. a L, 1b=1 generated by playing against π -1, using any self-agent policy It is used to characterize the overall behavioral pattern of क on an episode-wise basis. See the construction process of Depak in App. C
) is episode-wise in-context data,
Detopk (so. 9. 1,k St-1, a, () is step-wise in-context data, generated by the current episode 1,k -1, and π I. It represents the step-wise specific behavior pattern of π -1.k,
involving
Furthermore, for each timestep t, we collect the Return-To- To-Go (RTG) obtained by the self-agent, denoted G...t.+ => -t= 7d -tr 7 -'- ag 1,k.,
,o, ",. _., where a.kk, π khe
1.8 - and V.l.k.+ = =EIG2 To end with, the training data for timestep t is obtained as:
D = (st, ay an at 11,
1,k,*
G. G..
After preparing the training data, we use supervised learning to pretrain an actor To(a) [s D to learn the BR against π - an opponent imitator (= st, Dt) ) to imitate the opponent's s policy, and a critic (s,, Dft) to estimate the state value of self-agent. Notably, all these components condition on D. as their in-context data. For each episode, the optimization objectives are as follows:
max Es; [-
Tog st, (;)
1,4 1 D2)) 1o -1,
max E 1-7
min F J (Va D+)- G...ks)) 2
The left side of Fig. illusstrates OMIS's architecture and its pretraining procedure. Based on the understanding of ICL in eecision-making we design our architecture upon causal Transformer 67
4.2 Decision-Tinee Time Search with In-Context Components
Following the best practices in OM, we assume the testing environment features unknown non- stationary opponent agents, which we denote as &. Unknown indicates that the self-agent is unable to ascertain the true policy Æ employed by Q. Non-stationaryy implies that & switches its policy between episodes in some way, with each switch involving randomly sampling म from [[iest
Ics is used to mark data in prpkkk h is an index but not timestep.

---

of Following the general setup in the DTS domain, we assume the ground truth transition dynamic P available [75 76 12 13 46 38 Based on the pretrained in-context components Tg. Vas Pa- and P. we conduct DTS to play against Φ. At each timestep I, we perform M times of rollouts with length L for each legal action á of self-agent P This is done to estimate the value (s,, a]) for each a; under current true opponent policy म -- and current self-agent policy Te. The self-agent then executes the legal action with the highest Q value in the real environment. Our expectation is that through such DTS, we can refine the original policy wy better adapt to +.
The specific process of the DTS is as follows: for each timestep t, we first construct in-context data D2 (D D. about &, and its construction method almost identical to D mentioned in Sec. 4.1 However, since मं is unknowable, we make a slight modification: D is constructed by sampling consecutive segments from the most recent trajectories in which & participated.
After constructing Dr. for any given legal action a). we sample the opponents' action by a," Ho (-|s,, D.) and transition using P to obtain 8 and ft. We append (st, a, to the end of D to obtain the updated step-wise in-context data D and in-context data De+1 (Dei, D D(sep) Following, the 1-th step of the rollout for å (1 e [L)), we sample self-agent action and opponent action using the following two formulas, respectively:
بi+ T Dc+s),
(6)
1,+ ~ 1o(11$1111 D(t1).
(7)
Transitioning with P yields 864141 and ft+.. Next, we append (ăr44, a,z1) to the end of Diap to obtain D1tt+1 and Det1+1 = (Dopi, D After completing the L-th step, we use V4+++1 = (8 D++)+1) to estimate the value of the final search state. When finishing M times of rollouts for â}, we obtain an estimated value for à by:
M
L+L
1 (ssr, at) = M
r'-t Tearch
F/. + Tearch
L+1
Ve+L+1
(8)
Here, "isearch is the discount factor used in the DTS. After completing rollouts for all legal actions the self-agent at timestep t, we obtain our search policy by maximizing Q:
scarrhh (5t) := arg max Q(5 a!)) a.
(9)
In practical implementation, we observe that using nearch directly is not always effective. This is because we cannot totally precisely estimate the opponents' policy and the state value, making the results obtained from the DTS not sufficiently reliable across all states. To this phenomenon, we propose simple yet effective mixing technique to balance the search policy and the original actor policy in deciding the action to be exccuted:
Tiix (st) :- [Tsencc($t) (|Q(st, secarcts (s))ll > at ~ (-(s Dz), otherwise
(10)
Here, is a threshold hyperparameter. The main motivation for the mixing technique is as follows. We consider the expected return of the action selected by the DTS as the confidence of the search policy. When the confidence exceeds a certain threshold, we tend to consider that weeurch has relatively high probability of achieving better results than no: otherwise, we prefer to use the original policy ng. See the testing procedure of OMIS on the right side of Fig.
4.3 Theoretical Analysis
Our theoretical analysis unfolds in the following two aspects. (1) We propose Lem. 4.1 and Thm. 4.2 to prove that OMIS without DTS (denoted as OMIS w/o S) converges to the optimal solution when facing true opponent policy -1 I and it recognizes the opponent policy as the policy in
m with a minimum certain form of KL divergence from #-1 when facing --1 & I (2) Building upon Thm. 4.2 we further propose Thm. 4.3 to prove the policy improvement theorem of OMIS with DTS, ensuring that it leads to enhancements in performance.
2 is used to mark the terms during DTS.
5

---

To begin with, we instantiate a Posterior Sampling in Opponent Modeling (PSOM) algorithm (see [.I) based on the Posterior Sampling (PS) algorithm [6]]. where PSOM can be proven to share the same guarantees of converging to the optimal solution as PS. Based on some reasonable assumptions, we prove that OMIS w/o is equivalent to PSOM.
Lemma 4.1 (Equivalence of OMIS w/o and PSOM). Assume that the learned He is consistent and the sampling of s from pre is independent of opponent policy, then given म. aadd its D, we have PD : P( _ ; ) for all possible ?--
Here, 'consistent' indicates that the network's s fitting capability is guaranteed. (; de- notes the probability distribution on all the trajectories involving x-1 during pretraining. & = a)") is self-agent history, where a 1 is sampled from the BR to the opponent
as "t..a"......." ST, policy recognized by PSOM. The proof is provided in App. D.2
Theorem 4.2. When म = =1, E II if the PS algorithm converges to the optimal solution, then OMIS w/o S recognizes the policy of d
1.k i.e., Te. pdg and V converge to π xl.hi,, 1,k and Vllk,, respectively; When n & 11 OMIS w/o S recognizes the policy of 4 the policies in 11 Inaiin with the minimum D Dxz(P(a
[=] P(a 1|, ")).
Based on this theorem, when OMIS w/o S faces seen opponents, it accurately recognizes the opponent's ' policy and converge to the BR against it; when facing unseen opponents, it recognizes the opponent's policy as the seen opponent policy with the smallest KL divergence from this unseen opponent policy and produces the BR to the recognized opponent policy. The proof is in App. D.3
Theorem 4.3 (Policy Improvement of OMIS's DTS). Given मt and ity D, suppose OMIS recognizes & as म.. and V*_, is the value vector on S, where V(s) == (u, D), <[aas) := no(als, D). Let
Gr. be the L-step DTS operator and n' E Gv. then V=L V holds component-wise.
Within, OMIS's s DTS can be viewed as Gr, and n' corresponds to Tscarch in Eq. 9 Thm. 4.3 indicates that OMIS's DTS is guaranteed to bring improvement, laying the foundation for performance stability, Additionally, OMIS's DTS avoids gradient updates. The proof provided in App. D.4
Analysis for generalization in OM. In OM, generalization can be typically defined as performance when facing unknown opponent policies (e.g., opponents like 4). Existing approaches lack rigorous theoretical analysis under this definition of generalization. In Lem. 4. we proved that OMIS w/o is equivalent PSOM. In Thm. 4.2, proved that PSOM can accurately recognize seen opponents and recognize unseen opponents as the most similar to the seen ones. Since OMIS w/o S is equivalent to PSOM, OMIS w/o S possesses the same properties. Additionally, Thm. 4.3 proved that OMIS's s DTS ensures performance improvement while avoiding instability, These theoretical analyses potentially provide OMIS with benefits in terms of generalization in OM.
Experiments
5.1 Experimental Setup
Environments. We consider three sparse-reward henchmarking environments for OM as shown in Fig. (S App.Efoor detailed introductions of them):
Predator Prey (PP) is a competitive environment with a continuous state space. In PP, the self-agent is prey (green) whose goal is avoid being captured by three predators (red) as much as possible. There are two obstacles (black) on the map. The challenge of PP lies in the need to model all three opponents simultaneously and handle potential cooperation among them.
Level-Based Foraging (LBF) is mixed environment in a grid world. In LBF, the self-agent is the blue one, aiming to eat as many apples as possible. The challenge of LBF is that cooperation with the opponent is necessary to eat apples of higher level than the self-agent's (the apples and agents' levels are marked in the bottom-right). LBF represents a typical social dilemma.
OverCooked (OC) a cooperative environment using high-dimensional images as states. OC, the self-agent is the green one, which aims at collaborating with the opponent (blue) to serve dishes as much as possible. The challenge of OC lies the high-intensity coordination required between the two agents to complete series of sub-tasks to serve dish successfully.

---

Baselines. We consider the following representative PFAs, TFAs, and DTS- based OM approach:
DRON [2]]: Encode hand-crafted features of opponents using a Mixture-of- Expert network while also predicting opponents' actions auxiliary task (this is the most performant version in (2]).
Use the observations and actions of the self-agent to reconstruct those of the opponent through auto-encoder, thereby embedding the opponent policy into latent space.
LIAM 63:
(VAE) reconstruct the opponent's future actions and condition on the embedding generated by this VAE to learn Bayesian meta-pon meta-policy.
MeLIBA 107 Use Variational
Meta-PG 3 Execute multiple meta- graadieg steps to anticipate changes in opponent policies to enable fast adaptation during testing.
Meta-MAPG [41] Compared to Meta-PG, it includes new term that accounts for the impact of the self-agent s current policy on the opponent's future policy.
MBOM 101: Use recursive reasoning in environment model to learn opponents at different recursion levels and combine them by Bayesian mixing.
OMIS w/o S: variant of OMIS, where OMIS directly uses To based on D, without DTS.
Use scripted opponent model estimate the opponent's ''s actions and apply MCTS for DTS. We adopt OMIS w/o S as its original self-agent policy. This is a DTS-based OM approach. Within, DRON, LIAM, and McLIBA belong to PFAs; Meta-PG, Meta-MAPG, and MBOM belong to TFAs. See neural architecture design of all approaches in App. F For a fair comparison, we implement MBOM and SP-MCTS using the ground truth transition
SP-MCTS 88
the environment model.
Opponent policy. We employ a düiersity-driven Population Based Training algorithm MEP [004] to train a policy population. Policies from this MEP population are used to construct I and [ ensuring that all opponent policies are performant and exhibit diversity. We measure and visualize the diversity of opponent policies within the MEP population in App. G
We randomly select 10 policies from the MEP population to form I Then, we categorize opponent policies in the MEP population into two types: 'scen' denotes policies belonging to I and 'unscen' denotes policies not be- longing to IItrain, We set up opponent policies with different ratios of [seen: unseen] to form ITust e.g., [seen: : unseen] [0 10] signifies that [Teest is composed of 10 opponent policies that were never seen during pretraining.
(a) PP
(b) LBF
(c) OC
Figure 2: The benchmarking environments.
During testing, all opponents are the unknown non-stationary opponent agents & mentioned in Sec. 4.2 & switches policies by sampling from 11 every E episodes.
Specific settings. For the pretraining stage, we train all approaches for 4000 steps. For the testing stage, all approaches use the final checkpoints of pretraining to play against & for 1200 episodes. All bar charts, line charts, and tables report the average and standard deviation of the mean results over 5 random seeds. See all the hyperparameters in App.
5.2 Empirical Analysis
We pose a series of questions and design experiments to answer them, aiming to analyze OMIS's opponent adaptation capability and validate each component's ss effectiveness.
Question 1. Can OMIS effectively and stably adapt to opponents under various IT configurations?
We set up 5 different [seen: : unseen] ratios to form [ess and we show the average results of all approaches against & corresponding to each ratio in Fig. OMIS exhibits a higher average return and lower variance than other baselines across three environments, highlighting its effectiveness and stability in opponent adaptation under different lest configurations. It can be observed that OMIS w/o S outperforms existing PFAS (e.g., MeLIBA) in most cases, validating that pretraining based on ICL exhibits good generalization testing with unknown opponent policies.
The results also indicate that OMIS improves OMIS w/o S more effectively than SP-MCTS does. SP-MCTS sometimes even makes OMIS w/o S worse (e.g., in OC and parts of PP). This might be because (1) the opponent model of SP-MCTS, which estimates opponent actions, is non-adaptive and (2) the trade-off between exploration and exploitation in the MCTS is non-trivial to optimize.
7

---

MAPG
MBOM
DRON
OMIS
Predator Pray
Level Based Foraging
OverCooked
0:55
180
25
0.50
160
-75
-50
0.40
0:35
120
100
140
-100
0:25
Ang.
-125
-150
-175
10:10
5:10
0.00
10.0
10:3
0 10:0
5:10
0.30
Ratio of unsen))
Ratio
[seen: unseen)
Ratio [seen:ursees
Figure 3: Average results of testing under different (seen: : unseen] ratios, where E
20.
MctaiPG
Mata MARG
MBOM
LAM
McLiBA
DRON
ONIS w/o
SPIMCTS
OMIS
Predator Prey
Level-Based Foraging
OverCooked
200
0.55
180-
Avg, Roturn
0.50
160-
140-
0.00
0.35
0.25
0.20
40-
20
10
20
30
Number of n=3 Switching Policies
Number of Switching Policies
Number n=1 Switching Policles
Figure 4: Average results against each true opponent policy during testing, where [seen unseen]
[10: : 10] and E = 20. Each point in X-axis denotes a policy switching of t, totaling 60 times. Y-axis denotes the average return against the corresponding switched #-1
Question 2. Can OMIS adapt well to each one of the true policies adopted by h
In Fig- 4 we provide the average results of all approaches against each true opponent policy म --1 employed by & corresponding to ratio of [seen: unseen] = [10 10). Similar to the observations Fig. 3 OMIS exhibits higher return and lower variance than other baselines across various true opponent policies in PP, LBF, and DC. TFAs (e.g., Meta-PG) generally show significant performance gaps when facing different true opponent policies. This is likely due to the continuous switching of opponent policies, making finetuning during testing challenging or ineffective.
Question 3. How does OMIS work when the transition dynamics are learned?
We include a variant named Model- Model-Based OMIS (MBOMIS) to verify whether OMIS can work effectively when the transition dynamics unknown and learned instead. MBOMIS uses the most straightforward method to learn transition dynamic model P: given state and action a. predicting the next state s' and reward using Mean Square Error (MSE) loss. P. is trained using the (s, a, s') tuples from the dataset used for pretraining OMIS w/o S. The testing results against unknown non- stationary opponents are shown in Fig. [ Although MBOMIS loses some performance compared to OMIS, it still effectively improves over OMIS w/o S and generally surpasses other baselines. We also provide quantitative evaluation results of P's estimation during testing in Tab. | Observations show that P generally has small MSE value in predicting the next state and reward without normalization.
Question 4. Is OMIS robust the frequency of which switches policies?
We employ different frequencies for & to switch policies, i.e., E 2, 5, 10, 20, dynamic (abbreviated as dyna). Here, E = dyna indi- cates that & randomly selects from 2, 5, 10, 20 at the start of each switch as the number of episodes until the next switch. Fig. shows the average results against Tb with different switch- ing frequencies. OMIS and OMIS w/o ex- hibit a degree of robustness to different policy switching frequencies E of in PP, LBF, OC. Notably, as E increases, their performance gen- erally shows a slight upward trend. This suggests that OMIS could gradually stabilize its adaptation to true opponent policies by accumulating in-context data.
w
Predator Prey
Level Based Foraging
OverCooked
150
-50
245
-75
-103-
535
120
-125
10
10 20
Switch &requencees
Figure 7: Average results during testing when & adopts different switching frequencies, where [seen: : unseen] = [10: 10).
Question 5. ls each part of OMIS's method design effective?
We design various ablated variants of OMIS: (1) OMIS w/o (See Sec. 5.Tp: (2) OMIS w/o mixing: variant where the mixing technique is not used, i.e., using Tsearch instead of mmixi (3) OMIS w/o
8

---

Predetor
Predetor of Pay
Leuedibased foreging
170
0.5
125
1.00
1.6
160 L855 1010 6.D0
12. 1655 10.18:
S.. 0:10
166 125 10 EL12
10
30
92
60 20 Number of *Seisding Polices
25
40
26
Rall of (HeNe) suntsen]
(a)
(b)
Figure 5: Results of OMIS using learned dynamics against unknown non-stationary opponents. (a) Average results of testing under different [seen: unseen] ratios, where E = 20. (b) Average results
against each true opponent policy during testing, where [seen: unseen] = [10: : 10) and E = 20.
Table I:
Quantitative evaluation results of the learned dynamic P's estimation during testing. The results are calculated during testing under different [seen: unseen] ratios, where 20.
Jeeen suseeen
Emvironmnnt
Fvshattio Matric
10: 0
10: 5
10: 100
5 10
D 10
Predator Pray
Avg. Next State MSE
Avg. Reward MSE
0,13836
0,00409 1: 0.00002
+ 0.01425
0,24080
0.00133
= 0,05558
0,00011
0.0010 0,00005
0.15288 0.0164]
0.26276 + 0.03761
0,0062 1 0.00004
0,22110 + O 0.00667
0.00445 1: 0.00002
Levol-Utred Foraging
Aug. Nest State MSF
Avg. Reward MSF
0.05750 +: 0.0187
0.00004 + 0.00000
0.00044
0.00433
= 0.00000
0.00074
0.07702
0.00004 = 0.00000
0.00160
0,05255 + 0.00154
0100004 + 0.00900
0.00003 + 00000
0.04923 + 00066
DownCooked
Avg. Nest State MSE
Avg. Reward MSE
0.R02Z +
0.00000 + 0.00000
0.00225
0.000000 +: 0,00000
0.00000
0.00028
0.00001
0,00000
0.00028 +
0100000 + 0.00000
0.00000 + 0.00000
GMI w/o D
OMIS w/o 02
QMS w/o
DMIS w/o mixing
DMS
Prodator Pray
Level-Baser Baaed Foraging
OverCooked
175
150
100
-80
0.60
0.45
0.35
100
125
120
0.30
340
0.20
50
160
0:15
0.10
500
1000 150 2000 2900 3000
500
1000
1500
2000 2500 3000 4000
500
1000 1500 2000 2900 3000 4000 Training Step
Training Step
Training Step
Figure 6: Average performance curves during pretraining against all policies in 1
S. Detepk a variant without DTS where De input is excluded from the model: (4) OMIS w/o S, Depi,; variant without DTS where Depi.k input is excluded from the model.
Fig. 6 shows the average performance curves during pretraining for these variants against all policies in mi In PP, LBF, and OC, OMIS w/o S consistently performs a lot worse than OMIS. This indicates that the DDS effectively improves the original policy of my. OMIS w/o mixing exhibits a notable performance decrease compared to OMIS in LBF and OC. This suggests selective searching based on confidence is more effective than indiscriminately. It can be observed that Depi.k and D.uup. both play crucial roles in OMIS's s adaptation to opponents, with Depi. making greater contribution. This could be because capturing the overall behavioral patterns of opponents is more important than focusing on their step-wise changes.
Question 6. Can OMIS effectively characterize opponent policies through in-context data?
For each policy in Irrain we visualize OMIS's attention weights of Depi, over the final 20 timesteps in an episode in Fig. Each position of tokens in Depi,& has weight indicated by the depth of color.
In all three environments, the attention of OMIS exhibits the following characteristics: (1) Focusing on specific positions of tokens in Depi. for different opponent policies; (2) Maintaining a relatively consistent distribution for a given opponent policy across various timesteps within the same episode. This implies that OMIS can represent different opponent policies according to distinct patterns of different in-context data. We also provide quantitative analysis on OMIS's s attention weights in App.
Question 7. How well does the in-context components of OMIS estimate?
We collect true opponent actions and true RTGs as labels, using Accuracy and MSE as metrics to evaluate the effectiveness of μO's 's and V. 's estimations, respectively. In Tab. 2 we provide estimation results of the in-context components μl: Va, during testing under different [seen: .unseen] ratios. It can be observed that ne is estimated relatively accurately in DC, while V., is estimated relatively accurately in PP and LBF, However, Ms does not estimate very accurately in PP and LBF. This indicates that the functioning of OMIS does not necessarily depend on very precise estimates. In all three

---

Predator Proy
Level Level-Based Based Foraging
OverCooked
25
25
25
Positions of Tokenn
20
A.00
15.
4.0
15
6.5
20.
10
10
8.050
5.05
5.00
5.00
5,000
Opporent Policy
Opporent Policy in mmaan
Opponent Policy ra
Figure 8: Attention heatmaps of OMIS when playing against different policies in I[
Table 2: The estimation results of the in-context components OMIS during testing, where 20.
Env.
Predator Pray
Level-Bazed Foraging
OverCooked
[seen: : unseen]
10: 0
0:
10: :
10
0: 10
10
10: 0
0:
10. :
Avg. Auc. 1
6044 0.9
0.6
53.0 1: 0.9
63.1 0.8
56.3 +:
59.2 4
79.8 4 0.3
64.4 0.6
71991:04
Arg. MSB
0.01 + 0.00
0.07 + 0.02
0.03 + 0.01
000 + 0.00
0.01 + 0,00
0.01 + 0.00
0.12 + 0.20
0.20 + 0.27
0.36 + 022
Predatoc Pray
Level Based Foraging
OverCooked
lopil
canes
Battig
canor
dasing wurd
Figure 9: Visualization of OMIS's DTS when playing against unseen opponent policy.
environments, the estimation accuracy for purely unseen opponents does not decrease significantly, further confirming the good generalization of ICI..
Question 8. How does OMIS's DTS work in real games?
Fig. 9 visualizes the OMIS's DTS process at particular timestep during game against an unseen opponent policy in three environments. We only illustrate two legal actions as an example. It can be observed that OMIS's & DTS promptly evaluates each legal action, predicts the opponent's 's actions during the DTS, and ultimately selects the most advantageous action. We notice the following interesting phenomena: In PP, DTS enables the self-agent to avoid being captured by opponents who use an encirclement policy; In LBF, DTS allows the self-agent to cooperate with opponents to eat apples with higher level than itself; In OC, DTS helps prevent the self-agent from blocking the path of collaborators, allowing them to serve dishes smoothly.
6 Discussion
Summary. this paper, we propose OMIS based on ICL and DTS, aiming to address the challenges of limited generalization abilities and performance instability issues faced by existing OM approaches. The foundations of OMIS lie in two stages: (1) We employ ICL to pretrain Transformer model consisting of three components: actor, opponent imitator, and critic. We prove that this model converges in opponent policy recognition and has good properties in terms of generalization; (2) Based on the pretrained in-context components, we use DTS to refine the policy of the original actor. gradient updates and provides improvement guarantees. Extensive experimental results in three environments validate that OMIS adapts effectively and stably to unknown non-stationary opponent agents.
This DTS avoids (the
Limitations and future work. (1) We only considered opponents with non-stationary switches among several fixed unknown policies during testing. OMIS might face challenges adapting to opponents who are continuously learning reasoning; (2) This work focuses on the setting of perfect information games. Effectively incorporating ICL and DTS for OM in imperfect information games a challenging and meaningful research problem; (3) OMIS only utilizes a naive DTS method to refine its policy. We will continue to explore how to apply more advanced DTS methods to the OM domain for more effective adaptation to unknown opponents. A more in-depth discussion is in App.
10

---

Acknowledgments
This work is supported in part by the National Science and Technology Major Project (2022ZD01 16401); the Natural Science Foundation of China under Grant 62076238, Grant 62222606, and Grant 61902402; the Jiangsu Key Research and Development Plan (No. BE2023016): and the China Computer Federation (CCF))Tencent Open Fund.
References
[1] Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. In Advances in Neural Information Processing Systems, page 35151-35174, 2023.
[2] Ekin Akyürek, Dale Schuurmans. Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In International Conference on Learning Representations, 2023.
[3] Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environ- ments. In International Conference on Learning Representations, 2018.
[4] Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A compre- hensive survey and open problems. Artificial Intelligence, 258:66-92 95, 2018.
[5] Ioannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K Hubert, and David Silver. Planning in stochastic environments with learned model. In International Conference Learning Representations, 2021.
[6] Robert Axelrod. Effective choice prisoner's dilemma. Journal of conflict resolution, 24 (1):3-25, 1980.
[7] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. Stat, 1050:21, 2016.
[8] Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers statis- ticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.04633 2023.
[9] Anton Bakhtin, David Wu, Adam Lerer, and Noam Brown. No-press diplomacy from scratch. Advances in Neural Information Processing Systems, 34:18063-18017 2021.
[10] Nolan Bard, Michael Johanson, Neil Burch, and Michael Bowling. Online implicit agent modelling. In International Conference on Autonomous Agents and MultiAgent Systems, pages 255-262, 2013.
[11] David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? In Advances in Neural Information Processing Systems, pages 1542-1553, 2022.
[12] Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365 (6456):885-890, 2019.
[13] Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforcement learning and search for imperfect-information games. In Advances Neural Information Processing Systems, pages 17057-17069, 2020.
[14] Micah Carroll, Rohin Shah, Mark Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. In Advances in Neural Information Processing Systems, page 5174-5185, 2019.
[15] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances Neural Information Processing Systems, pages 15084 15097, 2021.

---

[16] Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Shared experience actor-crit critic for multi-agent reinforcement learning. In Advances Neural Information Processing Systems, 2020.
[17] Zhongxiang Dai, Yizhou Chen, Bryan Kian Hsiang Low, Patrick Jaillet, and Teck-Hua Ho. R2-h2: Recursive reasoning-based bayesian optimization for no-regret learning in games, In International Conference on Machine Learning, pages 2291-2301 2020.
[18] Ivo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. Policy improvement by planning with gumbel. In International Conference on Learning Representations, 2021.
[19] Anthony DiGiovanni and Ambuj Tewari. Thompson sampling for markov games with piece- wise stationary opponent policies. In Uncertainty in Artificial Intelligence, pages 738 7488 2021.
[20] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL? Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:/611.02779 2016.
[21] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model- meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1 1126-1135 2017.
[22] Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. International Conference on Autonomous Agents and MultiAgent Systems, pages 122-130, 2018.
[23] Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. DiCE: The infinitely differentiable Monte Carlo estimator. In International Conference on Machine Learning, pages 1524-1533, 2018.
[24] Haobo Fu, Ye Tian, Hongxiang Yu, Weiming Liu, Shuang Wu, Jiechao Xiong, Ying Wen, Kai Li, Junliang Xing, Qiang Fu, et al. Greedy when sure and conservative when uncertain about the opponents. In International Conference on Machine Learning, pages 6829 9899,,88,888 2022.
[25] Kitty Fung, Qizhen Zhang, Chris Lu, Timon Willi, and Jakob Nicolaus Foerster. Analyzing the sample complexity of model- free opponent shaping. ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023.
[26] Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. In International Conference on Learning Representa- tions, 2022.
[27] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? study of simple function classes. In Advances in Neural Information Processing Systems, volume 35, pages 30583-30598, 2022.
[28] Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learn ing policy representations in multiagent systems. In International Conference on Machine Learning, pages 1802-1811, 2018.
[29] He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforcement learning. International Conference on Machine Learning, pages 1804-1813, 2016.
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[31] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.0m (tt::1606008415,, 2016.
12

---

[32] Pablo Hernandez-Leal, Matthew E Taylor, Benjamin S Rosman, Luis Enrique Sucar, and E Munoz de Cote. Identifying and tracking switching, non-stationary opponents: A bayesian approach. In AAAI Conference on Artificial Intelligence Workshop on Multiagent Interaction without Prior Coordination, pages 560-566, 2016.
[33] Zhang- Weii Hong, Shih- Yang Su, Tzu-Yun Shann, Yi-Hsiang Chang, and Chun- Yi Lee. A deep policy inference q-network for multi-agent systems. In International Conference on Autonomous Agents and MultiAgent Systems, pages 1388-1396 2018.
[34] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5149-5169 2021.
[35] Hengyuan Hu, Adam Lerer, Noam Brown, and Jakob Foerster. Learned belief search: Effi- ciently improving policies in partially observable settings. arXiv preprint arXiv:2 09086, 2021.
[36] Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In International Conference on Machine Learning, pages 4476-4486. PMLR, 2021.
[37] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Flyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys, (0(2):1-35, 2017.
[38] Athul Paul Jacob, David J Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Jacob Andreas, and Noam Brown. Modeling strong and human-like gameplay with ki- regularized search. In International Conference Machine Learning, pages 9695-9728. PMLR, 2022.
[39] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey contrastive self-supervised learning, Technologies, 9(1::2, 2020.
[40] Yuheng Jing, Kai Li, Bingyun Liu, Yifan Zang, Haobo Fu, QIANG FU, Junliang Xing, and Jian Cheng. Towards offline opponent modeling with in-context learning. In The Twelfth International Conference on Learning Representations, 2023.
[41] Dong Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian sopez-Cot, Gerald Tesauro, and Jonathan How. A policy gradient algorithm for learning learn in multiagent reinforcement learning. International Conference on Machine Learning, pages 5541-5550, 2021.
[42] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[43] Michael I Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steiger- wald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In International Conference on Learning Representations, 2023.
[44] Jonathan N Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. arXiv preprint arXiv:2306. 14892, 2023.
[45] Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, al. Multi-game decision transformers. In Advances in Neural Information Processing Systems, pages 27921- 27936, 2022.
[46] Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games. In Proceedings of the AAAI conference on artificial intelligence, pages 7187-7194, 2020.
13

---

[47] Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping differentiable games. In International Conference on Learning Representations, 2019.
[48] Wenzhe Li, Hao Luo, Zichuan Lin, Chongjie Zhang, Zongqing Lu, and Deheng Ye. A survey on transformers in reinforcement learning. Transactions on Machine Learning Research, 2023.
[49] Yingcong Li, M Emrullah Idiiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and implicit model selection in in-context learning. In International Conference on Machine Learning, page 19565-19594, 2023.
[50] Weiming Liu, Haobo Fu, Qiang Fu, and Yang Wei. Opponent-limited online search for imperfect information games. In International Conference on Machine Learning, pages 21567-21585, PMLR, 2023.
[51] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[52] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-compptiiie environments. In Advances in Neural Information Processing Systems, pages 1-12, 2017.
[53] Christopher Lu, Timon Willi, Christian A Schroeder De Witt, and Jakob Foerster. Model-free opponent shaping. In International Conference Machine Learning, pages 14398-14411, 2022.
[54] Yongliang Lv, Yuanqiang Yu, Yan Zheng, Jianye Hao, Yongming Wen, and Yue Yu. Limited information opponent modeling. In International Conference on Artificial Neural Networks, pages 511-522. Springer, 2023.
[55] Luckeciano C Melo. Transformers are meta-reinforceemnnt learners. In International Confer- ence on Machine Learning, pages 15340-15359, 2022.
[56] Long Short-Term Memory. Long short-term memory. Neural Computation, 98): 2010.
[57] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018.
[58] Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisy, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):508-513. 2017.
[59] Samer Nashed and Shlomo Zilberstein. A survey of opponent modeling in adversarial domains. Journal of Artificial Intelligence Research, 73:277-327, 2022.
[60] Yazhe NIu, Yuan Pu, Zhenjie Yang, Xueyan Li, Tong Zhou, Jiyuan Ren, Shuai Hu, Hongsheng Li, and Yu Liu. Lightzero: A unified benchmark for monte carlo tree search in general sequential decision scenarios. Advances in Neural Information Processing Systems, 36, 2024.
[61] lan Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems, pages 3003-3011, 2013.
[62] Georgios Papoudakis and Stefano V Albrecht. Variational autoencoders for opponent modeling in multi-agent systems. arXiv preprint arXiv:2001.10822 2020.
[63] Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial observability for deep reinforcement learning. In Advances in Neural Information Processing Systems, pages 19210-19222, 2021.
[64] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Bench- marking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Neu- ral Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), page 10707-10717, 1021.
14

---

[65] Keiran Paster, Sheila Mcllraith, and Jimmy Ba. You can't t count on luck: Why decision transformers and rvs fail in stochastic environments. In Advances in Neural Information Processing Systems, pages 18966-38973 2022.
[66] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick, Machine theory of mind. International Conference on Machine Learning, pages 4218-4227, 2018.
[67] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[68] Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4257-4266, 2018.
[69] Allan Raventos, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. In Advances in Neural Information Processing Systems, page 1-13, 2023.
[70] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron. Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions Machine Learning Research, 2022.
[71] Benjamin Rosman, Majd Hawasly, and Subramanian Ramamoorthy. Bayesian policy reuse. Machine Learning, 104:99-123 127, 2016.
[72] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Master- ing atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604-600) 2020.
[73] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint UrXiv:1707.06347 2017.
[74] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):
484-489, 2016.
[75] David Silver, Julian Schritwieser,, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of Go without human knowledge. Nature, (55077676)334-3588 2017.
[76] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 11101111111 1144, 2018
[77] Samuel Sokota, Gabriele Farina, David J Wu, Hengyuan Hu, Kevin Wang. Zico Kolter, and Noam Brown. The update equivalence framework for decision-time time planning. In The Twelfth International Conference on Learning Representations, 2023.
[78] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process opti- mization in the bandit setting: no regret and experimental design. In International Conference on Machine Learning, pages 1015-1022, 2010.
[79] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 151))1199-1198, 2014.
[80] Gerald Tesauro. Td-gammon, a self-teaching backgammon program, achieves master-level play. Neural computation, 6(2):215-219, 1994.
15

---

[81] Gerald Tesauro and Gregory Galperin. On-line policy improvement using monte-carlo search. In Advances in Neural Information Processing Systems, page 1068-1074, 1996.
[82] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-994, 1933.
[83] Yuandong Tian, Qucheng Gong, and Yu Jiang. Joint policy search for multi-agent collaboration with imperfect information. Advances in Neural Information Processing Systems, 33:19931- 19942, 2020.
[84] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, page 6000-6010, 2017.
[85] Adam R Villaflor, Zhe Huang, Swapnil Pande, John M Dolan, and Jeff Schneider. Addressing optimism bias in sequence modeling for reinforcement learning. In International Conference on Machine Learning, pages 12270-222833 2022.
[86] Friedrich Burkhard Von Der Osten, Michael Kirley, and Tim Miller. The minds of many: Opponent modeling in a stochastic game. In International Joint Conference on Artificial Intelligence, pages 3845-3851, 2017.
[87] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151-35174. PMLR, 2023.
[88] Jane X Wang, Zeb Kurth-Nelsson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning reinforcement learn. arXiv preprint arXiv:1611.05760 2016.
[89] Jannis Weil, Johannes Czech, Tobias Meuser, and Kristian Kersting. Know your enemy: Investigating monte-carlo tree search with opponent models in pommerman. arXiv preprint arXiv:2305. 13206. 2023.
[90] Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive rea- soning for multi-agent reinforcement learning. In International Conference on Learning Representations, 2019.
[91] Ying Wen, Yaodong Yang, and Jun Wang. Modelling bounded rationality in multi-agent interactions by generalized recursive reasoning. In International Joint Conferences on Artificial Intelligence, pages 414-421, 2021.
[92] Timon Willi, Alistair Hp Letcher, Johannes Treutlein, and Jakob Foerster. Cola: consistent learning with aepoonent-leamingg awareness. In International Conference on Machine Learning, pages 23804-23831, 2022.
[93] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Transformers: State-of- the-art natural language processing. In Conference on Empirical Methods in Natural Language Processing:
System Demonstrations, pages 38-45. 2020.
[94] Zhe Wu, Kai Li, Hang Xu, Yifan Zang, Bo An, and Junliang Xing. L2e: Learning to exploit your opponent. In International Joint Conference on Neural Networks, pages 1-8, 2022.
[95] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2021.
[96] Bing Xu, Naiyan Wang. Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations 2015.
in convolutional network. arXiv preprint
[97] Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.0412 2023.
16

---

[98] Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Dichotomy of control: Separating what you can control from what you cannot. In International Conference on Learning Representations, 2023.
[99] Tianpei Yang, Jianye Hao, Zhaopeng Meng, Chongjie Zhang, Yan Zheng, and Ze Zheng. Towards efficient detection and optimal response against sophisticated opponents. In Interna- tional Joint Conference Artificial Intelligence, pages 623-629, 2019.
[100] Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34:25476-25488, 2021.
[101] Xiaopeng Yu, Jiechuan Jiang, Wanpeng Zhang, Haobin Jiang, and Zongqing Lu. Model- based opponent modeling. In Advances in Neural Information Processing Systems, pages 28208 28221, 2022.
[102] Luyao Yuan, Zipeng Fu, Jingyue Shen, Lu Xu, Junhong Shen, and Song-Chur Zhu, Emer- gence of pragmatics from referential game between theory of mind agents. arXiv preprint arX7v 2020.
[103] Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. arXiv preprint arXiv:2306.0999 099277 2023.
[104] Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population-basee training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6145-6153, 2023.
[105] Stephen Zhao, Chris Lu, Roger B Grosse, and Jakob Foerster. Proximal learning with opponent- learning awareness. Advances Neural Information Processing Systems, pages 26324- 26336, 2022.
[106] Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjie Fan. A deep bayesian policy reuse approach against non non-stationary agents. In Advances Neural Information Processing Systems, pages 962-972, 2018.
[107] Luisa Zintgraf, Sam Devlin, Kamil Ciosek, Shimon Whiteson, and Katja Hofmann. Deep interactive bayesian reinforcement learning via meta-learning. In International Conference on Autonomous Agents and MultiAgent Systems, pages 1712-1714, 2021.
17

---

Table of Contents for the Appendix
Extensive Related Work
19
Pseudocode of OMIS
21
Construction Process of Deplk
D Proofs of Theorems
21
D.I Algorithm of Posterior Sampling in Opponent Modeling
21
Proof of Lemma 4.1
22
D.3 Proof of Theorem 4.2
25
D.4 Proof of Theorem 4.3
27
Detailed Introductions of the Environments
29
F
Architettue Design
29
Diversity of Opponent Policies
30
H
Hyperparameters
33
H.1 Hyperparameters for Opponent Policies Training
33
H.2 Hyperparameters for In-ContextLearring-badd Pretraining
33
H.3
Hyperparameters for Picisiin-Timee Search with In-Context Components
34
Quantitative Analysis of Attention Weights Learned by OMIS
34
In-depth Discussion
36
Compute Resources
36
Broader Impacts
36
18

---

A Extensive Related Work
Opponent modeling. The recent research in opponent modeling based on machine learning method- ologies can be broadly categorized as follows:
(1) Opponent Modeling with Representation Learning: Embed the opponent policies into latent space using representation learning methods to enhance the crriiiionmmaking capabilities of the self-agent. In the study by Grover et al. [28], imitation learning (37] and contrastive learning [39] were utilized to produce policy embeddings for opponent trajectories. Subsequently, these embeddings were integrated with reinforcement learning for policy optimization. Comparable initiatives, exemplified by He et al.
229 and Hong et al. [33], employed auxiliary networks to encode manually crafted opponent features, predict opponent actions, and finetune the policy network to enhance overall performance. Papoudakis et al. [63] suggested employing an autoencoder to leverage the self-agent's 's observations and actions for reconstructing the opponent's 's observations and actions. This approach aims to learn embeddings that facilitate decision-making In comparison, Papoudakis and Albrecht [62] and Zintgraf al.
107 utilized Variational AutoEncoders (VAE) to capture the high-dimensional distribution of opponent policies.
(2) Opponent Modeling with Bayesian Learning: Detect or deduce the opponent policies in real-time using Bayesian methods and subsequently generate responses based on the inferred information. Bard et al. [0] initially trained a mixture-of-expert counter-strategies against a predefined set of fixed opponent policies. They then utilized bandit algorithm during testing to dynamically select the most suitable counter-strategy. Building on BPR+ 7 32]. Zheng et al. [106] introduced a rectified belief model to enhance the precision of opponent policy detection. Furthermore, they introduced distillation policy network to achieve better results. DiGiovanni and Tewari [99 utilized Thompson sampling (82) and change detection methods to address the challenge of opponent switching between stationary policies. Fu et al. [24] employed a bandit algorithm to select either greedy or conservative policies when playing against non-stationary opponent. The greedy policy underwent training via VAE in conjunction with conditional reinforcement learning and was continuously updated online through variational inference. In contrast, the conservative policy remained fixed and robust policy. Ly et al. [54] introduced a similar approach to exploit non-stationary opponents.
(3) Opponent Modeling with Meta-learning: Leveraging meta-leaming methods [34], train against set of given opponent policies to adapt to unknown opponent policies during testing swiftly. While most meta-reinforrcement learning methods presume that tasks in training and testing exhibit similar distribution, this category investigates the possible application of metza-reinforcement learning in the context of competing with unknown non-stationary opponents. Al-Shedivat et al. 3 introduced gradient-based meta-learning algorithm designed for continuous adaptation in non-stationary and competitive environments, showcasing its efficacy in enhancing adaptation efficiency. Building upon analysis on Al-Shedivat et al. 33 Kim et al. [41] introduced a novel meta multi-agent policy gradient algorithm designed to effectively handle the non-stationary policy dynamics inherent multi-agent reinforcement learning. Zintgraf et al. [107] introduced a meta-learning approach for deep interactive Bayesian reinforcement learning in multi-agent settings. This approach utilizes approximate belief' inference and policy adaptation to enhance opponent adaptation. Wu et al. [94] put forward a meta- learning framework called Learning to Exploit (L2E)) for implicit opponent modeling. This framework enables agents to swiftly adapt and exploit opponents with diverse styles through limited interactions during training.
(4) Opponent Modeling with Shaping Opponents' Learning: Considering opponents' learning (i.e., conducting gradient updates), estimating the mutual influence between the future opponent policy and the current self-agent's 'ss policy. Foerster et al. 22 proposed an approach named LOLA, which modeled the one-step update of the opponent's policy and estimated the learning gradient of the opponent's policy. Foerster et al. (23] introduced a Differentiable Monte-Carlo Estimator operation to explore the shaping of learning dynamics for other agents, building upon the approach presented by Foerster et al. [22]. Letcher et al. 47 further integrated stability guarantees from LookAhhead with opponent- shaping capabilities from Foerster et al. [22] to achieve theoretical and experimental improvements. Kim et al. [41] also presented term closely related to shaping the learning dynamics of other agents' policies. This considers the impacts of the agent's current policy on future opponent policies. Lu et al. (53] proposed a meta-learning approach for general-sum games that can exploit naive learners without requiring white-box access or higher-order derivatives. Willi et al. [92] introduced Consistent LOLA (COLA), new multi-agent reinforcement learning algorithm that addresses inconsistency
19

---

issues with Foerster et al. [22]ss approach. COLA learns consistent update functions for agents by explicitly minimizing a differentiable measure of consistency. Zhao et al. [TO5] proposed proximal LOLA (POLA), an algorithm that addresses policy parameterization sensitivity issues with L.OIA and more reliably learns reciprocity-Net basee cooperation partially competitive multi-agent environments. Fung et al. [25] further presented that the sample complexity of Lu et al. (53]' 's algorithm scales exponentially with the inner state and action space and the number of agents.
(5) Opponent Modeling with Recursive Reasoning: By simulating nested layers of beliefs, predicting the opponent's 's behavior, and generating the best response based on the expected reasoning process of the opponent towards the self-agent. Wen et al. [90] and Wen et al. [9]] suggested conducting recursive reasoning by modeling hypothetical nested beliefs through the agent's 's joint Q-function. Their work
demonstrated enhanced adaptation in stochastic games. Dai et al. [17] introduced recursive reasoning by assuming agents select actions based on GP-UCB acquisition functions [78]. This approach achieved faster regret convergence in repeated games. Yuan et al. [102] proposed an algorithm that utilizes ToM- recursive reasoning and adaptive reinforcement learning. This approach enables agents to develop pragmatic communication protocol to infer hidden meanings from context and enhance cooperative multi-agent communication. Yu et al. [001] proposed a model-based opponent modeling approach that employs recursive imagination within an environment model and Bayesian mixing to adapt to diverse opponents.
(6) Opponent Modeling with Theory of Mind (ToM): Reasoning about the opponent's 's mental states and intentions to predict and adapt their behavior. This involves modeling their beliefs, goals, and actions to understand opponent dynamics comprehensively. Von Der Osten et al. [86] introduced multi-agent ToM model designed to predict opponents' actions and infer strategy sophistication in stochastic games. Building upon Zheng et al. [106]* 's Bayesian online opponent modeling approach, Yang et al. [99] proposed a ToM approach. This approach leverages higher-level decision-making to play against opponents who are also engaged in opponent modeling. Rabinowitz et al. [66] and Raileanu et (68] also delved into methodologies for modeling an opponent's mental state. Rabinowitz et al. [66] utilized three networks to reason about agent actions and goals, simulating a human-like like ToM. Raileanu et al. (68) proposed utilizing their own policy to learn the opponent's goals in conjunction with the opponent's 's observations and actions.
Our work aims to pioneer a new methodology: Opponent Modeling with Decision- -ime Search (DTS). We explore how DTS can work in the context of opponent modeling, intuitively. DTS can make our policy more foresighted. To imbue DTS with opponent awareness and adaptability, we developed a model based on in-context learning to serve as the foundation for DTS.
Transformers for decision-making. There is an increasing interest in leveraging Transformers for
by framing the problem as sequence modeling (97 48 Chen et al. [T5] introduced Decision Transformer (DT), model that predicts action sequences conditioned on returns using a causal Transformer trained on offline data. Subsequent studies have explored enhancements,
such as improved conditioning [26 65). and architectural innovations [85 Another interesting direction capitalizes on the generality and scalability of Transformers for multi-task learning [45] 70).
Transformers applied to decision-make making have demonstrated meta-learning capabilities well (55)
Recently, Lee et al. [44] introduced a Transformer-based in-context learning approach that, both empirically and theoretically, surpasses behaviors observed in the dataset in terms of regret, performance metric where DT falls short [1] 98]. Incorporating these insights, our work employs a causal Transformer architecture to maximize the model's 's ability for in co-tentex learning, specifically in the realm of opponent modeling.
20

---

B Pseudocode of OMIS
Algorithm 1 Opponent Modeling with In-context Search (OMIS)
1: Prepare all the opponent policies +/
2: Train an opponent policy population using the MEP algorithu to construct Il and I1
3: /+ in-context-learning -aase pretraining (Section 4.1) +/
4: Train the BR against each opponent policy in I using the PPO algorithm and get {BR( 11,)) 1R-1-
5: Initialize we with parameters U&s with parameters ø, and V. with parameters w.
6: while not converged do
7: Sample a opponent policy π 1 ~v II play an episode against it using BR(T 1,)) == = (als), and collect the training data ( 17 where D == (st, DI , I.K." a _-1k G..h.t)
8: Calculate losses based on Eqs. (3) to (5), and backpropagate to update 0, d,
9: end while
10: /- Decision-time search with in-context components (Section 4.2) +/
the policy switching frequency for unknown non-stationary opponent agents
12: for num_ swnnsswichhh in max_ foox_switchie wwwttwwnggggnttteeeeee number do
13: Sample true opponent policy x ~ 11 11 using uniform distribution, and set & a-1
14:
for ep in [E] do
15:
for t [T) do
16:
Decision-time search actions Rollout L steps for M times using Ne, Mae V., and estimate the value for at by
17:
for a; in a do
18:
++L Σ
at) =
Nrench
fb + Tranch Ve+z41
19:
end for
20:
Take real actions
21:
The opponents according b while simultancously the self-agent acts according
(St) = Towerth (8+), [[Q(se, (s:)] where Treach (st) == arg max ((s., a!).
aj ~ TT(.1se Dz), otherwise
a
22:
end for
23:
end for
24:
end for
C Construction Process of Depi.k
The construction of Depik is follows:
1. Sample C' trajectories from all historical games involving π -1,k
1,6 h.t - 2. For each trajectory, sample consecutive segments {(S 9x-1 Ih-mh where h, is the starting timestep:
3. Concatenate these segments together.
The construction of Depik stems from two intuitions: Firstly, -1i''s gameplay style can become more evident over continuous timesteps, so we sample consecutive fragments. Secondly, -ikk can exhibit diverse behaviors across different episodes, so we sample from multiple trajectories.
D Proofs of Theorems
D.1 Algorithm of Posterior Sampling Opponent Modeling
We instantiate Bayesian posterior sampling algorithm in the context of opponent modeling, referred to as Posterior Sampling Opponent Modeling (PSOM). In the PSOM algorithm, we use opponent trajectory (8o, ap 8T-1, arr_), which consists of consecutive (s, 11) tuples, to construct the in-context data D. Following up, describe the PSOM algorithm in most general way 44).
21

---

Given the initial distribution of opponent policies Ho f- Пpe. where Im is the probability distribu- tion on Iltrain, for e [C]:
1. Sample an opponent policy qe by II, and compute the BR policy _. for the self-agent;
2. Interact using the self-agent with u/ against the opponent with true opponent policy ă and use the opponent trajectory (50, 4p ..... 1, ar-1) to construct D.
3. Update the posterior distribution IT,(T P(
D.2 Proof of Lemma 4.1
Lemma 4.1 (Equivalence of OMIS w/o S and PSOM). Assume that the learned Tι is consistent and the sampling of s from pm independent of opponent policy, then given ã- "and its D, we have P, π PSOM) = P(t+ D, म - ;oo) for all possible E
Proof. In this section, we use π to denote opponent policies posteriorly sampled from Ug.e by the self-agent and use #- to denote the true opponent policy interacted with during the testing stage of OMIS. In the proof, we abbreviate OMIS without DTS OMIS. For clarity and ease of understanding. all trajectory sequences are indexed starting from in this section (originally starting from in the main text). We abbreviate D D in the proof, as D is sufficient for completing the proof. Define & ($1, af ..... sT, 1.,") as self-agent history, where T denotes the maximum length of this history (i.e., horizon for each episode) and a-.* is sampled from the best response policy π 1. against π -1 -Tpe=( (-: denotes the probability distribution on all the trajectories involving --1 during pretraining.
Let π -1 1pe and D contain sampled trajectory fragments of - and let Squery ES, &Saat* A', - (S x A')T-1 and [0, T 1] be arbitrary, the full joint probability distribution during OMIS's pretraining stage can be denoted as:
(D; ( DD Squayy a "'"" Ipre ( 'T
')62($1:1 S (Squary (aa"" Squeryy
pre
x Unif(O, T
II iE[T]
7.''" (a.
(11) ) which is independent of opponent policy. Squery is set to the uniform over S. In addition, we sample t ~ Unif(0, T 1] and truncating & from _ (or, equivalently, sample 4 A((S x A directly).
Herein, Gr E A(S
We define the random sequences and subsequences of the self-agent trajectory under PSOM algo- rithm (tsoM D) (SPSOM A PSSM St SPSOM A! A.PMM) This trajectory is generated in the
following manner:
"SSOM P( D SPSOM P.
A 1.PSOM
~ π (111-(11111 (-15, (-IS. TpsuuM) A, 1.POOM ~ # (-1sss, )
1,
S PSOM P(-ISPSOM A! I.PSOM A, L.POMM ) 1 2. (1
Within, P denotes the initial distribution on S. Analogously, we define the random se- quences and subsequences of the self-agent trajectory under OMIS algorithm as Epre D) (STre A pre St A! A!meee). This trajectory is generated in the following manner:
spre P, 1
Ppre (-1SST D.
Ep (i - 1; D)); 2 ~ -+SS") 21,
4+1 ~ p
A! pme
A, pre
2.
To simplify matters, we will refrain from explicitly referencing D for = in our notations, except when required to avoid confusion. Next, we introduce a common assumption to ensure the neural network fits the pretraining data distribution.
22

---

Assumption D.1 (Learned Te is consistent). For any given D, 1; D)),
Prre(A pre mm (i (i- 1; D))
(A Pm 1 D, ppr 1: D)) for all possible A?tre
Upon Assump.
D.I. we will limit our attention to pree for the rest of the proof.
To prove that v P() prove that
= P( D, mo) (Le., Lemma 4.1), is equivalent to
P(EPsOM(T) { P( -
(12)
We will prove that AA E (T),
P(EPsOM(t) ( = P(=pe (t) = (
(13)
using Mathematical Induction and then introduce a lemma for the evidence of Eq. (2)
To begin with, propose a lemma to assist in proving Eq. 33 for the base case when / 1.
Lemma D.2. If the sampling of from Tpr is independent of opponent policy, then Ppre(T) (D) P(TpsOM = x ID).
Proof. Assuming the sampling of from Tpm is independent of opponent policy, we have:
P(TPsOM π DD) x IT n (nm+1))pp P)D(D n 1
(14a)
x Ipr (T ))
II क (a,
(14b)
JE[[PI]
pre
II
(aj (18)) T (15) (s3)
(14c)
]E[[D]]
=Ippe (T TT (D;
(14d)
X (D))
(14e)
denotes that the two sides are equal up multiplicative factors independent of -- Eq. (14a)
derived through the Bayesian posterior probability formula. Eq. (14b) uses the fact that s in posterior sampling is independent of opponent policy. Eq. (14c) holds because of the assumption that the sampling of s from Tp is independent of opponent policy. Eq. (14c) uses the definition of Tpee Eq. 14e) is derived through the Bayesian posterior probability formula.
Now, we prove that Eq. 13 holds when & = 1:
P(EPsOM(1 (1) = +1) = P(S 81, A
1.PSOM = a
(15a)
p(si)
P(A 1(A
1.POMM al, TPSSM - 1SSSSOM =s1 =s)ddT
-1
(15b)
p((1)
1 r''*(a(ls1: PPsoM x ID $1 )dr
(15c)
p(si)
***(afls 1) PPSPM (TPSOM = - -DDdd
(15d)
-((1))
n ['a((alss Ppo DD)d (D))dr
(15e)
p(s)) (a D)
(15f)
P(Eme E--
(15g)
Eqs. [5a) to (5c)., (55 and (15g) are derived using Bayesian law of total probability and conditional
probability formula based on Eq. (II). Eg. (15d) holds because the sampling s is independent of
- Eq. 15ep is derived through Lem. D.2
Next, we start proving Eq. (1) for the other cases when / * 1. We utilize the inductive hypothesis to demonstrate the validity of the entire statement. Suppose that P(EpsoM - 1) {
23

---

Pt - 1) = 5t-1 ) since
P(Epsov(() = 61) =
P! - P(
SPOOM St. A, A..SSO = of SPsoSs (6- -01 1) = 5%-1)
and
P(Epo((t) [/)]=
P(Epool - 1) = P(S) 86: A! = =a o (64-1] = f(-1);
to prove that P(ErsoM (1) {t) P(Epre (t) = (, ) it equivalent to prove:
PlS 8t, A I.PSO at EpSOM (t 1) 1)
(16)
= P(S St, A 1 pre a, =pro - 1) = -)--)).
By expanding Eq. (16). we can get:
P P(StSOM = A₂ PSOM a 1) -
P(s = P
PSOM a/ 15 S5PSMM St, EpsoM(t -1 = cl-1)
(17a)
=f0 P(se|si-1, 2, (1 (
Jai-1
.SSOM P(A.
0e, TPSOM - 1 8t, EpsoM 1) -1-1)d
(17b)
In Eq. (17b). the first integral term the same for PSOM and OMIS, while the term inside the second integral term satisfies:
P(4 PSSM a, PSSOM =
- -|S/SOM St; EpSOM 1) = = 1, (at [88; x (( = π 11 = 81, EpsoM - 1) = 4-
(18)
Based on Eq. and Eq. (8), to prove that Eq. (16) holds, it equivalent to prove:
P P(TPSOM
1|
SPSS = St, EpSOM (t - 1) = 4 = Pre - 1/,,, D, (6
(19)
We prove that Eq. 19 holds through the following derivation:
P( SSSSSM St, EpsoM( (1- 1) = 61-1)
P(SPSOM 8t, Ep (1-)) 67 TPSOM 1 ) N -
(20a)
P(SPSOM 81: EpsomI 1) = E(-1)
Ppelte ( D)
II
P(s a (at
(20b)
kE[t-1]
x Ppre(t (
II kE(t-1]
~ (aHlss:
(20c)
P pre -T- D) D)Squery (
x (af(app iE(t-1]
II
(20d)
(T st, D.
(20e)
Eq. (00 is derived through the Bayesian posterior probability formula. In Eq. (0b decompose
the probability of observing the sequence observations s and actions a Eqs. (0ch and (0dd use
the fact that the sampling of is only related to the true opponent policy and is independent of
o Eq. (20eh is derived by the definition of Ppre -111. D, D.
Therefore, we finish the proof of P(EpsoM(() { P(Epre()) {!), where
((EPsoM(t) = =
24

---

P( (lo / T (a Ppree ( D, 1
(21a)
= (t - = - Pme(a (at D. D.
(21b)
P( (t) = 3.
(21c)
Based on Mathematical Induction, Eq. 13 holds for any t E [T). Hence, Eq. 12 is satisfied. This concludes the proof.
D.3 Proof of Theorem 4.2
Theorem 4.2. When Æ = = - E 11 if the PS algorithm converges to the optimal solution, then OMIS w/o recognizes the policy of Q as π 1.k i.c., To, pig, and Vu converge to n xi.k,,, x -1,k, and VLk.*, respectively; When Ā-1 & T]train, OMIS w/o S recognizes the policy of & as the policies in Mh with the minimum DEL (Pcz(P(aal ->) P(a 1
Proof. In the proof, we abbreviate OMIS without DTS as OMIS. We denote the in-context data consisting of (s, a a-1) tuples as D, and the in-context data consisting of (s, ,", r²) where s' is the next state of transitioned to. Note that the original PS algorithm uses D' while PSOM and OMIS use D to recognize the policy of +
8
tuples as D',
To begin, we propose a lemma and its corollary to prove the convergence guarantee of the PSOM algorithm and analyze its properties in opponent policy recognition.
Lemma D.3. Let f(T ; D) Sam-1 P(s, a : D) log(P(a (ls, x ")dsdd and क..
arg min ET f(T (m= D), then VF- E { {f * ;( D)}, we have P(# (s.a h)a)
p
P(r. (s.a 11n)))
Proof. Here, n.. denotes the equivalent class of opponent policies to which PSOM converges with non-zero probability. a1s,a=1; 1)) is the distribution of (s, a a- tuples D. n is the number of the (s, (s,a")) tuples. To prove that P(= P(==' (8.0 bind
P. under the given conditions, it equivalent to P(r. (1mm))))n)))
prove:
P( ( a ))mm) Pg +00.
L 1,a log
(22)
P(T. 1(s,, a ((s,a--1)nn) 1=m)
By expanding Eq. (22 we can get:
P(T P(T (ss, (1s,a-))a )1on) log
P(a
Lx 1,a - log
log(
(23)
P(T [(s, ((s,a-1)10)
TT
P(a 1 T& .8)
i=1
According to the definition of m- and the condition f(n fm= D) * { D), have:
Eis ))P((-))-- Σ log( P(a | 8) . :;; f( | D) C> 0. (24)
P(a- |.
Here, C is positive constant. Therefore, based the law of large numbers, have lim P(] ---i. - >>>e e) = 0, where is any positive number. Hence, Eq. 22 is satisfied, and the proof ends.
Corollary D.4 (Corollary of Lem. D.3). When --1 II we have #-1 E mp When क é Ilaain ..- is the equivalent class of policies in I[ with the minimum DKr, ")]lP(( " "|,,
Proof. Since P(s, D) P(s: 11) holds, we have
/ P(s,a" log( (a(a -lls, ( )dsda
म.. arg
min uii
(25a)
25

---

arg
min
P(5: logg )dsdd-
(25b)
x Ell
s,a-1
arg
min Ellinn
P(s: [s, log ((ag(((a - dsdda
9,a-1
P(s; D)P(a 1aP log ))dsddd
(25c)
log ())
arg
min Elllmn
P(s: - |s,
(og(P(a-1) =,, dsda
(25d)
8,a-1
arg
min
(s ||, 1))=s
(25e)
arg
min E (PL ||, |||,==)||P(( PP ||,, न - Eellini
(25f)
When D the in-context data of the opponent policy म = 0 in Eq. 25t). and ग.
Mlaaain
DRr. (P(a-1/s 11 policies that have the same action distribution When D is the in-context data of an opponent policy ग g Iuaain ग the equivalent class of oppo-
is the equivalent class of opponent
Le., Va - P(a P(a- (,,,,,)
P(a-11s, o
nent policies that minimizes the expected KL divergence (K I|s 1) P(a I|, म 1)). O
Using a similar proving method as in Lem. D.3 it can be proved straight- forward that the PS algorithm
converge: Let
f(T f(m=1::')
arg wrg.in,,14e Ellu f(x D'), then Va-1 E { If(=If(( DD) * f(r 1-11
P(s, a aaaa,st s'1 D'); log(P(s") P(s"",r'|
rr ls al, a ")ssda ds' dr'
and
;')}, have
P(s=')(s,a ,r")) 4 0.
P( (s.00 "K..0 """,),)
Next, we introduce lemma prove that if the PS algorithm converges to the optimal solution, PSOM converges to the optimal solution.
Lemma D.5. Given s,a Laa,,, Æ-1 _T. if Va Va- ,(aa ss, و P(a-'Is, holds, it can be deduced that Vs ,r', P(s", rr|ss, T P(s") , 18, -1), but the reverse is not true.
Proof. For the forward deduction (i.e., =), we have:
Vs5 r1 P(s" ,r'ss,a (,,,ammm)))
Σ P(a 18, 1 r r''s,,",s',a"" a
(26a)
P(a P(a-'1s,5 ( "'
(26b)
P(5" r2 12, , a
(26c)
For the backward deduction (i.e., =), counterexamples exist. For example, when A-1 C A-1 and 1 = P(a P|a- - Ea-IeA-, P(a lss क.,"
p( r as, 8 a- takes equal values for some a P(a |
Va E A-1 A
Em-IEA-1 P(a - hold, P(a (s,-((sTT,1) = P(a-1|s, (-) does not necessarily hold for all a E A-1 This means when म E uai the PS algorithm may lead to distributions on opponent policies with non-zero probability other than the equivalence class of -1 resulting in potential suboptimality compared to PSOM.
According to Lem.
D.5
π. C .!-|_ Based Cor. D.4 have -ー ., Thus,, conclude Hence, if PS converges to the optimal solution, PSOM converges to the
that --1 E .. C
optimal solution.
Based on Lemma 4.1, can be inferred that OMIS is equivalent to PSOM. Thus, all the proofs in this section also hold for OMIS. We have the following conclusions:
26

---

1. Based on Cor. D.4 and Lem. D.5 when カー E II if the PS algorithm converges to the optimal solution, then OMIS converges to the optimal solution. If the true opponent policy is π -1. OMIS recognizes the current policy of + as π -l. In this case, converges to xt L.k,, Similarly to the PSOM algorithm App. D.1 when we replace n1.* with a- and VI,,, we can derive the algorithms with the same theoretical guarantees. Thus, H& and V converge to π -1. and VL&k,, respectively.
1.
2. Based on Cor. D.4 when æ-1 & Ie OMIS recognizes the current policy of + as the policies in IT with the minimum KL divergence D >>>>>>> P(( 1s, t )))
This concludes the proof.
D.4 Proof of Theorem 4.3
Theorem 4.3 (Policy Improvement of OMIS's DTS). Given 7 and its D. suppose OMIS recognizes as n. and V 1 is the value vector on S, where V(s) == (s, D), (a]s)) == To(als, D). Let OL be the L-step DTS operator and x' E ( then V n.
2 V holds component-wise.
n, "
Proof. To begin with, we do not consider the mixing technique the proof. Based on Theorem 4.2, given π and its D, OMIS recognize the policy of t as ग._ which means Ta, Há, and V. converge to x nT. and V respectively. When # E 11train since the labels in the pretraining data may not be optimal, there space for improvement in the (i.e., π .!")). When & 11train т may not be the best response against म -1 thus there is still space for policy improvement. Furthermore, disregarding the impact of De on Pgg P.g. can be treated as fixed policy during the DTS process. Thus, the virtual environment for the DTS is stationary.
L-greddy policy w.r.t the value function V,,,, belongs to the following set of policies,
arg max max No II TE..AL--
Σ
Thaschh M(s, ((8)) + Treanch Va, (sr.)
(27a)
= arg max To
R(So, To( +=))))==)))))))) + Tseurch ( (T (s11)
(27b)
where the notation Ry. 1 means that we condition on the trajectory induced by the choice of and the starting state Su = . The म.. terms in R means
actions (#o(s)), Tı (s.)...., opponents take actions by pe conditioned on D. We define Tx as the operator choosing actions using -² for one step, where n1 is any self-agent policy. We define T the Bellman optimality operator, where
TV R
max T V - a²
(28)
Following up, we define T (shown in Eq. (7bb the L-step Bellman optimality operator, where
TVV
max
R(st, (((s): म, 77eacch Vn- (SL)
(29)
To.
Since the argument in Eq. 27b a vector, the maximization is component-wise. i.e., we want find the choice of actions that will jointly maximize the entries of the vector. Thus, the L-greedy policy chooses the first optimal action of a non-stationary, optimal control problem with horizon L. Since is maximized to select actions during OMIS's 's DTS, Eq. 9 can be considered equivalent to Eq. (7
The set of L-greedy policies w.r.t. V= i.e., the L-step DTS operator, G1 (V=V..)), can be expressed as follows:
VV F. 1, ग TE V= def T T''TL-IV
(30a)
We use Rt. 1 0. ...... 1 to denote w each step for simplicity as they can be different based on 'In Eq. (8). the number of rollout steps is actually L + as we need to perform rollout of I step for each legal action first. However, this does not affect the conclusions.
27

---

VV n. 9L = {
= V ,
(30b)
For Eq. (30a). the operator T T represents choosing actions using π in the first step and selecting the optimal actions from all possibilities in the subsequent L steps. For Eq. (30b), the policy x' derived from G2 (V s, 1) satisfies that choosing actions using in the first step and selecting the optimal actions from all possibilities in the subsequent L steps is equivalent to choosing all possible optimal actions in all steps.
We observe that
V T
TV Sr. ".
(31)
Then, sequentially using Eqs.
30ap. 306
and 31
we have
V Vー = ( ... TVVx T
Txx (TL-LVE-)
(32)
mx
This leads to the following inequalities:
V n. - STT (T (T'-
(33a)
S TT (T4-ITVZ, n. )) = 7 (TL
x--) ) (T (T4-IV 1) = To (Tn'TL-iV)) vT n,
(33b)
(33c)
n
1 x lim
(T vo Vil
(33d)
Within, Eq. (33a) holds because of Eq. 32 Eq. (33b is due to Eq. (31 and the monotonicity of Tx and T (and thus of their composition), Eq.
33ch is derived by Eq. (32), and Eq. (33d) is due
to the fixed point property of T Lastly, notice that VT_, V= if and only if (see Eq. (31)
.. x_1 TV" V which holds if and only it is the optimal policy. This concludes the proof.
28

---

E Detailed Introductions of the Environments
Predator Prey 15 is a competitive environment with a three vs. one setup and continuous state space, The environment consists of three predators (in red), one prey (in green), and two obstacles (in black). The goal of the predators is to capture (iee. collide with) the prey as much as possible, while the goal of the prey is to be captured as little as possible. The environment features sparse rewards, where each time predator captures the prey, the capturing predator receives reward of 10, and the prey receives a reward of - 10. Additionally, the environment provides a small, dense reward to the prey to prevent it from running out of the map boundaries. Here, the prey is the self-agent, and the three predators serve as opponents. From the perspective of the self-agent, the environment is highly unstable, there are three opponents with unknown policies in the environment. The challenge in this environment is that the self-agent must model the behavior of three opponents simultaneously and adapt to various potential coordination strategies employed by the opponents (e.g., surrounding from three different directions). For specific implementation of this environment, adopt the open-source code of Multi- Particle Environment, which is available at https:/ thub. com/ openai mult t
www.pppattiieeeeep envs
Level-Based Foraging [16 64] is mixed environment in 9x9 grid world containing two players: the self-agent (in blue) and the opponent (in black), along with five apples (in red). At the beginning of each episode, the two players and the five apples are randomly generated in the environment and assigned a level marked in their bottom-right corner. The goal of the self-agent is to eat many apples possible. All players can move in four directions or eat apple. Eating an apple can be successfully done only under the following conditions: one or two players are around the apple, and all players who take the action of eating apple have a summed level at least equal to the level of the apple. The environment has sparse rewards, representing the players' contributions to cating all the apples in the environment. The environment is essentially a long-term social dilemma and can be viewed as an extension of the Prisoner's ' Dilemma [6]. The challenge in this environment that the self-agent must learn to cooperate to eat high-level apples while greedily eating low-level apples simultancously. For specific implementation of this environment, we adopt the open-source code of forrgiggn which is available at https: / tighhb...... com wwwm///eeeaaeeeeeeeeeeeeeaeeeeee foraging
OverCooked [4] is a cooperative environment with high-dimensional images as states. One of the chefs is the self-agent (green), while the other chef is the opponent (blue). The two chefs must collaborate to complete a series of subtasks and serve dishes. All players share the same sparse rewards, earning 20 for each successful dish served. The more successful the dish servings, the higher the reward. The goal of the self-agent is to collaborate as effectively as possible with the other chef to maximize the return. The challenge in this environment is for the self-agent to not only be able to complete subtasks such as getting onions, putting onions into the pot, and serving dishes but also to coordinate intensively with the opponent. It requires the self-agent to allocate subtasks effectively with the opponent, ensuring that it does not negatively impact the opponent (e.g.. not blocking the opponent's 's path). For specific implementation of this environment, we adopt the open-source code of Overcooked-Al -AI, which is available at https: //gi: //tgtggt...bttt// thub. com/ HurnarCCmmatiiDllll//ooorookoee ai
F Neural Architecture Design
For OMIS, we adopt the neural architecture design as follows:
The buckbone of the OMIS architecture is mainly implemented based on the causal Transformer, i.e., GPT2 (67] model of Hugging Face [93]. The backbone is a GPT2 decoder composed of self- attention blocks. Each self-attention block consists of a single-head attention layer and feed-forward layer. Residual connections [30] and LayerNorm 7 are utilized after each layer in the self-attention block. Within each attention layer, dropout [79 is added to the residual connection and attention weight.
In the backbone, except for the fully connected layer in the feed-forward layer (the feed- forward layer consists of a fully connected layer that increases the number of hidden layer nodes and a projection layer that recovers the number of hidden layer nodes), which consists of 128 nodes with GELJU (31 activation functions, the other hidden layers are composed of 32 nodes without activation functions. The modality-specife linear layers for self-agent actions, opponents actions, and RTGs comprise 32
29

---

Testing
13
D
ViS-i
V.
II pest
j
causal transformer
&1
S11
PLOI
VO-
Figure 10: The architecture of OMIS during testing.
nodes without activation functions. The modality-specifie linear (and convolutional) layers for states comprise 32 nodes with LeakyReLU (96) activation functions.
For input encoding, states 8, self-agent actions w², opponents actions a a- RTGs G are fed into poodality-specifff linear layers. For OC, additional convolutional layers added before the linear layers to encode the state s better. positional episodic timestep encoding is added. We adopt the same timestep encoding in Chen al. [15]. In addition, agent index encoding is added to each token to distinguish the inputs from different agents.
For output decoding, the sequences of embedded tokens are fed into the backbone, which autoregres- sively outputs the self aglnt actions a', opponents actions a-1, values V the positions of state tokens using causal self-attention mask. The who outputs at, the us who outputs a a- and the Ve who outputs V all consists linear layers.
G...A. Il the input sequence is l.k. 11, Gl.,. st), ($1, a, 1,1.E 1.k ..... šH, 1-1,k St-B+1, O4-B11 B+1 4,B+11 1--..1.... 0--1,0-11 .,-1 where is the maximum sequence length as Transformer model has a token capacity. The output
During pretraining, for each timestep t given π 11, 1.k.,
11, B+1
prediction sequence is (a (d.- Ot-B1 V--B+1 a V2- The output label sequence 1.k is (4,- "I-B+1 G
Gl.k,*
1.k.a
a0-i G!-11)
11,
d-1
the input sequence is (31, ā, ...,šH "H St-B+1, a, B-1 O Vi-B+1.... $1-1 0 11 Vi-1,s), where â=! is the true actions Φ. The output sequence is (a/-B+1> a B B+1..... a. a Vi-1). We
During testing. for each timestep i given & 1-1 क
demonstrated the architecture of OMIS during pretraining in Fig. see the architecture of OMIS
during testing in Fig. 10
For all the baselines, we adopt the neural architecture design as follows:
We replace the original backbone of the baselines (e.g., linear layers, recurrent layers, LSTM [56.. and more) with the same GPT2 backbone as OMIS. For input encoding and input encoding, we encode and decode states s and actions using the same modality encode and decode rewards r using the modality-specific layers used to encode RTGs in OMIS. Note that we only modified the neural architectures of all the baselines ensure fair comparisons. All the baselines are still pretrained and tested according to their respective methodologies.
layers OMIS. We
G Diversity of Opponent Policies
As mentioned in Sec. 5.1, we run the Maximum Entropy Population-basee training algorithm (MEP) to generate a diversified opponent policy population. Nevertheless, quantitative analysis is still necessary to measure the similarity/disimilariyy between different opponent policies within the MEP population. We calculate the pair-wise KL. divergence between different opponent policies to measure their dissimilarity. The results for PP, LBF, and OC are shown in Figs. [1 to respectively.
Thiis timestep can be the real timestep, and it also can be virtual timestep during the DTS.
30

---

Predator Prey
1.20
95
06
1:31
1:05
1.19
1.65
2.22
1:51
3.5
2:10
1.88
2.25
2.42
2:38
2:07
2.58
1.95
2.25
1.41
1.24
1.40
2.75
1:32
2:32
2.22
0.00
1.05
1.75
2.02
1.84
2.39
3.25
3.87
99
2.00
3.24
2.62
2.36
3.70
2.69
2.17
1.49
2.91
0.00
2.06
1.06
1.01
1.95
1.15
.000
.990
14
2:25
1.45
3.0
11.14
Opponent Policy
1.13
2:75
2.5
2.89
3.34
3.33
3.00
2:35
83
2.88
3.72
3.39
2.73
2.93
0.00
1.84
2.48
2.09
1.30
1.56
2.58
2.05
2.05
2.3.
2:37
2.33
2.39
2.00
3.88
2.46
2.03
0.90
2.09
2:93
1.42
2.54
2.322
5.52
1:20
2:17
1:90
11110
1.48
8.82
8.99
2.20
9.99
1.59
1.48
1.25
1.68
0.00
1.33
1.85
2.90
1.37
1:74
1.32
1:24
0.95
1:48
1.48
2.0
2.85
2.26
2.66
1:65
1:83
0.91
1.5
1.83
2:16
2.3.7
2:36
0.50
24
0.00
2.200
3.53
3.55
3.08
3.45
2.00
3.00
3.44
3.37
3:28
2.29
2.40
2.5
9:99
8.69
0.00
2:08
2:29
2.38
2.18
1.88
2:49
1.24
8.88
9:44
7.
8.39
1.78
1.98
1.0
1.70
2.26
1.96
2.25
0.00
2:36
1.56
2.31
3.97
2.43
3.66
3.79
2.53
1.27
2.44
2.39
2.00
1.02
1.38
1.04
2.00
2.62
2.30
1.39
1.03
2:38
1:3
0.5
2:37
2.35
1.42
2.05
0.00
1.93
2.31
2:35
1.58
2.22
1.63
2.74
2.05
1.08
1.28
2:23
2.09
136
1.89
0.94
2.74
0.88
1.77
1.77
15.15
16
1-111
18
0.0
17
Opponent Policy
Figure 11: Pair-wise KL divergence of all policies within MEP population in PP
For any given policies n; and nj we estimate the KI. divergence between them by:
n, (a(s))
DKL(Ti]TT) EswP(s)
Σ (((als) s) log LaA
(34)
TT (a
Here, P(s) denotes the state distribution. Ideally, P(s) should cover the entire state space S. However, practical situations, covering the entire state space in even slightly large environments can be intractable.
To maximize the coverage of the state space by P(s), employ the following sampling method: Within the MEP population, there total of 20 opponent policies. For each opponent policy æ -.k we sample 1000 episodes. In these 1000 episodes, the opponents' policy are fixed to x 1I, while the self-agent traverses through all the opponent policies, resulting in the self-agent using per opponent policy for 50 episodes.
In Figs. 11
denotes seen opponent policies, while π -1. k 11, 12..... 20 denotes unseen opponent policies. The lighter the color in the heatmap, the higher the KL divergence value, indicating lower similarity between the two policies.
to 13 T
.., 10 1,2
In the and OC environments, there is relatively large dissimilarity between all pairs of opponent policies. Assuming dissimilarity threshold of 1.0 (i.e., two policies are dissimilar if their KL diver- gence is greater than 1.0), the dissimilarity rates for and are 93.75% and 91.5%, respectively. In contrast, the dissimilarity rate for LBF is 66. 25%. indicating relatively smaller differences between opponent policies. This could be attributed to fact that the state space of LBF much smaller than PP and OC, making it difficult for well-trained opponent policies to exhibit significant behavioral diversity. Nonetheless, overall, we can consider the MEP opponent policy population we generated to be adequately diverse.
31

---

- Foraging
1,20
1.06
1:19
3.5
0.82
0.97
0.00
0.65
0.84
3.15
4.56
1.36
2.80
2.52
0.31
3.12
1.23
0.82
1.50
2.72
2:97
2.84
0.00
1.39
2.12
0.66
2.44
1.54
3.32
0.72
1.98
1.32
1.09
2.56
2.68
0.67
0.64
0.67
1.22
2.85
0.00
3.99
1.92
0.86
1.09
3.0
1.15
1.06
1111
Opponent Policy
2.5
0:78
0.61
1.76
5.95
0.93
0.52
52
3.20
2:25
2.33
2.74
0.61
0.00
0.47
1.62
2.22
2.00
2.15
2.36
0.65
1.59
0.45
1.40
1.60
0.92
0.59
AM
0.78
1.59
1.45
2.31
0.90
0.07
0.55
0.98
3.80
0.86
3.52
1.40
0.47
1:45
2.0
1.10
2.52
1.00
9.05
0.80
99
1.26
888
2.50
6.00
0.66
0.84
0.83
0.88
0.87
9:93
0.72
1:93
1.54
00
2:23
2.33
2.00
1.03
1.5
0.86
1:23
2:30
9.
0.04
0.00
000
7.79
1.49
7:73
3.30
2.09
3.07
0.00
0.46
2.62
1.39
8.08
0.62
000
8.00
41
2.59
0.94
0.59
0.88
0.55
0.80
3:44
0.57
9.91
1.24
0.70
1.54
1.0
0.45
1.80
1.89
3.09
0.00
3.33
0.87
2.78
1.50
1.63
0.49
0.48
0.43
1.25
2.65
0:85
2.55
1.71
0.57
1.42
2.36
2.23
2.60
1.01
0.56
1.54
1.3
3.30
2:37
1.56
0.00
2.03
1.41
0.96
0.43
0.89
0.89
1.51
1.86
1.26
0.41
0.49
0.44
1:2
1.58
0.37
254
1.39
0.54
1.30
15.1
16
0.0
Opponent Policy
Figure 12: Pair-wise KL divergence of all policies within MEP population in LBF
OverCooked
2.85
0.97
1.94
2:25
3.78
3.30
2.47
2.15
3.37
2.53
3.30
3.18
3.98
2:23
3:22
2:50
2:34
2:27
2.37
0.00
1.69
1.78
0.96
1:18
2:33
3.32
1.37
3:48
3.76
2.20
1.00
2.66
2.53
2.05
1.62
3.45
3.99
1.69
0.00
2:24
5
18
2.64
3.90
1.15
1.68
2:30
3.34
2:48
3.56
.009
1.38
2.52
3.65
1.89
1.22
1.05
3.99
3.45
0.00
5.76
3.25
17
0.99
2:88
3.86
2.63
2.69
3.05
1.99
3.62
2.35
2.73
2.35
2.36
2.55
4.08
3.87
3.79
1.17
2.85
3.99
3.42
2.2.
2.25
3.09
1.58
2.28
.99
2.06
0.00
2.00
2:22
3.30
3.76
3:23
5.09
5.10
1.88
0.80
3.80
2.84
3:74
2.39
1:59
2.89
2.77
0.00
1:97
0.67
3:3.3
1.52
2.28
3.21
Opponent Policy
1.94
2.66
2.50
3.33
2.77
3.20
2.66
3.37
0.79
3.34
2.97
0.00
2.69
1.89
2.36
3.28
1.70
2.55
3:59
-4.12
2.05
3.74
3.02
2.66
3
11
1.11
2.32
n7 -1.10
2.22
2.96
2.1.
3.95
2.60
2:91
2.21
1.76
2.
1.89
2.35
3.33
77
3.26
1.47
1.48
1.29
2.40
3.43
3.08
3.38
0.00
1.95
2.26
0.87
2.70
1.35
2.48
3.32
0.97
1.44
2.42
2.45
3.42
1.62
1.62
2.50
2.94
2.53
2.99
0.00
3.74
2.366
3.87
2.82
3.22
2.15
2.70
2.53
2.96
2.07
2.81
3.30
-2
1.7
3.40
2.2.3
2.25
3.80
8.68
1.778
2.29
2:97
2.28
9:29
3.00
2.06
1.22
2.72
1.10
0.00
2.50
2.22
1.80
3.35
3.66
1.76
1.64
2.67
2.37
0.93
1.71
0.88
0.89
1.62
2.63
3.47
1.97
1.67
0.00
1.144
2.55
3.38
1.08
3.43
1.28
2.06
1.07
2.52
2.84
1.02
1.34
7.20
2.36
1.39
2.33
3.09
2.55
0.00
2.45
1.26
3.35
9.25
9.10
5.16
1.30
2.65
3.64
3.33
1.60
3.67
1.26
2.88
1.54
2.64
2.96
2.73
1.82
3.50
2.39
1.1
1.14
1.15
1.11
Opponent Policy
Figure 13: wiir-wise KL divergence of all policies within MEP population OC
32

---

H Hyperparameters
H.1 Hyperparameters for Opponent Policies Training
As mentioned in Sec. 5.1 we employ diversity-driven Population-Based Training algorithm MEP [04 to train a policy population, which further used create the opponent pol- icy sets I[ and Il For specific implementation of MEP, we adopt the open-source code of mwx.iummmmmmmaammmmmmmmmmmm mawximm_enttroy-aeeeeee wwwpppyooopaaaoonnnooom based_ training. which is available at https://github com/ rui wwwzuuuiiiaiii t/ maximaimmm www.eent-oooolalytteeaeeeeeapaae based_ seassd__ralie training For the three environ- ments, we use the same hyperparameters as this open-source code train the MEP populations.
H.2 Hyperparameters for In-Context-Learning- Pretraining
Hyperparameter Name
LBF
dimensionality states
(5, 4, 20) (image-like)
dimensionality of actions
borizon for each episode (T)
100
50
400
agent index of the sell--gent agent indexes of the opponents
0
0
0, 1,2
1 50000
lotal number of episodes training BRs
50000
50000
discount factor training BRs
1.0
betch for training updating epochs
1096
4096
4096
training step training BRs
learning rale of the actor for training BRs
5 10-4
10
x 10
learning rate of the critic training BRs
10
x
x 10
number
layers training
additional convolutional layers
number of nodes of hidden layers training BRs
32
clipping factor of PPO 73 for training BRs
0.2
0.2
0.2
muximum norm the gradients for training BRs (clip exceeded)
5.0
5.0
5.0
number of opponent policies 1h (J)
10
10
10
soquence longth episode-wis: in-context data D (H)
number of trajectories randomly sampled to consiruct D (C)
3
3
maximum sequence longth for OMIS's GPT2 backbone (B) (see App. For detailed
descriptions)
reward scaling factor for pretraining Fg feg. Va, (all the rewards are multiplied by roduce variance training)
100
100
total oumber of training steps for pretraining sté, Mar Vi
1000
4000
4000
discount factor for pretraining pig. V. (t)
1.0
1.0
1.0
batch size for pretraining
V
61
641
64
number of updating epochs each training step pretraining Re,
10
10
weighting coefficient for pretrnnning iee
1.0
1.0
1.0
weighting
pretraining H4p
weighting coefficient peetraining V__
warm up epochs for pretraining To Ma. V., (the learning rate is multiplied by
um specheel to allow to increase lincarly during the initial warm-up opochs of training) wame-g specte
10000
10000
10000
learning rate Adam W optimizer pretraining Fe- He. Va.
x
6
weight decay coefficient for AdamW optimize for pretraining me. Hé. Va.
1 x 10
x 10
x 10
maximum norm of the gradients for pretraining number of nodes of hidden layers for OMIS's GPT2 backbone (sec App. for detailed
V..
0.5
0.5
0.5
descriptions)
dropout factor for OMIS's GPT2 backbone
0.1
0.1
0.1
number of sell-attention blocks for OMIS's ' GPT2 backbone number of attention head for OMIS's GPT2 backbone random seeds
3
3
0, 1,2,,,44
0,1,2,3,4
0,1,2,3,4
33

---

H.3 Hyperparameters for Decision-Tiin Time Search with In-Context Components
Hyperparameter Name
LBF
total number episodes testing
1200
1200
1200
number of rollouts for self-agent's each legal action for DTS (M)
length each rollout DTS (I.)
discount factor for DTS (hhcdd)
0.7
0.7
0.7
threshold of mixing technique for DTS (r)
10
0
0
sequence length episode-wise in-context data D (H)
15
13
15
number of the most recent trajectories used to construct D) (C)
maximum sequence length for OMIS's s GPT2 backbene (B)
20
20
20
number of nodes of hidren layens for OMIS's GPT2 hackhone dropout facter for OMIS's GPT2 buckbone number of self allention blocks OMIS's GPT2 backhome number of attention head OMIS's GPT2 backbonc seeds
32
32
32
0.1
0.1
0, 0,1,2,3,4
0, 3,
0, 0,1,2,3,
I Quantitative Analysis of Attention Weights Learned by OMIS
To rigorously evaluate whether OMIS can effectively characterize opponent policies, we conduct quantitative analysis of the attention weights learned by OMIS by calculating the pair-wise Pearson Correlation Coefficients (PCC) between the attention vectors. The relevant results are shown in Fig. [1] The first column the heatmaps of the pair-wise PCC statistics of all attention vectors, and the second column shows the corresponding p-value plots for the statistics in the first column, with pairs marked in white for < 0.05 and black otherwise.
The observations reveal that the attention vectors of the same opponent policy have strong pair-wise correlations (i.e., statistics close to and p 0.05) across multiple timesteps. contrast, the attention vectors of different opponent policies generally have no strong pair-wise correlations with each other. Although there is some pair-wise correlation between the attention vectors of different opponent policies, each opponent policy generally has the strongest pair-wise correlation with its own other attention vectors. These observations indicate that the attention weights learned by OMIS can be distinguished by different opponent policies and maintain consistency for the same opponent policy to some extent. Therefore, this analysis further demonstrates OMIS's ability to represent opponent policies based on in-context data.
34

---

Predator Prey Statistic
Predator Pray Pvalue Lsc DS white)
0111
Opponent Policy Tisee
Opporten Policy in
A-1 p 675 m-
Opponent Palicy in m
Oppanent Palicy in maaa
Foraging
Foraging
Opponent Policy [laah
opppnentt Policy In
Opponent Policy in Ma
Opponent Policy in
OverCooked Statictic
OverCooked Pvalue (sc 0.05 white)
2-1.1
Opponent Policy piras
Coooono Pollcy
74
g-14
p-1
6-
Opponent Policy p/mm
Opporent Policy nm
Figure 14: Pair-wise PCC statisties and p-values between the attention weights learned by OMIS. The attention vectors on D are calculated over the final 20 timesteps against each opponent policy.
35

---

In-depth Discussion
In Sec.Blof the main text, we analyzed this study's limitations and future work from four perspectives. Herein, we would like to point out that there are currently many potential feasible solutions for each aspect. The OMIS proposed in this paper can be viewed as a complete framework that tackled the main problems in existing OM works during the pretraining and testing stages. This framework can be modified for other settings (such as the opponents are learning, imperfect information, etc.). Moreover, this framework also represents minimalist approach, focusing generic opponent modeling settings, while more complex settings can considered as new research problems to explore in the future.
(1) For the settings where opponents are learning, according to the observations in Laskin al. 43]. ICL has the ability to model sequence taken during the learning process. Therefore, we can potentially model continuously updating opponents by using the complete (s, a-1) sequences during opponent learning in-context data. Strictly speaking, regardless of the type of opponent, as long as we have their in-context data and their best response policy, we can use the OMIS framework to learn to respond to that opponent. Another possible solution is to leverage the idea of Opponent Modeling with Shaping Opponents' Learning [2 the opponent's gradient updates during testing to shape their learning process.
92 1OS 25 (see App. A explicitly modeling 53
(2) For imperfect information settings, there is vast of research in the field of imperfect information 35 38 (50,, with many mature methods that can be adapted to work within the OMIS framework. Yet, this adaptation non-trivial, as such DTS methods often require explicit or learned beliefs about the true state, introducing significant additional computational complexity. Interestingly. recently proposed
online search [58, 12
13 83
Framework [77] suggests that we can effectively search in imperfect information settings without relying on beliefs.
(3) For more complex decision-time searches, numerous advanced DTS methods 75 76 12 2. [46 38 36
13
660 can seamlessly integrate with our framework. This because the OMIS pretraining stage learns all the key components needed for DTS: actor, critic, and opponent imitator. The actor provides good prior decision for the self-agent during the DTS, the critic estimates the value given terminal state during the DTS, and the opponent imitator estimates the most probable action for the opponent during the DTS.
5
100
18
K
Compute Resources
CPU: AMD EPYC 7742 64-Core Processor
GPU: NVIDIA GeForce RTX 3090 24G x8
MEM: 500G
Maximum total computing time: pretraining + testing ~ 40h
L
Broader Impacts
This paper presents work that aims to advance the field of Machine Learning. There are many potential socictal consequences of our work, none of which feel must be specifically highlighted here.
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. Please see Abstract and Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the limitations of the work performed by the authors. Please see Sec. 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't seknowleddee in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
37

---

Answer: [Yes]
Justification: For each theoretical result, the paper provides the full set of assumptions and a complete (and correct) proof. Please see Sec. 4.3 and App. D
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper fully discloses all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or con- clusions of the paper (regardless of whether the code and data are provided or not). Please see Sec. [.T and App. H
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
38

---

5. Open access to data and code
Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The paper provides open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results. Please see our project website in Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-sourge benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and basclines. If only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
The paper specifies all the training and test details (c.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary understand the results. Please see Sec. 5.1 and App.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments. Please see Sec. 5
Guidelines:
The answer NA means that the paper does not include experiments.
39

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The paper provides sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments. Please see App. K
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't i make it into the paper).
paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https::/neuuri: cttp:pp.cccpppppppe http://s....pb..b..ttccaaaccaaas
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics htt.ghhhtppp....orrrr..cc/c///aa
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper diseuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
40

---

Justification:
The paper discusses both potential positive societal impacts and negative societal impacts of the work performed. Please see App. L
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research does not release data models, so the paper has such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected. Please see our project website in Abstract.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
41

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thooee com comddatasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
42

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
43


Reviews:
[{"review_id": "a8ZzoAZzo3", "reviewer": "Reviewer_xSNm", "summary": "The paper addresses the challenges of opponent modeling in multi-agent environments, particularly the difficulties in generalizing to unknown opponent policies. The authors propose a Opponent Modeling with In-context Search (OMIS), which combines in-context learning-based pretraining and decision-time search. OMIS utilizes a Transformer model with three components: an actor, an opponent imitator, and a critic, to enhance decision-making. The method proves to converge in opponent policy recognition and generalize well without search, while offering performance stability with search. Empirical results show that OMIS outperforms existing approaches in competitive, cooperative, and mixed environments.", "questions": "- Line 39: It's disconnected from the previous paragraph. How does pre-training with a transformer model deal with any of the issue abovementioned?\n- Line 41: How is in-context learning related to pre-training issues? \n- Line 42: How are these three components related to pre-training issues? The current version only describes how the proposed method is implemented but didn't explain \"why\" it should be implemented in this way. A good introduction should sell your main insight instead of describing your implementation.\n- Line 48: Again, what issues are you talking about there? What are limited generalization abilities? What are good properties? What are performance instability issues? All these need explanation. I guess that you refer instability issue to the sentence in Line 38, \"TFA always perform unstably when facing unknown opponents...\". If so, instability performance is too ambiguous.\n- Line 147: The phrase \"self-agent is unable to ascertain the true policy $\\bar{\\pi}^{-1}$ employed by $\\pi^{-1}$\" reads weird. What do you mean by true policy is unclear. Isn't $\\bar{\\pi}^{-1}$ just a policy chosen by a non-stationary player?\n- Line 158: What are D^epi and D^step? \n- Line 159: $D^epi$ samples segments? It reads weird. \n- Section 5.2: How many random seeds do you train? What does the error bar represent?\n- Line 311, Section 5.2: From the heatmap shown in Figure 6, I don't think there are clear patterns between each opponent policy. Since this claim is only backed up by qualitative observation, I don't think you can say the results imply OMIS can represent different opponent policies. One suggestion is to run a statistical test (e.g., randomization test) to test the significance of the hypothesis on each attention vector.\n- Line 319: I initially had a question, \"What do you mean by using opponent actions and true RTGs as labels?\" but I figured out what $\\mu$ and $V$ mean by looking back to the method section. I suggest the author write the role of each math symbol before you refer to it if the definition of this math symbol is far away from the current text.", "limitations": "Yes, it's addresed.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Using in-context learning for opponent modeling is novel and interesting, though the proposed method is somewhat limited by the need of dynamics and the presentation is unclear.", "weaknesses": "- The proposed method requires transition dynamics while the baselines compared with it do not require, which makes experiments unfair. It makes more sense to take model-based approaches (e.g., MCTS) as a baseline. This may be a reference: https://arxiv.org/pdf/2305.13206.\n- The presentation is unclear. Notations are unnecessarily complex and the reasoning in some paragraphs are unclear. See my questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer **xSNm**\n\nThank you very much for your valuable feedback. In response to your comments, we would like to make the following clarifications and feedback. We hope our explanations and analyses can eliminate concerns and make you find our work stronger.\n\n> the proposed method is somewhat limited by the need of dynamics\n> \n\nWe include results and corresponding analyses for the version of OMIS where the dynamics are learned. Please refer to the **Global Response**.\n\n> The proposed method requires transition dynamics while the baselines compared with it do not require, which makes experiments unfair …\n> \n\nWe compare our work with the paper [1] you mentioned. Please refer to the **Global Response**.\n\n[1] https://arxiv.org/pdf/2305.13206\n\n> The presentation is unclear …\n> \n\nIn fact, the presentation of our paper received consistent approval from reviewers G1Ke, eKwk, and YkYd. However, as you mentioned, there is still significant room for improvement in terms of clarity. We apologize for any inconvenience this may have caused and appreciate your suggested advice. We will reorganize the logic and simplify the mathematical symbols in the revision. We will address your concerns one by one in our subsequent responses.\n\n> (1) Line 39: … How does pre-training with a transformer model deal with any of the issue abovementioned?\n(2) Line 41: How is in-context learning related to pre-training issues?\n(3) Line 42: How are these three components related to pre-training issues? …\n> \n\nYour suggestion is very pertinent; the transition in Lines 39-42 is indeed not smooth. Actually, we explained the motivation for our approach at the beginning of the methodology section (Lines 106-120). In the revision, we will move the overview of the motivation to Lines 39-42 to emphasize the rationale behind our work rather than its implementation.\n\nThe in-context-learning (ICL)-based pretraining we used theoretically provides good generalization guarantees: the pretrained model can accurately recognize seen opponents and recognize unseen opponents as the most familiar seen ones to some extent. This theoretical property endows our approach with the potential to address pretraining issues effectively.\n\nFor the three components: the actor is the core of the ICL-based pretraining, ensuring good generalization to handle pretraining issues. The critic and opponent imitator are indispensable modules for OMIS search during testing.\n\n> Line 48: Again, what issues are you talking about there? What are limited generalization abilities? What are good properties? What are performance instability issues? …\n> \n\nWe apologize for the misunderstanding caused by our unclear expression. In the revision, we will clarify that \"limited generalization abilities\" refers to the lack of theoretical guarantees for generalization during the pretraining stage in existing approaches. \"Good properties\" refer to the characteristics of OMIS described in Theorem 4.2—accurately recognizing seen opponents and recognizing unseen opponents as the most similar to the seen ones. \"Performance instability issues\" refer to the problem mentioned in Line 38, as you guessed, which is also reflected in the results of our experiments, such as Figure 3.\n\n> Line 147: … What do you mean by true policy is unclear. Isn't $\\bar{\\pi}^{-1}$ just a policy chosen by a non-stationary player?\n> \n\nHere, $\\pi^{-1}$ can be understood as the non-stationary opponent agent, while $\\bar{\\pi}^{-1}$ is a mnemonic representing the actual policy used by $\\pi^{-1}$ at a given time, which is unknown to the self-agent. We will emphasize the distinction between the two symbols in the revision.\n\n> Line 158: What are $D^{\\text{epi}}$ and $D^{\\text{step}}_t$?\n> \n\nBoth $D^{\\text{epi}}$ and $D^{\\text{step}} _ t$ are generated from the interactions between OMIS and the non-stationary opponent $\\pi^{-1}$ during testing. $D^{\\text{epi}} = \\{(\\tilde{s} _ h, \\tilde{a} _ h^{-1})\\} _ {h=1}^{H}$ is episode-wise in-context data, constructed similarly to the process in Appendix C, except that $(s, a^{-1})$ tuples are sampled from the most recent $C$ trajectories in which $\\pi^{-1}$ participated. $D^{\\text{step}} _ t = (s _ 0, a _ 0^{-1}, \\dots, s _ {t-1}, a _ {t-1}^{-1})$ is step-wise in-context data. We will add these explanations to the revision.\n\n> Line 159: $D^{\\text{epi}}$ samples segments? …\n> \n\nThe expression here is indeed inaccurate. We mean that $D^{\\text{epi}}$ is constructed by sampling several consecutive segments from the opponent's trajectories.\n\n> Section 5.2: How many random seeds do you train? What does the error bar represent?\n> \n\nWe trained with 5 random seeds, as stated in the \"Specific settings\" paragraph of Section 5.1. The error bars represent the standard deviation of the test results across these above 5 random seeds.\n\n> Line 311, Section 5.2: … Since this claim is only backed up by qualitative observation, I don't think you can say the results imply OMIS can …\n> \n\nWe incorporate your suggestion and add a quantitative analysis of attention weights learned by OMIS. Please refer to the **Global Response**.\n\n> Line 319: I initially had a question, \"What do you mean by using opponent actions and true RTGs as labels?\" …\n> \n\nYour suggestion is beneficial; we will emphasize the meaning of each symbol before it appears in the revision.\n\nAll your comments have greatly contributed to the improvement of our paper. If you have any new comments, please feel free to provide them, and we will promptly address them. If you find that we have addressed your concerns, we hope you will reconsider your rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XXkUNAcJCm", "reviewer": "Reviewer_G1Ke", "summary": "This paper addresses the problem of opponent modeling and leverages in-context learning to tackle the challenges posed by opponents using non-stationary and unknown policies. Specifically, the proposed method, OMIS, employs PPO to train a best-response policy for each opponent policy in the training set. OMIS then uses these best-response policies to generate in-context data. Based on this data, OMIS trains three in-context components using transformers: the self-agent actor, the opponent imitator, and the critic. With these components, OMIS can perform multiple rollouts for each action and select the action with the highest expected return to interact with opponents. By relying on in-context data, OMIS effectively mitigates the adverse effects of non-stationary and unknown opponent policies.", "questions": "1. OMIS requires virtual transition dynamics, which are exact replicas of the true dynamics, to perform rollouts. However, if the virtual transition dynamics need to be learned from data, as in model-based RL approaches, how will OMIS perform under these conditions?\n\n2. Lemma 4.1 assumes that \"the sampling of $s$ from $\\tau_{pre}^{-1}$ is independent of the opponent's policy.\" However, $\\tau_{pre}^{-1}$ is the probability distribution over all trajectories involving the opponent's policy during pre-training. Thus, the opponent's policy should influence the distribution $\\tau_{pre}^{-1}$, and sampling from $\\tau_{pre}^{-1}$ should be correlated with the opponent's policy. So is the assumption reasonable?\n\n3. Equation 10 considers the expected return of the action selected by the search as the confidence of the search policy. How does this design address the issue of value function overestimation?", "limitations": "The authors have discussed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is meticulously designed and clearly articulated, ensuring the reliability and validity of the results.\n\n2. The paper provides a comprehensive review of existing literature, showcasing a deep understanding of the relevant background.\n\n3. The paper is well-written, logically organized, and structured, making it straightforward for readers to comprehend and follow the arguments and conclusions.", "weaknesses": "1. This work implicitly assumes the presence of a perfect transition model. OMIS relies on this transition model to perform multiple rollouts and identify the best action. However, in many application domains, such transition models do not exist, making OMIS inapplicable.\n\n2. The mixing technique in Equation 10 is quite simple and may not generalize across different domains. Additionally, the hyperparameter requires fine-tuning for different environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer **G1Ke**\n\nThank you very much for your recognition of our paper's writing, methodologies, literature reviews, and the valuable feedback you provided. In response to your comments, we would like to make the following clarifications and feedback. We hope our explanations and analyses can eliminate concerns and make you find our work stronger.\n\n> (1) This work implicitly assumes the presence of a perfect transition model. OMIS relies on this transition model …, making OMIS inapplicable. \n(2) OMIS requires virtual transition dynamics ... However, if the virtual transition dynamics need to be learned from data, …, how will OMIS perform under these conditions?\n> \n\nWe include results and corresponding analyses for the version of OMIS where the dynamics are learned. Please refer to the **Global Response**.\n\n> The mixing technique in Equation 10 is quite simple and …, the hyperparameter requires fine-tuning for different environments.\n> \n\nThank you for pointing that out. In fact, more complex hybrid techniques combining both search and original policies have been proposed by others as well, such as [1]. However, their technique also introduces hyperparameters, and our experimental verification found that the approach proposed by [1] yielded comparatively poorer results. In contrast, our proposed mixing technique is a simple yet effective approach. Additionally, we empirically found that it is quite straightforward to find suitable values for the hyperparameter $\\epsilon$. The value of $\\epsilon$ only needs to follow one principle: *it should be slightly smaller than the absolute value of the sparse rewards in the environment.* We argue that our technique has an advantage building upon its simplicity and effectiveness.\n\n[1] Modeling strong and human-like gameplay with KL-regularized search. ICML, 2022.\n\n> Lemma 4.1 assumes that … sampling from $\\mathcal{T}_{\\text{pre}}^{-1}$ should be correlated with the opponent's policy. So is the assumption reasonable?\n> \n\nThank you for carefully reading our paper and raising this interesting question. We also hope you understand that *such a gap between theoretical analysis and practical algorithms is quite common in deep reinforcement learning.*\n\nYes, opponent policies do influence the distribution $\\mathcal{T} _ {\\text{pre}}^{-1}$. Here, we assume that sampling states $s$ from the distribution $\\mathcal{T} _ {\\text{pre}}^{-1}$ is independent of opponent policies. This is not a strong assumption. During the pretraining stage, we generate many trajectories w.r.t. each opponent policy. Therefore, given any opponent policy $\\pi^{-1}$ in $\\Pi^{\\text{train}}$, sampling $s$ from $\\mathcal{T} _ {\\text{pre}}^{-1}(·;\\pi^{-1})$ can be approximately considered as sampling over the entire state space $\\mathcal{S}$, thus making it independent of the opponent policy.\n\nIn our practical implementation, we made every effort to approximate this assumption. For example, we generated 1000 trajectories w.r.t. each opponent policy, utilized MEP to ensure sufficient diversity among opponent policies, etc. Despite these approximations, we found that both OMIS w/o S and OMIS achieved impressive results empirically. This suggests that they are not heavily reliant on this assumption to a certain extent.\n\n> Equation 10 considers the expected return ... How does this design address the issue of value function overestimation?\n> \n\nThis is an important question, and we greatly appreciate you bringing it up. \n\nGenerally, when optimizing the Bellman optimality equation in RL, using the max operation for temporal difference bootstrapping can lead to maximization bias, which further causes overestimation issues. However, *our search does not use the max operation for bootstrapping*, so there should be no overestimation problem.\n\nEven if overestimation occurs, we also utilized a search-specific discount factor $\\gamma _ {\\text{search}}$ (see Eq. 8) to balance the negative impact of overestimation in the value function. When $\\gamma _ {\\text{search}}$ is small, we rely more on relatively reliable rewards to estimate the action value $\\hat{Q}$ and less on the value function. Our experiments found that this approach effectively addresses the overestimation issue in the value function, resulting in stable performance improvements from the search process.\n\nAll your comments have greatly contributed to the improvement of our paper. If you have any new comments, please feel free to provide them, and we will promptly address them. If you find that we have addressed your concerns, we hope you will reconsider your rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ARt15ggHv", "reviewer": "Reviewer_eKwk", "summary": "This paper proposes a novel approach to opponent modeling called OMIS, which combines ICL and decision-time search to improve performance and stability in three distinct game settings over baselines. It also shows ablations that validate the need for specific components such as mixing technique, search, episode-wise in-context data for episodes, and step-wise in-context data.", "questions": "- Why do you specifically select 10 policies from the MEP population to form the training policies? \n- Why do the test policies need to contain seen policies? When testing, why not have all unseen policies?", "limitations": "Yes.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well written paper with great attention to detail\n- Novel combination of known components (ICL + In-context search) \n- Strong empirical results and ablations \n- Extensive comparison to known baselines \n- Extensive Appendix with proofs, codebase, and website\n- Question and Answer format helps to pre-empt some questions.", "weaknesses": "- The theoretical explanations in Section 3 are dense. It’s unclear if this section provides additional benefit within the Methodology section. You may consider moving Section 4.3 to the beginning of the Methodology section, so that the transition between the methods and experiment are clearer. Alternatively, you could summarize the key steps in a summary paragraph at the beginning of the Experimental setup to explain why you have chosen these Environments and Baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer **eKwk**\n\nThank you very much for your recognition of our paper's methodologies, theories, empirical results, paper writing, and the valuable feedback you provided. In response to your comments, we would like to make the following clarifications and feedback. We hope our explanations and analyses can eliminate concerns and make you find our work stronger.\n\n> The theoretical explanations in Section 3 are dense. It’s unclear if this section provides additional benefit within the Methodology section. You may consider moving Section 4.3 to the beginning of the Methodology section …\n> \n\nThank you very much for your valuable suggestions. The connection between our methodology and experiments is indeed not smooth. In the revision, we will move Section 4.3 to the beginning of the Methodology section to make the transition between the methods and experiments clearer.\n\n> Why do you specifically select 10 policies from the MEP population to form the training policies?\n> \n\nSelecting 10 policies from the MEP population is just one of the possible reasonable design choices. We follow this choice for the following reasons:\n\n(1) MEP is an efficient approach for generating a diverse and high-strength population of policies, making it a reasonable method for opponent policy generation.\n\n(2) For all opponent modeling approaches, we use the same 10 policies from the MEP population as training opponent policies, facilitating fair comparison with other baselines.\n\n(3) The specific selection of which 10 policies from the MEP population does not matter, as all policies have appropriate differences between each other, as can be observed from the quantitative analysis of policy diversity in Appendix G.\n\n(4) Considering computational resource constraints, we select only 10 policies as training opponent policies.\n\n> Why do the test policies need to contain seen policies? When testing, why not have all unseen policies?\n> \n\nAlthough testing against seen opponents may seem trivial, existing opponent modeling approaches still struggle to handle even seen opponents, as can be observed in Figures 3 and 4. This is because their methodological designs often rely heavily on intuition but lack rigorous theoretical analysis. In contrast, OMIS provides good theoretical properties in terms of generalization (see Theorems 4.2 and 4.3), ensuring accurate recognition of seen opponent policies during pretraining and providing the most appropriate responses. Our experimental results (Figures 3 and 4) further validate the effectiveness of our approach, as it generally outperforms other baselines against seen opponents.\n\nAll your questions and feedback have greatly contributed to improving our manuscript. With the valuable input from you and all other reviewers, the quality of our work can be significantly enhanced. We welcome further comments from you and will seriously consider your suggestions for revisions. If you feel that we have addressed your concerns, we hope you will reconsider your rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NlcG1FgMAI", "reviewer": "Reviewer_YkYd", "summary": "This paper introduces an approach to opponent modeling in multi-agent environments, aimming to address the challenges of generalization and performance instability when trained agents interacting with unknown opponents during the test time. The proposed method, Opponent Modeling with In-context Search (OMIS), leverages in-context learning-based pretraining to train a Transformer model, which includes three components: an actor learning best responses to opponent policies, an opponent imitator mimicking opponent actions, and a critic estimating state values. OMIS uses these pretrained components for decision-time search to refine the actor’s policy during testing. The paper theoretically proves that OMIS converges in opponent policy recognition and has good generalization properties under certain conditions, while the decision-time search improves performance stability without gradient updates. Empirical results in competitive, cooperative, and mixed environments show that OMIS effectively and stably adapts to opponents with unknown non-stationary policies.", "questions": "1. Could you clarify if OMIS needs to know the exact time points when opponents switch policies during the test stage?\n\n2. Is it possible to provide OMIS-dyna under all ratio settings?\n\n3. For OMIS w/o S, could you analyze the reasons why it can still outperform other baselines in LBF and OC but falls behind in competitive PP when the ratio of unseen opponents is high?\n\n4. Could you analyze the reasons why OMIS can outperform other baselines while the in-context components of OMIS do not estimate very accurately in PP and LBF?\n\n5. During the search stage, these levels of estimation errors can quickly accumulate into large compounding errors. Therefore, I wonder how the results of deep search can benefit from the pre-training of the in-context components of OMIS, especially the opponent imitator.", "limitations": "Please refer to the Questions part.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. Focus on Generalization to Unseen Opponents:\n   The paper focuses on a critical challenge in multi-agent reinforcement learning: the ability to generalize to unseen opponent policies during testing. The proposed OMIS approach leverages in-context learning and decision-time search to enhance the agent's adaptability and robustness against such opponents. \n\n2. Clear and Coherent Writing:\n   The paper is well written with logical structure and good prsentation apporaches, which makes it easy to follow.\n\n3. Comprehensive Literature Review:\n   The authors have conducted a comprehensive literature review, covering various approaches to opponent modeling, including representation learning, Bayesian learning, meta-learning, and decision-time search. \n\n4. Thorough Experimental Evaluation:\n   The empirical evaluation of OMIS is thorough and robust, encompassing a variety of environments (competitive, cooperative, and mixed) and comparing against several baseline methods.", "weaknesses": "1. Lack of novelty: I assume that two of the most important feature of this work is in-context learning-based pretraining and in-context search. However, I hardly find these two ideas or the combination of these two ideas applied to marl novel. \n\n2. Strong assumptions: OMIS relies on the access to the environment dynamics which is a strong assumption in real problems and it may give an unfair advatange to OMIS when compared to other baselines. In addition, I understand that the authors have conducted experiments to test OMIS's robustness to random E (which is the number of episodes before the opponents switching to another policy), but it is still unclear to me that whether OMIS requires to know the exact time points when opponent switch policies. \n\n3. Scalability: The experiments conducted in the paper are relatively simple and conceptual compared to the complexity of the OMIS, I am concerned with how OMIS can perform in more complex environments with regrad to associated cost of generating diverse training opponent policies population, collecting training trajectories, pretraining and computing resources for search.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer **YkYd**\n\nThank you very much for your recognition of our paper's problem setting, paper writing, experimental results, and the valuable feedback you provided. In response to your comments, we would like to make the following clarifications and feedback. We hope our explanations and analyses can eliminate concerns and make you find our work stronger.\n\n> **W1**. Lack of novelty: … However, I hardly find these two ideas or the combination of these two ideas applied to marl novel.\n> \n\nWe all know that *learning* and *search* are two major “magic keys” in machine learning, as well as common paradigms in MARL. We argue that the novelty of OMIS lies in using **in-context** **learning (ICL)** and **in-context** **search (ICS)** to address the challenges of OM, with \"**in-context**\" being the core aspect. In other words, we offer a new perspective that learning and search should be in-context (adaptive). We utilize ICL-based pretraining to respond to opponents based on **in-context data (ICD)** adaptively. Building upon this foundation, we further improve this adaptive capability through ICS. Our ICS is capable of inferring opponent actions, making responses, and estimating corresponding values based on the opponent's ICD. Lastly, both our proposed ICL and ICS are theoretically well-grounded, whereas many existing OM approaches lack theoretical analysis. To end with, we would like to argue that novelty is not always about creating entirely new methodologies. *Offering a systematic approach that effectively addresses a long-standing problem in a field is also a form of innovation.*\n\n> **W2**. Strong assumptions: OMIS relies on the access to the environment dynamics … it may give an unfair advatange to OMIS when compared to other baselines.\n> \n\nWe include results and corresponding analyses for the version of OMIS where the dynamics are learned. Please refer to the **Global Response**.\n\n> (1) **W2**. In addition, …\n(2) **Q1**. Could you clarify if OMIS needs to know the exact time points when opponents switch policies during the test stage?\n> \n\nOMIS does not require knowing the exact time points when opponents switch policies, as demonstrated in Figure 7, where the timing of policy switches is strictly unknowable to OMIS. Even in this situation, OMIS works well and generally outperforms other baselines (see Figures 3 and 4).\n\nIf the switch times were precisely known, it might be able to further enhance OMIS. Detecting whether opponents have switched policies is an independent research problem [1,2]. In fact, we could leverage these methods for detecting opponent switches to make $D^{\\text{epi}}$ as pure as possible, thereby achieving better result.\n\n[1] Efficiently detecting switches against non-stationary opponents. AAMAS, 2017.\n\n[2] A deep bayesian policy reuse approach against non-stationary agents. NIPS, 2018.\n\n> **W3**. Scalability: The experiments conducted in the paper are relatively simple, …\n> \n\nAs you mentioned, the experiments in our paper are generally not that complex. Despite this, *current algorithms in the OM domain still struggle to work well*, facing challenges such as having difficulty generalizing to unknown opponent policies and performing unstably (see Figures 3 and 4). In contrast, our approach performs well in these commonly used benchmarks. Furthermore, we would like to emphasize that *OverCooked (OC) is a relatively complex environment* with a high-dimensional state space where all states are image-like. Training even a workable policy on OC is actually challenging and requires more than a few computational resources.\n\n> **Q2**. Is it possible to provide OMIS-dyna under all ratio settings?\n> \n\nWe provide the results for OMIS-dyna under all ratio settings. Please refer to the **Global Response**.\n\n> **Q3**. For OMIS w/o S, … why it can still outperform other baselines in LBF and OC but falls behind in competitive PP when the ratio of unseen opponents is high?\n> \n\nThe possible reason for this result is that PP is a continuous environment with an uncountable state space, exacerbating the degree of out-of-distribution of unseen opponents' states. OMIS w/o S relies on $(s, a^{-1})$ tuples for generalization, whereas other baselines do not depend on this for generalization. This reliance introduces additional challenges for OMIS w/o S. In contrast, the state spaces of LBF and OC environments are countable, and many of the states of unseen opponents can be familiar (though the actions in those states may be novel). This makes generalization for OMIS w/o S easier.\n\n> (1) **Q4**. … why OMIS can outperform other baselines while the in-context components of OMIS do not estimate very accurately in PP and LBF?\n(2) **Q5**. … , I wonder how the results of deep search can benefit from the pre-training of the in-context components of OMIS, especially the opponent imitator.\n> \n\nWe argue that the necessity of search arises precisely because there are estimation errors in the opponent's actions and value predictions. Otherwise, we could directly solve for the best response to the current opponent's policy, and there would be no need for search.\n\nThe search domain generally does not emphasize the concept of compound errors; rather, it focuses on the trade-off between exploration and exploitation. In fact, the inaccuracy of the opponent imitator can be viewed as accurately sampling the opponent's actions most of the time while exploring other actions occasionally, meeting the trade-off need for search. Given that OMIS's critic is relatively accurate, combined with the reward signal during search, we can consider our value estimates to have relatively high confidence.\n\nAll your questions and feedback have greatly contributed to improving our manuscript. We welcome further comments from you and will seriously consider your suggestions for revisions. If you feel that we have addressed your concerns, we hope you will reconsider your rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response

We extend our heartfelt thanks to AC and all the reviewers for your diligent work in evaluating our manuscript. We are deeply grateful for the insightful feedback and recommendations from each of you. In this global rebuttal comment, we provide additional experimental results and relevant analysis requested by all reviewers. The figures and tables for the results are all in the attached PDF.

## **1 (to all reviewers) Results of OMIS using learned dynamics**

To relax OMIS's reliance on ground truth transition dynamics, we implement a Model-Based OMIS (MBOMIS) to test whether OMIS can work effectively when the environment model is unknown and learned. We use the most straightforward method to learn a transition dynamic model $\hat{\mathcal{P}}$: given a state $s$ and action $a$, predicting the next state $s'$ and reward $r$ using Mean Square Error (MSE) loss. We train $\hat{\mathcal{P}}$ using the $(s, a, r, s')$ tuples from the dataset used for pretraining OMIS w/o S.

The test results against unknown non-stationary opponents are shown in **Figure 1 of the PDF**. Due to the rebuttal time constraints, we initially provided results for the PP and LBF environments. We will supplement the OC results in future revisions. Observing **Figure 1(a) of the PDF**, we find that although MBOMIS loses some performance compared to OMIS, it still effectively improves over OMIS w/o S and generally surpasses other baselines. **Figure 1(b) of the PDF** shows a similar phenomenon, where MBOMIS, despite not reaching the level of OMIS, effectively adapts to each true policy employed by the opponents compared to other baselines.

We also provide quantitative evaluation results of the learned dynamic $\hat{\mathcal{P}}$'s estimation during testing in **Table 1 of the PDF**. Observations show that $\hat{\mathcal{P}}$ generally has a relatively small MSE value in predicting the next state and reward (without normalization). Although the reward error on PP and the next state error on LBF are relatively larger, the results in **Figure 1 of the PDF** indicate that this does not significantly negatively impact the search.

## **2 (to reviewer xSNm) Results against a new search-based baseline**

We add a search-based baseline, SP-MCTS [1], to make the experiments more comprehensive. The results are shown in **Figure 2 of the PDF**. In this experiment, the true transition dynamics are available to all approaches, and SP-MCTS uses OMIS w/o S as the blueprint policy. The results indicate that OMIS can effectively outperform SP-MCTS and more effectively improve OMIS w/o S. SP-MCTS can sometimes effectively improve OMIS w/o S (e.g., in LBF and parts of PP), while in other cases, it even makes OMIS w/o S worse (e.g., in OC and parts of PP).

We presume that this is because (1) Compared to OMIS, SP-MCTS's critic and opponent model do not adaptively adjust based on the opponent's information (in-context data). Additionally, we suspect that its performance could deteriorate further if a fixed policy, rather than OMIS w/o S, were used as the blueprint policy. (2) The trade-off between exploration and exploitation in the MCTS used by SP-MCTS heavily relies on the hyperparameter $c_{\text{puct}}$. It requires extensive hyperparameter tuning to function effectively. With default hyperparameters, it may sometimes (e.g., in OC) focus too much on exploration and insufficiently on exploitation, leading to a worse policy.

[1] https://arxiv.org/pdf/2305.13206

## **3 (to reviewer xSNm) Quantitative analysis of attention weights learned by OMIS**

To rigorously evaluate whether OMIS can effectively characterize opponent policies, we conduct a quantitative analysis of the attention weights learned by OMIS by calculating the pair-wise Pearson correlation coefficients (PCC) between the attention vectors. The relevant results are shown in **Figure 4 of the PDF**. The first row is the heatmaps of the pair-wise PCC statistics of all attention vectors, and the second row shows the corresponding p-value plots for the statistics in the first row, with pairs marked in white for $p < 0.05$ and black otherwise.

The observations reveal that the attention vectors of the same opponent policy have *strong pair-wise correlations* (statistics close to $1$ and $p < 0.05$) across multiple timesteps. In contrast, the attention vectors of different opponent policies generally have no strong pair-wise correlations with each other. Although there is some pair-wise correlation between the attention vectors of different opponent policies, each opponent policy generally has the strongest pair-wise correlation with its own other attention vectors. These observations indicate that the attention weights learned by OMIS can be distinguished by different opponent policies and maintain consistency for the same opponent policy to some extent. Therefore, this analysis further demonstrates OMIS's ability to represent opponent policies based on in-context data.

## **4 (to reviewer YkYd) Results of OMIS-dyna under all ratio settings**

We supplement the results of OMIS-dyna under all [seen:unseen] opponent ratio settings, as shown in **Figure 3 of the PDF**. Observations reveal that despite the unpredictable frequency of opponent policy switches, OMIS can generally achieve good adaptation results. Additionally, the results of OMIS-dyna are close to those of *OMIS-20* (where the opponent switches policies every 20 episodes, results shown in **Figure 2 of the PDF**). In both PP and OC, there is a slight performance decline as the proportion of unseen opponents increases, likely due to the out-of-distribution behavior of the unseen opponent policies.

Should there be any further questions or concerns after reviewing our responses, we welcome continued discussions in the forthcoming phase. Your ongoing engagement is greatly valued and appreciated. Thank you again for your time, expertise, and contributions to our paper.

Meta-review: This paper proposes opponent modeling with in-context search (OMIS), where a transformer-based policy is pretrained and enhanced during the test by planning. Empirical results show OMIS outperforms pretraining-focused and testing-focused baselines.

In the initial review, most reviewers have concerns about the assumption of truth transition dynamics. During rebuttal, the experiment of OMIS based on the learned model was conducted. The performance of OMIS is still better than the baselines. This clarifies this concern.  

There are still some concerns about the novelty and scalability after the rebuttal. For scalability, I agree that the paper will be stronger if it is additionally evaluated in more complex games than current environments. 

Currently, I think the merits outweigh the weaknesses. The authors should include the results of OMIS with the learned model in the final version as well as promised modifications.
Decision: Accept (poster)