Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Authors: Yuheng Jing
Kai Li
Bingyun Liu
Ziwen Zhang
Haobo Fu
QIANG FU
Junliang Xing
Jian Cheng
Venue: ICML 2025 poster (2025)
Keywords: Opponent Modeling
Offline
In-context Learning
Primary area: reinforcement_learning->everything_else
TL;DR: We propose TIPR, a plug-and-play framework that instantly refines policies during test time using an in-context horizon-truncated Q function, effectively improving offline opponent modeling algorithms trained on suboptimal datasets.

Abstract: Offline Opponent Modeling (OOM) aims to learn an adaptive autonomous agent policy that dynamically adapts to opponents using an offline dataset from multi-agent games. Previous work assumes that the dataset is optimal. However, this assumption is difficult to satisfy in the real world. When the dataset is suboptimal, existing approaches struggle to work. To tackle this issue, we propose a simple and general algorithmic improvement framework, Truncated Q-driven Instant Policy Refinement (TIPR), to handle the suboptimality of OOM algorithms induced by datasets. The TIPR framework is plug-and-play in nature. Compared to original OOM algorithms, it requires only two extra steps: (1) Learn a horizon-truncated in-context action-value function, namely Truncated Q, using the offline dataset. The Truncated Q estimates the expected return within a fixed, truncated horizon and is conditioned on opponent information. (2) Use the learned Truncated Q to instantly decide whether to perform policy refinement and to generate policy after refinement during testing. Theoretically, we analyze the rationale of Truncated Q from the perspective of No Maximization Bias probability. Empirically, we conduct extensive comparison and ablation experiments in four representative competitive environments. TIPR effectively improves various OOM algorithms pretrained with suboptimal datasets.

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Yuheng Jing" Kai Li 112
Bingyun Liu
Ziwen Zhang 12
Haobo Fu
Qiang Fu 3
Junliang Xing
Jian Cheng" 136
Abstract
1. Introduction
A fundamental task towards Artificial General Intelligence is the development of autonomous agents capable of mod- eling others. The line of work that focuses adversarial domains is commonly referred to as Opponent Modeling (OM), where we build a self-agent (i.e., autonomous agent) that models the behaviors, goals, or other properties of op- ponents to reduce its uncertainty about the enviromment and enhance its meaksion-making (He et al., 2016a; Albrecht & Stone, 2018; Papoudakis et al., 2021a; Nashed & Zil- berstein, 2022: Fu et al., 2022; Yu et al., 2022; Ma et al., 2024; Jing et al., 2024b; 2025). Among the research, Offline OM (OOM) is recently proposed learning paradigm (Jing et al., 2024a). OOM aims to learn an adaptive self-agent policy that can dynamically adapt to opponents based on the available opponent information using offline datasets. This new paradigm makes OM more practical and efficient, as it relaxes the reliance on interactions with the environment and opponent policies in the learning process.
Offline Opponent Modeling (0OM) aims to learn an adaptive autonomous agent policy that dynam- ically adapts to opponents using an offline dataset from multi-agent games. Previous work assumes that the dataset is optimal. However, this assump- tion is difficult to satisfy in the real world. When the dataset is suboptimal, existing approaches struggle to work. To tackle this issue, we propose a simple and general algorithmic improvement framework, Truncated Q-driven Instant Policy Refinement (TIPR), to handle the suboptimal- ity of OOM algorithms induced by datasets. The TIPR framework is plug-and and--pand nature. Com- pared to original OOM algorithms, it requires only two extra steps: (1) Learn context value function, namely Truncated Q. using the offline dataset. The Truncated Q esti- mates the expected return within a fixed, truncated horizon and is conditioned on opponent informa- tion. (2) Use the learned Truncated Q to instantly decide whether perform policy refinement and to generate policy after refinement during testing. Theoretically, we analyze the rationale of Trun- cated Q from the perspective of No Maximization Bias probability. Empirically, we conduct exten- sive comparison and ablation experiments in four representative competitive environments. TIPR ef- fectively improves various OOM algorithms pre- trained with suboptimal datasets.
in-
Existing work on OOM assumes that the pre-collected multi- agent game dataset is optimal (Jing et al., 2024a). Here, 'optimal' means that for each opponent policy in the offline dataset, all trajectories are generated by the opponent policy and its Best Response (BR) self-agent policy. However, the experiences can collect are often suboptimal, making it difficult to meet the above optimality requirement. By 'suboptimal'' we mean that within the dataset, the self-agent policy can be arbitrarily bad rather than being the BR to the opponent policy. As expected, we observe that when pre- trained on suboptimal datasets, existing OOM algorithms deteriorate dramatically. This contradicts the original in- tention of OOM, which is to improve learning efficiency through offline learning. Regarding this, we aim to explore how to make OOM effective using suboptimal datasets in this work. The concept of OOM with suboptimal data is re- flected in many real-world applications. E.g., NBA players study opponents' styles and strategies by analyzing game replays to identify their weaknesses, even though many re- plays include videos of losses against those opponents.
'Corresponding authors DC, Institute of Automation, Chi- nese Academy of Sciences 'School of Artificial Intelligence, University of Chinese Academy of Sciences *Tencent Al Lab "Tsinghua University School of Future Technology, University of Chinese Academy of Sciences c AiRiA. Correspondence to: Kai Li <kai.li@ia.e an.",.nn>..........aaa.>,,, Jian Cheng <jian.cheng @ in....cn>.
In Offline Reinforcement Learning (RL.), a mainstream methodoloy to acquire a policy potentially better than the one embedded in the dataset is to learn an additional
Proceedings of the 4rnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
In summary, we propose simple and general algorithmic improvement framework, Truncated Q-driven Instant Policy Refinement (TIPR), to enhance OOM algorithms in han- dling suboptimality induced by datasets. The TIPR frame- work is auug-and-play and requires only two extra steps compared to original OOM algorithms: (1) Truncated Q Training: Learn a foriztn-trunccted in-context action-value function, namely Truncated Q using the same dataset used for pretraining the OOM algorithm. (2) IPR: Use the learned Truncated Q to instantly decide whether to refine the original policy and generate the refined policy during testing.
Original Q
Truncated Q
Average MSE on Validation Set
Markov Soccer (MS)
Level-Based Foraging (LBF)
10a
10a
10
10¹
Physical Deception (PD)
Predator Prey (PP)
1
10a
10²
10²
100
0
2500 5000 7500 10000 2500 5000 7500 10000 Training Steps
100
Theoretically, we prove from the perspective of maximiz- ing the No Maximization Bias probability that, compared to Original Q, our Truncated Q is potentially more reasonable Q function. Empirically, we conduct extensive comparative and ablation experiments in four representative competitive environments. We construct offline datasets with varying degrees of suboptimality to pretrain a series of OOM algo- rithms and then test their performance against unknown non- stationary opponents. Our results demonstrate that under the TIPR framework, various OOM algorithms consistently achieve considerable performance improvements, regardless of the degree of suboptimality in the dataset.
Training Steps
Figure 1. The Mean Squared Error (MSE) curves on the validation sets for learning different Q functions with various suboptimal offfine datasets from different environments.
action-value function Q for Offline Conservative Learn ing (OCL) (Wu et al., 2019; Fujimoto et al., 2019; Kumar et al., 2020; Jin et al., 2021). Inspired by OCL, an intuitive way to improve the original policy learned by OOM algo- rithms from suboptimal datasets is to refine it through Q. However, in the context of OOM, learning a workable Q highly challenging. On the one hand, the involvement of opponents introduces additional action dimensions to Q. adding extra estimation complexity. On the other hand, the non-stationarity of opponents (i.e., opponents switching policies) makes Q' estimation unreliable.
2. Preliminaries
We use an n-agent Partially-Observaae Stochastic Game (8, { (A') P, [ { T, 7) to for- malize the multi-agent environment (Yang & Wang. 2020). Here, S denotes the state space. O' denotes the observation space of agent
[n]. A' denotes agent i's action space, A = II= A is the joint action space. P S x A X $ + [0, 1] is the transition probabilities. R ; S x A - R denotes the reward function of agent i. O' : S x A x O - [o, 1] de- notes the agent i's observation function. T is the full horizon for each game episode. 7 is the discount factor.
For the first challenge, we find that shortening the horizon over which Q estimates the expected return can significantly reduce the learning difficulty. Fig. shows the error curves of learning Q validation datasets. The error of learning the Original Q, which estimates the expected cumulative discounted reward (i.e., return) over the full horizon, is unacceptably large, , especially in environments with more opponents, e.g., PD and PP. In contrast, learning the Trun- cated Q, which estimates the expected return over a trun- cated horizon, results in aader-of-magnitude reduction in error. For the second challenge, we argue that learning in-context Q, conditioned on opponent information, is more beneficial for improving the reliability of the estimation under the current opponent policy than an unconditional Q.
We use the superscript 1 to denote terms related to the self-agent, i.e., the agent under our control, and the su- perscript -1 to denote terms related to the opponents, iee, all other agents. Both 1 and - represent terms in the joint space. The offline dataset is defined as T := Uk Th == {T² := (o!:",
.. aa 1,,k _...k .e. 1,6 "T=o' The set of offline opponent policies embedded in T. is denoted as 11 = { K Jk=1, where k [K] is the index of the opponent policy. T contains multiple trajectories 7* that re- sult from the interaction hetween the opponent policy n -- 11k and certain self-agent policies. Existing work assumes that for any k E [K], the actions a-. generated by the self-agent policy are optimal (Jing et al., 2024a). In this work, we make no such assumptions about the self-agent policies embedded in T. and they can be arbitrarily suboptimal.
In addition to how to learn Q, how to use Q also matters in OOM. The previously mentioned OCL emphasizes learning an improved policy on the offline trajectory distribution. However, under OOM's setting, opponents during testing can be entirely unseen, leading to severe distributional shifts. This can significantly degrade the performance of OCL on the test trajectory distribution. Considering this, we propose a novel method, Instant Policy Refinement (IPR), to im- prove the original policy of OOM algorithms during testing.
In OOM, the data used to characterize opponent policies is
2

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Ho o/. .P))))))))))))))) 03
lecess adi agest
Calculate Refinement Condition
uppprrmnt
Estimate Opponent Action o
100 (.)))))))0))))))0000)
-frbs /1
C:
maa Pj)
fnofot.a,
(66 +
Q
10 at 1 .2))00)
Causal Transformer Decuder
Sumple Ia-Conntext Data De i- (Ois 1 M from the Opponeut's Most Bocent I'Eajectottes
with Original OOM Policy
Decide to Refine or Not faclof Dr) > or
a
4j -
Cross
Allention
Yes
cBeol-s a ..)
Causal Transformer Encoder
Generate Self agent Action
4560.44..77 .D))
Stur: IPR Timestep during Tesking
Tusion
AUsion
Nusion
Reflaed Polley
Hoo, De)
Figure 2. Left: Neural network and learning process of Truncated Q. The network of Truncated Q is primarily composed of an Encoder and a Decoder. The Encoder encodes the In- Context Data D. which is used to characterize the opponent's policy, into latent variable z. The Decoder autoregressively uses Qc
This latent variable establishes a connection with the self-agent's input through and Qu at each timestep to predict the Confidence and Value of rewards be obtained over the next H steps. Right: Overall procedure of Instant Policy Refinement (IPR). At each timestep during testing, IPR first samples D from the opponent's 's recent trajectories and predicts the opponent's action using Opponent Imitator wen then decides whether the Refinement Condition (RC) is met based the prediction of Qc. If the RC is satisfied, IPR generates the refined policy #t using Qy; otherwise, IPR maintains the original OOM policy R.
typically called In- In-Context Data D. A straightforward way to construct D is to sample several (o- , a aa tuples from the trajectory of a given opponent policy. By this way, the set of D within T can be defined as DD == UeD mDx == (D= == (om² am MM > [0_ a ) ~ -k E T* The adaptive self- agent policy to be learned can be expressed as x (aa DD) which dynamically adjusts based on D to adapt the oppo- nent. Given opponent policy x-1 the action-value func- tion of self-agent can be defined as + (o', ,a", a)= E
learns a horizon in-context action- function, i... Truncated Q (see Sec. 3.1), to perform Instant Policy Refinement (IPR) during testing, thereby improving the pretrained OOM algorithm (see Sec. 3.2). Truncated Q is theoretically proven to have the advantage of maximizing the No Maximization Bias probability (see Sec. 3.3). Our TIPR is designed as a plug-and-play framework to address the suboptimality induced by T for OOM algorithms. The overview of our TIPR framework is illustrated in Fig. 2. We also provide the corresponding pseudocode in Algo. 1.
ETD y'r! The objective of OOM is to pretrain л using the dataset T so that it maximizes its expected action value when tested against a set of online opponent policies 17 i,e,, max, 00MP -I Qu, -1,,,x,(,,,))) (.,,,)) :)).
3.1. Truncated Q Training
Inspired by Offline Conservative Learning (OCL), which learns Q for conservative policy improvement to improve the policies embedded in datasets, we aim to learn a Q to im- prove OOM algorithms pretrained on suboptimal datasets. However, under the problem setting of OOM, learning a workable Q is highly challenging due to the following rea- sons: (1) Challenge 1: The multi-agent games involve oppo- nents, and an accurate Q requires modeling the opponents' actions a 1 This inherently introduces extra complexity to the learning of Q (2) Challenge 2: During testing, oppo- nents are non-stationary, meaning they can switch policies. In such cases, it is difficult to ensure the reliability of the Q estimation, as the current Q estimation may not correspond to the action values under the opponents' newest policy.
In Sec. A. we provide extensive related work on Opponent Modeling, In- Context Learning, Offline RL, Offline Multi- Agent RI., and Transformers for Decision- Bacision-Makiin.
3. Methodology
OOM algorithms typically follow a supervised pretraining paradigm, and their performance heavily depends on the self-agent policies embedded in T. However, in the real world, the collected T is often suboptimal, where the em- bedded self-agent policies can be arbitrarily poor. In such cases, the pretrained OOM algorithms could also be arbi- trarily suboptimal. In this work, we propose a simple and general algorithmic improvement framework, Truncated Q- driven Instant Policy Refinement (TIPR). The framework
Under the formalization of OOM, assuming Q denotes the Original Q parameterized by w. Then, Q can be learned by:
"T Q are self-agent's 's terms. Omit superseript for simplicity.
min E
at a [(((o),
G! (1)
3

---

h Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Here, Gt -i - represents the self-agent's full horizon Return-To-Go (RTG). From Eq. (1), we can intu- itively observe that the error in the optimization objective of Q accumulates primarily through the accumulation of rewards over time/ t' in G. Therefore, for a given sample (of, dj. a. ) the larger the absolute value of -t, the more difficult it becomes for Q to fit that RTG label Ct.
exp (-x)). We define the value loss Loy for learning Qy
using Mean Square Error (MSH) follows:
1 H-1 Σ H
at D) rtth)?
fov
h=0
To maximize Truncated Q's ability to recognize opponents through ICL, our neural architecture adopts a causal Trans- former (Radford et al., 2019). See more details on neural network design in Sec. E. Moreover, we introduce an aux- iliary Opponent Imitator hoअ (a ,D), which learns to estimate the opponents' action given o² through condi- tional imitation learning. for the use during the IPR process.
As mentioned in Challenge 1, inclusion of opponent agents further exacerbates the difficulty of learning Q. This effect becomes more pronounced as the number of opponents in- creases, as demonstrated in Fig. for PD (2 opponents) and PP (3 opponents). To address this challenge, we argue that truncating the horizon over which Q estimates the expected return from the full length to a fixed, truncated length can mitigate the cumulative error effect during the learning of Q. This truncation makes it feasible to learn a workable Q, as thoroughly validated by experimental results in Sec. 4.2.
D),
3.2. Instant Policy Refinement
Learning good Q is undoubtedly important, but how to Q to improve OOM algorithms is equally critical. A direct way is to adopt the OCL methodology, using Q for policy improvement while adding conservative constraints to en- sure the iteration does not deviate too far from the trajectory distribution of T. Existing OOM algorithms typically adopt supervised pretraining objectives, where these objectives can conveniently serve conservative constraints.
To address the aforementioned Challenge 2, we argue that a sound Q should estimate the action values under the current opponent policy. Such a Q can provide reliable estimations even in the presence of non-stationary opponents. There- fore, we design Q to be in-context, incorporating opponents' (o (o-1aa=) as additional conditional inputs. This design po- tentially enables Q to characterize the current opponent policy through In-Context Learning (ICL) (Lin et al., 2024). in-
Assuming original policy of OOM algorithm is denoted as म and parameterized by 0, the objective of learning a improved मे using OCL based on Q can be informally written as:
To sum up, we propose learning a context action-value function Truncated Q, denoted as Q, to overcome the challenges associated with learning Q in OOM. Assuming that Q estimates the expected return over truncated horizon of H. Q decomposes the prediction of the expected return within H steps into predictions of the specific rewards for each timestep. Here, Q is composed of Qc and Qv- For a specific future timestep h. Qc and @" predict the Confidence of obtaining the h-th reward and the Value of the h-th reward, respectively (see the Left side of Fig. 2). Specifically, we learn Q through the objective:
E max k-1 to bo 0
A (o al, o, D)
-1 . Loom]
~Tt DDD. (1
Here, Q (1:5))) = EM 7 / Qv(::.))) denotes the truncated cumulative value. LOOM typically adopts a cross- entropy form imitate the actions of self-agent sampled from T. which is also equivalent to certain Kullback- Leibler divergence constraint that prevents deviation from the tra- jectory distribution of T (Jing et al., 2024a). À and are the coefficients for the policy improvement term and the conservative constraint term, respectively.
E
min k- [K] (o at -0 (a Loe + 3- Lqul.
The above OCL method emphasizes improving the policy the trajectory distribution of T during offline training. However, in the context of OOM, opponents during test- ing are unknown and may even be entirely unseen. In such cases, severe distributional shifts can occur, rendering the improvements made offline ineffective on the new trajec- tory distribution. Therefore, rather than directly following OCL, we propose improving the original policy of the pre- trained OOM algorithm during testing through IPR (see the Right side of Fig. 2). During testing, IPR first determines whether to refine the original policy based on the estimated confidence. If refinement is deemed necessary, IPR then generates the refined policy based on the estimated value.
Here, a and 3 are the coefficients for different loss terms, and D represents the In-Context Data mentioned in Sec. 2. We define the confidence loss foe for learning Qc with Binary Cross Entropy (BCE) as follows:
1 H-1
Loe = Lucr (' at, a, D), D), (r # 0}),
Σ h=0
H
where CBCE(E. = - (y log(z) + (1 (1-y) v) log(1 - )) {(} is the indicator function, and the outputs of Qc are assumed to have been processed by Sigmoid function (x) 1/(1+ +
h
We provide the detailed procedure of IPR as follows.
4

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Algorithm 1 Truncated Q-driven Instant Policy Refinement
to generate self-agent action, i.e,, ot #(-lo}, Di).
Require: Offline dataset T,, original OOM policy के pre-
trained on T, truncation horizon
Generate the refined policy. If the RC is satisfied, IPR use the estimated value Qy to derive the refined policy R. Specifically, म generates self-agent action oj by
Ensure: IPR generated policy TIPR
1: /* Truncated Q Training (Section 3.1) */
2: Initialize Truncated Q model Q's parameters w
#(o) aT' Dt) arg max Vo(o) a af D.). (3) E'EAt
3: while Training is not finished do
4: Sample a batch of opponent policy index ~ [K]
5: Sample batch of training data and label TT
To summarize, at timestep i during testing, given of, the prepared Dr and a, the self-agent policy induced by the IPR can be expressed as:
(ol, az, ai [r/.)r -
6: Sample batch of In-Context Data D D* ~ T
7: Compute confidence loss Loe. value loss Lau and imitation loss Lor for Opponent Imitator noi
TIPr ( (ot, af D = { [#(o), ar Dr), RC is satisfied ( T(-1o}, De), otherwise
8: Update w using gradients of a Lqe + B. LQv + Coi
9: end while
Intuitively, IPR performs policy refinement (greedy) only when it has high confidence in the current value estimation, while maintaining the original policy (conservative) in other cases. This policy balancing mechanism validated in our experiments to effectively trade-off between greediness and conservativeness, achieving better performance improve- ments compared to always being greedy conservative.
10: /* Instant Policy Refinement (Section 3.2) #/
11: for each timestep / during testing do
Get self-agent observation of
13:
Sample In-Context Data D, from the most recent C
opponent trajectories
14:
Estimate opponent actions with Opponent Imitator:
a, Tor(- D.)
15:
Compute Refinement Condition (RC) fRe by Eq. (2)
3.3. The Rationale behind Truncated Q
16:
if fRc(OL; 0r D2) > 0 then
17:
Generate refined policy # by Eq. (3)
Truncated Q shortens the horizon for estimating the ex- pected return from the full length to fixed, truncated length. This truncation's 's impact the effectiveness of Q function requires further investigation. This subsection provides a theoretical analysis to justify the rationality of Truncated Q
18:
Act using refined policy: at # "1 D.)
19: else
20: Act using original OOM policy: at ~ a(- De)
21:
end if
22:
end for
When using Q to improve the original policy, what really matters is whether the learned Q can correctly choose the action with the highest true expected return. This is the most fundamental criterion for evaluating the effectiveness of the learned Q Let the truncated horizon random vari- able be denoted as h, and the neural network of Truncated Q be denoted as Qn. Given any 0², a- and D, we aim for the following Overall No Maximization Bias (NMB) Probability y(h) to be as large as possible:
23:
return IPR generated policy Zipr
Prepare & a a- At each timestep t, some preparations need to be made. First, we randomly sample M tuples of (o=),a=1) from the most recent C' historical trajectories of the opponent to construct In-Context Data D,, which helps characterize the current opponent policy. Next, we use the Opponent Imitator foi to sample a, ~ Tor(- Dt), as the opponents' action at timestep is unknown. After these preparations, we use the estimated confidence Qc to determine whether perform policy refinement.
(hh) = P(arg max Qn = arg max EGT)) a²
(4)
at
Gn denotes the horizon-truncated RTG random variable, while G7 denotes the original RTG random variable. It is straightforward to derive that y(h) satisfies that:
Decide whether to refine or not. We define the trun- cated cumulative confidence ()())))) = -H11
y(h) 2 (h)g((h),
Bern fern(Qcl where Bern(-) denotes sampling 1 or O from the Bernoulli distribution. Next, define the Re- finement Condition (RC) follows:
where f(h) = P(arg max Qn arg mnax IG,,),
(5)
a²
g(h) := P(arg max EG, = arg max EG,)
fac a (.)))= = Σ 1108(0, a, D.) > 0}. alEAl
Here, /(h) denotes the Empirical Risk NMB Probability, which is determined by the neural network's fitting process. g(h) denotes the Natural NMB Probability, which de- termined by the intrinsic properties of the environment's 's reward structure. Next, we present the following theorem:
(2)
If fRc(-...)2 > 0 (RC is satisfied), IPR performs policy refinement. Otherwise, IPR uses the original 00M policy
5

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Theorem 3.1. For any given o², at,a-=, D, we have ((h)) N
domly appears on the field at the beginning. MS requires players to be flexible in both offense and defense.
f(h) == 1 1- -21o. where no satisfies that
Level-Based Foraging (LBF) is two-player mixed-incentive game with discrete state space. In LBF, the self-agent aims to eat as many apples as possible. All players and apples have a level. LBF represents a typical social dilemma and necessitates cooperation with the op- ponent to eat apples of higher level than the self-agent's s. Physical Decept i on (PD) is a three-player (two op- ponents) game with a continuous state space. Self-agent aims to hit an unknown target landmark, where there is a fake landmark and target one. Opponents aim to prevent self-agent from hitting the target. Self-agent has to detect the opponents' deception and identify the real target. Predator Prey (PP) is a four-player (three oppo- nents) game with a continuous state space. Self-agent is prey aims to avoid being captured by three predators (opponents). There are two obstacles. The challenge of PP lies in the need to model all three opponents simulta- neously and handle potential cooperation among them.
5(In TT + 1) - In no ht/p://11111110 ||AID)
min
min
(6)
In 1/// //0p//111/// |
U²
24
4
The asymptotic complexity of the term A (h) is Ohh))
To facilitate a clear understanding of the rationale behind our theory, we provide the detailed explanations of all the notations used in Thm. 3.1 in Sec. B. The complete proof of Thh. 3.1 in Sec. B.2. From Thm. 3.1, we conclude that as the number of opponents increases (leading to larger |A] and (D), where D is the set of all possible D), f(h) decreases. This implies that fitting RG, with Qn becomes more challenging, which supports Challenge I mentioned in Sec. 3.1 and aligns with the results shown in Fig. 1. Based on Thm. 3.1, we propose the following proposition:
OOM Baselines. Considering that most existing OM ap- proaches follow online learning paradigm, we include OOM algorithms derived from puttrining-focusse OM ap- proaches adopted by Jing et al. (2024a) as baselines.
Proposition 3.2. For any given a, there exists an optimal truncated horizon h" E [T] that maximizes gh)), i.e., the lower bound of y(h).. Specificully, there exist func- tions g such that h* # T.
DRON-concat (He et al., 2016a): Encode hand-crafted features of opponents with linear network. It concate- nates the self-agent's 's and the opponent's hidden states to aid the downstream policy optimization.
The proof and intuitive analysis of Prop. 3.2 can be found
in Sec. B.3. According to Prop. 3.2, as h increases, f(h)
exhibits decreasing trend, while (hh) generally tends to
increase. Therefore, selecting an appropriate truncated hori-
DRON-MoE (He et al., 201 6a): Encode hand-crafted features of opponents with Mixture-of-Expert network while also predicting opponents' actions to model the opponents (the most performant version in their paper).
zon h for estimating the expected return enables us to trade
off between f(h) and g(h) to maximize y(h). This, in turn,
allows us to select the action with the highest EG, with
higher probability when improving the original policy. To
LIAM (Papoudakis et al., 2021a): Use the observations and actions of the self-agent to reconstruct those of the opponent through an auto-encoder, thereby embedding the opponent policy into a latent space to model opponents. Prompt-DT (Xu et al., 2022): Based on Decision Trans- former (Chen et al., 2021), sample expert task trajectories as prompts to obtain task adaptability through offline pre- training. We adopt the version in Jing et al. (2024a) that demonstrates great potential to handle OOM problems.
sum up, our theoretical analysis indicates that compared to
unthinkingly setting h to 7 (which is the case for Original
Q), learning Truncated Q is more reasonable.
4. Experiments
In this section, Sec. 4.1 provides a detailed description of the experimental setup. Sec. 4.2 poses a series of questions and presents empirical results to answer them, aiming to analyze the effectiveness of the TIPR framework.
TAO (Jing et al., 2024a): 1) Pretrain a well-structured opponent policy embedding with representation learning. 2) Learn to respond to the opponent policies based on the learned policy embedding with a Transformer model. 3) Adapt to unknown opponents with ICL during testing.
4.1. Experimental Setup
Environments. We consider four sparse-reward competi- tive multi-agent environmental benchmarks. See Sec. C for detailed introductions of these environments.
Opponent Policies & Offline Datasets. We employ diversity-driven Population-
Training algorithm MEP (Zhao et al., 2023) to train policy population. Poli- cies from this population are used to construct If and o Opponent policies generated using MEP have been shown to be performant and exhibit diversity (Jing et al., 2024b).
Markov Soccer (MS) is a two-player zero-sum game with a discrete state space. In MS, the self-agent's 's objec- tive is to move the ball towards the opponent's 's goal, with the opponent having the same objective. The ball ran-
6

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
DRON concat
DRON moe
_RON_moe
LIAM
Prompt DT
TAO
DIA
w/ TIPR (Performance Increment)
MS Optimal Ratio p: 0.2
MS Optimal Ratio p: 0.4
MS Optimal Ratio 0.6
MS Optimal Ratio p: 0.8
0.0
-0.2
0.4
-0.6
-0.8
-1.0
LBF Optimal Ratio 0.2
LBF Optimal Ratio 0.4
LBF Optimal Ratio p: 0.6
LBF Optimal Ratio p: 0.8
0.5
0.4
0.3
Average Return
0.2
0.1
0.0
PD Optimal Ratio p: 0.2
PD Optimal Ratio p: 0.4
PD Optimal Ratio p: 0.6
PD Optimal Ratio p: 0.8
150
125
75
25
Optimal Ratio p:
Optimal Ratio p:
Optimal Ratio p: 0.6
PP Optimal Ratio pt 0.8
0
500
-100
-150
-200
-250
-300
Seen
Unseen
Mixed
Seen
Unseen Test Mode
Mixed
Seen
Unseen
Mixed
Seen
Unseen Test Mode
Mixed
Test Mode
Test Mode
Figure 3. The average original results of various OOM baselines tested against unknown non non-stationary opponents, along with the average results after improvement using the TIPR framework. The different subplots various environments and offline datasets with different levels of suboptimality to pretrain the OOM algorithms. We use 'black error bars' to represent the Standard Deviation (SD) of the original OOM policy's 's performance and "grey error bars' represent the SD of the performance after applying TIPR.
We quantitatively measure and visualize the diversity of op- ponent policies within the MEP population used construct the offline datasets T in Sec. D.
MEP population that have never appeared in IT 3) Mixed: This II the union of the Seen and Unseen.
All figures and tables report the Mean and Standard Devia- tion (SD) of the results averaged over 5 random seeds. See all the hyperparameters in Sec. F.
Offline datasets T with varying suboptimality, measured by the Optimal Ratio are constructed. We define the ratio of the performance self-agent policy embedded in to the performance the BR policy against the opponent policy. Self- Selftt policies used to construct of different p are obtained by training with PPO (Schulman et al., 2017) for varying numbers of steps while keeping opponent policy fixed. The smaller p, the more suboptimal the dataset T is.
4.2. Empirical Analysis
Question 1. Can TIPR effectively handle the suboptimal- ity of OOM algorithms induced by the offline datasets?
Fig. shows the original testing results of all OOM baselines and their results after applying the TIPR framework. use white shading pattern to highlight the performance incre- ments achieved using the TIPR framework relative the original OOM policy. It can be observed that decreases, the performance of all OOM baselines generally drops dra- matically, reaching an unacceptable level (especially in PD and PF). After applying the TIPR framework, all OOM baselines achieved consistent performance improvements. The improvements achieved by TIPR remain stable across different p settings and are particularly pronounced in envi- ronments such PD. These results demonstrate that TIPR can effectively handle the suboptimality of OOM algorithms
OOM Pretraining & Testing Protocols. We pretrain all OOM baselines for 3000 steps. The final checkpoints of the pretrained OOM baselines were used to test against Un- known Non-stutionary Opponents for 2400 episodes. 'Un- known' indicates that the true policy of the opponents is unknowable the self-agent. "Non-stationary' means that the opponent switches its policy by sampling from I every 20 episodes. We set up three types of IT
1) Seen: This o is equivalent to IToff which contains 12 policies selected from the MEP population.
2) Unseen: This o contains 8 policies selected from the

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
OOM
OOM-Original Q
OOM-Truncated Q w/o
OOM-Truncated Q w/o Q
OOM-Truncated Q
MS IPR
LBF IPR
PP IPR
0.6
0.0
0.5
150
-50
-0.2
0.4
0.3
120
90
-100
150
Average Return
-0.4-
0.2
60
200
-0.6
0.1
250
0.0
300
MS- OCL
LBF OCL
PD OCL
PP. OCL
0.6
125
0.0
0.5
100
-50
-0.2
0.4
75
1000
0.3
-0.4
0.2
50
200
-0.6
0.1
25
250
0.0
300
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Optimal Ratio p
Optimal Ratio p
Optimal Ratio p
Optimal Ratio p
Figure 4. The average testing results of improving OOM algorithm using different variants of Q through both IPR and OCL methods, where the IT is set Mixed. Some hars have extreme values, making them difficult to observe clearly. E.g. in subplot 'PD IPR', the average returns of 'OOM-Oris Original Q' are close to O. In subplot 'PP- IPR", the average returns of (OO--rriging Q' are less than -000
Table 1. The quantitative metrics estimated values when improv- ing OOM algorithms during testing using Original Q and Truncated Q through the IPR method, where the 11 is set to Mixed.
distribution than making improvements offline.
Compared to Original Q can using Trun- cated Q more effectively improve OOM algorithms?
Question 3.
Evaluation Term
Optimal Ratio p
Env.
0.2
0.4
0.6
0.8
Let us continue observing Fig. 4. Focusing on the results of using Truncated Q and Original Q for improvements, we find: Regardless of whether the improvement is achieved through IPR OCL, 'OOM-Truncated" Q' generally out- performs 'OOM-Original Q' in effectively improving the policy. '0OM-Origine Q' can completely spoil the original OOM policy in many cases, such as in PD and PP.
Q MSE L QU MSE
63.9 18.7
97.5 + 52.4
15.9
70.5 + 14.1
MS
0.5 0.1
0.8 0.2
0.7 17700 0.2
0.7 + 0.1
Qu Acc. (50
99.3 19.31:00 0.2
99.0 1 0.2
99.2 1: 0.2
99.3 1: 0.1
Q MSE 1
46.1 = 28.1
23.6 1.5
29.8 + 9.7
42,0 + 14.0
CBF
Qu MSE
5.8 = 1.1
5.1 1.2
5.1 1.6
4.7 + 1.6
Qu Acc. (501
97.0 1: 0.1
97.4 1. 0.6
96.5 1: 1.0
97.1 1 1.4
L
4.3
77.8 34.4
253.9 54.7
372.1 +: 105.0
ED
Qu MSE
2.9:0.11
2.5 1::0044 0.4
2.6 2.6::00 0.3
2.4 + 0.3
Qu Acc. (7)
90.7 = 10.7
86.1 + 3.3
87.6 4.4
Table presents the statisties for predictions made when im- proving OOM algorithms using Original Q and Truncated Q through IPR during testing. We use MSE to measure the Q's value estimations and Accuracy to evaluate the precision of the Q's confidence. Truncated Q consistently demon- strates higher accuracy in its estimations across all games. In contrast, Original Q generally exhibits very low predic- tion accuracy and fails to produce meaningful predictions in PP. These observations support that Truncated Q is more effective than Original Q in improving OOM algorithms.
L
4.1c5 = 3.5e5
3.265 + 3.5e4
1.855 :: 2.1e5
3.1e5 4: 2.5e5
BP
QU MSE
3.3 0.4
2.9 3.2 0.2
2.5 4.5::33
2.1 +
Qe Acc. (7)
89.3 0.2
90.2 + 0.7
88.7
90.9 1.1
induced by the offline datasets.
Question 2. Compared to OCI,, can our proposed IPR more effectively improve OOM algorithms?
In Fig. 4, we present the testing results of improving the pol- icy of OOM algorithms using different Q variants through both IPR and OCL. We select the most representative OOM algorithm, TAO, for observation and analysis. 2 Let us focus on the results of improving OOM algorithm using Trun- cated Q through IPR and OCL, respectively. We observe that, compared to OCL, the IPR method generally provides more effective improvement to the original O0M policy. OCL can even degrade the performance of OOM algorithms in environments such as PD. This can be attributed to the severe distribution shift between the trajectory distribution during testing and T. As a result, performing improvements during testing is better suited to adapt to the new trajectory Due to space limits, subsequent ablations also focus on TAO.
Question 4. Do all the key design choices of Truncated Q contribute positively to improving OOM algorithms?
Fig. 4 also includes the testing results of improving OOM algorithm using ablated variants of Truncated Q. Within,
"Truncated Q w/o QC h, tion of confidence, thus always performing policy refine- ment (greedy). 'Truncated w/o D' indicates the variant where the in-context data input is removed. Focusing on the results of improvements made using IPR, we can find that both "Truncated Q w/o Q" and 'Truncated Q w/o D' generally suffer varying degrees of performance degradation compared to 'Truncated Q'. These observations demonstrate
a variant that removes the estima-
8

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
OOM
OOM-Truncated Q-H1
OOM-Truncated Q-H3
OOM-Truncated Q-H5
OOM-Truncated Q-H10
Average Return
MS
LBF
PD
PP
0.6
175
0.0
150
-0.2
0.4
0.3
100
125
80
-0.4
0.2
100
-0.6
0.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Optimal Ratio p
Optimal Ratio
Optimal Ratio p
Optimal Ratio p
Figure 5. The average testing results of improving OOM algorithms using Truncated Q with different truncated horizons H, where the 11TH is set to Mixed. We use different colors to represent Truncated trained with different values of H.
OOM
OOM w/ TIPR
Optimal Ratio: 0.2
Optimal Ratio: 0.4
Optimal Ratio: 0.6
Optimal Ratio: 0.8
MS
LBF
PD
PP
Average
-0.4-
-0.3
-0.2
-0.5
-0.1
-0.6-
-0.7-
-0.8
0.4
0.5
0.1
0.3
0.2
175
150
125
100
75
50
25
-300
-350-
-50
200
100
150
-50
1000
2000
3000
0
1000
2000
3000
1000
2000
3000
1000
2000
3000
Training Steps
Training Steps
Training Steps
Training Steps
Figure 6. The average testing results of improving OOM algorithms during their pretraining stage using the TIPR framework, where the Il set to Mixed. We use different colors to represent the datasets with varying Optimal Ratios used for pretraining.
that weighing the decision to refine based on confidence es- timation and characterizing opponent policies with D both contribute positively to the effectiveness of Truncated Q-
TIPR can reduce the number of training steps required for OOM algorithms to achieve the same level of performance.
5. Discussion
Question 5. horizons H for Truncated Q affect the improvement results?
How does the choice of different truncated
Summary. This paper investigates critical yet underex- plored problem: How to address the degradation in algo- rithm performance caused by the suboptimality of offline datasets in OOM. We propose TIPR, a simple and general algorithmic improvement framework that learns horizon- truncated in-context action value Truncated Q and performs IPR during testing to address this problem. Our theory justi- fies the rationality of Truncated Q by analyzing how the trun- cated horizon affects the No Maximization Bias probability. Experimental results demonstrate that TIPR effectively im- proves various OOM algorithms pretrained on suboptimal datasets, significantly enhancing their adaptability to un- known non-stationary opponents.
In Fig. 5, we present the testing results of improving OOM algorithm using Truncated Q learned with different trun- cated horizons Hl under the TIPR framework. We set four different values for H: 3, 5, and 10. We observe that TIPR's improvement OOM algorithm does not exhibit a strict correlation trend with the size of H. In different environments, an optimal value of H exists, which can be considered a tunable parameter. However, when the value of H becomes too large, it may lead to adverse improvements on the original OOM policy (e.g., in LBF and PD). When H T, Truncated degenerates into Original Q-
Question 6. Can TIPR potentially improve the pretraining efficiency of OOM algorithms?
Limitations and future work. In Poiani et al. (2023), trajectories of different lengths are truncated to estimate the action value Q. Unlike our work, their goal is to find the optimal combination of truncation lengths under given sampling budget to estimate Q as accurately as possible. Enlightened by their work, we think that, given a dataset, how find the optimal truncated horizon H for Truncated Q is problem worth further investigation. Furthermore, in this work, the testing opponents choose policy from a fixed set of policies for switching. An interesting future research direction would be exploring how to leverage offline datasets to handle opponents continuously updating their policies.
Fig. 6 shows the results of using TIPR to improve OOM algorithms during their pretraining process. Specifically, we periodically take checkpoints during the OOM algorithm's 's pretraining stage for testing. We find that using TIPR during pretraining consistently and effectively improves OOM algo- rithms, regardless of the p value of the T used for pretraining- Moreover, the improvements achieved by TIPR remain rel- atively stable from the beginning to the end of pretraining. This suggests that TIPR can enhance the pretraining effi- ciency of OOM algorithms. When is suboptimal, using

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Acknowledgements
Bai, C., Wang, L.,, Yang, Z.., Deng, Z.-H., Garg, A., Liu, P., and Wang, Z. Pessimistic bootstrapping for uncertainty- driven offline reinforcement learning. In International Conference on Learning Representations, 2021.
This work supported in part by the Strategic Priority Re- search Program of the Chinese Academy of Sciences (Grant No. XDA0480200), the National Science and Technology Major Project (2022ZD01 16401), the Natural Science Foun- dation of China (Grant Nos. 62222606 and 61 61902402), and the Key Research and Development Program of Jiangsu Province (Grant No. BE2023016).
Bai, Y., Chen, F., Wang. H., Xiong, C., and Mei, S. Trans- formers statisticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.04637 2023.
Bard, N., Johanson, M., Burch, N., and Bowling, M. Online implicit agent modelling. In International Conference Autonomous Agents and MultiAgent Systems, pp. 255- 262, 2013.
Impact Statement
This work advances the field of OOM by addressing the critical limitation of suboptimal offline datasets, which are prevalent in real-world multi-agent scenarios. The proposed TIPR framework enhances the robustness and adaptability of autonomous agents without requiring additional online interactions, thereby reducing resource consumption and potential risks in deployment. Ethically, the framework en- courages safer Al deployment by minimizing reliance on idealized data and promoting reliability under imperfect conditions. However, care must be taken to prevent misuse in adversarial or deceptive applications, particularly in com- petitive or high-stakes domains such as finance or security. Future societal implications include improved collabora- tion and competition among AI agents in diverse settings, fostering broader trust and utility in autonomous systems.
Bernardo, M. and Smith, A. F. Bayesian theory, volume 405. John Wiley & Sons, 2009.
Brandfonbrener, D., Whitney, W., Ranganath, R., and Bruna, J. Offline without off-policy evaluation. Advances in neural information processing systems, 11:4933-4945 2021.
Brandfonbrener, D., Bietti, A., Buckman, J., Laroche, R., and Bruna, J. When does return couur--oondiiooeed supervised learning work for offline reinforcement learning? In Ad- vances in Neural Information Processing Systems, pp. 1542-1553, 2022.
Bridle, J. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. In Advances in Neural Infor- mation Processing Systems, pp. 211-217, 1989.
References
Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. Transform- learn to implement preconditioned gradient descent for in-context learning. In Advances in Neural Informa- tion Processing Systems, pp. 35151-351744 2023.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, pp. 15584-150977 2021.
Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learning? inves- tigations with linear models. In International Conference on Learning Representations, 2023.
Christianos, F., Schäfer, L., and Albrecht, S. V. Shared expe- rience actor-critic for multi multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Al-Shedivat, M., Bansal, T., Burda, Y., Sutskever, I., Mor- datch, L., and Abbeel, P. Continuous adaptation via meta- learning in nonstationary and competitive environments. In International Conference on Learning Representations, 2018.
Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv: 151107289,, 2015.
Albrecht, S. V. and Stone, P. Autonomous agents modelling other agents: A comprchensive survey and open problems. Artificial Intelligence, 258:66-95, 2018.
Cui, Q. and Du, S. S. Provably efficient offline multi-agent reinforcement learning via strategy-wist bonus. In Ad- vances Neural Information Processing Systems, pp. 11739-11751, 2022.
Axelrod, R. Effective choice in the prisoner's ' dilemma. Journal of conflict resolution, 24(1):3-25, 1980.
Dai, Z., Chen, Y., Low, B. K. H., Jaillet, P., and Ho, T.-H. R2- b2: Recursive reasoning based bayesian optimization for no- no-regret learning in games. In International Conference on Machine Learning, pp. 2291-2301 2020.
Ba. J. L., Kiros, J. R., and Hinton, G.E. E. Layer normalization. Stat, 1050:21, 2016.
10

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
DiGiovanni, A. and Tewari, A. Thompson sampling for markov games with piecewise stationary opponent poli- cies. In Uncertainty in Artificial Intelligence, pp. 738- 748, 2021.
In International Conference on Learning Representations, 2022.
Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What transformers learn in-context? case study of simple function classes. In Advances in Neural Information Processing Systems, volume 35, pp. 30583-30598, 2022.
Dorfman, R., Shenfeld, I., and Tamar, A. Offline meta reinforcement iaaiffig-dentfffabiiitty challenges and ef- fective data collection strategies. In Advances in Neural Information Processing Systems, pp. 4607-4618, 2021.
Ghasemipour, S. K. S., Schuurmans, D., and Gu, S. S. Emaq: Expected-mas max q q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp.
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I. and Abbeel, P. RL Fast reinforcement learn- ing via slow reinforcement learning. arXiv preprint arXiv: 1.0779, 2016.
PMLR, 2021.
Grigsby, J., Fan, L.,, and Zhu, Y. Amago: Scalable in-context reinforcement learning for adaptive agents. In The Twelfth International Conference on Learning Representations, 2023.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. In Interna- tional Conference on Machine Learning, pp. 1126-11355 1135, 2017.
Grover, A., Al-Shedivat, M., Gupta, J., Burda, Y., and Ed- wards, H. Learning policy representations in multiagent systems. In International Conference on Machine Learn- ing. pp. 1802-1811, 2018.
Foerster, J., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, Learning with opponent- learning awareness. In International Conference Au- tonomous Agents and MultiAgent Systems, pp. 122-130, 2018a.
He, H., Boyd-Graber, J., Kwok, K., and Daumé III, H. Oppo- nent modeling in deep reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 1804-1813, 2016a.
Foerster, J.. Farquhar, G., Al-Shedivat, M., Rocktäschel, T., Xing, E., and Whiteson, S. DiCE: The infinitely differen- tiable Monte Carlo estimator. In International Conference on Machine Learning, pp. 1524-1533, 2018b.
He, K. Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016b.
Formanek, C., Jeewa, A., Shock, J., and Pretorius, A. Off- the-grid marl: framework for dataset generation with baselines for cooperative offline multi-agent reinforce- ment learning. arXiv preprini arXiv:2302.00521, 2023.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint NXX.:v666600841.. 2016.
Fu. H., Tian, Y., Yu, H., Liu, W., Wu, S., Xiong, J., Wen, Y., Li, K., Xing, J., Fu, Q., al. Greedy when sure and conservative when uncertain about the opponents. In International Conference Machine Learning, pp. 2022.
Hernandez-Leal, P., Taylor, M. E., Rosman, B. S., Sucar, L. E., and Munoz de Cote, E. Identifying and track- ing switching, non-stationary opponents: A bayesian ap- proach. In AAAI Conference on Artificial Intelligence Workshop on Multiagent Interaction without Prior Coor- dination, pp. 560-566, 2016.
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Hoeffding, W. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffd- ing, pp. 409-426, 1994.
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In Interna- tional Conference on Machine Learning, pp. 2052-2062, 2019.
Hong, Z., Su, S.-Y., Shann, T.-Y., Chang, Y.-.., and Lee, C.-Y. A deep policy inference nenetwork for multi-agent systems. In International Conference on Autonomous Agents and MultiAgent Systems, pp. 3388-1396. 2018.
Fung, K., Zhang, Q., Lu, C., Willi, T., and Foerster, J. N. Analyzing the sample complexity of model-free oppo- nent shaping. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023.
Hu, S., Fan, Z., Huang, C., Shen, L., Zhang, Y., Wang, Y., and Tao, D. Q-value regularized transformer for offline reinforcement learning. In Forty-first International Con- ference on Machine Learning, 2024.
Furuta, H., Matsuo, Y., and Gu, S. S. Generalized decision transformer for offline hindsight information matching.

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Jaiswal, A., Babu, A. R., Zadeh, M. Z.,, Banerjee, D., and Makedon, F. A survey on contrastive self-supervised learning, Technologies, 9(1):2 2020.
Laskin, M., Wang, L.., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S. S., Filos, A., Brooks, E., Gazeau, M., Sahni, H., Singh, S., and Mnih, V. In in-context reinforcement learning with algorithm dis- tillation. In International Conference on Learning Repre- sentations, 2023.
James, G., Witten, D., Hastie, T., Tibshirani, R., et al. An introduction to statistical learning, volume 112. Springer, 2013.
Lee, J. N., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. Supervised pretraining can learn in-context reinforcement learning. arXiv preprint arXiv:2306.14892, 2023.
Jiang, J. and Lu, Z. Offline decentralized multi-agent re- inforcement learning. arXiv preprint 1/81122008.01832 2021.
Janng J. and Lu, Z. Online tuning for offline decentralized multi-agent reinforcement learning. In AAAI Conference on Artificial Intelligence, pp. 8050-8059, 2023.
Lee, K.-H., Nachum, O., Yang, M. S., Lee, L., Free- man, D., Guadarrama, S., Fischer, I., Xu, W., Jang, E., Michalewski, H., et al. Multi- Bulti-game decision transformers. In Advances in Neural Information Processing Systems, pp. 27921-27936, 2022.
Jin, Y., Yang, Z., and Wang, Z. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pp. 5084-5096, PMLR, 2021.
Letcher, A., Foerster, J., Balduzzi, D., Rocktäschel, T., and Whiteson, S. Stable opponent shaping in differentiable games. In International Conference on Learning Repre- sentations, 2019.
Jing, Y., Li, K., Liu, B., Zang, Y., Fu, H., FU, Q., Xing. J., and Cheng, J. Towards offline opponent modeling with in-context learning. In The Twelfth International Conference on Learning Representations, 2024a.
Li, L., Yang, R., and Luo, D. Focal: Efficient fully fully-offline meta-reinforcemmet learning via distance metric learning and behavior regularization. In International Conference on Learning Representations, 2020.
Jing, Y., Liu, B., Li, K., Zang, Y., Fu, FU, Q. Xing, J., and Cheng, J. Opponent modeling with in-context search. In The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024b.
Li, W., Luo, H., Lin, Z., Zhang, C., Lu, Z., and Ye, D. A survey on transformers in reinforcement learning. Trans- actions on Machine Learning Research, 2023a.
Jing, Y. Li, K., Liu, B., Fu, H., Fu, Q., Xing, J., and Cheng, J. An open-ended learning framework for opponent model- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.
2025.
Li, Y., Emrullah Ildiz, M., Papailiopoulos, D., and Oymak, S. Transformers as algorithms: Generalization and implicit model selection in in-context learning. In International Conference on Machine Learning, pp. 19565-19594, 2023b.
Kim, D. K., Liu, M., Riemer, M. D., Sun, C., Abdulhai, M., Habibi, G., Lopez-Cot, S., Tesauro, G., and How, J. A policy gradient algorithm for learning to learn in multia- gent reinforcement learning. In International Conference on Machine Learning, pp. 5541 2021.
Lin, L., Bai, Y., and Mei, S. Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. The Twelfth International Con- ference on Learning Representations, 2024.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit gelarning. arXiv preprint 1xXiv:211000616 2021.
Littman, M. L. Markov games as a framework for multi- agent reinforcement learning. In Machine learning pro- ceedings 1994, pp. 157-163. Elsevier, 1994.
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabi- lizing off-policy q-learning via bootstrapping error reduc- tion. Advances in neural information processing systems, 32, 2019.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference Learning Representations, 2018.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conser- vative q-leariig for offline reinforcement learning. In Advances in Neural Information Processing Systems, pp- 1179-1191, 2020.
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mor- datch, I. Multi-agent actor-critic for mixed cooperative- competitive environments. In Advances in Neural Infor- mation Processing Systems, pp. 1-12, 2017.
I Lanctot, M., Lockhart, F.,, Lespiau, J.-B., Zambaldi, V., Upadhyay, S., Pérolat, J.. Srinivasan, S., Timbers, F,, Tuyls, K. Omidshafiei, S., et al. OpenSpiel: A frame- work for reinforcement learning in games. arXiv preprint arXiv: /908. 09453, 2019.
Lu, C., Willi, T., De Witt, C. A. S., and Foerster, J. Model- free opponent shaping. In International Conference on Machine Learning, pp. 14398-14411, 2022.
12

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Lv, Y., Yu, Y., Zheng, Y., Hao, J., Wen, Y., and Yu, Y. Lim- ited information opponent modeling. In International Conference on Artificial Neural Networks, pp. 511-522. Springer, 2023.
Paster, K., Mcllraith, S., and Ba, J. You can't count on luck: Why decision transformers and rvs fail in stochastic envi- ronments. In Advances in Neural Information Processing 2022.
Systems, pp.
Ma, L., Wang, Y. Zhong, F., Zhu, S.-C., and Wang, Y. Fast peer adaptation with context-award aware exploration. In Inter- national Conference on Machine Learning, pp. 33963- 33982. PMLR, 2024.
Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighhe regression: Simple and scalable off off-policy reinforcement learning. arXiv preprint 19X1v:990000077,. 2019.
Ma, X., Yang. Y., Hu, H., Yang. J., Zhang, C., Zhao, Q Liang, B., and Liu, Q. Offlin reinforcement learning with value-based episodic memory. In International Con- ference on Learning Representations, 2021.
Poiani, R., Metelli, A. M., and Restelli, M. Truncating trajectories in monte carlo reinforcement learning. In In- ternational Conference on Machine Learning, pp. 27994- 28042. PMLR, 2023.
Melo, L. C. Transformers are meta-reinforrement learners. In International Conference on Machine Learning, pp- 15340-15359, 2022.
Pong, V. H., Nair, A. V., Smith, L. M., Huang, C., and Levine. S. Offline meta-reinforrement learning with on- line sulf-suueri.sion. In International Conference Machine Learning, pp. 17811-17829, 2022.
Meng, L., Wen, M., Le, C., Li, X., Xing, D., Zhang, W., Wen, Y., Zhang, H., Wang, J., Yang. Y., al. Offline pre-trained multi-agent decision transformer. Machine Intelligence Research, 20(2):233-248, 2023.
Rabinowitz, N., Perbet, F., Song, F., Zhang, C., Eslami, S. A., and Botvinick, M. Machine theory of mind. In International Conference on Machine Learning, pp. 4218- 4227, 2018
Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P. A simple neural attentive meta-lcarner. In International Conference on Learning Representations, 2018
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Mitchell, E., Rafailov, R., Peng, X. B., Levine, S., and Finn, Offline meta-reinforcemmnt learning with advan- tage weighting. In International Conference Machine Learning, pp. 7780-7790 2021.
Raileanu, R., Denton, E., Szlam, A., and Fergus, R. Mod- eling others using oneself in multi-agent reinforcement learning. In International Conference on Machine Learn- ing, pp. 4257-4266, 2018.
Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Acceler- ating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.0913 09359, 2020.
Raventos, A., Paul, M., Chen, F., and Ganguli, S. Pretrain- ing task diversity and the emergence of non-bayesian in-context learning for regression. In Advances in Neural Information Processing Systems, pp. 1-13, 2023.
Nashed, S. and Zilberstein, S. A survey of opponent model- ing in adversarial domains. Journal of Artificial Intelli- gence Research, 73:277-327, 2022.
Pan, L., Huang, L., Ma, T., and Xu, H. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In International Conference Machine Learning, pp. 17221 17221-17237, 1,237,20222 2022.
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., BartB- G., Giménez, M. Sulsky, Y.. Kay, J., Springenberg. J. T., Fecles, T., Bruce, J., Razavi, A., Edwards, A., Heess, N., Chen, Y., Hadsell, R., Vinyals, O., Bordbar, M., and de Freitas, N. A generalist agent. Transactions on Machine Learning Research, 2022.
Papoudakis, G. and Albrecht, S. V. Variational autoencoders for opponent modeling in multi-agent systems. arXiv preprint arXiv:2001.10829 2020.
Rosman, B., Hawasly, M., and Ramamoorthy, S. Bayesian policy reuse. Machine Learning, 104:99-127, 2016.
Papoudakis, G., Christianos, F., Albrecht, S., and et al. Agent modelling under partial observability for deep re- inforcement learning. In Advances in Neural Information Processing Systems, pp. 19210-19222 2021a.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:
2017.
Papoudakis, G., Christianos, F., Schäfer, I.., and Albrecht, S. V. Benchmarking multi-agent deep reinforcement learning algorithms cooperative tasks. In Neural In- formation Processing Systems Track on Datasets and Benchmarks (NeurIPS), pp. 10707-10717, 2021b.
Shao, J., Qu, Y., Chen, C., Zhang, H., and Ji, X. Coun- terfactual conservative q learning for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-195)) 2014.
Wolf, T., Debut, L.,, Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-art natural language processing, In Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, pp. 38-45, 2020.
Tian, Q., Kuang, K., Liu, F., and Wang, B. Learning from good trajectories in offline multi-agent reinforcement learning. In AAAI Conference Artificial Intelligence, pp. 1672--11680, 2023.
Wu, C., Tang, P., Yang, J., Hu, Y., Lv, T., Fan, C., and Zhang, C. Conservative offline policy adaptation in multi- agent games, Advances in Neural Information Processing Systems, 36, 2024a.
Tseng. W.-C., Wang, T.-H. J., Lin, Y.-C., and Isola, Of- fline multi-agent reinforcement learning with knowledge distillation. In Advances in Neural Information Process- ing Systems, pp. 226-237, 2 2022.
Wu, J., Wu, H., Qiu, Z., Wang. J., and Long, M. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 31278-31291 2022a.
Vapnik, V. Statistical learning theory. John Wiley & Sons google schola, 2:831-842. 1998.
Villaflor, A. R., Huang, Z., Pande, S., Dolan, J. M., and Schneider, J. Addressing optimism bias in sequence mod- eling for reinforcement learning. In International Confer- ence on Machine Learning, pp. 22270-22283 2022.
Wu, Y., Tucker, G., and Nachum, O. Behavior regu- larized offline reinforcement learning. arXiv preprint arXiv: /911. 1161,, 2019.
Wu, Y.. Zhai, S., Srivastava, N., Susskind, J. M., Zhang, J., Salakhutdinov, R., and Goh, H. Uncertainty weighted actor-critic for offline reinforcement learning. In Inter- national Conference Machine Learning, pp. 1 11319- 11328. PMLR, 2021.
Von Der Osten, F. B., Kirley, M., and Miller, T. The minds of many: Opponent modeling in a stochastic game. In International Joint Conference on Artificial Intelligence, pp. 3845-38511 2017.
Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp. 35151-35174. PMLR, 2023.
Wu, Y., McMahan, J., Zhu, X., and Xie, Q. Reward poison- ing attacks on offline multi-agent reinforcement learning. In AAAI Conference on Artificial Intelligence, pp. 10426- 10434, 2023.
Wang, J. X., Kurth-Nelson, 7., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. Learning to reinforcement learn. arXiv preprint arXiv:1611. 1.0566, 2016.
Wu, Y.-H., Wang, X., and Hamaya, M. Elastic decision transformer. Advances in Neural Information Processing Systems, 36, 2024b.
Wu, Z., Li, K., Xu, H., Zang, Y., An, B., and Xing, J. L2c: Learning to exploit your opponent. In International Joint Conference on Neural Networks, pp. 1-8, 2022b.
Wang, Y., Yang, C., Wen, Y., Liu, Y., and Qiao, Y. Critic- guided decision transformer for offline reinforcement learning. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 38, pp. 15706- 15714, 2024.
Xiao, C., Wu, Y., Mei, J., Dai, B., Lattimore, T., Li, L., Szepesvari, C., and Schuurmans, D. On the optimal- ity of batch policy optimization algorithms. In Inter- national Conference on Machine Learning, pp. 1 11362- 11371. PMLR, 2021.
Wen, Y., Yang, Y., Luo, R., Wang, J., and Pan, W. Proba- bilistic recursive reasoning for multi- reinforcement learning. In International Conference on Learning Rep- resentations, 2019.
Wen, Y., Yang, Y., and Wang, J. Modelling bounded ratio- nality in multi-agent interactions by generalized recursive reasoning. In International Joint Conferences on Artifi- cial Intelligence, pp. 414 414-421, 2021.
Xiao, C., Wang, H., Pan, Y., White, A., and White, M. The in-sample softmax for offline reinforcement learning. In The Eleventh International Conference on Learning Rep- resentations, 2023.
Willi, T., Letcher, A. H., Treutlein, J., and Foerster, J. Cola: consistent learning with opponent-learnigg awareness. In International Conference on Machine Learning, pp- 23804-23831, 2022.
Xic, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Rep- resentations, 2021.
14

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Xu, M., Shen, Y., Zhang, S., Lu, Y., Zhao, D., Tenenbaum, J., and Gan, C. Prompting decision transformer for few- shot policy generalization. In International Conference on Machine Learning, pp.
in Neural Information Processing Systems, pp. 26324- 26336, 2022.
Zheng, Y., Meng, Z., Hao, J., Zhang, Z., Yang, T., and Fan, deep bayesian policy reuse approach against non- stationary agents. In Advances in Neural Information Processing Systems, pp. 962-972, 2018.
2022.
Yamagata, T., Khalil, A., and Santos- Rodriguz, R. Q- learning decision transformer: Leveraging dynamic pro- gramming for conditional sequence modelling in offline rl. In International Conference on Machine Learning, pp. 38989-39007. PMLR, 2023.
Zhu, Z., Liu, M., Mao, L., Kang. B., Xu, M., Yu, Y., Ermon, S., and Zhang, W. Madiff: Offline multi-agent learning with diffusion models, arXiv preprint arXiv:2305.17330 2023.
Yang, S., Nachum, O., Du, Y., Wei, J., Abbeel, P., and Schu- urmans, D. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129, 2023a.
Zhuang, Z., Peng, D., Liu, J., Zhang, Z., and Wang, D. Reinformer: Max-return sequence modeling for offline rl. In Forty-first International Conference on Machine Learning, 2024.
Yang, S., Schuurmans, D., Abbeel, P., and Nachum, O. Di- chotomy of control: Separating what you can control from what you cannot. In International Conference Learning Representations, 2023b.
Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., and Hof- mann, K. Deep interactive bayesian reinforcement learn- International Conference on
ing via meta-learning. Autonomous Agents and MultiAgent Systems, pp. 1712- 1714, 2021.
Yang. T., Hao, J., Meng, Z., Zhang, C., Zheng, Y., and Zheng, Z. Towards efficient detection and optimal re- sponse against sophisticated opponents. In International Joint Conference on Artificial Intelligence, pp. 623-629, 2019.
Yang, Y. and Wang. J. An overview of multi-agent reinforce- ment learning from game theoretical perspective. arXiv preprint arXiv:200 1.00533, 2020.
Yang. Y., Ma, X., Li, C.,, Zheng. Z., Zhang, Q., Huang, G., Yang, J., and Zhao, Q. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 14:1999911031,, 2021.
Yu, X., Jiang, J., Zhang, W., Jiang. H., and Lu, Z. Model- based opponent modeling. In Advances in Neural Infor- mation Processing Systems, pp. 28208-28222 2022.
Yuan, L., Fu, Z., Shen, J., Xu, L., Shen, J., and Zhu, S.-C. Emergence of pragmatics from referential game between theory of mind agents. arXiv preprint a0X7::2001.07752, 2020.
Zhang, R., Frei, S., and Bartlett, L. Trained trans- formers learn linear models in-context. arXiv preprint arXiv:2306.00 (0l.23306.09977, 2023.
Zhao, R., Song, J., Yuan, Y., Hu, H., Gao, Y., Wu, Y., Sun, Z., and Yang, W. Maximum entropy population-banl bbased training for zero-shot human-ai coordination. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6145-6152. 2023.
Zhao, S., Lu, C., Grosse, R. B., and Foerster, Proximal learning with opponent- learning awareness. In Advances
15

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
A. Related Work
Opponent Modeling (OM). Most prior OM work relies on a fully online learning paradigm, directly interacting with the environment and the opponent's 's policy rather than learning an opponent model from an offline dataset. The existing learning paradigms generally boil down to two stages: (1) Pretraining: pretrain an opponent model with designed OM methodology on training set of opponent policies; (2) Testing: deploying the pretrained opponent model in certain way on testing set of opponent policies to benchmark adaptability to unknown opponents.
Different OM approaches typically have their own distinct focuses:
1. Pretraining- Cetraining-Foocuse Approach (PFA) focuses acquiring knowledge of responding different opponents during pretraining and generalizing it to testing.
2. Testing-Focuud Focused Approach (TFA) centers on updating the pretrained opponent model during testing to reason and respond to unknown opponents effectively.
Methodologically, OM approaches based on Representation Learning (Jaiswal et al., 2020) and Non-meta- Hon-meta-gradien Meta- learning (Duan et al., 2016) typically fall under the category of PFA:
OM approaches based on Representation Learning aim to learn high-quality representations of opponent policies during pretraining to assist in policy optimization (He et al., 201 Hong et al., 2018; Grover et al., 2018; Papoudakis & Albrecht, 2020; Zintgraf et al., 2021; Papoudakis et al., 2021a; Papoudakis & Albrecht, 2020).
OM approaches based on Non-mela- von-mela-gradient Meta-learning attempt to use recurrent architectures to learn the internal structure of each opponent's policy and the differences between them during pretraining (Wang et al., 2016; Zintgraf et al., 2021).
OM approaches based on Bayesian Learning (Bernardo & Smith, 2009), Meta- teta-gradientt-arn Meta-learning (Finn et al., 2017), Shuping Opponents' Learning (Foerster et al., 2018a), Recursive Reasoning (Wen et al., 2019), and Theory of Mind (Rabinowitz et al., 2018) methodologies typically fall under the category of TFA:
OM approaches based on Bayesian Learning detect or infer the opponent policies in real-time using Bayesian methods and subsequently generate responses based on the inferred information (Bard et al., 2013; Rosman et al., 2016; Hernandez-Leal et al., 2016; Zheng et al., 2018; DiGiovanni & Tewari, 2021; Fu et al., 2022; Lv et al., 2023).
OM approaches based on Meta- bett--rrudient-baseed Meta-learning emphasizes leveraging the well-initialized solutions obtained during pretraining in the parameter space to fine-tune and quickly adapt to the test opponents (Al-Shedivat et al., 2018; Kim et al., 2021; Wu et al., 2022b).
OM approaches based on Shaping Opponents' Learning model opponents' updating gradients to estimate the mutual influence between the future opponent policy and the current self-agent's policy (Foerster et al., 201 8a;b; Letcher et al., 2019 Kim et al., 2021; Lu et al., 2022; Willi et al., 2022; Zhao et al., 2022; Fung et al., 2023).
OM approaches based on Recursive Reasoning simulate nested layers of beliefs, predict the opponent's 's behavior, and generate the best response based on the expected reasoning process of the opponent towards the self-agent (Wen et al., 2019; 2021; Dai et al., 2020; Yuan et al., 2020; Yu al., 2022).
OM approaches based on Theory of Mind concentrate on reasoning about the opponent's 's mental states and intentions to predict and adapt to their behavior (Von Der Osten et al., 2017; Rabinowitz et al., 2018; Raileanu et al., 2018; Yang et al., 2019).
Offline Opponent Modeling (OOM) is recently proposed learning paradigm that aims to use offline datasets, without interacting with the environment or opponent agents, to learn an adaptive self-agent policy that dynamically adapts to opponents based on opponent information. Jing et al. (2024a) was the first to propose OOM. first uses representation learning methods to learn embeddings of opponent policies and then employs In-Context Learning to train an adaptive self-agent policy. Jing et al. (2024a) assumes that the offline dataset is optimal, which is challenging assumption to satisfy
16

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
in real-world scenarios. We found that when this assumption is not met, the performance of OOM algorithms deteriorates to unacceptable level. Our work aims to propose general algorithmic improvement framework, TIPR, to relax this assumption and enable OOM algorithms to learn workable adaptive self-agent policies even from highly suboptimal offline datasets.
Previous online OM approaches belonging to PFA can be easily adapted to the OOM learning paradigm, as their methodolo- gies generally do not rely on interactions with the environment or other agents. We adopt the same OOM baselines as Jing et al. (2024a), which are reasonably modified from some representative PFAS. Interestingly, Wu et al. (2024a) recently introduced Constrained Self-Play for offline policy adaptation, which shares a similar objective with OOM. However, the setup of Wu et al. (2024a) is much simpler than OOM: it assumes that interaction with the environment is still possible and that the opponent adopts fixed, known policy during testing rather than unknown non-stationary policies.
In-Context Learning (ICL). Algorithmically, In-Context Learning (ICL) can be considered as taking a more agnostic approach by learning the learning algorithm itself (Duan et al., 2016; Wang et al., 2016; Mishra al., 2018; 8:; Laskin et al., 2023). Recent work investigates why and how supervised pretrained Transformers perform ICL (Garg al., 2022; Li et al., 2023b; Zhang et al., 2023; Ahn et al., 2023; Raventos et al., 2023). Xie et al. (2021) introduces Bayesian framework explaining how ICL works. Some work proves Transformers can implement ICL algorithms via in-context gradient descent (Von Oswald et al., 2023; Akyürek et al., 2023; Bai et al., 2023).
In terms of decision-making, ICL can endow the pretrained model Reinforcement Learning (RL) abilities an in-context way (Wang et al., 2016; Duan et al., 2016; Grigsby et al., 2023; Dorfman et al., 2021; Mitchell et al., 2021; Li et al., 2020; Pong et al., 2022; Laskin et al., 2023; Lee et al., 2023). Wang al. (2016); Duan et al. (2016); Grigsby et al. (2023) focus on the online me setting with the training objective to be the total reward. Furthermore, Dorfman et al. (2021); Mitchell et al. (2021); Li et al. (2020); Pong et al. (2022) focus on offline meta-RI., and their training objectives explicitly handle the distribution shift. Laskin et al. (2023) applies autoregressive supervised learning to distill (sub-sampled) traces of a single-task RL algorithm into a task-agnostic model. Lee et al. (2023) proposes supervised pretraining to empirically and theoretically demonstrate In-Context RL abilities. Lin et al. (2024) further introduce a theoretical analysis framework to explain the principles and working conditions of In-Context RL.
Existing ICL methods in decision-making primarily focus on task adaptation in single-agent settings. Our work is among the few exploratory studies focusing on adapting to non-stationary opponents in multi-agent scenarios. Jing et al. (2024a) was the first to propose the OOM problem setting. It introduced using ICL methods to learn an adaptive self-agent policy, which conditions on opponent information to imitate self-agent policies embedded in the offline dataset. However, the suboptimality of real-world offline datasets significantly reduces the effectiveness of OOM algorithms for the opponent adaptation. To address this issue, we propose an improvement framework based on the suboptimality of OOM algorithms induced by offline datasets. Inspired by the aforementioned ICL works, we train the action-value function Q in an in-confext manner. We found that Q learned this way, compared to unconditional Q. has higher confidence and can better handle the impact of opponent von-stationariiy. Such an ICL-based design choice effectively enhances Q's ability to improve the adaptation performance of various OOM algorithms.
in-context Q to handle
Offiine RL. The mainstream methodology in Offline RL is Offfine Conservative Learning (OCL), which performs policy iteration with the mixed policies embedded in the offline dataset as reference policies. OCL may incorporate soft hard conservative constraints to restrict updates within the trajectory distribution of the offline dataset. In offline single-agent RL, pessimism-driven OCL has been widely demonstrated to effectively reduce extrapolation error (Kumar et al., 2020; Fujimoto et al., 2019; Jin et al., 2021; Xiao et al., 2021; Kostrikov et al., 2021). Some have implemented OCL through statistical constraints (Wu et al., 2019; Peng et al., 2019; Brandfoonrennrr et al., 2021; Fujimoto & Gu, 2021; Nair et al., 2020), such as KL divergence. Other works have employed iamsa methods for conservative policy optimization (Fujimoto et al., 2019; Kostrikov et al., 2021; Ma et al., 2021; Wu et al., 2022a: Xiao et al., 2023). Additionally, some have used ensemble methods to encode the concept of conservatism (Kumar et al., 2019; Ghasemipour al., 2021; Wu et al., 2021: Bai et al., 2021).
In this work, we propose TIPR, which improves OOM algorithms by performing IPR through a truncctdd in-context Q to address the suboptimality induced by offline datasets. Methodologically, TIPR is in sharp contrast to OCI, as TIPR performs policy refinement instantly during testing rather than by optimizing an objective function during offline pretraining like OCL. Additionally, compared to conservatism, TIPR places greater emphasis on policy balancing- pereormiefoogggggeeeeee policy refinement when confidence in value estimation is high while retaining the original policy when confidence is low. Notably, we found that using our proposed Truncated Q for OCL can also effectively improve the original OOM policy to some extent.
17

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
In contrast, using Original Q for OCL not only fails to enhance the original OOM policy but often completely undermines it.
Offline Multi-Agent RL (MARL). Offline MARL has emerged an important research topic on multi-agent system in recent years (Cui & Du, 2022: Tian et al., 2023; Wu et al., 2023; Jiang & Lu, 2023; Formanek et al., 2023; Yang et al., 2021; Pan et al., 2022; Shao et al., 2024; Tseng et al., 2022; Meng et al., 2023; Zhu et al., 2023). This research focuses on developing effective policies solely from pre-existing datasets, without any interaction with multi-agent environments. The offline approach eliminates the need for costly or potentially hazardous online exploration, thereby enhancing the practicality of MARL. for real-world applications.
An vital challenge in offline MARL is to handle the distribution shift between the offline dataset and the true transition dynamics at test time (Jiang & Lu, 2021). This challenge is also referred to extrapolation error, where offline models may encounter out-offdistribution states or actions when deployed online. When opponents adopt policies not represented in the offline dataset, the performance of policies trained exclusively offline can deteriorate substantially. Recent studies have introduced several techniques to mitigate this challenge. Consistent with the ideas of many offline single-agent RL methods (Fujimoto et al., 2019; Kumar et al., 2020), some address extrapolation error through the concept of OCL (Yang et al., 2021; Pan et al., 2022; Shao et al., 2024). For example, Pan et al. (2022) introduced Offline MARL with Actor Rectification that uses zero-order optimization to update the actor based on the offline critic conservatively, There is also increasingly work adapting Transformer architecture for offline MARI.. For instance, Tseng et al. (2022) proposed distilling knowledge from a teacher Transformer model trained with global information into decentralized student to enable effective decentralized execution. Multi-Agent Decision Transformer (Meng et al., 2023) treats MARL as a sequence modeling problem and predicts actions in an autoregressive manner. Recently, there have also been explorations of developing offline MARL frameworks based on diffusion models for centralized training with decentralized execution (Zhu et al., 2023).
Offline MARL and OOM share a common goal of deriving effective policies from offline datasets in aulti-agent environments and applying them in lesting scenarios enhance efficiency and narrow the gap with real-world applications. Nevertheless, most offline MARL approaches focus on purely cooperative settings, while in OOM, we mainly focus competitive games where the environment features opponents. Moreover, OOM, either explicitly or implicitly, must incorporate the modeling of other agents in the environment to achieve udaptability, whereas offline MARL does not place significant emphasis on this requirement. This is because OOM typically assumes more challenging setting where the policies of other agents in the testing environment are unknown and non-stationary.
Transformers for Decision-Making. Growing interest has emerged in utilizing Transformers for decision-making by reframing the problem as sequence modeling. (Yang et al., 2023a; Li al., 2023a). Chen al. (2021) introduced Decision Transformer (DT),, model that predicts action sequences conditioned returns using causal Transformer trained on offline data. DT has opened up a series of research from a new perspective, that decision- making problems can be addressed through Return-Conditionel Supervised Learning (RCSL). Further research has investigated advancements such as enhanced conditioning (Furuta et al., 2022; Paster et al., 2022) and architectural improvements (Villaflor et al., 2022). Another promising avenue leverages the generality and scalability of Transformers for multi-task learning (Lee et al., 2022; Reed et al., 2022). Additionally, Transformers used in meaision-making have shown potential for meta-learning (Melo, 2022).
Interestingly, recent works have introduced methods from offline single-agent RL into RCSL address the issue of suboptimality in offline datasets (Yamagata et al., 2023; Wu et al., 2024b; Wang et al., 2024; Hu et al., 2024; Zhuang et al., 2024). The core ideas of these works can also be summarized as OCL. This work proposes TIPR, which differs from these prior works in the following aspects:
1. Methodological Difference from OCL: Unlike OCL, which performs iterative policy improvement during pretraining through an objective function, TIPR is the first to introduce IPR, conducting instant policy refinement during testing to improve the original policy.
2. Not an RCSL Method: TIPR does not belong to the RCSL class of methods, as Truncated Q is not conditioned on the Retur-TooGo. Brandfonbrener et al. (2022) discussed the issues of specifying conditioning functions in RCSL methods and how unreasonable conditioning functions can lead to arbitrarily poor results.
It is worth noting that most of the previously mentioned ICL works adopt a Transformer architecture, as Transformers have a natural advantage in sequence modeling tasks (Garg et al., 2022; Li et al., 2023b; Zhang et al., 2023; Duan et al., 2016; Grigsby et al., 2023). For instance, Lee al. (2023) proposed a Transformer-basee ICI. approach that outperforms
18

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
behaviors observed in the dataset, both empirically and theoretically, with respect regret, performance measure where DT falls short (Brandfonbrener al., 2022; Yang al., 2023b). Building on these insights, our Truncated Q utilizes causal Transformer architecture to enhance the model's ICL capabilities, particularly within the domain of OM.
B. Detailed Theoretical Analysis
B.1. Explanations of Notations
Table 2. Notations in Thm. 3.1
Notation
Description
h
The random variable of truncated horizon for the Truncated Q function.
The Empirical No Maximization Bias (NMB) probability, ie.,, the proba- bility that the action selected by the Truncated Q function aligns with the action that maximizes the true truncated expected return.
f(h)
A parameter used lower bound the empirical NMB probability with the inequality f(h) 2 1 2no.. where this parameter satis-
Ylo
fies the equation that ( S(nn 271 +1] Ia /4/4 [|]]
+
Ini
Inin
no/
2mm |
02
T.
A term related to the Q function's fitting error over the truncated horizon; its asymptotic complexity is O(h²).
A(h)
(VC) dimension of the function set L == {A(h) < E(o", a, D, Gn:w) I B(h), E 2}. where L(o², a, D. wp) := (Qn a, D:w) - G2)² denotes the loss func- tion for learning Truncated Q function Qn- A(h) is the infimum of E(0², a, D, G)) while Bhh is its supremum. f2 is the space of param- eter W. Notations using sans- fonts all represent random variables.
The
S
T
The size (number of trajectories) of the offline dataset T.
|0
The cardinality of the self-agent's observation space.
|A|
The cardinality the joint action space of all agents in the environment.
The cardinality of the In-Context Data set D, where D the set of all possible In- In-Context Data D.
D
Let 1(02, D) denote the number of samples (o' a, D) in the offline dataset T, then /iin = min,' ..D i(o", a. D) represents the minimum number of samples (o+,a a, D) in the offline dataset T.
Imin
constant satisfies equation that ming',, a _ G,, (o', a a (o a+ a 11, DD) = U, where al* is the self-agent action that maxi- mizes Gh. and a is the self-agent action with the second-highen Gn-
U
B.2. Proof of Thm. 3.1
Theorem 3.1. For any given o² a D. we have f(h) 2 f(h) 1 2no, where no satisfies that
S(n 20 1) In
1 Imin
/ | 1/2
A(h)()
Umin
21 21
The asymptotic complexity of the term (hh) is Ohh).
Proof. First, we define some notation for convenience in the derivation. Let the truncated horizon random variable be denoted as h, and let the neural network for Truncated Q be represented as Qn. with parameters ws and parameter space Q.
19

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Let the random variable for the self-agent's observation be denoted of, the random variable for the joint actions of all agents as a, and the random variable for the In-Context Data D. Let the set of all possible In-Context Data denoted as D {D= (om= am') )= a a) ) E (O-1 x A }}}
Without loss of generality, we use Mean Squared Error (MSE) to define the loss function for learning On as follows:
S(o', a, D, G): = (Qn(o', a D; - Gb)
8
Assuming that the infimum of Eq. (8) A(h) and the supremum of Eq. (8) is Bhh), we define the difference between the supremum and supremum as ((h) (h) A(h). Under general assumption, let the environment reward random variable follow certain distribution with values in [a, b]. Then, we have ah GA S öh. It is straightforward to derive that the asymptotic complexity of (h)) is Ohh)).
Ideally, we aim to find the optimal parameter w that minimizes the following risk functional R(w) within the function set == {A(h) = 2(o", , D, G): & B(h), E (?}.
R(w) = Σ P(o'. a. D.D) ( S(o', a, D, Gnsw)dF(GGlo D) = Σ , a) a, o',a,D
Here, F' represents the cumulative distribution function, and R(o", a, D(w) denotes the per-sample risk functionaa....
However, since the true distribution of (o' a, D) is unknown, we can only apply the Empirical Risk Minimization (ERM)
principle on the pre-collected offline dataset T to find the most suitable w. That is, we seek the parameter w that minimizes the following empirical risk functional Remp (w).
[T' Σ
Remp (w) =
2(0 a()) ((y, Gn(s):w) =
Σ
I(o",
DeRemp a, D(w).
(10)
TT
[T]
.!.a.D
Here, 1(07, a, D) represents the number of samples (o+ D) in the offline dataset T, and Remp a D(w) denotes the per-sample empirical risk functional.
Next, we introduce two lemmas to assist in the proof.
Lemma B.1. (Vapnik, 1998) Assume that {A < ((r; < B, w EΩ) is measurable, bounded real function set whose indicator set satisfies the measurability condition. Let 1 = {Øl inf M(r;w) I Φ S sup (x; and denote the annealed entropy of the indicator set as H020 & Then, the following inequality holds:
weDl P(sup() [ ((1;W)dF(s) T Σ (((():))) > s 4exp((
H9.2 (2[T))
(e - 1/[T))² |T) (B A)2
(11)
Eg. (11) can be equivalently expressed as follows.
For Yw, the following inequality holds with probability at least T7:
R(w)= S Remp (w) + (B (BB-)) (B) H (2 In n/4
(12)
[Ti
Lemma B.2. (Vapnik, 1998) Let the Vapnik- Chervonenkis (VC) dimension of the function set {A < ((xww) s B, w E 2} in Thm. be denoted as S. Then, the following inequality holds:
H 70.0 ([T)) < S(I
+ 1).
(13)
Substituting the previously defined loss function Eq. (8) and sample data (o", a, D) into Thm. B.1 and Thm. B.2, we have:
lio lio D.D)
P(sup max / / 2(0², D. ()))) D) 1(o1, a, D) a..
Σ
S(o a D Gn()) > cf
20

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
< Σ P(sup( WESt G.
2(o", a, D, Gn; Gr:))dF(GNlo a, D) a a, D)
2(0 a()) (()) Gn(s) > e)
(14a)
at D
H (21(o' . D)) (e
(E - 10-1//l"', a. D))2 ,
a, D)}
(14b)
?(ol,a.D)
(A(h))2
T $ 4 $40 1101| |A| (Inn
(e - m/mmmn)
S + 1)
(A(h))² m
(14c)
Here, /iin = m a. ((o' a, D) represents the minimum number of samples (o', clearly holds. Eq. (14b) can be derived from Eq. (11) in Thm. B.1. Eq. (14c) follows from Eq. (13) in Thm. B.2, where represents the VC dimension of IL.
a, D) in the offline dataset T. Eq. (14a)
According to Eq. (12) in Thm. B.2, the above Eq. (14) can be equivalently written as follow.
For Vw, o a, D, the following inequality holds with probability least 1 7p:
5(n
21T1 8
+ 1) - In 1
Rio',an a, DWw Rom a, D() - V
(15)
Amin
min
According to Hoeffding's Inequality (Hoeffding. 1994), we have:
1(o a.D)
/ / a,D
Pf Pmmax(
E(o', a, D, GR: D) 1(o 1(o', a, D)
2(0 a(1): ((), Gn(): c}
i=1
1(o ..D)
Σ P
L(o a. D, G. a, [(o a, D)
(() a() (()), Gn()): > cF
(16a)
aD
Gm
2c21(o', D,
exp(
(16b)
(A(h))²
a.D
2 }.
< 00 | ||||
cxpl- expt-
(16c)
(A(h))
Eq. (16a) and Eq. (16c) clearly hold. The above Eq. (16) can be equivalently written as follow.
For Vo?, a, D, the following incquality holds with probability at least my:
Remp (o', D(w") - , al,"") A(hi)) = In 1n |
(17)
24
As mentioned earlier, in practice, only find w by minimizing (w) (see Eq. (10)). However, strictly speaking, we should find w by optimizing the following objective:
min max (Remp(o' (o a, D(w) Remp a, D|w*) )).
(18)
Without loss of generality, assuming that wt is found using Eq. (18), we can derive that:
max (Remp(O) (0', D)w+) - Remp (ot a, D(w=)) < max (Remp(o', a, D|w*) ) Remp (ommpo',,, alw"") = 0. ..D
(19)
a
Based Eqs. (15), (17) and (19), we can derive the following formula.
The following inequality holds with probability least 1 m
max (R(o', a, D(w+) Rio', a, D(w*)) ..D
21

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
max (R(o', a. D/ emmp (o', a D(w")))
o' ..,D
max (R'm a, D[w+) Remp(o' D[w"))
o' a'D
+ max (Remp (o' , w Rio a, D(w")
(20a)
o' a,
S(In 2 + - 1n
1
nn 0 ||||||||II
$ 1
+
min
mmin
2
+ max (Remp(o' (o", a, D/w Remp (0', D[w")
(20b)
a a.
2 S(n
+ 1) In 1
1
In / O0 |AIIA
<ō: = (1)
(
8
(20c)
Liin
Amin
21
Eq. (20a) clearly holds. Eq. (20b) can be derived from Eq. (15) and Eq. (17). Eq. (20c) follows from Eq. (19).
From Eq. (20), we have:
vo D, , D(w") R(oo a, DIw") S 8.
(21)
According to the Bias- Variance Decomposition (James et al., 2013), we have:
a a, D, Rio", a, D(w) = Ec, 2(o", a, D, Gkow) = (Qn D:w) EE,))² VGh.
(22)
Substituting Eq. (22) into Eq. (21), we obtain:
Yot a, D, a, IEG,)²
-,,,) s 6.
(23)
When the neural network Qn converges, the bias of w", ie., the second term of the left side in Eq. (23), can be considered Therefore, can derive the following formula.
For Vo a, D, the following incquality holds with probability at least 17:
i2(("', a, D EG, I Vo.
(24)
Assume that
min (G aa-"",a-.100 G a U, o
(25)
where 2* is the self-agent action that maximizes Gn, and 2+ is the self-agent action with the second-highest Gn, Observ- ing Eqs. (24) and (25), it is straightforward to derive the following formula.
When 2/8 U, there least a probability of 27jo that Q, selects the action with the highest IGG,, i.e.
f(h) = P(arg max Qn arg max BG,) f(h) 1 27jo
(26)
Within, 7h satisfies that:
S(I 2[T] S
1) In no/ IAAI
1
no 700/00100 |0 D
U2
((h)(1
(27)
mmin
Imin
21
4
This concludes the proof.
B.3. Proof and Intuitive Analysis Prop. 3.2
Propusition 3.2. For any given a,,aa,1 ",,, D. there exists an optimal truncated horizon h" E [T] that maximizes ((h)g(h), i.c. the lower bound of ((h). Specifically, there exist functions such that h" #
22

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Proof. To ensure that Truncated Q selects the action with the highest EGT with maximum probability during policy refinement, need to find the optimal h (7] that maximizes the Overall No Maximization Bias (NMB) Probability y/h)..
However, directly maximizing y(h) is intractable, and we need to perform further transformations. Based on Eq. (5), we can maximize its lower bound (h)) For any given o², a a D.. this objective can be written as:
max ((h)g(h). h
(28)
Here, the Empirical Risk NMB Probability f(h) is determined by the neural network's fitting process on the dataset T, while the Natural NMB Probability g(h) is dictated by the intrinsic properties of the environment's 's reward structure.
Eq. (28) is still intractable, requiring further transformation. According to Thm. 3.1, for any given o', a f()) has tight lower bound denoted as f(h) = 1 2nj. The variation of f(h) with respect to h can approximately characterize the variation of (h). Thus, we can optimize the following objective as an approximation to optimize Eq. (28):
max ((h)
(29)
As h increases, A (h)) also monotonically increases, leading to a monotonic increase in Tuo which in turn causes f(h) to monotonically decrease. Therefore, f(h) is strictly monotonically decreasing function with respect to h.
Since g(h) is determined by the environment, we cannot explicitly formulate it. However, upon observation, it is easy to conclude that ghh) satisfies certain properties:
When h = 1, gh) attains its minimum value, denoted as Jmin-
When h = T, g() attains its maximum value of 1.
When 1 < h < T, we have Jmin S g(h) S 1.
can be observed that as h increases, the function g(h) generally exhibits an upward trend. Based on these properties, can be derived that there exists an h" [T] that satisfies Eq. (29).
Specifically, there exist functions g such that h" + T. Below, we provide some examples:
When g(h) 1, it clearly holds that h" 1.
There exist strictly monotonically increasing functions g(h) such that Jmin < g(h") < 1 and h* <.
This concludes the proof.
Intuitive Analysis. For better intuitive understanding, we further transform Eq. (29) for analysis. It known that for any given o², a1, D. the following holds:
yhh) 2 ((h)
(30)
Therefore, is straightforward to derive that:
(hy (- In f(h)) (- In gh)).
(31)
Maximizing yhh can be achieved by optimizing the following objective:
min( In f(h)) + g-)n
(32)
Fig. 7 presents schematic illustration of the variation in NMB probability h changes when g is certain strictly monotonically increasing function. In the case shown in Fig. 7, the optimal h needs to balance the trade-off between Iu f(h) and - In gh),, thereby approximately minimizing - In y(h), which in turn maximizes y(h). It can be intuitively observed that the optimal value of is not necessarily always T.
With the aid of Fig. 7, we can further intuitively analyze the benefits of IPR making refinement decisions based on value confidence Qc rather than unthinkingly refining the original policy in all cases. In the practical implementation of Truncated
23

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Schematic of h vs. NMB Probabilities
Negative Log of NMB Probability
0.4
0.3
gh))
([))
0.2
(n)
0.1
=
0.0
10
20
30
40
50
Truncated Horizon (h)
Figure 7. Illustration of the variation in NMB probability as h changes when g is a certain strictly monotonically increasing function.
Q. we empirically set a truncated horizon H. However, the true h* could potentially be a much larger value. In this scenario, the y( () values for many given samples (o* a=1,,,, D) may be very small, resulting in a low probability of selecting the action with the highest E after policy refinement. At this point, rather than using Truncated Q for refinement, it would be better to retain the original policy. Building on this reasoning, IPR uses value confidence Qc to balance the decision of whether to perform policy refinement.
24

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
(a) MS
(b) LEF
(c) PD
(d) PP
Figure 8. The environmental benchmarks.
C. Detailed Introductions of the Environments
Markov Soccer (MS) MS (Littman, 1994; Lanctot et al., 2019) is a sparse- two-player zero-sum game with a discrete state space. The game is played 4 grid in Fig. 8(a). The game begins with the self-agent (blue) and the opponent (red) in random squares in the left and right half (except the goals), respectively, and the ball goes to one of them randomly or none of them to random square in the middle column. The players have five actions: move N, S, W, E, or stand still (i.e., NO-OP), An action is considered invalid if it leads the player to shaded square (grey) outside the border. a player has the ball, possession will be exchanged and transition will not take place when the two players move into the same square. Both the self-agent's 's and the opponent's ss objectives are taking the ball into the other's 's goal (the self-agent's goal is blue, and the opponent's 's goal is red), and the game ends when this happens. The reward sparse, providing +10 for success, 10 for failure, and 0 for instances where the maximum timestep limit is exceeded, only at the end of the games. For specific implementation of this environment, we adopt the open-source code of OpenSpiel, which is available https: ttt.bb..tttttttggg com/ coom/deepminte nd/ open_spenp_pp spee_.
Level-Based Foraging (LBF) LBF (Christianos et al., 2020; Papoudakis et al., 2021b) is a sparse-reward two- player mixed-incentene incentiveeniveeeee game with a discrete state space, as shown in Fig. 8(b). The game is played on a 9x9 grid with the self-agent (in blue) and the opponent (in black), along with five apples (in red). At the beginning of each episode, the two players and the five apples are randomly generated in the environment and assigned level marked in their bottom-right corner. The goal of the self-agent is cat many apples possible. All players can move in four directions cat an apple. Eating an apple can be successfully done only under the following conditions: one or two players are around the apple, and all players who take the action of eating an apple have summed level at least equal to the level of the apple. The environment has sparse rewards, representing the players' contributions to eating all the apples in the environment. The environment is essentially long-term social dilemma and can be viewed as an extension the Prisoner's Dilemma (Axelrod, 1980). The challenge in this environment is that the self-agent must learn to cooperate to eat high-level apples while greedily eating low-level apples simultaneously. For specific implementation of this environment, adopt the open-source code of lb-foraging, which is available at ht tttpps:/ g thub. com/ semi table / b- foraging.
Physical Deception (PD) PD (Lowe et al., 2017) is a sewrse-reward three-player non-zero-sum game with a continuous state space. PD includes the self-agent (red), two opponents (purple), one normal landmark (black), and one target landmark (green) shown in Fig. 8(c). All agents observe position of landmarks and other agents. There are no borders present on the entire game map. The self-agent's objective is to hit the target landmark which is unknown to him (i.e.. the self-agent does not know which one of the landmarks is the target one) as many times as possible. The opponents' objective is minimize the times that the self-agent hits the target landmark while maximizing the times that they themselves hit the target landmark. Therefore, the opponents need to cleverly deceive the self-agent, while the self-agent must astutely detect the opponents' deception. When the self-agent hits the target landmark, it receives sparse reward of 10, while the opponents receive -10. When the opponents hit the target landmark, they receive a sparse reward of 10. For specific implementation of this environment, we adopt the open-source code of Multi-Agent Particle Environment, which available at https: ///g/// //gitt github.. com/ opwnai/////////ooaoo /wwlttmtaaaaa imultiagent www.annnttpppac -part ccle- envs. Additionally, we introduce a collision mechanism to the original environment to increase the randomness and dynamics of the environment.
25

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
Predator Prey (PP) (Lowe al., 2017) sparse-reward four-player non-zero-sum game with continuous state space, as shown in Fig. 8(d). The environment consists of three predators (in red), one prey (in green), and two obstacles (in black). The goal of the predators is to capture (i.e., collide with) the prey much as possible, while the goal of the prey is to be captured as little as possible. The environment features sparse rewards, where each time a predator captures the prey, the capturing predator receives reward of 10, and the prey receives a reward of - 10. Additionally, the environment provides a very small, dense reward to the prey to prevent it from running out of the map boundaries. Here, the prey is the self-agent, and the three predators serve as the opponents. From the perspective of self-agent, the environment highly unstable, there are three opponents with unknown policies in the environment. The challenge in this environment that the self-agent must model the behavior of three opponents simultaneously and adapt to various potential coordination strategies employed by the opponents (e.g., surrounding from three different directions). For specific implementation of this environment, we adopt the open-source code of Mulli-Agent Particle Environment, which is available at ht https::/ /gittubb com/ comn/opeaiiiic /mmltiagent part icle- envs.
D. Diversity of Opponent Policies
MS
LBF
1:25
1.25
1:21
0.00
0.76
2.12
0.91
3.30
3:13
0.84
1.62
0.00
8
3.15
1.22
1.22
0.00
1.21
0.84
1.92
0.82
3.81
2.05
0.00
1.43
0.84
Opponent Policy
m-3.
1:15
1:46
1:33
1:38
2.01
1.06
0.00
1.35
1.39
2:01
1.39
Opporent Policy
1.22
2.02
1.09
2.51
3.00
3.39
3.59
0.00
2.37
0.90
2.63
3.22
1.28
2.63
2.33
1.47
1.28
0.00
1.52
2.53
1.32
2.53
1.19
1.33
1.60
1.34
1.36
0.97
1.43
0.00
3.20
2:24
2.56
2.23
1.16
1.60
1-2
1.19
0.00
1.22
1.25
0.87
4.79
4.08
1.10
0.00
2.45
3.50
4.69
0.83
4.52
1.68
0.87
1.17
1.35
1.31
1.34
1.31
2.05
1.38
1.29
0.00
2.47
1.03
1.36
2.49
2.37
2.15
2.03
2.61
1.39
1.24
0.00
1.32
2.34
1.4
3.00
2.34
1.86
1.42
1.46
3.00
1.45
1.30
0.00
1.05
1.47
1.11
1.32
3.53
2.03
2.24
2.00
1.63
1:30
m
Opponent Policy
Opponent Policy
PD
PP
1.67
1.70
1.73
0.00
0.89
0.97
3.13
0.00
1.11
1.10
1.66
1.18
3.50
2.54
4.14
1.53
6.03
0.42
2.09
2.74
0.00
6.42
1.50
1.78
1.72
0.00
1.78
1.28
0.99
0.00
1.20
Opponent Palicy
Opponent Policy
2-
1.39
8.66
1.85
0.00
2.65
0.89
1.45
1.20
2.19
2.82
1.91
1.29
0.00
1.97
2.19
1.39
2.49
1.31
1.20
1.44
4.64
4.69
1.80
1.47
0.00
3.04
0.32
2.64
0.64
1.04
1.44
1.25
1.88
0.00
1.70
1.3
1.68
1.70
1.08
1.41
1.98
1.70
1.25
3.50
1.57
0.00
4.00
1.54
3.80
4.23
1.66
3.75
3.57
6.23
3.50
1-2
1.87
3.
1.50
4.17
1.54
4.02
4.46
2.09
2.75
0.89
1.50
7
m
Opponent Policy
Opponent Policy
Figure Pair-wise expected KL divergence opponent policies used construct offline datasets
As mentioned Sec. 4.1, we run the Maximum Entropy Population- based training algorithm (MEP) (Zhao et al., 2023) to
26

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
generate a diversified opponent policy population. Nevertheless, quantitative analysis is still necessary measure the similarityddssimiaarty between different opponent policies used construct the offine datasets T.
From the MEP population, 12 policies are selected as pretraining opponents (Seen Set), and 8 policies are selected as testing opponents in the Unseen Set. We calculate the pair-wise expected KL divergence between different pretraining opponents to measure their dissimilarity. The results for MS, LBF, PD, and PP are shown in Fig. 9, respectively.
For any given policies ग -1. and x 1. E Seen Set, we estimate the expected KL divergence between them by:
Σ π --aaaa 1a- 1o log π T (a (a-100) π I- (a-
(33)
LE--
Here, P)o- denotes the opponent observation distribution. Ideally, P(o- should cover the entire opponent observation 1 in even slightly large
space 0 However, in practical situations, covering the entire opponent observation space O-1 environments can be intractable.
To maximize the coverage of the opponent observation space by P(o- we employ the following sampling method: Within the Seen Set, there are total of 12 opponent policies. For each opponent policy π -'A we sample 1200 episodes. In these 1200 episodes, the opponents' policy are fixed to x -1, while the self-agent traverses through all the opponent policies, resulting in the self-agent using per opponent policy for 100 episodes.
In Fig. 9, the lighter the color in the heatmap. the higher the KL divergence value, indicating a lower similarity between the two policies. Assuming dissimilarity threshold of 1.0 (i.e., two policies are dissimilar if their expected KL divergence greater than 1.0), over 80% of the expected KL divergence values in all environments exceed this threshold. This indicates that the pretraining opponent policies are generally well-distinguished from one another. The dissimilarity rates for MS, LBF, PD, and PP are 92.35%, 80.56%. 91.67%, and 85.42%, respectively. Overall, we can consider the MEP opponent policies we select to construct the offline datasets T to be adequately diverse.
E. Neural Architecture Designs
E.1. Neural Architectures of Truncated Q
For the horizon- truncated in-context aation-value function we proposed, i.e., Truncated Q. we adopt the neural architecture design as follows:
Backbone: The backhone of the Truncated Q is mainly implemented based on the causal Transformer, i.e., GPT2 (Rad- ford et al., 2019) model of Hugging Face (Wolf et al., 2020). Both the Encoder and Decoder of the Truncated Q's neural architecture adopt this backbone. The backbone is a GPT2 model composed of 3 self-attention blocks. Each self-attention block consists of a single-head attention layer and a feed-forward layer. The Decoder's self-attention block includes an additional angle-head cross-attentin layer. Residual connections (He et al., 2016b) and LayerNorm (Ba et al., 2016) utilized after each layer in the self-attention block. Within each attention layer, dropout (Srivastava et al., 2014) is added to the residual connection and attention weight. In the backbone, the feed-forward layer consists of fully connected layer that increases the number of hidden layer nodes and projection layer that recovers the number of hidden layer nodes. Except for the fully connected layer in the feed-forward layer, which consists of 128 nodes with GELU (Hendrycks & Gimpel, 2016) activation functions, the other hidden layers are composed of 32 nodes without activation functions.
observations -1 and actions a are fed into modality linear layers, and a positional episodic timestep encoding is added. We adopt the same timestep encoding as in (Chen et al., 2021). Then, we use fusion linear layer to fuse the 0-1 a_ embedding tokens at each timestep into fused embedding tokens. Finally, the sequences of fused embedding tokens are fed into the backbone, which autoregressively outputs the per-timestep embedding tokens -1 corresponding to each (o a 2) tuple using a causal self-attention mask. The per-timestep embedding tokens -1 are inputed as key and vatue into the cross-attention layers of the Decoder. The fusion linear layer comprise 32 nodes without activation functions. The modality-specific linear layers for the opponents' observations o and actions a comprise 32 nodes with ELU (Clevert et al., 2015) activation functions.
Encoder:
The opponents'
Decoder: Self-agent observations o¹. opponents actions a self-agent actions a', and self-agent rewards r' are fed into modality-speciin linear layers. positional episodic timestep encoding is added. We adopt the same timestep
27

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
encoding in Chen et al. (2021). In addition, an agent index encoding is added each token to distinguish the inputs from different agents. The sequences of embedded tokens fed into the cross- attention layers of the backbone as query, which autoregressively outputs the action-value predictions Q at the positions of self-agent actions at tokens using a causal self- atlf-attentoon mask. The Opponent Imitator Toi of the Decoder autoregressively predicts opponents actions a at the positions of self-agent observations o tokens. The modality-specific linear layers for the self-agent observations o², opponents actions a a self-agent actions a', and self-agent rewards r comprise 32 nodes without activation functions. All the output heads use linear layers. The output head of Qc uses Sigmoid as activation functions.
Input & Output: Given timestep t, the input sequences for the Encoder and the Decoder are (o , ,ai "q''...,07 arr) and (oi-1-1: L+1 L+1 torical trajectory sequence length for the Decoder, as Transformer model has a token capacity. The output prediction
9--+:::7 "it--+:......- 9-11, a ob), respectively. Within, L is the in-episode his-
sequence is (Tou(--L+1)> Qcie- L+):: L- TT(())) Pc(o) Qveo)) During training, the corresponding output label sequence is (4,-1+1: (1) + 0f 141 (rd) ( ................................ or (1(r) + 03 ( Ard (+b)))=))))
L+H
E.2. Neural Architectures of OOM Baselines
For all the OOM baselines, we strictly follow the neural architecture designs adopted in Jing et al. (2024a):
DRON-coneat:
We use the GPT2 model mentioned in Sec. E.1 to implement the backbones of DRON-concat. DRON- concat's neural architecture contains 2 backbones, and the backbones' architectural design is identical to that of Truncated Q's Decoder, except that it does not include the cross-attention layer. One of the backbones is used to encode the sequence consisting of RTGs G observations o², and actions at, while the other backbone is used to encode the opponent's hand-crafted features (He al., 2016a). To be versatile in different environments, we consider using the opponent's 's previous action and the opponent's previous action frequency as the hand-crafted features. We concatenate the embedding tokens obtained by the two backbone encodings in the dimension of the hidden state and feed them to fusion linear layer to get concatenated embedding tokens. Within, the hand-crafted feature embedding tokens are aligned with the positions of a² embedding tokens, and other positions are filled with 0 and aligned with the positions of G', a² embedding tokens. Finally, given the concatenated embedding tokens, the actions a, are autoregressively predicted at the locations of observations of. Both the fusion linear layer and the linear layer of hand-crafted features comprise 32 nodes without activation functions, while the number of other hidden layer nodes and the activation function settings are identical to those of Truncated Q's Decoder. All the output heads use linear layers.
We use the GPT2 model mentioned in Sec. E. to implement the backbones of DRON-MoE. DRON- MoE's neural architecture contains 2 backbones, and the backbones' architectural design is identical to that of Truncated Q's Decoder, except that it does not include the cross-attention layer. One backbone is used encode the sequence consisting of RTGs G¹, observations o², and actions a', while the other backbone is used to encode G', , al sequence and the opponent's hand-crafted features. For the first backbone: After encoding G.,, of. ag. the of embedding tokens are fed to an expert linear layer and output a predefined number of expert of action probability distributions. For the second backbone: Before inputting, the opponent's 'ss observations o² according to the timestep, and they are fed to a mixing linear layer; after encoding o/, of and hand-crafted features, o embedding tokens are fed to an expert linear layer, and weight vector with a length of the number of experts is outputted. Finally, the obtained weight vector is used to perform a weighted summation of the obtained action probability distributions, and the actions af are predicted autoregressively. The mixing linear layer consists of 32 nodes without activation function, and the expert linear layer consists of 32 nodes with Softmax (Bridle, 1989) activation function. The number of other hidden layer nodes and the activation function settings are identical to Truncated Q's Decoder. All the output heads use linear layers. The predefined number of experts is set to 5.
DRON-MoE:
features are concatenated with the corresponding
We use the GPT2 model mentioned in Sec. E. to implement the backbones of LIAM. The architectural design of LIAM's backbone is the same as that of Truncated Q's Decoder, except that it does not include a cross-attentin layer. In addition to feeding the sequence consisting of RTGs G², observations o', and actions a into the backbone and predicting the action al in an autoregressive manner using a causal self-attention mask, we also employ an extra decoder to learn an auxiliary task (Papoudakis et al., 2021a). This auxiliary task involves reconstructing the opponent's s observations o, and actions a," using the observations of and actions af_1 of the self-agent. Specifically, we use the of token embeddings obtained through the backbone as input to the extra decoder to predict of and ar as the of token embeddings contain all the information of of and a!_ The extra decoder consists of two linear layers with 32
LIAM:
28

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
nodes and activation functions. The number of other hidden layer nodes and the activation function settings identical to Truncated Q's Decoder. All the output heads linear layers.
Prompt-DT: We implement Prompt-DT's neural architecture directly based on its open-source code, which is available We use high-quality data against opponent policies from offline datasets as expert demonstrations (i.e., prompts). High-quality data refers trajectories of the self-agent that rank near the top in terms of return across all episodes in which it competes against specifie opponent poliey. Specifically, select the top 20% for the high-quality data, sample 3 trajectory, and then sample consecutive fragments with a length of 5 from the trajectory as prompts. The linear layers for RTGs, observations, and actions in the prompts consist of 32 nodes without activation functions. The number of other hidden layer nodes and the activation function settings are identical to Truncated Q's Decoder. All the output heads use linear layers.
at https:: //gtthub...
TAO: We implement TAO's neural architecture directly based on its open-source code, which is available at (TAO Code]. ]. (1) TAO's Encoder: We implement TAO's encoder using the exact same backhone the Encoder of Truncated Q. The opponents' actions a rewards r r- and observations o-1 are fed into modality linear layers, and a positional episodic timestep encoding is added. We use a fusion linear layer to fuse the r,-1, oF embedding tokens at each timestep into fused embedding tokens. The sequences of fused embedding tokens are fed into the backbone, which autoregressively outputs the per-tinestep embedding tokens corresponding each (a, rr- o, tuple using causal self-attention mask. Then, input all the per-timestep embedding tokens output by the backbone, as key and value into the cross-attentien layers of TAO's decoder. The modality-specifft linear layers for the opponents' actions, rewards, and observations are composed of 32 nodes with ELU (Clevert et al., 2015) activation functions. (2) TAO's 's Decoder: We implement TAO's decoder using the exact same backbone as the Decoder of Truncated Q. The self-agent's s RTGs G',, observations o², and actions al are fed into modality-specifft linear layers, and a positional episodic timestep encoding is added. The sequences consisting embedding tokens of G. of at are fed into the cross- atssntttentoon layers of the backbone as query. autoregressively predicts actions a at the positions of of embedding tokens using causal self-attention mask. The modality-specinf linear layers for the self-agent' 's RTGs, observations, and actions are composed of 32 nodes without activation functions. All the output heads use linear layers.
29

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
F. Hyperparameters
F.1 Hyperparameters for Opponent Policies & Offline Datasets
Hyperparameter Name
MS
LBF
PD
PP
Dimensionality of observations
12
10
16
Dimensionality of actions
5
5
Horizon for each episode (T)
50
50
100
100
Agent index of the self-agent
0
0
Agent indexes of the opponents
1
1,2
0,1,2
Size of MEP population
4
4
Weighting coefficient for MEP's diversity objective
-
le - 2
le -
le -
Total number of episodes used for training MEP population
1e5
1e5
1e5
165
Total number of episodes used for training self-agent policies (these policies are used construct offline datasets T, the training objective is achieve the BR against the opponent)
1c4
1e4
1c4
1e4
Number of updating epochs at each MEP's and self-agent policies' training step
10
10
10
10
Batch size training MEP population and self-agent policies
4096
4096
4096
4096
Learning rate for training MEP population
5e
5e
4
5e
5e 4
Learning rate for training self-agent policies
8e
8e
5
8e
8e 5
Maximum norm of the gradients for training MEP population and self-agent policies
5.0
5.0
5.0
5.0
Clipping factor of PPO (Schulman et al., 2017) used for train- ing MEP population and self-agent policies
0.2
0.2
0.2
0.2
Number of opponent policies used construct the offline
12
12
12
12
dataset T (K)
Number of trajectories used construct the T for each opponent policy n
1e3
1e3
1e3
1e3
30

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
F.2. Hyperparameters for OOM Basclines Pretraining
Hyperparameter Name
MS
LBF
PD
PP
In-episode historical trajectory sequence length for the backbone (see Sec. E.2 for detailed descriptions)
20
20
20
20
Sequence length of the In-Context Data D (M)
15
15
15
15
Number of trajectories sampled to constreut D (C)
3
3
3
3
Total number of training steps
3c3
3e3
3c3
3e3
Number of updating epochs at each training step
10
10
10
Batch size
20
20
20
20
Warm-up epochs (the learning is multiplied by num_spocht+1 to allow it to increase linearly during the warm-up epochs
le4
le4
1e4
le4
initial warm-up epochs of training)
Reward scaling factor (all the rewards are multiplied
100
100
by roward scaling factor to reduce the variance of training)
Learning rate for AdamW (Loshchilov & Hutter, 2018) optimizer
le - 4
le 4
1e -
le - 4
Weight decay coefficient for AdamW optimizer
le
le 4
1e 4
1E
Maximum norm of the gradients (clip if exceeded)
0.5
0.5
0.5
0.5
Dropout factor for the backbone (GPT2 model)
0.1
0.1
0.1
0.1
Random seeds
0, 0,1,2,3,4
0, 0,1,2,3,4
0, 0,1,2,3,4
0, 0,1,2,3,4
F33 Hyperparameters for Truncated Training
Hyperparameter Name
MS
LBF
PD
PP
In-episode historical trajectory sequence length for the Decoder (I) (see Sec. E. for detailed descriptions)
20
20
20
Sequence length of the In-Context Data D (M)
15
15
15
15
Number of trajectories sampled to constreut D (C)
3
3
3
3
Total number of training steps
le4
1e4
1e4
le4
Number of updating epochs at each training step
10
10
10
10
Batch size
20
20
20
Weighting coefficient for confidence loss Lqe (a)
1.0
1.0
1.0
1.0
Weighting coefficient for value loss Lov (3)
1.0
1.0
1.0
1.0
Warm-up epochs
le4
le4
le4
le4
Learning rate for AdamW optimizer
5e -
5e 3
5e 3
5e 3
Weight decay coefficient for Adam optimizer
1e
le-4
1e-4
le 4
Maximum norm the gradients (clip if exceeded)
0.5
0.5
0.5
Dropout factor for the backbone (GPT2 model)
0.1
0.1
0.1
0.1
Random seeds
0, 0,1,2,3,4 0,1,2,3,4
0,
0, 1,2,3,4
31

---

Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement
F.4. Hyperparameters for Instant Policy Refinement
Hyperparameter Name
MS
LBF
PD
PP
Truncated horizon (H)
3
Total number of episodes for testing
2400
2400
2400
2400
Number of episodes between each opponent's
20
20
20
20
non-stationary policy switching
Sequence length of the In-Context Data D (M)
15
15
15
15
Number of historical trajectories of the opponent sampled construct (C)
3
Random seeds
0, 0,,,3,3,4
2,
0, 2,3,4 4
0, 0,,,,,3,4
32


Reviews:
[{"review_id": "PP9eodaFcq", "reviewer": "Reviewer_Chog", "summary": "This paper introduces a framework called Truncated Q-driven Instant Policy Refinement (TIPR) to improve Offline Opponent Modeling (OOM) algorithms trained on suboptimal datasets where the self-agent may not always select best-response (BR) actions to its opponents. Unlike prior OOM methods that assume optimal trajectories, TIPR addresses the reality that self-agent policies in offline datasets can be arbitrarily poor. The framework introduces Truncated Q, a horizon-truncated in-context action-value function that estimates returns over a fixed truncated horizon rather than the full episode, reducing estimation complexity and improving reliability. At test time, Instant Policy Refinement (IPR) dynamically refines the self-agent policy by using Truncated Q’s confidence estimates to decide whether policy updates are necessary. Theoretical analysis suggests that Truncated Q optimizes the No Maximization Bias (NMB) probability and there exist optimal truncated horizon lengths which can stabilize training along with ensuring effectiveness of Q-guided action selection. Empirical results across four competitive multi-agent environments show that TIPR consistently improves various OOM algorithms, even with highly suboptimal datasets. TIPR also outperforms conventional offline RL methods like Offline Conservative Learning (OCL) which struggle with distributional shifts at test time.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We'd like to clarify a few points in response, hoping our explanations address your concerns and strengthen our work.\n\n---\n\n> **(A) Theoretical Claims: …** the empirical risk bound in eq-4 assumes a specific distribution of Q-learning errors, which depends on well-behaved rewards and opponent policies. If the environment dynamics cause non-monotonic error growth …\n> \n\nOur theory (Thm. 3.1 and Prop. 3.2) fundamentally proves that there exists a trade-off between the truncated horizon $h$ and the No Maximization Bias probability. Specifically, the theorem reveals a structured relationship showing that extending the $h$ increases the complexity of accurately estimating returns, reducing the accuracy of Q-values in selecting optimal actions. Thus, using Truncated Q is more sound and effective than using the Original Q for policy refinement.\n\nImportantly, we clarify that **our theory does not explicitly assume or depend on any particular structure or specific distributions of Q-learning errors, nor does it require particular properties or assumptions regarding opponent policies**. The derived bounds in Eq. (4) are general and are based purely on standard statistical inequalities and complexity terms that arise naturally from the estimation problem itself.\n\nWe acknowledge that some environment dynamics may cause non-monotonic error growth. However, our theory doesn't specifically assume monotonicity or well-behaved rewards or opponents. Future work could explore more detailed analyses of environments with complex long-term strategic dependencies.\n\n> **(B) Theoretical Claims:** Further, the assumption that opponent modeling via in-context data is always reliable could be a strong assumption …\n> \n\nWe clarify that assuming reliable Opponent Modeling (OM) via *In-Context Data* (ICD) is not overly strong. TAO (Jing et al., 2024a) has theoretically and empirically proved that using ICD for OM is both feasible and effective, even with non-stationary opponents. Building on this evidence, our Truncated Q uses ICD to enhance OM reliability and performance.\n\n> **(C) Claims And Evidence:** The claim would be stronger if supported by an adaptive mechanism for choosing $h$ rather than manually testing different values (as in Figure 5).\n> \n\nFor possible automated mechanisms of choosing $h$, please refer to our response to reviewer **RP82**'s comment \"**(F) Questions For Authors:** 2. How to find\". You can use Ctrl + F to search for the sentence in quotation marks to quickly jump to there.\n\n> **(D) Experimental Designs Or Analyses:** … However, if the offline dataset lacks sufficient opponent diversity …\n> \n\nWe use MEP algorithm to generate opponent policies. From the MEP population, 12 policies were selected as training opponents (Seen Set), and 8 policies were selected as test opponents in the Unseen Set. **A quantitative analysis of the diversity among the 12 training opponent policies** is provided at [this link](https://ibb.co/XZnpC3cm).\n\nTo measure their distinctions, we use the **Pair-wise Expected KL Divergence** (PEKLD). If we take 1.0 as a threshold, over 80% of the PEKLD values in all environments exceed this threshold. This indicates that the training opponent policies are generally well-distinguished from one another.\n\n> **(E) Essential References Not Discussed:** … the problem formulation in [1] focuses on a similar setting from the perspective of principal-agent …\n> \n\n[1] is an interesting related work. We will cite [1] and discuss its connections with our work in our revision.\n\n> **(F) Other Strengths And Weaknesses:** 1 … Testing TIPR on more complex strategic settings …\n> \n\nFor experiment results on more challenging settings, please refer to our response to reviewer **LtaJ**'s comment \"**(B) Weaknesses:** 2 … the scalability of TIPR\".\n\n> **(G) Other Strengths And Weaknesses:** 2. The computational efficiency of TIPR is not discussed …\n> \n\nCompared to the original OOM algorithm, **TIPR only requires one additional forward pass using the Truncated Q at each timestep, which can be completed very quickly (on the order of milliseconds to seconds).**\n\nMany other policy refinement methods, such as decision-time planning (e.g., Monte Carlo Tree Search), perform a large number of forward passes to rollout numerous simulated trajectories to refine the policy. In contrast to them, the computational cost of TIPR is negligible.\n\n> **(H) Other Comments Or Suggestions:** … introduces the variable $M$ but has it been defined somewhere in the paper?\n> \n\n$M$ denotes the number of $(o^{-1}, a^{-1})$ tuples in the ICD $D$. This value determines how much information is used to characterize the opponent's policy and is typically set as a predefined constant. We will include this in our revision.\n\n---\n\nYour comments have greatly helped improve our paper. If you have further feedback, we're happy to address it. If your concerns are resolved, we hope you'll reconsider your rating.", "claims_and_evidence": "Claim: Suboptimal datasets degrade OOM performance\n- The paper presents experimental results across four competitive environments showing that existing OOM algorithms perform poorly when trained on suboptimal datasets. This supports the claim that suboptimality is a significant issue.\n\nClaim: Truncated Q is more reliable than Original Q\n- Theoretical analysis explains why truncating the Q function reduces estimation complexity and improves learning stability.\n- Experimental results (Figure 1, Table 1) show that Truncated Q has significantly lower Mean Squared Error (MSE) than Original Q, supporting the claim that it provides more reliable value estimates.\n\nClaim: TIPR consistently improves OOM algorithms across varying dataset optimality levels\n- Performance improvements in Figure 3 demonstrate that TIPR enhances multiple OOM algorithms, even with heavily suboptimal datasets.\n- The paper includes a thorough ablation study, showing that removing key components (confidence estimation, in-context opponent modeling) reduces performance.\n\nClaim: IPR is more effective than OCL for OOM improvement\n- Empirical results (Figure 4) show that IPR outperforms OCL across all environments.\n- The explanation that OCL struggles due to distributional shifts is reasonable and aligns with existing challenges in offline RL.\n\nClaim: Theoretical justification of Truncated Q maximizing the No Maximization Bias (NMB) probability\n- The analysis in Section 3.3 argues that Truncated Q is superior due to better balancing between empirical risk and natural NMB probability.\n- However, the derivation relies on assumptions about the structure of Q-learning errors and the properties of the opponent policies, which may not hold universally.\n- Empirical validation supports the effectiveness of Truncated Q, but additional experiments explicitly comparing different horizon truncation strategies could strengthen this claim.\n\nClaim: TIPR maintains stable improvements across all degrees of suboptimality\n- While the results show consistent performance gains, some environments (PP) exhibit high variance in TIPR's impact (Figure 3). More analysis on why TIPR works better in some settings than others would improve clarity.\n\nClaim: Optimal truncated horizon $h^*$ exists for every environment\n- Proposition 3.2 suggests that there is an optimal $h^*$, but finding it is non-trivial. This paper does not provide a method to determine $h$ optimally and instead treats it as a hyperparameter. The claim would be stronger if supported by an adaptive mechanism for choosing $h$ rather than manually testing different values (as in Figure 5).\n\nOverall, the core claims (that TIPR improves OOM algorithms, that Truncated Q is better than Original Q, and that IPR is more effective than OCL) are well-supported by both theory and experiments. However, some theoretical claims (like No Maximization Bias probability, optimal truncation horizon) rely on assumptions that may not always hold and would benefit from additional empirical validation.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I have briefly looked at the proof in the appendix. Reiterating the comment made in the \"Claims and Evidence\" section:\n\nThe proof relies on a structured relationship between truncation horizon and error growth, which may not generalize across all settings. Specifically, the empirical risk bound in eq-4 assumes a specific distribution of Q-learning errors, which depends on well-behaved rewards and opponent policies. If the environment dynamics cause non-monotonic error growth (for example due to long-term dependencies in strategic games), the proposed bounds might not accurately reflect reality. Further, the assumption that opponent modeling via in-context data is always reliable could be a strong assumption, especially since some environments may have non-stationary opponents whose strategies evolve in unseen ways.", "experimental_designs_or_analyses": "Yes, I checked the entire experiments section in the paper. \n\nThis work assumes that the opponent policies seen during training provide enough diversity such that Truncated Q generalizes well to new opponents. However, if the offline dataset lacks sufficient opponent diversity, then the confidence and value estimates of Truncated Q might be biased. This issue is hinted at in Table 1, where accuracy for the confidence estimates $Q_C$ drops in environments with multiple opponents (PP for example). The assumption that an in-context model can fully capture opponent dynamics may not hold if the training distribution is too narrow.", "relation_to_broader_scientific_literature": "Opponent modeling has traditionally relied on online learning, with early works like Deep Reinforcement Opponent Network (DRON) (He et al., 2016) and Latent Interaction-based Opponent Modeling (LIAM) (Papoudakis et al., 2021) focusing on learning opponent representations. More recent efforts, such as TAO (Jing et al., 2024), leveraged transformers for in-context learning to adapt to unseen opponents. However, previous Offline Opponent Modeling (OOM) approaches assumed that datasets contained optimal trajectories, a limitation that TIPR overcomes by enabling learning from suboptimal offline datasets. The idea of refining policies based on action-value functions is inspired by conservative offline RL methods, such as Conservative Q-learning (CQL) (Kumar et al., 2020) and Implicit Q-learning (IQL) (Kostrikov et al., 2021), but TIPR differs by performing refinement dynamically at test time rather than during offline training. This paper also contributes to the growing field of transformer-based RL, drawing inspiration from Decision Transformer (Chen et al., 2021) and Prompt-DT (Xu et al., 2022), but instead of using ICL for direct policy learning, TIPR applies it to improve action-value estimation through Truncated Q. Additionally, it addresses challenges in multi-agent RL related to opponent non-stationarity by introducing a more flexible confidence-based policy refinement mechanism, allowing real-time adaptation to shifting opponents. However, unlike adaptive truncation methods used in Q-learning (Poiani et al., 2023), TIPR does not yet provide an automated mechanism to determine the optimal truncation horizon $H$.", "essential_references_not_discussed": "Related to this paper's discussion on adapting the self-agent to unseen opponents, the problem formulation in [1] focuses on a similar setting from the perspective of principal-agent mechanism design using a meta-RL approach to few-shot test-time adaptation. \n\n[1] Banerjee, A., Phade, S., Ermon, S. and Zheng, S., 2023. MERMAIDE: Learning to Align Learners using Model-Based Meta-Learning. arXiv preprint arXiv:2304.04668.", "other_strengths_and_weaknesses": "Additional comments on some limitations in this paper:\n\n1. While the proposed approach is evaluated on smaller simulated environments for multi-agent RL, a discussion on real-world deployment challenges (for example, computational cost of real-time policy refinement) is missing. Testing TIPR on more complex strategic settings (like SMACv2) would better demonstrate its practical impact.\n\n2. The computational efficiency of TIPR is not discussed. Since IPR refines policies dynamically, it may be slower than standard OOM baselines.", "other_comments_or_suggestions": "The last paragraph in page 2 introduces the variable $M$ but has it been defined somewhere in the paper?", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ACJdYRkAh", "reviewer": "Reviewer_wBej", "summary": "This work aims to model opponent in multi-agent learning scenario from offline data. They argue that the offline data may be suboptimal and lead to suboptimal policies. To address this, they propose to learn a horizon-truncated in-context Q-function and whether to perform policy refinement. Truncated Q-driven Instant Policy Refinement (TIPR) maximize the No Maximization Bias (NMB) probability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback. In response to your comments, we would like to make the following clarifications and feedback. We hope our explanations and analyses can eliminate concerns and make you find our work stronger.\n\n---\n\n> **(A) Other Comments Or Suggestions:** The paper should define the concept \"policy refinement\" in a more concrete way …\n> \n\nIn *Offline Opponent Modeling* (OOM), existing algorithms typically assume that the offline dataset is optimal. We refer to the policy trained using these existing algorithms directly on the dataset as the **original policy**. When the dataset is suboptimal, we find that the performance of the original policy drops significantly. Our work aims to design a plug-and-play algorithmic framework to improve the performance of the original policy obtained through OOM algorithms—this is the definition of \"**Policy Refinement**\" (PR). We will clarify this in our revision.\n\nNotably, our work introduces the novel idea of performing PR instantly at test time, referred to as *Instant Policy Refinement* (IPR), rather than refining the original policy during the offline stage, as done in traditional *Offline Conservative Learning* (OCL) methods. As pointed out by Reviewer **Chog**, this helps mitigate distribution shift, and the advantage of IPR over OCL is also demonstrated in Fig. 4 (**Question 2** in Section 4.2).\n\n> **(B) Methods And Evaluation Criteria:** … However, it is still unclear to me why and how truncated Q value would address the sub-optimality of the dataset …\n> \n\nInspired by OCL methods, we aim to learn a Q-function to refine the original policy obtained through OOM algorithms. However, learning a workable Q-function in multi-agent environments is highly challenging, as the error in return prediction accumulates significantly when learning an Original Q (see Fig. 1). As you noted, this issue becomes more severe with a larger number of opponents—a claim that is also supported by our Theorem 3.1. In contrast, **our Truncated Q can effectively reduce this accumulation, making it possible to distinguish between optimal and suboptimal actions**.\n\nOur in-depth theoretical analysis (Theorem 3.1 and Proposition 3.2) demonstrates that, under the OOM problem setting, **an optimal truncated horizon exists that maximizes the probability of correctly identifying the best actions—formally, the No Maximization Bias probability**. This theoretical result supports the fact that using Truncated Q for PR is more sound and effective than using the Original Q.\n\nIn addition, our proposed IPR method can automatically decide whether to perform PR based on the confidence of action-value estimates from the Truncated Q. It effectively addresses the distribution shift problem commonly found in OCL methods: when uncertain, it falls back on the original policy (conservative), and when confident, it refines the policy (greedy). This trade-off mechanism is validated in Fig. 4 (**Question 4** in Section 4.2) to outperform both always being greedy and always being conservative.\n\n> **(C) Other Strengths And Weaknesses:** The results show that the value of fixed horizon doesn't impact the performance …\n> \n\nIn Fig. 5 (**Question 5** in Section 4.2), we analyze how the choice of different truncated horizons $H$ for the Truncated Q affects the improvement results. In fact, the results in Fig. 5 show that different values of $H$ lead to varying levels of improvement, and this effect is environment-dependent.\n\nWhen $H$ is set to a small value, the improvement from PR is limited. When $H$ equals $T$ (i.e., degenerating to the Original Q), PR generally degrades the performance of the original policy. However, when $H$ takes on a moderate value, PR is able to effectively improve the original policy. This observation supports our Proposition 3.2: there exists an optimal $H^{\\ast}$, and in general, $H^{\\ast}$ is not equal to $T$.\n\n> **(D) Questions For Authors:** Should we consider the method as few-shot adaptation method? …\n> \n\nYou've provided a very interesting insight—indeed, our method can be viewed as a form of few-shot adaptation. This is because our Truncated Q is built upon in-context learning and a Transformer-based architecture, which enables it to collect a small set of contextual samples at test time. Through the IPR method, we can adaptively decide whether to perform PR using the Truncated Q—based on the collected samples—without relying on gradient descent. This allows us to generate a refined policy in an adaptive and efficient manner.\n\n---\n\nAll your questions and feedback have greatly contributed to improving our manuscript. With the valuable input from you and all other reviewers, the quality of our work can be significantly enhanced. We welcome further comments from you and will seriously consider your suggestions for revisions. If you feel that we have addressed your concerns, we hope you will reconsider your rating.", "claims_and_evidence": "They present somewhat theoretical and experimental results to validate their claim. They provide results by isolating the performance improvement of truncated Q and iterative policy refinement across different testbeds. The experimentation includes different rational ablations.", "methods_and_evaluation_criteria": "The proposed method appears to perform better in terms of experimental evaluation. However, it is still unclear to me why and how truncated Q value would address the sub-optimality of the dataset. I understand truncated Q may reduce the complexity that arise with the increment of the number of opponents.", "theoretical_claims": "The proofs looks reasonable. However, as mentioned in the previous point, I am concerned about how truncated Q is a sufficient solution.", "experimental_designs_or_analyses": "The experiments are well-designed and throughly conducted.", "relation_to_broader_scientific_literature": "This paper nicely fits in between offline MARL, in-context learning, and transformer-based decision making models. However, it discusses a very specific problem opponent modeling via offline pre-training particularly at the presence of suboptimal data.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The results show that the value of fixed horizon doesn't impact the performance. It is not clear why this is the case. Also, it would be nice to see whether there is any difference between using lower value of h vs. values close to T.", "other_comments_or_suggestions": "The paper should define the concept \"policy refinement\" in a more concrete way as it is central to the paper.", "questions_for_authors": "Should we consider the method as few-shot adaptation method? It seems during test time it observes few samples first and then decide whether to perform policy refinement.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4fLyJbE6Tk", "reviewer": "Reviewer_LtaJ", "summary": "The paper proposes an offline opponent modeling approach that enhances the consistency of Q-functions by truncating the horizon length during Q-learning, incorporates in-context learning to mitigate distribution shift caused by sub-optimality in offline datasets, and employs test-time policy refinement to further improve policy performance. Experimental results across four tasks demonstrate the effectiveness of the proposed TIPR method and its key components.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We'd like to offer the following clarifications and hope our responses address your concerns and strengthen our work.\n\n---\n\n> **(A) Weaknesses:** 1. The authors use a MEP approach to generate 4 opponents …\n**Questions For Authors:** 4. … How are 20 policies sampled …\n> \n\nMEP uses Population-Based Training (PBT), which maintains a fixed population of 4 policies and evolves them through cross-play and natural selection. During training, we regularly saved checkpoints of all policies in the population, resulting in many policies. We then selected 20 diverse, representative ones (from weak to strong) to form the opponent policy pool.\n\n> **(B) Weaknesses:** 2 … the scalability of TIPR in complex environments (e.g., Google Research Football (GRF)) remains unverified.\n> \n\nRegarding scalability, we argue that **TIPR framework is fundamentally built upon highly scalable in-context learning methods and the Transformer architecture**, which perfectly aligns with the principles outlined in Richard S. Sutton’s \"*The Bitter Lesson*\" [1], emphasizing the inherent scalability of learning-based approaches.\n\nRegarding the choice of environments, although they are not highly complex, existing OOM algorithms still exhibit significant performance drops when the datasets are suboptimal. Moreover, these environments are widely recognized as benchmarks in the OM community and have been extensively used in prior works such as DRON, LIAM, TAO, OMIS [2], etc. While environments like GRF and SMACv2 are indeed more complex, they are typically used in MARL, focusing on agent communication and coordination, not the OM setting.\n\nTo further validate TIPR, we also ran preliminary experiments on the more challenging **OverCooked** (OC) environment [2,3], which has high-dimensional image inputs and requires intensive cooperation to complete a sequence of subtasks for serving dishes. The results can be found at [this link](https://ibb.co/XZX5X1m9). Due to time limits, we currently report results only on the representative OOM algorithm TAO, and we will include the full results in the revision. TIPR consistently improves the original OOM policy in OC, effectively addressing the suboptimality of datasets.\n\n[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html?ref=blog.heim.xyz, Rich Sutton.\n\n[2] Opponent modeling with in-context search, NeurIPS2024.\n\n[3] On the Utility of Learning about Humans for Human-AI Coordination, NeurIPS2019.\n\n> **(C)  Other Comments Or Suggestions:** 2. The MSE comparisons between $\\bar{Q}$ and $\\bar{Q}_V$ may be exacerbated …\n> \n\nOur main concern is the absolute Q-value error during Policy Refinement (PR) in unknown test environments. PR relies on the ranking of Q-values across all legal actions, since actions are chosen based on these rankings. Normalization doesn’t affect this ranking—it only scales values uniformly. Therefore, the reported MSE directly reflects confidence in action rankings for each method. Normalization wouldn't change qualitative conclusion that truncated Q-values improve ranking accuracy and thus enhance PR effectiveness.\n\n> **(D) Questions For Authors:** 1. How to read the colored and shaded areas of Fig.3 in MS and PP? …\n> \n\nWe use ***Black Error Bars*** (BEB) for the original OOM policy's Standard Deviation (SD) and *Grey Error Bars* (GEB) for the SD after applying TIPR. In both MS and PP, GEB consistently appears above BEB, with the shaded area showing performance gains. This shows TIPR consistently improves OOM performance. We'll refine Fig. 3 in the revision to address any clarity issues.\n\n> **(E) Questions For Authors:** 2. Why are the refine condition (RC) threshold and … set as 0?\n> \n\nThis design is intuitive: (1) **Indicator Condition**: If a legal action is predicted to yield non-zero reward within $H$ steps, its value estimate is considered high-confidence. (2) **RC**: If any action meets this, we deem it promising to use Truncated Q to refine the policy at that timestep. This helps in sparse-reward environments and doesn't negatively impact performance in dense-reward ones.\n\n> **(F) Questions For Authors:** 3 … So I wonder how the differences between the trained opponents are.\n> \n\nRegarding the diversity of the opponent policies, please refer to our response to reviewer **Chog**'s comment \"**(D) Experimental Designs Or Analyses:** … However, if the offline dataset\". You can use Ctrl + F to search for the sentence in quotation marks to quickly jump to there.\n\n> **(G) Questions For Authors:** 5. Does TIPR introduce significant additional computational overhead …?\n> \n\nFor this question, please refer to our response to reviewer **Chog**'s comment \"**(G) Other Strengths And Weaknesses:** 2. The computational efficiency\". \n\n---\n\nYour comments have greatly improved our paper. If you have further feedback, we'll address it promptly. If your concerns are resolved, we hope you'll reconsider your rating.", "claims_and_evidence": "Yes. The claims are reasonable and seem technically sound.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. I just roughly checked the proof in the appendix and had not found any issues.", "experimental_designs_or_analyses": "Yes. I checked the designs and analyses. Most of the experimental designs are reasonable.", "relation_to_broader_scientific_literature": "This work provides an opponent modeling based approach for MARL.", "essential_references_not_discussed": "I have not found yet.", "other_strengths_and_weaknesses": "**Strengths:**\nThis work integrates truncated Q-learning, in-context learning, and policy refinement to enhance opponent modeling in multi-agent reinforcement learning. TIPR serves as a plug-and-play framework compatible with many offline opponent modeling (OOM) methods, making it a promising tool for broader applications. The experimental design and analysis validated the effectiveness of TIPR.\n\n**Weaknesses:**\n\n1. The authors use a MEP approach to generate 4 opponents across all four environments, which possibly cover the opponent policy spaces. While this may be sufficient for the current task, it raises concerns about whether the results generalize to larger scenarios.\n2. The in-context learning module uses a dataset of sequence length 15 across four experiments. Given that Transformer architectures typically require large-scale pre-training data, the scalability of TIPR in complex environments (e.g., Google Research Football (GRF)) remains unverified.", "other_comments_or_suggestions": "1. The experiments may conduct on some larger-scale environments.\n\n2. The MSE comparisons between $\\bar{Q}$ and $\\bar{Q}_V$ may be exacerbated. As the original Q is used for a longer horizon, thus a normalization may be needed.", "questions_for_authors": "1. How to read the colored and shaded areas of Fig.3 in MS and PP? The shared areas show a permanence drop, which may contradict the legend.\n2. Why are the refine condition (RC) threshold and the inner cumulative confidence threshold (indicator condition) set as 0?\n3. If I understand correctly, the context length to identify the opponent's newest policy is 15 which is very fast with Transformer (assumed the opponent is seen or unseen but fixed during 20 episodes). So I wonder how  the differences between the trained opponents are.\n4. The Appendix E.1 states that the size of MEP population is 4. How are 20 policies sampled from this population for Seen and Unseen setting?\n5. Does TIPR introduce significant additional computational overhead (in terms of time and memory) during testing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "abyEhyGGhz", "reviewer": "Reviewer_RP82", "summary": "This paper proposes to learn a horizon-truncated incontext action-value function and a policy refinement mechanism to tackle the offline opponent modeling problem, espically when the training dataset is suboptimal. This paper has  analyzed the rationale of Truncated Q from the perspective of No Maximization Bias probability theoretically. And experimental results have demonstrated the effective of the algorithm.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. In response, we offer the following clarifications and hope our explanations address your concerns and strengthen our work.\n\n---\n\n> **(A) Theoretical Claims:** 1. What's the definition of $\\breve{Q} _\\mathsf{h}$ in Equation 2.\n> \n\n$\\mathsf{h}$ denotes the random variable of the truncated horizon, and $\\mathsf{G} _\\mathsf{h}(t) := \\sum _{t'=t}^{t+\\mathsf{h}-1} \\gamma^{t'-t} \\mathsf{r} _{t'}$ denotes the random variable of the horizon-truncated *Return-To-Go* (RTG). In Eq. (2), $\\breve{Q} _\\mathsf{h}$ denotes the random variable corresponding to the Truncated Q (i.e., the neural network we aim to learn), and its learning objective is to approximate $\\mathbb{E}\\mathsf{G} _\\mathsf{h}$.\n\n> **(B) Theoretical Claims:** 2. The explanations of the notations in Theorem 3.1 should be included in the main paper\nWeakness: 1. Each symbol used in the paper should be clearly defined …\n> \n\nWe create a notation table in [this link](https://ibb.co/m5QGry8L) to provide detailed explanations of all the notations used in Thm. 3.1. These definitions will be incorporated into the main text in our revision.\n\n> **(C)** Weakness: 2. The pseudocode of the algorithm is necessary in the main part.\n> \n\nWe provide the pseudocode for the TIPR algorithm framework at [this link](https://ibb.co/9H8Hjwt8). We appreciate your suggestion and will incorporate it into the main text in our revision.\n\n> **(D)** Weakness: 3. More intuitive explanation is needed as to why truncating Q-values is better.\n> \n\nFor this question, please refer to our response to reviewer **wBej**'s comment \"**(B) Methods And Evaluation Criteria:** … However, it is still unclear\". You can use Ctrl + F to search for the sentence in quotation marks to quickly jump to there.\n\n> **(E) Questions For Authors:** 1. Can you give a detailed explanation about why the error in the optimization objective of $\\bar{Q}$ accumulates …\n> \n\nIn Eq. (1), the original Q-value $\\bar{Q}$ regresses onto the RTG label $G_t^1$. As the horizon grows longer, estimation errors for individual rewards accumulate due to the cumulative summation of discounted rewards. This compounding of errors directly reduces the empirical accuracy of the learned $\\bar{Q}$, especially when datasets are suboptimal.\n\nFrom a theoretical view (Thm. 3.1), this accumulation manifests explicitly in the term $\\Delta(\\mathsf{h})$, whose complexity increases with the $\\mathsf{h}$ (specifically $\\Delta(\\mathsf{h})=O(\\mathsf{h}^{2})$). Consequently, as the $\\mathsf{h}$ extends, the Empirical Risk NMB Probability (the probability of correctly fitting truncated returns) decreases significantly due to this error accumulation.\n\nThe longer horizon in $\\bar{Q}$ reduces its reliability in identifying actions with the highest true returns, lowering the Overall NMB Probability. Truncating the horizon limits the error accumulation, improving action selection accuracy and policy refinement—especially in challenging suboptimal dataset settings.\n\n> **(F) Questions For Authors:** 2. How to find the optimal $H^{\\ast}$? …\n> \n\nOur primary focus in this work is to demonstrate, both theoretically and empirically, that **just truncating the horizon for Q-value estimation, without careful optimization of $H$, can already provide significant advantages over not truncating at all**. Theoretical insights from Prop. 3.2 explicitly show the existence of an optimal $H^{\\ast}$, validating the conceptual soundness of truncation.\n\nIdentifying the precise $H^{\\ast}$ is indeed an important direction for future research. Here are some possible methods:\n\n1. **Cross-Validation on the Dataset**: Empirically selecting $H^{\\ast}$ by performing cross-validation using subsets of the dataset to minimize prediction errors or maximize validation performance.\n2. **Adaptive Methods**: This approach adaptively adjusts $H$ during training or testing based on real-time confidence or uncertainty estimates. For example, an adaptive strategy could start with a small horizon and increase it if confidence stays high, or reduce it when uncertainty rises—balancing accuracy and computation dynamically.\n\n> **(G) Questions For Authors:** 3. Intuitively, h should not be too large or too small, and It should have robust performance within a certain range …\n> \n\nIn fact, the results in Fig. 5 (**Question** 5 in Section 4.2) clearly demonstrate that setting $H$ too large (e.g., $H = T$, which degenerates into the Original Q) or too small (e.g., $H = 1$) is suboptimal. A moderate value of $h$ tends to yield better performance.\n\nWe did not perform an exhaustive search for the optimal $H^{\\ast}$ in each environment. In Fig. 3, we used a moderate, untuned $H$. Still, Truncated Q effectively improves suboptimal original OOM policies over the Original Q.\n\n---\n\nYour comments have greatly helped improve our manuscript. If you have further feedback, we'll address it promptly. If your concerns are resolved, we hope you'll reconsider your rating.", "claims_and_evidence": "Yes, the claims made in the submission are supported by both theory and experiment results.", "methods_and_evaluation_criteria": "The evaluation criteria is make sense, but the proposed algorithm needs more explanation. Please refer to the Question part.", "theoretical_claims": "I have reviewed the theoretical section of the paper, but I have some confusions. For example:\n1. What's the defination of $\\breve{Q}_h$ in Equation 2.\n2. The explanations of the notations in Theorem 3.1 should be included in the main paper", "experimental_designs_or_analyses": "Experimental designs are good.", "relation_to_broader_scientific_literature": "The key contributions of this paper build upon and extend several important lines of research in offline opponent modeling (OOM), offline reinforcement learning (RL), and in-context learning.", "essential_references_not_discussed": "I am not very familiar with this field, so I am not sure.", "other_strengths_and_weaknesses": "Strength:\n1. The paper introduces truncated Q-value estimation, which is a interesting approach.\n2. This paper provides experimental validation and theoretical analysis, which can serve as an inspiration for future research.\n3. The paper is well-structured.\n\nWeakness:\n1. Each symbol used in the paper should be clearly defined. The main paper should be self-contained.\n2. The pseudocode of the algorithm is necessary in the main part.\n3. More intuitive explanation is needed as to why truncating Q-values is better.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. Can you give a detailed explanation about why the error in the optimization objective of $\\bar{Q}$ accumulates primarily through the accumulation of rewards over time $t'$ in $G_t^1$ in Equation 1.\n\n2. How to find the optimal $h^*$? Because experimental results show that truncating Q-values performs worse in certain values of h.\n\n3. Intuitively, h should not be too large or too small, and It should have robust performance within a certain range. However, this paper indicates that TIPR’s improvement on OOM algorithm does not exhibit a strict correlation trend with the truncated horizons. This is somewhat counterintuitive. What role does the randomness of hyperparameter tuning experiments play in the experimental results?\n.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a framework named Truncated Q-driven Instant Policy Refinement (TIPR) to improve Offline Opponent Modeling (OOM) algorithms trained on suboptimal datasets. Both theoretical analysis and experimental results strongly back the central assertions that TIPR enhances OOM algorithms, Truncated Q outperforms Original Q, and that IPR is more effective than OCL. After the rebuttal discussion, the reviewers consistently think this paper can be accepted.
Decision: Accept (poster)