Planning with Quantized Opponent Models
Authors: XiaoPeng Yu
Kefan Su
Zongqing Lu
Venue: NeurIPS 2025 poster ()
Keywords: opponent modeling
multi-agent
deep reinforcement learning
Primary area: deep_learning
TL;DR: QOM uses quantized autoencoders to model opponent types, updates beliefs in real-time via Bayesian inference, and enables efficient belief-aware planning.

Abstract: Planning under opponent uncertainty is a fundamental challenge in multi-agent environments, where an agent must act while inferring the hidden policies of its opponents. Existing type-based methods rely on manually defined behavior classes and struggle to scale, while model-free approaches are sample-inefficient and lack a principled way to incorporate uncertainty into planning. We propose Quantized Opponent Models (QOM), which learn a compact catalog of opponent types via a quantized autoencoder and maintain a Bayesian belief over these types online. This posterior supports both a belief-weighted meta-policy and a Monte-Carlo planning algorithm that directly integrates uncertainty, enabling real-time belief updates and focused exploration. Experiments show that QOM achieves superior performance with lower search cost, offering a tractable and effective solution for belief-aware planning.

Planning with Quantized Opponent Models
Xiaopeng Yu
Kefan Su
Zongqing Lu"
School of Computer Science, Peking University
Abstract
Planning under opponent uncertainty is fundamental challenge in multi-agent environments, where an agent must act while inferring the hidden policies its opponents. Existing type-based methods rely on manually defined behavior classes and struggle to scale, while model-free approaches are sample-inefficient and lack principled way to incorporate uncertainty into planning. We propose Quantized Opponent Models (QOM), which learn a compact catalog of opponent types via a quantized autoencoder and maintain a Bayesian belief over these types online. This posterior supports both a weight-d meta meta-policy and a Monte-Carke planning algorithm that directly integrates uncertainty, enabling real-time belief updates and focused exploration. Experiments show that QOM achieves superior performance with lower search cost, offering tractable and effective solution for belief-aware planning.
Introduction
Opponent modeling is considered an important aspect of intelligent behavior in partially observ- able multi-agent environments[[] Early robotics and game-Al systems constructed explicit rype libraries srararlssssaall sets of handcrafted behavior classes- and then fitted the observed action stream to the most plausible type online [36] 47). Advances in machine learning broadened this agenda,
giving rise to statistical recognizers that detect plays 44 formations [13, or higher- level strategies real-time strategy games [39
49 Complementary model-free approaches forgo explicit opponent representations and instead adapt the agent's policy through reinforcement learning (2): yet such adap-
tation sample-inefficient and brittle when observations are noisy or delayed. Crucially, regardless of whether the agent maintains explicit opponent beliefs, it must still plan ahead under uncertainty. MonteeCCalla Tree Search (MCTS) (8] is the de-facto planner for large sequential domains, but its branching factor explodes when every node must marginalize over many hypothetical opponent moves. Naïve belief propagation inside the tree quickly becomes intractable 14). Bridging expressive opponent modeling with tractable belief-aware planning therefore remains open challenge.
We address this challenge by proposing Quantized Opponent Models (QOM). Our core insight is to compress the vast policy space of potential opponents into finite, interpretable catalog of latent types discovered offline by a quantized autoencoder. During online interaction, the agent maintains a Bayesian belief over these types; by using pre-trained decoder, the model can compute likelihoods efficiently enough to support real-time updates. This belief used to sample opponent types in simulation, mix agent policies into soft best-response meta-policy, and guide planning through belief-informed prior. Consequently, every rollout (i) samples opponent actions from the current posterior, (ii) selects agent actions that promising under that posterior, and (iii) refines the belief along the simulated trajectory, allowing planning and inference to reinforce each other.
Our framework thus unifies three previously disparate strands of research. From type-based opponent modeling, it inherits interpretability while discarding manual type design; from Bayesian reasoning
"Correspondence zongqing. lu@pku. edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

it inherits principled uncertainty handling without incurring prohibitive inference costs; and from Monte-Carlo planning inherits strong look-ahead search while injecting belief information directly into the tree policy. We provide theoretical analysis of posterior concentration under uncertain opponents, and empirical results demonstrating competitive performance gains over state-of-the-arr baselines in benchmark games with partial observability and adversarial dynamics.
2 Related Work
Type-based opponent modeling has evolved from early behavior classification schemes that used handcrafted feature predicates for adversarial domains [6 47). Subsequent work explored declarative and similarity-basee formulations to transfer knowledge across game instances [48] 39), while approaches extracted relational sequential patterns anticipate high-level strategies in both real-time strategy games and sports domains [54] 6 31. Researchers further demonstrated online
adaptation of offensive tactics 227 26 and recognition of group activities from multimodal trajectories
[4 28 38). More recent studies integrate positional data streams to identify formations on-the-fly
or learn objective-driven agent behaviors from logged encounters [13 20
Recent hierarchical
formulations have also been proposed to couple opponent modeling with planning in structured goal spaces [16). Collectively, these works substantiate the effectiveness of representing opponents by finite catalog of "types"- an idea our Quantized Opponent Models embrace, while unifying type discovery and belief maintenance within a single quantized autoencoder framework.
Bayesian reasoning provides a complementary perspective, treating the opponent's policy (or latent parameters) as random variable and continuously updating posterior beliefs evidence accrues. Early vision-based systems inferred multi-agent activities through dynamic Bayesian networks 17 [8, and robot-soccer agents leveraged probabilistic motion models predict trajectories and openings (37 4]. Subsequent work applied hierarchical or domain-specifie priors to forecast build
orders in real-time strategy games 49 51) and to locate sweet spots in racket sports [55]. Recent advances scale Bayesian inference to negotiation 77]. normative coordination (34), inverse games [30]. and safety-aware exploitation under uncertain type belief's [29]. Our framework inherits this probabilistic tradition by maintaining an explicit posterior over latent policy types and amortizes likelihood computation enable real-time updates during planning rollouts.
MCP, especially MCTS, has become a widely adopted method for look-ahead decision making. Foundational variants such as Bayes-adaptive and information set MCTS 9 14] inspired extensions
to multi-agent POMDPs 3 21 and typed or open systems 33 12 Recent research refines value backup and exploration principles [1]], reuses beliefs across rollouts (50), adapts models online [60), and incorporates learning for path-planning or decentralized coordination [46] 10 [5). Risk- aware shielding, decision-theoree algorithmic toolbox [32
communication, and nested MCTS methods further widen the 19 433 Our method builds on this line of work by incorporating QOM's belief-weighter meta-policy and Bayesian belief updates directly into the agent action selection, thereby synthesizing advances in type abstraction, probabilistic reasoning, and MCP into a unified belief-aware planning algorithm.
3 Method
In this section, we introduce QOM for planning in multi-agent partially observable environments. Our key idea is to combine a compact, discrete representation opponent policies with principled belief tracking and planning under the unknown opponent policy. We first describe the QOM that captures opponent behavior, and then outline how to construct a meta-policy based on the learned latent types, and finally detail the integration of this belief model into an MCP framework for planning. The algorithm architecture is summarized in Figure
3.1 Preliminaries
defined as tuple g AS, O, O-,, P.O. R2 7), involving an agent and a set of other agents (opponents) A-,- We treat all opponents jointly as single composite agent -i with joint action space A_; and joint observation space O_,, where S the hidden state space, A, and A-; are the action spaces of agent i and the joint opponent, O and O_, are the observation spaces, P(s' s, a, a_;) is the transition
We consider partially observable stochastic game
15

---

(T) Rdd, then assigns it to the nearest codebook vector ek- Bellef Tracking
Offline Preparation
0, 0)
QOM
D (0)
6(K)
nooder
b(k)
Quantbars
oponnnt Simulation
Decoder
Action Selection
Ensrromment
optata
PUCT
((1)
Figure 1: Overview of the QOM framework. The agent maintains belief over latent opponent types, which is updated based on observed actions. The quantized autoencoder learns discrete representation of opponent policies, and the meta-policy is constructed mixture of soft best responses to these types. This meta-policy is then used for planning.
probability, O:(o: s') and O-i(o-; | s') are observation models, Rs(s, ais am;) is the reward for agent i, 7 [0, 1) the discount factor.
At each time step f, both agent and the opponent select actions a; a based on their own local observation-action histories h ft The agent's goal maximize the expected return:
8
J BT
E (s 2 (...)))
Besides the game definition, following resources are required: (i) an agent policy library II; { (1) //_ that stores set of candidate policies; (ii) an opponent policy library II_; { [-=jjj111 that
supplies diverse behaviors; and (iii) an environment simulator that produces transitions (s', r) (ss, ay, as) and is used both for offline data generation and for online rollouts during planning.
3.2 Quantized Opponent Models
To reason effectively in the presence of an unknown opponent, we introduce the Quantized Opponent Models (QOM), which models the opponent's 's policy using discrete latent variable representation. QOM approximates the opponent's potentially complex policy using set of representative policy "types," learned via quantized autoencoder (VQ-VAE) [2 where the raw policy sequences are modeled autoregressively.
Latent opponent types via quantized autoencoder. We approximate the space of opponent policies with finite set of latent types (1...., K). An offline-trained quantized autoencoder maps each opponent trajectory 7, collected by letting an agent policy x; interact with an opponent policy (.) continuous embedding The decoder fase(ek:)) produces step- wise likelihood ax h), enabling both fast simulation and likelihood evaluation. This discretization turns an intractable policy space into a tractable, interpretable catalog of opponent policies.
Belief over latent types. The belief by (k) is defined as the probability that the opponent follows type A;; it forms categorical distribution over the K latent types. All opponent-depende quantities, opponent quantized policy मे and rollout priors, are averaged under this distribution, so planning automatically balances the risks of different hypotheses. A broad belief encourages exploratory behavior, whereas sharply peaked belief lets the agent exploit its current inference. Thus by (k) is the single latent variable that ties together opponent modeling and decision making.

---

3.3 Meta-Policy Construction
Given the belief distribution (k) over latent opponent types and a set of agent policies IT, we define meta-policy for the agent mixture of soft best responses, reflecting the agent's expected performance against different opponent types. For each opponent type k, we compute soft response distribution over the agent policy set using a softmax rule:
during rollouts. The meta-policy is constructed belief-weighte
(0) 11) (
exp( R(,s) Ex exp(Rv,
(2)
where 3 is temperature parameter and Ri is the estimated expected return of agent policy x; when interacting with opponent type k. These values can be approximated via offline simulations, and represent a kind of payoff matrix between the agent and latent opponent policies.
(1)
Intuitively, o (m)) k) represents how suitable agent policy n. is response to a particular opponent type k. A higher R1, results in a higher selection probability, which controls how sharply the distribution concentrates around the best-responding policies.
The agent's 's overall rollout policy is then belief- weighte mixture over all opponent types and corresponding soft responses:
I (1) (T)
meta (a: h1:) =
by (k) k=1 1=1
Σ
k) o. (a; h.).
This meta-policy formulation provides a principled way to combine prior agent policies in a belief- aware manner. Rather than committing to a single policy or performing hard switching, the agent manner. This approach ensures robustness and adaptability,
combines its policies in a and is especially useful in planning contexts where simulated rollouts benefit from coherent and belief-aligned action selection.
Payoff matrix estimation. The meta-policy requires a payoff matrix R RLLK whose entry RI,
is the empirical return of agent policy n!!) against latent opponent type k. To compute those entries we first label every offline trajectory with type index and then aggregate the corresponding returns.
After the quantized autoencoder is trained, each collected opponent trajectory r is passed through the encoder fene to obtain embedding 2. The trajectory is mapped its closest codebook vector eg by assigning the index k* = arg min; || exl12. which yields a latent label consistent with the prototype policies used later in belief tracking. Given the labels, the payoff matrix is estimated by bucket-wise averaging:
Σ R LE
Gus, Gis,
Ni. = Ki MT kJ},
(4)
NE. Jo
where Gr, is the return obtained by n. in trajectory Tijj.
Trajectory collection. All trajectories used above are generated once offline by pairing policy (!) from the agent library IT; with policy (() from the opponent library IT,, and executing the environment model G. Each trajectory simultaneously (i) supplies data for learning and labeling latent types and (ii) records the returns needed for the payoff matrix, so the same dataset serves both purposes without extra simulations. This payoff matrix construction is performed entirely offline, removing its computational cost from the online decision-making loop.
3.4 Planning with Quantized Opponent Models
Once we have established a compact latent representation of the opponent's policies through QOM, we integrate this model into MCP framework for belief-aware planning. Our goal is to plan effective actions of the agent while reasoning over the uncertainty in the opponent's latent type. This integration requires three key steps: belief tracking, opponent simulation, and agent action selection.

---

Algorithm Planning with Quantized Opponent Models
1: Input: initial history hgma h belief greal (A:
2: for each environment step do
3: Initialize simulation with current history and belief
4: while within computational budget do
5: Set rollout history and belief from real environment values
6: while not terminal and within depth limit do
7:
Sample opponent type k ~ b(k)
8:
Sample opponent action a_i 7x(- h_2)
9:
Compute agent meta-policy oi and select agent action a, using PUCT
Simulate next step: (s', o4. o' r')
11:
Update histories: h2s h-s
Update belief b(k) using simulated opponent observation and Equation 5 13: end while
Backpropagate value and update Q, N end while
15:
Select of = arg maxa, ((hs, 0z)
Execute at, observe 0; ('-p. infer (1) and update A.sal N
(1+1) (t)
17:
18: Update belief (ka) using Equation ( apply belief smoothing
19:
end for
Belief tracking. During both online execution and internal simulation, the agent maintains a categorical belief (kk) over the K latent types. After observing opponent action a given history [ either from real interaction or simulated rollout, the belief is updated via Bayes' rule:
be+1 (k) α (k:) [a "
(S)
Here, (x ( ()) denotes the likelihood of the opponent action under latent type k, estimated by the learned decoder.
During online execution, belief updates are performed based the observed opponent actions, progressively refining the agent's 's uncertainty over latent types as interaction unfolds. This posterior refinement aligns with the assumptions underlying our theoretical analysis and supports the expected convergence behavior over time, see Section A
During rollouts, belief updates similarly performed every step. At each newly expanded node, the belief is inherited from the parent node and immediately updated according to the newly generated opponent action. Formally, if parent node at history h; carries belief (k)), then after simulating the
agent's action a; (t) and opponent action a " the newly created child node at h carries an updated belief b (k) according to Equation ( This consistent inheritance and updating of belief across
both real and hypothetical trajectories enables the agent to reason adaptively about latent opponent behavior over time.
To mitigate premature overconfidence caused by noisy observations decoder approximation errors, we apply belief smoothing after each update by interpolating the updated belief b (k) with the initial prior bo(k). Specifically, the smoothed belief is computed as bymmothh( = (1 - A)6 (k) + (b)(k), where À controls the degree of smoothing and (k) is the uniform distribution over types. This simple smoothing mechanism helps maintain exploration and prevents belief collapse, especially during the early, data-sparse stages of interaction. The dynamically refined belief at each history serves as the foundation for both opponent simulation and meta peta-policy construction throughout planning and execution.
Belief-driven simulation of opponent actions. In each simulation, the agent leverages the current belief distribution (k:) to model opponent behavior in a belief-aware manner. At each decision step during the rollout, a latent opponent type is sampled according the current belief, k ~ (x()). Given the sampled type, the corresponding decoder सe is used to simulate the opponent's action
based on the reconstructed rollout history ( following _ ox(: (: ()). Since the opponent's
5

---

private observations are not directly available, the agent reconstructs the opponent's 's history using the environment simulator & to infer plausible opponent observations during the deployment. The simulator generates next states and observations based on joint actions, enabling the agent to update h incrementally as simulation progresses.
This belief-driven simulation ensures that the planning evolves consistently with the agent's 's posterior opponent types. As the belief (k) becomes increasingly concentrated through interaction, the sampled opponent behaviors naturally adapt, leading to progressively more targeted planning.
Belief-informed agent action selection. For selecting its own actions during simulation, the agent adopts belief-informed variant of the Predictor Upper Confidence Bound for Trees (PUCT) algorithm 45 that incorporates the meta-policy as a dynamic prior. Specifically, for each candidate action o, at given agent history h4, the PUCT score is computed as:
T. (as 1b)
Uh, a.) = Q(hi, a.) + 1+ (h;,
(6)
where Qhhi a:) denotes the estimated action value of action a; in the agent history h4, (N,,, a;) is the corresponding visit count, n. (ap h;)) represents the prior probability of selecting under the meta meta-policy constructed from the current belief, and e > is an exploration constant. After each simulation, the value Q(h,, a,) is updated by backing up the cumulative return V along the search path, where Q(h;; a;) is incrementally updated as a running average weighted by the new sample, and V denotes the cumulative discount reward obtained after selecting action at history h;, Specifically, V is recursively computed during the rollout by discounting the future return and adding the immediate reward, following the standard form V <- + vV, where ra is the reward received and y E [0, 1) is the discount factor.
By integrating -. into the PUCT, the agent is guided towards actions that are promising given its belief about the opponent, while still allowing sufficient exploration of less visited branches. This mechanism ensures that search effort is adaptively focused on policies likely to perform well against the inferred opponent- type distribution, thereby improving planning efficiency.
Our method integrates belief tracking, opponent simulation, and meta-policy-guideed action selection into the planning algorithm enable adaptive opponent uncertainty. By reasoning about latent policy types and updating beliefs online, the agent can make progressively more informed decisions throughout planning and exccution. The simplified planning procedure is shown in Algorithm and the detailed algorithm is provided in Section B
4 Experiments
4.1 Environment
We evaluate method across four diverse multi-agent environments, covering range of interaction structures, including cooperative and competitive dynamics, varying levels of partial observability, and spectrum of strategic complexity.
The Pursuit- Evrrsion and Predator- Prey tasks 42 are discrete grid-world domains where agents act based on local observations and reactive strategies. Pursuit-Evasion is an asymmetric zero-sum game where the evader must reach a goal while avoiding capture by a pursuer. Predator-Pree is a cooperative setting requiring multiple predators to coordinate in order to capture the prey.
The Running -ritiiingwwttt-SSe with-Scissors (RWS) environment 5 53) extends rock-paper pappp-ppppsssseeee scissors into a spatial, partially observable gridworld. Two agents collect resources rock, paper, or scissors and can tag 3x3 area ahead to challenge the opponent, with outcomes determined by rock papeererr-scissors rules. The environment features concealed information and non-transitive payoffs, requiring agents to infer opponent intent and commit to -tuater-strategies through temporally extended behavior.
The One-on-One scenario 23 57 involves high-dimensional state and action spaces, where an attacker and goal-keeper compete in simplified football drill. This environment emphasizes fine-grained control and opponent-aware decision-making in high-dimensional state and action spaces.

---

QOM
POTMMCP
I-POMCP-PF
MBOM
Mean Return
0.75
Return
0.75
Mean Return
0.25
0.00
0.00
-0.50
0.2
Search Time
10
Search Time
Search Time
10
20
PE(Evader)
(b) PF(Pursuer)
(c) Kumning-Withe With-Scissors
Mean Return
1.0
Mean Return
0.4
Win Rate
0.3
0.6
0.4
0.
6.1
0.0
0.0
0.1
20
01
20
0.1
10
20
Search Time
Search Time
Search Time
(d) PP(two-agents)
(e) PP(four-agents)
One-on-One
Figure 2: Performance comparison across six multi-agent environments under varying search time budgets. Each point denotes the mean return over 300 episodes, with emror bars showing 95% % confidence interval. Our method (QOM) consistently outperforms all baselines across wide range of settings, particularly especially when computation time is limited. The x-axis corresponds to discrete search-time settings with non-uniform spacing in actual runtime; lines drawn to illustrate performance trends. Environments include: (b) Pursuit-Evasion from both evader and pursuer perspectives; (c) Running-With-Scitnn With-Scissors; [d] Predator-Prey with two and four agents; and (f)
One-on-One football task.
4.2 Experimental Setup
Policy Library. To obtain diverse set of strategies, we construct the policy library using PSRO
255. Unless otherwise specified, each environment includes a library of [I,] 10 agent policies and
50 opponent policies. The opponent's latent type size set to K 16 by default.
We compare QOM against the following baselines that reflect different paradigms of belief-aware planning and opponent modeling.
Baselines.
A state-of-the-art Monte-Carlo planning method that performs belief- aware decision making by maintaining particle filter over opponent policies. It integrates belief tracking with search-time planning to adapt to opponent policy.
POTMMCP 41
J-POMCP-PE. A variant of I-POMDPs [19] without explicit communication or recursion. It uses particle filtering to represent beliefs over opponent policies and performs planning via POMCP with uniform random rollout policy at the leaf nodes.
MBOM [57]. A machine Theory of Mind method that infers opponent intentions through online planning and simulation rollouts. MBOM does not rely on discrete policy library and instead models the opponent implicitly through predictive learning.
Each experiment conducted over 300 episodes, and the average return is reported as the final result.
We evaluate performance under varying search time budgets, scaled [0.1, 1, 10, 20) units of time. The unit time environment-specific and reflects the relative planning complexity of each domain. We ground-truth simulators for all environments except One-on-One. which uses learned simulator. Detailed experimental settings, including environment configurations, policy libraries, and hyperparameters, are provided in Section
4.3 Performance of static opponents
Figure Saauueerrepor reports the mean return under varying search time budgets. Across environments and time budgets, our proposed method QOM consistently achieves the highest performance. This indicates its
7

---

QOM POTWHCP
HPORCPPP KDOH
QOM POTHRCP
EPONCEPP MDOM
0-1--0
[0_
n. 130
[0. -20
Mean Return
Mean Return
Return
25
-.0
-75
53
13 Search Time (c) Scalability
Search Time
Search Time
(a) Adaptation
(b) Generalization
Figure 3: Evaluation in the Running- Sunning-With-- With-SSiissors environment under three conditions. In (a) the opponent switches its policy after unsuccessfully tagging the agent, testing the agent's ability to rapidly adapt to changing behaviors. In [b] the opponent uses policy not included in the predefined policy library, evaluating each method's s generalization to unseen policies. In [c] library size [II_;] assess the scalability and inference efficiency of QOM as policy library size increases.
we vary the policy
ability to efficiently leverage posterior beliefs over opponent types to guide decision making, even under tight computational constraints. In particular, QOM shows rapid improvement with increased search time in both the Pursuit-Evasion tasks and the Predator- demonstrating effective belief- driven planning in both asymmetric and cooperative-comperiiyy scenarios. РОТMMMP employs particle-filters posterior tracking to model opponent behavior. However, under low search time budgets, its effectiveness diminishes due the sample inefficiency of particle filtering, In contrast with QOM's compact rrpe-based belief model, which enables faster and more structured inference. I-POMCP-PF shows moderate performance gains but remains limited by its reliance on uniform rollout policies and shallow belief updates, especially in complex environments such as Figure Figure[(e) Meanwhile, MBOM exhibits competitive performance in simpler tasks but degrades in more strategic domains as Figure [ because it lacks an explicit policy library and struggles to generalize online. In the One-on-One setting as Figure 2(1) the performance variance higher across all methods due to the fine- ine-grained strategic adaptations required. Even here, QOM still maintains a slight advantage.
4.4 Adaptation to Switching Opponents
To evaluate adaptability, we design an experiment where the opponent switches its policy after unsuccessfully tagging the agent. This simulates a piecewise-stationay adversary, requiring the agent to detect behavioral changes and respond accordingly. As shown in Figure 3[a) QOM outperforms
all baselines. This highlights the efficiency of our Bayesian belief update mechanism in detecting type shifts and reallocating planning effort. In contrast, POTMMCP and I-POMCP-PF struggle to adapt rapidly due their reliance on particle filtering, which lags in capturing abrupt type transitions with limited samples. MBOM performs comparably at high budgets but lacks structured uncertainty modeling, resulting less reliable performance under tight time constraints.
4.5 Generalization to Unseen Opponents
We test generalization by deploying opponent that uses policy not included in the policy library. This scenario evaluates whether the agent can perform robustly when faced with -iitof-distributionn behaviors. As shown in Fib) QOM again demonstrates strong performance, outperforming baselines across all search budgets. Despite the unseen nature the opponent, the quantized opponent models extrapolate effectively, assigning probability mass to nearby types in latent space.
4.6 Scalability with Policy Library Size
To assess the scalability of belief inference, we vary the policy library size II_;| from 30 to 300 while keeping the test policy fixed. As shown in Figure (3c) QOM maintains stable and high performance across all library sizes, showing only minimal degradation, highlighting the scalability of the belief model. This demonstrates the model's 's ability to scale efficiently with growing policy library size. In contrast, purticle-based methods such as POTMMCP suffer from increasing posterior variance
8

---

GOM BerL
000
ooe
1. a
Mean Return
Mean Return
x
b. e
Mean
Search Time
Search
Search Time
Search Time
(b) π (8)
(c) Quantization
(d) K
Figure 4: Ablation study in the Running- Weth--Wiiihhhtn environment. In a) we vary the belief update temperature 3, observing that moderate entropy regularization improves adaptation, while extremes degrade performance. In (b) we compare different approximations of the belief-weighted opponent
policy # _ showing that marginalizing over type beliefs (QOM) outperforms heuristic alternatives such as choosing the best, worst, or random type. In [c) we evaluate different quantization methods used discretize opponent behavior, including FSQ. LFQ, and BSQ; all perform comparably, but VQ-based QOM achieves more consistent gains across budgets. (d) increasing the number of types K initially improves performance, but further increases lead to type redundancy.
and degraded precision when [T_x]] is large, due to the curse of dimensionality in sampling- -aased inference.
4.7 Ablation Study
Effect of Belief Temperature 3. We examine the impact of the entropy temperature in the belief update, which controls the sharpness of the posterior over opponent types. As shown in Figure [a) moderate values (e-g.. 3 = 1) yield the best performance, striking a balance between confident updates and robustness to uncertainty. When 8 = 0 (uniform), the model performs hard inference, which is brittle early episodes. Conversely, 8 > oo (greedy) leads confident planning, resulting in degraded performance. This result confirms the importance of uncertainty calibration in belief-aware planning.
Opponent Simulation policy (8) To validate the design choice of using a belief-weighted mixture of opponent types during simulation, we compare against three ablations: choosing the most likely type (Best-k), the least likely (Worst-k), and a random type. Note that Best-k and Worst-k are realizable strategies but post-hoc performance bounds, obtained by selecting the best or worst single opponent type in hindsight. As shown in Figure Figure 4[b] the QOM agent using soft marginalization consistently outperforms all alternatives. Heuristic approximations either inject high variance (Random) or suffer from brittle assumptions, especially under limited data. This highlights the benefit of maintaining structured uncertainty over opponent behavior throughout planning.
Quantization Method. We evaluate the influence of different quantization schemes used to dis-
cretize the latent opponent policy space: FSQ 333. LFQ (56), and BSQ 58 Figure/4())ssovvv shows that all methods yield comparable overall performance, but VQ-based QOM achieves slightly more stable results across search budgets. While structured quantization variants may offer theoretical benefits, the core advantage stems from converting continuous trajectory embeddings into discrete types that enable Bayesian reasoning. This suggests the proposed framework is flexible various encoder discretization strategies.
Number of Opponent Types K. We study how the number of discrete opponent types affects performance. As shown in Figure increasing K initially improves return, as larger type enables finer distinctions among opponent behaviors. However, beyond a certain point, additional types provide diminishing returns and may lead to type redundancy, where multiple types generate similar behaviors. This can increase the variance of belief updates and reduce planning efficiency. The result suggests that while increasing K allows modeling more diverse behaviors, excessive number of types leads small distances between them, making the belief less informative and weakening generalization. For more results from other environments, see Section D

---

QOM D0N-mot
00M
GDM w/b mein-pelit
Mean Return
Mean Return
Mean Return
0.1
10
10
10
Search Time
Search Time
Search Time
(a) Root vs Step Sampling
(b) With vs Without Meta-Policy
(c) Effect Quantization
Figure 5: Ablation studies of QOM variants. (a) illustrates that dynamic opponent sampling with belief updates each step enables faster adaptation and higher returns in the SoininngWitt-Seiisson
(b) indicates that the meta-policy substantially improves planning efficiency in the Running- Withh-
Scissors. (c) shows that QOM achieves better performance under tight search budgets owing to efficient belief updates and type compression in the Tursuit-Evasion
Root vs. Step-Level Opponent Sampling To examine the effect of dynamic belief refinement, we compare QOM-step (our default), which re-samples opponent types and updates beliefs at each simulation step, with QOM-root, which samples only once the root and fixes the opponent type throughout the rollout. While the latter approximates marginalizatton over types, it fails to capture evolving posterior beliefs as simulated trajectories unfold. As shown in Figure ((a) QOM-step achieves consistently higher returns across all search budgets, indicating that dynamic belief updates provide clear advantage by maintaining uncertainty latent opponent types during planning.
With vs. Without Meta-Policy We further assess the role of the meta-policy used during simulation. The idea meta-policy originates from POTMMCP; our method extends it by integrating learned quantized opponent model. To isolate this component, we compare the standard QOM (with meta- policy) against variant that uses uniform rollouts (QOM w/o meta-policy). As shown in Figure ((b) removing the meta -olliyy consistently degrades performance, confirming that belief-aware rollout selection enhances planning.
Effect of Opponent Quantization We include QOM-direct, which performs exact categorical Bayesian updates over 50 opponent policies without quantization, using the same PUCT prior, belief smoothing, and history reconstruction. As shown in Figure 5(c) under low computational budgets, QOM outperforms QOM-direct, while their performance converges at higher budgets. This pattern highlights that the quantized representation accelerates belief updates and focuses exploration, whereas the unquantized variant ineurs costly marginalization and slower posterior concentration when policy types are not easily separable.
5 Conclusion and Limitation
We introduced the Quantized Opponent Models (QOM), framework that combines type-based opponent modeling with Monte- Carlo Planning. By learning a discrete set of opponent types and maintaining a Bayesian belief online, QOM enables efficient and adaptive planning under uncertainty. Experiments show consistent gains strong baselines across multiple environments and settings.
However, QOM relies on a fixed policy library and assumes short-term stationarity of opponent policies. Its performance may degrade if opponent policies lie far outside the learned type space. Future work could explore continual type discovery and hierarchical dynamics settings.
Acknowledgments
This work supported by NSFC under Grant 62450001 and 62476008. The authors would like to thank the anonymous reviewers for their valuable comments and advice.
10

---

References
[1] Stefano V Albrecht and Peter Stone. Autonomous Agents Modelling Other Agents: A Compre- hensive Survey and Open Problems. Artificial Intelligence, 258:66-95 2018.
[2] Stefano V Albrecht and Peter Stone. Reasoning about hypothetical agent behaviours and their parameters. arXiv preprint arXiv:1906.11065 2019.
[3] Christopher Amato and Frans Oliehoek. Scalable planning and learning for multiagent pomdps. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[4] David Ball and Gordon Wyeth. Classifying an opponent's behaviour in robot soccer. In Australasian Conference Robotics and Aulomation, 2003.
[5] Charles Beattie, Thomas Köppe, Edgar A Dučfiez-Guzmáán and Joel Z Leibo. Deepmind lab2d. arXiv preprint urXiv:2011.07022. 2020.
[6] Grazia Bombini, Nicola Di Mauro, Stefano Ferilli, and Floriana Esposito. Classifying agent be- haviour through relational sequential patterns. In Agent and Multi-Agent Systems: Technologies and Applications: 4th KES International Symposium, KES-AMSTA 2010, Gdynia, Poland, June 23-25, 2010, Proceedings. Part I 4, pages 273-282. Springer, 2010.
[7] Shengbo Chang and Katsuhide Fujita. A scalable opponent model using bayesian learning for automated bilateral multi-issue negotiation. In Proceedings of the 2023 international conference on autonomous agents and multiagent systems, pages 2487-2489, 2023.
[8] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pages 72-83. Springer, 2006.
[9] Peter Cowling, Edward J Powley, and Daniel Whitehouse. Information set monte carlo tree search. IEEE Transactions on Computational Intelligence and Al in Games, 4(2):120-143, 2012.
[10] Aleksander Czechowski and Frans A Olichock. Decentralized mets via learned teammate models. IJCAI, 2020.
[11] Tuan Dam, Carlo D' D'Eramo. Jan Peters, and Joni Pajarinen. A unified perspective on value backup and exploration monte-carlo search. Journal of Artificial Intelligence Research, 81:511-577, 2024.
[12] Adam Eck, Maulik Shah, Prashant Doshi, and Leen-Kiat Soh. Scalable decision-theon planning in open and typed multiagent systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7127-7134, 2020.
[13] Takuya Fukushima, Tomoharu Nakashima, and Hidehisa Akiyama. Online opponent formation identification based on position information. In RoboCup 2017: Robot World Cup XXI 11, pages 241-251. Springer, 2018.
[14] Arthur Guez, David Silver, and Peter Dayan. Scalable and efficient bdyes-adaptive reinforcement learning based monte-carlo tree search. Journal of Artificial Intelligence Research, 48:841- 883, 2013.
[15] Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In AAAI, volume pages 709-715, 2004.
[16] Yizhe Huang. Anji Liu, Fanqi Kong, Yaodong Yang, Song-Chun Zhu, and Xue Feng. Efficient adaptation in mixed-motive environments via hierarchical opponent modeling and planning. International Conference on Machine Learning.. 2024.
[17] Stephen S Intille and Aaron F Bobick. A framework for recognizing multi-agent action from visual evidence. AAAI/IAAI, 99(518 1--52)) 1999.
[18] Stephen S Intille and Aaron F Bobick. Recognizing planned, multiperson action. Computer Vision and Image Understanding, 4143):414-445, 2001.

---

[19] Anirudh Kakarlapudi, Gayathri Anil, Adam Eck, Prashant Doshi, and Leen-Kinn Soh. Decision- theoretic planning with communication in open multiagent systems. In Uncertainty in Artificial Intelligence, pages 938-948. PMLR, 2022.
[20] Farzad Kamrani, Linus J Luotsinen, and Rikke Amilde Løvlid. Learning objective agent behavior using a data-driven modeling approach. In 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 002175-00218 IEEE, 2016.
[21] Sammie Katt, Frans Oliehoek, and Christopher Amato. Learning in pomdps with monte carlo tree search. In International Conference Machine Learning, pages 1819-1827. PMLR, 2017.
[22] Johannes TN Krebs. A large deviation inequality for 3-mixing time series and its applications to the functional kernel regression model. Statistics & probability letters, 133:50-58, 2018.
[23] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michat Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4501-4510, 2020.
[24] Mare Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan- Davis. OpenSpiel: A framework for reinforcement learning in games. CoRR, abs/ 1t///1988.. 009409933, 2019.
[25] Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game- steeetheereetc approach to multiagent reinforcement learning. Advances in neural information processing systems, 30, 2017.
[26] Kennard Laviers and Gita Sukthankar. A real-time opponent modeling system for rush football. In IJCAI Proceedings-Intermttioon Joint Conference on Artificial Intelligence, volume 22, page 2476, 2011.
[27] Kennard Laviers, Gita Sukthankar, David Aha, and Matthew Molineaux. Improving offensive performance through opponent modeling. In Proceedings of the AAAI Conference Artificial Intelligence and Interactive Digital Entertainment, volume 5, pages 58 58-63, 2009.
[28] Ruonan Li, Rama Chellappa, and Shaohua Kevin Zhou. Learning modtl densities on discriminative temporal interaction manifold for group activity recognition. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 2450-2457 IEEE, 2009.
[29] Tongxin Li, Tinashe Handina, Shaolei Ren, and Adam Wierman. Safe exploitative play with untrusted type beliefs. arXiv preprint arXiv:241 1..06779, 2024.
[30] Xinjie Liu, Lasse Peters, Javier Alonso-Mora, Ufuk Topcu, and David Fridovich-Keil. Auto- encoding bayesian inverse games. arXiv preprint arXiv:2402.08890 2024.
[31] Patrick Lucey, Dean Oliver, Peter Carr, Joe Roth, and lain Matthews. Assessing team strategy using spatiotemporal data. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1366-1374, 2013.
[32] Giulio Mazzi, Alberto Castellini, and Alessandro Farinell.. Risk-aware shielding of partially observable monte carlo planning policies. Artificial Intelligence, 324:103987, 2023.
[33] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309. /5505, 2023.
[34] Ninell Oldenburg and Tan Zhi-Xuan. Learning and sustaining shared normative systems via bayesian rule induction in markov games. arXiv preprint arXiv:2402. 13399, 2024.
[35] Alessandro Panella and Piotr Gmytrasiewicz. Interactive pomdps with finite-state models of other agents. Autonomous Agents and Multi-Agent Systems, 31:861-904, 2017.
12

---

[36] Patrick Riley and Manuela Veloso. On behavior classification in adversarial environments, In Distributed autonomous robotic systems 4, pages 371-380. Springer, 2000.
[37] Patrick Riley and Manuela Veloso. Recognizing probabilistic opponent movement models. In Robot Soccer World Cup, pages 453 -588. Springer, 2001.
[38] Adam Sadilek and Henry Kautz. Recognizing multi-agent activities from gps data. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1134-1139, 2010.
[39] Frederik Schadd, Sander Bakkes, and Pieter Spronck. Opponent modeling in real-time strategy games. In GAMEON, pages 61-70. Citeseer, 2007.
[40] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint /rXiv:/707.000
[41] Jonathon Schwartz, Hanna Kurniawati, and Marcus Hutter. Combining a meta meta-policy and monte-carlo planning for scalable type-based reasoning in partially observable environments. arXiv preprint arXiv:2306.06000 06067, 2023.
[42] Jonathon Schwartz, Rhys Newbury, Dana Kulić, and Hanna Kurniawati. Posggym, 2023.
[43] Jonathon Schwartz, Ruijia Zhou, and Hanna Kurniawati. Online planning for iiteracciivepoomlee using nested monte carlo tree search. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8770-8777. IEEE, 2022.
[44] Behjat Siddiquie, Yaser Yacoob, and Larry Davis. Recognizing plays in american football videos. University of Maryland, 2009.
[45] David Silver, Thomas Hubert, Julian Schritwieser,, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140-11444 2018
[46] Alexey Skrynnik, Anton Andreychuk, Konstantin Yakovlev, and Aleksandr Panov. Decentralized monte carlo tree search for partially observable multi-agent pathfinding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17531-17540, 2024.
[47] Timo Steffens. Feature-based declarative opponent opponent-modelling in multi-agent systems. In RoboCup 2003: Robot Soccer World Cup VII 7, pages 125-136. Springer, 2004.
[48] Timo Steffens. Similarity-based opponent modelling using imperfect domain theories. CIG, 5:285-291, 2005.
[49] Gabriel Synnaeve and Pierre Bessiere, bayesian model for opening prediction in rts games with application to starcraft. In 201 / IEEE Conference Computational Intelligence and Games (CI pages 281-288. IEEE, 2011.
[50] Ebert Theeuwes, Gabriele Venturato, and Gavin Rens. Belief re-use in partially observable monte carlo tree search. In The 17th International Conference on Agents and Artificial Intelli- gence, 2024.
[51] A Torkaman and R Safabakhsh. Robust opponent modeling in real-time strategy games using bayesian networks. Journal of Al and Data Mining, 7(1):149-159, 2019.
[52] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[53] Alexander Vezhnevets. Yuhuai Wu, Maria Eckstein, Rémi Leblond, and Joel Z Leibo. Options as responses: Grounding behavioural hierarchies in multi-agent reinforcement learning. In International Conference on Machine Learning, pages 9733-9742. PMLR, 2020.
[54] Ben G Weber and Michael Mateas. data mining approach to strategy prediction. 2009 IEEE Symposium Computational Intelligence and Games, pages 140-147. IEEE, 2009.
13

---

[55] Xinyu Wei, Patrick Lucey, Stuart Morgan, and Sridha Sridharan. Sweet-spot: Using spatiotem- poral data discover and predict shots in tennis. 7th Annual MIT Sloan Sports Analytics Conference, Boston, MA, 2013.
[56] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng. Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion-tokenizer is key to visual generation. arXiv preprint arX1v:2310.057.7,, 2023.
[57] Xiaopeng Yu, Jiechuan Jiang, Wanpeng Zhang, Haobin Jiang, and Zongqing Lu. Model-based opponent modeling. Advances in Neural Information Processing Systems, 35:28208-28221, 2022.
[58] Yue Zhao, Yuanjun Xiong, and Philipp Krähenbühl. Image and video tokenization with binary spherical quantization. arXiv preprint arXiv:2406.07548, 2024.
[59] Xinlong Zheng, Xiaozhou Zhang, and Donghao Xu. Speeding path planning via reinforce- ment learning mets for automated parking. 2024 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 5410-5415. IEEE, 2024.
[60] Maddalena Zuccotto, Edoardo Fusa, Alberto Castellini, and Alessandro Farinelli. Online model adaptation in monte carlo tree search planning. Optimization and Engineering, pages 1-22, 2024.
14

---

A Theoretical Analysis
In this section, we provide theoretical guarantees for the proposed QOM method, establishing how the autoencoder-bassd quantization affects learning performance. We follow the standard analysis structure with notation and assumptions, then present the key lemmas and theorems. Throughout, explicitly highlight the influence of the quantization error on posterior concentration.
Notation. Let μ² denote the true underlying distribution or model of the environment. In the context of QOM, μL" can be considered the true (unknown) distribution over high-dimensional observations. The autoencoder compresses observations into a finite codebook of K prototypical points (or latent states); we denote this approximate model distribution by p. The distribution f places probability mass Tt. on the k-th code (for k = l................... K), which aims approximate the true mass gy that μ assigns to the region represented by that code. We write (z( for the probability density (or mass) of observation under distribution μ.. Notation (L(#" I|)) denotes the Kullback-Leibe (KL) divergence of from jx".
Corresponding to the discussion in Section 3.4 the distribution f can be seen as the belief (k) given a fixed opponent pool { k=1- The observation æ represents the opponents' history z = (h_ a-1).
Therefore, the probability Pp (z) can be formulated as
Pa(x)= Σ
by (a-
&
4 corresponds to the unknown ground truth belief 6n- (l)
The QOM agent maintains a Bayesian posterior Pr(u) x1:)) over models / given the data X1:t up to time We define (T) := log T log log T, a slowly growing function of the horizon that will appear as correction factor due to temporal dependence. For clarity, |- denotes Eg norm unless stated otherwise.
Assumptions. We assume the following conditions hold in the QOM model setting:
Assumption I (Boundedness and ß-mixing). The sequence of observations (or contexts) {X,)=_ is generated from 3-mixing process with mixing coefficients decaying sufficiently fast (c.g. ((m) = O(e-eR) for some >> 0), ensuring weak dependence across time. Moreover, all relevant random variables are bounded: in particular, there exists B >> O such that for all and for all candidate models 14, the lig-liielihood ratio per observation is bounded by B:
Pu (X,) (X,)
log
3 B
almost surely.
(8)
Py-
This implies the rewards or losses are also bounded (for instance, if rewards r, E [0, B] or the maxi- mum reward gap is bounded by B). These conditions allow us to apply Bernstein-type concentration inequalities even though data are dependent over time.
Assumption 2 (Autoencoder multiplicatiie approximation). The autoencoder provides a multiplicative reconstraction guarantee: there exists a fixed & E 10, (the quantization error parameter) such that for every observation X, and its autoencoder reconstruction X,, the relative error uniformly bounded:
IX x. < E eIll,
Vt=
In words, the reconstruction is accurate up factor & of the original magnitude every instance. This assumption implies that the approximate distribution f induced by the codebook is close to the true distribution 1* in a multiplicative sense. For example, we can reasonably assume (perhaps by a Lipschitz continuity argument on pn (x:) that for all x,,
(1- (z-()) S Pa(2) S (1+
(10)
That is, the density under the QOM model does not deviate from the true density by more than a dif relative factor at any point. Finally, it in KL) lies within the support of our Bayesian prior formally, the prior assigns sufficient mass to a small KL neiheeibbordoodd of μ². This ensures the posterior is well-behaved around the truth.
assume the true model μ² (or distribution arbitrarily close to
Under these assumptions, we now derive step-by-step the main theoretical results: bound on the KL divergence due to quantization (Theorem [.. a posterior concentration rate (Theorem 2 Each result explicitly incorporates the 3-mixing correction ((T') and the quantization error &.
15

---

Lemma (KL divergence under quantization). Under Assumption 2, the KL divergence between the true distribution μ² and the autoencoder-quaanee model bounded by a small constant on the order of g². In particular, one may show
KI(P"IlA) S - llo((1 -E) = 5 +0(E) 1-
Consequently, for sufficiently small &, KI()*lI) is O(g) (and in particular O(52) to leading quadratic order).
Proof. Since Pa (x) everywhere pointwise approximates Pur (x) to within a (1 = factor, the KL divergence can be bounded by the worst-case log discrepancy. By definition,
(I(P"IIA f Pp. (xx) log Pe- (x:) dr.
(11)
Pa (x(x)
Given the multiplicative bounds, we have Pu)(2)
Pa(#) $ I-
1. Fur- L for all x. Thus log Pu- S log I-
thermore, on the other side (=) < (1 E)px-(a) (x) implies P.- 2 I+E so log Pv 2 log I+E Ps
-log(1 1 + E). Since - llog(1 + {) < - log(1 E) for 0 < E < 1, the maximal absolute diver- gence occurs when (z( is smaller than Pp- by factor (1 - E). In the worst case (if this held for all 2)), [I())) log - In reality Pa will not be uniformly at the extreme bound for all 2, so (IL()*llA will be strictly less than this worst-case. In any case, log I-g expands to
E+ & +... E/(1 E), giving the stated bound. Thus the divergence between the true distribution and the QOM's approximate distribution is on the order of the small quantization error &. as claimed.
Intuitively, Lemma I shows that the autoencoder-baat quantization does not significantly distort the problem from an cnormattio-theoretiii perspective: the true environment jx" is almost as likely under the quantized model class as it is under the truth (their KL divergence is very small for small f). This fact will be crucial proving posterior concentration, since it guarantees that the true (or near-true) model is not "ruled out" by a large divergence.
Theorem 2 (Posterior Concentration under QOM). Under Assumptions I and 2, the Bayesian posterior concentrates around the true model μ² at a rate accounting for the dependence and quantization, In particular, there exist positive constants Ci, C2 such that for sufficiently large T, with high probability (approaching as + oo):
Pr(u: KL( Ci ((T) X1:) S exp( C2
)
T
up lower-order terms. Equivalently, the posterior places almost all its mass on those models μ that satisfy KL( = O log T log log T ) In particular, the bulk of the posterior mass lies within an O Tog Tog neighhoooodoo of p². Combined with Lemma ] this implies that the posterior effectively concentrated near the true μe' (or an equivalent fa that is E-close to μ²' up to an asymptotic approximation error on the order c.
Proof. The proof is based on Bernstein-tyre -ttpe large deviation inequality for f-mixing sequences, which allows us to handle the dependent observations. At a high level, we need to show that for any candidate model that is significantly different from μz" (say, (L(u*))e) is larger than some e), the posterior probability of p becomes negligibly small as T grows. By Bayes' rule,
T
Pe (X,)
Pr(p X1a) x Pr(n)
II
Pue (X,)
16

---

where Pr(μ) in the numerator is the prior density at j.. Consider any model l with (L()"Ila)
?) > Define the lokeliiiiooo ratio sum:
T log
(x() Py- (Xt)
(T(
Σ
log
t=1
The expected value of this sum under the true distribution is negative and scales with T:
log Pa(X,)
E [Ir(N) Σ By-
Py- (Xx)
i=1
= = (T.K((1)))))) (from and Assumption 1 that (X,) is f-mixing process)
-TT
(12)
We want to show that Lr(H) is very likely be much less than 0 (i.e. strongly negative), so that Pa(X,) IP (x.) exp(LT(u)) is extremely small, overcoming even nonzero prior mass on To this end, we apply concentration inequality for ß-mixing processes. Using result of [2 large deviations for 8-mixing time series (a Bernstein-type inequality accounting for dependence), one can
show that for any E (0,1):
Pr (L((1))) > -TTn) s exp( C
Tn Φ(T)
for constant C > depending the mixing parameters (including Ug) and the bound B. In words, with probability at least - exp(- (C))) (TT), log-likelihood ratio for j is strongly negative (at least half of its negative expectation). When this event holds, the posterior weight of y exponentially small:
have Lr(u) < - meaning the
Pr(μ] X1:1) Pr(μ) exp(Lr (N)) Pr(u)) exp(-1Tm)
Now, provided the prior Рr(μ)) is not pathologically large (which we can ensure since the prior is fixed and 1) constant gap), the right-hand side decays super-pot poper--olynomilllly in T. We can union-bound this argument over suitable s-net covering the set of all models with KL(μ") [|p)) > to conclude that, with high probability, no such model has substantial posterior mass. More formally, for any 8 >>0 0:
Pr N²
Σ
Pr(u X1a) =
+ oo),
(IcKT.(p(1e))>]]
for arbitrary small 0. This implies that asymptotically, Pr(KL(H(|]p) > | converges to
0. By choosing e on the order of C e(T) (the "critical radius" where the exponential bound starts
T to kick in, balancing Tиj against (TT) in the exponent above), we obtain the stated rate of posterior contraction. summary, the posterior concentrates in KL-ball around μz' radius O log Tlog T). Finally, since 1u" is within & (KL-)ddsttance of some model in our support (by Lemma T f satisfies (L(H*) [] = O())), the posterior effectively centers around the ğ that approximates jt", which in turn is É-close to μ². Thus the presence of the quantization error & does not prevent concentration; it only imposes that the posterior cannot distinguish models closer than O to μ² thereby introducing small OgE) uncertainty floor even as T - co.
Theorem Theorem@esttann establishes that QOM's 's Bayesian posterior rapidly concentrates around the true environ- ment's distribution, up to the minor resolution limit imposed by quantization. The 3-mixing condition causes logarithmic slow-down (the ((T) log T log log T factor) compared to the iid case, but this is a very mild penalty. Importantly, the autoencoder's 's approximation error & enters only as an additive term in the final concentration radius, meaning that aside from an O(E) neighborhood that
17

---

the posterior cannot distinguish, the uncertainty shrinks at the usual (11/T) rate (up to log factors). In practical terms, as long as Ç small, the posterior quickly hones in on model that is essentially as good the truth for decision-making purposes.
B Algorithm
Algorithm 2 Planning with Quantized Opponent Models
1: Input: initial agent history hr opponent history A belief Greal (k)
2:
for each real environment step 0, 1, 2..... do
3: //Run simulations for action planning
4: while within computational budget do
5:
Initialize rollout history h; +- 11 hh: +- 1
6:
Initialize rollout belief b(k) <- greal (k)
7:
while node at h; is not terminal and depth has not exceeded maximum limit do if node h, is not yet expanded then
8:
9:
Expand new child nodes at h2 for each legal agent action
Initialize visit counts N (hi, a,) 0 and action-values Q(hi, a;)
11:
end
12:
Sample opponent latent type k b(k)
13:
Sample opponent action a_, ~ 7t(
h h_))
Compute meta-policy prior π. metu h2
15:
Compute PUCT scores as Equation 6 for all a;
Select agent action: a; arg maxa, Append (h; a,) to search path Step environment simulator: (s', df d.::
(h4, a;)
17:
Update rollout history: h; h4 U (0), h-i t h (o a-1)
20:
Update belief b(k) by Equation end while
22:
//Perform rollout evaluation and update planning statistics
for each (h2, a;) the search path (in reverse order) do
24:
Q(1y, a.) & Q(hi, az) x N(hs, aa) + V
N(h,,
N(h, a.) + N(hs, +1
25:
Update cumulative return: V +
26:
end for
27:
end while
28:
Select final action to execute: a: = arg max,, N N(h a:)
29:
//Execute selected action and update real histories and beliefs Execute af in the real environment
30:
Observe agent observation of (t+1) and opponent action a
31:
32:
Infer opponent observation _ by g
Update history: Agtal <- Meeal U (o) ((+1) .af) A.ral <- hr U (o (o_)a_0)
33:
34:
Update real-world belief graal (k) as Equation 5) and apply belief smoothing
35:
end for
18

---

C Experiments Settings
C.1 Environments
(a) PursuitEvasion
(b) PredatorPrey(two-ageets,
(c) Predato/Prey(four-geen)))
(d) Running-wie Sratinngwwtt--aeee with-Scissors
(c) One-on-One
Figure 6: Environments
To evaluate method across broad range of multi-agent scenarios, consider four partially observable environments spanning gridworlds, game-theoretic interactions, and high-dimensional control, All environments are sourced from established open-source platforms, including POSG-
Gym 42), DeepMind Lab2D 5) and Google Research Football 23 Figure 6 shows example visualizations.
Pursuit-Evann Evasion. This environment features two agents pursuer and mander-navigatiee maze-like gridworld with limited observability. At each timestep. agents receive localized and noisy observations: wall proximity in four directions, directional visual cues, and auditory signals within two-cell radius. The evader earns small rewards for advancing toward a safe zone, while the pursuer rewarded for capturing the evader and penalized for failing hinder progress. An episode ends when the evader escapes, captured, or 100 steps elapse.
Predator-Prey. We consider two variants: one with two predators and another with three. Agents operate in gridworld with deterministic dynamics and local 5 x 5 observations. Each episode includes three prey, and predators must coordinate to capture them. the two-agent case, both predators must occupy the prey's 's cell; in the three three- predator variant, at least three agents required. Episodes conclude when all prey are caught or after 50 steps. Success hinges on coordination and inferring prey intentions under partial observability.
This environment extends the classic rock-paper-sciiee paper-scissors game into spatial, time-extended setting. Agents navigate 13 21 grid containing 36 resource tiles-rock paper, scissors-hal of which are randomized each episode. Agents observe a 4 x local region and collect resources by stepping on tiles. Fach starts with one unit of each resource. Agents can "mark" 3 x 3 zone in front of them to challenge opponents, resolving confrontations via standard
Sumning-witt-Seissors...
19

---

RPS rules. Episodes last up to 500 steps end upon successful mark. Partial observability. randomized resources, and delayed interaction create rich strategic environment requiring agents to infer and adapt to opponents" evolving inventories.
One-on-One. In this soccer-inspired setting, an attacker and a goalkeeper face off duel. The attacker controls the ball and may dribble or shoot at any time, while the goalkeeper must intercept. The attacker earns +1 for scoring; otherwise, the reward goes to the goalkeeper. We control the goalkeeper, treating the attacker as non-stationary opponent. The continuous state-action space, visual inputs, and stochastic dynamics pose significant challenge for belief-aware planning under opponent uncertainty.
C.2 Policy Libraries and Payoff Matrix
1.00
0.75
Policis
Payolf
0.00
5-
-0.25
7-
0.50
13
14
18
19
20
23
24
0.50
Palciss
0.25
Payoff
0.00
8-
0075
-1.00
Opporenn Policies
(a) Empirical payoff matrix
1.00
0.75
Agent Poliches
0.00
5
-0.25
0.75
-1.00
2
6
10 11
13
14
15
Oppanent
(b) Latent-type payoff matrix
Figure Empirical payoff matrix and datent-type payoff matrix in Running- motning-witt--oe with-Scissoor
20

---

Building the policy libraries. Before planning begins, we generate both an agent policy library and an opponent policy library with Policy-Space Response Oracles (PSRO) 25 25|24] PSRO proceeds in rounds: every iteration it trains a best-response policy to the current mixture of opponents and then adds this new policy to the empirical game. After PSRO iterations we have 10 distinet agent policies and 10 corresponding opponent polices. All oracles are trained with Proximal Policy Optimisation (PPO) [40]. using the same network architecture and
reported in Section C.3 To widen behavioral coverage further enlarge the opponent library with intermediate PPO checkpoints saved during training. Including these extra snapshots yields fifty opponent policies in total, spanning broad spectrum of play styles from early exploration converged behavior.
Constructing the payoff matrix. Running PSRO already produces empirical payoff table that records the return obtained by each agent policy against every opponent policy. When Quantized Opponent Models (QOM) later need a payoff matrix indexed by latent opponent types, we simply reuse these existing simulation results, as Equation 4.. Each opponent trajectory is labelled with latent type by the quantized auto-encoder; returns from trajectories that share the same label averaged, giving payoff entry for that latent type. This recycling step eliminates the need for any additional rollouts while keeping the statistical link between the PSRO game and the aatent-type abstraction used by QOM. We plot the payoff matrix in Running-wing Seih--Siissor, as shown in Figure 7
Unseen Opponent Policies. For the generalization experiments, the unseen opponent policies generated from three additional rounds of PSRO training, independent those used to construct the QOM training library. These extra policies are excluded from the quantized opponent model and introduce novel strategic behaviors arising from different PSRO dynamics across runs. This setup enables evaluation of QOM's generalization to previously unseen opponent strategies beyond the fixed policy catalog.
C.3 Hyper-parameters
Table 1: Hyperparameters
Parameters
Pursuit-Evasion
Predator-Prey
RWS
One-on-One
hidden units
MLP(64. 32]
MLP(64. 32]
MLP(64, 32]
MLP(64. 32]
activation function
ReI.U
ReI.U
ReI.U
ReI.U
optimizer
Adam
Adam
Adam
Adam
PPO learning rate
0.000S
0.0005
0.0005
0.001
target update interval
10
10
10
10
value discount factor
0.99
0,99
0.99
0.99
GAF parameter
0.99
0.99
0.99
0.99
clip parameter
0.115
0.115
0.115
0.115
max grad norm
0.5
0.5
0.5
0.5
method
PSRO
PSRO
PSRO
Paired Policy Archive
meta strategy method
Q-rank
a-rank
a-rank
Policy Set
sims per entry gpsro iterations
100
100
100
10
10
10
11, 11:1
10
10
10
10
50
50
50
50
learning rate reconstruction weight
0.0001
0.0001
0.0001
0.0001
batch size letent types temperature 3
64
64
64
64
QOM
16
16
1
exploration constant e
2
2
depth
20
20
20
20
unit time
1s
Is
2s
5s
21

---

The encoder of the VQ-VAE is implemented as two-layer GRU with a hidden dimension of 64, which processes the opponent trajectories and maps them into continuous embeddings IR Fach embedding is quantized via nearest-neighhor assignment to codebook with K=16 entries, producing discrete latent type 6t- The decoder is GRU-based autoregressive model conditioned on eg; its hidden state is initialized from learned projection of cg. and it outputs see-wise likelihoods of opponent actions 7 (a_; h). The quantizer is trained using the standard VQ- VQ-VAE objective, which combines reconstruction, codebook, and commitment losses, optimized with straight-through estimator and exponential-moviee moving-average codebook updates.
All experiments were executed on two dedicated workstations. For the PE, PP, and RWS environments we used an Intel Core i7- 17-12700KF (12th Gen, 3.60 GHz) paired with an NVIDIA GeForce RTX 4060 Ti. The One-on-Opp One experiments ran on server equipped with an Intel Xeon E5-2620 v4 (2.10 GHz, cores) and NVIDIA TITAN Xp.
22

---

D Extra Results
D.1 Scalability
[n.1= 30
|n.,1 50
10.11
100
n 200
un_| 300
Mean Return
0.75
Mean Return
0.6
0.50
0.4
0.25
0.2
0.00
0.0
-0.25
-0.2
0.1
1
5
10
20
0.1
i
5
10
20
Search Time
Search Time
(a) PE(Evader)
(b) PE(Pursuer)
Mean Return
1.00
Mean Return
0.8
0.75
0.6
0.50
0.4
0.25
0.2
0.00
0.0
0.1
1
5
10
20
0.1
10
20
Search Time
Search Time
(c) PP(two-agents)
(d) PP(four-agents)
Mean Return
0.4
Win Rate
2.5
0.2
5.00
0.1
7.5
0.1
10
20
0.1
5
10
20
Search Time
Search Time
(c) Running-Wing With-Scissors
(f) One-on-One
Figure 8: Scalability with policy library size
Figure tracks how planner behaves when give it more thinking time while also varying how many opponent policies it must consider. When the library is tiny (III=] = 30), the curve levels off early: there simply
enough opponent styles in the set, the agent soon hits ceiling. At the other extreme (ITT-|l = 300), the first few points are lower because the agent must sift through many more hypotheses, but the time budget grows this setting catches up and finally matches the mid-sized libraries. Libraries of 50 100 policies combine wide coverage with a moderate inference cost, so they reach high returns the fastest across all six tasks. Most importantly, every curve keeps climbing and the error bars stay narrow, which means the belief update of QOM remains stable even when the number of possible opponents grows
23

---

D.2
Ablation of Belief Temperature 3
0
B= 0.1
B=1
B 5
B=
0.8
0.75
Mean Return
Mean Return
0.6
0.50
0.4
0.25
0.2
0.00
-0.25
0.0
0.1
10
20
0.1
20
Search Time
Search Time
(a) PF(Fvader)
(h) PF((ursuer)
Mean Return
1.00
Mean Return
0.8
0.75
0.6
0.50
0.4
0.25
0.2
0.0
0.1
5
10
20
0.1
1
5
10
20
Search Time
Search Time
(c) PP(two-agents)
(d) PP(four-agents)
Mean Return
2
Win Rate
0.3
-2-
0.2
0.1
0.0
0.1
5
10
20
0.1
5
10
20
Search Time
Search Time
(c) Running-With-Seise With-Seissors
(f) One-on-One
Figure 9: Ablation study on belief temperature $
For belief temperature 3, we conduct additional ablation across all environments Figure 9 Across all tasks, moderate entropy levels (e.g., 33 1) consistently lead to stronger performance, confirming the generality of the trend observed in the main text. Both overly sharp posteriors (6 = -oo) and overly diffuse ones (3 0) result in degraded returns win rates. This highlights the importance of planning in diverse strategic settings.
24

---

D.3
Ablation of Opponent Simulation Policy म
(k)
QOM
Best k
Worst_k
Random k
0.8
Mean Return
Mean Return
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-0.2
0.1
1
10
20
0.1
i
5
10
20
Search Time
Search Time
(a) PE(Evader)
(b) PE(Pursuer)
1.0
Mean Return
Mean Return
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
10
0.1
10
20
Search Time
Search Time
(c) PP(two-agents)
(d) PP(four-agents)
2.5
Mean Return
Win Rate
0.4
0.0
0.3
-2.5
0.2
-5.0
0.1
-7.5
20
0.1
5
10
20
Search Time
Search Time
(c) Running-Winde With-Scissors
(f) One-on-One
Figure 10: Ablation study on opponent simulation policy (R)
Figure 10 shows how different ways of using the latent type information affect planning quality. QOM reaches higher scores more quickly than the other three choices and keeps that lead until the longest time horizon. The advantage is largest when the time budget short: with only 0.1 or 1 unit of thinking time, QOM already gives clear positive return, while Worst-k remains near zero and Random-k rises only slowly. Variability across runs is also lower for QOM. indicating that sampling from the belief offers more stable balance between exploration and exploitation than any single-type shortcut.
25

---

D.4 Ablation of Quantization Methods
QOM
QOM-FSQ
QOM-LFQ
QOM-BSQ
Mean Return
0.75
Mean Return
0.6
0.4
0.2
0.0
-0.25
-0.2-
0.1
1
10
20
0.1
1
5
10
20
Search Time
Search Time
(a) PF((vader)
(h) PF((Prsuer)
1.0
Mean Return
Mean Return
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.0
20
Search Time
Search Time
(c) PP(two-agents)
(d) PP(four-agents)
Mean Return
2-
Win Rate
-2
0.2
0.1
0.0
-6
0.1
5
10
20
Search Time
Search Time
(e) Running- Boiminng-Wtt--ee With-Scissors
(I) One-on
Figure 11: Ablation study on quantization methods
For different quantization methods we keep the planning code unchanged and only swap the vector- quantizer with its compressed variants FSQ, LFQ, and BSQ. Figure I1 shows that heavier compression costs performance. The gap is most visible in pursuit-evasion and the two-agent patrol task, where fine-grained cues about the opponent are important early on. As the search budget grows, all curves move closer, yet QOM still ends on top or tied by the last point. Overall, the experiment confirms that keeping more information in those codes yields higher and more stable rewards, while stronger compression trades some of that gain for smaller codebooks.
26

---

D.5
Ablation of latent type size K
==8 8
1=
K=
32
0.75
0.75
Mean Return
Mean Return
0.50
0.50
0.25
0.25
0.00
0.00
-0.25
-0.25
-0.50
0.1
1
10
20
0.1
1
10
20
Search Time
Search Time
(a) PF((vader)
(h) PE(Pursuer)
Mean Return
Mean Return
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
-0.25
-0.25
0.1
Search Time
Search Time
(c) PP(two-agents)
(d) PP(four-agents)
Mean Return
0.4
Win Rate
0-
0.2
-2
0.0
4-
-6
-0.2
0.1
5
10
20
0.1
10
20
Search Time
Search Time
(c) Running- With-Scissors
(f) One-on-One
Figure 12: Ablation study on latent type size
Figure 12] studies how the number K of latent opponent types influences performance. When the codebook holds 16 types the agent posts the highest scores in every game and at every search-time setting. Using only 8 types comes close, but loses few points once the search grows longer, suggesting that some opponent behaviors are not represented. When the codebook contains more latent types, early returns dip, and even with extra search time the larger setting remains behind the smaller ones. These results point to simple trade-off: a small K can miss important strategies, while very large K spreads the belief over too many possibilities, making each simulation less focused. A medium value offers enough variety to recognise the opponent without overloading the search, giving the best balance between coverage and efficiency.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
28

---

Justification:
Section A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section
4 and Section C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification:
supplemental material
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwiiicgguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
30

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: After careful comparison with the NeurIPS Code Ethics, we were confident that no guidelines were violated.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All basclines and environments cited with the original paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
33

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
34


Reviews:
[{"review_id": "1pqrDwSRWq", "reviewer": "Reviewer_ETPj", "summary": "This paper proposes a method for planning in multi-agent stochastic games. The method learns an encoder that maps opponents' policies into a discrete set of latent vectors, each representing a distinct agent archetype. Importantly, this embedding space is significantly smaller than the original policy space. Together with the encoder, the method also trains a decoder that reconstructs a policy for each latent vector. The planner then uses the decoded policies to compute a belief over the discrete set of agent types, based on the likelihood of the opponents' action history. This belief is then used to sample policies and plan a best response. The empirical results reported in Figure 2 suggest that the proposed method outperforms the selected baselines.", "questions": "1. According to Algorithm 1, the opponent’s policy may change within a single rollout. Wouldn’t it make more sense to sample an opponent policy once at the root node and keep it fixed for the entire rollout? As it stands, the planner could be reasoning over trajectories that do not correspond to any specific opponent type, but rather to a mixture of types, which may not be meaningful or valid for planning.\n\n2. The paragraph starting at Line 169 explains that, to mitigate the effects of noisy observations or approximation errors, a smoothing operation is applied after the belief update to keep the belief close to the initial belief. Could the authors clarify why noise or approximation error would affect the belief in this way? This seems counterintuitive, since the belief is computed exactly using Bayes’ rule, and while noise may influence the observations, it should not affect the likelihood of actions under the decoded policies. Additionally, the ablation study (\"Effect of Belief Temperature\") does not clearly demonstrate the utility of this smoothing, as the error bars in Figure 4a appear to overlap significantly.\n\n3. The idea of constructing a meta-policy seems somewhat orthogonal to the main contribution of the paper. Could the authors clarify the motivation for including this component and how it connects to the core idea of quantized opponent models? Additionally, in Line 205, the authors argue that using the meta-policy should improve sample efficiency. While I agree that, if the set of agent policies is good, planning efficiency may improve, it seems that this merely shifts the computational burden from online search to offline policy training and payoff matrix estimation. Could the authors comment on this trade-off?", "limitations": "Limitations are covered in the last section.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**  \nThe paper tackles an important problem: modeling opponent policies to adapt and plan the best response to specific types of opponents. The idea of projecting opponent policies into a discrete set of policy archetypes is compelling and novel.\n\n**Weaknesses**  \n* Experiments: This is, in my opinion, the paper's biggest weakness, as some of the claims in the experimental section are not fully supported by the empirical evidence. A few points:\n  - There is significant overlap in the error bars in Figures 3 and 4, making it difficult to draw any definitive conclusions about the relative performance of the methods. Additional runs are needed to make such claims more reliable.\n  - Given that the opponent library consists of only 50 policies, it would be worthwhile to include a baseline that uses the library policies directly to update the belief.\n  - More details should be provided on how the agent and opponent policy libraries are constructed, along with evidence demonstrating that the policies are diverse.\n\n* Technical concerns: Some of the algorithmic choices and problem formulation decisions are questionable. Specifically:\n  - Sampling multiple opponent policies within a single trajectory rollout in Algorithm 1. See Question 1.\n  - Belief smoothing as described in Line 169. See Question 2.\n  - Meta-policy construction in Section 3.3 seems to shift the computation from offline to online. See Question 3.\n  - The payoff matrix is defined for each combination of agent and opponent types, but not for each state of the environment. As a result, the meta-policy favors agent policies that perform well on average across states, which may not be optimal for the agent's current state.\n  - Although the method targets partially observable stochastic games involving multiple agents, all other opponents are treated as a single composite agent, and only one belief is maintained across them. This raises the question of why the problem is not instead formulated as a two-agent game.\n\n**Minor Comments**  \n- In the first ablation study, is $\\beta$ intended to be $\\lambda$?\n\n**Related Work**  \n- A. Czechowski and F. A. Oliehoek, 2020. *Decentralized MCTS via Learned Teammate Models.*", "was_revised": true, "final_justification": "After reading the authors’ response, I still stand by my assessment. The initial experimental section was weak (e.g., overlapping error bars), lacked important baselines for comparison, and included strong claims that were not adequately supported by the results. The authors addressed some of these issues in their response and provided preliminary results. However, given that the changes required to the original submission would be quite substantial, I believe a new round of review would be necessary.", "rebuttal": "Thanks for your valuable comments. Below is a detailed response to your question, addressing each point individually. Owing to space constraints, we reference your comment using its opening words.\n\n> There is significant overlap ...\n\nThe RWS environment is highly stochastic, resulting in large variance and wide confidence intervals. Even with 300 runs, QOM outperforms baselines in Figure 3 (Adaptation and Generalization). Increasing to 1000 runs tightens CIs and reinforces statistical robustness. Figure 4, which analyzes hyperparameter and component sensitivity, retains valid trends despite observed variance.\n\nWe increased the number of runs to 1000. Results are shown below. This larger sample yields tighter intervals. Results for other environments will be added in the revision.\n\n**Table: Extended results for Figure 3(a) — Adaptation evaluation in RWS with 1000 runs**\n|Search Time|0.1|1|5|10|20|\n|---|---|---|---|---|---|\n|QOM|-3.91 ± 0.94|-2.82 ± 1.09|-1.53 ± 0.80|-0.09 ± 0.80|0.20 ± 1.07|\n|POTMMCP|-5.11 ± 1.08|-4.32 ± 0.79|-2.66 ± 0.89|-1.44 ± 1.01|-1.51 ± 0.90|\n|I-POMCP-PF|-5.60 ± 0.80|-4.34 ± 1.15|-3.85 ± 0.93|-3.54 ± 0.95|-2.93 ± 0.93|\n|MBOM|-4.08 ± 0.79|-3.15 ± 1.07|-1.44 ± 0.92|-1.03 ± 1.01|0.19 ± 0.95|\n\n> Given that the opponent library ...\n\nWe added QOM-direct, which performs exact categorical Bayes updates over 50 policies without quantization, using the same PUCT prior, belief smoothing, and history reconstruction. Under low budgets, QOM significantly outperforms QOM-direct; under higher budgets, performance converges. This suggests QOM’s type compression enables faster belief updates and focused exploration, while QOM-direct suffers from expensive marginalization and slow posterior concentration when policies are not clearly separable.\n\n**Table: Performance comparison between QOM and QOM-direct in PE(Evader)**\n|Search Time|0.1|1|5|10|20|\n|---|---|---|---|---|---|\n|QOM|0.10 ± 0.08|0.59 ± 0.07|0.69 ± 0.08|0.69 ± 0.07|0.72 ± 0.10|\n|QOM-direct|-0.26 ± 0.09|0.01 ± 0.06|0.06 ± 0.10|0.12 ± 0.09|0.16 ± 0.09|\n\n> More details should be provided ...\n\nPolicy libraries are built via PSRO: each round trains a best-response against a policy mixture. We obtain 10 agent and 10 opponent policies, all trained via PPO with shared architecture and hyperparameters. To broaden coverage, we add intermediate PPO checkpoints, resulting in 50 opponent policies spanning exploration to convergence. Details are in Appendix C.2.\n\n> According to Algorithm 1 ...\n\nWe understand the concern that, under Algorithm 1, a single simulation rollout may involve opponent actions that are generated from different latent types, rather than a single consistent policy throughout. This is indeed a deliberate design choice in QOM.\n\nWhile it is common in type-based planning methods (e.g., POTMMCP) to assume that the opponent adheres to a single fixed strategy sampled from a predefined library, QOM adopts a fundamentally different modeling philosophy. Rather than assuming the opponent follows a specific type from the quantized library, QOM maintains a posterior belief $b_t(k)$ over types and uses this belief to simulate opponent behavior at each decision step (Algorithm 1, line 7). In other words, each rollout samples an opponent action from a posterior‑predictive mixture, where the type is drawn per step according to current uncertainty. This simulates behavior from a belief‑weighted model $p(a_{-i}\\mid h)=\\sum_k b_t(k)\\tilde\\pi_k(a_{-i}\\mid h)$, and is consistent with our modeling approach in Section 3.4 and Equation (7).\n\nThere are two reasons we adopt this per-step marginalization. First, the quantized types in our library are not assumed to correspond to ground-truth opponent identities. Especially early in an episode, the posterior is uncertain or multi-modal, and marginalizing across likely types allows the planner to hedge its expectations. Second, this design is empirically justified: as shown in Figure 4(b), QOM — which marginalizes over types during simulation — outperforms heuristics that fix a specific opponent type during rollouts (e.g., “Best‑k”), across simulation budgets. In response, we will add a root-sampling variant as an ablation in the revision to evaluate its effect.\n\n> $\\beta$ intended to be $\\lambda$ ?\n> The paragraph starting at Line 169 ...\n\nThank you for pointing out the notation clash. In the first ablation, the parameter is the belief temperature used in the Bayes update of the type posterior. We now make this explicit by tempering Eq. (5) as $b_{t+1}(k)\\ \\propto\\ b_t(k)\\,\\big[\\tilde\\pi_k(a_{-i}^{(t)}\\mid h_{-i}^{(t)})\\big]^{\\beta},$ so that $\\beta=0$ recovers the uniform prior, $\\beta=1$ yields the standard Bayes update, and $\\beta\\!\\to\\!\\infty$ collapses to a greedy posterior. To avoid overloading notation, we use $\\beta$ only in Eq. (5) and no longer in Eq. (2); the smoothing weight $\\lambda$ is a separate post‑update mixing parameter and is unrelated to $\\beta$.\n\nRegarding above “close to the initial belief,” this was a misunderstanding arising from the earlier notation issue. The purpose of belief smoothing is not to bias the posterior toward the prior per se, but to ensure that all types retain non‑zero probability mass. Although the update follows Bayes’ rule, the likelihoods are approximate: (i) they come from a learned decoder $\\tilde\\pi_k(a\\mid h)$ over quantized types, and (ii) the opponent’s history $h_{-i}$ used by the decoder is reconstructed via the simulator under partial observability. Model misspecification and reconstruction error can make early posteriors spuriously sharp after a few highly discriminative observations; once certain types are driven near zero, recovery is difficult with finite samples. Smoothing therefore acts as a robust‑Bayes regularizer that avoids zero‑probability assignments while the agent is still gathering information. Our theoretical analysis also shows that the quantization error introduces an $O(\\xi)$ uncertainty floor, which further justifies a small amount of robustness in the early phase.\n\n> The idea of constructing a meta-policy ...\n\nWhile it may appear that the meta-policy component is orthogonal to the main contribution of quantized opponent modeling, we emphasize that it is a critical mechanism for realizing belief-aware planning within our QOM framework. Specifically, the meta-policy plays three key roles. First, it guides action selection during Monte Carlo rollouts. The meta-policy serves as a belief-weighted mixture of soft best responses, which directly informs the agent’s action choices during Monte Carlo rollouts. Rather than relying on uninformed or uniform rollout policies, it provides a principled prior that focuses the search on strategies likely to succeed under the current opponent belief, thereby improving planning efficiency and robustness. Second, it provides a principled mechanism for responding to opponent uncertainty. Beyond rollout heuristics, the meta-policy operationalizes a soft best-response to the posterior over latent opponent types. This allows the agent to act adaptively under uncertainty—balancing exploitation and exploration—without hard type commitment or brittle switching. In essence, it connects the Bayesian belief over opponent behavior to real-time decision-making. Third, it serves as the interface between offline learning and online planning. The meta-policy leverages the offline-estimated payoff matrix, computed from agent-opponent policy interactions, to synthesize strategic behavior during planning. It thus serves as the bridge between offline opponent modeling (via quantized types) and online planning under uncertainty, aligning the agent’s strategy space with its evolving beliefs. The design of meta-policy is inspired by the idea introduced in POTMMCP. Our work builds upon and extends this concept by integrating it into a learned quantized belief model.\n\nIn summary, the meta-policy is integral to the deployment of QOM: it enables efficient belief-aware planning by translating opponent uncertainty into informed action selection during planning. Without this mechanism, the benefits of the learned opponent models would not materialize in online decision-making. While this design shifts some computational burden to the offline phase, we believe it is a worthwhile and effective trade-off that significantly reduces online planning latency and improves real-time decision quality.\n\n> The payoff matrix is defined ...\n\nThe payoff matrix is computed offline across trajectories, not conditioned on state. The meta-policy then acts as a dynamic prior in PUCT, while rollouts incorporate current history and belief. This hybrid yields efficient adaptation. Extending to state-conditional matrices via value approximators is a promising direction, and we leave it for future work.\n\n> Although the method targets ...\n\nWe model all opponents as a single composite agent with a joint type belief—a common abstraction [1,2] to keep planning tractable in multi-agent settings. While a two‑player reformulation is possible, it would mask the genuinely multi‑agent interaction structure (e.g., coordination and interference among multiple pursuers in Predator–Prey) and reduce transfer to more general multi‑agent settings. Moreover, maintaining separate beliefs for $n$ opponents yields an exponential type space $K^n$, which is computationally prohibitive; a belief over joint types keeps complexity at $K$ while still capturing group behavior uncertainty. Our formulation therefore strikes a practical balance between expressiveness and tractability.\n\n[1] Anirudh Kakarlapudi, Gayathri Anil, Adam Eck, Prashant Doshi, and Leen-Kiat Soh. Decision theoretic planning with communication in open multiagent systems. PMLR, 2022.\n\n[2] Alessandro Panella and Piotr Gmytrasiewicz. Interactive pomdps with finite-state models of other agents. AAMAS, 2017\n\n> Related Work\n\nThank you for the suggestion. We will include this work in the revised version."}, {"review_id": "sTdS3pZyev", "reviewer": "Reviewer_VDJ5", "summary": "This paper presents a method for multi-agent planning by learning a finite set of opponent types from trajectory data using a VQ-VAE. The learning of this quantized autoencoder bases on a simulated offline dataset trajectories of the agents' interactions with the knowledge of the behavior policy libraries of both the agent and the opponents. The agent updates their belief over the opponents' type after this offline training, and makes decision with expectation on sampled types from this belief together with per-type policies derived from a pre-trained payoff matrix. Experiments on some multi-agent environments show that the proposed method achieves an effective performance with less searching cost.", "questions": "Please see the weakness section above.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n* The idea of modeling opponents' behavior into finite patterns via a vector-quantized autoencoder is interesting and novel.\n\n* The proposed method has all the merits inherited from type-based methods, Beyesian reasoning, and Monte-Carlo planning, while avoiding the drawbacks of manually defined behavior classes and sample inefficiency during online planning.\n\n* Ablation study shows reasonable and interpretable explanations over different design choices.\n\n## Weaknesses\n\n* Major weaknesses\n\n  * The success of the method depends heavily on (1) the quality of the codebook of the opponents' behavior from the quantized encoder trained with the offline trajectories; (2) the payoff matrix estimated from the offline trajectories. Any illustrations of the learned codebook and their correspondence with the opponent policies would be helpful to understand the quantization process.\n\n  * The success of the proposed method also relies on a good set of opponent policies that are close to the true opponents' behavior in real-time execution. However, the paper does not provide enough demonstration showing that with the proposed method's planning, the agent also faces similar opponents' behavior as in the offline training. It is claimed in this paper (at Line 231) that the opponent policy library has 50 policies including final trained opponents' policy and intermediate checkpoints. If I am understanding correctly, the opponent's policy can also include any agent's own individual policy and freely combine them as long as they share same state and action space. Therefore, the complexity of the opponent's policy library can be very high, depending exponentially on the number of agents.\n\n\n* Minor weaknesses\n\n  * In Figure 2 and 3, the x-axis does not represent the actual quantitative values along its direction. Each point is a different choice of search time but they are equally segmented. They are neither linear scale nor log scale. Therefore, the current plot does not reflect the actual true trends with respect to the increase of the search time. It would be better to use other representations than solid lines.\n\n  * It is unclear to me for different search time, what is the actual wall time used for the search. Are the actual wall time requires for the search linear with respect to the search time? If not, it would be better to clarify this in the paper.", "was_revised": true, "final_justification": "I am keeping my rating and would like the authors to provide some additional experiments in the revision.", "rebuttal": "Thank you for acknowledging our novel contributions as well as raising valuable questions. As follows, we address your concerns in detail.\n\n> The success of the method depends heavily on (1) the quality of the codebook of the opponents' behavior from the quantized encoder trained with the offline trajectories; (2) the payoff matrix estimated from the offline trajectories. Any illustrations of the learned codebook and their correspondence with the opponent policies would be helpful to understand the quantization process.\n\nAs shown in Figure 6, we visualize the payoff matrices between (i) agent policies and latent types (the learned codebook), and (ii) agent policies and opponent policies. However, as you pointed out, these matrices alone do not make the policy–type correspondence explicit. To address this, we directly measure behavioral similarity between opponent policies and latent types. While figures cannot be shown here, the analysis shows that more than half of the opponent policies align strongly with a single latent type, whereas the remainder spread their alignment over two to three types (multimodal behavior). We will include the corresponding illustration and analysis in the revision.\n\n> The success of the proposed method also relies on a good set of opponent policies that are close to the true opponents' behavior in real-time execution. However, the paper does not provide enough demonstration showing that with the proposed method's planning, the agent also faces similar opponents' behavior as in the offline training. It is claimed in this paper (at Line 231) that the opponent policy library has 50 policies including final trained opponents' policy and intermediate checkpoints. If I am understanding correctly, the opponent's policy can also include any agent's own individual policy and freely combine them as long as they share same state and action space. Therefore, the complexity of the opponent's policy library can be very high, depending exponentially on the number of agents.\n\nWe agree that many policy-library–based methods struggle to generalize to opponents outside the training population [1,2]. Classic BPR-style reuse relies on exact opponent matches and performs poorly when facing novel strategies. In contrast, QOM quantizes the opponent policy space into $K$ latent types and maintains a Bayesian posterior over them. This allows QOM to assign probability mass to nearby types and adaptively plan, even when the online opponent is not in the library. This generalization is demonstrated in our out-of-library test (Figure 3(a)).\n\nIn term of complexity, we treat all opponents as a single composite agent with a joint action space; consequently, online planning reasons only over K latent opponent types, rather than over combinations of per‑opponent policies. Constructing separate policy libraries for each opponent and composing them at decision time would, as you point out, cause an exponential increase in complexity. Exploring such factorized libraries is outside our current scope and left for future work.\n\n[1] Jing Y, Liu B, Li K, et al. Opponent modeling with in-context search. NeurIPS, 2024.\n\n[2] Lian J, Huang Y, Ma C, et al. Fusion-PSRO: Nash policy fusion for policy space response oracles. arXiv:2405.21027.\n\n> In Figure 2 and 3, the x-axis does not represent the actual quantitative values along its direction. Each point is a different choice of search time but they are equally segmented. They are neither linear scale nor log scale. Therefore, the current plot does not reflect the actual true trends with respect to the increase of the search time. It would be better to use other representations than solid lines.\n\nOur plots could misleadingly suggest a continuous quantitative x‑axis. In the revision, we will re-plot the figures by treating search time as a categorical variable and will remove all connecting solid lines.\n\n> It is unclear to me for different search time, what is the actual wall time used for the search. Are the actual wall time requires for the search linear with respect to the search time? If not, it would be better to clarify this in the paper.\n\nWe would like to clarify that our evaluation is based on wall-clock time. As stated in Section 4.2, the “search time budget” refers to real wall-clock runtime limits per decision step (e.g., 0.1s, 1s, 5s, etc.), which are implemented as actual timing constraints during online planning. The precise durations used for each environment are listed in the Hyperparameter Table in Appendix C. In practice, the actual wall time used scales approximately linearly with the search time budget. POTMMCP is evaluated under the same wall-clock constraints, ensuring a fair and consistent comparison. Thank you for the suggestion and we will clarify this more explicitly in the revision."}, {"review_id": "f0lFbrtGrq", "reviewer": "Reviewer_PqEB", "summary": "The paper addresses a core challenge for partially observed multi-agent environment, that is the uncertainty about an opponent's behavior. It proposes a framework where a \"catalog\" of opponent \"strategies\" is learned from data by using VQVAE. This allows an agent to maintain a Bayesian belief over a finite set of opponent \"strategies\" and update these beliefs at test time. During test time, the framework leverages the opponent model for planning in MCTS. Experimental results show strong performance compared to other baselines.", "questions": "1. Could the author clarify whether a ground-truth or learned simulator is used in the experiment?\n\n2. Bayesian update is used for updating belief over opponents. Have the author experiment with other belief updates such as expert switching [1], which alleviates the catch-up phenomenon in Bayesian update [2]?\n\n3. The paper mentions using a VQ-VAE and other alternatives to learn opponent \"strategies\". Could you elaborate on the architecture of the quantizer? \n\n4. How well is the learned \"strategies\" mapped to the opponent's policy library? And does a \"good\" mapping contributes to better performance? Could you provide some discussion?\n\n5. Since the framework is quite complex, will the author open source the code? \n\n[1] W. M. Koolen et al. Combining expert advice efficiently.\n\n[2] T. van Erven et al. Catching Up Faster in Bayesian Model Selection and Model Averaging.", "limitations": "Yes. More societal impact could be discussed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\nOverall the paper is well-written. It provides a good description for a very complex framework. Theoretical aspect is sound.\n\nExperimental results are strong. The proposed Quantized Opponent Models (QOM) consistently outperform state-of-the-art baselines across a variety of multi-agent environments. Ablations are extensive.\n\n**Weaknesses**\n\nThe proposed framework requires extensive offline preparation and potentially quite expensive in terms of compute. \n\nThe number clusters learned by VQVAE is still quite small, therefore, it's questionable whether the framework can be extended to work in more complex environments.", "was_revised": true, "final_justification": "The rebuttal has addressed my concern. I maintain the score, but downgrade the confidence level as I feel other reviewers are more familiar with the multi-agent frameworks than me.", "rebuttal": "Thank you sincerely for your feedback and appreciation.\n\n> Could the author clarify whether a ground-truth or learned simulator is used in the experiment?\n\nWe use ground-truth simulators for the Pursuit-Evasion, Predator-Prey, and Running-with-Scissors environments. For the One-on-One environment, we follow the common practice in prior work (MBOM) and use a learned simulator. This setup ensures consistency with the baseline implementations and allows for fair performance comparison. We will clarify this distinction in the revision.\n\n> Bayesian update is used for updating belief over opponents. Have the author experiment with other belief updates such as expert switching [1], which alleviates the catch-up phenomenon in Bayesian update [2]? [1] W. M. Koolen et al. Combining expert advice efficiently. [2] T. van Erven et al. Catching Up Faster in Bayesian Model Selection and Model Averaging.\n\nWhile we do not use expert‑switching updates, we explicitly address the catch‑up phenomenon with a lightweight forgetting mechanism. After each Bayes step (Eq. (5)) we apply belief smoothing by interpolating the posterior with a uniform prior, $b^{\\text{smooth}}_{t+1}=(1-\\lambda)\\,b_{t+1}+\\lambda\\,b_0$. This constant‑share (“leak”) step (i) maintains a non‑vanishing probability mass on alternative types, preventing premature lock‑in, and (ii) induces an effective memory of roughly $1/\\lambda$ steps, which shortens recovery after occasional switches or model mismatch. Its effect is largely neutral in very short interactions, but in piecewise‑stationary scenarios it reduces catch‑up time without changing the rest of the planner. Expert‑switching priors are a valuable direction and may further help in our Adaptation to Switching Opponents setting (Sec. 4.4). We will include a concise comparative experiment in the revision.\n\n> The paper mentions using a VQ-VAE and other alternatives to learn opponent \"strategies\". Could you elaborate on the architecture of the quantizer?\n\nThe encoder of VQ‑VAE is a 2‑layer GRU sequence model (hidden size 64) that reads the opponent trajectory and maps it to a continuous embedding $z\\in\\mathbb{R}^{32}$. We vector‑quantize $z$ by nearest‑neighbor assignment to a $K$-entry codebook ($K{=}16$), yielding a discrete type $e_k$. The decoder is a GRU‑based autoregressive model conditioned on $e_k$: we initialize the decoder hidden state with a learned projection of $e_k$ and predict per‑step opponent‑action likelihoods $\\tilde\\pi_k(a_{-i}\\!\\mid h)$. The quantizer is trained with the standard objective (reconstruction + codebook + commitment losses) using a straight‑through estimator with EMA codebook updates. We will revise and expand the appendix to include these architectural details and hyperparameters.\n\n> How well is the learned \"strategies\" mapped to the opponent's policy library? And does a \"good\" mapping contributes to better performance? Could you provide some discussion?\n\nAs shown in Figure 6, we visualize the payoff matrices between (i) agent policies and latent types (the learned codebook), and (ii) agent policies and opponent policies. However, as you pointed out, these matrices alone do not make the policy–type correspondence explicit. To address this, we directly measure behavioral similarity between opponent policies and latent types. While figures cannot be shown here, the analysis shows that more than half of the opponent policies align strongly with a single latent type, whereas the remainder spread their alignment over two to three types (multimodal behavior). We will include the corresponding illustration and analysis in the revision.\n\n> Since the framework is quite complex, will the author open source the code?\n\nYes, we plan to release the code, and the link will be available in the revision."}, {"review_id": "xyES4BBJVZ", "reviewer": "Reviewer_PHgn", "summary": "This paper proposes a new framework called Quantized Opponent Models (QOM) for planning in multi-agent environments with unknown and potentially non-stationary opponents. The core idea is to discretize the opponent policy space using a quantized autoencoder (VQ-VAE), mapping each policy to a latent type. During interaction, the agent maintains a Bayesian belief over these types. It then constructs a belief-weighted soft best-response policy, known as the meta-policy, and uses it to guide action selection within a Monte Carlo Tree Search (MCTS) planner. This design allows opponent modeling to be integrated directly into the planning process.\n\nThe paper presents a theoretical result showing that the posterior over opponent types concentrates under bounded quantization error, and evaluates the method across several multi-agent environments, including adversarial and partially observable settings. Empirically, QOM consistently outperforms baseline methods, especially under limited computational budgets, and maintains robust performance when facing switching opponents or previously unseen policies.", "questions": "1. How stable is the belief distribution $b_t(k)$ over short time horizons, and when the number of types $K$ is large and closely spaced, how reliably can the model distinguish between them?\n2. During MCTS rollouts, how is the opponent’s observation $o_{-i}$ inferred or approximated, given that it is not directly observable to the agent?\n3. In the “Adaptation to Switching Opponents” experiment, what exactly does “tag failure” refer to, and how does it determine when the opponent switches its policy?\n4. How are the unseen opponent policies constructed in the generalization experiments, and if the training set already covers diverse PSRO checkpoints, what makes these test-time opponents meaningfully novel?\n5. The ablation on quantization methods compares several discrete encoding schemes (e.g.,FSQ, LFQ, BSQ), but since the performance differences are small, what insight is this experiment meant to provide regarding the role of quantization in the overall method?\n6. Why is evaluation almost entirely based on performance versus search time, rather than more direct metrics like wall-clock runtime or tree size that would reflect computational cost?", "limitations": "The authors acknowledge some key limitations, such as the reliance on a fixed opponent strategy library and the assumption of piecewise-stationary behavior during interaction.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**  \n1. The proposed method of discretizing the opponent policy space using a quantized autoencoder is conceptually simple, and avoids the need for handcrafted type definitions.\n2. The integration of opponent modeling into the planning loop—through a belief-weighted prior in MCTS—is well-motivated and cleanly implemented.\n3. The empirical evaluation spans a reasonably diverse set of environments and includes ablation studies that clarify the impact of key design components.\n\n** Weaknesses**  \n1. I find your framework is quite similar to [1]. Like your work, [1] maintains beliefs about agents' types, includes an opponent simulation component (goal-conditioned policy in [1]) to predict opponents’ actions, and features a planning module to determine the best response. Both approaches use Bayes’ rule to update beliefs, with the opponent simulation component also acting as the likelihood term in Bayes' rule. Additionally, the opponent simulation component is used to simulate others’ actions during planning. Unlike [1], you use a pre-trained VQ-VAE to infer the types of opponents, while [1] represents opponent types using goals selected from a fixed goal set. Consequently, your approach requires both an agent library and an opponent library, as well as many samples to pre-train the VQ-VAE. In contrast, [1] requires expert knowledge to define a goal set for each environment. Additionally, you employ MCP for planning, while [1] uses MCTS. However, I may have overlooked some differences. Please clarify the novelty and advantages of your approach compared to [1], and ensure you cite it appropriately.\n2. The approach relies heavily on a fixed offline opponent strategy library, which may limit its applicability in settings where the opponent pool is open-ended or evolving. \n3. There is limited discussion of the computational cost introduced by belief updates, latent inference, and payoff matrix construction. While QOM is claimed to be more efficient than baselines, no runtime comparisons or profiling are provided.\n4. The method is evaluated only with pre-defined agents and policies. It is unclear how it would perform in more dynamic, co-adaptive environments where both agents and opponents are learning simultaneously.\n\n[1] Huang, Yizhe, et al. \"Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning.\" International Conference on Machine Learning. PMLR, 2024.", "was_revised": true, "final_justification": "The authors' response addressed some of my concerns and confusion regarding the methodological details. However, the use of MCTS in the planning module weakens QOM's applicability in partially observable environments. Therefore, I have decided to keep my original score unchanged.", "rebuttal": "Thank you for acknowledging our contributions as well as raising valuable questions.\n\n> I find your framework is quite similar to [1]. Like your work, [1] maintains beliefs about agents' types, includes an opponent simulation component (goal-conditioned policy in [1]) to predict opponents’ actions, and features a planning module to determine the best response. Both approaches use Bayes’ rule to update beliefs, with the opponent simulation component also acting as the likelihood term in Bayes' rule. Additionally, the opponent simulation component is used to simulate others’ actions during planning. Unlike [1], you use a pre-trained VQ-VAE to infer the types of opponents, while [1] represents opponent types using goals selected from a fixed goal set. Consequently, your approach requires both an agent library and an opponent library, as well as many samples to pre-train the VQ-VAE. In contrast, [1] requires expert knowledge to define a goal set for each environment. Additionally, you employ MCP for planning, while [1] uses MCTS. However, I may have overlooked some differences. Please clarify the novelty and advantages of your approach compared to [1], and ensure you cite it appropriately. [1] Huang, Yizhe, et al. \"Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning.\" International Conference on Machine Learning. PMLR, 2024.\n\nWhile both QOM and HOP [1] involve opponent modeling and planning, the two methods differ fundamentally in their problem formulation and modeling assumptions. HOP relies on a structured environment with an explicit goal space, modeling opponent behavior via goal inference and goal-conditioned policies. This hierarchical formulation is central to its mechanism and restricts its applicability to settings where goals are well-defined and observable. In contrast, QOM assumes no access to such goal structures. Instead, it operates in general partially observable multi-agent environments, modeling opponents through latent policy types learned via a quantized autoencoder. The agent maintains a Bayesian belief over these types, enabling efficient and scalable planning without relying on predefined goals. Thus, QOM addresses a more general setting and offers a flexible solution applicable across a wider range of domains where goals may be implicit or unavailable.\n\n> The approach relies heavily on a fixed offline opponent strategy library, which may limit its applicability in settings where the opponent pool is open-ended or evolving.\n\nWe would like to clarify that this reliance on an offline opponent policy library is a common feature shared by many existing approaches, and such methods can still be effective in practice—provided that the strategy library sufficiently covers the policy space. This highlights an important but often overlooked aspect: the quality and diversity of the policy library are critical. Our proposed method, QOM, partially addresses this issue by reducing the reliance on handcrafted opponent strategies. Instead of manually designing diverse policies, QOM leverages quantitative objective modeling to construct a structured opponent representation space, from which diverse and representative policies can be more effectively identified and utilized.\n\nRegarding settings with open-ended or evolving opponents, we acknowledge that this remains an open challenge for the community. While our current framework assumes a fixed opponent pool, it can be extended in future work to incorporate adaptive mechanisms such as online strategy expansion or continual opponent modeling. We leave it for future work.\n\n> There is limited discussion of the computational cost introduced by belief updates, latent inference, and payoff matrix construction. While QOM is claimed to be more efficient than baselines, no runtime comparisons or profiling are provided.\n\nThe payoff matrix construction is performed offline, which removes a substantial computational burden from the online decision-making loop. While belief updates and latent inference are executed online, they are designed to be lightweight and incremental, enabling fast and efficient planning during interaction. Our efficiency claim focuses on the online planning process, specifically in terms of search budget utilization. Compared to baselines, QOM achieves higher performance given the same per-step search budget. We will make this distinction explicit in the revision.\n\n> The method is evaluated only with pre-defined agents and policies. It is unclear how it would perform in more dynamic, co-adaptive environments where both agents and opponents are learning simultaneously.\n\nWhile most of our experiments are based on a fixed policy library, we would like to point out that we do consider partially dynamic settings—for example, in Figure 3(a), the opponent switches its policy during interaction, requiring the agent to adapt. We agree that fully co-adaptive environments, where both agents and opponents learn simultaneously, are more challenging. This remains a limitation for library-based approaches such as QOM and POTMMCP. However, when the policy library is sufficiently diverse and well-structured, it can provide strong coverage of the policy space. We believe this enables the agent to handle a wide range of dynamic opponent behaviors.\n\n> How stable is the belief distribution $b_t(k)$ over short time horizons, and when the number of types $K$ is large and closely spaced, how reliably can the model distinguish between them?\n\nFrom our empirical observations, when using a reasonably chosen number of types $K$, the belief distribution $b_t(k)$ typically shows preference for several likely types after a few observations, enabling effective and focused planning. We will add a visualization of belief dynamics over time horizons in the revision. Conversely, setting $K$ too large tends to over-fragment the latent space, spreading the belief across many similar types and thereby diluting the planning focus in each simulation. This phenomenon is discussed in Figure 11, where we show that excessively large $K$ leads to diminishing returns and even degraded performance due to type redundancy.\n\n> During MCTS rollouts, how is the opponent’s observation $o_{-i}$ inferred or approximated, given that it is not directly observable to the agent?\n\nThe opponent’s observation $o_{-i}$ is approximated during rollouts using the environment simulator, which generates plausible observations from simulated joint actions and next states. In each rollout step, we use the environment simulator to generate $o_{-i}$ by stepping forward with the sampled joint action, and update the opponent’s history accordingly. This iterative process enables belief tracking and simulation without directly observing private opponent observations.\n\n> In the “Adaptation to Switching Opponents” experiment, what exactly does “tag failure” refer to, and how does it determine when the opponent switches its policy?\n\nIn the Running-with-Scissors environment, “tag” is an action where the opponent marks a 3×3 zone ahead to challenge the agent, with outcomes decided by RPS rules. If the game doesn't end, the opponent switches policy. We will clarify this in the revision.\n\n> How are the unseen opponent policies constructed in the generalization experiments, and if the training set already covers diverse PSRO checkpoints, what makes these test-time opponents meaningfully novel?\n\nThe unseen opponent policies used in the generalization experiments are generated from three additional rounds of PSRO training, independent of those used to construct the QOM training policy set. These additional policies are not included in the quantized opponent model, and provide novel strategic behaviors that differ from the training set due to differences in PSRO dynamics across runs. This setup allows us to assess generalization beyond the fixed policy catalog. We will clarify this detail in the revision.\n\n> The ablation on quantization methods compares several discrete encoding schemes (e.g.,FSQ, LFQ, BSQ), but since the performance differences are small, what insight is this experiment meant to provide regarding the role of quantization in the overall method?\n\nThe experimental results indicate that QOM achieves consistently performance across different quantization schemes, with only marginal differences. This suggests that the effectiveness of QOM does not heavily depend on the specific choice of quantization strategy, highlighting its robustness and modularity — a valuable property for practical deployment.\n\n> Why is evaluation almost entirely based on performance versus search time, rather than more direct metrics like wall-clock runtime or tree size that would reflect computational cost?\n\nWe would like to clarify that our evaluation is based on wall-clock time. As stated in Section 4.2, the “search time budget” refers to real wall-clock runtime limits per decision step (e.g., 0.1s, 1s, 5s, etc.), which are implemented as actual timing constraints during online planning. To make this more concrete, the Hyperparameter table in Appendix C lists the precise wall-clock durations (in seconds) used as unit time for each environment. These are empirically calibrated to reflect the relative complexity of each domain and ensure comparability across methods. POTMMCP is evaluated under the same wall-clock constraints, ensuring a fair and consistent comparison. We will clarify this more explicitly in the revision."}]

Author rebuttal:

Meta-review: This uses a hybrid approach, where a finite number of opponents are model with neural networks, and a Bayesian prior is maintained over them.

+ The approach is interesting
- It is a bit complicated. Maybe a simpler version of the algorithm should have been tested as well (which the authors do in their rebuttal)
- The experimental results are not very convincing, i.e. the algorithm is not strictly better than the baseline.

The reveiwers are mainly positive. ETPj asks for some more experiments, but it is unclear to me what those would be.
Decision: Accept (poster)