Generative Evolutionary Meta-Solver (GEMS): Scalable Surrogate-Free Multi-Agent Reinforcement Learning
Authors: Alakh Sharma
Gaurish Trivedi
Kartikey Singh Bhandari
Yash Sinha
Dhruv Kumar
Pratik Narang
Jagat Sesh Challa
Venue: Accepted by TMLR ()

Abstract: Scalable multi-agent reinforcement learning (MARL) remains a central challenge for AI. Existing population-based methods, like Policy-Space Response Oracles, PSRO, require storing explicit policy populations and constructing full payoff matrices, incurring quadratic computation and linear memory costs. We present Generative Evolutionary Meta-Solver (GEMS), a surrogate-free framework that replaces explicit populations with a compact set of latent anchors and a single amortized generator. Instead of exhaustively constructing the payoff matrix, GEMS relies on unbiased Monte Carlo rollouts, multiplicative-weights meta-dynamics, and a model-free empirical-Bernstein UCB oracle to adaptively expand the policy set. Best responses are trained within the generator using an advantage-based trust-region objective, eliminating the need to store and train separate actors. We evaluated GEMS in a variety of Two-player and Multi-Player games such as the Deceptive Messages Game, Kuhn Poker and Multi-Particle environment. We find that GEMS is up to ~$\mathbf{6\times}$ faster, has $\mathbf{1.3\times}$ less memory usage than PSRO, while also reaps higher rewards simultaneously. These results demonstrate that GEMS retains the game theoretic guarantees of PSRO, while overcoming its fundamental inefficiencies, hence enabling scalable multi-agent learning in multiple domains.

Gaurish Trivedi Published in Transactions on Machine Learning Research (02/2026)
Generative Evolutionary Meta-Solver (GEMS): Scalable Freee Multi-Agent Reinforcement Learning
Alakh Sharma
f20240593@pilami. piwpiiiiiiiiiiiiiiiii.mmmmmm
Birla Institute of Technology and Science, Piloni
|20220028@pilamibbtts wwtp:ppaaaaiiibi-i--aaai.ie
Birla Institute of Technology and Science, Pilani
Kartikey Singh Bhandari
0.0 in
Birla Institate of Technology and Science, Pilani
Yash Sinha
Birla Institute of Technology and Science, Pilani
Dhruv Kumar
thwu.kuumrr@plln pitilliiiiiiittiiipioooomm
Birla Instätute of Technology and Science, Piloni
Pratik Narang
protik. mnaiik.narog@@pilan. bits bits-pilani, oc in
Birla Institute of Technology and Science, Pilani
Jagat Sesh Challa
jagatsesh@@piloni. bits- ciits-ppllmi.or oc. oc
Birla Institute of Technology and Science, Pilani
Reviewed on OpenReview: https: openrevveee net/ forum? id- ZuEJsXoBHD
Abstract
Scalable multi-agent. reinforcement learning (MARL) remains central challenge for AI. Existing population-based methods, like Policy-Space Response Oracles, PSRO, require storing explicit policy populations and constructing full payoff matricos, incurring quadratic computation and linear memory costs. We present Generative Evolutionary Meta-Solver (GEMS), a surrogate-free framework that replaces explicit populations with compact set of latent anchors and single amortized generator. Instead of exhaustively constructing the payoff matrix, GEMS relies unbiased Monte Carlo rollouts, multiplicative-weige meta-dynamics, and model-free empirical-Bernsteer UCB oracle to adaptively expand the policy set. Best responses are trained within the generator using an advantage-base based trust-region objective, eliminating the need to store and train separate actors. We evaluated GEMS in a variety of Two-player and Multi-Player games such as the Deceptive Messages Game, Kuhn Poker and Multi-Partichh environment. We find that GEMS up 6x faster, has 1.3x less memory usage than PSRO, while also reaps higher rewards simultaneously. These results demonstrate that GEMS retains the game theoretic guarantees of PSRO, while overcoming its fundamental inefficiencies, hence enabling scalable multi-agent learning in multiple domains.
"Joint first authors

---

Published in Transactions on Machine Learning Research (02/2026)
1 Introduction
Imaginc organizing a round-robin tennis tournament with hundreds of players (Fig. 1 Scheduling every match gives a complete ranking but is clearly inefficient: the number of matches grows quadratically, and the results table quickly becomes unwieldy.
Training AI agents in two-player zero-sum Markov games faces a similar challenge. Population-ban based mothods such as Policy-Space Response Oracles (PSRO) (Lanctot et al. (2017) maintain a growing set of k policies and explicitly construct k x k payoff matrix, where each entry records the expected outcome of policy against another. A meta-solver then updates the distribution over policies, analogous to computing player rankings from the tournament table.
While conceptually straightforward, classical PSRO suffers from three critical bottlenecks: O Memory overbead: stor- ing a new policy for each player leads to linear growth in storage; @ Computation overhead: filling in the full k x k payoff matrix quickly becomes infeasible; @ Scalability of new entries: adding new policy requires training and storing another separate actor. Prior work has mitigated some of these costs through selective best-response train- ing (Smith et al. (2021)) ), Double Oracle methods (McAleer eet al. (2021): h: Huang et al. (2022)), meta-game improve- ments (McAleer et al. (2022b) McAleer et al. (2022a)) )
Classical PSRO: SO0k)) matches, SOlk) memory
Monte Carlo rollouts, single generator, scalable expansion
GEMS
and knowledge transfer for agents (Smith et al. 20223]
Lian et al. (2024)). However, these approaches retain the core paradigm of explicit policy sets, leaving scalability fundamentally limited.
Generattr
Figure 1: Tournament analogy for policy populations. (Teft) PSRO: explicit & x k payoff matrix with all pairwise matchups. (Right) GEMS: compact anchor set of latent policies, with a single generator producing diverse strategies on demand.
GEMS overcomes these limitations while preserving the game-thooretic guarantees of PSRO. Analogously to run- ning a large tournament without scheduling every match, O GEMS maintains compact anchor set of latent codes that represent active "players". @ GEMS treats the pay- off matrix as conceptual and queries it through Monte Carlo rollouts, which are unbiased in nature. Meta-strategy is updated using mulliplicative-weighhh discrelization of replicator dynamics, which are akin to adjusting ranking based on samples. 4 Empiricial-
Upper Confidence Bound (UCB) [Maurer & Pontil 2009) selects promising new "players" from a candidate pool, thus expanding the population. @ Finally, GEMS incorporates an amortized generator with an Advantage-base based Best-Responss objective and Trust-region Regularization (ABR-TR) that eliminates the need for separate policies. We evaluated GEMS in variety of Two-player and Multi-Player games such the Deceptive Messages Game, Kuhn Poker Kuhn 2016) and Multi-Particle environments (Terry et al. 2021). We find that GEMS is up to 6x faster, has 1.3x less memory usage than PSRO, while also reaps higher rewards simultaneously.
Contributions. Relative to classical PSRO, GEMS achioves memory efficiency: GEMS replaces O(k) stored players with a single versatile generator. Computation efficiency: GEMS avoids PSRO'S quadratic payoff tables using Monte Carlo estimates, scaling per itcration with the number of sampled matches and the candidate pool size. o Scalable new entries: EB-UCB identifies strong candidates, and ABR-TR integrates them into the generator without adding new actors. 4 Theoretical guarantees: unbiased MC meta-gradients, instance-dependent regret bounds for EB-UCB, external regret bounds for setiiliicativ-weighhe dynamics, and
exploitability accounting for approximate best responses. Together, these advances transform the exhaustive "tournament book-keeping" of PSRO into a leaner, scalable framework, closer to how real tournaments operate, where only select matches determine rankings and strategies adapt efficiently from sparse outcomes.
2

---

Published in Transactions on Machine Learning Research (02/2026)
2
Related Work
Psro and its Variants. The seminal work Lancto et al. (2077] introduced PSRO as a general framework for multi-agent reinforcement learning. PSRO iteratively expands a population of policies by training a new best-response policy against a meta-game mixture of the existing population. A core challenge of PSRO is its reliance on a full h x k payoff matrix, where & is the number of policies in the population. This leads to O(12) computation overhead per iteration and (k) memory overhead to store individual policies, both of which
become bottlenecks in large-scale settings (Shao et. al., 2024; Zhou et. al., 2022). This has motivated a line of work on more efficient variants. For instance, EFFICIENT PSRO (E-PSRO) (Zhou et al. 2022) minimizes evaluation costs by formulating the solver propose training a best-response against a single opponent, reducing computation but potentially limiting diversity. The double oracle (DO) family of algorithms, including XDO (McAleer et al.] 2021 and EDO (uang et al.) 2022). focus on finding equilibria more efficiently, often by guaranteeing linear convergence or monotonic exploitability reduction (McAleer et al. 2022b). However, these methods can still suffer from issues like performance oscillations or slow convergence due to the need for a full set of strategies [Huang et al. 2022). In contrast, GEMS avoids these issues entirely by replacing the explicit payoff matrix with unbiased Monte Carlo rollouts and the discrele sel of policies wilh a single, amortized generalor.
unrestricted-rrsttiicke game. Similarly, /mithh et al. (2021)
Other work has addressed the computational cost of training new policies from scratch at each iteration, which is a significant bottleneck (Li ct al. 2023). Notably, PIPELINE PSRO (P2SRO) (McAleer et al.) 2020 mitigates this by parallelizing the training of best responses. To further address this, FUSION- SESION-PPRO PSRO (Lian et al., 2024) uses policy fusion to initialize new best responses, while STRATEGIC KNOWLEDGE TRANSFER (Smith al. 2023) explores transferring knowledge across changing opponent strategies. Our work tackles this problem differently by using a single amortized generator, which removes the need to store and train separate models for each policy.
Finally, some papers have focused on improving the meta-game solution or policy diversity. ALPUA-PPRO Muller et al. 2019) replaces the Nash meta-solver with a- a-Rank to scale PSRO to general-sum, many- many-player settings. A-PSRO (Hu et al. 2023) introduces an advantage-baser evaluation for strategy selection, providing a unificd objoctive across zero-sum and general-sum games. ANYTIME PSRO (MeAleor ct al., (022b) and SELF- PLAY PSRO (SP-PSRO) (McAleer al.) 2022a) aim to improve convergence by adding high-quality policies to the population. Other works introduce new metrics for policy diversity to ensure better approximation of the Nash Equilibrium (Tang et al. 2025 Yao et al., 2023). While these works aim to improve PSRO, they do not fundamentally change its core structure, which still relies on an explicit population of policies and a payoff matris.
Neural Population Learning. Recent work has aggressively explored unilying policy populations inlo single conditional networks to enable positive transfer and minimize storage. NEURAL. POPULATION LEARNING (NEUPL) (Liu al.) (222b)) represents an entire population of best-response policies using a single neural network that is explicitly conditioned on which strategy is being executed. Formally, this network defines a conditional policy [a x, that outputs a distribution over actions given the current state (or observation) 8, where a is learnable discrete strategy index identifying a particular population member. Addressing the limitations of discrete indexing, SIMPLEX-NEULL {Liu et al. 2022a) introduces a geometric formulation in which the policy conditions on continuous simplex vectors, enabling direct representation of mixed strategies within a single forward pass. Furthermore, NEUPL-JJPSO (Liu al. (2024) extends this paradigm to general- sum games by combining conditional policies with Join objectives, effectively learning correlated joint policies that approximate coarser equilibrium concepts such as CCE. However, despile amorlizing memory costs, these methods often retain the computational bottlenecks of empérical game theory, specifically the O(k2) cost of estimating pairwise payoffs to update the strategy distribution [Liu et al.) (2022)). Distinct from the conditional archilectures of the NEUPL family, GEMS introduces a fully generalive mela-solver that maps a latent geometry directly to policy parameters via a hypernetwork. This formulation allows GEMS to bypass explicit population management entirely, utilizing unbiased Monte Carlo rollouts and latent-space evolutionary optimization (OMWU) to solve the gome without constructing a full payoff matria.
3

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Theoretical comparison of GEMS and PSRO-style methods. Let N, denote the cumulative number of policies found by iteration Coval is the cost of one payoff evaluation (e.g., one episode), and k is the number of sampled opponents per iteration. Memory Scaling refers to the storage required for the meta-game state (payoff matrix / sampled entries). Eval. Cost refers to the complexity of evaluating new policy update. While standard PSRO variants scale quadratically in meta-game memory as the population grows (Nr x t), GEMS maintains constant meta- game scaling (O(1) via its generative parameterization.
Method
Payoff Matrix
Memory (Meta-Game)
Eval. Cost
(per iter.)
PSRO
Required (Full)
O(N?)
( Ceval)
Alpha-PSRO
Required
O(N7)
ON, Creal)
APSRO
Sampled
((N))-((
(I Cevat)
EPSRO
Sampled
((N)) O(N?)
O Ceval)
P2SRO
Sampled / local
((N))))(N3)
O Ceval)
NeuPL
Sampled online
((N))-O(N3)
( Ceval)
GEMS (Ours)
Not Required
O(1)
O Ceval)
Note: GEMS utilizes a fixed-size latent anchor set |Z] K (where < is constant hyperparameter) to represent the population. Therefore, its meta-game memory complexity remains ((1) relative training iterations whereas PSRO-based methods typically accumulate growing population history (N, x t). All methods additionally store policy parameters; the Memory Scaling column isolates mela-game (payoff/state) storage rather than network weights.
Methods for Multi-Agent Learning. Beyond the PSRO paradigm, other game- theoretic approaches exist for multi-agent learning. COUNTERFACTUAL REGRET MINIMIZATION (CFR) (Zinkevich et al., 2007) is a well-known method for extensive-form games, but it typically requires explicit state enumeration, making it challenging to scale to large or continuous domains. Some work has explored bridging the gap between PSRO and CFR, as seen in the unified perspective proposed by Wang el al. (2022). Other approaches include methods based on fictitious play, such as FICTITIOUS CROSS-PLAY (FxP) (Xu ct al. 2023), which combines self-play and PSRO to find global Nash equílibria. However, many self-play methods lack theoretical guarantees for general-sum games and can struggle with convergence in mixed cooperative-compettiie settings (Xu et al. 2023).
Our work contributes to this broader landscape by offering surrogate-free, amortized framework that is both memory and computationally efficient. While many variants of PSRO have been proposed, they have largely retained the core structure of maintaining on explicit policy population and a large payoff matriz. GEMS breaks from this paradigm by using an amortized generator and unbiased Monte Carlo rollouts, thereby sidestepping the fundamental scalability issues inherent to classical PSRO. As summarized in Table [ this design removes the need to store an explicit meta-game game payoff matrix and prevents meta-game state from growing with the population, while retaining a per rer-iteration evoluation profile comparable to sampled PSRO variants.
3 Proposed Method: GEMS
GEMS circumvents PSRO'S limitations by being surrogate-free H Instead of storing & explicit actor models and computing their full payoff matrix, it maintains a single generative model Go that maps low-dimensional latent codes to policies. Consequently, the method scales to massive conceptual populations while storing only the generator and set of latent "anchor" codes.
GEMS unfolds in an iterative loop: It first solves for the equilibrium of the current meta-game using noisy estimates, then expands the game by finding an approximate best response via a bandit-like oracle. The full procedure is detailed below.
While the core exposition focuses on two player zero-sum games for clarity, GEMS framework extends naturally to more general settings. This extensibility is demonstrated by benchmarking GEMS on two-player
In this context, "surrogate-free" signifies that GEMS does not maintain explicit, discrete set of policies to approximate the game.
4

---

Published in Transactions on Machine Learning Research (02/2026)
general-sum and, more broadly, n-player general-sum games. For the multi-player environments, the implementation leverages the PettingZoo library (Terry et al. 2021). The necessary modifications to the meta-game estimators, per-player oracles, and convergence guarantees are provided in Appendix II
Monte Carlo Rollout Evaluation Estimated under fixed meta-strategy (ax)
Estimated Meta Values
Start iteration /
Vector: 0, (policy-mmixture)
Scalar: fe (mixture self-play)
Advance iteration t t t +
Meta- Solver (OMWU) my 26, Ūc-1
Policies
T+-1 t OMWU
Anchor Set (Zn) Initialize: Zo 4
Amortized Generator G=>p=Gola Induces policy Ryp
Update: 7e+1 - Ze U {21}
06+1
Update
Expand 2,
Generator Training (ABR-TR)
New Anchar
Oracle (EB-UCB)
Trust region (KL to Ge + smoothness
21 arg UCC
Figure 2: At each iteration t, Monte Carlo rollouts evaluate the current policy mixture under a fixed meta mrea--stratggy at, producing estimated meta-values de (policy-to-mixture) and ê, (mixture self-play). An optimistic meta-solver updates the mixture via OMWU using the hint m, 26, - 0,-1. An EB-UCB oracle then selects new latent anchor # from the candidate set, which is incorporated through amortized generator training with trust-region objective (ABR-TR). The anchor set Z, is expanded accordingly, the generator induces updated policies Te, and the iteration advances to t + 1. Green ellipses denote temporal iteration boundaries rather than algorithmic operations.
Figure 2 provides a high-level overview of the GEMS algorithmic loop. Rather than explicitly maintaining a growing population of policies and their full payoff matrix, GEMS operates on a compact latent representation: a fixed-size generator coupled with an evolving set latent anchors. At each iteration, Monte Carlo rollouts estimate the meta-game values under the current mixture, which are then used by an optimistic meta-solver to update the population distribution. An EB-UCB oracle selects new latent anchor corresponding to an approximate best. response, which is incorporated via trust-region amortized generator training. The anchor set is expanded, the generator induces updated policies, and the process advances to the next iteration. This closed-loop structure captures how GEMS alternates between equilibrium refinement and controlled population expansion while avoiding explicil policy enumeration.
3.1 Formal Setup and Generative Representation
Consider a two-player zero-sum Markov game. Let r(T;, Ty) E [0, 1] denote the expected return for Player when policy n. faces тÿ. For a finite policy set A = {m....., mx). this induces a payoff matrix M E [-1,1]] where My = E[r(m, m.)]. Crucially, GEMS never explicitly constructs or stores this matrix M.
At each iteration GEMS maintains three core components:
An anchor set of latent codes Z, ( Zke C Rd,, where d is the dimension of the latent space, representing policies in low-dimensional space.
5

---

Published in Transactions on Machine Learning Research (02/2026)
2. A single generator network Go that maps latent code to policy parameters + Go(z), thereby defining a policy Te.-
3. A meta-strategy or E -k,-1; which is probability distribution over the current anchor set. Ze, representing the Nash equilibrium of the restricted game A, (FGa(=) :2 e 2,}, where A is the (k+ -imiennnio probability simplex
These components define the key quantities for analyzing the meta-game. The vector of expected payoffs for each anchor policy against the meta-stralegy o, is Mo,, and the expected value of the gamn at iteration is Ft = of Moz. A primary objective is to minimize exploitability, the incentive for any player to deviate from or:
(xploit(a.) = max e, Mo. - of Mor,
(1)
where ej is the i-th standard basis vector. GEMS iteratively refines and expands Ze to drive this exploitability toward zero.
3.2 Estimating the Meta-Game without the Matrix
Because GEMS does not access the true payoff matrix M, it estimates the values of e and fz through simulation. Adopting a two-time-scale assumption in which the meta-strategy o held fixed during estimation, GEMS employs Monte Carlo rollouts to obtain statistically sound estimates.
For each anchor policy E [kr], the algorithm samples n, opponents ja ae and executes m game episodes per pair, yielding returns Yi,,E E [0, 1 D. This process yields a simple yet powerful estimator for per-policy performance. To estimate the overall game value, the procedure additionally samples B independent policy pairs (6). ~ de X o, and averages their outcomes across m episodes each. The resulting estimators are
B
Pr
(b6, or x o
(2)
TI,
Bm 1=1
and constitute the empirical backbone of GEMS.
Lemma 3.1 (Unbiasedness and EmpiricealBernsteinn Bernstein Concentration). With rewards in [0, 1], the estimators are unbiased: [[ = (Mor), and E = of Mar. Moreover, for any 8 E (0, 1), with probability at least
00r, - (Moo) V
2 Varz, In(2/5)
+ 3lm(2/6) 7:m - | or Morl = =o(v
II(1//] Bm
(3)
n&m
where Var is the emptrical variance of aY,
Proof sketch. Unbiasedness follows from the law of total expectation over the sampling of opponents, and the concentration bounds apply the empirical -ernsteinn inequality for bounded random variables.
3.3 Solving the Meta-Gan Game via Optimistic Replicator Dynamics
Given the estimated payoffs é, and game value êt. GEMS updates the meta-stratogy from de 0++1- GEMS adopts the Optimistic Multiplicative Weights Update (OMWU) algorithm (Daskalakis & Panageas (2018) ), adaptive discretization of replicator dynamics. Rather than relying solely on the current payoff estimate Qz OMWU incorporates a predictive "hint" about the next payoff. Specifically, it forms the optimistic estimate = 26; - δ1-1, with fy = 0. The meta-strategy then updates according to
(1+1 & (t( (xp(nl 20,,, - 6 - -.. 1/8 >>0 0,
(4)
2G denotes the generator network parameterized by 6, which represents its learnable weights. TThe simplex A., E gn+1 : 2 0, Σ, ~; denotes the space of probability distributions over (n + 1) elements. 4j indexes anchor policies in the current population, indexes sampled opponents drawn from the meta-strategy at, and indexes independent. Monte Carlo vollouts (episodes) used estimate their expected returns.

---

Published in Transactions on Machine Learning Research (02/2026)
followed by normalization to ensure d4+1 regret. now scales with the cumulative variation of the payoff vectors rather than the iteration horizon T. Proposition 3.2 (External Regret of OMWU under Unbiased Noise). Assume payoffs in [0, 1]. For any sequence of payoff estimates ò, satisfying Ef os) Mo, and constant step size n, the average external regret obeys
Ak:-1 This optimistic step yields stronger theoretical guarantees:
1 T
max e, Mo. o, Moi) & 0 T
In kr
Σ t=1
lle
1 12o)
1
Σ
E Ell Mo,
(5)
i
where Ut Mot are the true expected payof's. The first term, driven by the variation of the meta-game, leads to faster convergence in slowly changing environments such as those induced by GEMS.
3.4 Finding New Strategies with a Bandit Oracle
Solving the restricted meta-game alone insufficient; progress requires introducing new, challenging policies into the population. GEMS casts this search as multi-armed bandit problem (Robbins) (1952)). The "arms" comprise finite pool of candidate latent codes Åt, and pulling an arm corresponds to evaluating the policy G2( against the current meta-strategy dr.
For latent code A,, define
fx(2) = Ejra, [T(TGoLe).T;))
(6)
GEMS estimates this value, Pe(2), together with its empirical variance Var,(2) via Monte Carlo rollouts. To balance exploration and exploitation efficiently, it employs empirical-Bernsteinn Upper Confidence
Bound (EB-UCB) (Maurer & Pontil 2009)), which leverages variance information for tighter bounds. A Jacobian penalty encourages smoothhess in the generator's 's latent space, aiding optimization, where (Go(z)
denotes the Jacobian matrix of the generator with respect to its input z. The score assigned to candidate 2 is
Var(()) ln(3/5,)
(l/6)))
UCBB BB (z) = Ae(z) + /
n.m -1 Ar|IJGo(=)11;:
(7)
n.m
where 6, = = is decaying confidence parameter and d 2 0 is penalty coefficient. GEMS selects:
arg max UCBER (2), zEA,
(8)
and augments the anchor set:
Zel1 Z. U {2t).
Theorem 3.3 (Instance-Dependee Oracle Regret). Assume rewards lie in [0, 1] and the bandit problem admits unique best arm z" with sub-optimality gaps A, > 0. Under the tuo-time-scal assumption (fized during selection), the cunalative regret of the oracle salisfies
- o
Σ In
A
E [())G((=))11))
(10)
2
If the Jacobian norm uniformly bounded, the second term can controlled by annealing XJ.
Proof sketch. Standard bandit-analysis arguments apply. With high probability, the EB-UCB provides valid confidence bounds; comparing the chosen arm 2t with the optimal arm 2 and summing over time yields the stated instanco-dependent rogret bound.
iIn the context the Optimistic Multiplicative Weights Update (OMWU) algorithm, n the step size, often referred as the learning rate or gain parameter.

---

Published in Transactions on Machine Learning Research (02/2026)
3.5 Training the Generator with Amortized Best Response
Once promising latent code of has been identified, GEMS must ensure that the generator Go can realize the associated high-performing policy while retaining its ability to produce previously effective policies. This is achioved via an Amortized Best-Respont trust-region methods in deep reinforcement learning such as TRPO and PPO (Schulman et al. (2015 2017
with a Trust Region (ABR-TR) objective, inspired by
Rather than training new network from scratch, GEMS fine-tunes the existing generator Go to maximize the performance of curated set of promising latent codes, drawn from a distribution tt, against the opponent mixture Ø-1 A Ki-divereence penalty against frozen, older generator o serves trust region, mitigating
catastrophic forgetting:
LABR-TR = BBzgg, jmattt AArooio (I((FG(()) TG,- (=)) - >r1JGG(2211
(11)
where A denotes a suitable advantage estimator, 3 > 0 is hyperparameter controlling trust-region strength and (z) denotes the Jacobian matrix of the generator with respect to its input 2. Taking a few gradient- ascent steps on LABR-TR each iteration amortizes best-response computation while preserving a stable, single generator.
3.6 Overall Exploitability Bound
The preceding components now combine to yield a convergence guarantee for the entire GEMS algorithm. To make this composition explicit, we first state a decomposition result that separates the average exploitability into four interpretable sources of error:
inherent regret of the OMWU mela-strategy solver (under time-varying payolls),
noise from Monte Carlo payoff estimation,
sub-optimality from bandit-based oracle (anchor) selection,
approximation error induced by amortized generator training.
Amortized best-response gap. Let TBR (m)) be the true best response to an opponent policy my. We define the amortized best-response error as the worst-case expected gap (over itcrations) between a true best response and the policy produced by the generator at the selected latent code zf:
EBR = =sup Bjson
[r (T) r(Feos=))) 2 0.
(12)
Exploitability decomposition. Let M denote the (implicit) meta-game payoff operator and define the true payoff vector i == Mo, at itcration t. Lot 62 be the Monto-Carlo estimator of 0, computed from nm rollouts per queried matchup (as in Section 3.2). Let z be the (possibly approximate) anchor selected by the EB-UCB oracle at iteration t, and let A₂ denote the suboptimality gap for anchor 2.
Proposition 3.4 (Exploitability decomposition). Assume rewards lie in [0, 1]. At each iteration t, the instantaneous exploitability satisfies
Exploit( ( <
- U1 0z- oi)
2 - 2
+
EBR amortized BR
max (2) we vo(21)
OMWU meta-dynamica term
MC estimetion arror
oracle suboptimalitty
(13)
Here of denotes the best fized mela-strulegy in hindsight for the (lime-varying) payoff sequence.
Proof skelch. Starting from the definition of exploitability as a best-response gap against o,, we add and
subtract (i) Monte-Carlo payofis ér (ii) the best-anchor payoff max, v2(2), and (iii) the generator-induee best response at 2 Triangle incqualities yield the MC term and the amortized BR term, while the oracle term captures the loss from selecting 2 instead of arg max, v2)... The remaining meta-dynamics term is controlled by the OMWU guarantee for time-varying payoffs.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Overall finte-poppulation bound.
We now upper bound each term in Proposition 3.4 using the corre-
sponding component guarantees.
Theorem 3.5 PoiniattPooullaiion Exploitability Bound). Assume rewards lie in [0. and that oracle selection
operates on a two-time-scale schedule. With OMWU step size v = (v kr/T) and Monte- Carlo budget salisfying Blllé, vel1zo] ((mm))-1/2) for each i, the average exploilabilily obeys
Exploit(or) s o
In kT
+
(((mm) 1/2
T
1=1
t=1
MC estimation error
OMWU regret
(14)
Σ
(T/6))
EBR
A.
amortized BR error
2
oracle regret
Proof (one line). Apply Proposition 3.4 then substitute the bounds from the OMWU time-varying regret guarantee, the Monte-Carlo concentration bound for ||ô, velloo; and the EB-UCB oracle regret, bound, and average over t 1.,....,
This bound corroborates the intuition behind GEMS. As the simulation budget grows (nm + oc) and generator training improves (Enn + 0), exploitability is ultimately driven by the no-regret property of the OMWU solver, achieving convergence guarantees competitive with traditional methods while avoiding explicit quadratic mela-game storage.
3.7 Generalization to n-Player and General-Sum Games
Although the GEMS framework is introduced in the two-player zero-sum (2P-ZS) setting for clarity, the same principles extend naturally to the n-player general-sum (NP-GS) case. The amortized generator, Monte Carlo rollouts, and bandit oracle remain intact; only the meta-game formulation generalizes.
,n} maintains meta-strategy (P)), To evaluate player p's i-th policy against the joint strategy of all other players, Σ, P. GEMS reuses single batch of shared game rollouts and computes an eaportancc-weightee estimate:
an n- r-player game, each player p E [I....,,]]
B m 111 = Bm
(
(15)
(1) (1)
where (()) = ,.....(())
((.))......))) a joint policy profile drawn from the full meta-stratogy Σ, =x
00 0, and
(P) denotes the return for player p.
b.
Each player then updates independently: applying Multiplicative Weights (or OMWU) to e(2) and invoking a separate EB-UCB oracle to discover new strategies. This decentralized process no longer targets exploitability (a 2P-ZS notion) but instead drives the time-averaged joint strategy toward an --Coase-Coorrelated Equilibrium (cCCE), the standard solution concept for general-sum games.
Full derivations, algorithmic details, and convergence proofs for the n-player extension appear in Appendix
9

---

Published in Transactions on Machine Learning Research (02/2026)
3.8 Step-Siae Size (ETA) Scheduler for the OMWU Meta- Meta-Update
The OMWU update in c
requires step size Nr 0. In GEMS, simple, explicit schedule (7 is used,
chosen from three options that trade off adaptivity and stability:
y0
const,
770
=
vl
sqrt,
with default no = 0.08, a =0.5.
(16)
Yoo
11+at & > 0, harmonic,
These schedules instantiate the optimistic MWU update
(1) x (+) exp (/ 20, - 6,-1, - -P:l)
(17)
followed by normalization to cnsure 0+41 E A
Rationale. 'const' keeps a fixed step size and adapts quickly to changes in the restricted game, but can overreact to noisy payoff estimates. 'qgrt' docays gently (n t-1/2), a standard choice in online Icarming that balances responsiveness and variance. 'harmonic' decays as me t-1,, yielding more conservative late-stage updates and improved noise suppression.
Interaction with other knobs. The global slowdown factor scales Vo - 7o/s / (and independently shrinks the ABR learning rate while enlarging the KL trust-region coefficient). The temperature 7 2 1 used in the generator's logits is orthogonal to the meta step size and only affects policy softness.
Variation-aware regret with scheduled (7²). Let v = Mo, denote the true expected payoff vector at iteration & and assume rewards lie [0, 1]. Under unbiased meta-game estimates ($3), optimistic mirror descent with the entropic mirror map (OMWU) and nonincreasing step sequence {m²} satisfies the variation-aware bound
T maxe, e, Mo. - (..M)) x
In Kr
TJt 2 [=1
T
01
11
+ T
M t=1
- (18)
T
T
Ter
T
stability term
variation
MC noise
where by is the (growing) number of anchors time T. Equation 18 formalizes the tradeoff encoded
by equation 16 larger 7/s reduces the stability term but amplifies sensitivity payolf variation and sampling noise, while smaller тe does the reverse. In practice, 'sqrt' performs well when the meta-game evolves moderately (frequent oracle additions), whereas 'harmonic' is preferred in low-noise, late-phase training.
Corollary (Recovering Prop. 3.2 from scheduled-step bound). Let V/2 Σt 110e o. -- 11.. If the step size is constant, 77 17, then equation [8] becomes
max e
T Mo. of Mo.)
In KT
+
V2
+
E [1l - Σ t=1
(19)
Tn
2T
T
Choosing 17" / 2 / yields
max e, Mo. - o. o,.oo.))
<
V2
Vinkr V2
1
E
[[l velloo].
(20)
T
T
which matches Proposition 3.2 up to constants.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm 1: Generative Evolutionary Mota-Solver (GEMS)
Require: Initial anchor set Zn- = {z1,...,
Generator Go, Meta-strategy do (uniform), Horizon T, Budgets NMC, B, M, Steps K KABR. Rates 1, Coeffs B, As.
,ho 3
Output 'Trained Generator Gg, Final Meta-strategy σT, Anchor set Zr.
for t t= , T do Phase 1: Sample opponent batch {j=] 0+-1
Meta-Game Estimation (Monte Carlo)
Compute empirical payoff vector 8g Bl via Eq. (2):
01,2 Nue E Ei= ((FGo((.) GGo();))
Estimate mean game value F, via B joint samples (i,j) 0--1 x 0--1
Phase 2: Meta-Strategy Update (Optimistic MWU)
Compute optimistic gradient estimate (with 0g = 0):
gr f 20, 6-11-22 7,1
Update meta-strategy probabilities Vi E (I,..., 1.-1
ox(1) (1) (1) 9:(1))
Phase 3: Population Expansion (EB-UCB Oracle)
Generate candidate pool A₃ via stochastic mutation of top anchors for each candidate 2 A, do
Estimate mean return (e(z) and variance (ar vs or Calculate acquisition score UCB(2) via Eq. (7):
UCB(2) Pc(z) + (2)
(z)
M
Select optimal candidate: 27 + arg ma UCB(:)
Update anchor set: Z, + Z1-1 f2
Phase 4: Amortized Best-Response (ABR-TR)
Store current generator state: Oold
for 1: 1,... KABR do Sample batch of anchors z Z, and opponents j de
Compute Advantages Axa,te via GAE
Update 0 by ascending regularized objective (Eq. 11): - 0 a A (TG.(TGG/ \r1)
return Go 2x
3.9 Algorithm
Algorithm 1 provides the full procodural rcalization of the method described in $ Each itcration consists of four phases that mirror the preceding subsections: (i) Monte Carlo estimation of the meta-game values (Eq. equation 2). (ii) an OMWU meta-update using the optimistic hint (Eq. equation 17], (iii) population expansion via EB-UCB anchor selection (Eq. (7) and Eq. equation 13), and (iv) amortized generator training with a KL trust region (Eq. equation 11.. The only persistent state is the generator parameters 6, the
anchor set 2,, and the meta-strategy de, enabling GEMS refine equilibria while avoiding explicit policy enumcration or payoff-matrix storage.
4 Experimental Results
4.1 Equilibrium Finding in a Deceptive Messages Game
Setup and Objective. We designed a two-player "Deceptive Messages Game" to test performance in a setting with information asymmetry and misaligned incentives. The game features Sender and Receiver.
11

---

Published in Transactions on Machine Learning Research (02/2026)
The Sender privately observes the identity of "best arm" (out of K arms with different stochastic payoffs) and sends a message to the Receiver. The Receiver uses this message to choose arm. Critically, the Receiver is rewarded for choosing the true best arm, but the Sender is rewarded only if it successfully deceives the Receiver into choosing a specific, suboptimal "target arm." This creates a zero-sum conflict where the Sender learns to be deceptive and the Receiver must. learn to be skeptical. The goal of this experiment is evaluate the ability GEMS to solve strategically complex games and find high-quality equilibria. We aim to determine which framework allows the Receiver to more effectively see through the Sender's deception and converge to optimal policy of always choosing the best arm, thereby mullifying the Sender's deceptive strategies. We compare GEMS against suite of strong baselines: PSRO, Double Oracle, Alpha-PSRO and A-PSRO for iterations. The runs are the averaged over seeds.
Alpha-PSRO
A-PSRO
Double Oracle
GEMS
PSRO
0.175
Sender Reward
0.150
Receiver Reward
0.75
0.125
0.70
0.100
0.075
0.60
0.050
0.025
0.55
0.000
Training Iterations
Training Iterations
536.0
300
Cumulative Time (s)
535.8
Memory (MB)
535.6
200-
535.4
150
535.2
535.0
50-
534.8
Training Iterations
Training Iterations
Performance in the Deceptive Messages Game Top Teft: GEMS Sender's ability to deceive converges to zero. Top Right: GEMS Receiver's performance converges to the optimal reward of 0.8, outperforming all PsRo-based basclines.
Figure 3:
Results and Analysis. The results in Fig. show a stark difference in the learned equilibria. The average reward for the GEMS Sender rapidly converges to zero, indicating a complete failure to deceive its opponent. In contrast, all PsRo-based basclines maintain a positive sender reward throughout training, indicating they sustain a partially successful deceptive strategy. Conversely, the GEMS Receiver's average reward quickly converges to approximately 0.8, the maximum possible value in the game. The receivers trained with PSRO variants improve but plateau consistently failing to achieve the optimal reward. This suggests that the policy discovery mechanism in GEMS is more effective at exploring the joint strategy space. The combination of the EB-UCB oracle exploring diverse latent space and the single amortized generator representing a continuum of strategics may prevent. The system from getting stuck in the poor local equilibria that can trap methods which expand their discrete policy sets more conservatively. Furthermore, GEMS is upto 35x faster as compared to the PSRO variants. This experiment highlights that beyond scalability benefits, GEMS also demonstrates a superior ability to find high-quality solutions in strategically deep games.
significantly lower, suboptimal performance level,
12

---

Published in Transactions on Machine Learning Research (02/2026)
4.2 Equilibrium Finding in Kuhn Poker
Setup and Objective. We benchmark GEMS in Kuhn Poker (Kuhn 2016), classic imperfect infor- mation game where agents must learn mixed strategies involving blufing. We evaluate performance using exploitability, which measures how close a policy is to the Nash Equilibrium (lower is better). GEMS is compared against a suite of strong PSRO variants over 40 training iterations for 5 seeds. This experiment is designed to assess GEMS'S ability to find near-optimal policies an extensive-form game with imperfect. information. The hypothesis that GEMS'S architectural approach, which is exploring a continuous latent space with single generator, will allow it to find low-esplottability strategies more efficiently and in fewer iterations than methods based on expanding discrete set of policies.
Alpho
PSRO
NcuPL
P2PSRO
GENS
1200
Cumulative Time (s)
Sppletability
Mesmary (14B)
800
200
206
Training Iterations
Training Iterations
Training Iterations
Figure 4: Equilibrium Finding in Kuhn Poker over 5 seeds [0-4]. GEMS rapidly converges to a significantly lower exploitability than strong PSRO basclines and NEUPL (Left), while demonstrating efficiency in cunulative training time (Right).
Results and Analysis. The results in Fig. 4 show the convergence of exploitability for each algorithm. GEMS demonstrates the fastest and most direct convergence to a low-cxploitability policy. By iteration 40, GEMS achieves an exploitability of approximately 0.18, significantly outperforming the next-best baseline, E- PSRO, which only reached an exploitability of 0.44. The other PSRO variants, while also showing improvement, converged at a considerably slower rate and achieved higher final exploitability within the 40-iteration budget. The superior performance in Kuhn Poker highlights GEMS'S strength in solving games that require nuanced, mixed strategies. The core of Kuhn Poker involves probabilistic actions like bluffing, which are difficult to represent as simple combination of a few deterministic policies. We argue that GEMS'S single amortized generator, operating over continuous latent space, is naturally suited to reprosenting these complex mixed strategies. In contrast, methods that expand a discrete set of policies, like the PSRO family, must approximate mixed strategy through a convex combination of many individual policies, which can require far more iterations to converge. The EB-UCB oracle effectively guides the search through the generator's 's latent space to find strategically potent policies quickly. This experiment demonstrates that GEMS's advantages extend beyond pure scalability to superior sample efficiency in solving canonical game- theoretic benchmarks.
4.3 Performance and Scalability on Multi- Tag
Setup and Objective. We conduct our analysis in the Simple Tag environment from PettingZoo (eery et al. 2021), where three cooperative pursuers must learn coordinated strategies, such as flanking, to capture a faster evader. This benchmark is designed to reward sophisticated coordination while punishing naive "herding" policies. We compare GEMS against classical PSRO on emergent behavior, mean return, and scalability (memory and time) over 100 iterations, averaged across 5 sceds. This experiment is designed to provide holistic comparison and test three foundational hypotheses. First, that GEMS overcomes the critical scalability bottlenecks of PSRO. Second, that this efficiency does not come at performance cost. Third, that GEMS learns policies of a higher strategic quality, both quantitatively (as measured by mean return) and qualitatively (as observed in emergent agent coordination).
13

---

Published in Transactions on Machine Learning Research (02/2026)
Results and Analysis. The results demonstrate clear advantage for GEMS across all aspects of evaluation. A qualitative analysis of agent trajectories (Fig. 5) reveals significant differences in strategy. The GEMS- trained adversaries exhibit coordinated flanking and cornering behaviors to trap the evader. In contrast, the Psro-trained adversaries often display a simpler "herding" behavior, pursuing the larget in less coordinated clump. This stralegic difference is reflected in the quantitative results (Fig. G GEMS consistently achieves higher mean agent return, stabilizing around 0, while PSRO'S average return fluctuates in lower range. Concurrently, GEMS is -6x faster and its memory usage remains flat at -250 MB, while PSRO'S memory grows to over 2350 MB and its cumulative time scales quadratically, The combined results show that GEMS provides a Pareto improvement over PSRO, achieving superior performance in solution quality, strategic complexity, and efficiency. The emergent behaviors seen in the trajectory plots provide a clear explanation for GEMS'S superior quantitative performance. The discovery of sophisticated, coordinated strategies like flanking directly translates to higher average returns. This suggests that GEMS'S exploration mechanism, driven by the EB-UCB oracle over diverse latent space, more effective at escaping the local optima that lead simpler behaviors like herding. The scalability results reconfirm that the single amortized generator and Monte Carlo sampling resolve the foundational bottlenecks of PSRO. Ultimately, GEMS presents complete advantage: It learns better strategies, which leads to higher returns, while requiring fraction of the computational resources.
GEMS
PSRO
Figure 5: Emergent agent trajectories in the multi-agent tag environment. Top row: GEMS. Bottom row: PSRO. Columns show uniformly sampled frames from a single rollout (frames 0, 10, 20, 30, 40, of the 50-frame GIF). This figure qualitatively compares the strategies learned by GEMS and classical PSRO. The top row shows that adversaries (red circles) trained with GEMS learn sophisticated, coordinated strategies like flanking and cornering to effectively trap the evader (green dot). In contrast, the bottom row shows that PSRO-trained agents adopt a less effective "herding" behavior, pursuing the target in a single, uncoordinated group. This clear difference in stralegic complexity consistent with the superior performance and higher returns achieved by GEMS, as reflected in the quantitative results.
5 Conclusion
Policy-Space Response Oracles (PsRO) and its many variants have driven much of the progress in population- based multi-agent reinforcement learning, but their reliance explicit policy populations and dense payoff matrices imposes fundamental barriers to scalability, In this work, we introduced GEMS, a surrogate-fre framework that breaks from this paradigm by maintaining compact anchor set, querying payoffs through unbiased Monte Carlo rollouts, and training policies via a single amortized generator.
14

---

Published in Transactions on Machine Learning Research (02/2026)
3900
GEMS
GEMS
Cumulativv Mean Return
Curulaaivv Time [5)
Memory (MB)
12048
-28-
17000
-46
7508
2008
GEMS PSRO
.44
-66-
50
330
30
66
300
130
Training Iterations
Traniing Iterations
Training Iterations
Figure 6: Performance and Scalability of Gems vs. Psro in Multi-Agent Tag. Compared to classical PSRO, GEMS achieves a higher and more stable mean return (Left), while maintaining a constant memory footprint (Middle) and near-linear cumulative training time (Right). The results show GEMS overcomes the core memory and computational bottlenecks of GEMS while learning more effective policies
Our approach removes the linear memory growth and quadratic computation overhead inherent to classical PSRO while preserving key game-theoretic guarantees. We provided theoretical analysis establishing unbi- asedness of meta gradients, regret bounds for EB-UCB policy selection, external regret for meta-dynamics. and mnite-population exploitability guarantocs. Empirically, GEMS demonstrates faster convergence, lower memory footprint, and improved scalability across challenging multi-agent benchmarks.
Returning the tournament analogy, GEMS shows that one does not need to schedule every possible match recruit new player for every playstyle. Instead, rankings can be inferred from manageable set of sampled matches, and versatile athletes can flexibly adapt to new strategies. In the same way, GEMS turns the exhaustive bookkeeping of PSRO into lean, adaptive process that scales naturally with problem complexity.
6 Limitations
While GEMS presents significant step toward scalable, surrogate-free multi-agent learning, we acknowledge the limitations that define the scope of this work and offer avenues for future research.
Benchmark Scope and Baselines: Our empirical evaluation focuses on demonstrating the fundamental scalability and game-solving advantages of GEMS over the PSRO paradigm. Consequently, some of our experiments (e.g., Multi-Agent Tag and Simple Spread) use classical PSRO as the primary baseline. This choice is deliberate, classical PSRO perfectly embodies the core O(k²) computational and O(k) memory bottlenecks that GEMS is designed to solve. To validate GEMS's strategic performance, we also benchmarked it against a suite of modern, stronger PSRO variants on complex games like Kuhn Poker and the Deceptive Messages Game. However, we did not conduct experiments on large-scale benchmarks such as the StarCraft Multi-Agent Challenge (SMAC). The prohibitive computational cost of training even a single bascline PSRO agent on such complex maps made a direct comparison infeasible with our available resources.
Hyperparameter Sensitivity: GEMS introduces a new set of hyperparameters related to the amortized generator, the ABR-TR objective (e.g., B, CABR-TR learning rate), and the EB-UCB oracle (e.g., candidate pool size |A,)). While we provide ablation studies the Appendix to analyze the sensitivity of key parameters in environments like Kuhn Poker and the Public Goods Game, a comprehensive, large-scale analysis of the interplay bet all hyperparameters across diverse game types is beyond the scope of this initial work.
Generator Capacity: The efficacy of GEMS relies on the capacity of the single amortized generator Go to represent rich, continuous space of diverse and effective policies. This work utilized standard MLP architectures for the generator. The performance of GEMS in much more complex domains (e.g., with high-dimensional observation/action spaces) may depend heavily on the choice of more
15

---

Published in Transactions on Machine Learning Research (02/2026)
advanced generator architectures (e.g., Transformers, diffusion models). Exploring the architectural limits of the generator is a key direction for future work.
Candidate Pool Generation: The EB-UCB oracle's ability to find effective new strategies is dependent on the quality of the candidate pool A, it searches over. Our current implementation relies on simple hcuristics for generating this pool (c.g., mutation and random sampling, detailed in the Appendix ablations). The performance of GEMS could potentially be improved by incorporating more sophisticated or guided methods for generating candidate latent codes.
References
Constantinos Daskalakis and loannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. arXiv preprint arXit:1807.02252 2018.
Elad Hazan. Introduction to online convex optimization, 2023. URL http::///aatxv.oo/////////////
Yudong Hu, Haoran Li, Congying Han, Tiande Guo, Mingqiang Li, and Bonan Li. A-psro: A unified strategy lcarning method with advantage function for normal-form games. arXiv preprint arXiv:2308. 12520, 2023.
Yihong Huang, Liansheng Zhuang, Cheng Zhao, and Haonan Liu. Efficient double oracle for extensive-form two-player zero-sum games.. In International Conference on Neural Information Processing, pp. 414 424. Springer, 2022.
Harold W Kuhn, A simplified two-person poker. Contributions the Theory of Games, 1:97-103, 2016.
Marc Lanclot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified thee-the approach to multiagent reinforcement learning. Advances neural information processing systems, 30, 2017.
Shuxin Li, Xinrun Wang, Youzhi Zhang, Wanqi Xue, Jakub Cerný, and Bo An. Solving large-scale pursuit- evasion games using pre-trained strategies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11594, 2023.
Jiesong Lian, Yucong Huang, Chengdong Ma, Mingzhi Wang, Ying Wen, Long Hu, and Yixue Hao. Fusion-psro: Nash policy fusion for policy space response oracles. arXie preprint arXiv:2405.2100 2024.
Siqi Liu, Mare Lanctot, Luke Marris, and Nicolas Manfred Otto Heess. Simplex neural population learning: Any-mixture bayes-optimality in symmetric zero-sum games. In International Conference on Machine Learning, 2022a. URL https://api. http:pppi.eemaaiicccccooaaa..oo wwtp://oo.....rreooooooooo...D: 2ttp//oorpusID:2422993311
Siqi Liu, Luke Marris, Daniel Hennes, Josh Merel, Nicolas Heess, and Thore Graepel. NeuPL: Neural population lcarning. In International Conference on Learning Representations, 2022b. URL https: :
//openreview. Soopenreviev.nntt/oomr/dddddld aett//orrummiiiIII333...
Siqi Liu, Luke Marris, Marc Lanctot, Georgios Piliouras, Joel Z. Leibo, and Nicolas Heess. Neural population learning beyond symmetric zero-sum games. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS '24, pp. 1247-1255, Richland, SC, 2024. International Foundation for Autonomous Agents and Multiagent Systems.
Andreas Maurer and Massimiliano Pontil. Empirical bernstoin bounds and sample variance penalization, 2009. URL httpss:////arit www.axiii.org//b.... 099000907797
Stephen McAleer, John B Lanier, Roy Fox, and Pierre Baldi. Pipeline psro: A scalable approach for finding approximate nash equilibria in large games. Advances in neural information processing systems, 33: 00238-202444 2020.
Stephen McAleer, John B Lanier, Kevin A Wang, Pierre Baldi, and Roy Fox. Xdo: A double oracle algorithm for extensive-form games. Advances in Newral Information Processing Systems, 34:23128 23139, 2021.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Stephen McAleer, John Banister Lanier, Kevin Wang, Pierre Baldi, Roy Fox, and Tuomas Sandholm. Self-play plaf-play psro: Toward optimal populations in two-player zero-sum games. arXiv preprint arXiv:2207.06641, 2022a.
Stephen McAleer, Kevin Wang, John Lanier, Mare Lanctot, Pierre Baldi, Tuomas Sandholm, and Roy Fox. Anytime psro for two-player zero-sum games. arv preprint arXi::2001.0700 2022b.
Paul Muller, Shayegan Omidshafiei, Mark Rowland, Karl Tuyls, Julien Perolat, Siqi Liu, Daniel Hennes, Luke Marris, Mare Lanctot, Edward Hughes, et al. generalized training approach for multiagent learning. arXiv preprint Xix: 12823, 2019.
Herbert E. Robbins. Some aspects of the sequential design of experiments. Bulletin of the Americon Mathemalical Sociely, 58:527-533 535, 1952.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp.
PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 700.063477 2017.
Zhengdao Shao, Liansheng Zhuang, Yihong Huang, Houqiang Li, and Shafei Wang. Purified policy space response oracles for symmetric zero-sum games. IEEE Transactions on Neurol Networks and Learning Systems, 2024.
Max Olan Smith, Thomas Anthony, and Michael P Wellman. Iterative empirical game solving via single policy best response. arXie preprint arXiv:2106.01901, 2021.
Max Olan Smith, Thomas Anthony, and Michael P Wellman. Strategic knowledge transfer. Journal of Machine Learning Research, 24(233):1199) 2023.
Hongsong Tang, Liuyu Xiang, and Zhaofeng He. Policy similarity measure for two-player zero-sum games. Applied Sciences, 15(5):2815, 2025.
Jordan Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S
Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicenie Vicente, et al. Pottingzoo: Gym for multi- agent reinforcement learning. Advances in Neural Information Processing Systems, 34:15032 15043, 2021.
Xinrun Wang, Jakub Cerny, Shuxin Li, Chang Yang, Zhuyun Yin, Hau Chan, and Bo An. A unified perspective on deep equilibrium finding, arXic preprint or arx.v2200 04930, 2022.
Zelai Xu, Yancheng Liang, Chao Yu, Yu Wang, and Yi Wu. Fictitious cross-play: Learning global nash equilibrium in mixed copperatie-cooppettive games. arXiv preprint arXi::2310.0855 2023.
Jian Yao, Weiming Liu, Haobo Fu, Yaodong Yang, Stephen McAleer, Qiang Fu, and Wei Yang. Policy space diversity for non-transitive games. Advances Neural Information Processing Systems, 36:67771-6777 2023.
Ming Zhou, Jingxiao Chen, Ying Wen, Weinan Zhang, Yaodong Yang, Yong Yu, and Jun Wang. Efffciintt policy space response oracles. arxiv preprint or 220: 00633, 2022.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. Advances in neural information processing systems, 20, 2007.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix Table of Contents
I. Mathematical Derivations and Prools
18
A. Formal Preliminario
10
B. Prolss for Mela-Game Estimation
C. Proofs for Meta-Gamt Solving
21
D. Proots for the Bandit Oracle
E. Proof of Overall Exploitability
24
II. Extensions to Two-Player General-Sum and N-Player General-Sum Games
Extension to Tloo-layer General-Sum Games
G. Proofs for Two-Player General-Sum
H.
Extension N-Player General-Sum Games
I.
Proofs for Part IIII (N-Player General-Sum Games)
32
III. Ablation and Analysis of experiments
J.
Run on Connect-1
K. Run on Hanabi
L. Coordination on Simple Spread
42
M. Coordination on Simple Tag
46
N. Run on Chess
49
O.
Run on Go
Ablation on Kuhn Poker
Q. Ablation on Public Goods Game
R. Ablation on Deceptive Message
S. Ablation on Oracle Selection
IV. Frequently Asked Questions
18

---

Published in Transactions on Machine Learning Research (02/2026)
Part
Mathematical Derivations and Proofs
This appendix provides the detailed mathematical analysis supporting the claims made in $3 We begin with formal definitions and then proceed to prove the lemmas, propositions, and theorems for each component of the GEMS algorithm.
A Formal Preliminaries
A.1 Two-Player Zero-Sum Markov Games
A two- pro-player, zero-sum, fmite-horizon discounted Markov Game is defined by a tuple M
(S, (A1, A2], P, Po)-
S is the state space.
Ap the action space for player p [1,
P:Sx x A + (SS) is the state transition function.
S x A1 x A2 + R is the reward function. Player aims to maximize the reward, and Player 2 aims to minimize it, such that Rí = and R2 = RR.
E [0, 1) is the discount factor.
Po ((S) is the initial state distribution.
A policy for player denoted Tp. is mapping from states to distribution over actions, TP S - A(Ap).
The expected return for Player playing policy my against policy Ty is:
(TT nT) E
Σ 7 R((,,,
(21)
For a finite set of k policies A {m..... m the associated payoff matrix is M E Rkxk, where Mij (TTs: By convention, assuine rewards are normalized to [0. 1].
A.2 Nash Equilibrium and Exploitability
A mixed strategy (or meta-stralegy) is probability distribution over the set of policies A, i.e., E A A Nash Equilibrium (NE) o² is a meta-strategy from which no player has an incentive to unilaterally deviate. In a two-player zero-sum game, this is equivalent to the minimax solution:
arg min max 01 Mo2
dIE -EAA-1
(22)
arg max min 01 Moz. o1 oG
The exploitability of a meta-strategy a measures the incentive for an opponent to play a best response. It is defined as the gap between the payoff of the best pure strategy response and the payolf of the meta-strategy itself.
Exploit(a) max(M (M Te: T- (-o (Ma)
kE
(23)
= max e Mo - o Mo.
ie k]
19

---

Published in Transactions on Machine Learning Research (02/2026)
Note: The term -a" Mo is the value of the game from Player 2's perspective. For symmetric game where players draw from the same population (M (M=-a -MM this simplifies to the expression in Eq. (1).
B Proofs for Meta-Game Estimation
B.1 Supporting Definitions: Empirical-Bernnn Bernstein Inequality
Before proving Lemma 2.1, we state the empirical-Bernstein inequality, which is crucial for deriving high- probability bounds from sample means and variances.
Thcorem B.1 EEmpirical-Bernstorn Incquality). Let X1,..., XN be i.t.d. random variables with meon 1t and variance o²_ Assume they are bounded, X; E [a, b]. Let X = N E X, be the sample mean and 0² N Enn (X; x)2 be the sample variance. Then for any (0, 1), with probubility at least 1 8:
2 In(3/))
3(b a) (n366)
X pl S
(24)
N
N-1
B.2 Proof of Lemma 2.1 (Unbiasedness and Concentration)
Lemma B.2 (Unbiascdness and Empirical-Bernstor Concentration). With rewards [0, 1], the estimators are unbiased: E[ (Mor); and B[f:]] of Mor. Moreover, for any d E (0, 1), with probobility al least - 8, the estimators concentrate around their true means:
2 2ar,, In(2/5)
(2/(2/
00. (Mor))
(25)
n&m
T4m 1
[n(1/8)
1 Mor
(26)
Bm
where Var is the empirical variance of (Yr,,,e).
Part 1: Unbiasedness of 0z,à
The estimator is 0g, 1 D E= Yist where opponent js is sampled as j. ~ 02. We use the law of total expectation.
1 = E
14:771
8=1 1=1
m E(Y.,.e]
(Linearity Expectation)
(27)
7:m
8=1
m
Bja a. jall (Law of Total Expectation)
7,m
Given fixed opponent js. the expectation of a single rollout Yes is the true expected return r(T,, Tg.) = M
E
Bj Bja [Maj.]
7&m
o, (i)
(Definition of Expectation)
74m
(28)
T4:77
acli)MM
1=1
= (Mo
20

---

Published in Transactions on Machine Learning Research (02/2026)
The proof for E[îi] of Mo, follows identical argument, where pairs (16. are sampled from de X dt. Part 2: Concentration of 6,2
The estimator dr, is the sample mean of N = 7,m random variables (Y,,.e). Each is a game return bounded in [0, 1]. These samples are i.i.d. drawn from the compound distribution defined by sampling an opponent j de and then sampling return from the matchup (T,, πz). We can directly apply the
Cuapirical-Benssted Inequality (Theorem B.1) with N nym, a 0, = Replacing the generic confidence parameter 3/6 with 2/8 (a minor variation common in literature) gives:
2Vrr,, In(2/0)
3lm(2/))
1ōt, - (Moo) S
+
(29)
14m
74m -
with probability at least 1 1 6.
Part 3: Concentration of fz
The bound for i. a standard application Hoeffding's incquality (or Bernstein's variance is considered). Since returns are [0, I], Hoeffding's inequality for the mean of Bm i.dd. variables states that for any e 0:
Mo, 2 e) s 2eex((22(Bm)) P P - o, Mool
(30)
Sctting this probability to 8, we get 5 = 22xpp(-22222 -2Bmc²). Solving for c:
In(2/5) =0 (v
(In(1/8) Bm
ln(6/2) = -2mm² = c=
(31)
2Bm
This confirms the simplified O(-) bound.
C Proofs for Meta-Game Solving
C.1 Proof of Proposition 3.2 (External Regret of OMWU with Unbiased Noise)
We provide a full derivation for the external regret bound when using the Optimistic Multiplicative Weights Update (OMWU) algorithm with noisy, unbiased payoff estimates. The proof proceeds two parts. First, we establish regret bound with respect the sequence of estimated payoffs é, Second, we translate this bound to the regret against the true expected payoffs U, Mo, by accounting for the Monte Carlo estimation error.
Let the estimated loss for policy i at time t be le,5 = 1 Ōz,i- The OMWU algorithm uses an optimistic loss estimate m. = 1 + (l. 11-1,, with in = O. The weight update rule is we+ (6) wr(1) exp(-1ma.))
Part 1: Regret Against Estimated Losses
Let W, = Eie wi) be the potential function. We analyze its evolution.
W411 = Σ WH+1(1) (1) = Σ (i) exp( (1mm)))
(32)
W, Σ (el exp(-mmme.)
Taking the natural logarithm, have:
Σ (1) expl (((())epp(mmm))
In(W++1) - In.)) = In
(33)
21

---

Published in Transactions on Machine Learning Research (02/2026)
Using the inequality In(E(e*]) 2 B[X] + ] for random variable X (Hoeffding's lemma for bounded
variables), where the expectation is over i ~ and X = -
In(W - In(,,) Σ (-)m))))))))))
(34)
= mi) 112 (or, mz)
Substituting m. = 14 (le f(-1):
(or, = (o), + (or, (4-1)
(35)
A key step in the OMWU analysis is to relate the second term to the loss of a fixed expert i. For any vector 2, we have the inequality (or, 2) - 2, I 2n
1 (In(or, ( - In e ((0")), A simpler path involves relating the
regret the squared difference of consccutive losses. Rearranging the potential function bound:
( In(W,) - In((++1) - -o
" (or.m)) m²)
(36)
"
For any fixed expert i", also have a lower bound on the potential: In(WT+1) 2 In(wT+ (i*) = In(wi (i*))
ni E My," Assuming wi (i) = 1/ki, we get In(Wr+1) 2 -I((k) - > 2i=1 M7
The standard OMWU analysis (e.g., in Hazan, "Introduction to Online Convex Optimization' shows that these steps lead to a bound on the regret against the estimated losses:
Hazan
2023
T Σ
T Σ les
In KT
1 113
(37)
1
1=1
Choosing an optimal learning rate 17
In kr
yields:
Σ
114, 1/11-11-1113 113
(or min Σ te,
I 2,
In kr
|10e
4
(38)
Part Translating True Regret
Now we translate this result to the true losses 4, = 1 -
The true external regret is Pdrac = Et=i(lor
(, Let the estimation error be A, = l, 4 Ur 0z.
T
Ryp
pirue
(lo 1--1--44 -Ai) -(e, - -.,-))
T
T Σ ((a,, A.s.)
(39)
Σ ((o,, (1) - 1.x
Regret on Estimates
Cumulative Error Term
The first term is bounded as derived in Part 1. We now bound the variation term using the triangle incquality:
d = 11(e- ( (t) + (1e - 4-1) +
< ||Ar|loo + ||4e - 4-111 + I|A
(40)
Substituting this into the bound from Part 1 introduces terms related to the estimation error. Taking the expectation over the sampling noise in our estimators 0z (and thus [c), and noting that E[A.] 0 due to unbiasedness, we can bound the expected true regret. The error terms accumulate, leading to the final bound.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Dividing by T and converting losses back to payoffs
(- =Σ Enr T) yields the proposition:
T max 04, (or, 0))
T
T Σ
0
In kn
1 - 0z- 11 Σ
Bllb; - 0z [oo]
(41)
T
This completes the proof.
Scope. Throughout this appendix we analyze OMWU with constant step size 1j in Prop. 3.2; the scheduled bound stated in $3.8 follows from the same potential-based argument and reduces to Prop. 3.2 by setting 7/ 1 and optimizing 1).
D Proofs for the Bandit Oracle
D.1 Proof of Theorem 3.3 Dussaance-Dependen Oracle Regret)
Theorem D.1 Dependeet Oracle Regret). Assume rewards in [0, 1] and that the bandit problem has unique best arm x* with suboptimality gaps A. = fe(=*) - ft(2) > Under the two-time-soale assumption (fized of during selection), the curalaatve regret of our oracle is bounded:
Σ IT
((((=))-((=2))) =
AJ
E [1]G6((2))1
(42)
Let's fix single bandit problem at meta-iteration (we drop the subscript t for f(z) and A, for clarity). Let 27 be the arm chosen at time t. The regret at this step is A:; (ignoring the Jacobian term for now). The total regret over steps of the oracle is Ri = Σi= A.p Let (T)) be the number times arm & pulled in T steps. Then Ry Exper (T)T),, Our goal to bound E[N,(T) for any suboptimal arm #*
An z is chosen at time if UCB,(=) 2 UCB,(=') for all L' A. For suboptimal chosen, it must be that UCB:(z) 2 (*"). Let (=(2) be the empirical mean for arm z after it has been pulled n₂ times. The UCB is:
* x* to be
2 Var(z) In(388)
UCB(2) = ((2) + C(n.. 8), where C(nz, 8) =
(43)
n.m
nmm
The concentration bounds from Lemma 2.1 (applied here to a single policy against the mix oe) imply that with probability at least 8:
A(z) C(n,, S f(z) s f(2) + C(n,,
(44)
Let's call the event that these bounds hold for all arms and all steps a 'good event' G. By setting 6, = t and taking a union bound, the probability of G high. We condition the rest of the proof on g.
If suboptimal arm z is chosen time 4, then UCB.(=) UCB,(=*). Using the bounds:
f(=') p+(=") C(n.r UCBi(=") (LCB for optimal arm)
< (=) = -+(=)) + C(n,, 6z)
(Suboptimal arm was chosen)
(45)
< f()) + +2C(n, &c)
(UCB for suboptimal arm)
This implies f(z*) - f()) < 20(n,, 8c), or A < (n 8z). The confidence term C(ny, 6t) decreases roughly as 1/ /ng. So, for the inequality A < 20(n,, ( 5,) to hold, n, cannot be too large.
2 Var(z) (2) ln36,))
In (3/6)))
4, 2
(46)
7mm
mm
23

---

Published in Transactions on Machine Learning Research (02/2026)
Assuming rewards in [0, 1], variance is at most 1/4. Simplifying, require n, to be roughly:
Vin(1/ór)
In(1/6.)
ln(t)
Vn, S
n1 00
=0
(47)
A
A
A2
This means that suboptimal arm z will be pulled at most O 0 T/) times. (Note: A lighter analysis for UCB variants gives O(nT/A.)). Let's follow the standard argument for UCB1 which leads to the tighter bound. A suboptimal arm is played at most 'c' times for some constant, and then only if ( s 2lnt
(2)1(2)+ + V N=-1(=) V (=-) Summing the number of pulls leads to the logarithmic dependency. For EB-UCB, the analysis similar but replaces the fixed variance proxy with the empirical variance, leading the same asymptotic form. The expected number of pulls for suboptimal arm 2 over T oracle steps is therefore (T)) T/Az).
2lnt_
The total expected regret is:
E R = Σ (T), = Σo 2722"
In
A. - -0
Σ 2212**
In T
(48)
A
2
Now, re-introduce the Jacobian penalty. The algorithm maximizes UCB (=) The regret defined with respect to the true arm values f()). The penalty term is an additive component in the objective, which translates to additive component in the regret. The total regret is the sum of the standard bandit regret and the expected penalty of the chosen arm.
T Σ
I
T A
- (a()) So
+
E E
[IJGG((E)) - [=GG(=") [
(49)
A
Since the norm is non-negative, we can bound the second part by simply summing the penalty of the chosen arm, which gives the stated result.
E Proof of Overall Exploitability
E.1 Proof of Theorem 3.5 (Finite-Population
Population Exploitability Bound)
Theorem E.1 PopulaaPopulltiion Exploitability Bound). Assume rewards in [0, 1] and the two-time--cale oracle selection. With optimistic meta-solver (OMWU) and sufficient Monte Carlo samples such that E[]l0, - Mox]loo] = (((m))(mm))))))) -1/2), the average exploilability is bounded:
T (nm) 11/2
1
T
Exploit(o,) 50
T
KT
F - 0z 11
T
Σ
OMWU Regret
MC Estimation Error
(50)
hT A
SBR
+0
Σ T
T
Amortized BR Error
Orucle Regret
The proof connects the exploitability the meta-strategy oe to the various regret and error terms of the algorithm's components. We start by decomposing the exploitability at iteration t. Let A (#1,..., Tk. be the set of policies in the anchor set.
Exploit(a,) max Ej-on [r(m, TT)] Ei (r
(51)
24

---

Published in Transactions on Machine Learning Research (02/2026)
Let TBR, arg max, By~o, [TT, my)] be the true best response to de. Let aet (eo)ee) be the policy added by our oracle. Let सe arg maX) e/ Mo, be the best pure strategy within the current anchor set. We decompose the exploitability follows:
(xploit(o:) En (TBBRR Ty)] o. Mor
max 6 Mo - oz Moi (k[ks
Eynor (roTnrr(TBnnn x,)l max e, Map
(52)
=E[k;]
Term I: Internal Exploitability
Term II: Population Gap
Bounding Term This is the external regret of Player 1 in the restricted game on A. From Proposition 3.2, the average regret of the OMWU algorithm using noisy estimates is bounded by the variation of the true payoff vectors:
Σ max 9 Mo. - or Ma. 0O? T
Kr Σ
111
+
Blll Mo+
(53)
T
sE(kg]
The first part is the OMWU regret, reflecting faster convergence rate in smoothly evolving meta-game, and the second is the MC estimation error.
Bounding Term II: This term captures how much better true best response is compared to the best policy already in our population. We can decompose this further:
Term II (Ejwar (T(BBR,) - Byso, :
(a) BR Approx. Error
+
E Rynar [M(=a
max Ejno, (oroorr(rrrrr [r(TGG(a)::TT)) =EA,
(b) Oracle Instant. Regret
(54)
max e T MM,, max Ejna, [T(TGG(s) =Ed,
(c) Progress Term
(E().)
Part (a) is exactly the amortized best-response error, EBR, from Eq. (10), assuming TBR, can be represented by some latent code.
Part (b) is the instantaneous regret of our bandit oracle. Its expected value what bounded in Theorem 3.3.
Part (c) is non-positive. Since the anchor set Ze is a subset the candidate pool A,, the maximum over A, must be greater than or equal to the maximum over Ze. So we can drop this term from the upper bound.
Summing everything and averaging over T:
Exploit(o,)
<
(Temm I + Term II,)
T
T
T
1
<
0
In KT
1t 1z
22
Bllle, Morloo
T
(55)
Torm
1
1
max f((2) f((2) SEA,
BR,
T
T
Term II
25

---

Published in Transactions on Machine Learning Research (02/2026)
The last term is the average instantancous oracle regret. The cumulative regret bound from Theorem 3.3 states Σ, B[max= ft(=) Jı(2;)) = OfE+,. ) Therefore, the average regret is 0 Σx a.)). Substituting this and the rate for MC error gives the final composite bound:
1
1 T
1 +0
T (nm) -1/2
Exploit(or) so
In KT
F0 1211113
Σ
1/2
T
1=1
OMWU Regret
MC Estimation Error
(56)
In A
EBR
T 27'2"
Amortized BR Error
Oracle Regret
This completes the proof. Each term corresponds to component of the algorithm, showing how errors from different sources contribute to the overall performance. As T + so, if the sample counts (n, m) increase and the generator training improves (éBR - 0), the average exploitability converges to zero, with the rate of convergence for the meta-game being accelerated by the optimistic updates.
26

---

Published in Transactions on Machine Learning Research (02/2026)
Part II
Extensions to Two-Player General-Sum and N-Player General-Sun Games
F
Extension to Two-Player General-Sum Games
We extend the surrogate-free GEMS framework two-player general-sum Markov games. Let rewards for each player be bounded in [0, 1]. For player E (1, 2}, we denote the other player by -p. The conceptual expected payoff matrices are M(2) [0, 1j and M [0, 1| where M = E|r (T. (1) (()) and
M = (x. ). At iteration i, each player p has a population of policies induced by an anchor set
- with a corresponding mixture strategy of Ax,(0)-1, where kp = (((()=
The value vector for player against player 2's mixture ofth M (*) Similarly, for player 2 against player I's mixture, it is of = (M)) of E 1R The expected value for each player under the joint mixture is T(P) =o(") M(
F.1 Monte Carlo Meta- Game Estimators (Both Players)
At a fixed iteration t, to estimate the value vector vi for player p. we sample opponents j. of-m) and run m episodes for each pair to obtain returns. The estimators are:
22
(1)
(1) (b..).)
is ~(
7&m
(57)
(
(2)
i, ~ (1)
t.
1=1
The mixture value for each player is estimated via joint sampling (bs. ~ of.1) (2)
(P)
(p)
(58)
Bm
Lemma F.1 (Unbiasedness and Concentration, General-Sum). With rewards in [0, 1], the estimators are unbiased: B[o{2]] ( ),, Blo!3) ((M(2) Tf ) and B(((") r!") For any (0, with probability at least - 6, we have concentration bounds based the empirical Bernstein inequality.
F.2 Optimistic Replicator Dynamics for Both Players
Each player (1,2) independently performs Optimistic Multiplicative- Weights Update (OMWU) step based on their noisy payoff estimates. This allows cach player to adapt more quickly to the opponent's evolving strategy by using the previous payoff vector as hint. The optimistic estimate for player is m(p) 20(p) 0 with ofF) 0. The update rule is:
(1) x (p) (1) exp mn ( s(p) - 1 11,1),
o (1)
n >
(59)
27

---

Published in Transactions on Machine Learning Research (02/2026)
Proposition F.2 (Per-Player External Regret with OMWU and Noisy Payoffs). Let payoffs lie in [0, 1]. The average external regret for each player p using OMWU is bounded by the varialion of their true payoff vectors:
7
1
Σ
max(v
( v//0)
<0
In
Rig.
T
Σ 110
11
(60)
T
110(
1 110
E
where ( is the true payoff vector for player p (i.e., u1
M (2) and v (o
F.3 Model-Free EB-UCB Oracles (Double-Orade Oracle Style)
For each player p, finite candidate pool of latent codes A C 2 is formed. The value of an "arm" A is its expected payoff against the opponent's current mixture:
1 (z)
(n(P) -
(61)
We estimate the mean ( (z) and variance Var, (p) (=) via rollouts and score each candidate using empirical- Bernstein UCB formula. new anchor 2," z-((p) selected and added to the player's anchor set.
Theorom F.3 (Per-Player Distanco-Depenentt Oracle Regret). Under a twoo-time-scale assumption, the cumulative regret of the EB-UCB oracle for player is bounded as stated in Theorem 3.3.
F.4 Overall Guarantee: e-Coarse-Corrlate Equilibrium
The time-averaged joint play of the players, (i((i.)) ( (f), converges to e-coarse- correlated equilibrium (e-CCE).
Thcorem F.4 (c-CCE of Time-Average Joint Play). Assume rewards in [0, 1] and the two-time-scale assumption. With per-player OMWU meto-solvers, the empirical distribution per is an e- e-CCE, with bounded by the sum of per-player errors:
Σ
0
In T
(p)
(p) -1-1111 11.
O
Σ
(m -1/2
T
(1.2))
OMWU Regret
MC Estimation Error
(62)
In T
BR,
A
(p)
Amortised BR Ervor
Oracla Regret
(p) where EBR the average best-response opprozimation ervor for player p. As T + do and Aimulation/trainiigg budgets increase, 6 0.
G Proofs for Two-Player General-Sum
G.1 Proof of Lemma 2 (Unbiasedness and Concentration, General-Sum)
Proof. (This proof remains unchanged as it concerns the estimators, not the update rule.)
0
28

---

Published in Transactions on Machine Learning Research (02/2026)
G.2 Proof of Proposition 2 (Per-Player External Regret with OMWU)
Proof. The proof adapls the standard analysis for Oplimistic MWU to our setting with noisy, unbiased feedback for an arbitrary player p. The proof proceeds in two parts.
Part 1: Regret Against Estimated Losses. Let the estimated loss for player p be 1.77 1
04 The
(p)
OMWU algorithm uses optimistic loss estimate m (p)
11 (1(2) - ( with (/²) 0. The weight
update for player p is w
(p) w (i) (0(0) (i)
(1) exp(-n(() (p)).
(1)) (m) (i) be the potential function for player p. Following standard potential function Let W = i= Wi analysis (as detailed in Appendix C.I), the regret of OMWU with respect to the observed sequence of losses is bounded by its variation:
Σ (0 (p) HP))
min
So
knp.
In
kpp,
Σ
((p)
1P)) 1
(63)
Part 2: Translating to True Regret, We now relate this bound to the regret against the true payoffs
(p) Let the true loss be (()) = 1 - u/ cumulative true regret for player p. R(P) is:
[p] and the estimation error be A! =(, (1)) Uz
((P)
(p)
-,60) The
T
(p)
(0 -o0(")100)) (1(1))
Σ 1=1
(o (p) (.)))
min (0)
max Uzz
(1 (())) - 1 (11)) -!7
(64)
T Σ
Σ (1ab (p) (())) {p)
(() (
(p)
Regret Estimates
Cumulative Error Term
The first term is bounded as derived in Part 1. For the variation term in that bound, we use the triangle inequality:
(1??)) - 11 S (1(2)) - 11 lbo + [1(P)) - 1 + || N
(65)
[|0(²) al || || + ||!! (2) 1| || 22
(66)
Taking the expeclation over the sampling noise in our estimators, and noting that E|A 0 due to unbiasedness, we can bound the expected true regret. The error terms accumulate, leading to the final result. Dividing by gives the statement in Proposition 60
G.3 Proof of Theorem 3 (Per-Playe Oracle Regret)
Proof. (This proof remains unchanged as it concerns the bandit oracle, which is decoupled from the meta- meta-game solver by the two-time-scale assunption.)
G.4 Proof of Theorem 4 (s-CCE)
Proof. A Lime-averaged joint strategy jor is an c-CCE if for each player and any deviating strategy n', the gain from deviating is small:
(2)) 2 Σ Pir (TT ; 2 1.
pr (1)
(5' (a', x1 cp
(67)
i,j
29

---

Published in Transactions on Machine Learning Research (02/2026)
where Ep Ep. The maximum gain for player p from deviating is bounded by their average external regret against the sequence of opponent strategies (o!-") 12=1-
max E yoOff ") (1 ( (1 wo swd) vs (bllo)( (m( - (/
Ep
(68)
This is the definition of player p's average exploitability of the sequence joint strategies. We decompose this term for each player p, analogous the decomposition in the proof of Theorem 3.4. The exploitability for player p at iteration is:
Exploit
max (u.")): (o, cgf")) + max E jmo!- (r(P) (Tt' 11) m/ (E))
max s (E(p((0)
(b))))
(69)
Internal Exploitability(1)
Population Gap(r)
Averaging over T and summing over players gives the total bound on 6.
Internal Exploitability: For each player p. this their external regret within the restricted game.
From Proposition 60 its average is bounded by the sum of the OMWU Regret and the MC Estimation Error.
2. Population Gap: For each player p, this term is decomposed further into their Amortized BR Error and Oracle Regret, following the same logic in the proof of Theorem 3.4.
Summing these four distinct error sources for each player p E (1,2) provides the composite bound for as stated in Theorem
H Extension to N-Player General-Sun Games
We now generalize the framework to players. Let P {L, .., n) be the set of players, and for any player p P, let -p:= P {p} denote the set of all other players. At each iteration t, every player p maintains an anchor set z(2) a generator Go,,, and a meta-strategy o(P) E AA,(t)-1- The joint mixture over all players is Σ qep of
For each player p, the conceptual payoff is represented by a hypermatrix M with entries corresponding to the expected reward for player p given joint profile of pure strategies. The expected payoff for player p's i-th policy against the joint mixture of all other players is:
(p) (i) = S
(T(
(p) (m (2) Poe-p) = EMMP)
(P) Lit-p
II o (9) (ig)
(70)
QE-P
and player p's expected value under the full joint mixture is
r(2) = o (i)o))))
H.1 Monte Carlo Meta-Game with Importance Weighting
To avoid forming the computationally intractable payoff hypermatrices, we use importanco-woighted estimators derived from single set of shared game rollouts. We draw B joint strategy profiles j() (ib ...., (")) E,, run m episodes for each profile, and obtain returns y-(p) for every player p. The estimators for the per-policy value vector and the mixture value are:
1(1
i
(p)
Bm
(p)
b.1
b=
t=1
(71)
m
(p)) Tt
Bm
63
30

---

Published in Transactions on Machine Learning Research (02/2026)
Lemma H.1 (Unbiasedness and Concentration for Players). For rewards in [0, 1], the estimators are (i) and EEft - T for all p, i. Por any ò E (0,1), with probabilily at least 1 - ô, have concentration bounds. However, the voriance of the importaance estimalor a(p) can be high if any policy's ' probability o (i) is small.
unbiased: B[0 = (i)
H.2 Per- Per-Player Optimistic Replicator and Oracle
Each player p runs an independent learning process to update their meta-strategy and expand their policy set.
OMWU Update: Each player uses the Optimistic MWU rule update their meta-strategy based on their individual payolf estimates. The optimistic estimate for player p is mi 20 -02-1: with
((P) 0.
L of ( (1) exp (1," (2) (2) 0(2)
(72)
EB-UCB Oracle: Each player p solves separate multi-armed bandit problem to find promising new anchor 2, 21()) The value an arm is its expected reward against the joint mixture of all other
players:
E
(T)
(TGo,
(x) (p)
{ oe toe-p))
(73)
Proposition H.2 (Per-Player External Regret with OMWU in n-Player Games). For any player p. assume payoffs lie in [o, 1]. The average enternal regret for player p using OMWU against the sequence of opponents' joint strategies is bounded by:
T In kpp
max(u
(P)x(u()))) -0(
o(2) 1/201
50
1101
2 1133
T
T
1=1
(74)
T
+
E
1100
(p)
11x0
Theorem H.3 (Per-Player Instance- Sestance-Dependent Oracle Regret). Under a two-lime-saane assumplion for n-players, where opponents' mixtures {a(4) Joe-p are considered fixed during player p oracle step, the cumulative oracle regret for each player is bounded:
E (2) (z*(e)) 11 (22 (2:1))))
Σ
ln
+
T A
E
||
(2, x(F)
)12)
(75)
(p)
(m)
H.3 Overall Guarantee: e-CCE for n Players
The time-averaged joint play, defined as Jor(f,,..,hh) = T Correlated Equilibriun (e-CCE).
Ih converges to an c-Coasee-
Theorem H.4 (eCCE of Time-Average Joint Play for n Players). Assume rewards in [0, and the n-player two-time-scale assumption. With per-player OMWU meta-solvers, the time- time-averaged distribution pr is on
31

---

Published in Transactions on Machine Learning Research (02/2026)
c-CCE, where the total deviation incentive is bounded by the sum of regrets and errors across all players:
T
T
kp
|
(P)
2
(p)
11
B Bllog
11 [1
T
p=1
1=1
OMWU Regret
MC Estimation Error
(76)
T
In T
T
E
BR,
+0
T
Σ
(p)
Amortized BR Ervor
Oracle Regret
If simalalion and training budgels grow such that the error lerms for each player go to zero, then - 0 as
T + 00.
Proofs for N-Player General-Sum
1.1 Proof of Lemma (Unbiasedness and Concentration, n-Players)
Proof.
(This proof remains unchanged as concerns the estimators, not the update rule.)
1.2 Proof of Proposition (Per-Player External Regret with OMWU)
Proof. The proof is structurally identical the proof of Proposition 2 for the two-player case. We present it here explicitly for the n-player setting for completeness. The analysis is performed for arbitrary player p n1.
Part 1: Regret Against Estimated Losses. Let the estimated loss for player p be (10) o.7 The OMWU algorithm uses an optimistic loss estimate mf() ((p) + (((2) (2)), with (6)) The weight
update for player p is (p) (i) =w(() (0(2) exp((1)(P))n ((!)).
The regret of OMWU with respect to this observed sequence of losses is bounded by its variation, following standard potential function analysis:
(
T (a (m)
Σ
((p)
So
In k
(tp) - ((2) 1
(77)
The "environment" from player p's perspoctive is the sequence of payoff vectors receives. The regret bound does not depend how this sequence is generated by the other players, only its properties (i.e., its variation).
Part Translating to True Regret. We relate this bound to the regret against the true payoffs _(P) Let the true loss be 1(0) 1 (p) and the estimation error be ((") = ((P) - -1(p) of ofp) The cunulative true regret for player p. R(p) is:
RlP)
(p) max U2. (0 (p) (1)))))))))
Σ
(o(0) (p) (.,1)) min (p)
t=1
1=1
(78)
Σ ((o(p) (p) ((P) min KPp)
((0 (p) APp) (p)
Regret on Estimates
Cumulative Error Term
The first term is bounded as derived Part 1. The variation term in that bound handled with the triangle inequality: ||_! 0 $ (P) I| + ||,(()) - bl + ||AA Taking the expectation over the sampling noise and dividing by T yields the final bound stated in the proposition.
32

---

Published in Transactions on Machine Learning Research (02/2026)
1.3 Proof of Theorem 5 (Per-Player Oracle Regret)
Proof. The proof is a standard bandit analysis, applied to the independent learning problem faced by each player p. Under the two-time-scale assumption, the joint mixture of all other players, 2:(-0) Bye- is considered fixed during player p's oracle selection phase at meta-iteration t.
(4)
For player p, the learning problem is stochastic multi-armed bandit task where:
The set of arms is the candidate pool of latent codes, A!
The reward for pulling arm =E (P) E is random variable (the outcome of a game rollout) whose expectation is the true arm value, f())(2) (z), as defined in Equation 73
This is standard bandit setting. The EB-UCB algorithm guarantees that the number of times suboptimal arm is pulled, NT))), is bounded logarithmically with respect to the lotal number of oracle steps T, i.e,, [[)+(T)] - (I (In T where ! is the suboptimality gap. The total regret is the sum of the expected regrets from pulling each suboptimal arm. The Jacobian penalty adds simple additive term this regret, leading to the final bound stated in the main text.
1.4 Proof of Theorem 6 (e-CCE for Players)
Proof. The proof generalizes the argument from the two-player case. A time-averaged joint distribution pr is c-CCE if for every player p {1...., n) and for every possible unilateral deviation to pure strategy Th the incentive to deviate is bounded:
Bi-pr (P) (TT T (P))) E
Ey
(79)
where e = Ep=1 Ep. The maximum gain for player p from deviating is bounded by their average external regret over the T iterations. Let's bound this gain, cp:
6
max E. - - (1(( (Tr', ( (
Ei (p((((=))
(80)
This is precisely player p's average exploitability over the sequence of play. We decompose this term for each player p into four components, following the structure of the proof of Theorem 3.4. For each player p, their average exploitability is bounded by the sum of:
Average Internal Exploitability: The regret. of OMWU within the restricted game z(2) which is bounded by Proposition 3. This gives the OMWU Regret and the MC Estimation Error terms.
2. Average Population Gap:
The gap between true best response and the best response within 2(²)) This is further decomposed into the A Amortized BR Error (by definition) and the Oracle Regret (from Theorem 5).
Thus, for each player p, their maximum deviation incentive ep is bounded by the sum of their four error terms:
4 00
In kp.rr
1| (p)
(p) 12 12
+ Σ
E (p)
of"2) + e(p) + R(P)
(81)
T
where EBR and R(()) are the average BR error and oracle regret, respectively. The total error e is the sum of these deviation incentives, == E Ep. Summing the bounds for each player provides the composite bound
for stated in Theorem 76
33

---

Published in Transactions on Machine Learning Research (02/2026)
Part III
Ablation and Analysis of experiments
All empirical evaluations presented in this paper were conducted on computing systems equipped with Intel 19 processors, utilizing either NVIDIA RTX A2000 GPU cluster (12GB VRAM) or an NVIDIA RTX 6000 Ada. Generation cluster (48GB VRAM).
Table 2: Comprehensive Summary of Gems Experiments. This table aggregates all environments evaluated in the paper, categorized by scale and type. It confirms that GEMS scales robustly high- dimensional latent spaces (d_ = 16) and large populations across competitive, cooperative, and game-theoretic ablation settings.
Environment
Type
Action Space
Latent Dim (d.)
Iterations (T)
Policies (|Zrl)
Compler Scale
Chess (y6)
Competitive
672
8
1,000
2,002
Go (9x9)
Competitive
82
8
200
402
Hanabi
Cooperative
A 20
8
200
402
Connect
Competitive
200
402
Continuous Control (Multi-particle) Cooperative Competitive
Simple Spread (MPE)
Continuous (5)
8
100
202
Simple Tag (MPE)
Continuous (5)
8
100
202
Game- Theoreticc and Ablations
Kuhn Poker
Imperfect Info N-Player
2, 4, 8
40
82
Public Goods Game
Continuous Continuous
8, 16
10
60
Deceptive Mean
Zero-Sum
8, 16
Note:
PPlicces (Zrl)* is calculated Ial players x T.
Initialization of the anchor set. Algorithm takes the initial anchor set Zg = ( as an input. We intentionally leave the choice of this initialization the implementer: anchors can be sampled from a prior the latent space, warm-started from pretrained policies, or constructed via domain-specific heuristics. In our experiments, normal distribution, i.., NO, I), with the initial meta-strategy on set to be uniform over Zo-
simple and standard choice and initialize anchors independently from standard
Initialization of the meta-strategy. The initial meta-strategy do is defined as a probability distribution over the initial anchor set Zn. In our experiments, we initialize on to be uniform over Zo, assigning equal probability mass to each initial anchor. This choice reflects the absence of prior preference among anchors at initialization and provides a neutral starting point for subsequent meta-strategy updates via optimistic multiplicative weights. Alternative initializations, such as biased warm-started distributions, can be incorporated without modifying the core algorithm.
Policy optimization in the amortized best-response. In Phase 4 of Algorithm I agents are trained via policy-gradient methods using advantage-base objectives. Concretely, the generator parameters 0 are updated by ascending a regularized policy objective that depends on estimated advantages, allowing any standard basecy-based reinforcement learning algorithm to be used this step. In our implementation, we instantiate this update using Proximal Policy Optimization (PPO) with generalized advantage estirmation (GAE), and train agents in PettingZoo environments accordingly. We emphasize that this choice is not intrinsic to GEMS, and alternative policy-gradient or trust-region methods could be substituted without altering the overall framework.
34

---

Published in Transactions on Machine Learning Research (02/2026)
Default hyperparameeeess for Kuhn Poker. For reproducibility. Table 3 reports the exact default hyperparameters used in our Kuhn Poker runs of GEMS (Algorithm 1 Unless otherwise stated, all Kuhn experiments in this paper use these settings: an outer-loop budget of 40=40 with k-=1 initial anchor per role (implying Zz = anchors for the two-player case), latent dimonsion d,=8, and fixed Monte Carlo oracle budgets for meta-garne estimation and EB-UCB expansion. Meta-strategy updates use OMWU with 7=0.03 and a constant schedule, while Phase 4 applics ABR-TR training for KABR=30 gradient steps per outer iteration with learning rate 10 and KL regularization weight 3kI.=0.05 (Table 3).
Table 3: Default hyperparameters for GEMS (Algorithm 1 on Kuhn poker. Defaults used in the reference implementation, grouped by phases: (1) MC meta-game estimation, (2) OMWU update, (3) EB-UCB expansion, (4) ABR-TR training.
Block
Parameter
Symbol
Default
Role
Budget / population
Iterations
iterations Init. anchors
Outer-loop horizon.
Initial auchors (per role)
Initial anchor set size.
Max anchors (per role)
Cap
Rana [2rl
32
Caps population size.
anchors (2-player)
Implied
2(kg + T)
[Zrl [Zo| + 27, Zol 2ho-
Generator lotent
Latent dimension Temperature
Latent size
d2
Latent code dimension. Sampling temperature.
Temp.
1.0
Phose 1: Meta-game estimation (MC)
Opponent batch size Rollouts per matchup
Opp. batch
Nac
Opponenta from 0t-1- Rollouts per matchup.
Rollouts
MMC
Bootstrap samples EMA smonthing
Bootstraps
B
128
Bootstrap resamples.
EMA coef.
OEMA
0.0
0 disables EMA.
Phose 2: Meto-strategy update (OMWU)
OMWU step size Schedule
Step size
0.03
OMWU learning rate. const/agrt/harnnn Caps logits.
Schodule
conat
Logit cap
Logit clip
50.0
Exponsion (EB-UCB)
Oracle opponent batch size
Opp. batch
Norante
Opponents from Rollouts candidate.
Oracle rollouts per candidate
Rollouts
Moracle
2
Mutated candidates
Mut. pool
[mmall
32
Mutation candidates.
Random candidates
Rand. pool
[raadd
Extra random candidates. Latent mutation radius.
Mutation scale
Mut. stddev
0.2
UCB confidence
Confidence
8g
0.5
UCB confidence.
Phose 4: ABR-TR training Training stops
Grad steps
KABR
30
Updates per outer iter. Auchors from Z,.
Anchors per batch
Batch anchors
Learning
Optimizer Ir KL, coef.
10
Step size for generator. Trust-rogion weight.
KI. penalty weight
AKL
0.05
New-opponent fraction
Mix frac.
Snow
0.25
Mix new vs. old.
Gradient clipping
Norm clip
Global grad-norm clip.
35

---

Published in Transactions on Machine Learning Research (02/2026)
J Run on Connect-4
Additional Analysis on Connect-4 (PottingZoo). Figures 7 present supplementary diagnostics for GEMS on the Connect-4 environment implemented using PettingZoo (connect_foo _f evaluated across four seeds (0-3). Figure 7 compares memory consumption and cumulative wall-clock time over training iterations for GEMS and PSRO. While PSRO exhibits steadily increasing memory usage and superlinear growth in cumulative runtime- consequence of explicit population expansion, payoff table maintenance, and repeated best-response computation- GEMS mainlains near-constant memory throughout training and achieves substantially lower cumulative wall-clock time. This behavior is consistent with the amortized nature of GEMS, where a single generator implicitly represents and updates the evolving policy population without requiring explicit storage or enumeration of all past strategies.
Figures g and g visualize principal component projections of learned representations across sceds, providing insight into the relationship between behavioral convergence and representational structure. Figure g the PCA embeddings of distribbisti features reveal a continuous, non-linear manifold structure rather than disjoint clusters. Since Connect-4 is solved game (perfect information, zero-sum) where optimal play dictates specific minimax trajectories, the observed strong overlap between players is theoretically expected: it indicates that GEMS has converged to the shared, symmelric geomelry of optimal play. The smooth gradients visible in the embeddings suggest that the generator Go does not merely memorize single solution point, but captures the smooth topology of valid strategies loading to the Nash equilibrium.
In contrast, Figure shows that the corresponding latent-space embeddings remain more dispersed across seeds and players. This increased dispersion indicates that, while the induced action distributions converge the singular optimal bchavior required by the solved game mechanics, the underlying latent reprosentations preserve diversity. This prevents the generator from collapsing into narrow region of the latent space, ensuring it retains the expressivity to generate counter-strategies if the opponent were to deviate from optimality.
Taken together, these results illustrate a key property of GEMS: Behavioral convergence emerges at the level of action distributions (reflecting the solved nature of the game), while representational diversity retained within the latent space. This decoupling enables GEMS to scale efficiently without sacrificing expressivily, supporting stable learning dynamics and reducing the risk of premature mode collapse. Combined with the resource efficiency observed in Figure 7 these findings reinforce the advantage of amortized, generator-based mota-solvers over classical PSRO-style approaches in multi-agent zero-sum games.
GENS
PSRt
PSRL
2009
Mermory (168
3039
3009
375
326
Training
Training Iberations
Figure 7: Connect 4: Resource Usage. Memory consumption (left) and cumulative wall-clock time (right) over training iterations for GEMS and PSRO.
36

---

Published in Transactions on Machine Learning Research (02/2026)
Connect 4: Action Embeddings (Seeds 0-3)
Action-distribution space embedding
Action-distributioon space embedding
player_0 0
player 0
player_1
player 1
18
42
act entrrpy
eentrop
PCA-2
PCA-2
-02
-20
-41-
00
-02
18
-04
-31
PCA-1
PCA-1
Action-disrribution space embedding
Action-distribttion space embedding
player_0
18
player_ 0
38
player
player 1
422
entro
entropy
PCA2
PCA-2
1.
-02
PCA-1
Figure 8: Connect-4: 4: Learned Strategy Manifold. PCA visualization of slate-conditional action distributions across four random seeds (0-3). Unlike discrete population methods, the embeddings reveal a continuous, non-linear manifold structure, indicating that the generator Go captures the complex geometry of the solution space. The smooth gradients the projection correspond coherent variations in strategic aggression and confidence, confirming that GEMS maintains structured diversity and avoids mode collapse while converging to symmetric equilibrium behaviors.
37

---

Published in Transactions on Machine Learning Research (02/2026)
Connect 4: Latent Embeddings (Seeds 0-3)
Latent space embedding
Latent space embedding
player_0 player_1
player player
16
3.4
PCA-2
PCA-2
18
PCA-1
PCA-1
Latent space embedding
Latent space embedding
player_0 player
18
player_0 player
18
PCA2
PCA-2
11
1.0
Figure 9: Connect 4: Latent-Space Embeddings. PCA visualization of learned latent representations across four seeds (0-3). Compared to action space, the latent space remains more dispersed, suggesting preserved representational diversity despite similar cooperative behavior.
38

---

Published in Transactions on Machine Learning Research (02/2026)
K Run on Hanabi
Additional Analysis on Hanabi (PettingZoo). Figures 10 12 present supplementary diagnostics for GEMS on the cooperative Hanabi environment implemented using PettingZoo, evaluated across four seeds (0-3). Figure 10 reports memory consumption and cumulative wall-clock time over training iterations for GEMS and PSRO. As in the Connect-4 setting, PSRO exhibits steadily increasing memory usage and superlinear growth in cumulative runtime, reflecting the cost of explicit population expansion, policy storage, and repeated equilibrium computation. In contrast, GEMS maintains near-constant memory throughout training and achieves substantially lower cumulative wall-clock time, despite the increased complexity of Hanabi arising from partial observability, information asymmetry, and larger joint action spaces. These results highlight the scalability benefits of GEMS in cooperative multi-agent environments where classical PSRO-style methods incur significant computational overhead.
Figures [1 and 12 further analyze the learned representations through PCA visualizations action-distribbution features and latent embeddings, respectively. In contrast to perfect-informationn games like Connect-4 where symmetric self-play leads to overlapping embeddings, Figure revealss for the two players occupy distinct, complementary regions of the manifold. This separation is geometrically consistent across seeds and is expected: Hanabi is game of asymmetric information, where each player possesses private knowledge (their partner's cards) that strictly differentiates their optimal policy from that of their partner.
reveals that the action-distribution embeddings
Consequently, the plots demonstrate that GEMS has successfully learned a complementary coordination protocol. The smooth, symmetric gradients visible in the plots suggest the generator has discovered a continuous "convention" actions. This confirms that GEMS avoids the mode collapse often seen in generative cooperative learning, instead maintaining the structured diversity required to solve partially observable settings where no single static policy is optimal.
shared latent topology that maps distinct agent perspectives to compatible joint
Similarly, Figure 12 shows that the corresponding latent-space embeddings exhibit a marked contrast: unlike the separated action spaces, the latent codes for both players are densely intermingled. This indicates that the generator utilizes shared latent protocol (or codebook) to encode the game state, from which distinct, complementary roles emerge in the action space. The preservation of diversity within this shared latent region suggests that GEMS maintains rich internal representation capable of supporting asymmetric coordination without fracturing into disjoint policies.
Taken together, these results demonstrate that GEMS achieves efficient and stable learning in Hanabi by discovering structured, complementary manifolds. The ability to learn these implicit conventions without explicit communication channels underscores the suitability of amortized, generator- meta-solvers for complex cooperative multi-agent environments with partial observability.
100
CEMS
CEMS
PQRS
8000
PGR.
1000
Meroory (HN0)
Cumulative
100
310
3500
Training Berations
Training Iterations
Figure 10: Hanabi: Resource Usage. Memory consumption (left) and cumulative wall-clock time (right) over training iterations for GEMS and PSRO.
39

---

Published in Transactions on Machine Learning Research (02/2026)
Hanabi: Action Embeddings (Seeds 0-3)
Action-distribution space embedding
Action-distribution space embedding
player_0 0 player_1
player 0
player
42
24
23
act entropy
PCA-2
PCA-2
50
-41
11
-01
-10
PCA-1
PCA-1
Action-distribution space embedding
Action-distribution space embedding
player_0
player_ 0
player
player
-10
0.-
3.6
action entropy
PCA2
PCA-2
3.4
3.2
05
33
-33
Figure 11: Hanabi: Cooperative Coordination Manifold. PCA visualization of action-distribution features for both players across four random seeds. The embeddings reveal a coherent joint manifold with smooth entropy gradients. While the separation between players reflects the asymmetric information inherent Hanabi, the symmetric and complementary geometry of the clusters demonstrates that the generator Go captures unified coordination protocol (or "convention") shared by both agents, enabling zero-shot coordination despite distinct individual perspectives.
40

---

Published in Transactions on Machine Learning Research (02/2026)
Hanabi: Latent Embeddings (Seeds 0 0-3)
Latent space embedding
Latent space embedding
player_0 player_
player player
24
10
PCA-
PCA-2
-18
PCA-1
PCA-1
Latent space embedding
Latent space embedding
player_ player
player_0 player
10
PCA2
PCA-2
action
05
Figure 12: Hanabi: Latent-Space Embeddings. PCA visualization of learned latent representations across four seeds (0 (0-3). Compared to action space, the latent space remains more dispersed, suggesting preserved representational diversity despite similar cooperative behavior.
41

---

Published in Transactions on Machine Learning Research (02/2026)
L Coordination on Simple Spread
Sctup. Our second experiment is conducted in the simple_spread environment, a classic cooperative benchmark from PettingZoo. The task requires three agents to collaboratively "cover" three distinct target landmarks in a 2D space. Agents are rewarded for minimizing their distance to any landmark but are penalized for colliding with each other. successful outcome requires the agents to learn a decentralized "divide and conquer" strategy, assigning themselves to unique targets and navigating to them efficiently. This environment is designed to test. emergent cooperation and lask division. We again compare GEMS and PSRO qualitative behavior, mean return, memory, and computation time, averaged over 5 sceds.
Objective. This experiment aims to achieve two goals. First, to confirm the significant scalability benefits of GEMS, as demonstrated in the previous experiment, but now in purely cooperative setting. Second, to evaluate which algorithm is more effective at discovering the complex, coordinated strategies required for cooperative success, as measured by both quantitative rewards and qualitative analysis of agent behaviors.
L4688
2116
PSRO
PSRO
Cumulative Mean Return
Cumulative Time (c)
LO049
Memery (MB)
1000
0009
1200
1700
40e9
PSRO
1500
Training Iterations
Training Iterations
Training Pheaations
Figure 13: Simple Spread
Figure 14: Simple Sprcad run on Seed
42

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 15: Simple Spread run on Seed 1
Figure 16: Simple Spread run on Seed 2
43

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 17: Simple Sprcad run on Seed 9
Figure 18: Simple Spread run on Seed 4
GEMS again outperforms PSRO both effectiveness and efficiency. In the ghost-montage comparisons for seeds 0 4 (Figures 14 18 GEMS on top, PSRO on bottom), GEMS agents learn efficient, coordinated strategy: they quickly partition landmarks and move directly toward them, and when two agents collide they separate and re-plan rather than dithering while still staying close to the black-dot targets (agents rendered in blue). By contrast, PSRO agents exhibit weaker coordination: typically one one-Iwo agents hover near larget while another drifts away from objectives or circles indecisively, an undesirable failure mode visible in multiple seeds (e.g., Fig. [4 15p. This qualitative advantage aligns with the aggregate plot (Figure 13): GEMS achieves higher (less negative) mcan return with lower variance over time. In terms of scalability, results match prior sections: GEMS is over 6x faster in cumulative time (~2,000s vs. ~13,000s) and maintains flat memory profile, while PSRO 's resource usage scales poorly.
Results. Across
44

---

Published in Transactions on Machine Learning Research (02/2026)
reinforce central claim: GEMS is better suited for cooperative multi-agent tasks that require implicit role assignment. The EB-UCB oracle's exploration over a diverse latent space helps discover complementary roles (distinct landmark assignments) and preserves them under contact-agents momentarily repel when they touch, then settle back near their respective black-dot objectives. Simpler procedures in PSRO more often collapse to near-symmelric yet suboptimal behaviors (e.g., multiple agents chasing the same landmark while another disengages). Coupled with GEMS surrogate-free design which yields lower wall-clock time and stable memory these qualitative and quantitative results position GEMS robust, scalable framework for cooperative MARL.
Analysis. These rollouts (Figures 14 18
45

---

Published in Transactions on Machine Learning Research (02/2026)
M Coordination on Simple Tag
Simple Tag (Appendix). Across sceds 0-4 (ghost montages: GEMS on top, PSRO on bottom), GEMS reliably produces coordinaled pursuit: the red laggers sell-organize inlo a roughly triangular encirclement that narrows angles on the green runner, adjusting spacing to cut off escape lanes; the green agent, in turn, exhibits purposeful evasion (arcing and zig-zag trajectories), and captures do occur when the enclosure closes. Under PSRO, by contrast, the taggers seldom sustain stable formation: transient "trianglo-ish" shapes appear but collapse before contaimment, leaving wide gaps through which the green agent easily escopes. While the green agent under PSRO shows less consistently trained avoidance than under GEMS, the taggers' lack of persistent coordination dominates the outcome, captures rare and evasion is commouplace. Overall, these qualitative rollouts corroborate the main-paper paper claim: GEMS induces higher-level cooperative tactics (persistent enclosure and angle closing) that the PSRO baseline fails to discover or maintain.
DE CUSSER
Figure 19: Simple Tag run on Seed 0
46

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 20: Simple Tag run on Seed 1
Figure 21: Simple Tag Seed 2
47

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 22: Simple Tag run on Seed 3
Figure 23: Simple Tag run Seed 4
48

---

Published in Transactions on Machine Learning Research (02/2026)
N
Run on Chess
We evaluate GEMS on the chess_ environment from the PettingZoo library, which provides a full two-player Chess setup following the FIDE ruleset, including castling, en passant, pawn promotion, and draw conditions. Each agent alternately controls either the white or black pieces and receives observations corresponding to the complete board state, enabling direct competition through turn-based play.
Across the early stages of training, most games between GEMS agents resulted draws. However, as iterations progressed, the system began exhibiting increasingly decisive outcomes alternating between strong white and black victories. This trend can be observed in Table 4 where the balance of wins and losses evolves from neutrality to a dynamic dominance-shift pattern. The accompanying performance metrics in Figure 24 further illustrate this evolution: the per-iteration time steadily decreased from approximately 25 to 9s as the training stabilized, while memory usage remained nearly constant throughout 1000 iterations. The cunulative time curve also followed an almost linear trend, confirming consistent. computational scaling.
Figures 25 33 visualize selected games during training. Initially, black exhibits early dominance, leveraging material advantages through aggressive captures. Yct, as learning progresses, white develops comter-sttategiis involving ional control and endgame foresight. A particularly striking sequence shows white using rook- based checkmate against black, who had earlier promoted pawn to a queen high-ighghhhhhgggggggggg GEMS's robustness in resolving complex, materially imbalanced positions to decisive terminal states without succumbing to policy collapse.
Time per Iteration Step
Cumulative Time per iteration Step
Memory per Reration Step
25.0
14000
12000
10:0
Caruddle Timu 11
100000
Memooy (MB)
15.75
27.5
00.00
00000
1605
4000
1600
2000
1655
10.0
400
850
1000
202
400
500
1000
200
500
500
2000
Rorotion
Reration
Rocadon
Figure 24: System metrics across training iterations. Memory usage remains stable throughout, per-step computation time gradually decreases the model stabilizes, and cumulative time increases linearly with iteration count inte--iiiiiiiinggen consistent computational efficiency.
49

---

Published in Transactions on Machine Learning Research (02/2020)
W
MA
RAA
Ci
RAA
RAL
i
RAR
RAR
AAA
AAU
RUU
i
di
4
C
R/V
RAG
RAA
RAA
Figure 25: Chess rum for 1000 iterations. Part 1/9
Early-game states at 1000 iterations. GEMS agents demonstrate preference for central occupation (controlling d4/e4 squares) and developing minor pieces to active squares. The divergence in pawn structures indicates that the agents are exploring asymmetric board configurations rather than collapsing into identical opening lines.
50

---

Thansactions on Machine Learning Research (02 (02/2026)
Published in Transactions
1
W
W
AAA
RAA
RM
RIA
RAUR
RAL
RAA
RAR
AAA
RAR
it
Figure 26:
Chess rum for 1000 iterations. Part 2/9
Transition to middlegame. The agents exhibit coordinated piece placement, with bishops and knights occupying supported diagonals and outposts. The board states reflect a learned policy of structural preservation, avoiding premature breakage of pawn chains while maintain piece connectivity.
51

---

Published in Transactions on Machine Learning Research (02/2026)
1-
RAA
RIRA
|
Figure 27:
Chess rum for 1000 iterations. Part 3/9
Complex middlegame configurations. Play involves tension around the center and semi-open files. The agents demonstrate an ability to navigate trade-offs between material retention and spatial activity, resulting non-trivial board states that require calculation tactical exchanges.
52

---

Published in Transactions on Machine Learning Research (02/2026)
|
|
|
|
|
|
|
|
Figure 28: Choss rum for 1000 iterations. Part 4/9
Late middlegame progression. Major pieces (Rooks/ (Rooks/Queens' Queens) increasingly control key files. The policy appears to prioritize King safety while initiating simplification sequences, transitioning the game from complex tactical middlegames toward solvable endgame states.
53

---

Published in Transactions on Machine Learning Research (02/2026)
|
|
|
Figure 29:
Chess rum for 1000 itcrations. Part 5/9
Endgame transition phases. The board states show reduced material with increased King activity. The agents demonstrate the capability to coordinate remaining pieces to restrict opponent mobility, suggesting the emergence of basic endgame principles without explicit tablebase supervision.
54

---

Published in Transactions on Machine Learning Research (02/2026)
|
|
RPA
RAU
Figure 30: Choss rum for 1000 iterations. Part 6/9
Sparse board configurations. GEMS converges on simplified states characterized by Rook and King maneuvering. The agents avoid random moves that surrender material, maintaining equilibrium in drawn positions or pressing advantages in uneven splits.
55

---

Phibisbiee in Cimmsaction on Machiin Ceammngg  Reseencii
(02/2026)
RAA
|
188
Formation of mating nets. The dominant agent utilizes worddinate the edge of the board. These sequences indicatt that the policy successfully Sapeesssillyppo propagates value value from terminal states back to these pre-terminal confiinnnnneee
Figure 31: Chess run for 1000 iterations. Part 7/9
www.diateedcheee checks to drive the opponent's 's King to
back
configurations.
56

---

in Tobuiatto
conaaattionn Machhia on
(02/2020) Reseenter fasnanug
RAAA
RAS
Figure 32: Chess run for 1000 itcrations Part& 8/9
T I Pre-terminal tactical motifs. The geetts execute forcing lines involving heavy pieces to convert spatial ad advantages into checkmate opportunities. The cossisiiiin stable convergente toward valid winning strategies.
Co of these motifs across differentt seeds sunggesti suggests
57

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 33: Chess run for 1000 iterations. Part 9/9
Terminal states. GEMS reliably reaches definitive checkmate configurations (e.g., King + Rook mates). This confirms that the training framework avoids the failure mode of indefinite cycles or stalemates, successfully resolving the game loop.
58

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Training progression of GEMS over 1000 iterations in Chess. Legend: denotes a win, and "L" denotes a loss.
R denotes a draw, "W"
Iter White Black Iter
White Black]
Iter
White Black || Iter
White Black
Iter
White Black
8
9
10
13
16
17
18
19
20
21
23
24
25
26
28
L
W
29
30
W
31
32
33
34
35
41
42
43
44
45
51
52
53
54
55
58
59
60
61
63
64
65
66
67
68
69
70
74
76
77
78
79
80
W
83
84
86
87
88
W
89
90
91
92
93
94
95
96
98
99
100
101
102
103
104
105
W
111
112
113
T
W
114
115
W
L17
118
119
120
121
122
123
124
W
125
126
127
128
129
130
131
W
132
133
134
135
W
136
137
138
139
L
W
140
146
147
148
149
150
151
152
153
154
155
156
W
157
158
W
159
160
161
162
163
W
164
165
166
167
168
169
170
171
W
172
173
174
175
181
182
W
183
184
185
186
187
158
W
189
190
191
W
192
193
194
195
W
196
W
197
198
199
200
W
201
202
203
204
W
205
W
206
207
208
209
210
W
213
214
215
216
217
W
218
219
220
221
222
223
224
225
226
227
W
228
229
W
230
231
232
233
234
W
235
W
W
238
239
240
241
212
243
214
245
W
247
248
249
250
251
W
252
253
254
255
W
256
257
258
259
W
260
261
262
263
W
264
265
W
266
W
267
268
W
269
270
274
W
276
277
278
W
279
W
280
W
281
282
253
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
W
304
305
W
311
312
W
313
W
314
315
316
319
320
321
322
323
324
W
325
328
W
327
328
329
W
330
W
331
W
332
333
334
W
335
336
W
337
W
338
339 L
W
340
W
W
346
W
347
W
348
349 L.
W
350
351
353
354 W
L
355
356
357
358
359
L
W
360
361
362
363
364
W
T.
365
W
W
368
369
370
59

---

Published in Transactions on Machine Learning Research (02/2020)
Iter
White Black
Iter
White Black
Iter
White Black
Iter
White Black
||
Iter
White Black
371
T.
W
372
373
W
I.
374
375
W
376
W
L
377
378
W
L
379
380
383
T
386
L
W
387
388
389
W
L
390
391
392
393
394
395
396
397
398
399
400
W
401
W
402
403
W
404
405
W
406
W
408
W
409
L
W
410
111
W
112
113
114
W
L
415
421
122
123
124
L
W
425
W
126
427
428
429
430
431
W
432
433
434
435
436
W
437
438
T.
W
439
T.
W
440
W
W
442
443
W
444
W
L
445
116
117
W
448
W
149
W
450
453
454
455
156
W
157
W
158
W
159
W
160
461
462
463
464
W
465
466
467
468
469
470
W
471
472
473
W
474
W
475
W
178
179
W
480
481
W
482
483
484
W
485
W
488
489
490
191
492
493
W
194
L
W
495
W
496
W
497
498
W
499
W
300
W
501
W
502
503
504
505
W
506
W
507
508
W
509
L
W
510
513
L
515
516
W
517
518
519
L
W
520
521
522
523
W
521
W
L
525
526
W
527
528
W
529
530
531
532
533
534
535
536
537
538
539
540
5-11
542
5-13
L
W
5-14
545
W
W
350
551
W
552
W
553
LELL
L
W
554
555
W
W
557
558
559
360
561
W
562
563 W
564
W
565
566
567
568
T.
569
W
570
571
W 572
573
L
574
W
375
576
W ST7
578
S79
580
586
587
W
588
I.
W
589
W
590
591
W
392
593
594
395
596
W
597
598 W
L
W
599
W
600
W
601
W
602
603
604
W
605
608
609
W
610
W
611
W
612
W
613
614
615
W
621
622
W
623
W
624
625
W
628
629
W
630
631
W
632
633
634
W
635
W
636
W
637
W
638
L
W
639
L
W
640
W
6-13
611
6455
646
W
647
W
648
649
650
656
657
658
L
W
659
W
660
662
663
L
664
W
665
666
W
667
668
W
669
W
670
W
671
W
672
673
674
W
675
W
681
W
682
W
683
684
685
690
W
691
692
693
LI
W
694
LL
W
695
W
696
697
698
699
W
700
701
W
702
703
704
705
706
W
707
708
W
709
L
W
710
W
713
716
717
718
719
720
721
723
725
726
727
W
728
729
730
731
732
733
734
W
735
W
737
738
739
L
W
740
741
W
742
W
743
741
W
L
745
W
751
752
753
754
755
T.
W
757
758
759
760
60

---

978 Published in Transactions on Machine Learning Research (02/2026)
Iter
White Black
Iter
White Black
Iter
White Black
Iter
White Black
Iter
White Black
761
W
I.
762
763
W
I.
764
I.
W
765
766
L
W
767
768
W
L
769
770
773
776
777
778
L
W
779
L
W
780
W
L
T81
782
753
781
W
785
756
W
787
788
789
W
790
W
791
W
792
793
794
W
795
W
796
797
798
W
799
800
W
801
W
802
903
804
L
W
805
W
810
811
W
812
W
S13
814
S15
816
S17
S18
819
W
820
821
522
823
W
824
W
825
826
827
828
$29
830
831
832
833
834
W
835
W
836
W
837
838
W
839
W
L
840
843
W
L
845
816
W
$17
$418
8419
W
850
W
851
852
853
854
W
855
856
W
857
858
W
859
W
860
S61
862
863
864
T
W
865
W
L
868
869
W
870
871
W
ST2
W
873
W
874
W
L
875
ST7
ST8
879
W
880
881
W
882
883
W
8S4
885
887
888
W
889
W
890
891
892
893
894
W
895
89G
W
897
898
W
899
L
W
900
W
906
907
908
W
909
W
L
910
W
911
912
913
LIRI
914
915
916
W
917
918
W
919
920
921
922
923
T
W
924
W
L
925
W
926
W
927
928
W
929
W
930
931
932
933
934
L
W
935
W
941
942
943
LINIEI
W
944
L
W
945
W
9-16
947
948
W
949
W
L
950
951
W
952
W
953 W
954
L
W
955
W
956
957
958 W
T.
959
W
960
W
961
962
963 W
L
964
965
W
966
W
967
W
968 W
969
970
W
W
976
W
L.
977
I.
W
979
W
980
L.
W
981
W
982
983
984
985
986
W
987
W
988
989
W
L
990
991
W
992
993
T.
W
994
W
T.
995
W
998
W
1000
61

---

Published in Transactions on Machine Learning Research (02/2026)
O
Run on Go
Figures 34 36
present supplementary analysis of GEMS on the Go environment implemented using PettingZoo.
We evaluate training dynamics over 200 iterations (matching the run used to generate the figures below), using the same logging and evaluation protocol as our Chess analysis.
per-itcration runtime, cumulative wall-clock time, and memory usage. The per-iteration runtime exhibits bounded variability without a persistent upward drift, yielding approximately linear increase in cumulative wall-clock time. Memory usage remains essentially stable with only minor incremental changes, indicating that GEMS maintains consistent computational and memory overhead over training. This stability is particularly notable in Go, where large board states and long- horizon rollouts typically amplify the overhead of population- based game-solving pipelines.
Figure reports system-lovel metrics over training itcrations:
Figure 35 illustrates emergent spatial structure in Go rollouts after training. Shown are board states sampled from a single rollout of trained agents after 200 training iterations, with successive frames corresponding progressively later timesteps within the rollout. The evolution from sparse, locally uncoordinated placements to coherent territorial regions highlights the emergence of non-trivial spatial coordination induced by the learned policies. Importantly, this structure arises without hand-crafted Go priors, explicit territory heuristics, or domain-specific shaping, suggesting that GEMS enables agents to discover meaningful spatial organization purely through self-play and amortized policy generation.
Figure 36 analyzes the learned representations via PCA projections of two embedding spaces for both agents. In action-distribution space (left), the embeddings clearly separate into two disjoint clusters with essentially overlap, indicating that the learned policies concentrate into two qualitatively distinct behavioral modes rather than collapsing to single averaged style. Within each cluster, the action entropy varies smoothly (color gradient), suggesting structured but still stochastic action selection inside each mode.
In contrast, the fatent-code embeddings (right) remain broadly dispersed without collapsing to a small region, reflecting substantial representational diversity. This separation petraatioommmiimmmomeioo structure in behavior together with non-collapsod latent representations- particularly desirable Go, where there are many strategically distinct yet viable ways to play. Overall, the plot consistent with GEMS maintaining multiple distinct strategic hypotheses (no representational mode collapse) while preserving coherent, structured behavior.
Taken her, these results suggest that GEMS scales robustly to high-dimensional spatial games such as Go, maintaining stable system-level performance while preserving structured, diverse latent representations over training.
MyP Yo Mammand
1191
Curulattiv Time (5)
11:2
Memary (MB)
1250
159
1020
257
125
Training Iterations
Training Iteraiios
Training Iterationn
Figure 34: System-level metrics training iterations. Per-iteration runtime exhibits bounded variability without long-term drift, leading to a near-linear growth in cumulative wall-clock time. Memory usage remains stable with only minor incremental increases, indicating that GEMS maintains consistent computational and memory overhead throughout training.
62

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 35: Emergent spatial structure in Go rollouts using GEMS. Shown are board states sampled from single rollout of agents trained via the GEMS MARL framework after 1000 training iterations. Frames are uniformly sampled across the trajectory (frames 1, 9, 17, 25, 33, 41, 49, 57, 65, 73, 81, 89, 97, 105, and 113), arranged left-to-right and top-to-bottom in temporal order. The evolution from sparse placements to coherent territorial regions highlights the emergence non-trivial spatial coordination induced by the GEMS policies, despite the absence of hand-crafted Go priors.
Action-distrbution space embedding
Latent space embedding
0.06
0.04
PCA-2
-0.04
-0.06
Blakk
black
00.08
white
white
-0.05
0.00
0.10 0.15
Figure 36: Go embeddings after training: behavioral modes without latent collapse. Each point is embedding from one of the two agents (black/wiie wwhite), projected to 2D via PCA; color indicates action entropy. Left (aclion-distrihution space): embeddings form two clearly separated, non-overlapping clusters with smoothly varying entropy, indicating two distinct behavioral modes. Right (latent space): embeddings remain broadly dispersed rather than collapsing, reflecting substantial representational diversity consistent. with maintaining multiple viable strategic hypotheses.
63

---

Published in Transactions on Machine Learning Research (02/2026)
P
Ablation on Kuhn's Poker
Tables
enumerate the sweeps used for Figs. 37
38
Boldface rows indicate the configurations that
achieved the lowest exploitability within each panel. Across the grid, the strongest settings typically combined a small/zero exploration scale (1 E {0, 0.06)) with simple schedulers (const/ /sttt//arahahicc minimal autaaiionraanoon pools, and either least_ _aas or worst_ev replacement (panel dependent).
EMA toggle (ablation only). Our main Kuhn Poker runs disable Estimated Moving Average (FMA) (ema=0). For this ablation, we enabled EMA probe stability: given meta-estimates (VP,r) and coefficient 3 (0, 1), update
vr - 31-3)Vv + BVP
i p 3) Br.
(82)
Impleementatinnally, this follows deeta_ssimmst exact in our code. With EMA on (8 E {0.2, 0.5, 0.8), see the EMA column), we observed the lowest exploitability among all tested settings (boldface). systematic treatment of EMA choice (schedule, bias-correction, and scope) deferred future work.
Table 5: Configuration details corresponding Fig.
37
Sub-Agure
acheduler
EMA
Steps
LR
FKL
Mutation Pool
Random Pool
Replacement
(6)
0.08
hormouic
0.2
1c.4
0.0
least Lano
0.12
0.5
1e-4
0.0
(d)
0.04
hormouic
0.0
20
Go-4
1e-2
worst_ oV
0.06
0.8
1e-4
0.0
(8)
conett
1e-4
(h)
0.12
harmantc
0.5
10
3e-4
5m-3
worst oV
coust
1c-4
least
harmonic
1e-4
(K)
0.06
#qrt
0.2
1e-1
0.0
O
worst PV
bormonic
1c.4
comes
1c.4
(5)
0.06
conet
0.8
20
5e-4
1e-2
least 2038
barmonic
Gc-d
1e-2
harmonic
1e-2
m009
(g)
0.12
aqre
0.5
1e-4
0.0
least_maae ma84
harmonic
5ed
16-2
aqri
(t)
0.12
aqrt
0.8
10
3e-4
5e-3
leart__mas
Table 6: Configuration details corresponding to Fig. 38
Sub-Agure
scheduler
RMA
Steps
I.IL
FXL
Mutation Pool
Random Pool
Replacement
const
0.0
1e-4
0.0
(c)
0.10
comss
0.2
1c-4
0.0
woral ov
0.06
harmonic
0.0
Be-4
1e-2
(Q)
0.08
conss
0.8
lc.d
0.0
worst ov
0.06
0.2
lc.4
0.0
leaa
(8)
0.0
conss
0.0
lcdd
0.0
least m055
0.04
0.0
10
3c-4
Se-3
lcas
64

---

Published in Transactions on Machine Learning Research (02/2026)
20
10-
10-
3.0
05
0.5
64-
644
60
00-
60-
(3)
(6]
3.0
1.0
3.0
300
08-
0.5
64-
0.-
60-
60
(e)
611
180
(N)
3.0
3.0
3.0
3.0
08
6.0
0.0
6.0
10
01
3.0
3.0
3.0
3.0
044
5.4
6.0
0.0
6.0
00
(m)
[n)
(p)
3.0
3.0
3.0
3.0
0/4
6.
0.0
63
00
(2)
01
Time
Memory (ME)
Exploitsbility
Figure 37: Ablation on Kuhn's Poker part 1/2, mean of sceds. Legend: Memory, at Y-axis defines 1256.005 MB, 0 defines MB, @ Time, at Y-axis defines 0.01 S, defines 0 S, @ Exploitability, Y-axis defines 0 defines O, and O Iterations defined X-axis going from 0 to 40.
65

---

Published in Transactions on Machine Learning Research (02/2026)
104
648
64-
62
62-
0.2
660
11
(3)
(b)
10
(5)
3.0
1.0
3.0
004
64-
62
62-
6.2
13
10
(6)
180
(N)
3.0
1.0
62
0.2
000
35
20
35
Time (11
Mamory (MR)
Explotability
10
Figure 38: Ablation on Kuhn's Poker part 2/2, mean of 5 seeds. Legend: Memory, at Y-axis defines 1256.005 MB, 0 defines MB, @ Time, at Y-axis defines 0.01 S, defines 0 S, @ Exploitability, Y-axis defines 0 defines 0, and @ Iterations are defined on X-axis going from 40.
66

---

Published in Transactions on Machine Learning Research (02/2026)
Q
Ablation on Public Goods Game
This ablation studies which parts of FAST FEMS-OMWE maller most for outcome quality versus compute, so that our defaults are principled and robust rather than hand-tuned. Concretely, we vary algorithmic and stability knobs 7,, eta_ sta_sched (const/sqrt, /qrt//hrrmoic)) ema, www_gradd cap, temperature т; population-gron pool_mut, _oo__rndd, latent size zdim; and ABR-TR tightness abr--arrr-- _br_ abr abr_steps, and ßKL. We sum- marize effects with terminal and speed--snssttve metrics: welfare _arst,_s_ttt__ coop_ coop_last (final social welfare and cooperation) and their learning-curve integrals welfare auc, coop_auc, alongside efficiency measures _aggg.bb_...__aaaaee
Public Goods Game Ablation:
Purpose.
and exploration knobs oracle __n oracle __eriod,
Sime_total_see avg and ram _ea___ea______ mb_ avg. A full factorial sweep is possible, but would require substan- tially more compute than we currently have available, so, we run structured partial variations (one-factor and a few targeted pairs) across multiple seeds and report mean from the multi-seed runner; the intent is to characterize sensitivities and trade-offs, not to claim a single global optimum. In practice, the ablation clarifies (i) step-size scheduling and mild smoothing (harmonic d)t with modest EMA) stabilize meta-updales without slowing progress unduly, (ii) moderate 3KL in ABR ABR-TR improves welfare without large time/ time/RAM penaltics, and (iii) small increases in growth while preserving exploration. All runs log the exact configuration to CSV for reproducibility and allow further probing of interactions as needed.
and balanced lat_tmuttoooooo. rand control population
Public Goods Game: overview & metrics. The Public Goods Game (PGG) is an 7-player social- dilemma benchmark. Each player chooses to contribute (cooperate) or withhold (defect). Let a, E 0, denote player i's contribute docision, E; aj the total contributions, multiplier r > 0, and per-contribution cost. Q (homogeneous case uses C c). The one-round payoff to player i is
Ui -s - C2 aai and the social welfare
W
Σ 1=1
m = = - Σ
Ga 2=1
(83)
When & <<<<> (homogeneous costs), each individual gains by defecting (Tree-riding) while the group gains by coordinating on cooperation this the core tension PGG exposes.
In our runs, the generator outputs the probability of cooperation; we track both outcome quality and learning speed/ slabilily via four summary metrics:
welfare last: the mean social welfare W at the final training iteration (averaged over profile samples and seeds). Higher is better; if parameters make W negative, "less negative" still indicates improvement.
coop last: the final average cooperation rate + Pr[a;=1].
welfare_ auc: the (trapezoidal) area under the welfare- verrss-ttoss curve. This rewards methods that reach good welfare earlier and keep it stable-a joint measure of speed and robustness.
coop auc: the analogous AUC for cooperation rate.
All four are reported as ttd across seeds by the multi-seed runner, making it straightforward to compare algorithms and syper-paramete settings both efficiency and cooperative outcomes. iters in child = 10: seeds in child 0,1,2,3,4; iters 10; players 5.
67

---

Published in Transactions on Machine Learning Research (02/2026)
Table 7: Ablation (Public Goods Game)
time. total _ec ong
pook _nb.
onacle porlod
welfare Lav
welfare
coup. Joae
comp, auc
c0ort beta k1
delta
eta nched
log nob
muv grad
oracle -o
poot mus
slow
top rond
pool
slow
two
ucb.
D. 6.0053 2031542633 42635 0.000N99117
D. 0088591154 1 1. 08533225111 0. 0-388817455 T.
6.007937888
0.006! 77473 0,050917745
0.031302772
08066862877 0.063860629
3.000755760 O. 899938799
0.
4. 7377797713 O. 1006622 10021
D. 683
4 0
997799778 O. 089779975 1.252421230
T 22000000007
1. 122200000
T. 12200000
0 1240 8125 0.00025
1241. 642186 0.0005
1240. 736719 0.0005
1340 909375
1240. 864688
0.0005
0..000255
20 0.02 1 0.0005 0.0 0.05
0.000 1 0.0000 0.0 0.00
02 0.0005
01 0.0005 0.0
10.0005
0.0 0.00
0.08
comet
ponett
const
0
0
3
0.05
0.1
O
0.03121 22933 12063121904
007981112 D. 00007500
02 a. 0.082130220 0.022120277
0.DOR 77445- 3-0-000-00577 7745
G21356287
0.000517745
D. 963860620
O. 001881612
0.060934263
D. 430287730
D. 977777077
3. 0637102110.0
4. 428422511
S08817474
0. 442342253
O.
O. 1.33600
D. 4318981743 1743
0.366047713
0.08992878
1. 2677620666
1
1. 1256717255
1222211551
1240. 986719 0. 0.6006
1240 683594 0 00025
1240. T33594 0.0005
1240. 714944
1240. 785156 0 00g25
12:10 8625 0.5055
0. 00005022552
20 0.00000000 02 1 0.0005 0.8 0.04 harmonto
20
20
20 0.01 1 0.0005 0.5 0.08 bartaroaee 0
20 0.01 1 0.0005 0.8 0.08
20 02 1 0. 0.0005 0.5 0.04
0.01 1 000005 0.0-0.08
0.02 1 0-0005 0.0 0.0-0.05
harmonse
const
const 0
0 3
0
0
0
0 0.1
3
3 0.05
3 0.05
3 0.1
2
2
2
2
2
2
1
D
D
T
1
1 O
1 5
16
S
1B
155576736
U. 118587876
1240. 905594 0.0005
0.01 1 0.0005 0.0 0.08
const
3
O
18
0.088910244 O. 0.00k0910S 4. 265299203
1-42652902 1-252139187
1240. 96 1565 0002502255
02 1 0.0005 0.0 0.04
conet
3
1
O
10
88088442288 0.000934263 O. 683710212 O. 0..08837100 1021 1 2777671846
1240. 687969 0.00n5
0.01 1 0.0005 0.8 0.08
harmonic
0.1
2
1
O.
10
D-0S1 0.031122
77077 3--30-47773
1240
80025
0.0005
0 0.01
1
comett
2
06G384218
008538421
4. 706677842
0.47065785
266106569
1240. 602188 00000025522
02 0.0005
0.04
O
006458697 00054587
874286636
O. 087425684
226150048
1240. 07 094 0.0005
1 0.0005
0.08
3 0.1
D. 0222797625 O. 0622799621
D.012: 79432 0-061217885 711121443
093790585 D. 00097965
0457896690 064575867
3. 5587731171
D. 7384311/3
1 2226629336
4. 0556600959
1 038748019
548711996
5286551514 O. 452865186
0. 0.282222834
O. 1-3544871199
0.
O. 4.406660955
O. 183874801
1. 1.1258-31001
1. 1.3180115558
1. 5.318738081
372322064
1.50113511
4554
454
1240 $08594 00025
1240. 702344 0.0005
1240. 759375 00025
1240. 834885 D 80025
1240 728805 0.0055
1240. $15315 0.0085
20
20 0.01 1 0.00ps 0.0 0.08
20
20 0.02
20 0.00
0.01
0.01
02 0.0005
1 0.0005 0.0 0.08
1 0.0005 D.8 0.05
1 0.0005 D.8 0.08
1 0.0005 0.0 0.04
barmonto
harmonse
harmowic
conet 0
cazat
posset
0.0
0
0 3 0.05
0003
3 0.1
5
2
Z
Z
Z I
2
D
O
O
1 5
1.
1
1 O
O-O
18
10
10
16
085384213 O. 068588421
4. 706878492
0.47065785
1 844771710
1240. 771094 0. 00025
20
02 1 0 0.0005 0.0.0
003st
0
3
2
2
2
O
1
5
S
556458 O. 00054587
D. 8732866366
O. 087498684
1 326258
1240 678125 0.0005
20 0:01 1 0 0.0005 0 0.08
harmonie
0
3 0.1
2
2
T
S
0064780689 0.064578967
4. 528651814
O. 45248658881
1. 2.296406222
1240. 711719
0..0002552
02 1 0.0005 0.8 0.04
harmonic
0.05
2 2
L O
&
022797628 0.0622796221
036746019 O. 0.188674801
1. 1.854440002
1240. $40825
0.6005
0.01 1 0.0005 0.0 0.08
09223
3
2 2
O.
1G 1G
0696996655 D. 00099905
058669959 O. 40266099G
1. 856069088
1240. 967519 0000002522
02 1 0. 0.0005 0.0 0.04
2
2
O
L
0.047578335
79432 0.00 2001211843 750431142
0.35488711
0175915114
.2878005988
0.0005
90025
1 0.0005 0.6
0.0005
barmense
barmonse
2
2 2
1
capet
D. 041826296 O. DD41136699 4
0.0488 D08877711 O. 9.0644567761 S. F11358419 0.57
0.1 11 2 0.01 1177225 5. 1573-1515 O. 5555734149
0.0997 BBUB a. 0.36097 BOOS D. 649623338 O. 085496235T
O. 0.04149977 O. 0.06G41 4837 4. 11332201 O.
1.0144 017441394 0.0G1 744189 O. 9361 3226332238222 O. 0888618337
44 47534 2030120T8 2.003301200 1205
41TOTE98S O. 441707
401584202
J. 1.291794720
1. 333359103
1 3448556701
1. 299304295 1240. 527344 0.0066
1. 278528048 1240. 721994 0. 00025
2992001366
340657478
1240. 548435 O 00923
1240. $05469 C. 0.6065
1240. 714063 00025
1240. 902341
1240. 564063
00025
20 0.0005
20
20
20
20 02 1 0.0005 0.5 0.04
20
0.01 1 0.0005 0.8 0.08
0.02 1 0.0005 0.5 0.04
01 1 0.0005 0.0 0.08
02 1 0.0005 0.0 0.04
02 0.0005 0.0 0.03
0.0005
0.08
harmonie
hurmonic
harmonte
harmonse
const 0
cozat 0 0003
0
0
0
D 0 0.05
3
3 0.3
3 0,1
3 0.05
Z
2
I O
T 1
T O
T O
1 O
L
1B
1G
18
1G
S
0-88870787 O. 002837078 328133807 O. 3328133381
3653366177 77 1240 680625
20 0000
01 1 000005
0.008
const 0
3
2
O
D.1 1127725 G.01 3157725 S. 1573-1515 O. 51511788114
1. 17
1240. 767188 00025
02 1 0.0005
0.04
conet
3
O
D.004473564 0B0347954 D. N030120TH O. 0O0000
208
22200382388
1240. 726563
0.0005
0.01 1 0.0005 0.8 0.08
harmonte
3 0.1
1
041 825393 0.00411118888 CC4482669444
417076996
O. 4417777G
1. 2222298321
1240 81875 00025
02 1 0. 0.0005 0.8 0.04
barmonie
0.0G
2
1
1
0.01 394 744139
093613327
1. 318811531
1240 320313 0.0005
1 0. 0.0005
0.0 0.08
const
3
O
1G
SDST 0.0064144
018342055
000252255
0.0005
popett
D. 0776011 4925 0-367501
D. 0200R7A07 0. 1-002269741
D.1 16555 O 0.0123044155
0.043677618 0664667661 B. 7-11383415
004512455
0.009716
00086831110 O. 06649691122
030784413 O. 063307 1S1 50T336221
104556423-00 O. 060451642
400
5 4 4867793394
1 131 707378 O. 113170738
O. 8.773337118
4. 4.207 2207443076 O. 0.420744300
64D621336
O. 0.4486775702
O. 088766672211
0. 1.5348766331
O. 50033662
O. S.SS1185343
0604662337
4486679930
1
1 2925 97726 1240. 687656 0.6065
1 325511721
1
1.8751434066
1. 1.2291 81865
1.34766677 1240. 714063
266421800
1240. 784375 00025
1240. 545315 0. 000050022222
1240. 774219 0 00025
1240. 753125 0.0005
1240. 527944
1240 902314 0-0055
0.000025522
20
20 0.00000000 02 1 0.0005 0.5 0.04
20 0.02 1 0.0005 0.0 0.04
30
20
20 0.0
0.01 1 0-0005 0.0 0.08
0.01 1 0.0005 0.5 0.08
02 0.0005 0.0 0.05
02 0.0005
1 0.0005
0.0 0.08
burmonic
barmoane barmonio const 0 0.0
hurmonie
comett
conat
0.0
0 0 0.1
0 0 0.05
0
0
3
2
O
O
1.
O.
L.6
O
1
16
10
10
O 0.08896815 0.060825815
6.5566504 0. 0688846804
1. 1.879670661
1240. 762344 0.0006
20 0.01 1 0.0005 0.5 0.08
harmaciic 0 3 0.1
2
1
16 16
0.03803511 11 0.005003511
3. 841003360 O. 3841666667 1-2996-10115
56 1240. T76563 0. 00025
02 1 0.0005 0.8 0.05
harmente
0 3 0.05
0.0307845113 4,063078431
507339224 O. 0.1807333633
1. 1321631813
1240. 781 4D5 0.0005
0.01 1 0.0005 0.0 0.08
comet
3
2
O
0.128046.h6 he
5.012304415
b. 5.44876127 0.5 0.544876151
1. 1.355440922
1240. 84375 000002255
02 1 0.0005 0.0 0.04
canet
3
2
2
O
10040154933
000451142
875367918 O. 087536792
129258728
1240 628806 0.0005
1 0. 0.0005 0.08
barmonic
0.1
2
0-4
0 0.04
harmonte
0.020607407 0.0023699411
111170T3T5 0.115170738
..34500088
1240. 132813 0.0085
1 0.0005
comst
L.
D 0.01522221 1
D 1 0.060152225
0.076014922 402501442522000 0.007601403 4 20T443076 O. 44320
0.05801511 11 0.065005h11
0.00E96815 0.000985515
211923200 1 0452505330
O. 1172009330
S. 8410516648 O.
0.65546504
O. 0.068866804
0.017240222
0.3841053367
2 2.857672119 1240. T86719 0.0005
855008870
1,28168634
299151307
1240. 8T8125 00022 20 0.00 02 1 0.0005 0.8 0.04
1240. 936719 0002002552
1240 70625 0.0065
40 0.01 1 0.0005 0.0 0.08
02 0.0005 0.04
1 0.0005 0.5 0.08
harmonse
comet 0
0
3
3 0.05
0.1
O
O
16
1G
18
O. 01060a5os D. 0 O. 400124063
D COOP2RSSE 2.45a-03
011222200 O. 80102521
O. 081531858
1. 2015043366
O. 24001
O. 00881533185
O. 3303501322
O. 1845250877
2 834643824
D. 3.33 03963
3 1122588718
2.37458836
1240.
1240 733156 D 80023
12400-773
1240.85
785166
D. 80025
40 0-0055
0.0006
40 0.01 1 0.0005 0.0 0.08008
40 0.00000000 02 1 0.0005 0.0 0.04
10 0. 0-02 1 0.0005 0.8 0.05 harmonte
0.01
1 0.0005 0.8 0.08
harmonic
const
const 0. 3
0
0 3 0.05
0
0
3 0.1
2
2
1
O
O
T
1
1
O
5
1G
0.0001300 1027 O. D0C389180S
J. 716800273
O. 171689026
2 A35034132
D2
1240
686719
0. 00025002522
40 02
1 0. 0.0005 0.04
coast 0
3
2
I
O
1G
0.00169102 0010 0.000100102 D. 103430038 O. 010545059
2. 351188858
1240. 553908 0.0005
0.01 1 0.0005
0.8 0.08
harmonse
0.1
2
18
0.01200210 102000000 0.001 200218 L. 108425272
O. 1105442224
2. 9117781772
1240. 192969 00000005550
02 1 0.0005 0.8 0.04
harmonse
0.08
2
L
10
1 0.000152225 O. 0.172400
339 O. 017240084
2 A654422058
1240. 771875 0.c0n6
0.01 1 0.0005 0.0 0.08
const
3
2 1
O
OO
0.00 #23344
443250889 O. 94525067
3322580787
80025
1 0.0005
0 0.01
canst
D00255363 2.45e-05
08-1331888
00815338S
2.35093627
1240
0.0005
1 0.0005
0.8 0.08
harmonse
harmosse
01 010508508
001000851
400124063 O 0.04002200 2406 2 8.336776888
1240. 530469 0. 0.0005
01 1 0.0005
0.0 0.08
0
16
000913000 1007 003391905
216896273 O. 171655925
2.3455616
1240. 785835 00025
02 0.0005 0.000
5
O.
D. 086622222 O. 2206652222
O. 0.00100102 L02 O. 0.001691022
O. 200100010
O. 2 O. 0.0816L12182 2 3551
O. 348
1 105-22227 O. 0.1054555922
O. 1105-42228 2 323055166 06
2 3.305444222 4236
1240. 330544 0. 00025
1240. 198438 0.0065
40 0.02
40 0.01 1 0.0005 0.8 0.08
1 0.0005 0.8 0.04
hurmonie
harmonie D 0.3 0.1
D 0.33300 0.05
1.6
16
10
D D.Dons 13542
0.0625 15B33
b-14
D. -D01702681 O. 0.OBUITUM TUUI1UZ26822
1. 12222-2222 O. 172224327
2. 312303310
2.37188077
380112074 32074
1240. 755125 0.6088 30 0.01
1240 905825 D. 80025
1241 5876 0.6065
40 0.01
40 0.02
1 0.0005 0.8 0.08
1 0.0005 0.0 0.08
1 0.00ps D.D 0.04
harmonse
const
canat 0
0 3 0.1
0.0 3
Z
2
2 O
O
1
005007222 0. 00000066722
1. 2776344577 O. 12776L144
2 342204618
1240 569219 0. 00025 40 02
1 0. 0.0805 0.5 0.04 barmonie
0 0 3 0.00
C
2
2
2
1
1 5
0.01991000 10067 O. 2917
D. 357 O. 008786017
7 9. 3124755471
1940-727344
0.0005
40 0.01 1 0.0005 0.0 0.08
0.0.0.8
const 0.
3
2
2
D
1
16
D. OSS1 89402- O. 3031 18846 1. 558488052 O. 155945906 D. 366680098
12:00 7125 0. 00025
02 1 0.0005 0.0 0.04
comset
3
2 2
O
16
002724900 O. 0.060272401 O. 111010116 O. 00117010112 2. 305494022
971875 0.0005
0.01 1 0.0005
0.8 0.08
hurmociic
0.1
2
2 1
10
019667682 0.061966768 234394093 O. 0.12348947 2 266290177
1240. 741466
00025
0. 02 1 0.0n05 0.5 0.04
hurmonie 0 0 0.05
2
2 2 1
L.
1G
0.006822226 0.00oes2225
3111112111 0.031511211 1218
1240.
0.0005
0.01 1 0.0005
0.08
conet
2
D.028418633
O. 1.17222432722
0.0005
0.04
conett
2 O
230513582
6.146-05
543044
0.0085
1
D08G87232
000969722
276544507 O. 1376654848
9 1.009166777
1240.
950781
00025
02
0.0005
0.6 0.00
harmente
0.01 21191166 19666667000
O. 0.001 291667 3879950172
13 O. 0008796617
2.33807653
1241.0
0.0005
1 1.00005
0.0 0.08
0
1
2 2 1.0
16
68

---

Published in Transactions on Machine Learning Research (02/2026)
sime. total DOC. avp
oracle po lod
welfare 2.00
coup Just
coup Auc
etn rchhed
muzn goad
poo! mot
pool Pxbd
detsao
abe
cowt
mull
alow
tal
top
adimm
0.0911
0.003118946 1, 1,630489052 0. 160048900 2. 8888877028 1240.
890004
0000555
40 0.02 1 0.0005
0.0 0.04
coast
0
16
0002734900 0. 0.000272241 0. 1117010106
0 2 2.299126663
1240.
508435
0.0006
0:01
1 1000005
0.8 0.08
0
0.1
2
0.0296876622 0.061965763 -354124693 0-125429477 2-294610247
71875
0.80025
0.02
0.0005
0.8 0.09
harmosse
0
0.007332874
0-060T33257
02282202187 0.028536217 2. 2.369500180 s9 1241. 516408
0.00n5
0.01 0.0005 0.0 0.08
comatt
0-0389/474
0-0634P1474
0-000633220
5-33e-05
0.04528769
L.00452877
222542948
10222642042
2.831411142
1240.
0.0085
0.0005
L O.
0.031591400 403 00315914
1.0160072G
0.1G1600727
1240. 904685
00025
02 0.0005
D 0 O. 0.060133804
O. 0842446658 O. 0.0084248682
2 4.702655479
12400 T1875 0.0005
0.01 1 0.0005 0.8 0.08
harmonse
0 0.1
18
D.0 6.016348574 O. 0.061534857
1.02745011 O, 0.10P274502
2 4.226659441
1240. 7T4219 00025 40 0.02 1 0.0005 0.8 0.04
harmenic
0 0.05
1 1
10
D. 4 D. 0.300T35257 O. 286302107 O. 0.028680217 2 2.897258656
1241.90013 980168 0.6065
0.01 1 0.0005 0.0 0.08
copat
0 0
O 1. 1
O 1
I. A39211315 0-063894174
O. 0.1662213
2 2.3821 92755
1241 821219 80025
10 0.02 1 0.00PS 0-05
const
0 0
O
D. DOUSSSSUS
9.335-05
O. DISTOTOS1
O. 008T077911 2 2,335633115
1240 789219 0.5055
40 0.01 1 0.000s D.8 0.08
harmense
0
a 0.1
2
1
1.0
D0SC62821
D. 000966282
1 201094464
O. 120109444 2.31300728
1241. 507913 00025
02 1 0. 0.0005 0.5 0.04
barticane 0
0 0.05
2
1
004623769 O. 000482377
O. 2225429U3 O 022264294 2 3663336617
1240 717185
0005
0:01 1 0 0.0005 0.0.0.05
coast 0 0
O
16
CRT 403
00315914
1536007266 O. 15 100727
9 3711551348
1240 785835
00025
02 1000--00000000000 000005
0-08
const
0
O
1 0.000133804
0.084246678 O. 0084249682
2 333399464
1240. 533594
0.0006
0.01
0.0005 0.8 0.08
harmacii
O. 4 0.06 684987 0.02745011 O. 11777740222
2 265339251
028966
00025
1 0.0005
0.04
harmonic
10
D.D4TUSAD84
20885535 000925555
28/1501911
O. 5012
2 250206776
1240 552344
00025
0 0.0005
harmonto
5
00R221 407 010822141
3664433677
O. 030663386
9 3666-66514
1240 738281
0000
0.0005
0.08
C282226866 D. 062922664
1.6304080
0.16384086
2 4333306440
1240. 703366
00025
0.
1 0.0005
0.0 0.04
1
O
002771278 O. 060277127 0. 0.116466946 0.011646090
2 298221258
1240 6876
0.6065
0.0
1 0.0005
0.8 0.08
burimeaie
0.1
1
O. 0.012767998 0. 061276799 1.113148996
0.1318149
2.363634220
1240.662G
00020022
0. 02 1 0.0005 0.8 0.04
bonmonae
0.00
2
1.D 10
69

---

Published in Transactions on Machine Learning Research (02/2026)
R Ablation on Deceptive Mean
We conducted an ablation study for the Deceptive Message game to analyze the sensitivity of GEMS to its core hyperparameters. Our investigation focused on key parameters governing the Amortized Best-Response (A (ABR) training, including the learning rate (abr _abr_ir), update steps sabr_steps), and KL-divergence coefficient (beta_kl) as well as the generator's latent dimension (zdim) and the Jacobian regularization penalty Hambda_jac)..
Table 8: Ablation (Deceptive Mean)
ram peak mu. nn
receiver last
meta J
sender Instt:
sender auc
Lime lutal
lambda jac
stops
adv ema
receiver
beta kd
abr le
inner
xdim
0. 0.0048828133 0.730976563
0.213747987
3.637925569
3.797561526
766.8417969
0.001
100.0
0.05
0.01
2.0
0.0
32.0
8.0
0.000976563
0.776320313
0.167490632
3.677008441
3.490059376
766.799
0,001
100.0
0.05
0.01
0.0
32.0
16.0
0. 0.03000225
0. 0.752929688
0.208329613
3.636479725
3.519 120212
766.908203
0.001
100.0
0.1
0.01
2.0
0.0
32.0
8.0
0.00 0.000s000
0. 0,001953125
0.7731375
0.151023457
3.691782171
3.178062987
767.00 767.0566106
0,001
100.0
0.1
16.0
0.00-1882813
0.750976563
0.213747387
3.637925569
3.465744257
766. 866.8632813
0.001
100.0
0.05
0.01
4.0
0.0
32.0
8.0
0.000976563
0.778320313
0.167490632
3.677008441
3.471985459
767.0722656
0,001
100.0
0.05
0.01
32.0
16.0
0. 0.00300625
0.752929688
0.208329613
3.636479725
3.5193591 122 766.95700 13
0.001
100.0
0.1
0.01
4.0
0.0
32.0
8.0
0.001953125
0.7734375
0.151025457
3.691782171
3.539175749
766.9375
0.001
100.0
0.1
0.01
4.0
0.0
32.0
16.0
0.005859375
0.76171875
0.187596763
3.494975376
4.1 16945505
766.8417968
0,001
100.0
0.05
0.0
64.0 8.0
0. 1.058553755
0. 0.766601563
074001
3.5 3.545335929
4.1 1100067773
766.9746094
0.001
100.0
0.05
0.01
2.0
0.0
64.0
16.0
0. 0.00-1822113
0.765625
0.1769021
3.508331751
4.123156428
8632813
0,001
100.0
0.1
0,0078125
0.762695313
0.192496811
3.535875717
4. 4.120264649
896.7897788
0.001
100.0
0.1 0.01
2.0
0.0
64.0 16.0
0.005850375
0.76171875
0.187596763
3.494975376
4.084610581
766.9765625
0.001
100.0
0.05
4.0
0.0
64.0
0. 0.058559755
0.766601565
0.200 174231
3. 3.545335929
4,109117566
9466949218
0.001
100.0
0.05
0.01
4.0
0.0
64.0
16.0
O. 1.004882813
0.765625
0.176902149 49 3.508334751
4.1 298499777 766.8691408 406
0.001
100.0
0.1 0.01
4.0
0.0
64.0 8.0
0.0078125
0. 0.762695313
0.192496811 1 3.535876717
4.13603615 158 766.8856719 1g
0.001
100.0
0.1
0.0
64.0 16.0
O. 0.030273138
0.689153125
0. 0.3989611677
3.1106754
18883199993
766 939-1511
0.0005
100.0
0.05
0.01
2.0
0.0
32.0
8.0
0,02734375
0.712890625
0.412892042
3.030212674
3.482968065
766.8671875
0.0005
100.0
0.05
32.0
0.03125
0.6875
0.397256236
3. 3.113605088
3.475835323
766.8925781
0.0005
100.0
0.1 0.01
2.0
0.0
32.0 8.0
0. 1.
0.71875
3.048 0.391349073
3.475960 135 767.0703125
0.0005
100.0
0.1
2.0
0.0
32.0
16.0
0. 0.0302734338
0.689453125
0.398964667
3.1106754
3. 3.194435072
766 8066406
0.0005
100.0
0.05
0.01
4.0
0.0
32.0
8.0
0.02734375
0.71
0.412892042
3.03021 2674 3.538249969
767.0585938
0.0005
100.0
0.05 0.01
4.0
0.0
32.0 16.0
0.03125
0.6875
0.397256236
5.20233935 1.366008
766 736.8378906 0.0005
100.0
0.1
0.0
32.0 8.0
0. 0082222013
0.71875
0.391319073
3. 3.018147322
3. 185667413
766 8632813
0.0005
100.0
0.1 0.01
1.0
0.0
32.0 16.0
0,03125
0.703125
0.41984623
3.008728387
4.0912076233
766.8574219
0.0005
100.0
0.05
2.0
0.0
0. 0.029296875
0.70703125
0.433813333
3.06261 4619
4.1 14554763
767.021484
0.0005
100.0
0.05
0.01
2.0
0.0
64.0 16.0
0.03125
0. 0.705078125 125
0.4197 1549
3.019714162
4. 17322803
766.9511719
0.0005
100.0 0.1 0.01
2.0
0.0
64.0 8.0
0. 0.032226633
0. 0.111001663
O. 0.1108-10818
3. 3.033551022
1. 15797449
766.92 92382828811 0.0005
100.0
0.1 0.01
2.0
0.0
61.0
16.0
0.031 25
0.703125
0.41984623
3. 3.008728387
4. 1.091 78257
766.828122
0.0005
100.0
0.05
0.01
4.0
0.0
64.0 8.0
0. 0.029296875
0.70703125
0.433813333
9.06261-6619
A.101922652
766 8183591
0.0005
100.0 0.05
16.0
0.03125
0.705078125
3. 3.019714162 0.41971549
4.11270225
766. 166.8984375
0.0005
100.0
0.1 0.01
4.0 0.0
64.0 8.0
0.032226563
0.704101563
0.440840818
3.053851022
4.102519035
766.8339844
0.0005
100.0
0.1
0.0
0.002920688
0.783203122 125
0.08143827
3. 3.804508408
4.592607737
766 766.9003906
0.001
200.0
0.05
0.01
2.0
0.0
32.0 8.0
0. 0.008888813
O. 0.779966875
0.1031684477
3. 8.676696227
4.610334158
766. 8359375
0.001
200.0
0.05
0.01
2.0
0.0
32.0 16.0
0. 0.008835988
0.78125
0.078916951
3. 3.813918651
4.561136138
766.83007 8300781
0,001
200.0
0.1 0.01
2.0
0.0
32.0 8.0
0. 0.004862813 13
0.778320315 13 0.094999513
3. 6803666 159 5988444355
766.796875
0
200.0
0.1 0.01
2.0
0.0
32.0 16.0
0. 2.002929688
7.77233122
0,081438871
8.808808-00
1.561905011
166887
0,001
200.0
0.05
32.0
0. 1.00-882883
0.779296875
0.1031684447
3.6 1.670696177
4.548526525
766.9376
0.001
110012200
200.0
0.05
0.01
4.0
0.0
32.0 16.0
0.006835038
0.78125
0.078946951
3.81391 8651 4.592769027
767.0585038
0.001
200.0
0.1
0.0
32.0
0.004862813
0. 0.778320313 13 0.094999513
3. 3.680365 159
4.618440155 151
766.92 1875
0.001 1..01100000
200.0
0.1 0.01
4.0
0.0
32.0 16.0
0.0
0,805661063
0.02203125
3. 879985872
202621220
767. 0332031
0.001
200.0
0.05
0.01
2.0
0.0
64.0 8.0
0.002929688
0.791992188
0.075109614
3. 3.967348489
1877666818
766. 7890625
0,001
200.0
0.05
2.0
0.0
64.0 16.0
0.0
0.8046875
0.020729167
3. 8.888508077
5. 56398416
766.8964375
0.001
200.0
0.1 0.01
2.0
0.0
64.0 8.0
0.002929688
0. 0.791992188
0.079345725
9.900021037
8.176624066
9003906
0,001
200.0
0.1
0.0
0.805664063
0.02203125
3. 3.879985872
5. 1.18808338
766 9003906
0.001
200.0
0.05
0.01
4.0
0.0
64.0 8.0
0.002929688
0.791 0221992180 188
0.075109614
3.907348480
5. 95022225
760.8632813
0.001
200.0
0.05
0.0
64.0
16.0
0.0
0,8046875
0.020729167
3. 888508007
5, 2010624393
767. 0670041
0.001
200.0
0.1 0.01
4.0
0.0
64.0 8.0
0. 8.002999688
0. 0.791992188
0.0793415225
3. 3.900021077
5.167958856
766. 81375
0.001
200.0
0.1
0.01
1.0
0.0
64.0
16.0
0.013671875
0.759765625
0.233534138
3.491052739
4.574347615
766.8769531
0.0005
200.0
0.05
0.01
0.0
32.0 8.0
0.008789063
0.760742188
0.237348755
3.381854229
4.594873428
767.046875
0.0005
200.0
0.05
0.01
2.0
0.0
32.0
16.0
0.015625
0.759765625
0.2345107
3.502512135
4.615203023
766.9911719
0.0005
200.0
0.1
2.0
0.0
32.0
0.01171875
0.755859375
0.240472116
3.358276201
4.573800206
766. 766.9645438
0.0005
200.0
0.1
0.01
2.0
0.0
32.0
16.0
0.013671875
0.759765625
0.233534138 38 3.491052739
4.676252937
760.8632813
0.0005
200.0
0.05
0.01
4.0
0.0
32.0
8.0
0.008789063
0. 0.760712188
0.237318755
3. 3,381851129
1.60927546
767. 0977099701 0.0005
200.0
0.05
0.01
1.0
0.0
32.0
16.0
0.015625
0.759765625
0.2345107
3.502512135
4.562783718
766.919923
0.0005
200.0
0.1
0.01
4.0
0.0
32.0
8.0
0.01171875
0. 0.755859375
0.240472116
3.358276201
4.630032897
886.8847656
0.0005
200.0
0.1
0.01
4.0
0.0
32.0
16.0
0.00390625
0.797851563 1563
0. 0.135991033
3.614750257
5. 1.190222199
766.861322 1328
0.0005
200.0
0.05
0.01
2.0
0.0
64.0
8.0
70

---

Published in Transactions on Machine Learning Research (02/2026)
ram peak, Lub .u
Line total sen
sender last
rocalver Jast
sender auc
lambba jac
aby. steps
adv em21
resdiver
abr Ir
beta ki
innor
meta.
adim
0.009229888
0.7890625
0.241206324
3.576210672
5.176317168
706.86 86686652344
0.0005
200.0
0.05
0.01
2.0
0.0
64.0
16.0
0. 003390625
0.796875
0.125327201
3.619901337
5.203107407
766. 186.9238281
0.0005
200.0
0.1
0.01
2.0
0.0
61.0
8.0
0.002929688
0.787109375
0.252284713
3.56288221
5.175059795
766.9140625
0.0005
200.0
0.1
2.0
0.0
0. 0.00300625
0.797851563
01123591093
3.614750257
5.206146717
766.9824219
0.0005
200.0
0.05
0.01
4.0
0.0
64.0
8.0
0. 000029088
0.7890625
0.241205324
3.576210672
5.198497653
760.8398438
0.0005
200.0
0.05
0.01
4.0
0.0
64.0
16.0
O. 0.00390625
0.796875
0.125327201
3.619901337
5.191039161
766 8378906 0.0005
200.0
0.1 0.01
1.0
0.0
61.0
8.0
O. 2.00920688
109375
0.252284713 13 3.56288221
5.200975243
766.8867188
0.0005
200.0
0.1 0.01
4.0
0.0
64.0
16.0
0. 1.001882813
7.75976663
0.209971877
3.6115173301
3-18:1597415
766.85 8374199
0.001
100.0
0.05
32.0
O. 0.001953125
0.7734375
0.171805641
3. 3.671931845
3.451884599
766.98 9511791
0.001
100.0
0.05 0.005
2.0
0.0
32.0 16.0
0,00390625
0.75390625
0.200136232
3.649967637
3.479956746
766.9042969
0.001
100.0
0.1 0.005
2.0
0.0
32.0
0.001953125
0.774414063
15623377
3.687826008
3. 3.488405466
7969799
0.001
100.0
0.1 0.005
2.0
0.0
32.0 16.0
0. 1.0-8888813
O. 7570976563
0,209971877
3. 6.615173301
3. 3.199933411
766. 8652344
0.001
100.0
0.05 0.005
4.0
0.0
32.0 8.0
0. 0.001533255
0.7734375
0.171805611
3.671331845
4936007779
766.945667 9459125
0,001
100.0
0.05 0.005
4.0
0.0
32.0
16.0
0. 00390625
0,75390625
0.200136232
3. 6449667637
3 3.50344944
766.841 7969
0.001
100.0
0.1 0.005
4.0
0.0
32.0
8.0
0. 0.001953125
0.771111063
0.15623379
3.687826008
3.179611635
766 876088
0,001
100.0
0.1 0.005
0.0
32.0
0. 0.005859375
0.764648458
0.185098763
3.501440219
4.079826832
766. 866.8730469
0.001
100.0
0.05 0.005
2.0
0.0
64.0 8.0
0.004852813
0.765625
0.19841044
3.547859269
4.141643167
766.9645438
0.001
100.0
0.05 0.005
2.0
0.0
64.0 16.0
0.005859875
0.765625
0.177693718 18 3.50501984 1841
1933133331
766 880.8808594
0.001
100.0
0.1 0.005
2.0
0.0
64.0 8.0
0,0078125
0. 7600811888
0.192957112
3. 3.55072115
1.103323817
766 8632813
0.001
100.0
0.1 0.005
2.0
0.0
61.0
16.0
0,005859375
0.764648438
0,185096763
3.501440219
4.093967438
766.877 873.673
0,001
100.0
0.05 0.005
4.0
0.0
64.0 8.0
0.004862813
0.765625
0.19841044
3.547859269
4.092059493
766.9082031
0.001
100.0
0.05 0.005
4.0 0.0
64.0 16.0
0,005859373
0,765625
0.177693718
3.505019841
4.07618-273
9199299
0.001
100.0
0.1 0.005
0.0
64.0
0.0078125
0.760742188
0.192957412
3.55072115
4.161067963
767.003906
0.001
100.0
0.1 0.005
4.0
0.0
64.0 16.0
0.029296875
0.688476563
0.393779673
3. 0.103399988
3.47068882
766.890625
0.0005
100.0
0.05 0.005
2.0
0.0
32.0
8.0
0.02734375
0.712890625
0,411135931
3. 3,030212674
3. 3-499367029
766. 9921999977
0.0005
100.0
0.05 0.005
2.0
0.0
32.0
16.0
0.03125
0.6875
0.397716837
3. 8.114880398
3. 1903644166
766.9101566
0.0005
100.0
0.1 0.005
2.0
0.0
32.0
8.0
0. 0.028320313
0.719726563
0.391344973
3.054277964
3.492498398
766.8339844
0.0005
100.0
0.1 0.005
2.0
0.0
32.0
16.0
0. 0.029296875
0.688476563
0.393779673
3. 1333999888
3.5061
766.8066406
0.0005
100.0
0.05 0.005
4.0 0.0
32.0
8.0
0.02734375
0.712890625
0.411135951
8.03012267
3.485206127
766.7988281
0.0005
100.0
0.05 0.005
0.0
32.0
0.03125
0.6875
0.397716637
3.114880596
3.504174113
766.861 766.861328/
0.0005
100.0
0.1 0.005
4.0 0.0
32.0 8.0
0. 0.028320313
0.719726563
0.3913449073
9.054277964
3.48001945
760.8085938
0.0005
100.0
0.1 0.005
4.0
0.0
32.0 16.0
0,03125
0. 7.11100563
06419866623
3. 3.0101921199
1.108761052
766.91600 9166156 0.0005
100.0
0.05 0.005
2.0
0.0
61.0 8.0
0.031255
0. 0070078812
D. 4311778824 3.064188056
4.202890396
766.9355469
0.0005
100.0
0.05 0.005
2.0
0.0
64.0
16.0
0.03125
0. 0.706054688 0.417979379
3.022753463
4.087831497
766.8613281
0.0005
100.0
0.1 0.005
0.0
64.0 8.0
0.03125
0. 704101163 0.440352537
3. 3.052575512
4. 131600
766.9726563
0.0005
100.0
0.1 0.005
2.0 0.0
64.0 16.0
0,03125
O. 704101563 0.41984623
3.010492179
4.099819779
766. 9492188 0.0005
100.0
0.05 0.005
0.0
64.0
0.03125
0.705078125
0.431778274
3.0641
4. 1.11238661
767.0527344
0.0005
100.0
0.05 0.005
4.0 0.0
64.0 16.0
0.03125
O. 706054688 0.417979379
0.027553663
4.113277912
766 970707 0.0005
100.0
0.1 0.005
4.0
0.0
64.0 8.0
0.03125
O. T.111116633
0. 0.113352537
3.052575512
1.119800811
767. 097.094
0.0005
100.0
0.1 0.005
4.0
0.0
61.0
16.0
0.00390625
0. 0.784179688 0.07845433
3.813977245
4.564417839
766.8300781
0.001
200.0
0.05 0.005
2.0
0.0
32.0
8.0
0.002929688
0. 0.78320312 125 0.1 12201097 3.684513977 3977
4.55967021
766.9160156
0.001
200.0
0.05 0.005
0.0
32.0
16.0
0. 0.004882813
O. D.783203122 125 0.07
3.8 3.816934612
4.642746329
766.92 766.921878 1875
0.001
200.0
0.1 0.005
2.0
0.0
32.0
8.0
0.004852813
0.779296875 0.093263402
3.684325663
4.574479938
767.0039063
0.001
200.0
0.1 0.005
2.0
0.0
32.0
16.0
0. 0.3000225
0. 07784179688
0.07845433
3.8 13977225
4.564909458
766 8320313
0.001
200.0
0.05 0.005
4.0
0.0
32.0 8.0
0. 000929688
0. 7..2233125
0.112201097
3. 6.645519777
1.595101476
766 9199199988
0.001
200.0
0.05 0.005
1.0
0.0
32.0
16.0
O. 0.00-1882113
0. 7.732031225 0.071694878
3.816931612
4.575098038
766.9642999
0,001
200.0
0.1 0.005
4.0
0.0
32.0 8.0
0.004882813
0. 7.77999687 0.093263402
3. 6.684325663
4.580031991
766.9023458
0.001
200.0
0.1 0.005
4.0
0.0
32.0
16.0
0.0
0. 0.866400255
0.020295 33 8.88961531 1531
5.265 15499 766.923628
0.001
200.0
0.05 0.005
0.0
64.0
8.0
0.0
0. 0.793446313
0.07736926
3. 9.09882361
5. 6.19077108
871.888
0.001
200.0
0.05 0.005
2.0
0.0
64.0 16.0
0.0
0.8046875
0.016822917
3.885627
5.201019526
766.9609375
0.001
200.0
0.1 0.005
2.0
0.0
64.0 8.0
0.0
O. 793944313
0.065121528
3.921 165
5.193174481
766 8808594
0.001
200.0
0.1 0.003
2.0
0.0
64.0
16.0
0.0
O. 8066660625
0.020295139
3. 88961531
5.199734433
766.9140625
0.001
200.0
0.05 0.005
1.0
0.0
64.0
8.0
0.0
O. 0.793945318
0.07736926
3. 3.900982364
5,205765486
766.9570313
0.001
200.0
0.05 0.005
4.0
0.0
64.0
16.0
0.0
0.8046875
0.016822917
3.885627 126
5. 5.192848086
766.8886719
0.001
200.0
0.1 0.005
4.0
0.0
64.0
8.0
0.0
O. 7.79940313
0.065 121528 2.921266161 155
187784677
767.0644531
0.001
200.0
0.1 0.005
0.0
64.0
16.0
0.0 0.012695313
O. 76726693313
0.231309715
3. 199925369-1
1.587309361
766 866.8378906
0.0005
200.0
0.05 0.005
2.0
0.0
32.0
8.0
0. 0.009765625
0.76171875
0.233089303
3.394772003
4.590754867
766.9121004
0.0005
200.0
0.05 0.005
2.0
0.0
32.0
16.0
0.012695313
0.76171875
0.231781967
3. 1186716801
1.591512561
766.939-1531
0.0005
200.0
0.1 0.005
2.0
0.0
32.0
8.0
0. 0.010742188
0. 42188
0.233377585
3.382513951
4.563710451
767.08125
0.0005
200.0
0.1 0.005
2.0
0.0
32.0
16.0
0.012695313
0. 0.766695313
0.231309745
3.499253694
4.560121536
766.859375
0.0005
200.0
0.05 0.005
4.0
0.0
32.0 8.0
0. 0.009765625
0.76171875
0.233089303
3.394772003
4.59262 1326
766.8652344
0.0005
200.0
0.05 0.005
4.0
0.0
32.0
16.0
2.012695313
0.76171875
0.234781967
3.486716801
4.588880777
766 8007813
0.0005
200.0
0.1
32.0
0.010712188
02768742188
0.233577585
3. 3.382513951
4.37550178
766 12199
0.0005
200.0
0.1 0.005
40
0.0
32.0 16.0
0.00390625
0.797851563
0.121854982
3.611278035
5.235417843
766.8730469
0.0005
200.0
0.05 0.005
2.0
0.0
64.0
0. 0000976563
0.7890625
0.23591163
3. 3.58168289-1
5, 1,162270712
766 996099993388
0.0005
200.0
0.05 0.005
2.0
0.0
61.0
16.0
0.00390625
0.797851563
0.125.227204
3.622892618
5. 5.165045619
856.85
0.0005
200.0
0.1 0.005
2.0
0.0
64.0
8.0
0.001953125
0.788085938
0.239 14672
3.5738821 13 5.201363683
766.8847656
0.0005
200.0
0.1 0.005
2.0
0.0
64.0
16.0
0. 0.0300025
0. 0569
0.121854982
3.61 278035
6.172132372
766.994 766.9941400 406
0.0005
200.0
0.05 0.005
4.0
0.0
64.0
8.0
0. 0.000976663
0.7890625
3.58168289-1 0.23591163
5.199103951
766. 766.921875
0.0005
200.0
0.05 0.005
16.0
0.00390625
0.797851563
0.125322704
3. 3.622892688
5.173853517
766.91 9179688
0.0005
200.0
0.1 0.005
4.0
0.0
64.0
8.0
0.001953125
0.788085938
0.239114672
3.573882113
5.177394509
766.8203125
0.0005
200.0
0.1 0.005
0.0
64.0
16.0
71

---

Published in Transactions on Machine Learning Research (02/2026)
ram peak, Lub .uu
Uine total sen
sender last
rocalver Jast
sender auc
lambda jac
aby. steps
adv em10
resdiver
abr Ir
beta ki
innor
meta.
adim
0. 1.008789033
0.73828125
0.234759823
3.582373689
3.185162735
766.953125
0.001
100.0
0.05
0.01
2.0
0.01
32.0
8.0
0. 0.007666225
0.76171875
0.214881581
3.61147658
3.503759861
766.808590
8088938
0.001
100.0
0.05
0.01
2.0
0.01
32.0
16.0
0.015625
0.73046875
0.244402282
3.574520543
3.476381302
766.8730469
0.001
100.0
0.1
2.0
0.01
32.0
0.002929688
0.771484375
0.165628322
3.683584449
3.504372954
766.9472666
0.001
100.0
0.1
0.01
2.0 0.01
32.0
16.0
O. 098789003
0.73828125
0.234759823
3.582373689
3.492276192
766.94330 730.9443554
0.001
100.0
0.05
0.01
4.0 0.01
32.0
8.0
O. 0.007665255
0.76171875
0.21488/1511
3.61147658
3.177153182
766 8515625
0,001
100.0
0.05
0.01
1.0 0.01
32.0
16.0
0.015625
0.73046875
0.244402282
3.574820543
3.462974429
766.7929688
0.001
100.0
0.1
0.01
4.0 0.01
32.0
8.0
0. 1.002929688
03771
0.165628322
6.835814499
3.19:1990706
766 9550781
0,001
100.0
0.1
16.0
0. 0,03996625
0. 0.763671875
0.186622201
3. 4959559998
2.06501941
766. 9042969
0.001
100001110
100.0
0.05
0.01
2.0 0.01 64.0
8.0
0.005859375
0.762695313 0.190373352
3.543969893
4.111203074
766.9624219
0,001
100.0
0.05
2.0 0.01
16.0
0. 0.030622
0. 75666601663
0. 0867
3.512755855
4.090454459
766.8203
0
100.0
0.1 0.01
2.0 0.01 64.0
8.0
0. 0.0078125
0. 752929688 0.21913159
3. 3.164209195 195
4.129907111
766.97666 97669765625
0,001
100.0
0.1 0.01
2.0 0.01
64.0
16.0
0. 0.00390625
0. 763671875 0.186622201
3. 1959551998
1.090277076
766.88 8915313
0,001
100.0
0.05
0.01
4.0 0.01
61.0
8.0
O. 0.05859375
0. 762666631
0. 0.190333352
3. 5.449669933
4. 1 139833689
76669966938
0.001
100.0
0.05
0.01
4.0 0.01
64.0
16.0
0,00390625
766601563 0.1764113568
3.512755855
1.091886661
766. 106
0,001
100.0
0.1
0.0078125
0. 0.752929688
0.21913159
3. 3.464269195
4, 1105113029
826.828125
0.001
100.0
0.1 0.01
4.0 0.01 64.0 16.0
0.01953125
0.711914063
0.406643105
3.163368808
5.509997457
766.8496094
0.0005
100.0
0.05
2.0 0.01 32.0
0.043945313
0.674804688 0.488205295
2. 2.925471717
3. 94498627765
766.9296875
0.0005
100.0 0.05
0.01
2.0 0.01 32.0 16.0
O. 0.02050883
O. 738981375
0. 1011682107
3. 3.161223083
3. 1115819697
766, 8867188 0.0005
100.0
0.1 0.01
2.0 0.01
32.0 8.0
0.0-1992188
0.68457U313
D.466933594
2.957788983
4887766735
766.9667969
0.0005
100.0 0.1 0.01
2.0 0.01 32.0 16.0
0.01953125
0.711914063
0.406643105
3.163368808
3.499094486
766.8027344
0.0005
100.0
0.05
0.01
4.0 0.01 32.0 8.0
0.0-3945331
0.67480-6685
0.488205295 2.925471717
3.48725915
767.0683594
0.0005
100.0
0.05
0.01 32.0
0. 0.020507833
0.7089844375
0.409682407
3. 3.161523083
3.473590493
767 767.0058594 0.0005
100.0
0.1 0.01
4.0 0.01 32.0 8.0
0.041992188
0.684570313
0.466933594 2.957768083
3.51 1683583
786.9941400 406 0.0005
100.0
0.1 0.01
4.0 0.01 32.0 16.0
0,03125
1118667188
0.422857851
3. 02996662--
4.102645857
766 931693 0.0005
100.0 0.05
0.01
2.0 0.01 64.0 8.0
0. 0.037109375
0.69921875
0. 0.167707097
3.0213234488
1.101898211
767.0316685
0.0005
100.0
0.05
0.01
2.0 0.01 64.0 16.0
0. 0.030273438
0.716796875
0.416727209
3.D4256475
4.077229142
766.9355469
0.0005
100.0
0.1
0.01 64.0 8.0
0. 0.040039633
0.694335938
0.483740632
3.00651 8521
4.094444156
766.8378006
0.0005
100.0
0.1 0.01
2.0 0.01 64.0
16.0
0,03125
0.713867188
0.422857851
8.09866204
4.107426643
776.9776563
0.0005
100.0 0.05
0.01 64.0
0.037 109375
0.69921875
0.467807097
3.024323448
4.088187575
767. 1386719
0.0005
100.0
0.05
0.01
4.0 0.01 64.0 16.0
0. 0.030273438
0.716796875
0.416727209
3.042564755
33665677
766.84375
0.0005
100.0
0.1 0.01
4.0 0.01
64.0 8.0
0.010039063
0.691335538
0. 0.183710832
3. 0005188211
1.127311628
766.81375
0.0005
100.0
0.1
0.01
1.0 0.01 61.0 16.0
0,0078125
0.779296875
0.086942319
3. 3.802173239
4.544541717
766.9042969
0.001
200.0
0.05
0.01
2.0 0.01 32.0
8.0
0.008789063
0.772460938 0.139713054
3.63791 2371
4.565919995
766.9628906
0.001
200.0
0.05
0.01
0.01 32.0 16.0
0. 0.002920688
0. 0.78320312 225 0.073504243
3. 3.86331
3.863313226 3226
4.57320559
766.9121004
0.001
200.0
0.1 0.01
2.0 0.01 32.0 8.0
0.001953125
0.784179688
0.126850641
3.6 1.663778332
4.571519375
766996
0.001
200.0
0.1
2.0 0.01 32.0
0.0078122 25
0.779296875
0.086944210 19 3.802173239
4.599365115
766.90625
0.001
200.0
0.05
0.01
4.0 0.01 32.0 8.0
0. 1.08779003
0. 0.772400938
0.139713054
3.637912377 2371
4.587291718
766. 8632813
0.001
200.0
0.05
0.01
4.0 0.01 32.0
16.0
O. 0.009299888
0. 7.783233125
0.0735042213
3.86 3.863313266
1.550252318
766 9023138
0,001
200.0
0.1 0.01
1.0 0.01 32.0 8.0
0. 0.001953125
0. 0.784179688 0.126830641
3.663778832
4.877753067
766.9160156
0.001
200.0
0.1 0.01
4.0 0.01
32.0
16.0
0.0
0.805664063 0.0202951 13 3.880446473
5.178888162
766.9101563
0.001
200.0
0.05
2.0 0.01 64.0 8.0
0.0
0.79296875
0.1052419993
3. 8.805329644
5.235127091
766.9160156
0.001
200.0
0.05
0.01
2.0 0.01
64.0
16.0
0.0
0.805664063
0.022465278
3.883667675
5.218698025
766.8508594
0.001
200.0
0.1
2.0 0.01
64.0
0.000976663
1.791992188 188
0.1 02152056604
3.797 862 16
5.181499692
766.921875
0.001
200.0
0.1 0.01
2.0 0.01
64.0 16.0
0.0
0,805661063
0.020295139
3. 8880116733
5. 169565088
766.8925781
0.001
200.0
0.05
0.01
1.0 0.01
61.0 8.0
0.0
0.79296875
0.105241993
3.8 3.805329684
5,176501393
766.953125
0,001
200.0
0.05
0.01
4.0 0.01 64.0
16.0
0.0
0.805664063
0.022465278
3. 3.883867675
5.181358099
766.875
0.001
200.0
0.1 0.01
4.0 0.01 64.0
8.0
0.000976563
0.791 902 88
0.1 12056804
3.7971 86216
5. 183052063
766.8671875
0.001
200.0
0.1
0.01 64.0
16.0
0. 0.006835938
0.771414063
0. 0.239691353
3.5 3.5-153-6912
4 4.59584856
767 767.0449211
0.0005
200.0 0.05
0.01
2.0 0.01 32.0 8.0
0.006835038
0.767578 15 0.259089737
4.57220912 3.30466137 137
766.8730469
0.0005
200.0 0.05
0.01
2.0 0.01 32.0
16.0
0. 1.00789003
0.774414063
0.240667916
3 3.534573811
4.576325177 18S
766 766.9121094
0.0005
200.0
0.1
0.01
2.0 0.01 32.0
8.0
0. 0.008359388
0. 0.761618-138
0.259012111
3.382881026
4.61 L661667522
766.8515628
0.0005
200.0
0.1
0.01
2.0 0.01 32.0
16.0
0. 0.006835938
0.774414063
0.239691353
3.545346912
4.548844612
766. 8652344
0.0005
200.0 0.05
0.01
4.0 0.01 32.0 8.0
0. 0.068335388
0.767578125
0.259089787
3.39466137 337
4.57443428
766.9101563
0.0005
200.0
0.05
0.01
4.0 0.01 32.0
16.0
0. 0008789033
0.774414063
0.240667916
3.534573811
4.56365
1443
766.8945313
0.0005
200.0 0.1
4.0 0.01 32.0
8.0
0. 0006835938
0.7616184138
0.259012145 11 3. 1.382881026
1.5703959117
766. 8925781
0.0005
200.0
0.1 0.01
1.0 0.01 32.0 16.0
0. 0.004882813
0.79199218 188
0. 1.130633117
3.590356213 13 68795666
760.8652344
0.0005
200.0
0.05
0.01
2.0 0.01 64.0 8.0
0. 0.05555975
0.771411063
0.2529835-12
3. 113388895
5, 2129975233
766.90625
0.0005
200.0
0.05
0.01
2.0 0.01
61.0
16.0
0.00390625
0.793945313
0.131299426
3. 608041399
5 20837337
866.8339844
0.0005
200.0
0.1
0.01
2.0 0.01
64.0
8.0
0. 1.004882813
0.7734375
0.253770771
3.485601261
5.190756202
886.8944844
0.0005
200.0
0.1 0.01
0.01
64.0
16.0
0. 0.004862813 13
021199214 188 0. 0333663321
3. 3.590356213 13
5. 150092546
766.9257813
0.0005
200.0
0.05
0.01
4.0 0.01
64.0
8.0
0. 0.005859875
0.774414063
0.262983542
3.493388295
5. 2088888861
766 9707037
0.0005
200.0
0.05
0.01
64.0
16.0
0.00390625
0.793915313
12612991126
3. 6.680193999
5. 5.236365676
766.98 95911511
0.0005
200.0
0.1 0.01
4.0
0.01
61.0
8.0
0.004852813
0.7734375
0.253770771
3.485601261
5.219882965
766. 186.9996069
0.0005
200.0
0.1
0.01
64.0
0. 0.005859375
0.7411406225
0.224260116
3.6124632241
3.17030735
766.81 8111
0.001
100.0
0.05 0.005
2.0 0.01
32.0
8.0
0. 0.010742188
0.76171875
0.215372865
3.617039222
3.490041375
766.8730469
0.001
100.0
0.05 0.005
2.0 0.01
32.0
16.0
0.017578125
0.727539063
0.245378844
3.584327921
3.494503975
766.8828125
0.001
100.0
0.1 0.005
2.0 0.01
32.0 8.0
0. 0.00300625
O. 1.770507813
06 (166116600 16603
3. 6699371678
3.497444153
766. 186.9824219
0.001
100.0
0.1 0.005
2.0
0.01
32.0
16.0
0.005859373
0.741140625
0.224260116
3.612163241
3867211992
766. 7871091
0.001
100.0
0.05
0.005
0.01
32.0
0.010742188
0.76171875
0.215372865
3.617039222
3.518079996
766.8613281
0.001
100.0
0.05
0.005
4.0
0.01
32.0
16.0
72

---

Published in Transactions on Machine Learning Research (02/2026)
0.017578125 0.727539063 0.245378844 3.584327921 3.484863997 836.8378906
73

---

Published in Transactions on Machine Learning Research (02/2026)
S Ablation on Oracle Selection
A key design choice in GEMS is the use of an EmpiricallBernsteinn UCB (EB-UCB) oracle (Section 3.4, Eq. 7) to select new policies. A natural question is why this specific, variance-aware bandit algorithm was chosen over simpler or alternative methods, such as standard UCB1 or Thompson Sampling.
The "Shifting Meta" Experiment While the oracle GEMS solves a stationary sub-problem within each fixed iteration (satisfying the assumptions of Theorem 3.3 the global problem across training is non-stationary. As the meta-strategy or evolves and the generator Ga is fine-tuned, the expected value of any given latent policy detecting when previously suboptimal policy has become optimal and, conversely, abandoning a previously optimal policy that is no longer effective.
Ar changes. The oracle must therefore be highly adaptive, capable of quickly
To simulate this dynamic, we designed a simple "Shifting Meta" bandit game. The game consists of 3 arms.
Phase 1 (Time Steps << 1000): Arm 1 is the unique optimal policy with the highest expected reward.
Meta Shift (Time Step 1000): The underlying meta-gamt abruptly changes.
Phase 2 (Time Steps I 2 1000): Arm 0 becomes the new unique optimal policy.
Critically, designed Arm 0 to also have higher reward variance than the other arms. This setup tests algorithm's ability to abandon a "safe," well-explored, but now-suboptimal arm in favor of "riskier," high-variance arm that has become optimal.
A Analysis of Results
The results are shown in Figure 39
Cumulative Regret (Top): In Phase 1, all algorithms (except Greedy) successfully identify the optimal arm and achieve low regret. After the meta-shift at = 1000, the limitations of non-adaptive oracles become clear. Greedy, being purely exploitative, never adapts and accumulates massive regret. Standard UCB1, which is not variance-aware, also accumulates a large amount regret as it is too "confident" in the old optimal arm. In contrast, the variance-aware methods (EB-UCB, UCB-V) and Thompson Sampling quickly adapt, and their cumulative regret begins to decrease they exploit the new, higher-value optimal arm.
Adaptability (Bottom): This plot provides the clearest justification. After the shift, standard UCB1 takes approximately 500 time steps (from 1000 to 1500) to reliably switch to the new optimal arm. Thompson Sampling also slow, taking over 250 steps. In stark contrast, both EB-UCB and UCB-V adapt almost instantaneously. Their sensitivity to variance means they never became over-confident in the old arm and were able to quickly recognize the value of the new, high-variance optimal arm.
74

---

Published in Transactions on Machine Learning Research (02/2026)
Oracle Performance on the "Shifting Meta" Game
Greedy
1500
UCBIV
1250
Thompson Samplin
Curmisttio Regret
750
500
-250
Adaptability: Frequency Selecting the Optimal Policy
1.0
0.8
Opdrall Choice 95 (MA over 50 steps)
0.6
0.4
UCB1
EB-UCB
Thompson Sampinn
1000
Time Steps
Figure 39: Performance of different oracle algorithms on the "Shifting Meta" game, averaged over 50 runs. The meta-game shifts at = 1000. (Top) Cumulative Regret. Lower is better. Note the poor performance of Greedy and Thompson Sampling after the shift. (Bottom) Adaptability, measured as the 50-step moving average of picking the correct optimal arm. The variance-aware oracles (EB-UCB and UCB-V) adapt almost. instantaneously, while UCB1 and Thompson Sampling are significantly slower.
This experiment demonstrates that simple UCB1 oracle is unsuitable for the non-stationary problem faced in GEMS, its over-confidencce would cause it to lag significantly behind the evolving mela-game. The superior adaptivity of variance-awan aware oracles is critical. We chose EB-UCB as it not only demonstrates this stato-of-tho-art adaptability but also aligns directly with the thcoretical concentration bounds (based on empirical Bernstein) used in our analysis (Section 3.4).
Conclusion
75

---

Published in Transactions on Machine Learning Research (02/2026)
Part IV
Frequently Asked Questions (FAQs)
1. Q: Why did you not benchmark GEMS on the StarCraft Multi-Agen Challenge (SMAC)?
A: Our work introduces GEMS as a direct, scalable framework to overcome the fundamental ineffi- ciencies of the Policy-Space Response Oracles (PSRO) paradigm, namely its quadratic computation and linear memory costs. Therefore, our primary objective was to benchmark GEMS directly against classical PSRO and its most relevant variants (e.g., Double Oracle, Alpha-PsRO, A-PSRO) in domains that clearly expose these bottlenecks and test game-theorees solution quality, such as Kuhn Poker, Deceptive Messages Game, and Multi-Agent Tag.
2. Q: Why did you choose Optimistic Multiplicative Weights Update (OMWU) over standard Multiplicative Weights Update (MWU)?
A: We chose OMWU because it provides stronger theoretical guarantees and faster convergence in our setting. Unlike standard MWU, OMWU incorporates a predictive "hint" about the next. payoff vector. This optimistic step results in an average external regret bound that scales with the cumulative variation of the payoff vectors (O l|0 ez .ll2)) rather than the time horizon T, as shown in Proposition 3.2. Since GEMS is designed to induce slowly changing meta-game, this property leads to faster convergence.
3. Q: How can GEMS be considered "surrogate-free"
A: We use the term "surrogate-free" to signify that GEMS does not maintain the two key surrogates of classical PSRO: an explicit, discrete population of k policies, which requires O(k) memory, and @ the full k x k payofl malrix, which requires O(k²) computation. Instead, GEMS replaces this entire structure with single amortized generator (Ge) and compact set of latent "anchor" codes (Zi).
Q: Does the single amortized generator (Gg) suffer from catastrophic forgetting?
A: We explicitly mitigate this risk using our Amortized Best-Response with a Trust Region (ABR- TR) objective (Section 3.5). This objective is designed to fine-tune the generator to produce new, high- petforrrrrrm policies while retaining its ability to generate previously effective ones. It achieves this by incorporating a KT-divergence penalty against, frozen, older version of the generator (0-), which serves trust region and prevents catastrophic forgetting.
5. Q: What is the main computational bottleneck of GEMS?
A: GEMS successfully replaces the O(k²) computational overhead of PSRO. The new computational cost is dominated by the number of Monte Carlo rollouts required per iteration. This cost scales with the number of sampled matches used to estimate the mela-game (controlled by ni; m, and B in Section 3.2) and the size of the candidate pool ([Ar)) evaluated by the bandit oracle (Section 3.4). This simulation-based cost is fundamentally more scalable than constructing the full payoff matrix.
6. Q: How do the theoretical guarantees of GEMS compare to classical PSRO, given its use of approximations?
A: GEMS retains the core game-theoretic convergence guarantees of PSRO. Our overall exploitability bound (Theorem 3.4, Section 3.6) cleanly decomposes the average exploitability into four interpretable terms: the external regret of the OMWU meta-solver, the noise from Monte- Carlo estimation, 3 the sub-optimality of the bandit oracle, and (4) the approximation error from the amortized generator. As the simulation budget (nm) grows and the generator training improves (EBR 0), the latter three terms vanish, and the overall exploitability is driven by the no-regret property of the meta-solver, ensuring convergence.
76

---

Published in Transactions on Machine Learning Research (02/2026)
7. Q: Why use an EB-UCB bandit oracle instead of just computing single, direct best response (BR)?
A: The goal of population expansion is to efficiently find new, challenging policies to add to the game
($ 3.44.. Simply training a single BR can be computationally expensive. Instead, we cast this as a multi-armed bandit problem over pool of candidate latent codes A,- We use the Empirical-Bernstede UCB (EB-UCH UCB) oracle because efficiently balances the exploration-exploitaiio trade-off by using sample variance to achieve tighter confidence bounds. This allows GEMS to select promising new policies from the candidate pool more efficiently and effectively explore the latent strategy space.
8. Q: What is the purpose of the Jacobian penalty (A₂)?
A: The Jacobian penalty (A+IIJGo==)IIP)) is a regularizer applied during both the oracle selection (Eq. and the generator training steps (Eq. [1]. Its purpose is to encourage smoothness in the generator's 's latent space by penalizing large gradients of the generator's 's output with respect to its latent imput 2. This smoothness aids in stabilizing the optimization process.
9. Q: Why did you benchmark against PSRO variants and not other modern MARL algorithms like MAPPO or QMIX?
A: GEMS is specifically proposed to solve the fundamental scalability bottlenecks inherent in population-based, game-theoretic approaches, for which PSRO is the seminal framework (Section 2). Algorithms like MAPPO or QMIX, while effective, address a different problem (e.g., decentralized exccution with centralized training for a fixed number of agents) and do not typically maintain solve mela-game over explicit population of policies. Therefore, to scientifically validate our claims, the most relevant and direct baselines are classical PSRO and its state-of-the-art variants.
10. Q: How does GEMS extend from two-player zero-sum (2P-ZS) games to the n-player general-sum (NP-GS) case?
A: The core components of GEMS extend naturally to the NP-GS setting, detailed in Section 3.7 and Appendix Part II. The main generalizations are: (1) Each of the players maintains their own independent meta-strategy of (P) (2) We use a single batch of shared game rollouts and an aypoortance-eiihhee estimator (Eq. 14) to efficiently compute each player's per-policy value vector (P) against the joint strategy of all other players. (3) Each player then independently runs their own OMWU update and EB-UCB oracle. (4) This decentralized process drives the timo-averaged joint strategy loward an Cwaaaa--CCrrellaae Equilibrium (e-CCE), the standard solution concept for general-sum games.
11. Q: The ablation tables for the Public Goods Game (Table 4) and Deceptive Messages Game (Table 5) present many results across different hyperparameter settings, but the paper docsn't explicitly state which configuration definitively "best." What is the main takeaway from these ablations?
A: The primary purpose of the extensive ablation tables (Table for PGG, Table 5 for Deceptive Messages) is to ensure transparency and aid reproducibilltty by documenting GEMs'S sensitivity to its core hyperparameters. Identifying single, universally optimal configuration across all games and metrics is challenging, as a full factorial sweep is computationally infeasible. These tables showcase the results of structured variations, illustrating the inherent trade-offss involved (e.g., between convergence speed, solution quality, and computational resources). We provide this detailed data to allow readers to observe these sensitivities directly and understand the impact of different choices. Determining the absolute optimal settings for every possible scenario remains an open area, and these tables serve a valuable resource for guiding future work or tuning GEMS for specific applications. Our main experiments utilize configurations found to be effective for demonstrating the core advantages of GEMS.
12. Q: What are the limitations introduced by GEMS's use of Monte Carlo estimation and amortized generator?
A: GEMS introduces two main sources of approximation to enable O(1) memory scaling:
77

---

Published in Transactions on Machine Learning Research (02/2026)
First, Estimation Noise: Payoffs are estimated via Monte Carlo sampling. In sparse-reward domains, this can increase the variance of the meta-gradient. However, our results show that even noisy estimates guide the population toward effective strategies significantly faster than exact methods, as the sampling cost scales lincarly (O(k)) rather than quadratically (O(k²)).
Second, Amortization Gap: The generator approximates the best-response manifold. A potential limitation is that an under-parameterrized generator could fail to capture niche counter-strategies, We address this by explicitly conditioning the generator on a growing set of "anchors" (Zt), which forces the network to maintain diverse modes. Empirically, our 1,000-iteration Chess experiments demonstrate that the generator successfully maintains over 2,000 distinct policies without collapsing, validating its expressive capacity.
13. Q: How does the theoretical analysis account for the non-stationarity of rewards as the meta-stratogy evolves?
A: We address the global non-stationarity of the learning problem by decomposing GEMS into two distinet timescales, ensuring that the assumptions for our regret bounds are locally satisfied:
Inner Loop (Stationary Sub-problem): Within any single meta-iteration t, the opponent meta-strategy o, held fixed. Consequently, when the oracle selects an anchor E A,, it faces a mathematically stationary reward distribution defined by this frozen opponent. This stationarity ensures that the instance-dependent regret bounds (Theorem 3.3) are valid for the anchor selection step.
Outer Loop (Dynamic Meta-Game)) The non-stationarity arises only across iterations as o, evolves. This global dynamic is explicitly handled by the OMWU meta-solver, which is designed to minimize dynamic regret in time-varying games (Proposition 3.4).
Thus, while the overall landscape shifts (motivating our use of adaptive bandits like EB-UCB in the "Shifting Meta" experiment); the specific sub-problem addressed by the oracle each step remains stationary and theoretically amenable to our bounds.
14. Q: When do Monte-Carlo rollouts become a computational bottleneck, and how does GEMS handle this tradc-off?
A: Monte-Carlo (MC) sampling effectively trades a "hard" computational wall for a "soft" sampling cost. We analyze this trade-off as follows:
The Quadratic Bottleneck (PSRO): Standard population-based methods require computing an exact payoff matrix. This operation scales quadratically, O(N),, with the number of iterations N (as the population grows indefinitely.. For complex games, this quickly becomes computationally intractable.
The Constant- Timmtant-Timee Scalability (GEMS):
In contrast, GEMS maintains a compact, fixed-size anchor sct (|A) < N). Consoquently, the computational complexity per itcration is O(1) with respect to the total history of strategics.
The Trade-off (Sampling Variance):
While the complexity class is constant, the wall-clock time depends on the sampling difficulty. The cost is proportional to O(k Cevat), where k is the number of samples and Ceval the cost per episode. This can become a bottleneck in environments with extreme stochasticity (roquiring high k) or very long episodes (high Cevat).
The Solution (EB-UCB): To mitigate this, GEMS employs the EB-UCB oracle. It adaptively allocates the sampling budget k, spending compute only when the estimator is uncertain (high variance) or the strategy is promising. This ensures that even when Crval is high, GEMS avoids wasting resources on sub-optimal strategies, preserving practical scalability where matrix-based methods have limitations.
78


Reviews:
[{"review_id": "SJwzk5SNiM", "reviewer": "Reviewer_jsEg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes GEMS (Generative Evolutionary Meta-Solver), a framework for scalable multi-agent reinforcement learning (MARL) that aims to address computational bottlenecks in Policy Space Response Oracles (PSRO). The core idea is to replace PSRO's explicit policy population (O(k) memory, O(k²) payoff matrix computation) with: (1) a single amortized generator G_θ that maps latent codes to policy parameters, (2) a compact anchor set in latent space, (3) Monte Carlo rollouts for payoff estimation instead of full matrix construction, (4) Optimistic Multiplicative Weights Update (OMWU) for meta-strategy optimization, and (5) an Empirical-Bernstein UCB oracle for selecting new anchors.\n\nKey Strengths:\n\n- Clear problem motivation addressing real scalability issues in PSRO\n- Comprehensive appendix with proofs, ablations, and extensions to n-player general-sum games\n- Breadth of experimental domains (Kuhn Poker, Chess, Go, Hanabi, PettingZoo environments)\n- Transparent acknowledgment of limitations in Section 6\n- Some potentially novel algorithmic components (EB-UCB oracle for MARL, ABR-TR training objective)\n\n\nKey Weaknesses:\n\n- Critical missing citation: Neural Population Learning (NeuPL, Liu et al., ICLR) proposes essentially the same core contribution (single conditional network representing policy populations with transfer learning)\n- Outdated baselines: comparisons primarily against \"classical PSRO\" (2017), missing modern methods like EPSRO, P2SRO, Self-Adaptive PSRO \n- Inconsistent statistical reporting: missing error bars on important results\n- Contradictory claims: paper claims efficiency advantage but states SMAC experiments were \"computationally infeasible\"\n- Chess/Go experiments lack comparison to specialized methods (AlphaZero, etc.)", "claims_and_evidence_yn": "No", "claims_explanation": "The paper's central novelty claim (introducing a \"surrogate-free\" framework with an amortized generator to replace explicit policy populations)is undermined by the existence of Neural Population Learning (NeuPL, ICLR), which proposes the same core idea: representing an entire policy population within a single conditional network to enable memory efficiency and transfer learning across policies. This prior work is not cited or compared against.\n\nThe experimental evidence has several gaps:\n\n- Baseline currency: Comparisons focus on \"classical PSRO\" from 2017. More recent methods like EPSRO, Pipeline PSRO, and Self-Adaptive PSRO are not included. Without these comparisons, it is unclear whether GEMS advances the state-of-the-art.\n- Statistical rigor: Key results (e.g., Kuhn Poker exploitability, Chess win rates) lack error bars and significance tests. The number of random seeds is not consistently reported. The authors acknowledge that \"full factorial hyperparameter sweep was computationally infeasible,\" raising reproducibility concerns.\n- Contradictory efficiency claims: The paper claims GEMS is more efficient than PSRO, yet states that SMAC experiments were skipped due to \"prohibitive computational cost.\" If GEMS is 6x faster than PSRO, and PSRO-based methods have been applied to SMAC, this contradiction is unexplained.\n- Theoretical concerns: Theorem 3.3's instance-dependent regret bound for the EB-UCB oracle assumes stationary reward distributions, but, if I understand correctly, in GEMS the payoff distribution for each anchor changes as the meta-strategy evolves. The paper does not address whether this non-stationarity invalidates the bound.\n- Uninterpretable results: Chess and Go experiments (1000 iterations) claim \"emergent strategic depth\" but provide no comparison to AlphaZero or Elo ratings, making the results uninterpretable.", "audience_yn": "Yes", "audience_explanation": "Scalable multi-agent reinforcement learning is an important and active research area. The problem of reducing PSRO's computational and memory costs is practically significant, and population-based game-theoretic methods are widely used in domains ranging from poker to real-time strategy games.\n\nEven if the paper's novelty claims require revision, the experimental breadth, extensive ablations, and extensions to n-player general-sum games could provide useful empirical insights. The combination of OMWU, EB-UCB oracles, and trust-region training in this context may interest practitioners seeking scalable MARL solutions.\n\nHowever, the paper's value to the community would be significantly enhanced by proper positioning relative to NeuPL and comparison against modern baselines.", "requested_changes": "- Cite and compare against Neural Population Learning (NeuPL): This ICLR paper proposes the same core idea of representing policy populations in a single conditional network. The paper must: (a) add thorough discussion in Related Work, (b) clearly articulate what GEMS adds beyond NeuPL, and (c) provide head-to-head experimental comparison on at least one domain (e.g., Kuhn Poker). Also cite Simplex-NeuPL and NeuPL-JPSRO.\n- Add modern baselines: Include comparisons against at least: (a) EPSRO (b) one of Pipeline PSRO, Anytime PSRO, or Self-Adaptive PSRO. The current comparison against only \"classical PSRO\" from 2017 is insufficient.\n- Add error bars and significance tests: All main results must report mean +/- standard deviation over multiple random seeds. \n- Resolve SMAC contradiction: Either run SMAC experiments (if GEMS is truly more efficient than PSRO) or provide a detailed computational cost comparison explaining why SMAC is infeasible for GEMS but not for classical PSRO.\n- Address non-stationarity in EB-UCB analysis: The instance-dependent regret bound (Theorem 3.3) assumes stationary rewards. Discuss whether/how this bound holds when payoffs change as the meta-strategy evolves, or acknowledge this as a limitation.\n- Recommend default hyperparameters: Given the many tunable parameters, please provide a recommended default configuration for practitioners.", "broader_impact_concerns": "No major ethical concerns are identified. The work is foundational research on multi-agent learning algorithms.", "additional_comments": "This paper tackles an important problem and demonstrates substantial effort through its comprehensive appendix, diverse experiments, and theoretical analysis. The writing is clear and the limitations section is commendably honest.\n\nHowever, the paper cannot proceed to acceptance without addressing the NeuPL gap. The core contribution (amortized policy generation via a single network) was published at ICLR, and this directly impacts the novelty assessment. The authors may have developed GEMS independently, but the positioning must acknowledge and differentiate from this prior work.\n\nIf the authors can demonstrate that GEMS outperforms NeuPL (perhaps through the EB-UCB oracle or ABR-TR objective), and can show improvements over modern PSRO variants with rigorous statistics, the paper would make a solid contribution to the scalable MARL literature.\n\nI encourage revision and resubmission after addressing the critical items above. The infrastructure built for Chess/Go experiments suggests the authors have strong engineering capabilities that could yield compelling results with proper baselines.\n\nMinor notes:\n\n- The FAQ section in the appendix is a nice touch and shows good faith engagement with potential reviewer concerns\n- The n-player general-sum extension (Appendix H) appears non-trivial and could be highlighted more", "extra_scores": {}, "extra_text": {}}, {"review_id": "EexHav5i1k", "reviewer": "Reviewer_r2ko", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes GEMS, a surrogate-free population-based MARL framework that overcomes the scalability limitations of PSRO by eliminating explicit policy populations and dense payoff matrices. The method leverages unbiased Monte Carlo payoff estimation, a compact latent anchor set, and a single amortized generator to achieve improvements in memory and computational efficiency while preserving core game-theoretic guarantees. The authors provide theoretical guaranteens including regret bounds and exploitability guarantees under approximate best responses. Extensive experiments in both zero-sum and general-sum benchmarks show that GEMS converges faster, uses less memory, and often achieves higher-quality strategies than PSRO and its variants.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides theoretical guarantees in the form of formal theorems accompanied by detailed proofs. The experimental results are presented comprehensively and in sufficient detail.", "audience_yn": "Yes", "audience_explanation": "The topic of multi-agent reinforcement learning (MARL) is of interest to the TMLR community.", "requested_changes": "1. While the overall framework is described clearly, it would be beneficial to include the full algorithm box in the main paper rather than relegating it to the appendix. In particular, explicitly specifying the inputs and outputs of each module (e.g., anchor set maintenance, Monte Carlo payoff estimation, meta-strategy update, EB-UCB selection, and generator training) and how they are connected would significantly improve clarity and accessibility.\n2. The paper presents several theoretical guarantees for different components of the method, including Monte Carlo rollouts, OMWU meta-dynamics, EB-UCB selection, and assumptions on the generator’s approximation quality. While Theorem 3.4 provides a final regret bound, it is stated primarily as an end result and does not explicitly show how this bound is composed from the guarantees of the individual components. It would strengthen the theoretical exposition to include a theorem (or proposition) in the main text that clearly decomposes the overall regret into contributions from these parts, making the interplay between the different guarantees more transparent.\n3. While the empirical results convincingly demonstrate that GEMS outperforms PSRO-style baselines in practice, it is unclear whether any theoretical comparison can be drawn. If possible, the authors could include a table summarizing theoretical properties (e.g., memory complexity, computational complexity, regret/exploitability guarantees, assumptions) of GEMS versus PSRO and its variants, which would complement the experimental findings.\n4. Although Monte Carlo sampling avoids constructing dense payoff matrices, it would be helpful to further discuss whether and when Monte Carlo rollouts may become a computational bottleneck. A more explicit analysis or discussion of the trade-off between sampling cost and efficiency gains would improve the reader’s understanding of the method’s practical scalability.\n5. Figure 5 provides an insightful qualitative comparison of emergent behaviors, but additional explanation in the caption or main text would help readers from different backgrounds interpret the trajectories. In particular, explicitly describing what behaviors (e.g., flanking vs. herding) should be observed and how they relate to the quantitative performance would improve the figure’s interpretability.", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "kOwsy9XTG1", "reviewer": "Reviewer_Asaz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**SUMMARY** The paper proposes GEMS, a framework for learning in multi-agent settings that aims to overcome the scalability limitations of classical PSRO methods. Instead of maintaining a growing population of independently trained policies, GEMS uses a neural generator that maps latent codes to policies. New strategies are discovered by sampling candidate latent vectors—via mutation of existing ones and random exploration—and selecting promising candidates with a bandit-style oracle. Payoff estimates between policies are obtained by Monte Carlo rollouts, and the equilibrium mixture over strategies is updated using Optimistic Multiplicative Weights Update, which avoids constructing the full payoff matrix. Although approximate, the method comes with convergence guarantees toward Nash equilibria of the restricted game.\n\n***STRENGTHS***\nThe method is conceptually simple and the paper provides convergence guarantees. First, instead of training a winning policy from scratch, it creates a 'latent space of policies' or a generator off which to generate candidate policies. Second, instead of calculating a payoff matrix in a round-robin tournament, it samples payoffs from a fewer tournament steps. This provides a 'good enough approximation' of the best-response policy that is theoretically grounded.\n\n**WEAKNESS** Approximations come with accuracy costs. For example, the rewards matrix of each policy is approximated by sampling some evaluation steps. For longer and complicated games, where the sucess signal is obtained far deep in the game, the Monte Carlo rollouts can be almost as costly as playing the game to the end.\n\nThe second approximation is on the generator network. As discussed by the authors in the 'Limitations' section, the expressivity and diversity of the generator network is not guaranteed. The best-response policy is obtained through a search mechanism akin to genetic algorithms plus a random variant. This depends on the pool of policies already parametrized in the generator network. For example, if we have very few policies or policies of the same kind, the scope of the generator network is limited in diversity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "**Are the claims made in the submission supported by accurate, convincing and clear evidence?**\n\nThe generator’s ability to represent an expanding set of diverse strategies is assumed but not demonstrated at large scale, and, as the authors acknowledge, neither its expressivity nor its resistance to catastrophic forgetting is theoretically guaranteed. Although the paper includes one experiment in the Multi-Agent Tag environment that reaches approximately 100 policies, this is still a modest population relative to the much larger scales the method is designed to target, and the experiment does not analyze whether the generator successfully maintains distinct behaviors across all anchors. Moreover, the evolutionary search for new latent codes relies heavily on the diversity already present in the anchor set; while random sampling is intended to mitigate this dependence, the empirical evaluations consistently use very small latent dimensions (d = 2) and relatively limited policy populations (typically 6–40, with only a single case reaching ~100), leaving the broader claims of large-scale scalability insufficiently substantiated by the presented evidence", "audience_yn": "Yes", "audience_explanation": "Yes. The paper addresses a well-known scalability limitation in population-based multi-agent reinforcement learning and proposes a conceptually appealing alternative that replaces explicit policy populations with a generative latent-policy model. This line of work is of interest to researchers studying game-theoretic RL, equilibrium computation, evolution-based methods, and scalable multi-agent systems. Although the empirical validation remains limited in scale, the central idea is novel and relevant enough that a portion of the TMLR audience would likely find the methodological contribution and theoretical analysis worthwhile", "requested_changes": "1. The paper introduces many moving pieces at once. It will be very beneficial to include a pseudocode, or share the codebase or even some infographics of how the different parts operate together.\n\n2. The figure style like figure 4 are hard to parse. It is not showing what it intends to show.\n\n3. The paper is targetted to an audience very familiar with the topics and each of the techniques presented. A more colloquial language will appeal to a wider audience in the multi agent community.\n\n3. A table comparing the dimensions of the policies explored on each of the experiments will make the claims stronger. Example:\n\n| Experiment           | Latent Dimension (d) | Number of Policies Explored | Notes                                                                 |\n|----------------------|-----------------------|------------------------------|-----------------------------------------------------------------------|\n| Deceptive Messages   | 2                     | ~6                           | Very small population; early proof-of-concept.                        |\n| Kuhn Poker           | 2                     | ~40                          | Largest fully analyzed small game; still modest scale.                |\n| Public Goods Game    | 2                     | ~20                          | Cooperative setting; limited anchor expansion.                        |\n| Simple Spread        | 2                     | ~10                          | Coordination environment; generator evaluated on small policy set.    |\n| Multi-Agent Tag      | 2                     | ~100                         | Largest experiment; no analysis of whether generator preserves diversity. |\n| Chess                | 2                     | <10                          | Minimal experiment; demonstrates feasibility but not scale.           |", "broader_impact_concerns": "None", "additional_comments": "Marginally above the acceptance threshold. More experiments showing the performance on a big number of policies and grraphs clearly showing the diversity of the beest-response policies will be beneficial.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)