Reevaluating Policy Gradient Methods for Imperfect-Information Games
Authors: Max Rudolph
Nathan Lichtlé
Sobhan Mohammadpour
Alexandre M Bayen
J Zico Kolter
Amy Zhang
Gabriele Farina
Eugene Vinitsky
Samuel Sokota
Venue: ICLR 2026 Poster (2026)
Keywords: imperfect-information games
two-player zero-sum games
reinforcement learning
multi agent
game theory
Primary area: reinforcement learning
TL;DR: We show that generic deep policy gradient methods may be stronger than previously understood for imperfect-information games.

Abstract: In the past decade, motivated by the putative failure of naive self-play deep reinforcement learning (DRL) in adversarial imperfect-information games, researchers have developed numerous DRL algorithms based on fictitious play (FP), double oracle (DO), and counterfactual regret minimization (CFR). In light of recent results of the magnetic mirror descent algorithm, we hypothesize that simpler generic policy gradient methods like PPO are competitive with or superior to these FP-, DO-, and CFR-based DRL approaches. To facilitate the resolution of this hypothesis, we implement and release the first broadly accessible exact exploitability computations for five large games. Using these games, we conduct the largest-ever exploitability comparison of DRL algorithms for imperfect-information games. Over 7000 training runs, we find that FP-, DO-, and CFR-based approaches fail to outperform generic policy gradient methods.

Published as a conference paper at ICLR 2026
REEVALUATING POLICY GRADIENT METHODS FOR GAMES
Max Rudolph":! Nathan Lichtlé:,, Sobhan Mohammadpour" Aleeandre 2.leadd Bayen?,
J. Zico Kolter", Amy Zhang', Gabriele Farina', Eugene Vinitsky", Samuel Sokota" l University of Texas at Austin, UUniversity of California, Berkeley,
Massachusetts Institute of Technology. *Carnegie Mellon University,
'NYU Tandon School of Engineering
mrudolphêcs utexas edu,, nathan lichttee@@aaiill com, somo@mit. edu, gfarina @m turtttt.c edd,, vinitsky eugene@gmai 1 com, "Equal contribution
cmu, cmu.edu
ABSTRACT
In the past decade, motivated by the putative failure of naive self-play deep reinforcement learning (DRL) in adversarial inperfectt-informaton games, re- searchers have developed numerous DRL algorithms based fictitious play (FP), double oracle (DO), and counterfactual regret minimization (CFR). In light of re- cent results of the magnetic mirror descent algorithm, we hypothesize that simpler generic policy gradient methods like PPO are competitive with or superior to these FP-, DO-, and CFR-based DRL approaches. To facilitate the resolution of this hy- pothesis, we implement and release the first broadly accessible exact exploitability computations for five large games. Using these games, we conduct the largest-ever exploitability comparison of DRL algorithms for imperfect-informaatin games. Over 7000 training runs, we find that FP-, DO-, and CFRRbased approaches fail to outperform generic policy gradient methods.
INTRODUCTION
An imperfect-information game (IIG) one in which there is information asymmetry between play- ers or in which two players act simultancously. Two-player zero-sum "hsee in which two players with opposite incentives compete against one another- are among the most well-studied IIGs. This is in part due to the fact that, for these games, there is sensible and tractable objective: minimizing the amount by which a player can be exploited by a worst-case adversary.
A natural approach to model- free deep reinforcement learning (DRL) in such games is to deploy single-agent algorithm in self-play. However, because imperfect information induces cyclical best response dynamics, such approach can fail catastrophically, yielding policies that are maximally exploitable. So as to avoid this outcome, most existing literature focuses on adapting pre-existing tabular algorithms with established equilibrium convergence guarantees, such as fictitious play (FP) (Brown 195T; Robinson 195T). double oracle (DO) (McMahan et al., 2003). and counterfactual regret minimization (CFR) (Zinkevich al. 2007). to DRL.
Unfortunately, constructing effective DRL approaches from these tabular algorithms has proven challenging. FP and DO require expensive best response computations at each iteration (i.e., each iteration requires solving an entire reinforcement learning problem) and can exhibit slow conver- gence function of iteration number (Zhang and Sandholm 2024). While CFR converges more quickly, its immediate adaptation to model-free reinforcement learning requires importance weight- ing over trajectories (Lanctot et al. 2009), resulting high variance feedback which function approximation is poorly suited. Moreover, none of FP, DO and CFR enjoy last-iterate convergence, creating layer indirection extract the desired policy.
Recently, Sokota et al. (2023) demonstrated the promise of an alternative algorithm, a policy gradient (PG) method called magnetic mirror descent (MMD). They showed that MMD achieves competitive
Code available at g (tchub.com/aabrffiee
com coton//athannttt. 1ete//1G--0 RI. Benchmar's and https: a-spiel

---

Published as a conference paper at ICLR 2026
performance with CFR in tabular settings, while retaining the deep learning compatibility native to reinforcement learning algorithms.
In this work, expound the relationship between MMD and other generic deep PG methods such as PPO (Schulman et al., 2017). Like MMD, the improvement steps of these other generic PG methods 1) maximize expected value, 2) regularize the policy, and 3) control the size of the update. These parallels lead us to the question: Given that MMD performs well as a DRL algorithm for IIGs, shouldn't these other generic PG methods perform well, too?
We believe that the answer to this question is yes. To further this idea, we put forth the following hypothesis:
With proper tuning, generic PG methods are competitive with or superior to FP-, DO-, and CFR-bused DRI approaches in IIGs.
The confirmation of this hypothesis would have large ramifications both for researchers and practi- tioners. It would foree researchers who currently dismiss approaches like PPO to revisit fundamental beliefs. And, for practitioners, it would justify the use of simpler PG methods, potentially offering both cleaner implementations and improved performance.
However, assessing the above hypothesis is not straightforward. Doing so demands exact ex- ploitability computations for games large enough be representative of settings to which DRL is relevant, but small enough that such computations are efficient. Unfortunately. due to the com- plexity of implementing these computations for large games, few are publicly available, even in OpenSpiel (Lanctot et al. 2019)-11 the standard library for DRL in IIGs. This issue has plagued the field, forcing researchers resort to tenuous metrics and making progress difficult to measure.
We address this issue by implementing exact exploitability computation for five large games: 2DSF Liar's Dice, 3x3 Dark Hex, Phantom Tic-Tac-Toe, 3x3 Abrupt Dark Hex, and Abrupt Phantom Tic- Tac-Toe. These games have millions of infostates tens of thousands more than many games for which exploitability is typically reported, such as Ledue Hold' em (Southey et al., 2005).
Using these benchmarks, we undertake the largest- exploitability comparison of DRL algorithms for IIGs, spanning 7000 total runs across 350 hyperparameter configurations, requiring over 345, $45,000 CPU hours, and including NFSP (Heinrich and Silver 2016), PSRO (Lanctot et al.) 2017). ESCHER [McAleer ct al., 2023), R-NaD (Perolat ct al. 2022), MMD (Sokota et al. 2023), PPO (Schulman et al. 2017) and PPG (Cobbe et al. 2021). Over these runs, all of which included in the paper, NFSP, PSRO, ESCHER, and R-NaD fail to outperform the generic PG methods (MMD, PPO, PPG).
We release implementations for computing exact exploitability and head-to-head values, which require under minutes on commodity hardware, with OpenSpiel-compatible Python bindings, as well as efficient implementations of tabular solvers and best responses. We encourage the commu- nity to use these tools both to test our hypothesis and as benchmarks for future research.
2
PRELIMINARIES
To start, we set context by giving a formalization of IIGs and exploitability, and an overview of model-free DRL approaches to IIGs.
2.1 GAME FORMALISM
IIGs are often formalized as perfect-recall extensive-form games (EFGs) (Kuhn 2003). Our formal- ism is equivalent to perfect-recall EFGs, but possesses greater superficial similarity to traditional reinforcement learning notation. We notate game as a tuple (S, A, 0, T, R. T, O, (max), where is the set of game states; A is the set of actions; O is the set of observations: Il = U, ( x A)* x O A + IR is the reward function for player (and the cost (S) is the transition function: O: S + O is the observa-
is the set of information states; R: function for player 2): T: S x A tion function, which determines the acting player's 's observation; C: + {1, 2} is the player choice function, which determines the acting player; f is the time step at which the game terminates.
The players interact with the game using policies म. : + A(A) for E {1, 2) mapping information states to distributions actions.
2

---

Published as a conference paper at ICLR 2026
The objective of player 1 and loss of player 2 is the expected return J(m) Ex Els (R(S",A)
Directly optimizing the expected return only possible given a particular opponent or distribution of opponents. In practice, identifying this distribution may be prohibitively expensive altogether infeasible-suce a distribution may not even exist.
The standard resolution to this issue is the pursuit of minimax guarantees. The metric associated to minimax guarantees is called exploitability:
expl(T) maxx; [[T T2) minn; I [T1,
2
Exploitability is the expected return of a worst-case opponent, called best response, playing half of the time as player I and half as player 2. Joint policies with exploitability zero are Nash equilibria.
Exploitability is an attractive resolution both because: 1) if exploitability zero is achieved, no oppo- nent can do better than tie in expectation (assuming the player plays each role half of the time); and 2) in complex games, it tends be the case that even achieving low, positive exploitability leads to reliably winning against opponents in practice.
2.2 ALGORITHMIC APPROACHES
Existing model-free DRL approaches to producing low exploitability policies for two-player zero- sum IIGs are largely based on one naive self-play, best responses, CFR, and regularization.
2.2.1 NAIVE SELF-PLAY
Of these approaches, naive self-play is appealing for its simplicity; however, is easy to show that can produce catastrophic results. Some value-based algorithms cannot even express non- deterministic policies. This makes it impossible for them to achieve low-exploitability in many games (e.g., in pock-pape-sscisonne deterministic policies are maximally exploitable). While PG methods can express non deneerrmministcc policies, their learning dynamics generally cycle, diverge or exhibit chaotic behavior, rather than converge to Nash equilibria (Cheung and Piliouras. 2019).
2.2.2 BEST RESPONSES
A next-simplest alternative is to leverage tabular game-theoretic algorithms that rely on best- response computations. These game-theoretic algorithms support plug and play workflow with model-free DRL, by substituting the exact best response for an approximate best response com- puted via DRL. And, unlike naive self-play, they come with convergence guarantees.
Fictitious play (FP) (Brown 1951 Robinson 1951) is one such game-theoretic algorithm. At each iteration 6, FP computes a best response ni the average of the previous iterations #z-1 avg(mı,.... .t- This average is over the sequence form of the policy, meaning that it is equiva- lent policy that uniformly samples T uniform([1, (fl,..,,f
1}} at the start of each game and then plays according to тr until the conclusion of the game. The exploitability of the average म. of sequence of policies generated by FP converges to zero as grows large.
Double oracle (DO) [MMMahan et al.) 2003) is another such game-theoreti algorithm. Like FP, DO works by iteratively computing best responses. However, rather than computing best responses to the average of the previous iterates, DO computes best responses to Nash equilibria or of so-called metagame. This metagame is a one-shot game in which players select policies 7(1), (2) from among the previous best responses { } and receive as payoff the expected value (=,(1((()))) of playing these policies against one another. A policy this metagame is a mixture Te previous best responses { 1 } that is enacted by sampling лт at once at the start each game, then playing according to गт until the conclusion of the game. A policy of this metagame d is unexploitable if none of the previous best responses [MI,..., TT-1) ) exploits it. The exploitability sequence of metagame Nash equilibria guaranteed to converge to zero as t grows large.
Unfortunately, both FP and DO can suffer from slow convergence. Requiring a large number of iterations is especially concerning in the context of DRL because each iteration requires solving
3

---

Published as a conference paper at ICLR 2026
an entire reinforcement learning problem. Despite numerous works DRL variants of both FP Qin et al.
(Heinrich and Silver
2016
2019; Vinyals et al.]
[219 Goldstein and Brown 2023: Hu
et al. Li et al.
2023a) and DO Lanctot al.] 2017 McAleer et al.] 2020 2021 Smith al.
2021
McAleer et al.
2022a Dinh et al.
2022 Zhou et al.
2022 Yao et al. 2023 Tang et al. 2023
McAleer et al. 2024
Li el al.
[222aa] Lian 2024), it is not clear that this issue has been addressed.
2.2.3
COUNTERFACTUAL REGRET MINIMIZATION
Counterfactual regret minimization (CFR) Zinkevich et al. 2007), widely regarded as the gold stan- dard for tabularly solving IIGs, generally converges faster than FP DO. Instead of relying on best responses, CFR is grounded in the principle of regret minimization. A key result of regret minimiza- tion that, in two-player zero-sum games, the average strategy of two regret minimizing learners converges to Nash equilibrium. CFR applies this principle by independently minimizing regret at each information state. By appropriately weighting feedback, CFR ensures regret minimization Nash equilibrium.
across the entire game, thus guaranteeing convergence
Because CFR is more involved than FP and DO, it does not admit the same plug and play workflow with single-agent model-free DRL. Instead, new methods (Steinberger et al. 2020 Gruslys et al. 2020 McAleer et al.) 2023) were designed specifically for CFR. Unfortunately, these new meth- ods come with weaknesses. The most straightforward method (Steinberger et al., (020), for instance, relies on importance sampling over trajectories, which leads to high variance in feedback, especially in games with many steps. This makes it challenging for function approximators to learn. Although McAleer et al. (2023) resolve this issue, their resolution necessitates a uniform behavioral policy, making it difficult to achieve sufficient coverage of relevant trajectories.
2.2.4 REGULARIZATION
Motivated by the limitations of FP-, DO-, and CFR-based methods, Perolatt et al. (2021) proposed an alternative approach centered around regularization. Unlike previous methods, their approach designed to converge in the last iterate, aligning it more closely with typical DRL techniques, Perolat
ctt al. (202T)'s work has been influential, serving as inspiration for several subsequent studies (Fu
al. 2022; Perolat al. 2022, Sokota et al.) 2023] Meng et al. 2023 Liu et al., 2024). Among
them, Sokota et al. (2023) introduced a policy gradient algorithm called magnet mirror descent
(MMD), characterized by the following update rule:
T++ argmax E ((A) aKL(T, p) 1 KL(, Tt), A~-
"
where ((A) is the value of action A, a is regularization temperature, is stepsize, p is a magnet policy, and is KL divergence. implemented as a standard tabular self-play reinforcement learning algorithm. achieves performance competitive with that of CFR in IIGs. (Sokota et al. (2023) also gave empirical evidence that MMD is performant as DRI. algorithm.
surprising result, Sooooa al., (2023) demonstrated that MMD,
3 THE POLICY GRADIENT HYPOTHESIS
The surprising performance of magnetic mirror descent invites reflection. Consider the functions served by the components of Equation
(A maximizes the expected return.
-aKL(T, p) regularizes the updated policy (for uniform this acts similarly to an entropy bonus).
_KLL m.) constrains the update size.
Far from being unique to MMD, these functions have long been present in deep PG methods, such as TRPO (Schulman et al. 2015), PPO (Schulman et al., 2017), and PPG (Cobbe et al. [002]). As with MMD, the update rules of these algorithms include terms that maximize expected value and encourage entropy; they also possess mechanisms that, by various means, control the update size.
Yet, despite this shared ethos, other generic PG methods-typiceee PPO-appear appear in literature only as underperforming baselines. This raises the question: Why has MMD performed well, while these other generic PG methods have not?
PPO
4

---

Published as a conference paper at ICLR 2026
There are multiple answers worth considering. One possibility is that the differences between MMD and other generic PG methods are material. Unlike most PG methods, which use some combination of gradient clipping and forward KL divergence. MMD employs reverse KL divergence to regulate update size. In tabular settings, where competitive performance requires high precision, this dis- tinction is manifestly important. However, the context of deep learning, it is less clear that this
variation matters (Engstrom et al., 2020 Hsu et al. 2020; Chan et al. 2022).
A second possibility is that these generic PG methods are fully capable of performing well, but have not been run with good hyperparameters. Ever-present reasons that baselines tend to perform poorly, such as the expense and difficulty of tuning and the structural disincentives to reporting strong results for baselines, could be at work.
But there are also domain-speciffe factors that make this second possibility particularly plausible:
There is putative belief that generic PG methods do not work in IIGs and that more game theo- retically involved algorithms are required. Due to this belief, tuning may have been confounded by confirmation bias.
Works that report generic PG method baselines often rely head-to-heade results. The intransitive nature of these results increases the complexity of tuning and may thereby have hampered it.
The hyperparameter regime for which (Sokota et al.) (2023) showed MMD to effective involves more entropy regularization than is typically used for PG methods in single-agent problems. Thus, it is possible effective hyperparameter configurations were overlooked due to their regime being too far removed from those of single-agent settings.
Based on these reasons, this work posits that generic PG methods are fully capable of performing well, a position we formalize under the following hypothesis.
The Policy Gradient Hypothesis. Appropriately luned policy grudient methods that share an ethos with magnetic mirror descent are competitive with or superior model-free deep reinforcement learning approaches based on fictitious play, double oracle, or counterfactual regret minimization in two-player zero-sum imperfectiinformaiion games.
The confirmation of the policy gradient hypothesis would have ramifications on both research and practice. On the research side, would vindicate generic PG methods, which have generally either been dismissed as unsound or relegated to the role of sacrificial baseline. Practically, it would lead to simpler implementations and performance improvements in applications where FP-, DO-, and CFR- based DRL algorithms are currently employed. These applications include autonomous vehicles (Ma et al. 2018),, network security (Lu et al. 2020 Xue et al.) 2021), robot confrontation (Hu al. eavesdropping (Guo et al. (2022). 2, radar (anti-)jamminn
2022).
eLi al. 2022a by Geng et al. 2023
2024
intrusion response (Hammar and Stadler 2023), acrial combat (He et al. 2023 Wang al.
2024
pursuit-evasion games (Li et al.) [2023b] 2024b). racing (Zheng et al. 2024). language model
alignment (Gemp et al. 2024 and red teamingg (Ma et 2024).
BENCHMARKS
Unfortunately, evaluating DRL algorithms for adversarial IIGs is not well standardized. Due to the difficulty of efficiently implementing exploitability for large IIGs, there is dearth of accessible implementations.
As result, the literature predominantly resorts to some combination of two unsatisfactory metrics. The first is exploitability in the same small EFGs that served as benchmarks for tabular solvers, such as Ledue Hold' em (Southey et al. 2005) (see Table 4. Benchmarking in these small games is un- satisfactory because success requires learning precise near-Nash policy for handful of repeatedly revisited information states fundamentally different challenge from large games, where success requires learning a policy that is strong (but not necessarily numerically close to Nash) for a vast number of unvisited information states.
The second is head-to soad-to-hedd evaluations in large games. These evaluations are unsatisfactory because the intransitive dynamics characteristic of imperfecttinformationn games (e.g., reck-beats-seissors-
5

---

Published as a conference paper at ICLR 2026
petts--pppp-----eeeeee muddle interpretations of relative strength. Exacerbating the issue, there is no community coordination on head-to-head opponents, precluding direct wross-work comparisons.
This predicament has created undesirable state of affairs. Many works claim "state-of-the-ad performance, but even experts in specific algorithm classes are often unsure which instances are actually effective. To address this predicament, five large games: 2D5F Liar's Dice (LD2D5F), Phantom Tic-Tac- Toe (PTTT), 3x3 Dark Hex (DH3), Abrupt Phantom Tic-Tac-Toe (APTTT), 3x3 Abrupt Dark Hex (ADH3).
out to implement efficient computations for
In 2D5F Liar's Dice, each of two players privately rolls two dice with five faces. Players then alternate making claims about the total number of dice across both players that show given face value (e.g., "two dice showing four"). A legal bid (in the ruleset OpenSpiel adopts) must either increase the quantity or increase the face value while not decreasing the quantity. On a turn, instead of bidding, a player may also call "liar," whereupon all dice are revealed; if the last bid is no greater than the revealed count of the specified face, the the bidder wins, otherwise the caller does.
In Tic- Tic-Tac-Toe Toe, which is played on 3-by- grid with square cells, the objective to get three-in-a-row horizontally, vertically, or diagonally. If the board filled without ei- ther player having three-in-a-row, the game ends draw. An example game is shown in Figure [ (left) in which the first moving player (red) wins via the diagonal.
Figure 1: Tic-Tac-Toe (left) and 3x3 Hex (right).
Table I: Game quantities. Positive Nash values (i.c., expected values for player at Nash equilibria) mean that player I has a structural advantage.
In 3x3 Hex, which is played bus with hexagonal cells, the first moving player's objective is connect the top edge of the board to bottom edge, while the sec- ond moving player's objective is to connect the left edge of the board to the right edge. Draws are not possible because the board provably cannot be filled without a player winning (Gale) 1979. Hex Theorem), An ex- ample game is shown in Figure [ (right) in which the first moving player (red) wins by connecting the top and bottom edges.
rhom-
Game
States
Info. states
Nash value
LD2D5F
235.93M
15.73M
0.01770 + 1.8e
DH3
19.12B
6.07M
1.00000 + 0.0
ADH3
29.31B
27.33M
0.38443 2.1e -4
PTTT
19.93B
5.99M
0.66665 + 2.6c
APTTT
27.12B
23.31M
0.55017 1.4e
In the dark and phantom variants these games, actions are concealed from the non-acting player, creating imperfect information. These inperfect-information variants have two rulesets classical and abrupt- which differ in how they handle the situation in which the acting player attempts to place piece cell occupied by their opponent's 's piece. In classical, the acting player must select a different cell. In abrupt, the acting player loses their turn.
These games have at least three significant strengths benchmarks:
1. OpenSpiel (Lanctot al. 2019) support: LD2D5F, DH3, PTTT, ADH3 already implemented in OpenSpiel, making it easy for other researchers to use them. To address APTTT's absence, we provide our own game implementation.
2. Precedent: Existing work uses LD, PTTT, and ADH3 for head-to-hear evaluations (Sokota et al,, 2023 McAleer et al. 2023 Meng et al. 2023 Liu et al. 2024), providing external evidence of their relevance.
3. Size:
These games allow model-free DRL algorithms to be trained in minutes or hours on com- modity hardware. But, as detailed in Table they have millions of infostates, making them quite large by the standards of game solving benchmarks.
Indeed, the size of these games makes it nontrivial to implement exploitability computations. To ac- commodate their magnitude, we use sequence- form representations, which are approximately mul- tiple orders of magnitude more compact than the explicit game trees, alongside dynamic payoff matrix generation to circumvent explicit matrix storage. This memory-efficiett approach enables usage commodity hardware.

---

Published as a conference paper at ICLR 2026
To optimize the runtime of LD2D5F, exploit the facts that actions are public and chance outcomes are independent and located at the root of the game to use dynamic programming to calculate the payoff at all public states in linear time, similarly to Brown et al. (2020). To optimize the runtime of the 3x3 games, we distribute computation across multiple threads after the first two moves, using 18 buffers prevent concurrency conflicts across the 81 possible opening sequences. Depending the machine and game, this implementation delivers exploitability computation (and head-to-head evaluation) times of roughly 30 to 90 seconds.
5 EXPERIMENTS
While our nigh-performanee exploitability computations open the door to evaluating model-free DRI. for IIGs more rigorously, the novelty of these benchmarks necessitates particular care in ex- perimental design and interpretation. Simply showing that tuned PG methods perform "well" on these benchmarks would not suffice as evidence for the policy gradient hypothesis] as there are no numbers for FP-, DO-, or CFR-based DRL algorithms against which to compare. The obvious securrs--wiihe we take-is to report our own results for FP, DO, and -FRe DRL algorithms. However, since we have hypothesized these algorithms will underperform, should be carefully scrutinized. To aid readers in this endeavor, we provide detailed documentation of algorithm implementations, hyperparameter tuning procedure, and hyperparameter tuning results, in addition to final results.
role in tuning them
5.1 AL.GORITHMS AND IMPLEMENTATION
We choose algorithms to evaluate using two criteria. First, that the collective set of algorithms in- cludes representatives from FP-based algorithms, DO-based algorithms, CFR-based algorithms, and generic PG methods, as make experiments sufficiently comprehensive provide evidence for or against the policy gradient hypothesis Second, that algorithms be implemented by a reliable external source in fashion requiring little adaptation possible for OpenSpiel (Lanctot al. 2019) compatibility, so as to reduce the probability of an implementation error.
In accordance with these criteria, the algorithms
select are NFSP (Heinrich and Silver 2016).
PSRO ([anctot et al. 2017). ESCHER (McAleer et al. (203)) R-NaD (Perolat et al., 2022), ) PPO (Schulman al. 2017). 7). PPG (Cobbe et al. 2021 1}, and MMD (Sokota et al. 2023). This selection includes one Fa-based algorithm (NFSP), one DO-based algorithm (PSRO), one CFR-based algo- rithm (ESCHER), three generic PG methods (PPO, PPG, MMD), and one non-standard PG method (R-NaD) inspired by Perolat et al. (202I). Algorithm implementations were sourced from Open- Spiel (Lanctot et al.) 2019), except PPG and ESCHER, which we sourced from CleanRL (Huang et al. 2022) and the official ESCHER repository (McAleer et al. 2023). respectively.
Of the algorithm implementation code that we wrote or modified, some of the larger poten- tial sources of error include moving NFSP from TensorFlow (Abadi et al.) 2015) to PyTorch (Paszke et al. 2019), adding multi-agent support for PPO and PPG, and implementing MMD as modification of PPO. To corroborate the correctness of these pieces of code, we repro- duce the NESP exploitability results for Leduc Hold' em from OpenSpiel, as detailed Ap- pendix D.I. and the MMD approximate exploitability results for ADH3 and PTTT from Sokktal [.2] We enable external verification of the correctness of these pieces of code, and the rest of our algorithm implementations, by open sourcing our codebase at felgithub.co com
et al. (2023), as detailed in Appendix
RL- Benchmark
5.2 DESIGN
We design our training runs as series of two launches: hyperparameter tuning launch followed by a maximization -ias-ffree evaluation launch.
For the hyperparameter tuning launch, for each combination of the 5 games and 7 algorithms, we test 50 hyperparameter configurations for 10 million steps over 3 seeds. To create these configura- tions, modify the default positive real-valued hyperparameters by independent randomly sam- pled powers of multiplying for the vast majority of hyperparameters, but exponentiating for some hyperparameters upper bounded by 1. We take the default values for hyperparameters from
7

---

Published as a conference paper at ICLR 2026
2030 Liars Doce (LD22235)
3ed Abrupt Lix (AD83)
3x3 Dark lox (DM3)
Abngt Plazicm To-Tac-Tor
Hoxton Tie-lar-Ste
(APTIT)
(PTTT)
PPO
NPSP
PSRO
MAD
Ridaa
PSRO
390
Ny
Rolla
NPSP
Ridaa
Neop
PSRd
Exsliiabille policy
Figure 2: Exploitability results. For each combination of game and algorithm, the box-and-whisker pair depicts the distribution of final exploitability over the runs from the hyperparameter tuning launch (left) and evaluation launch (right) with square-root y-axis scale. R-NaD, NFSP, ESCHER, and PSRO failed to outperform generic PG methods (MMD, PPO, PPG).
the implementation source in almost all cases. The most notable exceptions to this are the network architecture and optimizer, for which we impose 3-layer 512-hidden unit fully connected network and Adam (Kingma and Bal 2015) on every algorithm, and the entropy coefficient, for which we apply the value from Sokota et al. (2023) to all generic PG methods. We provide comprehensive list of hyperparameters, default values, and links to the specific lines of code of the external sources from which those default values were taken in Appendix G
For the evaluation launch, we run the 5 strongest hyperparameter configurations from the first launch (measured by lowest average final exploitability), with 10 fresh seeds, for million steps, for each combination of the 5 games and 7 algorithms. From these evaluation runs, report both ploitabilities and head-to-head comparisons. For the head-to-head comparisons, we select the policy with the lower median final exploitability over the 10 seeds for each hyperparameter configuration, as well as an approximate Nash equilibrium policy computed using discounted CFR (Brown and Sandholm 2019a), and compare these selected policies for each pair of algorithms for each game.
5.3 RESULTS
We summarize two the main results: exploitability performance and read-to-head performance.
Exploitability Figure plots the distribution of final exploitabilities for each game and algorithm in pairs of box-and- plots for the hyperparameter tuning launch (left) and the evaluation launch (right). Over these runs, ESCHER was uniformly noncompetitive. PSRO was largely non- competitive, showing poor performance in ADH3, PTTT, and APTTT and having only sporadic success in DH3, for which several hyperparameter configurations found Nash equilibria on some seeds but not others. NFSP and R-NaD approached matched the performance of generic PG methods in select cases, but typically underperformed them. The generic PG methods performed the strongest, and were roughly on par with one another.
Head-to-Head Comparisons Figure plots head-to-han -eadd comparisons for each pair of algo- rithms. Relative performance patterns are similar to those for exploitability: The generic PG meth- ods matched or defeated other algorithms, while maintaining rough parity among themselves; R- NaD either approached the performance of the generic PG methods underperformed them, while NFSP and, especially, ESCHER and PSRO were noncompetitive.
6 DISCUSSION
The experimental results unambiguously support the policy gradient hypothesis) Not only did the FP-, DO-, and CFR-based DRL algorithms fail to outperform the generic PG methods, they, along- side the nonstandard PG method, were largely noncompetitive.
8

---

Published as a conference paper at ICLR 2026
2000 Laua's Misa Sier SLARSSSP
XA2 Abrigt Dack EACANEL
Alrigh
PARGOCIO
0.00
0.4
9.23
0:00
0:22
0125
8:10
CLAS
9913
0,22
022
CURS
Figure 3: Head-to-head evaluations. The number in each cell is the expected return of the row algorithm against the column algorithm when each plays half of the games the first moving player. R-NaD, NESP, ESCHER, and PSRO failed to outperform generic PG methods (MMD, PPO, PPG), which are segregated by the dashed red lines.
However, we emphasize that we make no claim-and should not be interpreted as having claimed- that we have established truefthe policy gradient hypothesis pr any other speculation discussed in this work. A prudent interpretation of our experiments gives consideration to multiple limitations.
First, we only performed experiments on five games, four of which are somewhat homogeneous. We do not believe that the relative performance of the algorithms we evaluated would be universally consistent across adversarial IIGs at large (indeed, it is not even consistent across these five games).
Second, we evaluated only one FP-based algorithm (the original NESP), one DO-based algorithm (the original PSRO), and one CFR-based algorithm (the most recent adaption to DRL), As discussed in Section 2.2 there are dozens of variants of these algorithms, especially PSRO. Any one of these variants may possess important innovations that result in improved performance on our benchmarks.
Third, the experimental results reflect our particular experimental conditions, including: 1) the de- fault hyperparameter values, 2) the hyperparameters we tuned, 3) the manner in which we tuned hyperparameters, and 4) performance evaluation at 10 million steps. These conditions are indeed particular, and should not be misconstrued to represent optimal algorithm performance. is possible that strong performing hyperparameters exist for NFSP, PSRO, ESCHER, or R-NaD, but were not found by our hyperparameter sweep because they are too many orders of magnitude away from the defaults, or because they require too many different defaults to be changed simultaneously, or be- cause they involve different categorical hyperparameters- which we did not tune. It is also possible that the hyperparameters we used would achieve strong performance, given more training steps- possibility consistent with preliminary experiments, which suggested that all algorithms continue to decrease exploitability beyond 10 million steps.
To power community efforts to address or investigate these second and third sets of limitations, as well to give creators of existing and future DRL algorithms for IIGs the opportunity demon- strate the efficacy of their algorithms, we release our exploitability computations in a Python pack- age with OpenSpiel-compatible bindings at github..hhho.o com/ waaarfarinn/eete aapp-a-sppi ip e We detail this package, called exp-a- a- a-spiel, 1. in Appendix A pertains the policy gradient hypothesis] but also more generally for future IIGs research.
We encourage its use not only as it
7 CONCLUSION
This work hypothesizes and presents evidence that generic deep PG methods are strong approaches for adversarial IIGs. Our results, along with those of Yu et al. (2022), who demonstrated PPO's effectiveness in cooperative IIGs, Sokota et al. (2023). who demonstrated MMD's D's effectiveness in adversarial IIGs, add to growing body of evidence for the possibility that single, simple PG method could serve as a universal algorithm for DRL in games. We believe this possibility is an aspiration worth pursuing.

---

Published as a conference paper at ICLR 2026
8 ACKNOWLEDGMENTS
This work was supported by Award #2125858 NRT-AI: Convergent, Responsible, and Ethical Arti- ficial Intelligence Training Experience for Roboticists; NSF #2340651; NSF #2402650; DARPA #HRO01 12490431; ARO W911NF-24-1-0195, the NYU IT High-Performance Computing re- sources, services, and staff expertise; and ONR grant #N000142212122
We thank Jeremy Cohen and Alexander Robey for helpful feedback.
REFERENCES
George W. Brown. Iterative solution of games by fictitious play. In Activity Analysis of Production and Allocation. 1951.
Julia Jean Robinson, An iterative method of solving a games. Classics Game Theory, 1951.
H. Brendan McMahan, Geoffrey Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by an adversary. In International Conference Machine Learning (ICML), 2003.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In Neural Information Processing Systems (NIPS), 2007.
Brian Hu Zhang and Tuomas Sandholm. Exponential lower bounds the double oracle algorithm in zero-sum games. In International Joint Conference on Artificial Intelligence (IJCAI), 2024.
Mare Lanctot, Kevin Waugh, Martin Zinkevich, and Michael Bowling. Monte carlo sampling for regret minimization in extensive games. In Neural Information Processing Systems (NIPS), 2009.
Samuel Sokota, Ryan D'Orazio, J. Zico Kolter, Nicolas Loizou, Marc Lanctot, Ioannis Mitliagkas, Noam Brown, and Christian Kroer. A unified approach to reinforcement learning, quantal re- sponse equilibria, and two-player zero-sum games. In International Conference on Learning Representations (ICLR), 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Mare Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers. Karl Tuyls, Shayegan Omidshafici. Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis. Open- Spiel: framework for reinforcement learning in games. 2019.
Finnegan Southey, Michael Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, and Chris Rayner. Bayes' bluff: opponent modelling in poker. In Conference on Uncertainty in Artificial Intelligence (UAI), 2005.
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect- information games, 2016.
Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pero- lat, David Silver, and Thore Graepel. A unified sameetheoretic approach to multiagent reinforce- ment learning. In Neural Information Processing Systems (NIPS), 2017.
Stephen McAleer, Gabriele Farina, Mare Lanctot, and Tuomas Sandholm. ESCHER: Eschewing importance sampling in games by computing history value function to estimate regret. In Con- ference on Learning Representations (ICLR), 2023.
Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T. Connor, Neil Burch, Thomas Anthony, Stephen McAleer, Romuald Elie, Sarah H. Cen, Zhe Wang, Audrunas Gruslys, Aleksandra Malysheva, Mina Khan, Sherjil Ozair, Finbarr Timbers, Toby Pohlen, Tom Eccles, Mark Rowland, Mare Lanctot, Jean-Baptiss Lespiau,
10

---

Published as a conference paper at ICLR 2026
Bilal Piot, Shayegan Omidshafiei, Edward Lockhart, Laurent Sifre, Nathalie Beauguerlange, Remi Munos, David Silver, Satinder Singh, Demis Hassabis, and Karl Tuyls. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 2022.
Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In Interna- tional Conference on Machine Learning (ICML), pages 2020-2027. PMLR, 2021.
Harold William Kuhn. Lectures on the theory of games. 2003.
Yun Kuen Cheung and Georgios Piliouras. Vortices instead of equilibria in minmax optimization: Chaos and butterfly effects of online learning in zero-sum games. In Conference on Learning Theory (COLT), 2019.
Rong-Jun Qin, Jing-Cheng Pang, and Yang Yu. Improving fictitious play reinforcement learning with expanding models, 2019.
Oriol Vinyals, Igor Babuschkin, Wojeiech Marian Czameeki, al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 2019.
Maxwell Goldstein and Noam Brown. AdaptFSP: Adaptive fictitious self play, 2023.
Chaohao Hu, Yumlong Cai, Weidong Li, and Hongfei Li. Fictitious ellf reinforcement learning with expanding value estimation. International Conference on Robotics, Intelligent Control and Artificial Intelligence (RICAI), 2023.
Huale Li, Shuhan Qi, Jiajia Zhang, Dandan Zhang, Lin Yao, Xuan Wang, Qi Li, and Jing Xiao. Nfsp-plt: Solving games with a weighted nfsp-per- method. Electronics, 2023a.
Stephen McAlcer, JB Lanier, Roy Fox, and Pierre Baldi. Pipeline psro: A scalable approach for finding approximate nash equilibria in large games. In Neural Information Processing Systems (NeurIPS), 2020.
Stephen McAleer, JB Lanier, Kevin A Wang, Pierre Baldi, and Roy Fox. Xdo: double oracle algorithm for extensive-form games. In Neural Information Processing Systems (NeurIPS), 2021.
Max Smith, Thomas Anthony, and Michael Wellman. Iterative empirical game solving via single policy best response. In International Conference on Learning Representations (ICLR), 2021.
Stephen McAleer, Kevin Wang, John Lanier, Mare Lanctot, Pierre Baldi, Tuomas Sandholm, and Roy Fox. Anytime psro for two-player zero-sum games, 2022a.
Le Cong Dinh, Stephen McAleer, Zheng Tian, Nicolas Perez-Nieves, Oliver Slumbers, David Henry Mguni, Jun Wang, Haitham Bou Ammar, and Yaodong Yang. Online double oracle. Transactions on Machine Learning Research (TMLR), 2022.
Ming Zhou, Jingxiao Chen, Ying Wen, Weinan Zhang, Yaodong Yang, Yong Yu, and Jun Wang. Efficient policy space response oracles, 2022.
Jian Yao, Weiming Liu, Haobo Fu, Yaodong Yang, Stephen McAleer, Qiang Fu, and Wei Yang. Policy space diversity for non-transitive games. In Neural Information Processing Systems (NeurIPS), 2023.
Xiaohang Tang, Le Cong Dinh, Stephen McAleer, and Yaodong Yang. Regret-minimizing double oracle for extensive-form games. In International Conference on Machine Learning (ICML), 2023.
Stephen McAleer, JB Lanier, Kevin A. Wang, Pierre Baldi, Tuomas Sandholm, and Roy Fox. Toward optimal policy population growth in two-player zero-sum games. In International Conference on Learning Representations (ICLR), 2024.
Pengdeng Li, Shuxin Li, Chang Yang. Xinrun Wang, Xiao Huang, Hau Chan, and Bo An. Self- adaptive psro: Towards an automatic population- game solver. In International Joint Con- ference Artificial Intelligence (IJCAI), 2024a.
Jiesong Lian. Fusion-PSRO: Nash policy fusion for policy space response oracles, 2024.
11

---

Published as a conference paper at ICLR 2026
Eric Steinberger, Adam Lerer, and Noam Brown. Dream: Deep regret minimization with advantage baselines and model-free learning. 2020.
Audrūnas Gruslys, Marc Lanctot, Rémi Munos, Finbarr Timbers, Martin Schmid, Julien Perolat, Dustin Morrill, Vinicius Zambaldi, Jean-Baptiste Lespiau, John Schultz, Mohammad Gheshlaghi Azar, Michael Bowling, and Karl Tuyls. The advantage regret- pegretgmatching actor-critic, 2020.
Julien Perolat, Remi Munos, Jean-Baptiste Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, Georgios Piliouras, Marc Lanctot, and Karl Tuyls. From poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization. In International Conference on Machine Learning (ICML), 2021.
Haobo Fu, Weiming Liu, Shuang Wu, Yijia Wang. Tao Yang, Kai Li, Junliang Xing, Bin Li, Bo Ma, Qiang Fu, and Yang Wei. Actor-critic policy optimization in large-scale imperfect-information game. In International Conference on Learning Representations (ICLR), 2022.
Linjian Meng, Zhenxing Ge, Pinzhuo Tian, Bo An, and Yang Gao. An efficient deep reinforcement learning algorithm for solving imperfect information extensive-form games. AAAI Conference on Artificial Intelligence (AAAI), 2023.
Zongkai Liu, Chaohao Hu, Chao Yu, and Peng Sun. Regularization is enough for last-iterate con- vergence in zero-sum games, 2024.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), 2015.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations (ICLR), 2020.
Chloe Ching-Yuu Yun Hsu, Celestine Mendler-Dünner, and Moritz Hardt. Revisiting design choices in proximal policy optimization. 2020.
Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl diver- gences. Journal of Machine Learning Research (JMLR), 2022.
Xiaobai Ma, Katherine Driggs-Campbell and Mykel J. Kochenderfer. Improved robustness and safety for autonomous vehicle control with adversarial reinforcement learning. In IEEE Intelligent Vehicles Symposium (IV), 2018.
Ziyue Lu, Guoming Tang, Baochao Chen, Bangbang Ren, Sheng Chen, and Deke Guo. Ai-aided game: Enhancing the defense performance of scale-free network via deep reinforcement learning- In 2020 IEEE 22nd International Conference High Performance Computing and Communica- tions; IEEE 18th International Conference on Smart City; IEEE 6th International Conference on Data Science and Systems HHPCC//SarCCty/DDSS) 2020.
Wanqi Xue, Youzhi Zhang, Shuxin Li. Xinrun Wang. Bo An, and Chai Kiat Yeo. Solving large- scale extensive-fore network security games via neural fictitious self-play. In International Joint Conference on Artificial Intelligence (ICAI), 2021.
Mingxi Hu, Siyu Xia, Chenheng Zhang, and Xian Guo. Robot confrontation based on policy-space response oracles. In International Conference CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), 2022.
Delin Guo, Lan Tang, Lvxi Yang. and Ying- Chang Liang. Eavesdropping game based on multi- agent deep reinforcement learning. In International Workshop on Signal Processing Advances in Wireless Communication (SPAWC), 2022.
Kang Li, Bo Jiu, Wenqiang Pu, Hongwei Liu, and Xiaojun Peng. Neural fictitious self-play for radar antijamming dynamic game with imperfect information. IEEE Transactions on Aerospace and Electronic Systems, 2022a.
12

---

Published as a conference paper at ICLR 2026
Huayue Li, Zhaowei Han, Wenqiang Pu, Liangqi Liu, Kang Li, and Bo Jiu. Counterfactual regret minimization for anti- jammiig game of frequency agile radar. In Sensor Array and Multichannel Signal Processing Workshop (SAM), 2022b.
Jie Geng, Bo Jiu, Kang Li, Yu Zhao, Hongwei Liu, and Hailin Li. Radar and jammer intelligent game under jamming power dynamic allocation. Remote Sensing, 2023.
Jie Geng, Bo Jiu, Kang Li, Yu Zhao, Chao Wang, and Hongwei Liu. Multiagent reinforcement learning for antijamming game of frequency-agile radar. IEEE Geoscience and Remote Sensing Letters, 2024.
Kim Hammar and Rolf Stadler. Scalable learning of intrusion response through recursive decom- position. In Jie Fu, Tomas Kroupa, and Yezekael Hayel, editors, Decision and Game Theory for Security, 2023.
Shaoqin He, Yang Gao, Baofeng Zhang, Hui Chang, and Xinchen Zhang. Advancing air combat tactics with improved neural fictitious self-play reinforcement learning. In De-Shuang Huang, Prashan Premaratne, Baohua Jin, Boyang Qu, Kang-Hyun Jo, and Abir Hussain, editors, Ad- vanced Intelligent Computing Technology and Applications, 2023.
Dinghan Wang, Longmeng Ji, Jingbo Wang. Zhuoyong Shi, Jiandong Zhang, Qiming Yang, Guo- qing Shi, Yong Wu, Yan Zhu, and Jinwen Hu. Dogfight advantage occupancy method based on imperfect information self-play. In International Conference Control & Automation (ICCA), 2024.
Shuxin Li, Xinrun Wang, Youzhi Zhang, Wanqi Xue, Jakub Černý, and Bo An. Solving large-scale pursuit-evasion games using pre-trained strategies. AAAI Conference on Artificial Intelligence (AAAI), 2023b.
Pengdeng Li, Shuxin Li, Xinrun Wang, Jakub Cerný, Youzhi Zhang, Stephen McAleer, Hau Chan, and Bo An. Grasper: A generalist pursuer for pursuit-evasion problems. In International Confer- on Autonomous Agents and Multiagent Systems (AAMAS), 2024b.
Hongrui Zheng, Zhijun Zhuang, Stephanie Wu, Shuo Yang, and Rahul Mangharam. Bridging the gap between discrete agent strategies in game theory and continuous motion planning in dynamic environments. 2024.
Ian Gemp, Roma Patel, Yoram Bachrach, Mare Lanetot, Vibhavari Dasagi, Luke Marris, Georgios Piliouras, Siqi Liu, and Karl Tuyls. Steering language models with game-theoretic solvers. In Agentic Markets Workshop at ICML, 2024.
Chengdong Ma, Ziran Yang, Hai Ci, Jun Gao, Minquan Gao, Xuehai Pan, and Yaodong Yang. Evolving diverse red-team language models in multi-round multi-agent games, 2024.
David Gale. The game of hex and the brouwer fixed-point theorem. The American mathematical monthly, 1979.
Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong, Combining deep reinforcement learning and search for imperfect-informattee games. Advances in neural information processing systems, 33:1 7057-17069 2020.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research (JMLR), 2022.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fermanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg. Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015.
13

---

Published as a conference paper at ICLR 2026
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vitito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems (NeurIPS). 2019.
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Noam Brown and Tuomas Sandholm. Solving minimization. AAAI Conference on Artificial Intelligence (AAAI), 2019a.
information games via discounted regret
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang. Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO cooperative multi-agent games. In Neural Information Processing Systems Datasets (NeurIPS) and Benchmarks Track, 2022.
I. Romanovskii. Reduction of game with complete memory to a matrix game. Efficient computa- fion of equilibria for extensive two-person games, 1962.
Bernhard von Stengel. Efficient computation of behavior strategies. Games and Economic Behavior, 1996.
Daphne Koller, Nimrod Megiddo, and Bernhard von Stengel. Efficient computation of equilibria for extensive two-person games. Games and Economic Behavior, 1996.
Samid Hoda, Andrew Gilpin, Javier Pena, and Tuomas Sandholm. Smoothing techniques for com- puting nash equilibria of sequential games. Mathematics of Operations Research, 2010.
Oskari Tammelin. Solving large imperfect information games using CFR+. 2014.
Noam Brown and Tuomas Sandholm. Solving imperfect-infformaioo games via discounted regret minimization. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019b.
Gabriele Farina, Christian Kroer, Noam Brown, and Tuomas Sandholm. Stable-predictive optimistic counterfactual regret minimization. International conference on machine learning (ICML), pages 1853-1866 PMLR, 2019.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Mille and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
Stephen MeAleer, JB Lanier, Kevin Wang, Pierre Baldi, Roy Fox, and Tuomas Sandholm. Self- play psro: Toward optimal populations in two-player zero-sum games, 2022b. URL https: //axxiv..rg/abs//2000
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Rad- ford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines, 2018.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg. Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. International Conference on Machine Learning (ICML), 2018
Joseph Suarez, Pufferlib: Making reinforcement learning libraries and environments play nice, 2024.
Denys Makoviichuk and Viktor Makoviychuk. rl-games: A high-performance framework for rein- forcement learning, 2021.
Jayy Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research (JMLR), 2022.
14

---

Published as a conference paper at ICLR 2026
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dumning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018.
Daniel Hennes, Dustin Morrill, Shayegan Omidshafiei, Remi Munos, Julien Perolat, Marc Lanctot, Audrunas Gruslys, Jean-Baptise Lespiau. Paavo Parmas, Edgar Duenez-Guzmanl al. Neural replicator dynamics. arXiv preprint 1xxi::1906.00190, 2019.
15

---

Published as a conference paper at ICLR 2026
A THE EXP-R-SPIEE PACKAGE
We release a Python library to efficiently compute exploitability of policies for the five benchmark games introduced in Section 4 The library supports outputting state representations compatible with OpenSpiel (Lanctot et al. 2019), making it straightforward to evaluate the exploitability of policies trained using the latter library.
A.1 EXPLOITABILITY COMPUTATION
Internally, exp-a-spiel evaluates the exploitability of policies by carrying out computations us- ing the so-called sequence-form representation of the five games (Romanovskii 1962; von Stengel, 1996 Koller et al. 1996). This representation is roughly 1000 times more compact than the actual game trees, allowing us to compute exact exploitability (and head-to-head values) on commodity hardware. This representation is comprised of two objects: (i) the treeplexes X, of the two play- ers (Hoda et al.) 2010), and (ii) the sequence-form payoff matrix A of the game. Every policy for player admits an equivalent vector x X' (resp. corresponding to player can be computed closed form via the bilinear form the size of the games, the sequence-fone payoff matrix A is not stored explicitly in memory by asppel but is rather materialized on the fly in a multi-threade
y for player 2), and the expected payoff
æ Ay. Due to
fashion as needed.
At high level, given two policies Tı, T2 for the players, exp a a-spiel computes exploitability by performing the following.
1. First, πi and T2 converted into their sequence-form equivalents E X. Y. This step requires memory and runtime that scale linearly in the number of information states of the games. The latter is in the order of 10" as shown in Table
2. Then, the vectors 91 = Ay, O2 == -AT x are computed. These are the gradients of the utility function of the game. We accelerate the computation by playing the first two actions of the game and then each thread calculates the gradient assuming the first two moves separately and then the gradients are safely reduced. is worth noting that while there are 81 possible pairs of first moves for the players, only 18 buffers are required to avoid all possible concurrency conflicts. Depending on the machine and the game, calculating the gradients take roughly between 30 and 90 seconds. The compute time for calculating best response on various games presented Table 2]
3. Finally, exploitability is computed according to the formula
max & g1 + max y 92- BEX
yEy
The optimization problems can be solved in closed form using memory and runtime linear in the number of information states of the games, by using a standard greedy algorithm on treeplexes.
A.2 STATE- THE-ART TABULAR SOLVERS
exp a--aasppp spiel also includes implementation of the following state-of-the-art tabular solvers:
1. CFR+ (Tammelin) 2014) substitutes the regret matching (RM) algorithm in CFR with regret matching+ (RM+), leading to better performance empirically. The main difference between RM and RM+ is that RM+ clamps the regrets to be non-negative.
2. Discounted CFR (DCFR) (Brown and Sandholm 2019b), in similar vain CFR+, dis- counts the regrets to reduce the effect of previous actions with large (absolute) regret. Prac- tically speaking, iteration f, rescale the negative regrets by 1/(1 t*) and the positive regrets by 1 - 1/(1 + 1*). We use the default value of a = 1.5 and 3 = 0.
3. Predictive CFR (PCFR) (Farina al. 2019) uses optimistic regret minimizers achieve empirical and theoretical faster convergence rates for CFR. Practically, we observe the gradients twice to calculate the behavior policy of CFR.
These online, first-order optimization algorithms that tabularly refine strategies strategies-reprrssetted in requence-form-te taking steps in the direction of the gradient utility. Table [able reports the ex- ploitability of equilibrium computed by the methods across the 3x3 games and Figure 4 shows the
16

---

Published as a conference paper at ICLR 2026
Table 2: Runtime for computing best response with exp-a-spiel and OpenSpiel in seconds.
"OMM' indicates that the process out of memory before timing out. All results in this table were run on an AMD EPYC 7413 24-Core Processor with 256 GB memory.
Game
OpenSpiel
exp-a-spiel
Liar's Dice 1d4f
0.03
Liar's Dice 1d5f
0.03
Liar's Dice 1d6f
0.04
Liar's Dice 2d3f
10
0.04
Liar's Dice 2d4f
527
0.45
Liar's Dice 2d5f
OOM
11
Classical PTTT
OOM
Abrupt PTTT
OOM
27
Classical DH3
OOM
47
Abrupt DH3
OOM
47
(DR))
Abrugt
Phatomm (PTT)))TT
mapt Plantori Too-Toe-Ti
0,1000
Exploiabiitt
0.0010
1000
10000
10
1000
10000
10
1000
10030
10
1000
10000
10
1000
1000
Teeiitons
CFR
CFR
DCFR
PCRR
PCFR
PDCER
Figure 4: Performance of various tabular methods.
17

---

Published as a conference paper at ICLR 2026
Table 3: Exploitability after 512 iterations of different algorithms 3x3 board games
Game
Algorithm
Value
Exploitability
CFR
0.9999993
0.0000004
CFR+
0.9999993
0.0000004
3x3 Dark Hex (DH3)
DCFR
0,9999993
0.0000004
FP
0.9999997
0,0000002
PCFR
0.9999993
0.0000005
PCER+
0.9999993
0.0000005
PDCER
0.9999993
0.0000005
CFR
0.3841795
0.0295619
CFR+
0.3844054
0.0011629
3x3 Abrupt Dark Hex (ADH3)
DCFR
0.3844321
0,0010063
FP
0.3580483
0.1929647
PCFR
0.3844107
0,0011789
PCFR+
0.3843936
0,0006123
PDCER
0.3844073
0.0192829
CFR
0.6664844
0.0107796
CFR+
0.6666324
0.0016106
Phantom Tic-Tac-Toe (PTTT)
DCFR
0.6666511
0.0004070
FP
0.6622969
0.1016475
PCFR
0.6660216
0.0011638
PCER+
0.6662418
0.00193377
PDCER
0.6662910
0.0157007
CFR
0.5501850
0.0162306
CFR+
0.5501825
0.0004936
Abrupt Phantom Tic-Tac- Toe (APTTT)
DCFR
0,5501728
0,0003577
FP
0,5428041
0.2871734
PCFR
0.5501960
0.001 1708
PCFR+ 0.5501886
0.0004846
PDCER
0.5502121
0.0040427
convergence of DCFR and PCFR+. We note that Dark Hex has winning deterministic strategy for the first moving player. We analyze this strategy Appendix
B SOLVING DARK HEX
We investigate deterministic winning strategy for the first moving player in Dark Hex The value of this policy against uniform policy is 1, implying that it wins against all possible deterministic strategies, including any possible best response to it. The strategy is shown in Figure 5 where, at each information state, list of actions to try in that order is given. If the first action fails, the next action is played, and so on. After playing an action. there are two possible outcomes: the action is playable, information is gained. In general, the size of the list of actions is equal one plus the number opponent pieces that have been played but have not been observed; thus, the last action has to be playable. Beyond the computational proof that the strategy in Figure is optimal,
18

---

Published as a conference paper at ICLR 2026
Try cells and
Try cells 8, 7, and
Try cells and
Red wins
Red wins
Red wins
Try cells 4 and
cells and
Red
Red wins
Red wins
cells and
Red wins
Red wins
Figure 5: A deterministic strategy for player 1 that always wins. The gray dashed lines denote hidden action of the blue player.
can show using the fact the game of Hex (with perfect information) cannot end draw (Gale 1979 Hex Theorem). Since the first player always wins in the reachable information states, it is impossible the invisible pieces are a way that the opponent has already won.
This strategy is not applicable for Abrupt 3x3 Dark Hex as the board in the abrupt version does not correspond to board of hex. For instance, player 2 might never put piece on the board.
Since the game of Hex can be won by the first player, we argue that any deterministic winning Nash equilibrium for Dark Hex must also be an optimal strategy in the perfect information game, otherwise there must exist deterministic strategy for the opponent that wins against this Nash equilibrium. Thus at any information state, the action played must be either invalid or a winning move for all states that are compatible with that information state (i.e., it is possible that they the state of the board at that information state). This gives rise to an algorithm for finding deterministic Nash equilibrium by backtracking over which action play. We suspect that this algorithm does not.
19

---

Published as a conference paper at ICLR 2026
have any solution beyond 3 by Hex and that Dark Hex does not have any winning deterministic strategy.
C ALGORITHM IMPLEMENTATION DETAILS
We considered the following algorithms in this work: NFSP, PSRO, R-NaD, ESCHER, PPO, PPG and MMD. Below we give details about the implementation of each of them.
Source Implementation): We use the implementation of NFSP in OpenSpiel Laanctot et al. 2019). This implementation learns distinct models for player and player 2. For consistency with the other algorithms, we rewrote some of the algorithm to use PyTorch (Paszke et al. 2019) instead of TensorFlow (Abadi et 2015).
NESP
PSRO (Source Implementation): We use the implementation of PSRO in OpenSpiel Laanctot al. 2019). This implementation learns distinet models for player and player 2. For consis- lency with the other algorithms, we replaced the default OpenSpiel Tensorflow DQN agent (source) with the OpenSpiel PyTorch DQN agent (source).
R-NaD (Source Implementation): We use the implementation of R-NaD OpenSpiel (Lanctot al. 2019). This implementation learns single model that used for both player I and player 2. Natively, this algorithm is written in JAX (Bradbury et al. 2018). For convenience during evaluation. we convert the neural networks learned in JAX to equivalent PyTorch models. We verified this conversion by ensuring that, for the same inputs, the outputs of the models are equivalent.
ESCHER (Source Implementation): We the implementation of ESCHER from the original paper (McAleer et al., 2023). This implementation learns a single policy network that is used for both player and player Natively this algorithm is written in Tensorflow. For convenience during evaluation we convert the neural networks learned in Tensorflow to equivalent PyTorch models and verify by ensuring each layer weight and bias is equivalent.
PPO (Source Implementation) We use the implementation of PPO in OpenSpiel, which is itself modification from the CleanRL PPO made to work with OpenSpiel games and legal action masking. We further modify it to support self elay, and we learn a single model for both players.
PPG (Source Implementation) We the implementation of PPG in CleanRL, which modify to make it work with OpenSpiel games, legal action masking and to support self-play. We learn single model for both players.
MMD We use our PPO implementation and simply add a backward KI. term in the PPO loss. We learn single model for both players.
D RESULTS SUPPORTING IMPLEMENTATION CORRECTNESS
D.1 NFSP
We verified this re-implementation by comparing the exploitability of strategies learned in the orig- inal file with ours on Ledue Hold' em, see Figure
D.2 MMD
We verified that we obtain results consistent with the original MMD implementation (Sokota et al, 2023). Namely, after 10M steps training obtain approximate exploitability of around 0.14 and 0.20 on PTTT and ADH3 respectively. For reference, Table in Sooota et al. (2023) reports an approximate exploitability of 0.15 + 0.01 and 0.20 + 0.01 for PTTT and ADH3 respectively.
20

---

Published as a conference paper at ICLR 2026
NESP Implementation Verification
Library
PyTorch (ours)
TassodFlow (OpenSpiel)
Exploitablity
10'
10
10
10'
Step
Figure 6: Exploitability performance of the TensorFlow NESP implementation in OpenSpiel and our PyTorcl adaptation on Leduc Hold'en aggregated over 2 seeds.
Table 4: Related work and their use of exact or approximate exploitability when evaluating their method.
Approx. Exploitability or Head-to-Heat
Paper
Exact Exploitability
NFSP Heinrich and Silver
2016
Ledue
PSRO
Lanctot et al. 2017)
Ledue
ESCHER (McAleer et al.
2023
2x2 Battle Ship!, Ledue
Dark Chess, DH5, PTTT
RNaD Perolat al.
2022
Stratego
MMD Sokota et al.
2023
LDID4F, Leduc, ADH2
ADH3, PTTT
Qin et al.
(0199)
Kuhn Poker
AdaptFSP Goldsteen and Brown
2023
Ledue
Hu et al.
2023
Leduc
6-, 3-, 2-player poker,
NSFP-PLT (Li et al.
2023a)
Ledue
Pipeline PSRO McAAeer et al.
2020
Leduc
Barrage
XDO McAAler et al. 2021
Ledue
Smith et al.
2021 Anytime McAler et al.
Leduc
2022a
Leduc
Online DO Dinh et al. 2022)
Zhou et al.
2022
Leduc
PDO Yao et al. 2023)
Ledue
LDID6F,
SP-PSRO McAleer et al.
2022b
2x2 Battle Ship!, Ledue
Self-Adaptive PSRO (Li al.
2024a)
LDID6F, Leduc
Fusion PSRO Lian
2024
LD1D6F, Leduc
Dream Steinberger et al. 2020
Ledue
Flop Hold em
ARMAC (Gruslys et al.
2020
LD1D6F, Leduc
Head's up
FoReL (Perolat
2021
LDID6F, Leduc
Meng et al. (2023)
Leduc
PTTT. DH5, DH4
Liu et al. 2024)
Leduc
21

---

Published as a conference paper at ICLR 2026
Table 5: Size of games for which exact exploitability is reported in prior work.
Number of States (including terminal)
Game
Number of Infosets
ADH2
94
471
Leduc
936
9457
LDID4F
1024
8181
2x2 Battle Ship!
3286
10069
LDID6F
24576
294883
1.00
Exploiabaiity
0.50
0.25
0.012..025
0,050
0.100
0.200
0,400
Entropy coefficient
Figure 7: Average (dash-dotted green) and median (solid purple) exploitability of generic policy gradient methods across all games function of entropy coefficient, with shaded interquartile range and square-root x-axis. Vertical dashed red lines show the default entropy coefficient values for PPO in widely used DRL libraries. Results broken down by game are shown in Figure 12
Table 6: Entropy coefficients used in popular reinforcement learning libraries for policy gradient algorithms.
Library
Entropy Coefficient
Stable Baselines Hiill al.
2018
0.0
CleanRL (Huang et al.
2022
0.01 or 0.001
RLlib Liang et al. 2018
0.0
OpenSpiel Lanctot al.
2019
0.01
PufferLib Suarez 2024)
0.01
RL-Games Makoviichuk and Makoviychuk;
2021
or 0.01
Tianshou WWeng al.
2022
0 or 0.01
22

---

Published as a conference paper at ICLR 2026
E ENTROPY COEFFICIENT ANALYSIS
In Section we suggest that one possible reason for the poor performance of generic PG methods too far
in existing literature is that the entropy coefficients required achieve good performance removed from typical choices. To investigate, we plot exploitability as a function of entropy coef- ficient in Figure 7 The results are consistent with this possible explanation: Not only do unilateral changes in entropy coefficient produce drastic changes in exploitability, but the entropy coefficients with the best average performance are between 0.05 and 0.2, larger than any of the default en- tropy coefficients for PPO in Stable Baselines (Hill et al., 2018), CleanRI. (Huang et al. 2022), RIIib (Liang al., 2018), OpenSpiel (Lanctot al., 2019), Pufferl.ib (Suarez, (024)). RI-GGames Makovvichuk and Makoviychuk 2021). and Tianshou (Weng et al. 2022). which, as detailed in Table 6 range between and 0.01.
F ADDITIONAL RESULTS
The evolution of exploitability over 10M training steps for 5250 hyperparameter tuning runs is shown in Figure Each algorithm-gaat -aangggm pair, includes 150 runs 50 hyperparameter sets with seeds each. The variance across these seeds per set is visualized in Figure Furthermore, we com- pute the approximate importance of each hyperparameter in the tuning process, which report in Figure TO Given the high importance we obtained for the entropy coefficient in the policy-gradient algorithms, in Figure [2] we specifically analyze the average exploitability tropy coefficient. We observe that on average we obtain the best exploitability results for entropy coefficients much higher than standard ones detailed Table We also performed some additional experiments a wider hyperparameter sampling range for the number mebbbrrtrriinn wwwriiinngeeee sodes hyperparameter in PSRO, shown in Table 7 Figure reports the training durations and memory usage of the evaluated algorithms. Note that these metrics depend hardware and implementation optimizations and are provided for reference only.
function of the
Table 7: This table shows the performance PSRO on our benchmark games using a wider sam- pling range for the number -t sttttrrinnn www.trrninnnniiiiio..o. each oracle trains. Exploitability is presented as percentiles across 50 different hyperparameter sam- ples. Results for the original sampling range presented in the main results of the paper ([225, 8000)) are shown for comparison
hyperparameter that determines for how long
Game
Sampling Range
expl. @ 10%
expl. @ 25%
expl. @ 50%
expl. @ 75%
ADH3
(2500, 160000)
0.929014
ADH3
125, 8000]
0.750014
0.894206
APTTT
2500, 160000]
0.934627
APTTT
125, 8000]
0.795704
0.893034
0.988603
DH3
2500, 160000]
0.969553
DH3
125, 8000]
0.72258
0.862278
PTTT
25500, 160000]
0.88345
PITT
125, 8000)
0.786203
0.886317
0.99254
We show the training exploitability curves for the 1750 runs of the evaluation launch in Figure [3 The variance across seeds for each hyperparameter set be seen in Figure g Additionally, Fig- ure reports the training exploitability curves for the best hyperparameter set of each algorithm- game pair well as the variance 10 seeds for each set.
23

---

Published as a conference paper at ICLR 2026
201P tar Dir
10 Afric Dork (AOt))
Pheaaee TaTee Theeeeeeeee TeI
Abrot Phanen
8000
FPO
Figure 8: Exploitability step for all 5250 runs of the hyperparameter tuning launch, broken down by game and algorithm.
24

---

Published as a conference paper at ICLR 2026
20 LieV
D. DIDI
Abran No (APID)
Thesee des-
1.30
1947
Figure 9: Exploitability for all 5250 runs of the hyperparameter tuning launch, broken down by game and algorithm, then grouped by set of hyperparameters. The boxes-and-whisker each show the variance over 3 seeds for one set of hyperparameters, and are ordered by decreasing average exploitability.
25

---

Published as a conference paper at ICLR 2026
1060 Lanr 1764
SaATTT 2013
Figure 10: The top 10 most influencial hyperparameters from the hyperparameter tuming launch, broken down by game and algorithm. The sweep results provide mappings from hyperparameter sets exploitabilities, which each hyperparameter coefficient representing its importance in the prediction. This importance reflects the hyperparameter's impact on exploitability: high value indicates strong influence, while a low value suggests minimal impact. Note: inner rlrrrlaaaae aageet was abbreviated to inner.
train random forest regression model. This model assigns
Figure 11: Memory usage (RAM) and wall-time training duration statisties from the hyperparameter tuning launch, broken down by games and algorithm.
26

---

Published as a conference paper at ICLR 2026
20:00 Lizes Dice (LD2DS5F)
2xd Malugt Datk: (A.DEK3)
3x3 Dack [D113)
Abngg: Phantem To--ha--Toe
Phanton: To--ho--Toe (PTTT)
(APTTT))
0.75
0.25
0.00
Entrogy coefiiciett
MMD
PPC
Figure 12: Figure separated on a per game and algorithm basis.
201F San' Bus
20 Dub Nes DOOH
20 ABri Det (ADTIT
Fbraae Tala--he
Abrutt Flacos TheTeele AETTT
2
Figure 13: Exploitability vs. step for all 1750 runs of the evaluation launch, broken down by game and algorithm.
27

---

Published as a conference paper at ICLR 2026
Thase TaTTTT
P30
10
4
5
4
194
Figure 14: Exploitability for all 1750 runs of the evaluation launch, broken down by game and algorithm, then grouped by set of hyperparameters. The wwiissen-wwisskes each show the variance over 10 seeds for one set of hyperparameters, and are ordered by decreasing average exploitability.
Figure 15: Variance across seeds for the best-performing hyperparameter set in the evaluation launch, broken down by game and algorithm. The best set is defined as having the smallest final exploitability, averaged over seeds. Plots show mean exploitability over 10M training steps (solid line) with shaded region representing the standard deviation range across 10 seeds.
28

---

Published as a conference paper at ICLR 2026
G ALGORITHM HYPERPARAMETERS
Here, we provide an overview of the hyperparameters for the algorithms employed in the experiments. For each hyperparameter being swept, for each run, we sample N Uniform(1 -3,, 11,-3-2, 2, - 1, ))) For positive real-valued hyperparameters, we adjust the hyper- parameter by multiplying its default value by 2% rounding to an integer when necessary. For hy- perparameters that may otherwise egress [0. 1] when they should not, we exponentiate the default value the power of 2 rather than multiplying. All algorithms were trained for million steps and with the same neural network architecture (3 fully connected layers of 512 neurons each).
G.1 NFSP
Tuned hyperparameters:
replay bul.la..buffer...accttt
Description: Size of DQN replay buffer.
Default value: x 10a
Source: OpenSpiel NFSP Ledue example,
Maximum value: 5 x 10S, imposed to ensure that the replay buffer capacity cannot be significantly larger than the reservoir buffer capacity, in alignment with the ratio of default values.
2. reservoir buffer. www.errccaaccittt
Description: Size of reservoir buffer.
Default value: x 106
Source: OpenSpiel NFSP Ledue example,
Maximum value: x 108, imposed due memory constraints.
3. min buffer_size.ee learn
Minimum number of instances in buffer before training.
Default value: 1000
Source: OpenSpiel NFSP Ledue example.
4. anticipatory param
Description: Probability of using the RL best response as policy,
Default value:
0.1
Source: OpenSpiel NFSP Leduc example.
5. batch size
Description: Number of transitions to sample at each learning step.
Default value: 128
Source: OpenSpiel NFSP Leduc example.
6. everre
Description: Number of environment steps between learning updates.
Default value: 64
Source:
OpenSpiel NFSP Leduc example.
7. s1. ral_eerniingrrte
Description: Learning rate for supervised learning network.
Default value: 0.01
Source:
OpenSpiel NESP Ledue example.
8. inner rl agent epsilon decay duration
Description: Number of game steps over which exploration decays.
Default value: x 107
Source: OpenSpiel NFSP Ledue example.
29

---

Published as a conference paper at ICLR 2026
The epsilon decay duration is in the Ledue Example set to decay completely throughout training (2 x 10 games). We mimic this behavior by setting the decay duration to the total number of training steps (1 10? steps).
9. inner x1 ragentt.......... rearning.rate
Description: Learning rate for Q-network.
Default value: 0.01
Source: OpenSpiel NESP Leduc example.
10. inner r1. ale.tenttttt..m ba.chhsiiee
Description: Batch size of Q-network.
Default value: 128
Source:
OpenSpiel NFSP Ledue example.
11. inner r1 al..gent..... hupdate.target network.ever.
Description: Number of steps between target network updates.
Default value: 19200
Source: OpenSpiel NFSP Ledue Example
12. inner rl agent epsi start
Description: Initial exploration value.
Default value: 0.06
Source: OpenSpiel NESP Leduc example.
13. inner simer_rl_agee .aaaagen.epsst popsillo_endd
Description: Final exploration value.
Default value: 0.001
Source: OpenSpiel NFSP Leduc example.
Fixed hyperparameters:
1. optimizer str
Description: Supervised learning network optimizer.
Value: Adam
2. loss str
Description: Loss function for Q-network.
Value: MSE
G.2 PSRO
Tuned Hyperparameters
1. s sims www.pperrttt..
Description: Number of games to play for estimating the meta game.
Value: 1000
Source: OpenSpiel PSRO example
2. number training episodes,
Description: Number of episodes over which to train each oracle. Value: 1000
Source: OpenSpiel PSRO RL oracle file.
3. inner rl agent epsilon decay duration
Description: Number of game steps over which exploration decays.
Default value: x 107
Source: OpenSpiel NESP Ledue example.
30

---

Published as a conference paper at ICLR 2026
the epsilon decay duration in the Ledue Example is set to decay completely throughout training (2 x 10" games). We mimic this behavior by setting the decay duration to the total number of training steps (1 10? steps).
Note:
4. inner camer_r__agee ragenttt.......... rearning.rate
Description: Learning rate for Q-network.
Default value: 0.01
Source: OpenSpiel PSRO v2 example
5. inner rl agent batch size
Description: Batch size for Q-network.
Default value: 128
Source: OpenSpiel DQN Default
6. inner r1. al..aget..... update tapget..targe.......... network.eveey
Description: Number of steps between target network updates.
Default value: 1000
Source: OpenSpiel PSRO v2 Example
7. inner r1 epaagent.eession......
Description: Initial exploration value.
Default value: 0.06
Source: OpenSpiel NFSP Leduc example.
8. inner rl agent. epsi lon end
Description: Final exploration value.
Default value: 0.001
Source: OpenSpiel NFSP Leduc example.
Fixed Hyperparameters
1. optimizer str
Description: Oracle agent learning network optimizer
Value: Adam
2. loss foss.str
Description: Loss function for Q-network
Value: MSE
3. braininggsstratogg..ellcoo.
Description:
Determines against which oracle agents train in the next iteration
Value: Probabilistic (distributed according to the meta strategy)
G.3 R-NAD
Tuned hyperparameters:
1. batch size
Description: Number of samples in each training batch.
Default value: 256
Source: OpenSpiel R-NaD defaults
2. learning.rate
Description: Learning rate for the optimizer.
Default value: 10
Source: OpenSpiel R-NaD defaults
3. clip gradient
31

---

Published as a conference paper at ICLR 2026
Description: Maximum gradient value for clipping.
Default value: 10,000
Source:
OpenSpiel R-NaD defaults
4. target
www.wntwtwwo...m tazget.network.at avv
Description: Smoothing factor for target network updates.
Default value: 0.001
Source:
OpenSpiel R-NaD defaults
5. eta reward_transforr.
Description: Scaling factor for regularization in reward transformation.
Default value: 0.2
Source: OpenSpiel R-NaD defaults)
6. entropy schedule size value
Description: Schedule for updating the regularization network.
Default value: 50,000
Source: OpenSpiel R-NaD Leduc defaults.
7. c.vtrace
Description: Coefficient for V-trace importance weights (Espeholt et al. 2018).
Default value: 1.0
Source: OpenSpiel R-NaD defaults
Fixed hyperparameters:
1. trajectory max
Description: Number of steps after which games are truncated.
Value: Disabled
2. beta
Description: Size of the gradient clipped threshold in the NeurD gradient clip- ping (Hennes et al. 2019).
Default value: 2.0
Source: OpenSpiel R-NaD defaults
3. clip
Description: Size of clipping for the importance sampling in the NeurD gradient clip- ping (Hennes al. 2019)).
Default value: 10,000
Source: OpenSpiel R-NaD defaults
G.4 ESCHER
Tuned by peerpaarseeters
1. num.traversals
Description: Number of game plays for regret function learning.
Default value: 1,000
Source: Hyperparameters for Phantom TTT and Dark Hex 4 from McAleer et al. (023).
2. num_val.5 Seal.f.ttraversal
Description: Number of game plays for value function learning.
Default value: 1,000
Source: Hyperparameters for Phantom TTT and Dark Hex from McAleer et al. 2023
32

---

Published as a conference paper at ICLR 2026
3.
regret.traing staretttraiinstepss
Description: Number of training steps for regret network.
Default value: 5,000
Source: Hyperparameters for Phantom TTT and Dark Hex from McAleer et al. (023).
4. val www.train.n. train.steps
Description: Number of training steps for value function network.
Default value: 5,000
Source: Hyperparameters for Phantom TTT and Dark Hex 4 from McAleer et al. 2003).
5. policy net train.steep
Description: Number of training steps for the policy network.
Default value: 10,000
Source: Hyperparameters for Phantom TTT and Dark Hex from McAleer et al. (2023).
6. batch size regret
Description: Batch size for regret network learning.
Default value: 2,048
Source: Hyperparameters for Phantom TTT and Dark Hex 4 from McAleer et al. (2023).
7. batch size_val
Description: Batch size for value function network.
Default value: 2,048
Source: Hyperparameters for Phantom TTT and Dark Hex from McAleer et al. (023))
8. learning.rate
Description: Gradient descent learning rate.
Default value: x 10
Source: ESCHER Codebase
Notes: No value found in paper.
9. val expl
Description: Uniform policy mixing rate for off-policy exploration for value network.
Default value: 0.01
Source: ESCHER Codebase
Notes: No value found in paper.
Fixed hyperparameters:
We re-use all the fixed hyperparameters from the ESCHER Codebase As there are many, we do not list them here.
G.5 PPO
Tuned by perparameetess
1. learning rate
Description: Optimizer learning rate.
Default value: 2.5 x 10-4
Source:
OpenSpiel's PPO Implementation
2. num_steps
Description: The number of steps to rum each environment per policy rollout (i.e. the batch size is steps num envs).
33

---

Published as a conference paper at ICLR 2026
Default value: 128
Source: OpenSSiel'ss PPO Example Implementation
3. nww mumumiiiaatthes
Description: The number of minibatches (i.e. the minibatch size is round (num_steps num_envs / num.minibatches).
Default value:
Source:
OpenSpiel's ' PPO Example Implementation
4. update.epochs
Description: Number of policy update epochs (i.e. how many times go through the whole batch in each iteration).
Default value:
Source: OpenSpiel's PPO Example Implementation
5. cl ip coef
Description: Clipping coefficient in the PPO loss.
Default value:
0.1
Source: OpenSpiel's 's PPO Example Implementation
6. ent coef
Description: Coefficient for the entropy bonus in the loss.
Default value: 0.05
Source: Inspired from (Sokota et al. 2023
Note: This value is larger than in OpenSpiel's 's PPO Implementation because we have found that entropy bonuses lead much better policies. However, the default value of 0.01 is still within our sweeping range.
7. vf.coef
Description: Coefficient for the value term in the loss.
Default value: 0.5
Source: OpenSpiel's PPO Example Implementation
8. max grad norm
Description: Maximum norm of the gradient allowed during gradient clipping.
Default value: 0.5
Source: OpenSpiel's ' PPO Example Implementation
Fixed hyperparameters
1. num num_enve envs
Description:
The number of parallel game environments.
Default value:
Source: OpenSpiel's 's PPO Example Implementation
2. anneal 1r
Description: Toggle for learning rate annealing for the policy and value networks.
Default value: True
Source:
OpenSpiel's PPO Example Implementation
3. gamma
Description: Discount factor y for the return.
Default value: 0.99
Source: OpenSpiel's PPO Example Implementation
Note: We do not sweep this parameter due to the short-horizon nature of the games.
4. gae lambda
Description: Coefficient À for the general advantage estimation (GAF).
34

---

Published as a conference paper at ICLR 2026
Default value: 0.95
Source: Open PPO Example Implementation
Note: We do not sweep this parameter due to the nature of the games.
5. norm adv
Description: Toggle for advantage normalization before computing the loss.
Default value: True
Source: OpenSpiel's PPO Example Implementation
6. cl ip vloss
Description: Whether or not to clip the values in the value loss computation.
Default value: True
Source: OpenSpiel's PPO Example Implementation
7. target k1
Description: Target KL divergence threshold for early stopping of training epochs.
Default value: None (disabled)
Source:
OpenSpiel's s PPO Example Implementation
G.6 PPG
We consider the same parameters in PPO (Section G.5 with the addition of the following ones:
Tuned hyperparameters
1. iteeation
Description: Number of policy updates in the policy phase (Nz).
Default value: 32
Source: CleanRL implemmtttonnn and Cobbe et al. (2021)
2. e policy
Description: Number of policy updates in the policy phase (Ex).
Default value:
Source:
CleanRL implementation and Cobbe al. 2021
3. v.value
Description:
Number of value updates in the policy phase (Ev).
Default value:
Source:
CleanRL implementation and Cobbe et al. (2021).
4. auxiliary
Description: Number of epochs to update in the auxiliary phase (Eaux).
Default value: 6
Source: implemmntton and Cobbe al. 2021
5. beta clone
Description: Behavior cloning coefficient.
Default value: 1.0
Source: CleanRL implemetaton and Cobbe al. (201
6. wun.auxrrolootts
Description:
Number of mini-batches in the auxiliary phase.
Default value:
Source: CleanRL implementation
7. n.aux grad accum
Description:
Number of gradient accumulations in each mini-bateh
Default value:
Source: CleanRL. implementation
35

---

Published as a conference paper at ICLR 2026
G.7 MMD
We consider the same parameters as in PPO (Section G.5 with the addition of the following ones:
Tuned hy hyperparameters
1. k1 coef
Description: Coefficient of the reverse KI. divergence in the loss function.
Default value: 0.05
Source: (Sokota et al. 2023
We use a constant value for the reverse KL coefficient (as well as for the entropy coefficient) instead of a custom schedule.
Note:
36


Reviews:
[{"review_id": "ZDcGzlNmBK", "reviewer": "Reviewer_op4U", "summary": "The paper revisits a common claim that policy gradient methods in self-play perform poorly in IIGs.  The paper does a thorough comparison of a set of 6 DRL methods for IIG (3 representative of adaptations of tabular methods with theoretical guarantees in tabular settings, and 3 policy gradient methods).", "questions": "No questions; see weaknesses for suggestions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is important work!  It's very important to be sure the community is not trapped by oft-repeated claims that are just known, but not truly validated.  The goals of this paper are important.  But, what is most impressive about the paper is the carefulness in the transparency of the comparison, recognition of their own biases, aim for reproducibility, and the limitations of what the empirical results justify.  This is commendable.\n\nThe following line is one of my favourites a paper focused on empirical results: \"However, since we have hypothesized these algorithms will underperform, our role in tuning them should be carefully scrutinized. To aid readers in this endeavor, we provide detailed documentation of our algorithm implementations, hyperparameter tuning procedure, and hyperparameter tuning results, in addition to final results.\"\n\nThe section on limitations at the end is also valuable.  More papers should be quick to say what they aren't saying the way Section 6 does in the third sentence.", "weaknesses": "I think the biggest weakness of the paper is the paper comes at this work with a very strong bias and preconceived expectation to the result.  I am glad the authors recognize this bias, but I think that bias gives an edge ot the writing that is not needed in the text.  I believe the authors gave all algorithms a fair shake.  More importantly they are transparent about the shake they gave them.  But you don't get that picture through the first half of the paper.  I found myself immediately bristling at the line, \"(and, not least, the disinclination of researchers toward showing baselines outperforming their algorithmic contributions)\", thinking that the authors were just going to be doing the same thing but the other direction.  In the end I was pleasantly surprised by the care taken.  But, I would highly recommend the authors think about reframing the paper to take a more neutral objective tone (while still admitting their own hypothesis).  I would see value in this paper even if the results had gone the other way, and confirmed oft-repeated notions that have not been rigorously tested.\n\nMy only quibble with the experimental design was the forced choice of Adam to be used for all algorithms.  Do you have a clear table as to which of these 6 algorithms use Adam as the default optimizer?  That might be helpful to rule out that as a source of difference.\n\nI'll use the rest of this section to make some suggestions about how the paper could possibly be further improved.\n\nThere's an awful lot of repetition in the statements and ideas.  I think the paper probably could be a page shorter and have the same impact.  Which would then open up space for some valuable additions.\n\nI think it would be worth discussing what soundness results exist in this space.  I find the biased edge in writing seems to be dismissing trying to extend tabular methods with soundness guarantees into the world of parameterized approximations, which seems like an odd choice.  Wouldn't we want some theory to explain what works in practice?  I think it would be valuable to discuss what soundness guarantees we do have for any of the explored methods, both in tabular settings and for the particular application to either games (for PG methods) or function approximation (for other methods).  This would add considerably to clarifying the current state of the field.  The paper argues that policy gradient methods get \"dismissed as unsound or relegated to the role of sacrificial baseline\", but this paper doesn't offer anything to suggest that they are sound, right?  Now I suspect that the extensions of FP, DO, and CFR don't have guarantees outside tabular, but this clarity would be helpful.\n\nI would also like to see more discussion about why you think the poor performance of policy gradient methods in IIGs was so prominent given your empirical results.  You pose early in the paper that maybe it has to do with the hyperparameter space, where MMD used more entropy regularization than typically used.  Well, was that the reason?  Can you conclude anything about the reason from your experiments?  Even if it's just saying, we didn't find the entropy regularlization parameter to be a significant factor in policy gradient methods performance, would add something.\n\nIt would be nice to see some discussion of whether the methods actually achieve low absolute exploitability.  Are any of these small enough to run a tabular method on?  Or could you add one game that is?  It would be nice to know if PG methods are actually doing a good job of finding low exploitability strategies or if the non-PG methods are doing a particular poor job, despite their tabular counterparts being very effective.  Again, it seems there is a missed opportunity to clarify what gaps still exist in the field.  Similarly, could you show trends of exploitability as the network size increases across methods?  Are of these methods able to see scaling with compute and make use of additional capacity to reduce exploitability?\n\nI think another valuable addition would be a brief discussion about DeepStack, Libratus, and/or Student of Games.  These are methods that absolutely scale to large games, with NLHE being many orders of magnitude bigger than the games posed in the benchmarks, while also having some theory (with expected limitations due to approximation quality of network architectures).  They are also not model-free and add decision-time search as a key component, so they are outside the scope of the paper's claims.  But it still seems useful to discuss in order to give a more comprehensive picture.\n\nI think that's the summary of my suggestions: the authors do a very good job of giving evidence for their stated main claim, but there's an opportunity to be much more... to give a comprehensive picture of where we as a field are at in algorithms for learning in IIGs.  This is a good paper, but that would be a great one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mTmuwQQBOP", "reviewer": "Reviewer_rrhL", "summary": "This paper puts forward a hypothesis that policy gradient (PG) methods based on (divergence-regularized) self-play can outperform reinforcement learning (RL) implementations of fictitious play (FP), double oracle (DO), or counterfactual regret minimization (CFR) in solving two-player zero-sum imperfect-information games. The authors empirically verify this hypothesis through extensive experimental analysis under benchmark games. Under exact exploitability computations or head-to-head evaluations, PG methods like magnetic mirror descent (MMD) consistently outperform FP/DO/CFR-based methods.", "questions": "Could you make a more detailed comparison between the implementations of MMD (based on mirror descent) and R-NaD (based on FTRL) to explain their performance discrepancy? Since gradient descent, multiplicative weights, Hedge, and FTRL have inherent relationships, MMD and R-NaD could be similarly viewed as a kind of regularized FTRL from the game-theoretic perspective.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper makes significant efforts in aligning the existing game RL algorithms and fairly evaluating them under the strict exploitability measure.\n\n2. The elaborations on the experimental settings and choice of benchmarks make the conclusion convincing.", "weaknesses": "1. My major concern about this paper is that the proposed hypothesis is not novel. To me, the hypothesis is somewhat equivalent to saying last-iterate convergence is superior to average-iterate convergence under current game RL implementions. I think it has been common sense since recent game-theoretic research has a significant focus on last-iterate convergence (e.g., [1,2]), which usually implies a linear convergence rate. In some small games, the last iterates can converge exponentially faster than the average iterates, even under the same learning dynamics (e.g., optimistic FTRL). The superiority of MMD in larger games is therefore predictable.\n\n2. The benchmarks do not include real-world games like Battleship and No-Limit Texas Hold’em. To me, these games are more interesting because they could have more complicated transitivity structures in the policy space. Performance in these real-world games could be different from in the benchmark games considered in this paper. For example, it remains unclear whether MMD can also solve Stratego as R-NaD.\n\n[1] Kenshi Abe, et al. Adaptively perturbed mirror descent for learning in games. ICML, 2024.\n\n[2] Runyu Lu, et al. Divergence-regularized discounted aggregation: Equilibrium finding in multiplayer partially observable stochastic games. ICLR, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SJi7IWbM36", "reviewer": "Reviewer_WVeg", "summary": "The paper presents empirical evidence that commonly used policy gradient algorithms, such as PPO and PPG, which have some similarity to MMD, quickly converge to low exploitability strategies in a range of imperfect information zero-sum games. Moreover, they show that within a restricted budget, they converge much closer to the Nash equilibrium, than selected methods based on CFR, Fictitious play, and Double Oracle algorithms.", "questions": "Q1)  Do PPO/PPG match the performance of MMD even on smaller games?\n\nQ2) Do you know of specific counterexamples where PPO/PPG fail to converge? \n\nQ3) Is the plot in Figure 2 \"the hyperparameter tuning launch\" or the \"evaluation launch\"?\n\nQ4) Do I understand correctly that you run PSRO with at most 8000 steps per oracle training?\n\nQ5) Can you provide a compute time comparison between your BR implementations and the OpenSpiel one?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The paper presents interesting and important empirical evidence that may encourage people to try a simpler and more standard solution to solve their problems.\n  * The paper makes a rigorous effort to compare to several baselines.\n  * The paper promises to publish a more efficient BR computation for OpenSpiel, which might be useful", "weaknesses": "Summary: I believe the empirical results are interesting and likely worth publishing. However, the writing of the paper is not ideal. The authors try to condense the main message of the paper to a short paragraph they call a \"hypothesis\". However, for me, a hypothesis is a formal statement that can be proven to be true or false. The \"hypothesis\" in this paper is a very vague observation that is subject to many interpretations and many of them are clearly not true in general. \n\nMoreover, there are some mistakes, likely caused by insufficient proofreading after adapting the paper to the new template, and several misleading statements. I would be happy to recommend accepting the paper after a major revision, but in the current state, I lean towards rejecting.\n\n\nDetailed comments: \n\n\"Hypothesis\" is IMO not the right word for what the paper provides. I would be fine if it is called an \"observation\" and softened by some \"tend to\" \"often\", or something similar. If it were to stay a \"hypothesis\", it would have to specify much more clearly what is \"share an ethos with magnetic mirror descent\" and list specific classes of methods \"based on FP, DO, CFR\", since it is a huge diverse set of algorithms.\n\nI also do not like the authors over-selling the contribution of implementing efficient best response, calling it \"first broadly accessible exact exploitability computation\" and not properly reviewing prior efforts to scale-up BR computation, or comparing it to the BR computation available in other frameworks. I would expect mentioning at least [A], arguing why an approximate solution, such as [B] is not sufficient, and comparing execution speed to at least OpenSpiel BR, if not also some other frameworks. Otherwise, it is hard to accept that as a contribution. \n\nThe paper IMO falsely claims that \"only a small number of works make the effort to report exploitability in large games\". A very large portion of papers on approximating equilibria in the last over a decade use exploitability as the key metric and report it on large poker endgames, Phantom TicTacToe itself, and other games of comparable size. It is certainly nice to publish a new, efficient implementation compatible with OpenSpiel and presumably much faster than the default implementation, but I do not consider similar claims to be an appropriate characterisation of the existing work.\n\nMoreover, the paper should establish that one or two orders of magnitude that their BR implementation may buy compared to the existing ones is crucial to deliver their results. Would similar experiments on slightly smaller games deliver substantially different results? It claims that it is \"a fundamentally different challenge\", but gives a very brief statement to support that and does not show or specify whether it becomes fundamentally different at 10^8, 10^9 or 10^10 states.\n\nI also consider misleading the claim that \"DO can suffer from slow convergence (...) according to the (...)  exponential lower bound for DO\", after talking about perfect recall EFGs up to that point in the paper. The boud they cite is not about this class of games. The paper explicitly states in the introduction that there are efficient variants of DO for EFGs.\n\n\nTo conclude, I would love to see this paper published, with more fair treatment of the existing work, more exact definition of \"the policy gradient observation\", and experiments on smaller variants of the games showing whether it starts to hold only form some scale of the games up, or whether it holds also for smaller games. Moreover, I would avoid implying that the methods are better than all CFR, DO, and FP based methods, and focus on using these methods as mere baselines that are outperformed.\n\n\n[A] Johanson, Michael, et al. \"Accelerating best response calculation in large extensive games.\" IJCAI. Vol. 11. 2011.\n\n[B] Timbers, Finbarr, et al. \"Approximate Exploitability: Learning a Best Response.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)