Divergence-Regularized Discounted Aggregation: Equilibrium Finding in Multiplayer Partially Observable Stochastic Games
Authors: Runyu Lu
Yuanheng Zhu
Dongbin Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: partially observable stochastic game
Nash distribution
divergence regularization
hypomonotonicity
last-iterate convergence
Nash equilibrium
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: This paper presents Divergence-Regularized Discounted Aggregation (DRDA), a multi-round learning system for solving partially observable stochastic games (POSGs). DRDA is based on action values and applicable to multiplayer POSGs, which can unify normal-form games (NFGs), extensive-form games (EFGs) with perfect recall, and Markov games (MGs). In each single round, DRDA can be viewed as a discounted variant of Follow the Regularized Leader (FTRL) under a general value function for POSGs. While previous studies on discounted FTRL have demonstrated its last-iterate convergence towards quantal response equilibrium (QRE) in NFGs, this paper extends the theoretical results to POSGs under divergence regularization and generalizes the QRE concept of Nash distribution. The linear last-iterate convergence of single-round DRDA to its rest point is proved under the assumption on the hypomonotonicity of the game. When the rest point is unique, it induces the unique Nash distribution defined in the POSG, which has a bounded deviation from Nash equilibrium (NE). Under multiple learning rounds, DRDA keeps replacing the base policy for divergence regularization with the policy at the rest point in the previous round. It is further proved that the limit point of multi-round DRDA must be an exact NE (rather than a QRE). In experiments, discrete-time DRDA can converge to NE at a near-exponential rate in (multiplayer) NFGs and outperform the existing baselines for EFGs, MGs, and typical POSGs.

Published as a conference paper at ICLR 2025
DIVERGENGE-REGULARZZD DISCOUNTED AGGREGA- TION: EQUILIBRIUM FINDING IN MULTIPLAYER PAR- TIALLY OBSERVABLE STOCHASTIC GAMES
Runyu Lu Yuanheng Zhu Dongbin Zhao
School of Artificial Intelligence, University of Chinese Academy of Sciences 2 Institute of Automation, Chinese Academy of Sciences 1urunyul 7 7@ma ucas ac cn {yuanheng zhu, dongbin. zhao 8 ac.
ABSTRACT
This paper presents Eivergence-Regularie Discounted Aggregation (DRDA), a multi-round learning system for solving partially observable stochastic games (POSGs). DRDA is based on action values and applicable to multiplayer POSGs, which can unify normal-form games (NFGs), extensive-form games (EFGs) with perfect recall, and Markov games (MGs). In each single round, DRDA can be viewed as a discounted variant of Follow the Regularized Leader (FTRL) under a general value function for POSGs. While previous studies on discounted FTRL have demonstrated its last-iterate convergence towards quantal response equilib- rium (QRE) in NFGs, this paper extends the theoretical results to POSGs under divergence regul regularization and generalizes the QRE concept of Nash distribution. The linear last-iterate convergence single-round DRDA to its rest point is proved under the assumption unique, it induces the unique Nash distribution defined in the POSG, which has a bounded deviation from Nash equilibrium (NE). Under multiple learning rounds, DRDA keeps replacing the base policy for divergence regularization with the policy the rest point the previous round. It is further proved that the limit point of multi-round DRDA must be an exact NE (rather than a QRE). In experiments, discrete-time DRDA can converge NE near-exponential rate in (multiplayer) NFGs and outperform the existing baselines for EFGs, MGs, and typical POSGs.
the hypomonotonicity of the game. When the rest point is
INTRODUCTION
While a wide range of game-the learning dynamics, including no-regret dynamics and best- response dynamics, are primarily analyzed static normal-form games (NFGs), many real-world games are dynamic and usually considered under different game representation. For example, simultaneous games like pursuit-evasion games (Li et al. 2024) and fighting video games (Tang al. 2023) can be formulated Markov games (MGs), especially when there is infinite horizon. Imperfecctinfoomaaion sequential games like Texas Hold' em are commonly formulated as extensive- form games (EFGs) with perfect recall. In view of this requirement, many of the recent studies have tried to extend the theoretical results established in NFGs to dynamic games (Kroer et al. 2020; Farina al., 20216). Counterfactual regret minimization (CFR) (Zinkevich et al.) (007). as a well-known example, decomposes global regrets into local ones under counterlactual value and enables regret minimization within time complexity polynomial in the size of the game tree. However, when it comes to infinite-horizon games like MGs, such generalizations are not feasible. Instead, another line of algorithms based on best-response computations, including Fictitious Play (FP) [Bownn] 1951), Policy Space Response Oracle (PSRO) (Lanctot et al. (017), and Exploitability Descent (ED) (Lockhart et al. 2019). are known to be applicable. As an extension of MG, partially observable stochastic game (POSG) introduces imperfect information and is capable of unifying NFG, (perfect-recall) EFG, and MG. However, it is not clear a basic learning dynamic under general action value be directly applied to POSGs without sacrificing its convergence properties in NFGs.

---

Published as a conference paper at ICLR 2025
In recent years, last-iterate convergence has become increasingly interesting research target (Wei ct al. 202T; Lee et al. 202I). Compared to the well-examined concept of average-iterate convergence, which CFR and FP are based upon. last-iterate convergence means that there is no need to preserve policies in time average. This is an ideal property for further extension to deep reinforcement learning (DRL), as it is intractable to time-average function approximators like neural networks. Much of the existing research on last-iterate convergence is related to Follow the Regularized Leader (FTRL) (Shalev-Shwartz & Singer 2006)), which a basic learning framework that has close relationship with various learning dynamics in game theory. The widely studied replicator dynamics (Taylor & Jonker 1978) as well as Hedge, multiplicative weights, and gradient methods can be captured as special cases of FTRL
& Lugosi 2006 Arora et al., 201 2, Hazan et al. While proved that vanilla FTRL can cycle NFGs and EFGs (Mertikopoulos et al.
2016).
2018
Perolat et al. (02T), some of its variants exhibit last-iterate convergence. For example, Daskalakis & Panageas 2019) prove that the optimistic variant of FTRL exhibits last-iterate convergence to Nash equilibrium (NE) in the two-player zero-sum games with the unique equilibrium. To avoid the uniqueness assumption, some recent works also focus on solving the "regularized" game and directly establishing last- convergence to the quantal response equilibrium (QRE, which can be viewed as an approximation to NE) (McKelvey & Palfrey 1995) of the original game. These results are
applicable to EFGs (Liu et al. 2023 Sokota al. 2023) or MGs (Cen et al. [021 2023), but they are generally restricted to two-play two-player zero-sum games. To deal with multiplayer games, we also focus on some "discounted" variants besides the works on reward regularization.
A discounted variant of FTRL, first examined in Leslie & Collins (2005), is proved to approximate Nash equilibrium under a shrinking temperature parameter in two-player zero-sum MGs (Sayin et al. 102T Even for multiplayer games, existing theory suggests that it can converge to the solution concept of Nash distribution, a specific form of QRE defined in NFGs, under a range of distinct assumptions {Coucheney et al. 2015 Gao & Pavel 2021). While discounted FTRL exhibits strong convergence properties, it is still at the price of pingying&& a "perturbation" to the equilibrium point. Actually. it is completely unknown whether discounted FTRL can approximate exact Nash cquilibrium in multiplayer POSGs. Recently, a series of works on two-player zero-sum or monotone games (Perolat et al., 2021 Abe et al. 2022 2023) have shown that the perturbation arising from invergence-regulariize
rewards can be mitigated using a technique of multi-round learning, which repeatedly replaces the base policy (or "anchor" policy; seeDacob al. (2022); Bakhtin et al. (202)) with the policy at the rest point in each round. Inspired by these results, we introduce multi-round cellibrruumleaarinng system named Divergence-Regullrriee Discounted Aggregation (DRDA), based on the continuous -imeuuuuuu--eeeeeee dynamic of discounted FTRL. We further examine the last-iterate convergence of DRDA in the POSGs with certain hypomonotonicity (a property more extensively examined in optimization; see Alomar & Chavdarova (224)), as well as its relationship to Nash equilibrium.
Specifically, the contribution of this paper includes:
Employing the advantage value (a type of action value) at each decision point in POSGs, we propose single-round DRDA as extension of (NFG-based or MG-based) discounted FTRL. By defining a generalized Nash distribution (GND) that extends Nash distribution to POSGs, we first prove the relationship between the rest point of learning dynamic and QRE concept in POSGs: every GND induces a DRDA rest point in POSGs (Theorem 1).
By assuming local hypomonotonicity, which is a relaxation of the widely used concept of global monotonicity (Gorbunov et al.. 2022b Cai et al.) 22022), we prove that single-round DRDA converges to its rest point in any local X-hypomonotone region at a linear rate when the regularization (temperature) parameter in DRDA is greater than À (Theorem
2). Compared to the existing works (Perolat et al., 2021 Sokota et al.. 2023 Abe et al.
2024h, our convergence result is directly based on the action value for POSGs (possibly with imperfect information and an infinite horizon), while requiring a less stringent assumption.
By assuming the uniqueness of DRDA rest point, we establish an explicit NashConv bound en log K. Σ,'' for the rest point of single-round DRDA (Theorem 3). By showing that this gap can vanish as the rest-point sequence of DRDA converges in multiple rounds, we further prove that the limit point of multi-round DRDA is an exact Nash equilibrium (Theorem 4).
Through experiments, we show that discrete-tine implementation of multi-round DRDA converges to Nash equilibria in various finite-action games under the POSG formulation. For NFGs, DRDA can converge a near-exponential rate. For multiplayer EFGs, DRDA can
2

---

Published as a conference paper at ICLR 2025
outperform the baselines including extragradient CFR (Farina et al. 2024). moving magnet magnetic mirror decent (Sokota et al. 2023), and regularized Nash dynamics (Perolat et al.) 2022). Besides, DRDA significantly outperforms the general methods in infinit-hhriiznn Markov game and the existing baselines in typical POSG environments.
2 PRELIMINARIES
2.1 PARTIALLY OBSERVABLE STOCHASTIC GAME
We use the formulation of partially observable stochastic games (POSGs) general framework to describe a class of finite-action games possibly with imperfect information and an infinite horizon. Definition 1. A partially observable stochastic game is a tuple (N,S,O,A,1) 2, [t- p.,,,
N = 21, ... , n is the set of all players. S is the set of all global states E S. O is the set of all joint observations expressed as ã = (a');EN A". P is the state transition distributions, with a subsequent state generated by 8141 P(s,, .e). Z the observation distributions, with a joint observation generated by oi+1 ((s áz, S(+1). R' is the reward function for player i, with an instant reward generated by r+4 = R' (st, de. St+1)- p the initial state distribution, assigning each s E a probability p(s) to be the initial state Su. ? E (0, is the discount factor, and T is the termination time. For finite-horizon games, 7 = 1, and is finite. For infinite- horrize games, Y < 1, and + XC
(o') E O. A the set of all players' actions, with the joint action
Inherently, POSG is capable of describing simultaneous games like NFGs and MGs. Furthermore, since sequential moves can be regarded as simultaneous moves with empty actions for the waiting players, sequential games like EFGs can also be converted to equivalent POSGs.
2.2 BASIC CONCEPTS & NOTATIONS
The following concepts and notations will be frequently used in this paper.
History. As POSG contains imperfect information, the true state of the game time step / S T should be expressed as a history h, = (so. oo 0o 81, oi, ã.,- ... St, o,) E rather than a single current state sy. As comparable concept, the trajectory used in the field of reinforcement learning (RL) can be viewed as a history with rewards: T = (so. oo, Co $1, Fi, 01. ã1, ... St, Fe, oz).
Decision point. In POSG, player i makes his/her decisions based his/her past observations and actions z (ob. ob. of. ".. , of). In the case of imperfect-informattee games, each of can be viewed infinite-horizon games, some z (even with different t) are equivalent under the optimal strategy and can be regarded the same decision point, which we use any one of the x (or x²t) to denote. For example, in a Markov game, player i is at the same decision point when the current observation (the
information set and corresponds to multiple true histories ht In the case of
last term in x²) is the same. We use X to denote the set of all decision points and require the number of distinct decision points |X'| to be finite throughout this paper.
We borrow the notation from the literature of EFGs and write h4 æ (viewing xi as an information set) if h is not contradictory to the existing terms in z/. For & > t, we write h2 I h4 to express hg as a successor of hy in the game tree. Similarly, we write x 3 2 to express x as a successor of x
In the special case of perfect-information games, the of of any player can uniquely determine hy- Furthermore, we can simply use s; to represent both fs and z; in this case, as the POSG is reduced to an MG. The expression of NFG, EFG, and MG under the POSG formulation is shown in Table
Table 1: Conversion of NFG, EFG, and MG under the framework of POSG
NFG
EFG
MG
Termination Time
T= 1
T' depth of game tree
T' 0C
Discount Factor
7
Dynamics Reduction
|S| = |0] =1
P(st, (,) = P(s,, ag
(*+)
xt = hh 81
For EFGs, we use i(s:) to denote the current player under global state
St.
3

---

Published as a conference paper at ICLR 2025
Policy. At each decision point z', player i's policy (x") is probability distribution over A. Player i's complete policy is a combination of the policies at all x' E X". We use the joint policy म (Λ' Ree II to denote the combination of all players' complete policies and use मt to denote the combination of all players' policies except player i's. By Definition | the probability of visiting history h4 (8o; o0; do; 81, 01,01,"' ,8t, 02) under a joint policy म is computed as follows:
II at (xk ak) ) ((SL,
Pr (h, f) = p(so) (so, σo)
Σ 1=
|
(((s),0),+. @x), 8k-1) 2 ((sk, Ck. 8k+1), 6k+1)
Value functions Advantage. POSGs, the value function history he under a joint policy is defined as (he) E Ek - Ihe, The corresponding state-action value function is defined as Q4 (h. at) = E |re+1 + (he4a)) [ ai. #1 Borrowing the notation from RL literature (Sutton& & Barto 2018). define history-based advantage as A af) Q' a;) Vi(h)).
Utility & Nash equilibrium. The individual utility in POSGs is defined as the expected value over the initial states: (#t) = Eno [vo)) where hg = (sg, og). with sp ~ p and Ø 2(s)).
As commonly used solution concept in game theory. Nash equilibrium (NE) is joint policy nash where no player can increase his/her own utility by unilaterally deviating from his/her own policy. Specifically, for any player i E N and any individual policy x' E IT'. it holds that u² aTFnnst) 2 u' (TT, A Based on the Nikaido-Isoda function (Nikaidō & Isoda 1955 define NashConv(ā)
Σ==1 max (w'(xt', -- n² (7)). NashConv measures the deviation of म with respect to Nash equilibrium. Clearly, WashConv(Rnash) O.
2.3 FOLLOW THE REGULARIZED LEADER & DISCOUNTED FTRL
Follow the Regularized Leader. Follow the Regularized Leader (FTRL) is a widely examined cquilbbrium-leaminn dynamic. NFGs, the continuous-tin time FTRL (Mertikopoulos et al. (2088) can be expressed as an ODE of the score ÿ with a non non-negative continuous time variable & (different from the discrete time step used in the formulation of POSG):
{
v w/ (#t)
(I)
n. = (3'))
where w/ (a)) = (a () is player i's expected utility under a pure strategy =f(+) with T/ (a) = 1. and of (v/)) = arg max (=', vt) -
(0))-=((())) (T(") is the policy selection that maps the
current score function (-) into the policy space ((A).
The penalty function ((())) guarantees that the o²(.) in 1 is well-defined in the sense that the argmax
results in a singleton. Under entropic regularization (i.e., (m²) = > T (aa)) log x'' {aa) )), FTRL
equivalent multiplicative weights (Hedge) (Cesa-Bianee & Lugosi 2006 Arora et al. 2012) as
well as replicator dynamics in evolutionary game theory (Taylor & Jonker 1978).
Discounted FTRL. While FTRL exhibits average-iterate convergence, the last iterate of FTRL may not converge. Actually. it features cycling behavior that is robust to the choice of regularization, util- ity transformations, and game restrictions (see Mertikopoulos et al. (2018)). To avoid the recurrence behavior, a "discounted" variant of FTRL can be used instead (Leslie & Collins 2005):
d/ w w' v/ म: = o
Discounted FTRL uses (xt)) yt instead of (Tt.) as the derivative of the score Th. Consequently, the past aggregation of utility will be discounted as increases. This guarantees last-iterate conver- gence towards the solution concept of Nash distribution in NFGs (Gao & Pavel 2021; Coucheney et al. 2015). Nash distribution is defined as the NE policy under a utility function perturbed by entropic regularization ă' (Tf) u² u()) o' ('("". where of (i²) Eg'e x' x'(a) log x' (a'). It is
specific form of quantal response equilibrium (QRE) (McKelvey & Palfrey 1995) and can be viewed as relaxation of the original NE concept.
4

---

Published as a conference paper at ICLR 2025
Rest point. To formally examine the last-iterate convergence of continuous- dynamic, we need to use the concept of rest point. For discounted FTRL (2). (yr: #f) is a rest point if both y = w ( and 7 of (yf) are satisfied for all E N. Intuitively, it means that 3/ y,, म. म, (t 2 0) is solution to the ODE, Under entropic regularization, it can be proved that the policy at the rest point of
discounted FTRL is equivalent to the concept of Nash distribution in NFGs Bee & Pavel (202T)
EYVERGENCE-REGULANE
DISCOUNTED AGGREGATION
As we have mentioned, while the last-iterate convergence property of discounted FTRL is desirable. it is restricted NFGs and comes at the price of perturbing the equilibrium point. In view of this, we further employ general action value and combine the idea of multi-round learning to construct Divergence-Regubee Regularized Discounted Aggregation (DRDA), a learning system for solving POSGs.
3.1 SINGLE-ROUND DRDA
As POSG concerns imperfect information, define advantage value of (7) kind of action value that averages all histories' advantages at a decision point based on the visitation probability Pr (A(f)):
Σ Pr (h) ah,
o' (+)
hEr
Σ Pr (hl hE.*
With the definition of (FF), we analyze the following decision-point-lett ODE in POSG.
Definition 2. A single-round DRDA under regularization parameter >> O and base policy Thase is a continuous-tin dynamic expressed as the following ODE with non-negative time variable
{
i = n² (+te) - vi.
-. -'=
where uf(xx ..) is the score function at the decision point x² E X', (Ft)((a :) is the advantage value defined by (.) and a² (x") is a policy choice map under divergence regularization:
o² (v))(a")= = arg max
(x)), - <DKL ( (5 (x'))= (()es (a)))A((A)
(5)
Note that the advantage value of (wt) in the single-round DRDA (4) is comparable to w (Ft) in the discounted FTRL (2.. When T = | 10| = 1, the POSG is reduced to an NFG, and we have o' (Z)) a) = w (a) - u' u'(f))).
From the integral form of the score function, we can see that a single-round DRDA is a discounted aggregation of the past (T(Te) under a factor ((trr);
S' (t-rr) (Ft) d')dr
ai'e'aa) -
(6)
Different from vanilla FTRL, the introduction of the exponentially decaying factor suggests the boundedness of the score function:
Lemma 1. In single-round DRDA, the score ÿ is bounded.
Besides, KL-divergence DKL (= = =EPEA T''', (x', a²) log ('((aa")
Ti (2 the
regularizer (penalty) function in (5) and requires Thase to be of full support. Correspondingly, we have an equivalent expression for (wi), which is in form similar the well-known softmax:
Lemma 2. In single-round DRDA, o² ('()) has closed-form expression: Zbuse(x' (x',a²) exp (2',
o' d' a²)
Σ Thase (xt, exp (f b)) DEA
It is direct to verify that the advantage value i7(-) is continuous. With the continuity of (-)) 7.. know that ((F())) is also continuous. By Brouwer's fixed-point theorem (Florenzano 2003). J. fixed point of ((F(-) exists. The existence of the rest point of single-round DRDA is directly proved.
5

---

Published as a conference paper at ICLR 2025
3.2 MULTI- DRDA & DISCOUNTED AGGREGATION
Based on Definition 2 we further construct multi-round DRDA to extend the learning process to multiple rounds, using an idea of repeatedly replacing the base policy Whose (see Perolat al. (2021 ; Abe et al.] [2022] (023)).
). This technique can help the overall learning process to approach Nash equilibrium rather than QRE Specifically, a round corresponds to a time-evolving process of 4 until reaching its rest point (equilibrium point). The ODEs in different rounds differ in the base policy Thase for divergence regularization.
Definition 3. A multi-round DRDA under regularization parameter 0 and initial point po is an iterative process of calling an oracle M. In the 1-th round (I 2 1), M takes (Jo, Mo) Pt-1 as an imput and outputs it (. where (Ur, Re) is the rest point of the single-round DRDA starting from (Jo, To) under the specified e and Tbase mo.
To examine if multi-round DRDA can find the exact Nash equilibrium rather than QRE, we also need to define the counterpart of single-round DRDA under = O. We call it Discounted Aggregation (DA), which corresponds to the problem 1/ = (Fee) - vi under the restriction:
N5 x1(x²) E arg max (A))))A(A)
(<)() v(("))) Vax x
(8)
Note that the hardmax policy selection above implies that oे is satisfactory policy when only the actions with the highest score y/ are assigned non-zero probability. In Section 4 we will see that every NE in the POSG actually corresponds to solution of DA. Furthermore, when the solution (rest point) of DA is unique, the POSG has a unique NE corresponding to the unique rest point.
4 THEORETICAL ANALYSIS
In this section, we examine the convergence properties of DRDA. We will characterize the rest point of single-round DRDA with QRE concept in POSGs and prove hypomonotinicity-baae
condition for single-round DRDA to converge to the rest point at a linear rate. Furthermore, we will show that under the uniqueness assumption DA rest point, the limit point of multi-round DRDA is actually an exact Nash equilibrium rather than a QRE. All omitted proofs are provided in Appendix
4.1 GENERALIZED NASH DISTRIBUTION
As type of quantal response equilibrium (QRE) (McKelvey & Palfrey 1995). Nash distribution (Leslie & Collins 2005) in NFGs corresponds to a solution concept close to Nash equilibrium but under certain utility perturbation. Here, we first generalize this concept in the context of POSGs.
Definition 4. A generalized Nash distribution (GND) under regularization parameter e 2 0 and base policy bhase is a joint policy th. For any player i E N and any individual policy n² E II::
T-1 Σ
u' u'(
E
DKL (T)( (zi)
Σ ArEat
Pr (helft))
1=0
T-1
n' ( )
DKL (=' (x))
Σ hreaf
Pr (holfts)
m
Note that the GND introduces utility perturbation based on the KL--diergence under Wasse and the visitation probability Pr (h,) (.) under म.. When e = 0, the GND becomes a Nash equilibrium. Also note that when T = S = 0 = 1, the POSG is reduced to an NFG, and the GND is equivalent to the concept of Nash distribution defined in NFGs if Those is set to be a uniform policy. However, please note that the GND defined in POSGs is not the same the Nash distribution defined in the (exponentially large) NFG representation of the POSG.
Now use the following lemma to show an important inequality for GND. This inequality connects the solution concept of GND with the rest point of single-round DRDA, Its proof involves applying some existing results about action values to POSGs (see Appendices B. and

---

Published as a conference paper at ICLR 2025
Lemma 3. Given GND R. under and Tbases it holds for any player E N, any individual policy T' II, and any individual decision point zi E X²:
Σ Pr ( NeEn
Σ
n T((r,, A. (lee a/) e DRr. (#4 (z=))
Σ NgEa
Pr (
Σ
n (x/ A/s (he,af) - =DuL. (x())ll
The inequality above is further used to prove the following theorem, which claims that every GND must correspond to rest point of our learning dynamic. This theorem will be subsequently used to relate the rest point back to the solution concept of Nash equilibrium.
Theorem 1. In POSG, every GND tt, under
induces rest point (B(*.), A.) in single-round DRDA, and every GND F. under e = 0 (i.e., Nash equilibrium) induces a DA rest point (ū(T..) T.).
4.2 CONVERGENCE OF SINGLE-ROUNT DRDA
Now we analyze the conditions for single-round DRDA to converge to its rest point in POSGs. Our analysis primarily relies on two properties: the strong convexity of the penalty function and the local hypomonotonicity of the game. First, we need to show that, similar to other forms regularization used in FTRL,, the divergence regularization used in single-round DRDA 5 is 1-strongly convex.
Definition 5. A function f(-) on compact convex set C is K -trrongyy convex for any x' E C and any 3 [0, 1], f(az + (1 a)2²) S
a)f(==) - 1j a)ll= 2'1??
Lemma 4. DKL (M|1I)) = E. T (a)) log x(a) is 1-sirongly with respect to under & norm.
p(a)
Second, we need to examine the hypomonotonicity of the game. This property is a bit different from cocoercivity, which is also used, e.g., in analyzing the convergence of the extragradient method (Gorbunov al., (022a). The concept of hypomonotonicity relaxes monotonicity, a more strict but widely used concept in the existing convergence analyses (Gorbunov et al. 2022b Cai et al. 2022
Definition 6. An operator F; Rd - I is called X-hypomonotone (A 2 0) ffr any x, x' E IR:
(x x' F(x) F(x') 2 -1Il2 11lx->111 2'112
Note that being o-hypomonotone means being monotone. With the hypomonotonicity of a proper value operator, we can define the hypomonotonicity of a POSG on local policy set.
Definition 7. POSG is locally X-hypomonotone (A 2 0) on policy Nocat C TT if the negative advantage value - v(-) is X-hypomonotone (A 2 0) on Nioca-- l.e., for any गı, T2 E Tooeat
( - (² (Fae) S
Σ EEN
(10)
IEN
where the inner product sums over all (ks lu, at)) triples.
Using the strong convexity of the penalty function, we can now derive a general condition for the linear convergence of single-round DRDA towards its rest point under local hypomonotonicity.
Theorem 2. Assume that rest point (Te, Fe) of single-round DRDA under >> 0 is contained in
positively invariant compact set Q that induces a local policy set IIsoca = (o(T) ÿ E )), where >-hopomonotonicity is satisfied. Ife > X, then the policy of the DRDA starting from an arbitrary To E Ω will converge to सt at a linear rate. Specifically, there exists a non-negative energy function ((ie satisfying ((T) 22 V)(i) with मे = o(T) = म. when V( = 0.
The energy function V(Te)) is constructed based on a concept called Fenchel coupling Mertikopoulos & Zhou, 2019), which has lower bound under a strongly convex penalty function (e.g., KL- divergence). We decompose the derivative of the energy function into one term that can be upper- bounded using the assumption of hypomonotonicity and two negative Fenchel coupling terms. Under
7

---

Published as a conference paper at ICLR 2025
<> X, the derivative is non-positivy and the energy function approaches zero exponentially fast. We complete the proof by showing that the policy at the points with zero energy is exactly me.
Note that local hypomonotonicity only requires the inequality (10) to hold in a local region of policies 1otaa C. II. When Hocll = 1I, we say the game is globally hypomonotone. Actually, the following proposition guarantees that global lypomonnttoniccty is common property for at least all NFGs. Proposition 1. Every NFG is globally X-hypomonotone under sufficientll large X.
To further demonstrate that typomoootonicctty can still be a realistic assumption when the game is dynamic, we also numerically estimate the order of magnitude of the global hypomonotonicity value by sampling Eier (x] (8:)- (1a)) We provide the numerical results Appendix D
Eiex -1l
4.3 RELATIONSHIP TO NASH EQUILIBRIUM
Recall that, under e O, Theorem D limplies that every Nash equilibrium induces a rest point of DA Thus, the existence of the DA rest point can be guaranteed by the existence of NE. Furthermore, the rest point unique, then the policy at the rest point must be the unique Nash equilibrium of the game. Under this uniqueness assumption, the problem of finding Nash equilibrium can be reduced to finding the rest point of DA. Formally, we have the following theorem:
(8.
Theorem 3. If the rest point (Te; H.) in DA is unique, then म. corresponds an exact Nash equilibrium. Under >> O and Tbaser if the rest point (ūr, F.)) in single-round DRDA is unique, and there exists generalized Nash distribution in the POSG, then the NashConv of î.y is at most max en log K. E.n,, where IC iEN,z'EX' .EA Thaze [a)"aa"
Theorem 3 relates the DRDA rest point to Nash equilibrium under the assumption of its uniqueness. To find an approximate Nash equilibrium, we theoretically require the regularization parameter to be close to zero. the POSG is (strictly) monotone, then by Theorem 2 we can use an infinitesimal in DRDA while still guaranteeing its convergence since have À 0. In this case, a single round of DRDA is sufficient to find an arbitrarily precise approximate Nash equilibrium.
On the other hand, we can establish more general guarantee for multi-round DRDA (Definition 3) Note that when the rest-point sequence (pt)220 converges, the regularization terms approach zero at the rest points. Using TheoremB] we further prove that the limit point of multi-round DRDA is an NE rather than a QRE under the uniqueness assumption.
Theorem 4. If the policies in the rest-point sequence (pt)120 generated by multi-round DRDA converge (under a properly selected e >> 0 and po), and the rest point of DA is unique, then the corresponding policy sequence must converge to Nash equilibrium.
While Theorem 4 does not provide exact condition for multi-round DRDA converge, it suggests that an arbitrarily precise approximate NE can always be found as long as the overall learning process converges. In Appendix [.2] we also provide some evidence show that the uniqueness assumption could be removed, e.g., when the game is individually concave. In Section 5 we will further show that multi-round DRDA actually converges to Nash equilibrium a fast rate in a variety of games.
EXPERIMENTS
Considering the representation capability of the POSG framework. DRDA can be applied to broad class of finite-action games. We implement DRDA in a range of games commonly formulated as NFGs, EFGs, MGs, and POSGs to verify the hingle-round/multi-ronnn convergence and equilibrium- finding capability of DRDA. The implementation details, parameter settings, and game descriptions are placed in Appendices E.I E.3 and F respectively. Since the per-iteration time complexity of time DRDA (SDRDA; ATgorithm 1 is standard O([?) when we use dynamic programming to compute the advantage value, it is fair to compare its multi-round version (simply referred to as DRDA in the experiment) with other algorithms under the same total iterations.
Among the comparative algorithms, magnetic mirror descent (MMD), which exhibits last-iterant convergence to QREs in two-player zero-sum (strictly) monotone games (Sokota al. 2023).
8

---

Published as a conference paper at ICLR 2025
monto game
(eCT B00 por srdey
ETAL Fast Besill
127
122
107
KND lest caraaa
2004
NoshCo1e
NoshiCov
NanhCoore
10"4
10
0.5
2.5
35
2.5
3.5
4.5
0.5
1.5
2.5
35
45
Saretrrs
-10
Sarators
110
Berado15
Figure 1: NashCony learning curves in NFG
has an intrinsic relationship with single-round DRDA if we consider another way of discretization (see Appendix E.2) Besides, an extended version of MMD uses the idea of moving magnets to find exact Nash equilibrium, while DRDA employs the idea of multi-round learning to achieve the same goal. Since MMD shares certain similarities with DRDA, we use its moving- magnett version (simply referred to as MMD in the experiment) as a common baseline for NFGs and EFGs.
5.1 NORMAL-FORM GAME (NFG)
NFGs are games where each of the players makes one decision simultaneously. As single-round DRDA can be viewed as discounted variant of FTRL NFGs, we compare with vanilla FTRL and the FTRL variant under the optimistic gradient method (OFTRL for short; see see Boone & Mertikopoulos (2024)). We also compare the variant of regret matching under the extragradient method (ERM* for short; see Farina et al. (2024)) and use NashConv as the performance metric for all algorithms. The payoff settings for three NFG scenarios (i.e., 2-action matrix game, 3-action bimatrix game, and 3-action 3-player game) are provided in Appendix F.I and the learning curves shown in Figure []
In the 2-action matrix game and 3-action bimatrix game, while the average iterates of FTRL can converge to Nash equilibrium, its last iterates cannot. OFTRL well as ERM* and MMD, however, exhibit last-iterate convergence to NE at an exponential rate. For DRDA, the last iterates converge to a rest point in each single round separated by the vertical lines. This aligns with the linear convergence result in Theorem 2 Moreover, the rest-point policy sequence converges and approaches NE under multiple learning rounds. This is consistent with the statement in Theorem 4 Besides, the overall convergence rate is exprnexpo since the curve drawn by the stationary policies is roughly linear.
In the 3-player NFG (with 3d 27 joint actions), MMD still converges, and ERM* converges after an oscillation. However, the average iterate of FTRL and the last iterate of OFTRL no longer converge to NE Actually, there no theoretical guarantee for them to work in multiplayer games. In comparison, the single-round/mulli-n
convergence of DRDA is still guaranteed in this scenario.
5.2 EXTENSIVE-FORM GAME (EFG), MARKOV GAME (MG)& & TYPICAL POSG
EFGs are games where each of the players makes multiple sequential decisions based imperfect information about the global history. Since each player in an EFG may have multiple decision points, the size of the action space in the equivalent NFG representation can be exponential in the size of the game tree. Therefore, it is computationally impractical to directly apply some of the NFG- based methods to EFGs without considering the game-tree structure. Extending the idea of regret matching, counterfactual regret minimization (CFR) (Zinkevich et al., 2007) is the most commonly used equilibrium-learinng algorithm in EFGs. Empirically, CFR* (see (Bowling et al. 2015 has an improved average-iterattellst-ttie
performance. Regularized Nash dynamics (R- NaD for short; ) is a mulli--ound learning algorithm that also exhibits last-iterate convergence in EFGs. Here, compare CFR*, (stable) predictive CFR* (PCFR* for short; see Farina et al. 222 la 2024)). extragradient CFR (ECFR* for short), and R-NaD in multiplayer Kuhn poker scenarios. The game details are provided in Appendix F2
see Perolat et al. (2021 2022

---

Published as a conference paper at ICLR 2025
Bibyerr
107
Sypayer podaer
SOCCEF 3301
107
leckll Bast Recsa)
1,000 for lnoes
16
10.
10/13
10.00
Figure 2: NashConv learning curves in (multiplayer) EFG and (infinite-horizon MG
As is shown in Figure (left & mid), while the average iterates of CFR* converge, its last iterates oscillate. R-NaD has a multi-round learning pattern close to DRDA, but the process is much slower and suffers from oscillation in the 4-player scenario. While ECFR and MMD have impressive performance in NFGs, they do not work when it comes to multiplayer EFGs. PCFR* exhibits certain last-iterate convergence in the 3-player case but oscillates when comes players. The 20-round DRDA clearly outperforms the other algorithms in 3 8-player/4-playee Kuhn poker.
Markov games can be regarded as a special case of POSGs, where each player has full observation of the global state. The Markovian property guarantees that it is sufficient to use current states to represent histories and decision points. Since there are finite distinct decision points, tabular DRDA can solve infinite-horizon horizon MGs like the soccer game (see Appendix F.3), where we compare DRDA with three other general methods of equilibrium learning: FP (Brown) 1),1), PSRO (Lanctot et al., 2017), and ED (Lockhart et al. 2019). While both FP and PSRO require preserving history policies, ED only uses last-iterate policies like DRDA. Since the evaluation of value functions requires repeated dynamic programming in each iteration, we only run total of 100 iterations (5 rounds for DRDA) to save time. As shown Figure 2 (right), while the 5-round DRDA has not converged in each single round of 20 iterations, the overall convergence is significantly faster than the comparative algorithms.
Besides EFG and MG, we also run multi-round DRDA in typical POSGs like the tiger games (see Wiggers 2015)) and find that it consistently outperforms the existing methods like heuristic search value iteration (HSVI) and sequence form linear program (SFLP). The comparison details are placed in Appendix [:4 Figure 5 provides the NashConv learning curves of 10-round DRDA under different regularization parameters e {0.05, 0.1, 0.15] in Adversarial Tiger with time horizon H E {2, 4).
6 CONCLUSION
In this paper, a multi-round equlibrium-learning system, based on discounted FTRL, and named DRDA, is introduced under the framework of POSG. We define a generalized Nash distribution in POSGs and further show that the rest point of single-round DRDA can be characterized by this QRE concept (Theorem DD. We prove linear last-iterate convergence of DRDA its rest point in a single round under the assumption of game hypomonotonicity (Theorem 2. We further prove that the limit point of multi-round DRDA must be Nash equilibrium under the uniqueness assumption (Theorem 3 and Theorem 4.. In our experiments, the discrete-time implementation of multi-round DRDA manages to approach Nash equilibrium in various games represented by POSGs. The last-iterate DRDA is consistent with theory. Besides, like magnetic
convergence of woaggc--oounmmmui-rroooott mirror descent and extragradient regret matching, multi-round DRDA achieves ncar-exponentic rate in NFGs. For EFG, MG, and typical POSG, DRDA outperform the existing baseline algorithms.
Limitation & Future work. One limitation of this work is that we have not theoretically examined the condition for the policy sequence of multi-round DRDA converge despite the simulation results. is also left for future work to examine whether the requirement of unique equilibrium point can be removed in establishing the relationship to Nash equilibrium in multiplayer games.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 62293541 and Grant 62136008, in part by Beijing Natural Science Foundation under Grant 4232056, and in part by Beijing Nova Program under Grant 20240484514.
REFERENCES
Kenshi Abe, Mitsuki Sakamoto, and Atsushi Iwasaki. Mutation-driven follow the regularized leader for last-iterate convergence in zero-sum games. In Uncertainty in Artificial Intelligence, pp. 1-10. PMLR, 2022.
Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, Kentaro Toyoshima, and Atsushi Iwasaki. Last-iterate convergence with full and noisy feedback in two-player zero-sum games. In International Confer- ence on Artificial Intelligence and Statistics, pp. 7999-8028. PMLR, 2023.
Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, and Atsushi Iwasaki. Adaptively perturbed mirror descent for learning in games. In International Conference on Machine Learning, 2024.
Khaled Alomar and Tatjana Chavdarova. On the hypomonotone class of variational incqualities. In OPT 2024: Optimization for Machine Learning, 2024.
Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: meta- algorithm and applications. Theory of computing, 8(1::121-14, 2012.
Anton Bakhtin, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina, Alexan- der H Miller, and Noam Brown. Mastering the game of no-press Diplomacy via human- reguanreggllrizedd reinforcement learning and planning. International Conference on Learning Representations, 2023.
Victor Boone and Panayotis Mertikopoulos. The equivalence of dynamic and strategic stability under regularized learning games. Advances in Neural Information Processing Systems, 36, 2024.
Michael Bowling, Neil Burch, Michael Johanson, and Oskari Tammelin. Heads-up limit hold'en em poker is solved. Science,
2015.
Mario Bravo, David Leslie, and Panayotis Mertikopoulos. Bandit learning in concave N-person games. Advances in Neural Information Processing Systems, 31, 2018.
George W Brown. Iterative solution of games by fictitious play. Act. Anal, Prod Allocation, 13(1): 374, 1951.
Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Finite- last-iterate convergence for learning in multi-player games. Advances in Neural Information Processing Systems. 15:33904-33911 2022.
Shicong Cen, Yuting Wei, and Yuejie Chi. Fast policy extragradient methods for competitive games with entropy regularization. Advances in Neural Information Processing Systems, 34:27952-27964, 2021.
Shicong Cen, Yuejie Chi, Simon S Du, and Lin Xiao. Faster last-iterate convergence of policy optimization in zero-sum Markov games. In International Conference on Learning Representations, 2023.
Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Pierre Coucheney, Bruno Gaujal, and Panayotis Mertikopoulos. Penalty-regulated dynamics and robust learning procedures in games. Mathematics of Operations Research, 40(3):611-633 2015.
Constantinos Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. 10th Innovations in Theoretical Computer Science, 2019.

---

Published as a conference paper at ICLR 2025
Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. Advances in Neural Information Processing Systems, 34:27604-2766 2021.
Aurélien Delage, Olivier Buffet, Jilles Dibangoye, and Abdallah Saffidine. HSVI can solve zero- sum partially observable stochastic games. Dynamic Games and Applications, 14(4):751-805, 2024.
Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Faster game solving via predictive blackwell approachability: Connecting regret matching and mirror descent. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp- 5363-53633333 5371, 2021a.
Gabriele Farina, Christian Kroer, et al. Better regularization for sequential decision spaces fast convergence rates for Nash, correlated, and team equilibria. In EC'21: Proceedings of the 22nd ACM Conference on Economics and Computation, 2021b.
Gabriele Farina, Julien Grand-Clément, Christian Kroer, Chung- Wei Lee, and Haipeng Luo. Regret matching+: (In)stability and fast convergence in games. Advances in Neural Information Processing Systems, 36, 2024.
Monique Florenzano. General equilibrium analysis: Existence and optimality properties of equilibria. Springer Science & Business Media, 2003.
Dian Gadjov and Lacra Pavel. On the exact convergence to Nash equilibrium in hypomonotone regimes under full and partial-decision information. IEEE Transactions on Automatic Control, 68 (0::4539-45533 2023.
Bolin Gao and Lacra Pavel. On passivity, reinforcement learning, and higher order learning in multiagent finite games. JEEE Transactions on Automatic Control, 66(1):121-136, 2021.
Eduard Gorbunov, Nicolas Loizou, and Gauthier Gidel. Extragradient method: O(1/K) last- iterate convergence for monotone variational inequalities and connections with cocoercivity. In International Conference on Artificial Intelligence and Statisties, pp. 366 402.402.1 PMLR, 2022a.
Eduard Gorbunov, Adrien Taylor, and Gauthier Gidel. Last-iterate convergence of optimistic gradient method for monotone variational inequalities. Advances in Neural Information Processing Systems, 15:21858-21883 2870, 2022b.
Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends@ in Optimiza- tion, 2(3-4):157-325, 2016.
Athul Paul Jacob, David Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Jacob Andreas, and Noam Brown. Modeling strong and human gameplay with KL-regularized search. In International Conference on Machine Learning, pp. 9695-9728. PMLR, 2022.
Daphne Koller, Nimrod Megiddo, and Bernhard Von Stengel. Fast algorithms for finding randomized strategies game trees. In Proceedings of the twenty-sixth annual ACM symposium on Theory of computing, pp. 750-759, 1994.
Christian Kroer, Kevin Waugh, Fatma Kılin- Karzan, and Tuomas Sandholm. Faster algorithms for extensive-fore form game solving via improved smoothing functions. Mathematical Programming, 179 (1-2):385 417, 202. 2020.
Harold W Kuhn. simplified two-person poker. Contributions to the Theory of Games, 1:97-103, 1950.
Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. unified game-theoretic approach to multiagent reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
Chung- Ching-Wei Lee, Christian Kroer, and Haipeng Luo. Last-iterate convergence in extensive-form games. Advances in Neural Information Processing Systems, 34:14293- 2021.
David S Leslie and Edmund J Collins. Individual Q-earning in normal form games. SIAM Journal Control and Optimization, 44(2):95-514, 2005.
12

---

Published as a conference paper at ICLR 2025
Pengdeng Li, Shuxin Li, Xinrun Wang, Jakub Cemy, Youzhi Zhang, Stephen McAleer, Hau Chan, and Bo An. Grasper: A generalist pursuer for pursuit- evasion problems. In Proceedings of the 23rd International Conference Autonomous Agents and Multiagent Systems, pp. 1147-1155, 2024.
Mingyang Liu, Asuman Ozdaglar, Tiancheng Yu, and Kaiqing Zhang. The power of regularization in solving extensive-form games. In International Conference Learning Representations, 2023.
Edward Lockhart, Mare Lanctot, Julien Pérolat, Jean-Baptiste Lespiau, Dustin Morrill, Finbarr Timbers, and Karl Tuyls. Computing approximate equilibria in sequential adversarial games by exploitability descent. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 464 464-470, 2019.
Runyu Lu, Yuanheng Zhu, Dongbin Zhao, Yu Liu, and You He. Last-iterate convergence to approxi- mate Nash equilibria in multiplayer imperfect information games. IEEE Transactions on Neural Networks and Learning Systemx, pp. 1-15, 2024. doi 10. T097TNNIS
Richard D McKelvey and Thomas Palfrey. Quantal response equilibria for normal form games. Games and economic behavior, 10(1);6-38 1995.
Panayotis Mertikopoulos and Zhengyuan Zhou. Learning in games with continuous action sets and unknown payoff functions. Mathematical Programming, 173:465-507, 2019.
Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In Proceedings of the Twenty-Ninth Anmual ACM-SIAM Symposium on Discrete Algorithms, pp. 2703-2711 SIAM, 2018.
Hukukane Nikaid6 and Kazuo Isoda. Note on non-cooperative convex games. 1955.
Julien Perolat, Remi Munos, Jean- Baptistt Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, al. From Poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization. In International Conference on Machine Learning, pp. 8525-8535, PMLR, 2021.
Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome Comnor, Neil Burch, Thomas Anthony, al. Mastering the game of Stratego with model-free multiagent reinforcement learning. Science, (7866333) 996000999999000000 2022.
Muhammed Sayin, Kaiqing Zhang, David Leslie, Tamer Basar, and Asuman Ozdaglar. Decentralized Q-learning in zero-sum Markov games. Advances in Neural Information Processing Systems, 34: 18320-18334, 2021.
Shai Shalev-Shwartz and Yoram Singer. Convex repeated games and fenchel duality. Advances in Neural Information Processing Systems, 19, 2006.
Samuel Sokota, Ryan D'Orazio, Zico Kolter, Nicolas Loizou, Marc Lanctot, Ioannis Mitliagkas, Noam Brown, and Christian Kroer. A unified approach to reinforcement learning, quantal re- sponse equilibria, and two-player zero-sum games. In International Conference on Learning Representations, 2023.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Akira Takayama and Takayama Akira. Mathematical economics. Cambridge University Press, 1985.
Zhentao Tang, Yuanheng Zhu Dongbin Zhao, and Simon M. Lucas. Enhanced rolling horizon evolution algorithm with opponent model learning. IEEE Transactions on Games, 15(1):5-15, 2023. To. 1,9//0,,20000022222222
Peter D Taylor and Leo B Jonker Evolutionary stable strategies and game dynamics. Mathematical Biosciences, 40(1-2):145-155 1978.
Chen Yu Wei, Chung- Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In International Conference on Learning Representations, 2021.
13

---

Published as a conference paper at ICLR 2025
Auke Wiggers. Structure in the value function of two-player zero-sum games of incomplete informa- tion. Master's 's Thesis, University of Amsterdam, 2015.
Yuanheng Zhu and Dongbin Zhao. Online minimax Q network learning for two-player zero-sum
Markov games. JEEE Transactions on Neural Networks and Learning Systems, 33(3):1228-1241, 2022.
1469
Yuanheng Zhu, Weifan Li, Mengchen Zhao, Jianye Hao, and Dongbin Zhao. Empirical policy optimization for n-player Markov games. IEEE Transactions on Cyherneties, 53(10):6443 6455, 2023. TO. CWWP/CCBB.022.1112007
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. Advances in Neural Information Processing Systems, 20, 2007.
14

---

Published as a conference paper at ICLR 2025
A LAST-ITERATE CONVERGENCE OF MORE FTRL VARIANTS
Besides discounted updates, the ideas of optimistic gradient methods and extragradient methods can also be employed mitigate the recurrence behavior of FTRL and facilitate last-iterate convergence.
Note that the rest point of ( is (ūr, 7 pair, where Dr is a solution to the variational inequality problem (VIP) of F (1) 0 under F()) ti (ū)). and the discrete-time counterpart of [ can be written as Vi+1 = Bi (T))
Optimistic gradient method. Instead of using the current gradient F()) update, the optimistic gradient method uses ( The update formula then becomes Øc+1 = W (g) 7 This method turns the original FTRL into the optimistic FTRL (OFTRL).
Extragradient method. Instead of using F (3te,, the extragradient method uses the gradient at the point after one update attempt: Fl (z.)). The update formula then becomes M++1 F F (T.)). This turns the original FTRL into what we call extragradient FTRL (EFTRL).
In the field of optimization, both the optimistic gradient method and the extragradient method guarantee (1/ last-iterate convergence to the solution to the VIP under monotonicity assumptions Gorbunov et al. 2022b a). where I denotes iterations. For OFTRL, near-optimal Ó(1) regret bound that corresponds to (1/IZ) convergence rate is also proved (Daskalakis et al. 2021 and last-iterate convergence to Nash equilibrium is guaranteed in two-player zero-sum games under the unique equilibrium assumption (Daskalakis & Panageas (2019). For EFTRL, we find that it exhibits learning behavior similar to OFTRL in various game scenarios.
2022 demonstrate that divergence regularization Pr(h. - log Tbave (x) on rewards can lead to last-iterate convergence
Reward regularization. Besides the gradient methods, Perolat et al. 2021
for FTRL. With this idea, multi-round learning algorithm, regularized Nash dynamics (R-NaD), is also proposed to solve two-player zero-sum monotone EFGs. As DRDA is also multi-round learning system that is capable of dealing with EFGs, R-NaD is directly compared with multi-round DRDA in our experiment section.
15

---

Published as a conference paper at ICLR 2025
B AUXILIARY RESULTS
B.1 RELATED TO ACTION VALUES
The existing research reinforcement learning has established series of theoretical results when analyzing Markov decision processes (MDPs). As POSGs generalize MDPs by introducing multiple players and partial observations, some existing results related to action values in MDPs/POMDPs can also be generalized to POSGs. The following two lemmas will be used our subsequent proofs.
First, the advantage function has well-known property:
Lemma (Property Advantage).
Σ (rf, A4) =
Proof. By definition of the history-based advantage:
A/ (hc, at) Q+ at) V/((t)
Therefore, we have:
Σ
nt (at,
Σ
x' ( at) (Q% at) Vi((h)))
Σ
xe' ( at Q% (he, at) - Vi(le) Σ n (1),
of
(hht Vi(ha) =
Second, the well-known lemma of policy difference also holds in POSGs: Lemma 6 (Policy Difference).
Vi n x (h+) -
V
Σ k=
k-t A3s ab) / T-1
Proof. First, have:
Σ Tx at A's at)
Σ of
m at (hx, V2(hk))
R Ri(sk, des sk+1))
Pr 1h of, #-1) Σ
V2(h+)
Ak-1 3he
16

---

Published as a conference paper at ICLR 2025
Now can expand the RHS expectation:
E
k-t 1 ( ait) MMs mp. #-
7 --E (hx: | [T.. #1
Pr (Ma Tp. St
E ai
mt A4) ok)
Pr (hot2 nt. #) Rt (ski Tkk Sk+1)
hkttahe
h-t
Σ
Pr (hellhe o4. ") Vi(
k=t+1
he hk
Σ
Pr (helthe of. 4 (i(a
he (ha)
Vi n
(hr)
which derives the LHS value difference in the equality.
B.2 RELATED TO INDIVIDUAL CONCAVITY
Definition 8. A finite-action game is individually concave if for any E N, any # II= any प.. 2 II', and any E [0, 1):
u' - a) anj, .- (1 (1- a1- # + oal coal(zz, (m², 1
Note that the individual concavity defined here follows the common definitions in games with continuous action space (Bravo et al. 2018 Cai al. 2022). In the finite-action setting that we consider, the policy space corresponds to the continuous action space. In the context of POSGs, n² = (1 | + amT means that x (a* (x=")) = (1 - a)rf (x', + ant(x', at each decision point of E X The individual concavity is well-defined in the sense that π m2 E III + X² E I1' Also note that when T = |S| 10 = 1, the POSG is reduced to an NFG, and the individual concavity always holds since u' u'(m", Ā is linear in the individual policy n² in NFGs.
Under the assumption of individual concavity, is direct to prove the same NashConv bound as the one in Theorem for the rest point of DA single-round DRDA without relying the uniqueness of the rest point (see Theorem 5). It is therefore mathematically correct as well to replace the uniqueness assumptiio in Theorem 4 with individual concavity. This further suggests the feasibility of applying single-round/multi-ronn
DRDA to broader class of games with multiple Nash equilibria.
Theorem 5. If the POSG is individually concave, then every rest point (Tr, Tt.)) in DA induces a Nash equilibrium म.,, and every rest point (Ør, F.) in single-round DA (under >> 0 and Abase) induces joint policy स, with NashConv at most en log K. E, 7", where i'N,
max
These (x)
Proof. For any i E W' E II', and (0, 1]. define:
मे (a/ :) ((1 a)mf + ax ,,,)))
Using Lemma we have:
Vi x/, (ho) - -V2 (hv)) = E
Σ k=
k-t Al (he,, al) Pee ref 7,¹
17

---

Published as a conference paper at ICLR 2025
Therefore:
w((++) - u ( = Ebuo
Vi
,,, (hho)
- Ebho [ (An)
t Alr. (hes alt) pTp. tt.
Pr ( Σ
क5 oF of o4) (hx oi)
k=
hg
By assumption on the individual concavity:
n' x²( (1-a)u((ft + an', ,,.
Combining the two formulas above, have:
T
Pr (hx |T. N..
Σ Cff(ahe a (hu, af) 2 (u'r,,,, - (F
k=
h2
Using Lemma 5 we have:
Σ
a4)4
Σ (m² - + (hx.
Therefore:
Pr (t. :) Σ (m² (x) A4 (hes at) 2 u ( (T,
h4
oi
Letting + 0+, we have:
T-1
Σ
Pr (hec
Σ (T* - Ab) (hg, at) 2 u (,,",,))))) (TFT)) (11)
0m
of
Since (Ur, ) is a rest point of DA (c = 0) or single-round DRDA (c > 0), we have:
T. = ((r,) (x*) E arg max = (()))A(())
(< v (fx)) DRI. ( (x*) Thase (a())} N
By definition of (()) it holds for any i E N, E Xt:
Pr TT)
Σ ( A')44, (h, a²) eDRL (1l) (2*))
hE.!
2 Σ
Pr (1)
Σ (a" aa) (h) (h, +DoL (o (r'()")) (zx))
kEw
With inequality III
it is clear:
T-1
ut ( ( Se
Pr (hx (.,) (DEL
DKL.
k=0
hx
T
<
E
Pr (hx log < log KC
hu
When 0, the joint policy म, is clearly a Nash equilibrium.
18

---

Published as a conference paper at ICLR 2025
C OMITTED PROOFS
C.I PROOF LEMMA
Proof. Recall that the score ÿ in single-round DRDA (4) has the following integral form:
'v6 a²) + { ( d')dr
(e
When the POSG has a finite horizon, the value V is bounded since the reward functions { are bounded. When the POSG has an infinite horizon, the value Vi is still bounded since we have O 7 1 and Σ n
1 1-q- Therefore, in either case, the advantage A and advantage value
of (TF) are also bounded.
On the other hand, 1od (t = e-'1// = 1 e E [0, Since v' (H+) always bounded, with the basic property of integrals, directly prove the boundedness of the score y7.
C.2 PROOF OF LEMMA 2
Proof. First, we prove:
1 arg ((a) (EA(A)
(raa) log ulaa) erla)
aA
Write the corresponding optimization problem:
maximize
Σ
u(a) (r(a) log)
aeAA
m(a)= =1
a m(a) 2 Va E A
Using the Lagrange multiplier, we have:
L = Σ m(a) (r(a) - log - - E1uon u(a) -
aEA
DEA
ØL 0p(aa
= =(a(aa) ((ogg(() m(a)
=
p(a)
+ H(a)= = (a)- m(a) x (a)
By definition of o²)) 5 we have:
Σ (l a d'l a²) a'EA
xx(x²) o² (vi)(x))=
arg max 7(E))EA(A)
eDLL (T' (lasee
vilu'ta') ,a)) log It' (2(x"a a²) Thase (x),aa) a²)
Σ ( O'EA
arg max (AA))
yi(x'', ((la",.d a²) + log Thase (2tt a²) lllg (x' (x).at)
Σ a'EA
tt (at,aa)
arg max (A(A))
Therefore:
ni(x',a' x exp (Frile',a)) (2) + log whase (1", (os ,.,))
ax)exp = Thase (x', aaeaaa))l exp ( (Fvi(at, at)
19

---

Published as a conference paper at ICLR 2025
C.3 PROOF OF LEMMA 3
Proof. For any i E N and any own information set zi, define कt (,.,,T_)) Let मेz be joint policy that equals H. initially and switches to ff after player reaches orc By definition of GND:
T-1
Ebs V2, (ho)
DEL (o
Σ hr
(hx (Ta)
-
Am
"
T
2 Bho
Vig (1o)
Σ
DeL (e[(a))] (xt))
Σ
Pr (hulttt.)
k=0
by Exr
For both sides, eliminate the terms related to the histories outside of the subtrees whose roots belong to ri we have:
T-1
(he)
k-t
Σ
DEL (a
Σ
Pr (hx
A1E2
k=t
2 ==t
ALEE
T-1
2 Σ
(h+.)
DKL. (1((4)))
(1f)
Σ
Pr (halla))
h
k=!
hyE
For any x² E II' and a E [0, 1]. we let let = (1 - a) + ax and define:
T-1
g(a) = Σ
Pr (hn) E
Σ
DKL (#4(2))) (zf))
Σ
Pr (T)
k=t
ha hn
T-1
Σ
Pr(he V2 (h+) -
k-t
DKL.. (z1( (x+))
Σ
Pr Pr(hx(tt.)
h+E2t
k=t
ALE
Clearly, 9(0) 0 and Vo E (0, 1].) S 0, which yields:
Vg(a) la-o- SO
(12)
Using Lemma we have:
T-1 Σ
Vi (hx) - V4. (he)
x
k-t
Σ hh
Pr + hx/ Σ
(x (f),,)))A4 (hes att
k=
ot
Therefore:
T-1
Pr (he + +x/ Σ Ti(a at) (hx. (Han di))
g(a)) = Σ (hel
Σ
k-t Σ
k=
Nu The
T-1 k=t
(DEL (#f(a)))| (2tt) - DKL (re( ( (ztt))
Σ
Pr (hseltte)
ha Euf
Using Lemma 5 we have:
Σ
कf (ake at) A (hr,, ant) Σ (TT - (1), (he,a
4
20

---

Published as a conference paper at ICLR 2025
Then, compute the derivative:
V Vrr (h+ + hx
Σ
कं (1 ag) (ha, ab)+
T-1
Vg(a)= Σ
Pr (he a..)
Σ
k-t
oj
Pr - hp Σ
(x* ( af) (hes ab)
heE
b=
Na Ahe
a
T-1
Σ
(T' - 7 ab) (log (1 abb) - log Tass (1 att)
Σ
Pr(hustt))
k=t
%
by Eg
Let a + 0+ and simplify the equality:
Vg(a)la-00+
Σ
Σ
Pr Pr (hel
E (T T.) A4). (bxs at)
hEE
log Tr (ah, ab) log Thase (2) ak))
(TT* (x) ab)
Σ hxEnt
Pr (husl
b=t
Note that the following inequality always holds:
Σ
( at log n. (1)
ot)
Σ n' 7'(2 at) l
of
Therefore, with the inequality 12
we have:
Pr (hxl Σ (n² - (x) Ak) (hk, of)
k=t
m Exra
of
T-1
DKL (5 ( (x
Pr (hhl .6
DKL (1t (xth))
k=t
2R De
Exaa
Let = (x+)) for all ot under k 2 ++ and simplify the inequality:
Σ (h)
Σ
कं ( Ai) (hes ai) - +DDL (T.) (2))) (iae
E
at
> Σ
Pr (hel
Σ
T' (zt A3) (be,, at) EDEL (xi))l
h.E
21

---

Published as a conference paper at ICLR 2025
C.4 PROOF OF THEOREM 1
Proof. Let Ti = ((F.) and केe = T.. Compute ( using the definition of ((-)
3
arg max ((A)) (x') ( cDKL (n' (x")]l Thase (2)))
Σ
Pr (r(mm.)) A4. (h,
Σ n ('" aa)) a'EA
hea
eDKL. (T' (T (x¹))
= arg max *'()))A(A))
Σ Pr (hla.)
hEa!
Σ (h(
Σ
a' ((('taa) a') (h, eDAL (x(2))]
= arg max ((()))A(A)
NEW!
AEEEA
By Lemma
3 for any E N E II', " E X
Σ Pr (h)
Σ
nO ( (a))",a')AA A'z (h):a') - cDRL (xt (= (x*))
AEw!
A'EA
(h(
Σ (x² a (l), DrL. (r' (x") (ollbbooo("))
hEzi
aa'AA
For 6 >> 0, is clear that केe = ((%.).
For is clear that T((x') E arg max x (a)))A((A)
(T'()") Vx X
we prove that (f(H.), is a rest point single-round DRDA when 0 and is a rest point of DA when 0.
Since 3 = - -
-
C.5 PROOF OF LEMMA 4
Proof. With respect to & norm ||-II, we show that DKL (TIIM) = E. T(a) log pla) convex function of x under fixed μr, using the widely known result that the negative Gibbs entropy (TT) = Σ. (a)) log π (a) is 1-strongly convex (Example 2 in Shwartz Singer (2006)).
n(a)
is 1-strongly
By definition of strong convexity:
H(an + (1 a)m') S (ll((T) + (1 (T)(( 1 aa - - =
Since H(x) = DKL (T||K) + Σ ((a) log ((a), we have:
(1 - (|m) Σ (an + (1 (1-0))))(aa log m(a)
<aDKI + Σ (a) log u(a))
+ (1 DKL ( [14) + (1 - a) Σ n' (a) l(a)
1
2 a1 -
Simplifying the inequality, have:
DKL (aT + (1 a)rl S DKL (T]lK) + (1 - [KLL (T'lln) a(1 a x
which means that Dia. (<] (n)) is also 1-strongly convex with respect to e, norm ||-111-
22

---

Published as a conference paper at ICLR 2025
C.6 PROOF OF THEOREM 2
Proof. Construct a Fenchel coupling:
T F' =
((T(( eDEL
max of (EAE)AA(A))
K= Aa
T-1 ΣΣ (ct - <DuL (5' (xk))l (x))))))
k= hg
By Lemma 4 KL (x base (xx)) is an e-strongly convex function of x (xt) with respect
l norm. Besides, it is continuous A(A), Therefore, have the following inequality (Proposition 4.3 in Mertikopoulos Zhou 2019)): ( DRL. (#f (x(()) T hasse (xk)) }
max (E())AA(A))
((x( / (et) eDuu (a (r()) (x)e)
1
2 (1o((1))())))
2 (1((() - >><i))l
Enumerating all (k. h&) pairs and summing up the corresponding terms for both sides, we have:
F'(a', v¹) z 2 cllo'(0') - 1'112
(13)
Recall that (')) has an equivalent expression 7 Therefore, the corresponding max function is continuously differentiable with respect to y². Applying the envelope theorem (Theorem I.F.I in Takayama & Akira 1985)), we have:
Vy((e)) max H((D)A(())
((T(()))) cDuL (( = (x))
On the other hand, we have the derivative result:
V (itt ((r1(2)) (x+)) eDRL (T'()))) (=)))))))))))))))))))))))))))) =
Therefore, we have:
('' = d'('") m
For single-round DRDA 44. define a continuously differentiable energy function:
V(T.) Σ (T.
Compute its time derivative:
(( = Σ V y'
(T vi). vi)
NE
(o m -ē ( - vi)
EN
Σ
((T) n. vFxe)) - (5z n:, vi)
EN
where म. -.)).
23

---

Published as a conference paper at ICLR 2025
Since म. o(e)), by definition of Ø(-) (5) and the Fenchel coupling, we have:
(T) Té. vi) F (TT v1)
(DKL (v](2²) bbsse (xi)) DKL (= b (xt))
(m' me.v² v2) F (T v +
Σ Σ (DEL (T((=\)llbase (xk)) DEL (1i(2 (eese (zk))
k=
Summing up the two equations above, have:
(Té vi) + (T²² Te. v2) F ( vi) + F F*
(T
which implies:
(i7 Σ (TTz u - v/.) F' vi) - (xi.
iEN
Since (Te, He) is rest point of single-round DRDA, we have:
((T)
Σ
- ('e))
-
Σ
F
Σ (T(((()
iEN
iEN
iEN
Using the assumption on the local hypomonotonicity
10
and the inequality
13
it holds in Q:
1
V( S A
Σ
|
-2
|
- (A c)
x S
2
iEN
iEN
iEN
Using inequality
13
further have:
(i)) v 2(c - A1 v = 221 2
Note that V(ji) is non-negative. Therefore, with X, have:
((Te S ViTo exp (-2(1 11
which means (iT) approaches zero exponentially fast.
Also note that by
13 V(ū) = 0 Σ (1o(()²)) EEN
- क < = = = n..
As Ω is compact set positively invariant with respect to we prove that for single-round DRDA starting from arbitrary To E Ω, the policy always converges to म. at linear rate.
24

---

Published as a conference paper at ICLR 2025
C.7 PROOF OF PROPOSITION 1
Proof. Consider two joint policies (T²), ,... T²,
"") and ( , ,,") that differ only
in player j's policy. We hypothesize that there always exists constant Ljj such that:
v'(T", m. x²)) =>>//= S ni T² m
)) -
(14)
Now we verify this hypothesis under T= = 1 10| In this case, the POSG is equivalent to an NFG, with of (z)) a) = w (a) - u' ='(
for the unique decision point zd. where w (a) =
u' (má "F is player i's expected utility under pure strategy n/. (-) with x²((a) = 1.
Let ने = (T) T
,"
(m
, L = max Jur ( and we have:
)
o' ( S [+) - - = + (Ft (
- - ( - -((,))
When jj:
(f) - w -0
v (TQ グ3 (1²(a)) - =((0)) s -
( u' ut(+t)l== =
Σ aEA
When * j:
|wo (#) - w (
Σ
ul (T (- 11 - -2' (TT: #4. S |LA
ut ( o t) - r²" (= ,F.
max aed
aEd
[T'(F) x({+) = Σ n" ra) (1(( - u' (T x2,))
|n' (T( - w ( ;F.)))
s
max aed
lEd
Let o = (Tr नं カー मे2 = (T ". and we have:
(Ta, #1 un (Tá क, |w'(Fi) -'(F2)) =
Σ bed
n/ (mb (=( =2(O)) <L|T - -//1|
Therefore, it always holds:
| ( w/ ( |A| u(t) $2/A1//)) +11 S 2/1 1 - n' mf
where the last step follows Cauchy- Senehh--Shhwrzzinnne inequality.
Setting L;- = 2 = 2 2|A
} max
1 [ we can see that the inequality 14) holds. With this
#ell
inequality, we can decompose the inner product (गं - Tÿ. (+1) - ((ea) for two arbitrary joint
policies F = (T een and मे = ( Specifically, we have:
| o' - (F(2)
{mi n vz ( ... -() - o² (m) ... ,x,))
+ ( Th. v² (m) a, ... x²)) v² 2m m2. ...
.. Tf))
+ (Tf - T2z (m²) ... -
-- -
v' (m] ;
.| 2
JEN
25

---

Published as a conference paper at ICLR 2025
where the last step follows Cauchy-Schwarz Schwarz inequality and 14)
Setting À max EEN
Σ [ I.) ). we have:
JEN
Σ (मi T2. (+1) - 1/ (T2))
IEN
E [ T2 2111 oं
m
KEN JEN
IEN JEN
Σ
Σ
(13 L2))
| -
EN
JEN
|
T
iEN
Thus, we prove that every NFG is globally J-hypomonotone, where:
= max EEN
|
max #Ell
(+)) + max #Ell
| (=)
JEN
= LA| 2
max NEN
|u (x)())
+ max well
le (7)
JEN
26

---

Published as a conference paper at ICLR 2025
C.8 PROOF OF THEOREM 3
Proof. By Theorem D every generalized Nash distribution under e >> 0 or = 0 induces a rest point in single-round DRDA DA, respectively. If there exists GND H, that induces the unique rest point (Ur, 7,) in single-round DRDA, then मे must be the unique GND under the corresponding <> 0 and Those. Since Nash equilibrium always exists, if (yr, Ff) the unique DA rest point, then Ty must be the unique GND under e = 0 (i.e., the unique NE).
Therefore, by Definition for any E N and n' E 11::
u' (TT (,))')
u 5
(DKL (m((2))) (xt) DeL (1)) base (x)))
Σ (rolittt)
t=
hg Ex:
For the non-negative KL-divergence terms, we let
max
,AEA Thaso (xt,a) and further have:
DKL. (1t) bbsse (zt))
Σ
x x a) log T (at
Σ
n' (x a) log (1 a) S
aEA
aEA
Σ n' a' log mhose (v) a)
Σ
म (z), log
1
n' (1/, log K log K.
Tass (1t a)
aEA
aEA
Therefore:
T-1
( ='((tr) S
log Σ
Pr (Helltt)
t=0
x
heEr!
T-1
= e K
t=0
Note that Σ Σ
Pr (heltt.)) = (h. = 1, Vt 2 0. Therefore, we have:
27 N.Eo
he
T
NashConv(T
max s'ell
{u'(T', मे v²(,)) S en log
Clearly, when (7r, 7.) is the unique rest point of DA, corresponds to the case 0, which implies that ग. an exact Nash equilibrium.
C.9 PROOF OF THEOREM 4
Proof. By Definition 3 the rest points = (Vi, मt) under different in multi-round DRDA corre- spond to the solutions ÿ ((f) under different policy selections.
Let H. be the limit of the policy sequence (Fi) when / + co. Then, we have:
lim DKL (>)))))|
DKL (+ =
Therefore, as the policy sequence converges, the regularization term DEL.
in
5 approaches zero at the rest point (J(F), #). As a result, (V(Z.), 1, #.) corresponds to rest point of DA. By Theorem 3 the limit policy is Nash equilibrium when DA has unique rest point. [
27

---

Published as a conference paper at ICLR 2025
D GAME HYPOMONOTONICITY
Since hypomonotonicity (see Gao & Pavel 2021);
Gadjov Pavel (2023) less commonly used in the existing literature of game theory than the field of optimization, here we provide numerical simulations to show that it can be practically used to characterize multiplayer dynamic games.
y
The original Kuhn poker has two players with three ranks. Here we numerically test the global hypomonotonieity of its multiplayer variants, 3-player Kuhn poker with 5 ranks and 4-player Kuhn poker with 6 ranks. Both games are also used as test environments in our experiment section.
x104
3-player Kuhn poker
x104
4- 4-player Kuhn poker
5
4
Number
3
Number
2
1
0
-0.6
-0.4
-0.2
0 Value
0.2
0.4
0.6
-0.4
-0.2
0.2
0.4
Value
( Σ (nl -a
Figure Histograms of sampled
seN
in multiplayer Kuhn poker
|| |
(mi (FL)
By Definition the supremum of the computed value
can be an estimate
मं -
JNN
of global hypomonotonicit value À and thus reflect its order of magnitude when the sampling is sufficient. Therefore, we compute the value by randomly sampling 10 joint policy pairs (#1 T2). The histograms of the sampled value are shown in Figure
For 3-player Kuhn poker (with ranks), the computed value is in the range of |-0505566, 0.472063].
For 4-player Kuhn poker (with ranks), the computed value the range of [- [ 0.295455,, 0.349964).
Note that the maximum value is at the level of 0.1, which implies that the order of magnitude of global should not be too large. Therefore, is safe to say the local hypomonotonicity assumption used in our convergence analysis is realistic in multiplayer Kuhn poker. For more game scenarios, we show the results of the maximum and minimum values under the same number samples Table
Table 2: Maximum and minimum sampled values in more game scenarios
Game Scenario
Maximum Value
Minimum Value
2- 2-player Kuhn poker with 3 ranks
1.872479
-1.260188
2- player Kuhn poker with 4 ranks
0.861661
-0.824744
3 3-player Kuhn poker with 4 ranks
0.679113
-0.575718
2- 2-player Ledue poker with 2 suits and 3 ranks
0.022138
-0.019163
3- 3-player Leduc poker with 2 suits and ranks
0.002436
-0.003294
3-player Ledue poker with 3 suits and ranks
0.068833
-0.060182
The maximum values provide estimations for the order of magnitude of the global X. By Theorem 2
as long as À a finite quantity, we can always guarantee the convergence of single-round DRDA by setting sufficiently large On the other hand, Theorems 3 and suggest that a larger leads to larger NE gap for the rest-point policy. This can viewed as a trade-off in the selection of
28

---

Update T (xt, a²) end end Published as a conference paper at ICLR 2025
E IMPLEMENTATION DETAILS
E. DISCRETE- TIME ALGORITHMS
We the discrete-tine time counterpart of practical implementation of single-round DRDA. An iteration variable m in place of the continuous time variable t. With the introduction of a step size r, the single-round algorithm can be viewed an Euler method that solves the original ODE, and an iteration is exactly one step. The pseudocode for single-round DRDA is shown in Algorithm |
Algorithm I: Single-round DRDA (SDRDA)
Input: Initial score No. initial policy no. iteration number M, and regularization parameter
Set Thase 4o for m E {0, 1, ...., M 1} do
Compute all Pr (h|²m) and A (h, a²) for all E N (using dynamic programming)
for i E N do
for x² E 2 do
for a' E A do
Σ Pr( (baa")
Compute of (TTm)((' ,a) hEr
Pr(h|(m)
hEz
Update ii end for a' E Addo do
(2', f a²) (a', a²) + n (v' (T (x', "a'iaat at) (=' .'))
Thane (a' .a") expl & Bint (ce .at))
Σ base (zr expl Vonss (ar .0)
12
14
end
15
end
Output: Last-iterate (UMr FM)
For the multi-round algorithm, we use SDRDA (Algorithm) as the oracle M in Definition The pseudocode for multi-round DRDA is shown in Algorithm
Algorithm 2: Multi-round DRDA (MDRDA)
Input: Round number L, iteration number M, and regularization parameter e Initialize To and Hp for I {0. ... 1} do
11)+1, 11 SDRDA 5,,
end
Output: Last-iterate policy #i
E.2 RELATIONSHIP WITH MAGNETIC MIRROR DESCENT
Magnetic mirror descent (MMD) Sokota et al. 2023) is discrete-time learning dynamic based on mirror descent. Like single-round DRDA, MMD exhibits last-iterate convergence to QRE. Since mirror descent itself can be converted special form of FTRL by introducing a score y, here we try to mathematically compare MMD with DRDA.
For MMD, if we set the mirror map be negative entropy and the step size to be 1, it has the closed-form update formula πz x (T (T)ri," where ग, is the policy at discrete time t, is the magnet policy, 4: the Q-value, and is the regularization temperature. Note that p and qe in MMD are analogous to the base policy Thase and the advantage value v in DRDA, respectively.
When is set to be uniform policy, we have Te+1 oc (m Tे Through recursion, we can write Ek= -+1- qt.
Tt-1 e where y++1
29

---

Published as a conference paper at ICLR 2025
For DRDA, if we also set Those be uniform, it can be generally written as 7t-1 c in discrete time. If we further consider a form of discretization based on the integral formula (6) rather than the original ODE (4) and use Se to replace v(TT),, we directly have yt+1 = Σx= ( (1)++1= qt- Clearly, this score is the same that of MMD when enforce MMD through a specific discretization if we ignore the base policy or the magnet.
Thus, DRDA can be related to
Now use common notation P to indicate the base policy in DRDA. When p moves along time I and is not kept as a fixed uniform policy, however, the update formula T++1 X pre for DRDA
quite different from the update formula F X Pi 1 (T) for moving-magnet MMD. The aeoveementioned relationship can longer be established. That is to say, when employing moving magnet, the behavior of DRDA will be completely different from MMD.
From theoretical perspective, DRDA guarantees last-iterate convergence to QRE in all games with certain hypomonotonicity, while MMD restricted to two-player zero-sum or (strictly) monotone games. When considering finding an exact Nash equilibrium, multi-round DRDA theoretically requires the base policy p = Tbase to be a fixed rest-point policy in each round. In comparison, moving-magnet MMD (see Appendix H.5 of Sokota al. (2023)) directly learns in single round under a changing Pt. and the related paper does not provide theoretical guarantees for this choice.
E.3 PARAMETER SETTINGS
In Table 3 we show the parameter settings in our experiments on the discrete-time DRDA from Appendix B. Under the parameter settings, each single run of DRDA can be finished within one hour using a single Intel Core 12-1FF CPU.
Table Parameter settings DRDA
matrix game
bimatrix game
3-player game
Kuhn-3
Kuhn-4
soccer game
L
10
10
10
20
20
5
M
5000
5000
5000
1 1 10"
x 10"
20
6
0.1
0.1
0.3
0.02
0.015
0.1
1
0.005
0.005
0.005
0.001
0.001
0.002
In principle, an arbitrary regularization parameter can be used as long as it is greater than hypomonotonicity threshold for the game (see Theorem However, our experimental results show that a smaller gap for the rest point of single-round DRDA generally guarantees better performance of multi-round DRDA. Therefore, e should not be too large as well (see Theorem 3}. The setting above provides a moderate choice for and guarantees the performance of DRDA when the step size y is sufficiently small (no greater than the level of 0.01).
In Table we compare the NashConv of DRDA under different 17. When 17 is too large. DRDA will no longer converge (marked as " "-"). Also note that when 7) is overly small, the result can be not as good since DRDA has not reached its rest point in each single round under the same number of iterations. In each game scenario, a range of n can guarantee good performance of DRDA.
Table 4: Last-iterate NashConv of DRDA under different
7
matrix
bimatrix
3-player
Kuhn-3
Kuhn-4
soccer
0.001
1.4 x 10 9
1.9 10 9
0.014
1.1 10 10
1.3 10
6.9 10
0.002
2.0 10-11
1.6 10
2.0 10
1.1 x 10-
1.3 x 10 1
5.7 x 10
0.005
2.1 x 10-11
1.2 x 10-12
3.5 10-5
6.0 x 10-9
9.8 x 10-15
0.01
2.1 x 10-11
3.5 x 10-5
1.6 x 10 10-15
0.1
30

---

Published as a conference paper at ICLR 2025
F EXPERIMENT DETAILS
F.1 NORMAL-FORDD GAME (NFG)
The payoff matrices of the 2-action matrix game (zero-sum) and the 3-action bimatrix game shown in Tables 5 and 6 respectively.
Table 5: Payoff matrix of 2-action matrix game
Player 2, Action
Player 2, Action
Player Action 1
(1,-1)
(0,
Player 1, Action 2
22
(3, -3
Table 6: Payoff matrix of 3-action bimatrix game
Player 2, Action Player 2, Action 2
Player 2, Action 3
Player I, Action
(9, 6)
(3, 7)
(6,
Player I, Action 2
(7, 0)
(8, 2)
(4,
Player 1, Action 3
(2, 4)
(1, 6)
(3, 7)
The payoff tensor of the 3-action 3-player game is shown in Table 7
Table 7: Payoff matrix of 3-action 3-player game
Player 3, Action
Player 3, Action 2
Player 3, Action 3
Player 1&2, Action (1.1)
(9, 5, 5)
(8, 3, 8)
(3, 5, 9)
Player 1&2, Action (1,2)
(9,
5, 7)
(4, 10, 6)
Player 1&2, Action (1,3)
(10, 6,
(6, (6,3,0 0)
(8, 0, L)
Player 1&2, Action (2,1)
(1, 4,
(0, 1, 1)
(1, 2, 8)
Player 1&2, Action (2,2)
(8, 3,
(5,5,6)
[6, 0,0,9)
Player 1&2, Action (2,3)
(4,2,
(1,5,8)
(9,5,0)
Player 1&2, Action (3,1)
(4,9.9)
(9, 9,
(5, 1,8
Player 1&2, Action (3,2)
(6, 5,99
(10, 10, 0)
(8, 3,9. 9)
Player 1&2, Action (3,3) (6, 3, 9)
2,,
(9, 4)
F.2 EXTENSIVE-FORM GAME (EFG)
Since Kuhn poker (Kuhn 1950) is benchmark game from the extensive-form game literature, use its multiplayer variants as EFG test environments. For 3-player Kuhn poker, we increase the number of ranks from 3 to 5. For 4-player Kuhn poker, further increase the number of ranks
F.3 MARKOV GAME (MG)
We use a two-player zero-sum soccer game as the test environment for infinite-horizon MG. Figure an illustration of the game. The two players are marked with A and B. The player who keeps the ball is marked with cycle. Each player can choose an action from "up". "down", "left", "right", and "stay" at each time step. the two players collide after the simultaneous move, then the ball possession exchanges. When the ball carrier moves into the opponent's 's goal, the game terminates. 100. The
The winning player receives reward of +100 and the opponent receives a reward initial state distribution is set to be uniform distribution, and the discount factor is set to be 0.95.
F.4 TIGER GAME (TYPICAL POSG)
Adversarial Tiger and Competitive Tiger are two typical POSGs introduced by Wiggers (2015). We further test multi-round DRDA in the two games. For comparison purposes, 5-round DRDA
31

---

Published as a conference paper at ICLR 2025
goal
B
A
Figure 4: Illustration of soccer game
bumning ruraes Adeesral Tayer (ird)
hurring Eacaes Adbeceralllee (111)
DADA lesming Abeesiiil
12
22
2000 3003 3008 700 H8.0
1020 3003 5530 5030
2002 300E 1930 3000
Figure 5: NashConv learning curves in Adversarial Tiger
(103 iterations per round) under different time horizons H [2, 43 (using fixed parameter setting < 0.1, 0.01 for each game) and record the last-iterate NashConv as well as the time consumed.
According the existing results (reported in Table of Delage al. (2024) among heuristic search value iteration (HSVI), random search, informed search, sequence form linear program (SFLP), and CFR algorithms, SFLP (Kolerr et al. 1994) demonstrates the best performance in the tiger games under H E {2. 3, 4}. However, DRDA clearly outperforms SFLP when we use NashConv as an aligned metric to describe the reported results.
Table 8: Performance comparison between SFLP and DRDA in tiger games
SFLP (the best reported)
DRDA (tested)
Game Scenario
NashConv
Time
NashConv
Time
Adv Tiger H = 2
0.16
< 1 sec
< 10-4
< 1 sec
Adv Tiger H = 3
0.24
< I sec
10-4
< 1 sec
Adv Tiger
0.32
8 sec
0.017(0.010)
8(10) sec
Comp Tiger H
0.24
sec
10
< 1
Comp Tiger /I
3
0.36
48
0.028
3 sec
Comp Tiger H 4
0.48
14 min 0.024
5 min
Note that in Adversarial Tiger with H = 4, the 5-round DRDA actually terminates using 10 seconds, but we also show the NashCony achieved in 8 seconds (without parentheses) for comparison purposes.
32


Reviews:
[{"review_id": "EyYdFL71Ak", "reviewer": "Reviewer_W41z", "summary": "The paper introduces a new variant of multi-round discounted FTRL (called DRDA) which works in general (n-player, general-sum, partially observable, simultaneous or sequential action) games. The paper shows that the new algorithm converges (last-iterate) to a generalized Nash distribution (a Nash equilibrium of a regularized version of the game). Under an assumption (unique rest point), multi-round DRDA is proved to converge to an NE.", "questions": "Q1: Can you say anything about what equilibria is being selected by DRDA compared to other methods?\n\nQ2: How realistic is the local hypomonotonicity assumption?  I don’t think this is addressed in the text.\n\nQ3: How realistic is the single rest point assumption? Is the single rest point a feature of the game, learning dynamics, or both? Do the games in the  experiments section have this property? What happens if they do not have this property?\n\nQ4: The experiments look like single seed runs. Did you run a parameter sweep?\n\nMinor:\n\n* Line 16: “General value function”. I was unsure what this meant at first read through. A Non-zero sum value function?\n* Line 39: Go is more naturally formulated as an EFG. Is there a more natural example for MGs?\n* Line 69: If I am reading this right, “Nash distribution” is distinct from “Nash equilibrium”?\n* Line 109: Is the set of all joint observations ever used? It seems like a strange quantity.\n* Line 169/184/203/…: Please use inline math styling.\n* Line 169: I am curious if using sum over player convs (rather than max) is important? I realize both would be zero at equilibrium. \n* Line 183: Should a have a superscript: a^i? Because it is player i’s action, rather than a joint action?\n* Line 225: Is this not more like a q-value? Why use v rather than q for the notation?\n* Line 299: Does a GND always exist? It is not clear to me since any perturbation away from pi_base will result in: u^i(pi^i, pi^01_*) - u^i(pi_*) <= epsilon * (some negative value)\n\nScore:\n\nI am willing to raise the score after discussion with co-reviewers. Although I like the results, my main concern is how novel the contributions are compared to previous works. I would also like to be reassured that the method routinely converges -- I do not have a good intuition of how strong of an assumption single-resting point is. Furthermore, any answers to the questions I have asked will also likely improve my understanding and appreciation of the work.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "General games are hard. Any work that attempts to learn and prove convergence in this setting is important to the community.\n\nLast-iterate convergence is valuable, especially compared to average-iterate algorithms that are common in the field.\n\nExperimental results look impressive.", "weaknesses": "No discussion on the assumptions.\n\nTabular.\n\nSingle experiment seed runs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a2GBI3fiEq", "reviewer": "Reviewer_RHfE", "summary": "This paper introduces a new algorithm for finding a Nash equilibrium in partially observable stochastic games (POSGs), which include setting with an infinite horizon. \nThe algorithm, abbreviated DRDA, is inspired by previous work on finding epsilon-regularized Nash equilibria, which have limitations of a finite horizon and may not always have convergence guarantees in a multi-player setting.\nThey generalize the epsilon-regularized Nash equilibria to include the discounting factor of infinite-horizon POSGs, to form what they call a generalized Nash distribution (GND).\nDRDA can be formulated as an ODE, which is solved repeatedly in multiple rounds to converge to an NE in the limit.\n\nWhile the theory for convergence requires an assumption that is not clear if it can be satisfied in all games (rest point of DRDA is unique => GND has bounded deviation from NE) and the theory of multi-round DRDA converge is not well established, experiments show that the DRDA converges to a high precision in small games, at a near-exponential rate.\nMoreover, they provide linear convergence bounds under some hypomonotonicity assumptions.", "questions": "- How does multi-round DRDA depend on the gap of single-round DRDA from its rest point? If I understand correctly, this is unknown?\n- How do you explain the \"spikes\" in Figure 1/2 at the beginning of each new round?\n- In Figure 2, 3-player Kuhn, the \"levels\" (ignoring the spikes) are not monotonic for DRDA. I find it surprising. How is it possible it converges overall? Is there a measurable quantity that is monotonic? (perhaps regret? or gap from regularized eq?)\n- Finding NE outside of constant-sum games is PPAD-hard as I'm sure the authors are well aware of. Perhaps it is hard to remove the uniqueness assumption because of that?\n- Can you please run the algorithm on typical POSG games like the tiger, and compare to methods like HSVI?\n- Can you please compare the algorithm also with Smoothed Predictive Regret Matching+ (with restarts)? \n- How much tuning did the experiments require? The discussions is missing.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Paper is well written, given the high complexity of the topic.\n- They show linear convergence if epsilon > lambda, the hypomonotonicity constant. I find the hypomonotonicity intriguing. I think it's likely to  see future follow-ups on this work.", "weaknesses": "- The theorems 1, 2 and 4 look similar to the results found in [Sokota] and [Perolat], extended to the POSGs. I did not investigate the exact differences, but these links deserve more highlights. Connection to MMD is written about in the Appendix, but as the update equations are written in two different notations I found it hard to follow in a limited time.\n- Connected to previous the previous point, in experiments they compare with moving-magnet version of MMD, as L431 \"MMD is close to DRDA\", which they also abbreviate as MMD, while this version is substantially different from the non-moving-magnet base case. (Perhaps MMM would be a better abbreviation.) This makes me concerned that the real innovation of this paper is showing linear convergence under the hypomonotonicity assumption. \n- I would enjoy more thorough discussion of the experiment results, see also the questions.\n- Appendix E: The Euler method used for single-round DRDA -- SDRDA -- has a learning rate whose schedule is not mentioned in the paper and its relation to the rest point convergence in Theorem 2 is not clear. Also, in experiments they used MDRDA, which uses the Euler-based SDRDA. At first, based on the main paper text, I was under the impression that the discretization of Eq (6) is used for DRDA in the experiments, but the actual algorithm used is different. What is considered as an iteration exactly? One step of the Euler algorithm? It would be helpful to correctly label the algorithm in experiments as MDRDA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kWez7LLt9E", "reviewer": "Reviewer_KwQ7", "summary": "This paper introduces Divergence-Regularized Discounted Aggregation (DRDA), a multi-round learning method designed for partially observable stochastic games. DRDA builds on a discounted version of Follow the Regularized Leader, adapting it for POSGs to address imperfect information and infinite horizons. The authors expand on previous work showing last-iterate convergence to quantal response equilibrium by defining a generalized Nash distribution, which extends the QRE concept to POSGs via divergence regularization.", "questions": "1. The paper uses a more general POSG setting compared to traditional EFGs. What are the core challenges unique to this choice, and how do they affect algorithm design?\n2. Multi-round DRDA resembles a variant of Perturb-based methods from previous work [1]. What advantages does DRDA offer over these established approaches?\n3. Can prior methods like R-Nad [2] or APMD [1] be directly applied to POSGs, or are there specific limitations in these algorithms that DRDA addresses?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper builds on prior work by extending last-iterate convergence from NFGs and EFGs to POSGs, with theoretical proofs, broadening the method's applicability.\n2. It introduces the concept of a generalized Nash distribution, enhancing the QRE framework through divergence regularization.\n3. Convergence proofs are provided for both single-round and multi-round DRDA, strengthening the method's theoretical foundation.\n4. Experimental results demonstrate that DRDA outperforms existing methods across various benchmarks.", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yT69mEFNVE", "reviewer": "Reviewer_LUwx", "summary": "The submission studies a regularized variant of FTRL for solving for equilibria. The submission derives continuous-time limit point results for convergence to Nash equilibria in general-sum settings and compares experimentally to a variety of baselines.", "questions": "> On the other hand, \n\nOn the other hand compared to what?\n\n> For general-sum games, the last iterate of a “discounted” variant of FTRL, first examined in Leslie & Collins (2005), is proved to converge to the solution concept of Nash distribution (Coucheney et al., 2015; Gao & Pavel, 2021), a specific form of quantal\nresponse equilibrium (QRE) (McKelvey & Palfrey, 1995)) defined in NFGs.\n\nWhat's the catch here? Does this require exponential time for small quantal shocks?\n\n>  Therefore, the hypomonotonicity assumption used in our convergence analysis should not be regarded as a strong\nassumption.\n\nIs this claim based on experimental results for Kuhn poker? That would strike me as far from comprehensive enough to make such a sweeping claim.\n\n> These underestimates could reflect the order of magnitude of the true global hypomonotonicity value\nλ, which must be an upper bound for the local hypomonotonicity required in Theorem 2.\n\nWhy does it matter to provide an underestimate of an upper bound?\n\nHow did you tune the parameters of the baselines for the experiments?\n\nDid you use Q-values or counterfactual values for R-NaD and MMD for the experiments?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Converging to Nash in general-sum settings is a very difficult problem. Let me also say that I did not attempt to go through the submission's math. I evaluated the submission's soundness, presentation, and contribution as \"fair\" largely because I lack expertise to judge the novelty or correctness of the technical contribution. Similarly, I evaluated the submission as a 6 as I feel I lack the expertise to verify its value. Should other reviewers possess greater expertise I will defer to them, though I do feel confident saying that the writing is unclear in important places and the experimental results are weak. I'm also a bit concerned about the tension between the complexity results for computing equilibria in general-sum games and the claims made in the submission.", "weaknesses": "> solving partially observable stochastic games (POSGs), which unify normal-form games (NFGs), extensive-form games (EFGs),\nand Markov games (MGs).\n\nThis isn't correct. In contrast to EFGs, POSGs can neither express untimeable games nor imperfect-recall games.\n\n> While a wide range of game-theoretic learning dynamics, including no-regret dynamics and best response dynamics, were primarily analyzed in static normal-form games (NFG), many real-world games are dynamic and thus require to be solved under a different game representation\n\nInconsistent tense. Also, the reason they require a different representation is because treating them as NFGs is too expensive, not because they involve state.\n\n>  imperfect information game\n\nRequires hyphen.\n\n>  For example, the perfect information game of Go can be formulated as a Markov game (MG) by ignoring the action\nof the waiting player. The imperfect information game of Texas Hold’em is commonly formulated as\nan extensive-form game (EFG). In view of this requirement,\n\nThese sentences don't flow correctly.\n\n> As an extension of MGs, partially observable stochastic games (POSGs) introduces imperfect information and is capable of unifying NFGs, EFGs, and MGs.\n\nAgain, not true re EFGs.\n\n> forizon\n\n> The existence of the rest point of single-round DRDA is thus proved.\n\nIt's not clear to me why this is a non-trivial result.\n\n> Theorem 1. In a POSG, every GND ⃗π∗ under ϵ > 0 induces a rest point (⃗v(⃗π∗), ⃗π∗) in single-round\nDRDA, and every GND ⃗π∗ under ϵ = 0 (i.e., Nash equilibrium) induces a DA rest point (⃗v(⃗π∗), ⃗π∗).\n\nThis also seems kind of trivial to me.\n\n---\n\nOther weaknesses:\n- The results are continuous time, not discrete time.\n- The games studied in the experiments seem quite limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)