Title: A Finite-Sample Analysis of Payoff-Based Independent Learning in Zero-Sum Stochastic Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: Zero-sum stochastic games

payoff-based independent learning

best-response-type dynamics

finite-sample analysis
Abstract: In this work, we study two-player zero-sum stochastic games and develop a variant of the smoothed best-response learning dynamics that combines independent learning dynamics for matrix games with the minimax value iteration for stochastic games. The resulting learning dynamics are payoff-based, convergent, rational, and symmetric between the two players. Our theoretical results present to the best of our knowledge the first last-iterate finite-sample analysis of such independent learning dynamics. To establish the results, we develop a coupled Lyapunov drift approach to capture the evolution of multiple sets of coupled and stochastic iterates, which might be of independent interest.

A Finite-Sample Analysis of Payoff-Based Independent Learning in Zero-Sum Stochastic Games

Abstract

In this work, we study two-player zero-sum stochastic games and develop a variant of the smoothed best-response learning dynamics that combines independent learning dynamics for matrix games with the minimax value iteration for stochastic games. The resulting learning dynamics are payoff-based, convergent, rational, and symmetric between the two players. Our theoretical results present to the best of our knowledge the first last-iterate finite-sample analysis of such independent learning dynamics. To establish the results, we develop a coupled Lyapunov drift approach to capture the evolution of multiple sets of coupled and stochastic iterates, which might be of independent interest.

1 Introduction

Recent years have seen remarkable successes in reinforcement learning (RL) in a variety of applications, such as board games [1], autonomous driving [2], robotics [3], and city navigation [4]. A common feature of these applications is that there are multiple decision makers interacting with each other in a common environment. Although empirical successes have shown the potential of multi-agent reinforcement learning (MARL) [5, 6], the training of MARL agents is largely based on heuristics and parameter tuning and, therefore, is not always reliable. In particular, many practical MARL algorithms are directly extended from their single-agent counterparts and lack guarantees because of the adaptive strategies of multiple agents.

A growing literature seeks to provide theoretical insights to substantiate the empirical success of MARLand inform the design of efficient and provably convergent algorithms. Work along these lines can be broadly categorized into work on cooperative MARL where agents seek to reach a common goal [7-10], and work on competitive MARL where agents have individual (and possibly misaligned) objectives [11-22]. While some earlier work focused on providing guarantees on the asymptotic convergence, the more recent ones share an increasing interest in understanding the finite-time/sample behavior. This follows from a line of recent advances in establishing finite-sample guarantees of single-agent RL algorithms, see e.g., [23-26] and many others.

In this paper, we focus on the benchmark setting of two-player 1 zero-sum stochastic games, and develop best-response-type learning dynamics with provable finite-sample guarantees. Crucially, our learning dynamics are independent (requiring no coordination between the agents in learning) and rational (each agent will converge to the best response to the opponent if the opponent plays an (asymptotically) stationary policy [27]), and therefore capture learning in settings with multiple game-theoretic agents. Indeed, learning dynamics with self-interested agents should not enforce information communication or coordination among agents. Furthermore, we focus on the more challenging but practically relevant setting of payoff-based learning, where each agent can only

observe their realized payoff at each stage, without observing the policy or even the action taken by the opponent. For learning dynamics with such properties, we establish to the best of our knowledge the first last-iterate finite-sample guarantees. We detail our contributions as follows.

1.1 Contributions

We first consider zero-sum matrix games and provide the last-iterate finite-sample guarantees for the smoothed best-response dynamics proposed in [28]. Then, we extend the algorithmic idea to the setting of stochastic games and develop an algorithm called value iteration with smoothed best-response dynamics (VI-SBR) that also enjoys last-iterate finite-sample convergence.

Two-Player Zero-Sum Matrix Games. We start with the smoothed best-response dynamics in [28] and establish the last-iterate finite-sample bounds when using stepsizes of various decay rates. The result implies a sample complexity of O ( ϵ -1 ) in terms of the last iterate to find the Nash distribution [29], which is also known as the quantal response equilibrium in the literature [30]. To our knowledge, this is the first last-iterate finite-sample result for best-response learning dynamics that are payoff-based, rational, and symmetric in zero-sum matrix games.

Two-Player Zero-Sum Stochastic Games. Building on the algorithmic ideas for matrix games, we develop best-response-type learning dynamics for stochastic games called VI-SBR, which uses a single trajectory of Markovian samples. Our learning dynamics consist of two loops and can be viewed as a combination of the smoothed best-response dynamics for an induced auxiliary matrix game (conducted in the inner loop) and an independent way of performing minimax value iteration (conducted in the outer loop). In particular, in the inner loop, the iterate of the outer loop, i.e., the value function, is fixed, and the players learn the approximate Nash equilibrium of an auxiliary matrix game induced by the value function; then the outer loop is updated by approximating the minimax value iteration updates for the stochastic game, with only local information.

We establish the last-iterate finite-sample bounds for VI-SBR when using both constant stepsizes and diminishing stepsizes of O (1 /k ) decay rate. To the best of our knowledge, this appears to be the first last-iterate finite-sample analysis of best-response-type independent learning dynamics that are convergent and rational for stochastic games. Most existing MARL algorithms are either symmetric across players, but not payoff-based, e.g., [31-35], or not symmetric and thus not rational, e.g., [14, 36-38], or do not have last-iterate finite-time/sample guarantees, e.g., [39, 15, 40].

1.2 Challenges & Techniques

The main challenge in analyzing our learning dynamics is that it maintains multiple sets of stochastic iterates and updates them in a coupled manner. To overcome this challenge, we develop a novel coupled Lyapunov drift approach . Specifically, we construct a Lyapunov function for each set of the stochastic iterates and establish a Lyapunov drift inequality for each. We then carefully combine the coupled Lyapunov drift inequalities to establish the finite-sample bounds. Although a more detailed analysis is provided in the appendices, we briefly give an overview of the main challenges in analyzing the payoff-based independent learning dynamics in stochastic games, as well as our techniques to overcome them.

Time-Inhomogeneous Markovian Noise. The fact that our learning dynamics are payoff-based presents major challenges in handling the stochastic errors in the update. In particular, due to the best-response nature of the dynamics, the behavior policy for sampling becomes time-varying. In fact, the sample trajectory used for learning forms a time-inhomogeneous Markov chain. This makes it challenging to establish finite-sample guarantees, as time-inhomogeneity prevents us from directly exploiting the uniqueness of stationary distributions and the fast mixing of Markov chains. Building on existing work [23, 24, 41, 42], we overcome this challenge by tuning the algorithm parameters (in particular, the stepsizes) and developing a refined conditioning argument.

Non-Zero-Sum Payoffs Due to Independent Learning. As illustrated in Section 1.1, the inner loop of VI-SBR is designed to approximately learn the Nash equilibrium of an auxiliary matrix game induced by the value functions for the two players, which we denote by v 1 t and v 2 t , where t is the iteration index of the outer loop. Importantly, the value functions v 1 t and v 2 t are maintained individually by players 1 and 2 , and therefore do not necessarily satisfy v 1 t + v 2 t = 0 due to independent learning. As a result, the auxiliary matrix game from the inner loop does not necessarily

admit a zero-sum structure during learning. The error induced from such a non-zero-sum structure appears in existing work [15, 43], and was handled by designing a novel truncated Lyapunov function. However, the truncated Lyapunov function was sufficient to establish the asymptotic convergence, but did not provide the explicit rate at which the induced error goes to zero. To enable finite-sample analysis, we introduce ∥ v 1 t + v 2 t ∥ ∞ as a Lyapunov function in our coupled Lyapunov framework, which is customized to capture the behavior of the induced error from the non-zero-sum structure of the inner-loop auxiliary matrix game.

Coupled Lyapunov Drift Inequalities. In the existing literature of stochastic iterative algorithms [44, 24, 45, 46], when using a Lyapunov approach for finite-sample analysis, once the Lyapunov drift inequality is established, the finite-sample bound follows straightforwardly by repeatedly invoking the result. However, since our learning dynamics (for stochastic games) maintain multiple sets of stochastic iterates and update them in a coupled manner, the Lyapunov drift inequalities we establish are also highly coupled. Decoupling these Lyapunov inequalities is a major challenge. To overcome it, we develop a systematic approach for decoupling, which crucially relies on a bootstrapping argument. Specifically, we first use the Lyapunov inequalities in a direct way to establish a crude bound for each Lyapunov function. Then, we substitute the crude bounds back into the Lyapunov drift inequalities to obtain tighter bounds. See Appendix B.4 for a more detailed illustration of our decoupling procedure.

1.3 Related Work

Due to space constraints, here we only discuss related work in independent learning in matrix games and stochastic games, and single-agent RL. See Appendix A for a more detailed literature review.

Independent Learning in Matrix Games. Independent learning has been well-studied in the literature on learning in matrix games. Fictitious play (FP) [47] may be viewed as the earliest one of this kind, and its convergence analysis for the zero-sum setting is provided in [48]. Smoothed versions of FP have been developed [49, 50] to make the learning dynamics consistent [51, 52]. It was shown that the behavior of smoothed FP is captured by an ODE known as the smoothed best-response dynamics, which were also studied extensively in the literature [53]. Note that the Lyapunov function used to study the smoothed best-response dynamics is the regularized Nash gap [54, 53, 29], a variant of which is also used in our Lyapunov framework. To make the learning dynamics payoff-based, [28] developed a two time-scale variant of the smoothed best-response dynamics and established the asymptotic convergence. Moreover, no-regret learning algorithms, extensively studied in online learning, can also be used as independent learning dynamics for matrix games [55]. It is known that they are both convergent and rational by the definition in [27], and are usually implemented in a symmetric way. See [55] for a detailed introduction to no-regret learning in games.

Independent Learning in Stochastic Games. For stochastic games, independent and symmetric policy gradient methods have been developed in recent years, mostly for the case of Markov potential games [18, 19, 56]. The zero-sum case is more challenging since there is no single Lyapunov function to capture the learning dynamics (which is why we need to develop a coupled Lyapunov approach with multiple Lyapunov functions), such as the potential function in Markov potential games. For non-potential game settings, symmetric variants of policy gradient methods have been proposed, but have only been studied under the full-information setting without finite-sample guarantees [31, 32, 57, 33-35], with the exception of [58, 59]. However, the learning algorithm in [58] requires some coordination between the players when sampling, and is thus not completely independent; that in [59] is extragradient-based and needs some stage-based sampling process that also requires coordination across players. Best-response-type independent learning for stochastic games has been studied recently in [39, 15, 43, 60, 40, 61, 62], with [15, 43, 40, 61] tackling the zero-sum setting. However, only asymptotic convergence was established in these works, which motivated this work.

Finite-Sample Analysis in Single-Agent RL. The most related works (in single-agent RL) to our paper are those that perform finite-sample analysis for RL in infinite-horizon discounted Markov decision processes following a single trajectory of Markovian samples. See [63, 23, 41, 24-26, 6467, 10] and the references therein. Among these works, [23, 24] established finite-sample bounds for temporal difference (TD)-learning (with linear function approximation), and [25, 64, 65] established finite-sample bounds for Q -learning. In both cases, the behavior policy for sampling is a stationary policy. For non-stationary behavior policies as we consider, [41] established finite-sample bounds for SARSA, an on-policy variant of Q -learning, and [10] provided finite-sample bounds for offpolicy actor-critic, which is established based on a general framework of contractive stochastic approximation with time-inhomogeneous Markovian noise.

2 Zero-Sum Matrix Games

We begin by considering zero-sum matrix games. This section introduces algorithmic and technical ideas that are important for the setting of stochastic games. For i ∈ { 1 , 2 } , let A i be the finite action space of player i , and let R i ∈ R |A i |×|A -i | (where -i denotes the index of player i 's opponent) be the payoff matrix of player i . Note that in a zero-sum game we have R 1 + R ⊤ 2 = 0 . Since there are finitely many actions for each player, we assume without loss of generality that max a 1 ,a 2 | R 1 ( a 1 , a 2 ) | ≤ 1 . Furthermore, we denote A max = max( |A 1 | , |A 2 | ) .

The decision variables here are the policies π i ∈ ∆( A i ) , i ∈ { 1 , 2 } , where ∆( A i ) denotes the probability simplex supported on A i . Given a joint policy ( π 1 , π 2 ) , the expected reward received by player i is E A i ∼ π i ,A -i ∼ π -i [ R i ( A i , A -i )] = ( π i ) ⊤ R i π -i , where i ∈ { 1 , 2 } . Both players aim to maximize their rewards against their opponents. Unlike in the single-agent setting, since the performance of player i 's policy depends on its opponent -i 's policy, there is, in general, no universal optimal policy. Instead, we use the Nash gap and the regularized Nash gap as measurements of the performance of the learning dynamics, as formally defined below.

Definition 2.1 (Nash Gap in Matrix Games) . Given a joint policy π = ( π 1 , π 2 ) , the Nash gap NG ( π 1 , π 2 ) is defined as NG ( π 1 , π 2 ) = ∑ i =1 , 2 max ˆ π i ∈ ∆( A i ) (ˆ π i -π i ) ⊤ R i π -i .

Note that NG ( π 1 , π 2 ) = 0 if and only if ( π 1 , π 2 ) is in a Nash equilibrium of the matrix game (which may not be unique), in which no player has the incentive to change its policy.

Definition 2.2 (Regularized Nash Gap in Matrix Games) . Given a joint policy π = ( π 1 , π 2 ) and a constant τ > 0 , the entropy-regularized Nash gap NG τ ( π 1 , π 2 ) is defined as NG τ ( π 1 , π 2 ) = ∑ i =1 , 2 { max ˆ π i ∈ ∆( A i ) (ˆ π i -π i ) ⊤ R i π -i + τν (ˆ π i ) -τν ( π i ) } , where ν ( · ) is the entropy function defined as ν ( π i ) = -∑ a i ∈A i π i ( a i ) log( π i ( a i )) for i ∈ { 1 , 2 } .

A joint policy ( π 1 , π 2 ) satisfying NG τ ( π 1 , π 2 ) = 0 is called the Nash distribution [29] or the quantal response equilibrium [30], which, unlike Nash equilibria, is unique in zero-sum matrix games. As τ approaches 0 , the corresponding Nash distribution approximates a Nash equilibrium [68].

2.1 The Learning Dynamics in Zero-Sum Matrix Games

We start by presenting in Algorithm 1 (from the perspective of player i , where i ∈ { 1 , 2 } ) the independent learning dynamics for zero-sum matrix games, which was first proposed in [28]. Given τ > 0 , we use σ τ : R |A i | ↦→ R |A i | for the softmax function with temperature τ , that is, [ σ τ ( q i )]( a i ) = exp( q i ( a i ) /τ ) / ∑ ˜ a i ∈A i exp( q i (˜ a i ) /τ ) for all a i ∈ A i , q i ∈ R |A i | , and i ∈ { 1 , 2 } .

Algorithm 1 Independent Learning Dynamics in Zero-Sum Matrix Games

1: Input: Integer K , initializations q i 0 = 0 ∈ R |A i | and π i 0 = Unif ( A i ) .

2: for k = 0 , 1 , · · · , K -1 do

3: π i k +1 = π i k + β k ( σ τ ( q i k ) -π i k )

4: Play A i k ∼ π i k +1 ( · ) (against A -i k ), and receive reward R i ( A i k , A -i k )

5: q i k +1 ( a i ) = q i k ( a i ) + α k 1 { a i = A i k } ( R i ( A i k , A -i k ) -q i k ( A i k ) ) for all a i ∈ A i

6: end for

To make this paper self-contained, we next provide a detailed interpretation of Algorithm 1, which also motivates our algorithm for stochastic games in Section 3. At a high level, Algorithm 1 can be viewed as a discrete and smoothed variant of the best-response dynamics, where each player constructs an approximation of the best response to its opponent's policy using the q -function. The update for the q -function is in the spirit of the TD-learning algorithm in RL [69].

The Policy Update. To understand the update equation for the policies (cf. Algorithm 1 Line 3 ), consider the discrete version of the smoothed best-response dynamics:

In Eq. (1), each player updates its policy π i k incrementally towards the smoothed best response to its opponent's current policy. While the dynamics in Eq. (1) provably converge for zero-sum matrix

games, see e.g., [70], implementing it requires player i to compute σ τ ( R i π -i k ) . Note that σ τ ( R i π -i k ) involves the exact knowledge of the opponent's policy and the reward matrix, both of which cannot be accessed in payoff-based independent learning. This leads to the update equation for the q -functions, which estimate the quantity R i π -i k that is needed for implementing Eq. (1).

The q -Function Update. Suppose for now that we are given a stationary joint policy π = ( π 1 , π 2 ) . Fix i ∈ { 1 , 2 } , the problem of player i estimating R i π -i can be viewed as a policy evaluation problem, which is usually solved with TD-learning in RL [69]. Specifically, the two players repeatedly play the matrix game with the joint policy π = ( π 1 , π 2 ) and produce a sequence of joint actions { ( A 1 k , A 2 k ) } k ≥ 0 . Then, player i forms an estimate of R i π -i through the following iterative algorithm:

with an arbitrary initialization q i 0 ∈ R |A i | , where α k > 0 is the stepsize. To understand Eq. (2), suppose that q i k converges to some ¯ q i . Then Eq. (2) should be 'stationary' at the limit point ¯ q i in the sense that E A i ∼ π i ( · ) ,A -i ∼ π -i ( · ) [ 1 { a i = A i } ( R i ( A i , A -i ) -¯ q i ( A i ))] = 0 for all a i ∈ A i , which implies ¯ q i = R i π -i , as desired. Although Eq. (2) is motivated by the case when the joint policy ( π 1 , π 2 ) is stationary, the joint policy π k = ( π 1 k , π 2 k ) from Eq. (1) is time-varying. A natural approach to address this issue is to make sure that the policies evolve at a slower time-scale compared to that of the q -functions, so that π k is close to being stationary from the perspectives of q i k .

Remark. In [28], where Algorithm 1 was first proposed, the authors require β k = o ( α k ) to establish the asymptotic convergence, making Algorithm 1 a two time-scale algorithm. In this work, for finite-sample analysis and easier implementation, we update π i k and q i k on a single time scale with only a multiplicative constant difference in their stepsizes, i.e., β k = c α,β α k for some c α,β ∈ (0 , 1) .

2.2 Finite-Sample Analysis

In this section, we present the finite-sample analysis of Algorithm 1 for the convergence to the Nash distribution [29]. We consider using either constant stepsizes, i.e., α k ≡ α and β k ≡ β = c α,β α , or diminishing stepsizes with O (1 /k ) decay rate, i.e., α k = α/ ( k + h ) and β k = β/ ( k + h ) = c α,β α/ ( k + h ) . Let ℓ τ = [( A max -1) exp(2 /τ ) +1] -1 and L τ = τ/ℓ τ + A 2 max /τ . The requirement for choosing the stepsizes is stated in the following condition.

Condition 2.1. When using either constant or diminishing stepsizes, we choose τ ≤ 1 , α 0 < 2 ℓ τ ,

√

We next state the finite-sample bounds of Algorithm 1. See Appendix C for the proof.

Theorem 2.1. Suppose that both players follow the learning dynamics presented in Algorithm 1, and the stepsizes { α k } and { β k } are chosen such that Condition 2.1 is satisfied. Then we have the following results.

When using constant stepsizes, i.e., α k ≡ α and β k ≡ β , we have

where B in = 4 + 2 τ log( A max ) + 2 A max .

When using α k = α/ ( k + h ) and β k = β/ ( k + h ) , by choosing β > 4 , we have

The convergence bounds in Theorem 2.1 are qualitatively consistent with the existing results on the finite-sample analysis of general stochastic approximation algorithms [44, 46, 24, 65, 23, 42]. Specifically, when using constant stepsizes, the bound consists of a geometrically decaying term (known as the optimization error) and two constant terms (known as the statistical error) that are proportional to the stepsizes. When using diminishing stepsizes with suitable hyperparameters, both the optimization error and the statistical error achieve an O (1 /K ) rate of convergence.

Although Theorem 2.1 is stated in terms of the expectation of the regularized Nash gap, it implies the mean-square convergence of the policy iterates ( π 1 k , π 2 k ) . To see this, note that the regularized Nash gap NG τ ( π 1 , π 2 ) has a unique minimizer, i.e., the Nash distribution and is denoted by ( π 1 ∗ ,τ , π 2 ∗ ,τ ) . In addition, fixing π 1 (respectively, π 2 ), the function NG τ ( π 1 , · ) (respectively, NG τ ( · , π 2 ) ) is a τ -strongly convex function with respect to π 2 (respectively, π 1 ). See Lemma D.7 for a proof. Therefore, by the quadratic growth property of strongly convex functions, we have

As a result, up to a multiplicative constant, the convergence bound for E [ NG τ ( π 1 k , π 2 k )] implies a convergence bound of E [ ∥ π 1 k -π 1 ∗ ,τ ∥ 2 2 ] + E [ ∥ π 2 k -π 2 ∗ ,τ ∥ 2 2 ] .

Based on Theorem 2.1, we next derive the sample complexity of Algorithm 1 in the following corollary. See Appendix C.5 for the proof.

Corollary 2.1.1. Given ϵ > 0 , to achieve E [ NG τ ( π 1 K , π 2 K )] ≤ ϵ , the sample complexity is O ( ϵ -1 ) .

To the best of our knowledge, Theorem 2.1 and Corollary 2.1.1 present the first last-iterate finitesample analysis of Algorithm 1 [28]. Importantly, with only feedback in the form of realized payoffs, we achieve a sample complexity of O ( ϵ -1 ) to find the Nash distribution. In general, for smooth and strongly monotone games, the lower bound for the sample complexity of payoff-based or zeroth-order algorithms is O ( ϵ -2 ) [71]. We have an improved O ( ϵ -1 ) sample complexity due to the bilinear structure of the game (up to a regularizer). In particular, with bandit feedback, the q -function is constructed as an efficient estimator for the marginalized payoff R i π -i k , which can also be interpreted as the gradient. Therefore, Algorithm 1 enjoys the fast O ( ϵ -1 ) sample complexity that is comparable to the first-order method [72].

The Dependence on the Temperature τ . Although our finite-sample bound enjoys the O (1 /K ) rate of convergence, the stepsize ratio c α,β appears as c -1 α,β in the bound. Since c α,β = o ( ℓ τ ) (cf. Condition 2.1) and ℓ τ is exponentially small in τ , the finite-sample bound is actually exponentially large in τ -1 . To illustrate this phenomenon, consider the update equation for the q -functions (cf. Algorithm 1 Line 5 ). Observe that the q -functions are updated asynchronously because only one component (which corresponds to the action taken at time step k ) of the vector-valued q i k is updated in the k -th iteration. Suppose that an action a i is never taken in the algorithm trajectory, which means that q i k ( a i ) is never updated during learning. Then, in general, we cannot expect the convergence of q i k or π i k . Similarly, suppose that an action is rarely taken in the learning dynamics, we would expect the overall convergence rate to be slow. Therefore, the finite-sample bound should depend on the quantity min i ∈{ 1 , 2 } min 0 ≤ k ≤ K min a i π i k ( a i ) , which captures the exploration abilities of Algorithm 1. Due to the exponential nature of softmax functions, the parameter ℓ τ , which we establish in Lemma C.2 as a lower bound of min i ∈{ 1 , 2 } min 0 ≤ k ≤ K min a i π i k ( a i ) , is also exponentially small in τ . This eventually leads to the exponential dependence in τ -1 in the finite-sample bound.

A consequence of having such an exponential factor of τ -1 in the sample complexity bound is that, if we want to have convergence to a Nash equilibrium rather than to the Nash distribution, the sample complexity can be exponentially large. To see this, note that the following bound holds regarding the Nash gap and the regularized Nash gap:

which, after combining with Theorem 2.1, gives the following corollary. For simplicity of presentation, we only state the result for using constant stepsizes.

Corollary 2.1.2. Under the same conditions stated in Theorem 2.1 (1), we have

The last term on the RHS of Eq. (4) can be viewed as the bias due to using smoothed best-response. In view of Eq. (4), to achieve E [ NG ( π 1 K , π 2 K )] ≤ ϵ , we need τ = O ( ϵ ) . Since c α,β appears in the denominator of our finite-sample bound and is exponentially small in τ , the overall sample complexity

for the convergence to a Nash equilibrium can be exponentially large in ϵ -1 . In Appendix F, we conduct numerical experiments to investigate the impact of τ on this smoothing bias.

In light of the discussion before Corollary 2.1.2, the reason for such an exponentially large sample complexity for finding a Nash equilibrium is due to the limitation of using the softmax policies in smoothed best-response for exploration. We kept the softmax policy without further modification to preserve the 'naturalness' of the learning dynamics, which is part of the motivation for studying independent learning in games [73]. A future direction of this work is to remove such an exponential dependence on τ by designing an improved exploration strategy.

3 Zero-Sum Stochastic Games

Moving to the setting of stochastic games, we consider an infinite-horizon discounted two-player zerosum stochastic game M = ( S , A 1 , A 2 , p, R 1 , R 2 , γ ) , where S is a finite state space, A 1 (respectively, A 2 ) is a finite action space for player 1 (respectively, player 2 ), p represents the transition probabilities, in particular, p ( s ′ | s, a 1 , a 2 ) is the probability of transitioning to state s ′ after player 1 taking action a 1 and player 2 taking action a 2 simultaneously at state s , R 1 : S × A 1 ×A 2 ↦→ R (respectively, R 2 : S ×A 2 ×A 1 ↦→ R ) is player 1 's (respectively, player 2 's) reward function, and γ ∈ (0 , 1) is the discount factor. Note that we have R 1 ( s, a 1 , a 2 ) + R 2 ( s, a 2 , a 1 ) = 0 for all ( s, a 1 , a 2 ) . We assume without loss of generality that max s,a 1 ,a 2 | R 1 ( s, a 1 , a 2 ) | ≤ 1 , and denote A max = max( |A 1 | , |A 2 | ) .

Given a joint policy π = ( π 1 , π 2 ) , where π i : S ↦→ ∆( A i ) , i ∈ { 1 , 2 } , we define the local q -function q i π ∈ R |S||A i | of player i as q i π ( s, a i ) = E π [∑ ∞ k =0 γ k R i ( S k , A i k , A -i k ) ∣ ∣ S 0 = s, A i 0 = a i ] for all ( s, a i ) , where we use the notation E π [ · ] to indicate that the actions are chosen according to the joint policy π . In addition, we define the global value function v i π ∈ R |S| as v i π ( s ) = E A i ∼ π i ( ·| s ) [ q i π ( s, A i )] for all s , and the expected value function U i ( π i , π -i ) ∈ R as U i ( π i , π -i ) = E S ∼ p o [ v i π ( S )] , where p o ∈ ∆( S ) is an arbitrary initial distribution on the states. The Nash gap in the case of stochastic games is defined in the following.

Definition 3.1 (Nash Gap in Zero-Sum Stochastic Games) . Given a joint policy π = ( π 1 , π 2 ) , the Nash gap NG ( π 1 , π 2 ) is defined as NG ( π 1 , π 2 ) = ∑ i =1 , 2 ( max ˆ π i U i (ˆ π i , π -i ) -U i ( π i , π -i ) ) .

Similar to the matrix-game setting, a joint policy π = ( π 1 , π 2 ) satisfying NG ( π 1 , π 2 ) = 0 is called a Nash equilibrium, which may not be unique.

Additional Notation. In what follows, we will frequently work with the real vectors in R |S||A i | , R |S||A -i | , and R |S||A i ||A -i | , where i ∈ { 1 , 2 } . To simplify the notation, for any x ∈ R |S||A i ||A -i | , we use x ( s ) to denote the |A i | × |A -i | matrix with the ( a i , a -i ) -th entry being x ( s, a i , a -i ) . For any y ∈ R |S||A i | , we use y ( s ) to denote the |A i | -dimensional vector with its a i -th entry being y ( s, a i ) .

3.1 Value Iteration with Smoothed Best-Response Dynamics

Our learning dynamics for stochastic games (cf. Algorithm 2) build on the dynamics for matrix games studied in Section 2.1, with the additional incorporation of minimax value iteration, a well-known approach for solving zero-sum stochastic games [74].

Algorithmic Ideas. To motivate the learning dynamics, we first introduce the minimax value iteration. For i ∈ { 1 , 2 } , let T i : R |S| ↦→ R |S||A i ||A -i | be an operator defined as

for all ( s, a i , a -i ) and v ∈ R |S| . Given X ∈ R |A i |×|A -i | , we define val i : R |A i |×|A -i | ↦→ R as

Then, the minimax Bellman operator B i : R |S| ↦→ R |S| is defined as [ B i ( v )]( s ) = val i ( T i ( v )( s )) for all s ∈ S , where T i ( v )( s ) is an |A i | × |A -i | matrix according to our notation. It is known that the operator B i ( · ) is a γ - contraction mapping with respect to the ℓ ∞ -norm [74], hence it admits a unique fixed point, which we denote by v i ∗ .

A common approach for solving zero-sum stochastic games is to first implement the minimax value iteration v i t +1 = B i ( v i t ) until (approximate) convergence to v i ∗ [75], and then solve the matrix game max µ i ∈ ∆( A i ) min µ -i ∈ ∆( A -i ) ( µ i ) ⊤ T i ( v i ∗ )( s ) µ -i for each state s to obtain an (approximate) Nash equilibrium policy. However, implementing this algorithm requires complete knowledge of the underlying transition probabilities. Moreover, since it is an off-policy algorithm, the output is independent of the opponent's policy. Thus, it is not rational by the definition in [27]. To develop a model-free and rational learning dynamics, let us first rewrite the minimax value iteration:

where ˆ v ∈ R |S| is a dummy variable. In view of Eq. (5), we need to solve a matrix game with payoff matrix T i ( v i t )( s ) for each state s and then update the value of the game to v i t +1 ( s ) . In light of Algorithm 1, we already know how to solve matrix games with independent learning. Thus, what remains to do is to combine Algorithm 1 with value iteration. This leads to Algorithm 2, which is presented from player i 's perspective, where i ∈ { 1 , 2 } .

Algorithm 2 Value Iteration with Smoothed Best-Response (VI-SBR) Dynamics

1: Input: Integers K and T , initializations v i 0 = 0 ∈ R |S| , q i 0 , 0 = 0 ∈ R |S||A i | , and π i 0 , 0 ( ·| s ) = Unif ( A i ) for all s ∈ S .

2: for t = 0 , 1 , · · · , T -1 do

3: for k = 0 , 1 , · · · , K -1 do

4: π i t,k +1 ( s ) = π i t,k ( s ) + β k ( σ τ ( q i t,k ( s )) -π i t,k ( s )) for all s ∈ S

5: Play A i k ∼ π i t,k +1 ( ·| S k ) (against A -i k ) and observe S k +1 ∼ p ( · | S k , A i k , A -i k )

6: q i t,k +1 ( s, a i ) = q i t,k ( s, a i ) + α k 1 { ( s,a i )=( S k ,A i k ) } ( R i ( S k , A i k , A -i k ) + γv i t ( S k +1 ) -q i t,k ( S k , A i k )) for all ( s, a i )

7: end for

8: v i t +1 ( s ) = π i t,K ( s ) ⊤ q i t,K ( s ) for all s ∈ S

9: S 0 = S K , q i t +1 , 0 = q i t,K , and π i t +1 , 0 = π i t,K

10: end for

Algorithm Details. For each state s , the inner loop of Algorithm 2 is designed to solve a matrix game with payoff matrices T 1 ( v 1 t )( s ) and T 2 ( v 2 t )( s ) for each state s ∈ S , which reduces to Algorithm 1 when (1) the stochastic game has only one state, and (2) v 1 t = v 2 t = 0 . However, in general, since v 1 t and v 2 t are independently maintained by players 1 and 2 , the quantity

is in general non-zero during learning. As a result, the auxiliary matrix game (with payoff matrices T 1 ( v 1 t )( s ) and T 2 ( v 2 t )( s ) ) at state s that the inner loop of Algorithm 2 is designed to solve is not necessarily a zero-sum matrix game, which presents a major challenge in the finite-sample analysis, as illustrated previously in Section 1.2.

The outer loop of Algorithm 2 is an 'on-policy' variant of minimax value iteration. To see this, note that, ideally, we would synchronize v i t +1 ( s ) with π i t,K ( s ) ⊤ T i ( v i t )( s ) π -i t,K ( s ) , which is an approximation of [ B i ( v )]( s ) = val i ( T i ( v i t )( s )) by design of our inner loop. However, player i has no access to π -i t,K in independent learning. Fortunately, the q -function q i t,K is precisely constructed as an estimate of T i ( v i t )( s ) π -i t,K ( s ) , as illustrated in Section 2.1, which leads to the outer loop of Algorithm 2. In Algorithm 2 Line 8 , we set S 0 = S K to ensure that the initial state of the next inner loop is the last state of the previous one; hence Algorithm 2 is driven by a single trajectory of Markovian samples.

3.2 Finite-Sample Analysis

We now state our main results, which, to the best of our knowledge, provide the first last-iterate finite-sample bound for best-response-type payoff-based independent learning dynamics in zero-sum stochastic games. Our results are based on the following assumption.

Assumption 3.1. There exists a joint policy π b = ( π 1 b , π 2 b ) such that the Markov chain { S k } k ≥ 0 induced by π b is irreducible and aperiodic.

One challenge in our finite-sample analysis is that the behavior policies used for taking the actions are time-varying, due to the best-response nature of the dynamics. Most, if not all, existing finitesample guarantees of RL algorithms under time-varying behavior policies assume that the induced Markov chain of any policy, or any policy encountered along the algorithm trajectory, is uniformly geometrically ergodic [41, 76, 77, 59, 78-80]. Assumption 3.1 is weaker, since it assumes only the existence of one policy that induces an irreducible and aperiodic Markov chain.

We consider using either constant stepsizes, i.e., α k ≡ α and β k ≡ β = c α,β β , or diminishing stepsizes of O (1 /k ) decay rate, i.e., α k = α/ ( k + h ) and β k = β/ ( k + h ) = c α,β α/ ( k + h ) , where c α,β ∈ (0 , 1) is the stepsize ratio. In the stochastic-game setting, we redefine ℓ τ = [1 + ( A max -1) exp(2 / [(1 -γ ) τ ])] -1 , which, analogous to the matrix-game setting, is a uniform lower bound on the entries of the policies generated by Algorithm 2 (cf. Lemma D.1). We next state our requirement for choosing the stepsizes.

Condition 3.1. When using either constant or diminishing stepsizes, we choose τ ≤ 1 / (1 -γ ) and the stepsize ratio c α,β to satisfy c α,β ≤ min ( 1 60 L p |S| A max , c τ τ (1 -γ ) 2 34 |S| A 2 max , c τ ℓ 2 τ τ 3 (1 -γ ) 2 144 A 2 max ) , where c τ ∝ ℓ τ and L p > 0 are defined in Appendix B.3. In addition, when using α k ≡ α and β k ≡ β , we require α < 1 /c τ and β < 1 , and when using α k = α/ ( k + h ) and β k = β/ ( k + h ) , we require 2 β = 4 , α > 1 /c τ , and h > 1 such that α 0 < 1 /c τ and β 0 < 1 .

We next state the finite-sample bound of Algorithm 2. For simplicity of presentation, we use a ≲ b to mean that there exists an absolute constant c > 0 such that a ≤ bc .

Theorem 3.1. Suppose that both players follow Algorithm 2, Assumption 3.1 is satisfied, and the stepsizes { α k } and { β k } satisfy Condition 3.1. Then, we have the following results.

When using constant stepsizes, there exists z β = O (log(1 /β )) such that the following inequality holds as long as K ≥ z β :

When using α k = α/ ( k + h ) and β k = β/ ( k + h ) , there exists k 0 > 0 such that the following inequality holds as long as K ≥ k 0 :

Remark. Analogous to [29, 15], our learning dynamics are symmetric between the two players in the sense that there is no time-scale separation between the two players, that is, they both implement the algorithm with the same stepsizes.

Adetailed proof sketch of Theorem 3.1 is provided in Appendix B and the complete proof is provided in Appendix D. Next, we discuss the result in Theorem 3.1 (1). The bound in Theorem 3.1 (1) involves a value iteration error term E 1 , an optimization error term E 2 , a statistical error term E 3 , and a smoothing bias term E 4 due to the use of smoothed best-response in the learning dynamics. Note that E 1 would be the only error term if we were able to perform minimax value iteration to solve

the game. Since minimax value iteration converges geometrically, the term E 1 also goes to zero at a geometric rate. Notably, the terms E 2 and E 3 are orderwise larger compared to their matrix-game counterparts, see Corollary 2.1.2. Intuitively, the reason is that the induced auxiliary matrix game (with payoff matrices T 1 ( v 1 t )( s ) and T 2 ( v 2 t )( s ) ) that the inner loop of Algorithm 2 aims at solving does not necessarily have a zero-sum structure (see the discussion in Section 3.1 after Algorithm 2). Consequently, the error due to such a 'non-zero-sum' structure propagates through the algorithm and eventually undermines the convergence bound.

Recall that in the matrix game setting, we proved convergence to the Nash distribution (or the Nash equilibrium of the entropy-regularized matrix game). In the stochastic-game setting, we do not have convergence to the Nash equilibrium of the entropy-regularized stochastic game. The main reason is that, in order to have such a convergence, our outer loop should be designed to approximate the entropy-regularized minimax value iteration rather than the vanilla minimax value iteration as in Algorithm 2 Line 8 . However, in the payoff-based setting, since each player does not even observe the actions of their opponent, it is unclear how to construct an estimator of the entropy function of the opponent's policy, which is an interesting future direction to investigate.

Although the transient terms in Theorem 3.1 enjoy a desirable rate of convergence (e.g., geometric in T and ˜ O (1 /K 1 / 2 ) in K ), the stepsize ratio c α,β (which is exponentially small in τ ) appears as c -1 α,β in the bound; see Theorem 3.1. Therefore, due to the presence of the smoothing bias (i.e., the term E 4 on the RHS of the bound in Theorem 3.1 (1)), to achieve E [ NG ( π 1 T,K , π 2 T,K )] ≤ ϵ , the overall sample complexity can also be exponentially large in ϵ -1 . This is analogous to Corollary 2.1.2 for zero-sum matrix games. As illustrated in detail in Section 2, the reason here is due to the exploration limitation of using softmax as a means for smoothed best response, which we kept without further modification to preserve the naturalness of the learning dynamics. Removing such exponential factors by developing improved exploration strategies is an immediate future direction.

Finally, we consider the case where the opponent of player i (where i ∈ { 1 , 2 } ) plays the game with a stationary policy and provide a finite-sample bound for player i to find the best response.

Corollary 3.1.1. [Rationality 3 ] Given i ∈ { 1 , 2 } , suppose that player i follows the learning dynamics presented in Algorithm 2, but its opponent player -i follows a stationary policy, denoted by π -i . Then, we have max ˆ π i U i (ˆ π i , π -i ) -E [ U i ( π i T,K , π -i )] ≤ ˜ O ( ω 1 T ( γ +1 2 ) T + ω 2 K 1 / 2 + τ ) , where ω 1 and ω 2 are constants that are exponential in τ -1 , but polynpomial in |S| , A max , and 1 / (1 -γ ) .

Intuitively, the reason that our algorithm is rational is that it performs an on-policy update in RL. In contrast to an off-policy update, where the behavior policy can be arbitrarily different from the policy being generated during learning (such as in Q -learning [81]), in the on-policy update for games, each player is actually playing with the policy that is moving towards the best response to its opponent. As a result, when the opponent's policy is stationary, it reduces to a single-agent problem and the player naturally finds the best response (also up to a smoothing bias). This is an advantage of using symmetric and independent learning dynamics. One challenge of analyzing such on-policy learning dynamics is that the behavior policy is time-varying.

4 Conclusion and Future Directions

In this work, we consider payoff-based independent learning for zero-sum matrix games and stochastic games. In both settings, we establish the last-iterate finite-sample guarantees. Our approach, i.e., the coupled Lyapunov drift argument, provides a number of tools that are likely to be of interest more broadly for dealing with iterative algorithms with multiple sets of coupled and stochastic iterates.

Limitations and Future Directions. As mentioned before Corollary 2.1 and after Theorem 3.1, the convergence bounds involve constants that are exponential in τ -1 , which arise due to the use of the smoothed best response to preserve the naturalness of the learning dynamics. An immediate future direction of this work is to remove such exponential factors by designing better exploration strategies. In the long term, we are interested to see if the algorithmic ideas and the analysis techniques developed in this work can be used to study other classes of games beyond zero-sum stochastic games.

References

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature , 550(7676):354, 2017.

Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. Preprint arXiv:1610.03295 , 2016.

Shangding Gu, Jakub Grudzien Kuba, Yuanpei Chen, Yali Du, Long Yang, Alois Knoll, and Yaodong Yang. Safe multi-agent reinforcement learning for multi-robot control. Artificial Intelligence , 319:103905, 2023.

Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without a map. Advances in neural information processing systems , 31, 2018.

Lucian Busoniu, Robert Babuska, Bart De Schutter, et al. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C , 38(2):156-172, 2008.

Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control , pages 321-384, 2021.

Gürdal Arslan and Serdar Yüksel. Decentralized Q -Learning for Stochastic Teams and Games. IEEE Transactions on Automatic Control , 62(4):1545-1558, 2017.

Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Ba¸ sar. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning , pages 5867-5876, 2018.

Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning of localized policies for multi-agent networked systems. In Learning for Dynamics and Control , pages 256-266. PMLR, 2020.

Yizhou Zhang, Guannan Qu, Pan Xu, Yiheng Lin, Zaiwei Chen, and Adam Wierman. Global convergence of localized policy iteration in networked multi-agent reinforcement learning. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 7(1):1-51, 2023.

Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the Eleventh International Conference on International Conference on Machine Learning , ICML'94, page 157-163, San Francisco, CA, USA, 1994. Morgan Kaufmann Publishers Inc.

Michael L Littman. Friend-or-foe Q -learning in general-sum games. In International Conference on Machine Learning , volume 1, pages 322-328, 2001.

Junling Hu and Michael P Wellman. Nash Q-learning for general-sum stochastic games. Journal of Machine Learning Research , 4(Nov):1039-1069, 2003.

Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in neural information processing systems , 33:5527-5540, 2020.

Muhammed Sayin, Kaiqing Zhang, David Leslie, Tamer Basar, and Asuman Ozdaglar. Decentralized Q -learning in zero-sum Markov games. Advances in Neural Information Processing Systems , 34:18320-18334, 2021.

Yu Bai and Chi Jin. Provable self-play algorithms for competitive reinforcement learning. In International conference on machine learning , pages 551-560. PMLR, 2020.

Qiaomin Xie, Yudong Chen, Zhaoran Wang, and Zhuoran Yang. Learning zero-sum simultaneous-move Markov games using function approximation and correlated equilibrium. In Conference on Learning Theory , pages 3674-3682. PMLR, 2020.

Runyu Zhang, Zhaolin Ren, and Na Li. Gradient play in stochastic games: Stationary points, convergence, and sample complexity. Preprint arXiv:2106.00198 , 2021.

Dongsheng Ding, Chen-Yu Wei, Kaiqing Zhang, and Mihailo Jovanovic. Independent policy gradient for large-scale markov potential games: Sharper rates, function approximation, and game-agnostic convergence. In International Conference on Machine Learning , pages 51665220. PMLR, 2022.

Qinghua Liu, Tiancheng Yu, Yu Bai, and Chi Jin. A sharp analysis of model-based reinforcement learning with self-play. In International Conference on Machine Learning , pages 7001-7010. PMLR, 2021.

Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning-A simple, efficient, decentralized algorithm for multiagent RL. In ICLR 2022 Workshop on Gamification and Multiagent Solutions , 2022.

Constantinos Daskalakis, Noah Golowich, and Kaiqing Zhang. The complexity of Markov equilibrium in stochastic games. In The Thirty Sixth Annual Conference on Learning Theory , pages 4180-4234. PMLR, 2023.

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite-time analysis of temporal difference learning with linear function approximation. In Conference on learning theory , pages 16911692. PMLR, 2018.

R Srikant and Lei Ying. Finite-time error bounds for linear stochastic approximation and TD learning. In Conference on Learning Theory , pages 2803-2830, 2019.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Sample complexity of asynchronous Q-learning: Sharper analysis and variance reduction. IEEE Transactions on Information Theory , 68(1):448-473, 2021.

Zaiwei Chen, Siva Theja Maguluri, Sanjay Shakkottai, and Karthikeyan Shanmugam. Finitesample analysis of contractive stochastic approximation using smooth convex envelopes. Advances in Neural Information Processing Systems , 33, 2020.

Michael Bowling and Manuela Veloso. Rational and convergent learning in stochastic games. In International Joint Conference on Artificial Intelligence , volume 17, pages 1021-1026, 2001.

David S Leslie and Edmund J Collins. Convergent multiple-timescales reinforcement learning algorithms in normal form games. The Annals of Applied Probability , 13(4):1231-1251, 2003.

David S Leslie and Edmund J Collins. Individual Q -learning in normal form games. SIAM Journal on Control and Optimization , 44(2):495-514, 2005.

Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and economic behavior , 10(1):6-38, 1995.

Shicong Cen, Yuting Wei, and Yuejie Chi. Fast policy extragradient methods for competitive games with entropy regularization. Advances in Neural Information Processing Systems , 34:27952-27964, 2021.

Shicong Cen, Yuejie Chi, Simon Shaolei Du, and Lin Xiao. Faster last-iterate convergence of policy optimization in zero-sum Markov games. In The Eleventh International Conference on Learning Representations , 2023.

Runyu Zhang, Qinghua Liu, Huan Wang, Caiming Xiong, Na Li, and Yu Bai. Policy optimization for Markov games: Unified framework and faster convergence. Advances in Neural Information Processing Systems , 35:21886-21899, 2022.

Sihan Zeng, Thinh Doan, and Justin Romberg. Regularized gradient descent ascent for twoplayer zero-sum Markov games. Advances in Neural Information Processing Systems , 35:3454634558, 2022.

Liad Erez, Tal Lancewicki, Uri Sherman, Tomer Koren, and Yishay Mansour. Regret minimization and convergence to equilibria in general-sum Markov games. In International Conference on Machine Learning , pages 9343-9373. PMLR, 2023.

Yulai Zhao, Yuandong Tian, Jason Lee, and Simon Du. Provably efficient policy optimization for two-player zero-sum Markov games. In International Conference on Artificial Intelligence and Statistics , pages 2736-2761. PMLR, 2022.

Kaiqing Zhang, Xiangyuan Zhang, Bin Hu, and Tamer Ba¸ sar. Derivative-free policy optimization for linear risk-sensitive and robust control design: Implicit regularization and sample complexity. Advances in Neural Information Processing Systems , 34:2949-2964, 2021.

Ahmet Alacaoglu, Luca Viano, Niao He, and Volkan Cevher. A natural actor-critic framework for zero-sum Markov games. In International Conference on Machine Learning , pages 307-366. PMLR, 2022.

David S Leslie, Steven Perkins, and Zibo Xu. Best-response dynamics in zero-sum stochastic games. Journal of Economic Theory , 189:105095, 2020.

Lucas Baudin and Rida Laraki. Smooth fictitious play in stochastic games with perturbed payoffs and unknown transitions. Advances in Neural Information Processing Systems , 35:20243-20256, 2022.

Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for SARSA with linear function approximation. In Advances in Neural Information Processing Systems , pages 8668-8678, 2019.

Shangtong Zhang, R Tachet, and Romain Laroche. Global optimality and finite sample analysis of softmax off-policy actor critic under state distribution mismatch. Journal of Machine Learning Research , 23(343):1-91, 2022.

Muhammed O Sayin, Francesca Parise, and Asuman Ozdaglar. Fictitious play in zero-sum stochastic games. SIAM Journal on Control and Optimization , 60(4):2095-2114, 2022.

Guanghui Lan. First-order and Stochastic Optimization Methods for Machine Learning . Springer, 2020.

Zaiwei Chen, Sheng Zhang, Thinh T. Doan, John-Paul Clarke, and Siva Theja Maguluri. FiniteSample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learning. Preprint arXiv:1905.11425 , 2019.

Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review , 60(2):223-311, 2018.

George W Brown. Iterative solution of games by fictitious play. Activity Analysis of Production and Allocation , 13(1):374-376, 1951.

Julia Robinson. An iterative method of solving a game. Annals of Mathematics , pages 296-301, 1951.

D. Fudenberg and D. Kreps. Learning mixed equilibria. Games and Economic Behavior , 5:320-367, 1993.

Josef Hofbauer and William H Sandholm. On the global convergence of stochastic fictitious play. Econometrica , 70(6):2265-2294, 2002.

James Hannan. Approximation to Bayes risk in repeated play. Contributions to the Theory of Games , 3:97-139, 1957.

Drew Fudenberg and David K Levine. Consistency and cautious fictitious play. Journal of Economic Dynamics and Control , 19(5-7):1065-1089, 1995.

Josef Hofbauer and Ed Hopkins. Learning in perturbed asymmetric games. Games and Economic Behavior , 52(1):133-152, 2005.

Jeff S Shamma and Gürdal Arslan. Unified convergence proofs of continuous-time fictitious play. IEEE Transactions on Automatic Control , 49(7):1137-1141, 2004.

Nicolo Cesa-Bianchi and Gabor Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

Stefanos Leonardos, Will Overman, Ioannis Panageas, and Georgios Piliouras. Global convergence of multi-agent policy gradient in Markov potential games. In International Conference on Learning Representations , 2022.

Sarath Pattathil, Kaiqing Zhang, and Asuman Ozdaglar. Symmetric (optimistic) natural policy gradient for multi-agent learning with parameter convergence. In International Conference on Artificial Intelligence and Statistics , pages 5641-5685. PMLR, 2023.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive Markov games. In Conference on Learning Theory , pages 4259-4299. PMLR, 2021.

Ziyi Chen, Shaocong Ma, and Yi Zhou. Sample efficient stochastic policy extragradient algorithm for zero-sum Markov game. In International Conference on Learning Representations , 2021.

Muhammed O Sayin, Kaiqing Zhang, and Asuman Ozdaglar. Fictitious play in Markov games with single controller. In Proceedings of the 23rd ACM Conference on Economics and Computation , pages 919-936, 2022.

Lucas Baudin and Rida Laraki. Fictitious play and best-response dynamics in identical interest and zero-sum stochastic games. In International Conference on Machine Learning , pages 1664-1690. PMLR, 2022.

Chinmay Maheshwari, Manxi Wu, Druv Pai, and Shankar Sastry. Independent and decentralized learning in Markov potential games. Preprint arXiv:2205.14590 , 2022.

Eyal Even-Dar and Yishay Mansour. Learning rates for Q -learning. Journal of Machine Learning Research , 5(Dec):1-25, 2003.

Guannan Qu and Adam Wierman. Finite-time analysis of asynchronous stochastic approximation and Q -learning. In Conference on Learning Theory , pages 3185-3205. PMLR, 2020.

Zaiwei Chen, Siva T Maguluri, Sanjay Shakkottai, and Karthikeyan Shanmugam. A Lyapunov theory for finite-sample guarantees of Markovian stochastic approximation. Operations Research , 2023.

Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming , pages 1-48, 2022.

Yuling Yan, Gen Li, Yuxin Chen, and Jianqing Fan. The efficacy of pessimism in asynchronous Q -learning. IEEE Transactions on Information Theory , 2023.

Srihari Govindan, Philip J Reny, Arthur J Robson, et al. A short proof of Harsanyi's purification theorem. Games and Economic Behavior , 45(2):369-374, 2003.

Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3(1):9-44, 1988.

Josef Hofbauer and Sylvain Sorin. Best response dynamics for continuous zero-sum games. Discrete and Continuous Dynamical Systems Series B , 6(1):215, 2006.

Tianyi Lin, Zhengyuan Zhou, Wenjia Ba, and Jiawei Zhang. Doubly optimal no-regret online learning in strongly monotone games with bandit feedback. Preprint arXiv:2112.02856 , 2021.

Aleksandr Beznosikov, Eduard Gorbunov, Hugo Berard, and Nicolas Loizou. Stochastic gradient descent-ascent: Unified theory and new efficient methods. In International Conference on Artificial Intelligence and Statistics , pages 172-235. PMLR, 2023.

Drew Fudenberg and David K Levine. The Theory of Learning in Games , volume 2. MIT press, 1998.

Lloyd S Shapley. Stochastic games. Proceedings of the National Academy of Sciences , 39(10):1095-1100, 1953.

Stefan Banach. Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales. Fund. math , 3(1):133-181, 1922.

Sajad Khodadadian, Thinh T Doan, Justin Romberg, and Siva Theja Maguluri. Finite sample analysis of two-time-scale natural actor-critic algorithm. IEEE Transactions on Automatic Control , 2022.

Ziyi Chen, Yi Zhou, Rong-Rong Chen, and Shaofeng Zou. Sample and communication-efficient decentralized actor-critic algorithms with finite-time analysis. In International Conference on Machine Learning , pages 3794-3834. PMLR, 2022.

Tengyu Xu and Yingbin Liang. Sample complexity bounds for two timescale value-based reinforcement learning algorithms. In International Conference on Artificial Intelligence and Statistics , pages 811-819. PMLR, 2021.

Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems , 33:1761717628, 2020.

Shuang Qiu, Zhuoran Yang, Jieping Ye, and Zhaoran Wang. On finite-time convergence of actor-critic algorithm. IEEE Journal on Selected Areas in Information Theory , 2(2):652-664, 2021.

Christopher JCH Watkins and Peter Dayan. Q -learning. Machine learning , 8(3-4):279-292, 1992.

Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play. Advances in neural information processing systems , 33:2159-2170, 2020.

Ziang Song, Song Mei, and Yu Bai. When can we learn general-sum Markov games with a large number of players sample-efficiently? In International Conference on Learning Representations , 2022.

Weichao Mao, Lin Yang, Kaiqing Zhang, and Tamer Ba¸ sar. On improving model-free algorithms for decentralized multi-agent reinforcement learning. In International Conference on Machine Learning , pages 15007-15049. PMLR, 2022.

Qiwen Cui, Kaiqing Zhang, and Simon Du. Breaking the curse of multiagents in a large state space: RL in Markov games with independent linear function approximation. In The Thirty Sixth Annual Conference on Learning Theory , pages 2651-2652. PMLR, 2023.

Chen-Yu Wei, Yi-Te Hong, and Chi-Jen Lu. Online reinforcement learning in stochastic games. In Advances in Neural Information Processing Systems , pages 4987-4997, 2017.

Kaiqing Zhang, Sham Kakade, Tamer Ba¸ sar, and Lin Yang. Model-based multi-agent RL in zero-sum Markov games with near-optimal sample complexity. Advances in Neural Information Processing Systems , 33:1166-1178, 2020.

Gen Li, Yuejie Chi, Yuting Wei, and Yuxin Chen. Minimax-optimal multi-agent RL in Markov games with a generative model. Advances in Neural Information Processing Systems , 35:1535315367, 2022.

Qiwen Cui and Simon S Du. When are offline two-player zero-sum Markov games solvable? Advances in Neural Information Processing Systems , 35:25779-25791, 2022.

Qiwen Cui and Simon S Du. Provably efficient offline multi-agent reinforcement learning via strategy-wise bonus. Advances in Neural Information Processing Systems , 35:11739-11751, 2022.

Han Zhong, Wei Xiong, Jiyuan Tan, Liwei Wang, Tong Zhang, Zhaoran Wang, and Zhuoran Yang. Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. In International Conference on Machine Learning , pages 27117-27142. PMLR, 2022.

Zuguang Gao, Qianqian Ma, Tamer Ba¸ sar, and John R. Birge. Sample complexity of decentralized tabular Q -learning for stochastic games. 2023 American Control Conference (ACC) , pages 1098-1103, 2023.

David A Levin and Yuval Peres. Markov Chains and Mixing Times , volume 107. American Mathematical Soc., 2017.

Dimitri P Bertsekas and John N Tsitsiklis. Neuro-Dynamic Programming . Athena Scientific, 1996.

John M Danskin. The Theory of Max-Min and Its Application to Weapons Allocation Problems , volume 5. Springer Science & Business Media, 2012.

Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. Preprint arXiv:1704.00805 , 2017.

Amir Beck. First-order methods in optimization , volume 25. SIAM, 2017.

Constantinos Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. 10th Innovations in Theoretical Computer Science , 2019.

Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, Kentaro Toyoshima, and Atsushi Iwasaki. Lastiterate convergence with full and noisy feedback in two-player zero-sum games. In International Conference on Artificial Intelligence and Statistics , pages 7999-8028. PMLR, 2023.

A Extended Related Work

Continued from Section 1.3, we here discuss several other existing works that are relevant.

Recently, there has been an increasing study of MARL with sample efficiency guarantees recently [16, 82, 20, 17, 21, 83, 84, 22, 85]. Most of them focus on the finite-horizon episodic setting with online exploration, and perform regret analysis, which differs from our last-iterate finite-sample analysis under the stochastic approximation paradigm. Additionally, these algorithms are episodic due to the finite-horizon nature of the setting and are not best-response-type independent learning dynamics that are repeatedly run for infinitely long, which can be viewed as a non-equilibrating adaptation process. In fact, the primary focus of this line of work is a self-play setting where all the players can be controlled to perform centralized learning [86, 16, 82, 20, 17]. Beyond the online setting, finite-sample efficiency has also been established for MARL using a generative model [87, 88] or offline datasets [89-91, 67]. These algorithms tend to be centralized in nature and focus on equilibrium computation , instead of performing independent learning.

Finite-sample complexity has also been established for policy gradient methods, a popular RL approach when applied to solving zero-sum stochastic games [14, 36-38]. However, to ensure convergence, these methods are asymmetric in that the players update their policies at different timescales, e.g., one player updates faster than the other with larger stepsizes, or one player fixes its policy while waiting for the other to update. Such asymmetric policy gradient methods are not independent, as some implicit coordination is required to enable such a timescale separation across agents. This style of implicit coordination is also required for the finite-sample analysis of decentralized learning in certain general-sum stochastic games, e.g., [92], which improves the asymptotic convergence in [7]. In contrast, our learning dynamics only require the update of each player's policy to be slower than the update of their q -functions, but crucially we do not assume a time-scale separation between the two players, making our learning dynamics symmetric.

B Proof Sketch of Theorem 3.1

In this section, we present the key steps and technical ideas used to prove Theorem 3.1. The core challenge here is that Algorithm 2 maintains 3 sets of iterates ( { q i t,k } , { π i t,k } , and { v i t } ), which are coupled. The coupling of their update equations means that it is not possible to separately analyze them. Instead, we develop a coupled Lyapunov drift approach to establish the finite-sample bounds of Algorithm 2. Specifically, we first show that the expected Nash gap can be upper bounded by a sum of properly defined Lyapunov functions, one for each set of the iterates (i.e., the v -functions, the policies, and the q -functions). Then, we establish a set of coupled Lyapunov drift inequalities - one for each Lyapunov function. Finally, we decouple the Lyapunov drift inequalities to establish the overall finite-sample bounds. We outline the key steps in the argument below.

To begin with, we show in Lemma D.1 that the q -functions { q i t,k } and the v -functions { v i t } generated by Algorithm 2 are uniformly bounded from above in ℓ ∞ -norm by 1 / (1 -γ ) and the entries of the policies { π i t,k } are uniformly bounded below by ℓ τ > 0 . This result will be frequently used in our analysis. We next introduce the Lyapunov functions we use to analyze Algorithm 2. Specifically, for any t, k ≥ 0 , let ¯ q i t,k ∈ R |S||A i | be defined as ¯ q i t,k ( s ) = T i ( v i t )( s ) π -i t,k ( s ) for all s ∈ S . Let

Note that L sum ( t ) is introduced to deal with the fact that the induced matrix game the inner loop of Algorithm 2 is designed to solve may not be a zero-sum game due to independent learning. See the discussion in Section 1.2 and the paragraph after Algorithm 2. At the core of our argument is the following inequality (cf. Lemma D.4):

which motivates us to bound all the Lyapunov functions.

B.1 Analysis of the Outer Loop: v -Function Update

Motivated by Eq. (6), we need to bound L sum ( T ) and L v ( T ) . To achieve that, we establish Lyapunov drift inequalities for them. Specifically, we show in Lemmas D.5 and D.6 that

Suppose that the Additive Errors in the previous two inequalities were only functions of v 1 t and v 2 t , then these two Lyapunov drift inequalities can be repeatedly used to obtain a convergence bound for L sum ( T ) and L v ( T ) . However, the coupled nature of Eqs. (7) and (8) requires us to analyze the policies and the q -functions in the inner loop, and establish their Lyapunov drift inequalities.

B.2 Analysis of the Inner Loop: Policy Update

As illustrated in Section 2.1 and Section 3.1, for each state s , the update equation of the policies can be viewed as a discrete and stochastic variant of the smoothed best-response dynamics for solving matrix games [29]. Typically, the following Lyapunov function is used to study such dynamics [53]:

where X 1 and X 2 are the payoff matrices for player 1 and player 2 , respectively, and ν ( · ) is the entropy function. Specialized to our case, given a joint v -function v = ( v 1 , v 2 ) from the outer loop 4 and a state s ∈ S , we would like to use

as our Lyapunov function. Note that max s ∈S V v t ,s ( π 1 t,k ( s ) , π 2 t,k ( s )) = L π ( t, k ) . A sequence of properties (e.g., strong convexity, smoothness, etc.) regarding the Lyapunov function V X ( · , · ) is established in Lemma D.7. In the end, we show in Lemma D.8 that

where E t [ · ] stands for conditional expectation conditioned on the history up to the beginning of the t -th outer loop. To interpret the above, suppose that we were considering the continuous-time smoothed best-response dynamics. Then, the additive error term would disappear in the sense that the time-derivative of the Lyapunov function along the trajectory of the ODE is strictly negative. Thus, the three terms in the Additive Errors can be interpreted as (1) the discretization error in the update equation, (2) the stochastic error in the q -function estimate, and (3) the error due to the non-zero-sum structure of the inner-loop auxiliary matrix game.

B.3 Analysis of the Inner Loop: q -Function Update

Our next focus is the q -function update. The q -function update equation is in the same spirit as TD-learning in RL, and a necessary condition for the convergence of TD-learning is that the behavior

policy (i.e., the policy used to collect samples) should enable the agent to sufficiently explore the environment. To achieve this goal, since we show in Lemma D.1 that all joint policies from the algorithm trajectory have uniformly lower-bounded entries (with lower bound ℓ τ > 0 ), it is enough to restrict our attention to a 'soft' policy class Π τ := { π = ( π 1 , π 2 ) | min s,a 1 π 1 ( a 1 | s ) ≥ ℓ τ , min s,a 2 π 2 ( a 2 | s ) ≥ ℓ τ } . The following lemma, which is an extension of [10, Lemma 4], establishes a uniform exploration property under Assumption 3.1.

To present the result, we need the following notation. Under Assumption 3.1, the Markov chain induced by the joint policy π b has a unique stationary distribution µ b ∈ ∆( S ) [93], the minimum component of which is denoted by µ b, min . In addition, there exists ρ b ∈ (0 , 1) such that max s ∈S ∥ ∥ P k π b ( s, · ) -µ b ( · ) ∥ ∥ TV ≤ 2 ρ k b for all k ≥ 0 [93], where P π b is the transition probability matrix of the Markov chain { S k } under π b . We also define the mixing time in the following. Given a joint policy π = ( π 1 , π 2 ) and an accuracy level η > 0 , the η - mixing time of the Markov chain { S k } induced by π is defined as

where P π is the π -induced transition probability matrix and µ π is the stationary distribution of { S k } under π , provided that it exists and is unique. When the induced Markov chain mixes at a geometric rate, it is easy to see that t π,η = O (log(1 /η )) .

Lemma B.1 (An Extension of Lemma 4 in [10]) . Suppose that Assumption 3.1 is satisfied. Then we have the following results.

For any π = ( π 1 , π 2 ) ∈ Π τ , the Markov chain { S k } induced by the joint policy π is irreducible and aperiodic, hence admits a unique stationary distribution µ π ∈ ∆( S ) .

It holds that sup π ∈ Π τ max s ∈S ∥ P k π ( s, · ) -µ π ( · ) ∥ TV ≤ 2 ρ k τ for any k ≥ 0 , where ρ τ = ρ ℓ 2 r b τ µ b, min b and r b := min { k ≥ 0 : P k π b ( s, s ′ ) > 0 , ∀ ( s, s ′ ) } . As a result, we have

where we recall that t π b ,η is the η - mixing time of the Markov chain { S k } induced by π b .

There exists L p ≥ 1 (which was used in the statement of Theorem 3.1) such that

Remark. Lemma B.1 (1), (3), and (4) were previous established in [10, Lemma 4]. Lemma B.1 (2) enables us to see the explicit dependence of the 'uniform mixing time' on the margin ℓ τ and the mixing time of the benchmark exploration policy π b .

In view of Lemma B.1 (2), we have fast mixing for all policies in Π τ if (i) the margin ℓ τ is large, and (ii) the Markov chain { S k } induced by the benchmark exploration policy π b is well-behaved. By 'well-behaved' we mean the mixing time is small (i.e., small t π b ,η ) and the stationary distribution is relatively well-balanced (i.e., large µ b, min ). Point (i) agrees with our intuition as a large margin encourages more exploration. To make sense of point (ii) , since π ( a | s ) ≥ ℓ 2 τ π b ( a | s ) for all s and a = ( a 1 , a 2 ) , we can write π as a convex combination between π b and some residual policy ˜ π : π ( ·| s ) = ℓ 2 τ π b ( ·| s ) + (1 -ℓ 2 τ )˜ π ( ·| s ) for all s ∈ S . Therefore, since any π ∈ Π τ has a portion of the benchmark exploration policy π b in it, it makes intuitive sense that fast mixing of { S k } under π b implies, to some extent, fast mixing of { S k } under π ∈ Π τ . Note that, as the margin ℓ τ approaches zero, the uniform mixing time in Lemma B.1 (2) goes to infinity. This is not avoidable in general, as demonstrated by a simple MDP example constructed in Appendix E.

We define c τ = µ min ℓ τ , which was used in the statement of Theorem 3.1. With Lemma B.1 in hand, we are now able to analyze the behavior of the q -functions. We model the q -function update as a

stochastic approximation algorithm driven by time-inhomogeneous Markovian noise, and use the norm-square function

as the Lyapunov function to study its behavior. Note that L q ( t, k ) = ∑ i =1 , 2 ∑ s ∥ q i t,k ( s ) -T i ( v i t )( s ) π -i t,k ( s ) ∥ 2 2 . The key challenge to establishing a Lyapunov drift inequality is to control a difference of the form

for any q i ∈ R |S||A i | and i ∈ { 1 , 2 } , where F i ( · ) is some appropriately defined operator that captures the dynamics of the update equation; see Appendix D.5.2 for its definition. In the term (13), the random tuple ( S k , A i k , A -i k , S k +1 ) is the k -th sample from the time-inhomogeneous Markov chain { ( S k , A i k , A -i k , S k +1 ) } k ≥ 0 generated by the time-varying joint policies { π k } k ≥ 0 , and ( ˆ S, ˆ A i , ˆ A -i , ˆ S ′ ) is a random tuple such that S ∼ µ k ( · ) , A i ∼ π i k ( ·| S ) , A -i ∼ π -i k ( ·| S ) , and S ′ ∼ p ( ·| S, A i , A -i ) , where µ k ( · ) denotes the unique stationary distribution of the Markov chain { S n } n ≥ 0 induced by the joint policy π k . Lemma B.1 implies that µ k exists and is unique.

In the existing literature, when { ( S k , A i k , A -i k , S k +1 ) } is sampled either in an i.i.d. manner or forms an ergodic time-homogeneous Markov chain, there are techniques that successfully bound the term (13) [94, 24, 23]. To deal with time-inhomogeneous Markovian noise, building upon existing conditioning results [23, 24, 41, 76] and also Lemma B.1, we develop a refined conditioning argument to show that

where z k = t ( ℓ τ , β k ) is a uniform upper bound on the β k - the mixing time (i.e., the uniform mixing time with accuracy β k , see Eq. (12)) of the Markov chain { S n } n ≥ 0 induced by an arbitrary joint policy from the algorithm trajectory. Suppose that we are using diminishing stepsizes of O (1 /k ) decay rate (similar results hold for using constant stepsizes). Then, the uniform mixing property from Lemma B.1 (2) implies that z k = O (log(1 /k )) . As a result, we have lim k →∞ (13) ≤ lim k →∞ z k ∑ k -1 n = k -z k α n = 0 , which provides us a way to control the term in (13). After successfully handling (13), we are able to establish a Lyapunov drift inequality of L q ( t, k ) :

where C 0 is a (problem-dependent) constant, and we use the notation α k 1 ,k 2 := ∑ k 2 k = k 1 α k to simplify the notation. See Lemma D.12 for more details. When k is large, it can be shown that α k -z k ,k -1 ≤ 2 α k z k [65, Appendix 1.8].

B.4 Solving Coupled Lyapunov Drift Inequalities

Until this point, we have established the Lyapunov drift inequalities for the individual v -functions, the sum of the v -functions, the policies, and the q -functions in Eqs. (7), (8), (10), and (14), respectively. The last challenge is to find a way of using these coupled inequalities to derive the finite-sample bound. To elaborate, we first restate all the Lyapunov drift inequalities in the following:

To decouple the Lyapunov inequalities stated above, our high-level ideas are (1) using the Lyapunov drift inequalities in a combined way instead of in a separate manner, and (2) a bootstrapping procedure

where we first derive a crude bound and then substitute the crude bound back into the Lyapunov drift inequalities to derive a tighter bound. We next present our approach.

For ease of presentation, for a scalar-valued quantity W that is a function of k and/or t , we say W = o k (1) if lim k →∞ W = 0 and W = o t (1) if lim t →∞ W = 0 . The explicit convergence rates of the o k (1) term and the o t (1) term will be revealed in the complete proof in Appendix D.6, but is not important for the illustration here.

Step 1. Adding up Eq. (17) and (18), using Condition 3.1, and then repeatedly using the resulting inequality, we obtain:

Step 2. Substituting the bound for E t [ L π ( t, k )] in Eq. (19) into Eq. (18) and repeatedly using the resulting inequality, and we obtain:

which in turn implies (by first using Jensen's inequality and then taking total expectation) that:

where we recall that c α,β = β k /α k is the stepsize ratio. The fact that we are able to get a factor of O ( c 1 / 2 α,β ) in front of E [ L sum ( t )] is crucial for the decoupling procedure.

Step 3. Taking total expectation on both sides of Eq. (16) and then using the upper bound of E [ L 1 / 2 q ( t, K )] we obtained in Eq. (20), we obtain

By choosing c α,β so that O ( c 1 / 2 α,β ) ≤ (1 -γ ) / 2 , the previous inequality implies

which can be repeatedly used to obtain

Substituting the previous bound on E [ L sum ( t )] into Eq. (19), we have

Step 4. Substituting the bounds we obtained for E [ L π ( t, K )] , E [ L q ( t, K )] , and E [ L sum ( t )] in Eqs. (22) and (23) into Eq. (15), and then repeatedly using the resulting inequality from t = 0 to t = T , we have

Now that we have obtained finite-sample bounds for E [ L v ( T )] , E [ L sum ( T )] , E [ L π ( T, K )] , and E [ L q ( T, K )] , using them in Eq. (6), we finally obtain the desired finite-sample bound for the expected Nash gap.

Looking back at the decoupling procedure, Steps 2 and 3 are crucial. In fact, in Step 1 we already obtain a bound on E t [ L q ( t, k )] , where the additive error is O (1) E [ L sum ( t )] . However, directly using this bound on E t [ L q ( t, k )] in Eq. (16) would result in an expansive inequality for E [ L sum ( t )] . By performing Step 2 , we are able to obtain a tighter bound for E t [ L q ( t, k )] , with the additive error being O ( c 1 / 2 α,β ) E [ L sum ( t )] . Furthermore, we can choose c α,β to be small enough so that after using the bound from Eq. (20) in Eq. (16), the additive error O ( c 1 / 2 α,β ) E [ L sum ( t )] is dominated by the negative drift in Eq. (21).

C Proof of Theorem 2.1

The proof is divided into 4 steps. In Appendix C.1, we prove an important boundedness property regarding the iterates generated by Algorithm 1. In Appendices C.2 and C.3, we analyze the evolution of the policies and the q -functions by establishing the negative drift inequalities with respect to their associated Lyapunov functions. In Appendix C.4, we solve the coupled Lyapunov drift inequalities to prove Theorem 2.1. Moreover, we prove Corollary 2.1.1 in Appendix C.5. The statement and proof of all supporting lemmas used in this section are presented in Appendix C.6.

C.1 Boundedness of the Iterates

In this subsection, we show that the q -functions generated by Algorithm 1 are uniformly bounded from above, and the entries of the policies are uniformly bounded from below. The following lemma is needed to establish the result.

Lemma C.1. For any i ∈ { 1 , 2 } and q i ∈ R |A| i , we have

Proof of Lemma C.1. Given i ∈ { 1 , 2 } , for any q i ∈ R |A| i and a i ∈ A i , we have

̸

Since the RHS of the previous inequality does not depend on a i , we have the desired inequality.

We next derive the boundedness property in the following lemma.

Lemma C.2. It holds for all k ≥ 0 and i ∈ { 1 , 2 } that ∥ q i k ∥ ∞ ≤ 1 and min a i ∈A i π i k ( a i ) ≥ ℓ τ , where ℓ τ = [( A max -1) exp(2 /τ ) + 1] -1 .

Proof of Lemma C.2. We prove the results by induction. Since q i 0 = 0 and π i 0 is initialized as a uniform distribution on A i , we have the base case. Now suppose that the results hold for some k ≥ 0 . Using the update equation for q i k in Algorithm 1 Line 5 , we have

for any a i ∈ A i , where the last line follows from the induction hypothesis ∥ q i k ∥ ∞ ≤ 1 and | R i ( a i , a -i ) | ≤ 1 for all ( a i , a -i ) . As for π i k +1 , using the update equation for π i k in Algorithm 1 Line 3 , we have

The induction is complete.

C.2 Analysis of the Policies

Let V R : ∆( A 1 ) × ∆( A 2 ) ↦→ R be the regularized Nash gap defined as

where ν ( · ) is the entropy function. A sequence of properties regarding V R ( · , · ) are provided in Lemma C.7. For simplicity of notation, we use ∇ 1 V R ( · , · ) (respectively, ∇ 2 V R ( · , · ) ) to represent the gradient with respect to the first argument (respectively, the second argument).

Lemma C.3. It holds for all k ≥ 0 that

where we recall that L τ = τ/ℓ τ + A 2 max /τ .

Proof of Lemma C.3. Using the smoothness property of V R ( · , · ) (cf. Lemma C.7 (1)) and the update equation in Algorithm 1 Line 3 , we have for any k ≥ 0 that

where the last line follows from Lemma C.7 (2) and (3). Taking expectations on both sides of the previous inequality and using the condition that c α,β = β k α k ≤ min( τℓ 3 τ 32 , ℓ τ τ 3 32 A 2 max ) (cf. Condition 2.1), we have

The proof is complete.

C.3 Analysis of the q -Functions

We study the q -functions generated by Algorithm 1 through a stochastic approximation framework. For i ∈ { 1 , 2 } , let F i : R |A i | ×A i ×A -i ↦→ R |A i | be an operator defined as

Then, Algorithm 1 Line 5 can be compactly written as

Given a joint policy ( π 1 , π 2 ) , let ¯ F i π : R |A i | ↦→ R |A i | be defined as

Then, Eq. (24) can be viewed as a stochastic approximation algorithm for solving the slowly time-varying equation ¯ F i π k ( q i ) = 0 .

Lemma C.4. The following inequality holds for all k ≥ 0 :

Proof of Lemma C.4. For any k ≥ 0 and i ∈ { 1 , 2 } , we have

(This follows from the AM-GM inequality, where c 1 , c 2 > 0 can be arbitrary.)

Taking expectations on both sides of the previous inequality, we have

where the term E [ ⟨ F i ( q i k , A i k , A -i k ) -¯ F i π k ( q i k ) , q i k -R i π -i k ⟩ ] vanishes due to the tower property of conditional expectations. To proceed, observe

and

In addition, we have

where the last line follows from Lemma C.6. Using the previous 4 inequalities all together, we obtain

Summing up the previous inequality for i ∈ { 1 , 2 } , we have

where the last line follows from c α,β ≤ min( τ 2 ℓ τ 8 A 2 max , τℓ τ 128 A 2 max ) and β 0 ≤ τ 128 A 2 max (cf. Condition 2.1). The proof is complete.

C.4 Solving the Coupled Lyapunov Inequalities

For simplicity of notation, denote L q ( k ) = ∑ i =1 , 2 E [ ∥ q i k -R i π -i k ∥ 2 2 ] and L π ( k ) = E [ V R ( π 1 k , π 2 k )] . Then, Lemmas C.3 and C.4 state that

and

Adding up the previous two inequalities, we obtain

where the second inequality follows from c α,β ≤ ℓ τ (cf. Condition 2.1).

Constant Stepsizes. When using constant stepsizes, i.e., α k ≡ α and β k ≡ β , repeatedly using Eq. (25), we have for all k ≥ 0 that

where the second inequality follows from

Theorem 2.1 (1) follows by observing that L q ( k ) + L π ( k ) ≥ L π ( k ) = E [ NG τ ( π 1 k , π 2 k )] .

Diminishing Stepsizes. Consider using α k = α k + h and β k = β k + h . Recursions of the form presented in Eq. (25) have been well studied in the existing literature for the convergence rates of iterative algorithms [44, 24, 65]. Since β > 4 , using the same line of analysis as in [65, Appendix A.2], we have

Theorem 2.1 (2) follows by observing that L q ( k ) + L π ( k ) ≥ L π ( k ) = E [ NG τ ( π 1 k , π 2 k )] .

C.5 Proof of Corollary 2.1.1

We use Theorem 2.1 (1) to derive the sample complexity, and choose β = c α,β α with c α,β satisfying Condition 2.1. To achieve E [ NG τ ( π 1 K , π 2 K )] ≤ ϵ , in view of Theorem 2.1 (1), it is sufficient that

which implies β = O ( ϵ ) . It follows that K = O ( ϵ -1 ) .

C.6 Supporting Lemmas

Lemma C.5. For any i ∈ { 1 , 2 } and µ i 1 , µ i 2 ∈ { µ i ∈ ∆( A i ) | min a i ∈A i µ i ( a i ) ≥ ℓ τ } , we have

Proof of Lemma C.5. For any i ∈ { 1 , 2 } and µ i ∈ ∆( A i ) such that min a i ∈A i µ i ( a i ) ≥ ℓ τ , the Hessian of ν ( · ) satisfies

Therefore, the gradient of the negative entropy function ∇ ν ( · ) is 1 ℓ τ - Lipschitz continuous with respect to ∥·∥ 2 on the set { µ i ∈ ∆( A i ) | min a i ∈A i µ i ( a i ) ≥ ℓ τ } , which implies the 1 ℓ τ - smoothness of ν ( · ) .

Lemma C.6. For i ∈ { 1 , 2 } , we have for all µ i ∈ ∆( A i ) and µ -i ∈ ∆( A -i ) that

Proof of Lemma C.6. Recall that the negative entropy ν ( · ) is 1 -strongly concave with respect to ∥·∥ 2 . Therefore, given i ∈ { 1 , 2 } , fix µ -i , the function

is τ -strongly convex with respect to µ i . As a result, by the quadratic growth property of strongly convex functions, we have

Denote Π τ = { ( π 1 , π 2 ) ∈ ∆( A 1 ) × ∆( A 2 ) | min a 1 ∈A 1 π 1 ( a 1 ) ≥ ℓ τ , min a 2 ∈A 2 π 2 ( a 2 ) ≥ ℓ τ } . Lemma C.7. The function V R ( · , · ) has the following properties.

The function V R ( µ 1 , µ 2 ) is L τ - smooth on Π τ , where L τ = τ ℓ τ + A 2 max τ .

For any q 1 ∈ R |A 1 | and q 2 ∈ R |A 2 | , we have for all ( µ 1 , µ 2 ) ∈ Π that

Proof of Lemma C.7. Recall the definition of V R ( · , · ) in the following:

By Danskin's theorem [95], we have

both of which will be frequently used in our analysis.

where the second last inequality follows from Lemma C.5 and σ τ ( · ) being 1 τ -Lipschitz continuous with respect to ∥ · ∥ 2 [96], and the last inequality follows from ∥ R i ∥ 2 ≤ √ |A 1 ||A 2 | ≤ A max for i ∈ { 1 , 2 } . Similarly, we also have

It follows from the previous two inequalities that

which implies that V R ( · , · ) is an L τ - smooth function on Π τ [97], where L τ = τ ℓ τ + A 2 max τ .

The result follows from Lemma D.7 by setting X i = R i , i ∈ { 1 , 2 } , and by observing that R 1 +( R 2 ) ⊤ = 0 .

Using the formula of the gradient of V R ( · , · ) in the begining of the proof, we have

⟨∇ 1 V R ( µ 1 , µ 2 ) , σ τ ( q 1 ) -σ τ ( R 1 µ 2 ) ⟩ = ⟨-τ ∇ ν ( µ 1 ) + ( R 2 ) ⊤ σ τ ( R 2 µ 1 ) , σ τ ( q 1 ) -σ τ ( R 1 µ 2 ) ⟩ = τ ⟨∇ ν ( σ τ ( R 1 µ 2 )) -∇ ν ( µ 1 ) , σ τ ( q 1 ) -σ τ ( R 1 µ 2 ) ⟩ +( σ τ ( R 2 µ 1 ) -µ 2 ) ⊤ R 2 ( σ τ ( q 1 ) -σ τ ( R 1 µ 2 )) (This follows from the order optimality condition: R 1 µ 2 + τ ∇ ν ( σ τ ( R 1 µ 2 )) = 0 ) ≤ τ 2 c 1 ∥∇ ν ( σ τ ( R 1 µ 2 )) -∇ ν ( µ 1 ) ∥ 2 2 + τc 1 2 ∥ σ τ ( q 1 ) -σ τ ( R 1 µ 2 ) ∥ 2 2 + 1 2 c 2 ∥ σ τ ( R 2 µ 1 ) -µ 2 ) ∥ 2 2 + c 2 2 ∥ R 2 ( σ τ ( q 1 ) -σ τ ( R 1 µ 2 )) ∥ 2 2 (This follows from AM-GM inequality, where c 1 , c 2 > 0 can be arbitrary) ≤ τ 2 c 1 ℓ 2 τ ∥ σ τ ( R 1 µ 2 ) -µ 1 ∥ 2 2 + c 1 2 τ ∥ q 1 -R 1 µ 2 ∥ 2 2 + 1 2 c 2 ∥ σ τ ( R 2 µ 1 ) -µ 2 ) ∥ 2 2 + c 2 ∥ R 2 ∥ 2 2 2 τ 2 ∥ q 1 -R 1 µ 2 ∥ 2 2 (Lemma C.5) ≤ ( 1 c 1 ℓ 2 τ + 1 τc 2 ) V R ( µ 1 , µ 2 ) + c 1 2 τ ∥ q 1 -R 1 µ 2 ∥ 2 2 + c 2 ∥ R 2 ∥ 2 2 2 τ 2 ∥ q 1 -R 1 µ 2 ∥ 2 2 (Lemma C.6) ≤ 1 4 V R ( µ 1 , µ 2 ) + 4 τℓ 2 τ ∥ q 1 -R 1 µ 2 ∥ 2 2 + 4 ∥ R 2 ∥ 2 2 τ 3 ∥ q 1 -R 1 µ 2 ∥ 2 2 , where the last line follows by choosing c 1 = 8 ℓ 2 τ and c 2 = 8 τ . Similarly, we also have

Summing up the previous two inequalities, we obtain

where we used ∥ R i ∥ 2 ≤ √ A max for i ∈ { 1 , 2 } .

D Proof of Theorem 3.1

We begin by introducing a summary of notation in Appendix D.1. In Appendix D.2, we establish an important boundedness property regarding the q -functions, value functions, and the policies generated by Algorithm 2. In Appendix D.3, we bound the Nash gap in terms of the Lyapunov functions. In Appendices D.4 and D.5, we analyze the outer loop and the inner loop of Algorithm 2 and establish the Lyapunov drift inequalities. Finally, in Appendix D.6, we solve the coupled Lyapunov inequalities to obtain the finite-sample bound. The proof of all supporting lemmas are provided in Appendix D.7, and the proof of Corollary 3.1.1 is provided in Appendix D.8.

D.1 Notation

We begin with a summary of the notation that will be used in the proof.

Given a pair of matrices { X i ∈ R |A i |×|A -i | } i ∈{ 1 , 2 } and a pair of distributions { µ i ∈ ∆( A i ) } i ∈{ 1 , 2 } , we define

where ν ( · ) is the entropy function. Note that V X ( · , · ) is similar to V R ( · , · ) defined in Appendix C.2 in the setting of matrix games. However, we do not assume that X 1 + X 2 = 0 .

Given a pair of value functions ( v 1 , v 2 ) and a state s ∈ S , when X i = T i ( v i )( s ) , i ∈ { 1 , 2 } , we write V v,s ( · , · ) for V X ( · , · ) .

For any joint policy ( π 1 , π 2 ) and state s , given i ∈ { 1 , 2 } , we define v i ∗ ,π -i ( s ) = max ˆ π i v i ˆ π i ,π -i ( s ) , v i π i , ∗ = min ˆ π -i v i π i , ˆ π -i ( s ) , v -i π -i , ∗ ( s ) = min ˆ π i v -i π -i , ˆ π i ( s ) , and v -i ∗ ,π i ( s ) = max ˆ π -i v -i ˆ π -i , ˆ π i ( s ) . Note that we have v 1 ∗ ,π 2 + v 2 π 2 , ∗ = 0 and v 1 π 1 , ∗ + v 2 ∗ ,π 1 = 0 because of the zero-sum structure.

For i ∈ { 1 , 2 } , denote v i ∗ as the unique fixed point of the equation B i ( v i ) = v i , where B i ( · ) is the minimax Bellman operator defined in Section 3. Note that we have v 1 ∗ + v 2 ∗ = 0 .

For any t, k ≥ 0 and i ∈ { 1 , 2 } , let ¯ q i t,k ∈ R |S||A i | be defined as ¯ q i t,k ( s ) = T i ( v i t )( s ) π -i t,k ( s ) for all s ∈ S . In addition, let

which will be the Lyapunov functions we use in the analysis.

Given k 1 ≤ k 2 , we denote β k 1 ,k 2 = ∑ k 2 k = k 1 β k and α k 1 ,k 2 = ∑ k 2 k = k 1 α k .

Recall that z k = t ( ℓ τ , β k ) is the uniform mixing time defined in Lemma B.1 (2), where ℓ τ is the uniform lower bound of the policies. When using constant stepsizes, z k is not a function of k , and is simply denoted by z β . Observe that, due to the uniform geometric mixing property established in Lemma B.1 (2), we have z k = O (log( k )) when using O (1 /k ) stepsizes and z β = O (log(1 /β )) when using constant stepsizes. Let k 0 = min k : k ≥ z k , which is well defined because z k grows logarithmically with k .

D.2 Boundedness of the Iterates

We first show in the following lemma that the q -functions and the v -functions generated by Algorithm 2 are uniformly bounded from above, and the policies are uniformly bounded from below. In the context of stochastic games, we redefine ℓ τ = [1 + ( A max -1) exp(2 / [(1 -γ ) τ ])] -1 .

Lemma D.1. For all t, k and i ∈ { 1 , 2 } , we have (1) ∥ v i t ∥ ∞ ≤ 1 / (1 -γ ) and ∥ q i t,k ∥ ∞ ≤ 1 / (1 -γ ) , and (2) min s ∈S ,a i ∈A i π i t,k ( a i | s ) ≥ ℓ τ .

Proof of Lemma D.1. The proof uses induction arguments. Let i ∈ { 1 , 2 } .

Given t ≥ 0 , we first show by induction that, if ∥ v i t ∥ ∞ ≤ 1 1 -γ and ∥ q i t, 0 ∥ ∞ ≤ 1 1 -γ , we have ∥ q i t,k ∥ ∞ ≤ 1 1 -γ for all k ≥ 0 . The base case ∥ q i t, 0 ∥ ∞ ≤ 1 1 -γ holds by our assumption. Suppose that ∥ q i t,k ∥ ∞ ≤ 1 1 -γ for some k ≥ 0 . Then, by Algorithm 2 Line 6 , we have for all ( s, a i ) that

where Eq. (27) follows from the induction hypothesis ∥ q i t,k ∥ ∞ ≤ 1 1 -γ , our assumption that ∥ v i t ∥ ∞ ≤ 1 1 -γ , and max s,a i ,a -i | R i ( s, a i , a -i ) | ≤ 1 . The induction is now complete and we have ∥ q i t,k ∥ ∞ ≤ 1 1 -γ for all k ≥ 0 whenever ∥ v i t ∥ ∞ ≤ 1 1 -γ and ∥ q i t, 0 ∥ ∞ ≤ 1 1 -γ .

We next again use induction to show that ∥ v i t ∥ ∞ ≤ 1 1 -γ and ∥ q i t, 0 ∥ ∞ ≤ 1 1 -γ for all t ≥ 0 . Our initialization ensures that ∥ v i 0 ∥ ∞ ≤ 1 1 -γ and ∥ q i 0 , 0 ∥ ∞ ≤ 1 1 -γ . Suppose that ∥ v i t ∥ ∞ ≤ 1 1 -γ and ∥ q i t, 0 ∥ ∞ ≤ 1 1 -γ for some t ≥ 0 . Using the update equation for v i t +1 (cf. Algorithm 2 Line 8 ) and the fact that ∥ q i t,k ∥ ∞ ≤ 1 1 -γ for all k ≥ 0 (established in the previous paragraph), we have for all s ∈ S that

which implies ∥ v i t +1 ∥ ∞ ≤ 1 1 -γ . Moreover, we have by Algorithm 2 Line 9 that ∥ q i t +1 , 0 ∥ ∞ = ∥ q i t,K ∥ ∞ ≤ 1 1 -γ . The induction is now complete and we have ∥ v i t ∥ ∞ ≤ 1 1 -γ and ∥ q i t, 0 ∥ ∞ ≤ 1 1 -γ for all t ≥ 0 .

We first use induction to show that, given t ≥ 0 , if min s,a i π i t, 0 ( a i | s ) ≥ ℓ τ , then we have min s,a i π i t,k ( a i | s ) ≥ ℓ τ for all k ∈ { 0 , 1 , · · · , K } . Since min s,a i π i t, 0 ( a i | s ) ≥ ℓ τ by our assumption, we have the base case. Now suppose that min s ∈S ,a i ∈A i π i t,k ( a i | s ) ≥ ℓ τ for some k ≥ 0 . Then we have by Algorithm 2 Line 4 that

where the inequality follows from (1) the induction hypothesis, and (2) σ τ ( q i t,k ( s ))( a i ) ≥ ℓ τ , which follows from Lemma D.1 (1) and Lemma C.1. The induction is complete.

We next again use induction to show that min s,a i π i t, 0 ( a i | s ) ≥ ℓ τ for all t ∈ { 0 , 1 , · · · , T } . Since min s,a i π i 0 , 0 ( a i | s ) is initialized as a uniform policy, we have the base case. Now suppose that min s,a i π i t, 0 ( a i | s ) ≥ ℓ τ for some t ≥ 0 . Since this implies that min s,a i π i t,k ( a i | s ) ≥ ℓ τ for all k ∈ { 0 , 1 , · · · , K } , and in addition, π i t +1 , 0 = π i t,K according to Algorithm 2 Line 9 , we have min s,a i π i t +1 , 0 ( a i | s ) ≥ ℓ τ . The induction is complete.

D.3 Bounding the Nash Gap

Our ultimate goal is to bound the Nash gap

We first bound the Nash gap using the value functions of the output policies from Algorithm 2.

Lemma D.2. It holds that

Proof of Lemma D.2. Using the definition of the expected value functions, we have

The next lemma bounds the RHS of Eq. (29) using the actual iterates generated by Algorithm 2. Lemma D.3. It holds for i ∈ { 1 , 2 } that

Proof of Lemma D.3. For any s ∈ S and i ∈ { 1 , 2 } , we have

Since the RHS of the previous inequality does not depend on s , we have for i ∈ { 1 , 2 } that

It remains to bound the RHS of the previous inequality. Observe that for any s ∈ S and i ∈ { 1 , 2 } , we have

We next bound each term on the RHS of the previous inequality.

The 1 st Term on the RHS of Eq. (31). Using the definition of T i ( · ) , we have

The 2 nd Term on the RHS of Eq. (31). Using the definition of T i ( · ) , we have

The 3 rd Term on the RHS of Eq. (31). Bounding the third term requires more effort. To begin with, we decompose it in the following way:

We next bound each term on the RHS of the previous inequality. For the first term, we have by definition of T i ( · ) that

For the second term on the RHS of Eq. (32), using the Lyapunov function V v,s ( · , · ) (defined in Appendix D.1), we have

Using the previous two inequalities together in Eq. (32), we obtain

The 4 th Term on the RHS of Eq. (31). Using the definition of T i ( · ) , we have

Finally, using the upper bounds we obtained for all the terms on the RHS of Eq. (31), we have

Rearranging terms and using L sum ( t ) and L π ( t, k ) to simplify the notation, we obtain

Summing up both sides of the previous inequality for i ∈ { 1 , 2 } , we have

Using the previous inequality in Eq. (30), we have the desired result.

Combining the results in Lemma D.2 and Lemma D.3 in Eq. (28), we have the following result, which bounds the Nash gap in terms of the Lyapunov functions defined in Appendix D.1.

The next step is to bound the Lyapunov functions, which require us to analyze the outer loop and inner loop of Algorithm 2.

D.4 Analysis of the Outer Loop: v -Function Update

We first consider L v ( t ) and establish a one-step Lyapunov drift inequality for it.

Lemma D.5. It holds for all t ≥ 0 that

Proof of Lemma D.5. For i ∈ { 1 , 2 } , using the outer-loop update equation (cf. Algorithm 2 Line 8 ) and the fact that B i ( v i ∗ ) = v i ∗ , we have for any t ≥ 0 and s ∈ S that

Since the minimax Bellman operator B i ( · ) is a γ -contraction mapping with respect to ∥ · ∥ ∞ , we have from the previous inequality that

It remains to bound the second term on the RHS of Eq. (35). Using the definition of B i ( · ) , we have

where the last line follows from Eq. (33). Using the previous inequality in Eq. (35), we obtain

Summing up both sides of the previous inequality for i ∈ { 1 , 2 } , we have

To bound the last term on the RHS of the previous inequality, observe that

Therefore, we have

The proof is complete.

We next establish a one-step Lyapunov drift inequality for L sum ( t ) in the following lemma.

Proof of Lemma D.6. Using the outer-loop update equation (cf. Algorithm 2 Line 8 ), we have for any t ≥ 0 and s ∈ S that

where the last line follows from the definition of T i ( · ) . Since the RHS of the previous inequality does not depend on s , we have

The result follows from using Eq. (36) to bound the last term on the RHS of the previous inequality and then using L sum ( t ) and L q ( t, k ) to simplify the notation.

D.5 Analysis of the Inner Loop

In this section, we establish negative drift inequalities for the Lyapunov functions L q ( t, k ) and L π ( t, k ) , which are defined in terms of the q -functions and the policies updated in the inner loop of Algorithm 2. For ease of presentation, we write down only the inner loop of Algorithm 2 in Algorithm 3, where we omit the subscript t , which is used as the index for the outer loop. Similarly, we will write L q ( k ) for L q ( t, k ) and L π ( k ) for L π ( t, k ) . All results derived for the q -functions and policies of Algorithm 3 can be directly combined with the analysis of outer loop of Algorithm 2 using a conditioning argument together with the Markov property.

Algorithm 3 Inner Loop of Algorithm 2: from Player i 's Perspective

1: Input: Integer K , initializations q i 0 and π i 0 , and a v -function v i from the outer loop. Note that we have ∥ q i 0 ∥ ∞ ≤ 1 1 -γ , ∥ v i ∥ ∞ ≤ 1 1 -γ , and min s,a i π i 0 ( a i | s ) ≥ ℓ τ due to Lemma D.1.

2: for k = 0 , 1 , · · · , K -1 do

3: π i k +1 ( s ) = π i k ( s ) + β k ( σ τ ( q i k ( s )) -π i k ( s )) for all s ∈ S

4: Sample A i k ∼ π i k +1 ( · | S k ) , and observe S k +1 ∼ p ( · | S k , A i k , A -i k )

5: q i k +1 ( s, a i )= q i k ( s, a i )+ α k 1 { ( S k ,A i k )=( s,a i ) } ( R i ( S k , A i k , A -i k ) + γv i ( S k +1 ) -q i k ( S k , A i k ) ) for all ( s, a i ) ∈ S × A i

6: end for

7: Output: q i K and π i K

D.5.1 Analysis of the Policies

We consider { ( π 1 k , π 2 k ) } k ≥ 0 generated by Algorithm 3 and use V X ( · , · ) defined in Appendix D.1 as the Lyapunov function to study them. For simplicity of notation, we use ∇ 1 V X ( · , · ) (respectively, ∇ 2 V X ( · , · ) ) to denote the gradient with respect to the first argument (respectively, the second argument). Recall that Lemma D.1 implies that π k = ( π 1 k , π 2 k ) ∈ Π τ for all k ≥ 0 , where Π τ = { ( π 1 , π 2 ) ∈ ∆( A 1 ) × ∆( A 2 ) | min a 1 ∈A 1 π 1 ( a 1 ) ≥ ℓ τ , min a 2 ∈A 2 π 2 ( a 2 ) ≥ ℓ τ } . The following lemma establishes the properties of V X ( · , · ) .

Lemma D.7. The function V X ( · , · ) has the following properties.

For i ∈ { 1 , 2 } , fix µ -i ∈ ∆( A -i ) , the function V X ( µ i , µ -i ) as a function of µ i is τ - strongly convex with respect to ∥ · ∥ 2 .

V X ( · , · ) is ˜ L τ - smooth on Π τ , where ˜ L τ = 2 ( τ ℓ τ + max( ∥ X 1 ∥ 2 2 , ∥ X 2 ∥ 2 2 ) τ + ∥ X 1 + X ⊤ 2 ∥ 2 ) .

It holds for any ( µ 1 , µ 2 ) ∈ ∆( A 1 ) × ∆( A 2 ) that

For any u 1 ∈ R |A 1 | and u 2 ∈ R |A 2 | , we have for all ( µ 1 , µ 2 ) ∈ Π τ that

Proof of Lemma D.7. To begin with, we have by Danskin's theorem [95] that

Similar result holds for ∇ 2 V X ( µ 1 , µ 2 ) .

It is clear that the function V X ( · , · ) is non-negative. The strong convexity follows from the following two observations.

(i) The negative entropy -ν ( · ) is 1 - strongly convex with respect to ∥ · ∥ 2 [97, Example 5.27].

(ii) Given i ∈ { 1 , 2 } , the function max ˆ µ -i ∈ ∆( A -i ) { (ˆ µ -i ) ⊤ X -i µ i + τν (ˆ µ -i ) } as a function of µ i is the maximum of linear functions in µ i , and therefore is convex.

It follows that, for any i ∈ { 1 , 2 } , the function V X ( µ 1 , µ 2 ) is τ - strongly convex in µ i with respect to ∥ · ∥ 2 uniformly for all µ -i .

For any ( µ 1 , µ 2 ) , (¯ µ 1 , ¯ µ 2 ) ∈ Π τ , we have by Eq. (37) that

where Eq. (38) follows from Lemma C.5 and the Lipschitz continuity of σ τ ( · ) [96]. Similarly, we also have

Using the previous 2 inequalities, we have the following result for the full gradient of V X ( · , · ) :

The previous inequality implies that V X ( · , · ) is an ˜ L τ -smooth function on Π τ [97], where

Using the formula for the gradient of V X ( · , · ) in Eq. (37), we have

where Eq. (39) is due to the first order optimality condition X 1 µ 2 + τ ∇ ν ( σ τ ( X 1 µ 2 )) = 0 . To proceed, observe that the concavity of ν ( · ) and the optimality condition X 1 µ 2 + τ ∇ ν ( σ τ ( X 1 µ 2 )) = 0 together imply that

Therefore, we have from the previous 2 inequalities that

It follows that

where the last line follows from a 2 + b 2 ≥ 2 ab .

For any u 1 ∈ R |A 1 | , using the formula of the gradient of V X ( · , · ) from Eq. (37), we have

where the third last inequality follows from the 1 ℓ τ -smoothness of ν ( · ) on Π τ , the second last inequality follows from Lemma D.7 (1) together with the quadratic growth property of strongly

Similarly, we also have

Adding up the previous 2 inequalities, we obtain

where the last line follows from ∥ σ τ ( X 2 µ 1 ) -µ 2 ∥ 2 ≤ ∥ σ τ ( X 2 µ 1 ) ∥ 1 + ∥ µ 2 ∥ 1 ≤ 2 . Using Lemma D.7 (1) together with the quadratic growth property of strongly convex functions, we have

convex functions, and the last inequality follows from a 2 + b 2 ≥ 2 ab . Similarly, we also have for any u 2 ∈ R |A 2 | that

Adding up the previous two inequalities, we obtain

The proof is complete.

With the properties of V X ( · , · ) established in Lemma D.7, we now use it as a Lyapunov function to study ( π 1 k , π 2 k ) generated by Algorithm 3. Specifically, using the smoothness of V X ( · , · ) (cf. Lemma D.7 (2)), the update equation in Algorithm 3 Line 3 , and Lemma D.7 (3) and (4), we have the desired one-step Lyapunov drift inequality for L π ( k ) , which is presented in the following.

Lemma D.8. The following inequality holds for all k ≥ 0 :

Proof of Lemma D.8. We will use V v,s ( · , · ) (see Lemma D.1) as the Lyapunov function to study the evolution of ( π 1 k ( s ) , π 2 k ( s )) . To begin with, we identify the smoothness parameter of V v,s ( · , · ) . Using Lemma D.7 (1) and the definition of V v,s ( · , · ) , we have

Therefore, V v,s ( · , · ) is an L τ - smooth function on Π τ . Using the smoothness of V v,s ( · , · ) , for any s ∈ S , we have by the policy update equation (cf. Algorithm 3 Line 3 ) that

where the last line follows from Lemma D.7 (3) and (4). Since max i ∈{ 1 , 2 } ∥T i ( v i )( s ) ∥ 2 ≤ A max 1 -γ and

we have

Since the RHS of the previous inequality does not depend on s , we have

where the last line follows from τ ≤ 1 / (1 -γ ) .

D.5.2 Analysis of the q -Functions

In this section, we consider q i k generated by Algorithm 3. We begin by reformulating the update of the q -function as a stochastic approximation algorithm for estimating a time-varying target. For i ∈ { 1 , 2 } , fixing v i ∈ R |S| , let F i : R |S||A i | ×S × A i ×A -i ×S ↦→ R |S||A i | be defined as

for all ( q i , s 0 , a i 0 , a -i 0 , s 1 ) and ( s, a i ) . Then Algorithm 3 Line 5 can be compactly written as

Denote the stationary distribution of the Markov chain { S k } induced by the joint policy π k = ( π 1 k , π 2 k ) by µ k ∈ ∆( S ) , the existence and uniqueness of which is guaranteed by Lemma D.1 and Lemma B.1 (1). Let ¯ F i k : R |S||A i | ↦→ R |S||A i | be defined as

for all q i ∈ R |S||A i | . Then, Eq. (41) can be viewed as a stochastic approximation algorithm for solving the (time-varying) equation ¯ F i k ( q i ) = 0 with time-inhomogeneous Markovian noise { ( S k , A i k , A -i k , S k +1 ) } k ≥ 0 . We next establish the properties of the operators F i ( · ) and ¯ F i k ( · ) in the following lemma.

Lemma D.9. The following properties hold for i ∈ { 1 , 2 } .

It holds that ∥ F i ( q i 1 , s 0 , a i 0 , a -i 0 , s 1 ) -F i ( q i 2 , s 0 , a i 0 , a -i 0 , s 1 ) ∥ 2 ≤ ∥ q i 1 -q i 2 ∥ 2 for any ( q i 1 , q i 2 ) and ( s 0 , a i 0 , a -i 0 , s 1 ) .

It holds that ∥ F i (0 , s 0 , a i 0 , a -i 0 , s 1 ) ∥ 2 ≤ 1 1 -γ for all ( s 0 , a i 0 , a -i 0 , s 1 ) .

¯ F i k ( q i ) = 0 has a unique solution ¯ q i k , which is given as ¯ q i k ( s ) = T i ( v i )( s ) π -i k ( s ) for all s .

It holds that ⟨ ¯ F i k ( q i 1 ) -¯ F i k ( q i 2 ) , q i 1 -q i 2 ⟩ ≤ -c τ ∥ q i 1 -q i 2 ∥ 2 2 for all ( q i 1 , q i 2 ) , where c τ = µ min ℓ τ . See Lemma B.1 for the definition of µ min .

Proof of Lemma D.9. (1) For any ( q i 1 , q i 2 ) and ( s 0 , a i 0 , a -i 0 , s 1 ) , we have

For any ( s 0 , a i 0 , a -i 0 , s 1 ) , we have

where the last line follows from ∥ v i ∥ ∞ ≤ 1 / (1 -γ ) and | R i ( s 0 , a i 0 , a -i 0 ) | ≤ 1 .

We first write down the explicitly the operator ¯ F i k ( · ) . Using the definition of T i ( · ) , we have

Since µ k ( s ) ≥ µ min > 0 (cf. Lemma B.1 (4)) and diag ( π i k ( s )) has strictly positive diagonal entries (cf. Lemma D.1) for all s ∈ S and k ≥ 0 , the equation ¯ F i k ( q i ) = 0 has a unique solution ¯ q i k ∈ R |S||A i | , which is given as

Using the expression of ¯ F i k ( · ) , we have for any q i 1 , q i 2 ∈ R |S||A i | that

The proof is complete.

Next, we establish a negative drift inequality for L q ( k ) . Using ∥ · ∥ 2 2 as a Lyapunov function, we have by Eq. (41) that

where the last line follows from Lemma D.9 (4). The terms E [ ∥ q i k +1 -q i k ∥ 2 2 ] , E [ ∥ ¯ q i k -¯ q i k +1 ∥ 2 2 ] , E [(¯ q i k -¯ q i k +1 ) ⊤ ( q i k +1 -q i k )] , E [( q i k -¯ q i k ) ⊤ (¯ q i k -¯ q i k +1 )] on the RHS of Eq. (42) are bounded in the following lemma.

Lemma D.10. The following inequalities hold for all k ≥ 0 .

Proof of Lemma D.10. (1) For any k ≥ 0 , using Eq. (41) and Lemma D.9 (1), we have

The result follows by taking expectation on both sides of the previous inequality.

For any k ≥ 0 , using the definition of ¯ q k in Appendix D.1, we have by Lemma D.9 that

The result follows by taking expectation on both sides of the previous inequality.

For any k ≥ 0 , we have

where the last inequality follows from Part (1) and Part (2) of this lemma. The result follows by taking expectation on both sides of the previous inequality.

For any k ≥ 0 , we have

where c 1 > 0 is an arbitrary positive real number. We next bound the second term on the RHS of the previous inequality. For any s ∈ S , we have

Since the softmax operator σ τ ( · ) is 1 τ - Lipschitz continuous with respect to ∥·∥ 2 [96, Proposition 4], we have

We next analyze the term B 2 . Using Lemma D.7 (1) and the quadratic growth property of strongly convex functions, we have

Combine the upper bounds we obtained for the terms B 1 and B 2 , we obtain

Coming back to Eq. (43), using the previous inequality, we have

Choosing c 1 = 32 |S| A 2 max τ (1 -γ ) 2 in the previous inequality and then taking expectation on both sides, we obtain

The proof is complete.

We next consider the last term on the RHS of Eq. (42), which involves the difference between the operator F i ( q i k , S k , A i k , A -i k , S k +1 ) and its expected version ¯ F i k ( q i k ) , and hence can be viewed as the stochastic error due to sampling. The fact that the Markov chain { ( S k , A i k , A -i k , S k +1 ) } is time-inhomogeneous presents a challenge in our analysis. To overcome this challenge, observe that: (1) the policy (hence the transition probability matrix of the induced Markov chain) is changing slowly compared to the q -function; see Algorithm 3 Line 3 , and (2) the stationary distribution as a function of the policy is Lipschitz (cf. Lemma B.1 (3)). These two observations together enable us to develop a refined conditioning argument to handle the time-inhomogeneous Markovian noise. The result is presented in following. Similar ideas were previous used in [23, 24, 65, 41, 76] for finite-sample analysis of single-agent RL algorithms. Recall that we use α k 1 ,k 2 = ∑ k 2 k = k 1 α k to simplify the notation.

Lemma D.11 (Proof in Appendix D.7.2) . The following inequality holds for all k ≥ z k :

where z k is the mixing time of the Markov chain { S n } n ≥ 0 induced by the joint policy π k = ( π 1 k , π 2 k ) with accuracy β k ; see Eq. (11).

When using constant stepsize, we have z k α k -z k ,k -1 = z 2 β α = O ( α log 2 (1 /β )) . Since the two stepsizes α and β differ only by a multiplicative constant c α,β , we have lim α → 0 z 2 β α = 0 . Similarly, we also have lim k →∞ z k α k -z k ,k -1 = 0 when using diminishing stepsizes.

Using the upper bounds we obtained for all the terms on the RHS of Eq. (42), we have the one-step Lyapunov drift inequality for q i k . Following the same line of analysis, we also obtain the one-step inequality for q -i k . Adding up the two Lyapunov drift inequalities, we arrive at the following lemma.

Lemma D.12. The following inequality holds for all k ≥ z k and i ∈ { 1 , 2 } :

Proof of Lemma D.12. For i ∈ { 1 , 2 } , we have from Eq. (42), Lemma D.10, and Lemma D.11 that

where the second inequality follows from β k = c α,β α k with c α,β ≤ 1 . Since

we have

Summing up the previous inequality for i = 1 , 2 , we have

D.6 Solving Coupled Lyapunov Drift Inequalities

We first restate the Lyapunov drift inequalities from previous sections. Recall our notation L q ( t, k ) = ∑ i =1 , 2 ∥ q i t,k -¯ q i t,k ∥ 2 2 , L π ( t, k ) = max s ∈S V v t ,s ( π 1 t,k ( s ) , π 2 t,k ( s )) , L sum ( t ) = ∥ v 1 t + v 2 t ∥ ∞ , and L v ( t ) = ∑ i =1 , 2 ∥ v i t -v i ∗ ∥ ∞ . Let F t be the history of Algorithm 2 right before the t -th outer-loop iteration. Note that v 1 t and v 2 t are both measurable with respect to F t . In what follows, for ease of presentation, we write E t [ · ] for E [ · | F t ] .

Lemma D.5: It holds for all t ≥ 0 that

Lemma D.6: It holds for all t ≥ 0 that

Lemma D.8: It holds for all t, k ≥ 0 that

Lemma D.12: It holds for all t ≥ 0 and k ≥ z k that

Adding up Eqs. (46) and (47), we have by c α,β ≤ min( 1 L 1 / 2 τ , c τ τ 3 ℓ 2 τ (1 -γ ) 2 128 A 2 max , c τ ) (cf. Condition 3.1) that

D.6.1 Constant Stepsize

When using constant stepsizes, i.e., α k ≡ α , β k ≡ β = c α,β α , repeatedly using Eq. (48) from z β to k , we have

We next bound L π ( t, 0) + L q ( t, 0) . For i ∈ { 1 , 2 } , we have

and

It follows that

Using the previous inequality in Eq. (49), we have

which implies

Substituting the previous inequality on E t [ L π ( t, k )] into Eq. (47), we have

Repeatedly using the previous inequality, since c α,β ≤ c τ (cf. Condition 3.1), we have

which implies (by using Jensen's inequality) that

Substituting the previous bound on E t [ L q ( t, k ) 1 / 2 ] into Eq. (45) and then taking total expectation, we have

where the last line follows from c α,β ≤ c τ τ (1 -γ ) 64 A 2 max (cf. Condition 3.1). Since ∥ v 1 0 + v 2 0 ∥ ∞ ≤ 2 1 -γ , repeatedly using the previous inequality, we have for all k ≥ 0 that

Now we have obtained finite-sample bounds for L q ( t, k ) , L π ( t, k ) , and L sum ( t ) . The next step is to use them in Eq. (44) to obtain finite-sample bounds for L v ( t ) . Specifically, using Eq. (44), Eq. (50), and Eq. (51), we have

Repeatedly using the previous inequality from 0 to T -1 and then using L v (0) ≤ 4 1 -γ , we have

Our next step is to use the bounds we obtained for L q ( t, k ) , L π ( t, k ) , L v ( t ) , and L sum ( t ) in Lemma D.4. For simplicity of presentation, we use a ≲ b to mean that there exists a numerical constant c such that a ≤ cb . Using the previous inequality, Eq. (50), and Eq. (51), we have

The proof of Theorem 3.1 (1) is complete.

D.6.2 Diminishing Stepsizes

Consider using linearly diminishing stepsizes, i.e., α k = α k + h , β k = β k + h , and β = c α,β α . Repeatedly using Eq. (48), we have for all k ≥ k 0 := min { k ′ | k ′ ≥ z k ′ } that

Next, we evaluate the terms { ˆ E j } 1 ≤ j ≤ 3 . Terms like { ˆ E j } 1 ≤ j ≤ 3 have been well studied in the existing literature [24, 44, 65]. Specifically, we have from [65, Appendix A.2.] and β = 4 that

It follows that

which implies

Using the previous inequality on E t [ L π ( t, k )] in Eq. (47), we have

Repeatedly using the previous inequality starting from k 0 , since αc τ ≥ 1 (cf. Condition 3.1), we have

which implies (by using Jensen's inequality) that

Taking total expectation on both sides of the previous inequality and then using the result in Eq. (45), we have

where the last line follows from c α,β ≤ c τ τ (1 -γ ) 2 144 A 2 max (cf. Condition 3.1). Repeatedly using the previous inequality starting from 0 , we have

The next step is to bound L v ( t ) . Recall from Eq. (44) that

E t [ L v ( t +1)] ≤ γ L v ( t ) + 4 L sum ( t ) + 2 E t [ L 1 / 2 q ( t, K )] + 4 E t [ L π ( t, K )] + 6 τ log( A max ) .

Using Eqs. (52), (53), and (54) in the previous inequality, we have

Repeatedly using the previous inequality starting from 0 to T -1 , we have

Finally, using the previous inequality, Eq. (52), and Eq. (54) in Lemma D.4, we obtain

The proof of Theorem 3.1 (2) is complete.

D.7 Proof of All Supporting Lemmas

D.7.1 Proof of Lemma B.1

Lemma B.1 (1), (3), and (4) are identical to [10, Proposition 3]. We here only prove Lemma B.1 (2). Consider the Markov chain { S k } induced by π b . Since { S k } is irreducible and aperiodic, there exists

a positive integer r b such that P r b π b has strictly positive entries [93, Proposition 1.7]. Therefore, there exists δ b ∈ (0 , 1) such that

for all ( s, s ′ ) . In addition, the constant ρ b introduced after Assumption 3.1 is explicitly given as ρ b = exp( -δ b /r b ) . The previous two equations are from the proof of the Markov chain convergence theorem presented in [93, Section 4.3]. Next, we consider the Markov chain { S k } induced by an arbitrary π ∈ Π τ . Since

we have for any s, s ′ ∈ S and k ≥ 1 that

Since the previous inequality holds for all s and s ′ , we in fact have ℓ 2 τ P k π b ≤ P k -1 π b P π (which is an entry-wise inequality). Repeatedly using the previous inequality, we obtain

Following the proof of the Markov chain convergence theorem in [93, Section 4.3], we have

Since A max ≥ 2 (otherwise there is no decision to make in this stochastic game), we have ℓ 2 τ ≤ 1 2 . It follows that 1 -δ b ℓ 2 r b τ µ b, min > 1 / 2 . Using the previous inequality in Eq. (55), we have

We next compute the mixing time. Using the previous inequality and the definition of the total variation distance, we have

which implies as long as

D.7.2 Proof of Lemma D.11

For any k ≥ z k , we have

To bound the terms N 1 to N 5 on the RHS of the previous inequality, the following lemma is needed. Lemma D.13. For any positive integers k 1 ≤ k 2 , we have (1) ∥ q i k 2 -q i k 1 ∥ ∞ ≤ 2 α k 1 ,k 2 -1 1 -γ , and (2) max s ∈S ∥ π i k 2 ( s ) -π i k 1 ( s ) ∥ 1 ≤ 2 β k 1 ,k 2 -1 .

Proof of Lemma D.13. For any k ∈ [ k 1 , k 2 -1] , we have by Eq. (41) that

It follows that ∥ q i k 2 -q i k 1 ∥ ∞ ≤ 2 α k 1 ,k 2 -1 1 -γ . Similarly, for any k ∈ [ k 1 , k 2 -1] and s ∈ S , we have

which implies max s ∈S ∥ π i k 2 ( s ) -π i k 1 ( s ) ∥ 1 ≤ 2 β k 1 ,k 2 -1 .

We next bound the terms N 1 to N 5 . Let F k be the σ -algebra generated the sequence of random variables { S 0 , A i 0 , A -i 0 , · · · , S k -1 , A i k -1 , A -i k -1 , S k } .

The Term N 1 . Using the tower property of conditional expectations, we have

We next bound the two terms on the RHS of the previous inequality. Observe that

where we use E k [ · ] to denote E S ∼ µ k ( · ) ,A i ∼ π i k ( ·| S 0 ) ,A -i ∼ π -i k ( ·| S 0 ) ,S ′ ∼ p ( ·| S 0 ,A i 0 ,A -i 0 ) [ · ] for ease of presentation. To proceed, recall the following equivalent definition of total variation distance between probability measures p 1 , p 2 :

It follows that

Therefore, we have

It remains to bound the second term on the RHS of Eq. (58). Recall that we denote P π ∈ R |S|×|S| as the transition probability matrix of the Markov chain { S k } induced by a joint policy π . Using the definition of conditional expectations, we have

where the last line follows from Lemma B.1 (2). Observe that

Since P π as a function of π is 1 -Lipschitz continuous with respect to the ℓ ∞ -norm, we have

Using the previous inequality in Eq. (60), we have

Using the previous inequality and Eq. (59) together in Eq. (58), we obtain

The Term N 2 . For any k ≥ z k , we have by Lemma D.13 that

Using the definition of F i ( · ) , we have

Moreover, we have by Jensen's inequality that

Using Eqs. (62) and (63) together in Eq. (61), we have

The Term N 3 . For any k ≥ z k , we have

Observe that

where the last line follows from Lemma D.13 and

Therefore, we have

The Term N 4 . For any k ≥ z k , we have

where the last line follows from Lemma D.1. Using the definition of F i ( · ) , we have

It follows that

The Term N 5 . For any k ≥ z k , we have

where the last line follows from the same analysis as we obtain Eq. (59). As for the second term on the RHS of Eq. (64), using the definition of ¯ F i k ( · ) , we have

Using the previous inequality in Eq. (64), we obtain

Combining the upper bounds we derived for the terms N 1 to N 5 in Eq. (56), we have

where the last line follows from β k /α k = c α,β ≤ 1 60 L p |S| A max (cf. Condition 3.1).

D.8 Proof of Corollary 3.1.1

The following proof idea was previous used in [15] to show the rationality of their decentralized Q -learning algorithm.

Observe that Theorem 3.1 can be easily generalized to the case where the reward is corrupted by noise. Specifically, suppose that player i takes action a i and player -i takes action a -i . Instead of assuming player i receives a deterministic reward R i ( s, a i , a -i ) , we assume that player i receives a random reward r i ( s, a i , a -i , ξ ) , where ξ ∈ Ξ ( Ξ is a finite set) is a random variable with distribution µ ξ ( s ) , and is independent of everything else. The proof is identical as long as r i + r -i = 0 , and the reward is uniformly bounded, i.e., max s,a i ,a -i ,ξ | r i ( s, a i , a -i , ξ ) | < ∞ . Now consider the case where player i 's opponent follows a stationary policy π -i . We incorporate the randomness of player -i 's action into the model and introduce a fictitious opponent with only one action a ∗ . In particular, let the random reward function be defined as ˆ r i ( s, a i , a ∗ , A -i ) = R i ( s, a i , A -i ) for all ( s, a i ) , where A -i ∼ π -i ( ·| s ) , and let ˆ p ( s ′ | s, a i , a ∗ ) = ∑ π -i ( a -i | s ) p ( s ′ | a i , a -i , s ) . Now the problem can be reformulated as player i playing against the fictitious player with a single action a ∗ , with reward function ˆ r i ( i ∈ { 1 , 2 } ) and transition probabilities ˆ p . Using the same proof for Theorem 3.1, we have the desired finite-sample bound.

E On the Mixing Time of MDPs with Almost Deterministic Policies

Consider an MDP with two states s 1 , s 2 and two actions a 1 , a 2 . The transition probability matrix P 1 of taking action a 1 is the identity matrix I 2 , and the transition probability matrix P 2 of taking action a 2 is P 2 = [0 , 1; 1 , 0] . Given α ∈ (1 / 2 , 1) , let π α be a policy such that π ( a 1 | s ) = α and π ( a 2 | s ) = 1 -α for any s ∈ { s 1 , s 2 } . Denote P α as the transition probability matrix under π α . It is easy to see that

Since P α is a doubly stochastic matrix, and has strictly positive entries, it has a unique stationary distribution µ = 1 ⊤ / 2 .

We next compute a lower bound of the mixing time of the π α -induced Markov chain. Let e 1 = [1 , 0] ⊤ be the initial distribution of the states, and denote [ x k , 1 -x k ] ⊤ as the distribution of the states at time step k . Then we have

It follows that

which implies lim α → 1 t α,η = ∞ . Therefore, as the policies become deterministic, the mixing time of the associated Markov chain can approach infinity.

F Numerical Simulations

We first conduct numerical simulations to investigate the impact of choosing different τ , which is used to define the softmax operator in Algorithms 1 and 2. Our theoretical results indicate that there is an asymptotically non-vanishing bias due to using a positive τ . Intuitively, since a softmax policy always has strictly positive entries while a Nash equilibrium policy can have zero entries, we cannot, in general, expect the Nash gap to converge to zero.

To demonstrate this phenomenon, consider the following example of a zero-sum matrix game. Let

be the payoff matrix for player 1 , and let R 2 = -( R 1 ) ⊤ , where N > 0 is a tunable parameter. Note that this matrix game has a unique Nash equilibrium, which goes to the joint policy π 1 = (1 / 3 , 2 / 3 , 0) , π 2 = (0 , 2 / 3 , 1 / 3) as N → ∞ . In our simulations, we use constant stepsizes α k ≡ 0 , 5 and β k ≡ 0 . 01 and run Algorithm 1 for 100 trajectories (each has K = 2000 iterations). Then, we plot the average Nash gap (averaged over the 100 trajectories) as a function of the number of iterations k in Figure 1 for different temperatures τ . To enable a fair comparison, we use the normalized q -function to compute the softmax, that is, instead of directly using σ τ ( q i k ) in Algorithm 1, we use σ τ ( q i k / ∥ q i k ∥ 2 ) . As we can see in Figure 1, as τ increases, the asymptotic error also increases, which is consistent with our theoretical results.

F.1 Comparison with the Optimistic Multiplicative-Weights Update

The Optimistic Multiplicative-Weights Update (OMWU) was recognized as a popular learning algorithm for zero-sum matrix games [98]. Since OMWU in the payoff-based setting (or noisyfeedback setting) may not have last-iterate convergence [99], to enable a fair comparison, we will compare OMWU in the noiseless setting (which does enjoy last-iterate convergence [98]) with smoothed-best response dynamics. We start by writting down the algorithm.

OMWU: With initializations π 1 0 , π 1 1 (respectively, π 2 0 , π 2 1 ) that live in the interior of the probability simplex ∆( A 1 ) (respectively, ∆( A 2 ) ), OMWU updates ( π 1 k , π 2 k ) iteratively according to

where η ∈ (0 , 1) is the stepsize.

50

T=0.01

45

T=0.1

40

T=0.2

35

T=0.5

Gap

30 25 20

T=1

Nash

15

10

5

0

-5

0

500

1000

1500

2000

Number of Iterationsk

OMWU

1.8

DSBR

1.6

1.4

Gap

1.2

Nash

1

0.8

0.6

0.4

0.2

0

-0.2

0

50

100

150

Number of Iterationsk

The Discrete Smoothed Best-Response Dynamics (DSBR): With arbitrary initializations π 1 0 ∈ ∆( A 1 ) and π 2 0 ∈ ∆( A 2 ) , the discrete smoothed best-response dynamics update ( π 1 k , π 2 k ) iteratively according to

where β k is the stepsize.

We perform two sets of numerical simulations to compare OMWU and DSBR. Our first experiment is implemented on the rock-paper-scissor game, where the payoff matrix for player 1 is

and R 2 = -( R 1 ) ⊤ . As we see in Figure 2, the convergence rates of OMWU and DSBR are comparable. However, DSBR seems to be more stable compared with OMWU. Note that while we use softmax policies in DSBR, since the rock-paper-scissor game has a unique Nash equilibrium, which is also the unique Nash equilibrium of the entropy-regularized matrix game for any temperature τ > 0 , there is no smoothing bias and the Nash gap under DSBR does converge to zero.

OMWU

1.8

DSBR

1.6

1.4

Gap

1.2

Nash

1

0.8

0.6

0.4

0.2

0

-0.2

0

100

200

300

400

500

600

700

800

Number of Iterationsk

0.2

OMWU

0.15

DSBR

0.1

Gap

Nash (

0.05

0

-0.05

-0.1

600

650

700

750

800

Number of Iterationsk

In our second numerical simulation, we set the payoff matrix of player 1 to be

and R 2 = -( R 1 ) ⊤ , where we choose N = 100 . Note that as N →∞ , the unique Nash equilibrium goes to π 1 = (1 / 3 , 2 / 3 , 0) , π 2 = (0 , 2 / 3 , 1 / 3) . In this case, we also see from Figure 3 that DSBR is more stable compared with OMWU. However, since in this case, the Nash equilibrium has zero entries, due to the use of softmax policies, DSBR suffers from an asymptotically non-vanishing bias. This is clear from Figure 4, which plots the asymptotic behavior of Figure 3. We see that OMWU converges to zero while the Nash gap from DSBR converges to a positive real number.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a doubly smoothed best-response dynamics for two-player zero-sum Markov games, with matrix games as the degenerate case. Upper bounds of Nash Gap with bias and regularized Nash Gap without bias are presented.\n\nSTRENGTHS:\nThe technical sections are concrete and solid. The idea of simultaneously smoothly updating in the value (q) space and the policy (simplex) space appears novel. In such a way, each player is facing a much more stable environment than simply smoothing in the value space. As a consequence, the convergence results are natural outcomes of such an algorithmic design. Moreover, the Assumption 3.1 makes the results more appealing in a sense that it deviates from typically made assumptions in RL analysis.\n\nWEAKNESSES:\nSome major concerns:\n\n(1) The \"first form of smoothing\" in the policy space is very similar to the mixed strategy in fictitious self-play but not exactly the same. In fictitious self-play, the policies are mixed in the way of having $1-\\alpha_k$ vs $\\alpha_k$ probabilities to select the old policy vs the new best response. In comparison, here the authors simply averaged two stochastic strategies. Even with the explanations provided in the paper, it is still not very clear to me why the authors would want to deviate from the classical practice in learning Markov games (or extensive form games), which is backed by enormous amount of practical implementations.\n\n(2) While the authors claim that the learning dynamics are independent for the two players, the learning rates of them are tied through a constant, which makes the claim somewhat questionable. Please elaborate more about your definition of being \"independent\".\n\n(3) The bound for Nash gap is somewhat weak. Apart from the term $\\ell_\\tau$ that is exponential in $\\tau$, the constant $c_2$ also has both $O(\\tau)$ and  $O(\\tau^{-1})$ terms, which requires balancing. Overall, I think a corollary with the best choice of $\\tau$ is highly desirable to at least let readers to directly evaluate the strength of the bound.\n\n(4) The bound of regularized Nash gap is not at all surprising -- it appears rather standard in minimax (saddle point) optimization with regularized geometry. So the real emphasis should be the Nash gap itself. Although I see the authors discuss the natural difficulty for softmax policy algorithms to converge without bad dependencies on the temperature $\\tau$, it still feels quite vague why this is the case -- I believe a deeper illustration with math will greatly strengthen the paper -- it won't make the Nash gap look as unsatisfying as it appears now.\n\nQUESTIONS:\nPlease see the Weakness section.\n\nLIMITATIONS:\nYes, there is no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied a best-response type learning dynamics in two-player zero-sum stochastic games called doubly smoothed best-response dynamics. The dynamics uses smoothed value function updates with softmax smoothed best-response, and combines minimax value iteration. Ths dynamics is payoff-based, convergent, rational, and symmetric. The author also provided the first finite-sample analysis of these type of dynamics, showing convergence to Nash equilibrium up to smoothing bias.\n\nSTRENGTHS:\n1. Finite-sample analysis (or convergence rates analysis) for payoff-based (or noisy bandit feedback-based) learning dynamics in stochasitc games is an interesing question with practical importance. This paper contributes to this area by the first finite-sample analysis of a payoff-based best-response-type independent learning dynamics for zero-sum stochastic games. \n2. This paper is well-written and easy to follow. I appreciated it that the authors provided detailed discussion and high-level ideas of the algorithm and proof sketch for the main results.\n\nWEAKNESSES:\nThe claim that the proposed dynamics is *convergent* and *rational* seems not rigorous since the smoothing bias $\\tau$ persists over time and no convergence to exact Nash equilirium or best response is really proved. One possible fix might be introducing a decreasing parameter $\\tau_t$ so that exact convergence can be shown. This might significantly slow down the convergence of other bias terms because of the exponetionally dependence on $\\tau$. The proposed dynamics is thus very slow in terms of convergence to Nash equilirium and this is a weakness of the current work. \n\nMinor comments:\n\n1. Line 277, $Q(s,a^{i}, a^{-i})$ has not been defined.\n2. Under Line 284: one term should be $R^i(s, a^{i}, a^{-i})$.\n\nQUESTIONS:\n1. Do Algorithm 1 and 2 provide sublinear regret guarantee for one player when the other player might be an adversary?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose algorithms for learning the Nash equilibrium in two-player games and two-player stochastic games. The algorithm for games is effectively a single-time scale algorithm (Doubly smoothed best response dynamics). While , the algorithm for stochastic games (Double smoothed best response dynamics with value iteration) is a two-time scale approach. They show that these algorithm is independently implemented by each agent without any communication requirement and the rewards are based on actual payoffs obtained after each each action, rather than the full information setting. Also, they also show that the players reach a policy that is a best-response of the other players' (stationary) policies.\n\nSTRENGTHS:\nThe aim to reach convergence in last-iterate in zero-sum games and stochastic zero-sum games through independent learning is a challenging problem and the authors provide not only a finite sample guarantee, but also in addition, that the learned policies are best-responses to the stationary policies of other agents and achieve \"rational\" learning (Bowling and Veloso ' 2001). Also, the assumption about existence of a joint policy that induces an irreducible and aperiodic Markov chain is relatively weaker than other existing assumptions, that assume any policy always induces such an irreducible and aperiodic Markov chain or any policy created by the algorithm's trajectory is uniformly geometrically ergodic. They use coupled Lyapunov drift inequalities to guarantee convergence and finite sample guarantees.\n\nWEAKNESSES:\n1) If the game has a non-interior Nash, do you still get convergence? Lemma D.1 seems to say that all strategies are played with a probability lower bounded by a certain value.\n2) Is there any hope of getting a stronger probabilistic guarantee for at least the two-player zero-sum games setting? Essentially to understand the convergence of the iterates (in distribution or w.h.p)? instead of an expected sense as derived in this paper).\n3) Authors state that prior two-time scale approaches might require implicit coordination amongst players, however, their algorithm for the stochastic games is also two-time scale and it would help if they could clarify how they are able to avoid this in the algorithm and how they make up for this in the analysis.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThey have mentioned the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The authors focus on the problem of finite-sample convergence analysis of independent best-response-type learning dynamics in two-player zero-sum stochastic games. The dynamics are payoff based and stimulate the Shapley operator (that is known to be a contractive). Under the assumption that for any pair of policies, the induced Markov chain is irreducible and aperiodic, it is shown that in expectation the Nash gap is at most epsilon after 1/eps steps of the algorithm. Finally the authors provide results rationality type results (i.e., on the regret when the one player follows the dynamics and the other plays a stationary policy).

STRENGTHS:
I think that the result of 1/eps rate is quite interesting and surprising (I would expect a rate of 1/eps^2). Moreover the techniques seem highly non-trivial (design of a novel Lyapunov type function).

WEAKNESSES:
Sometimes the write-up is not self-contained, but this is because of the limited space.

QUESTIONS:
Can you please explain if your results carry over for other settings like Markov potential Games?

LIMITATIONS:
Not applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-13