Title: Learning Equilibria in Adversarial Team Markov Games: A Nonconvex-Hidden-Concave Min-Max Optimization Problem
Venue: NeurIPS 2024
Accepted: yes
Keywords: MARL

Convex RL

Nash Equilibrium

Non-smooth Optimization

Minimax Optimization

Hidden Convexity

Nonconvex-nonconcave

Markov Games

Stochastic Games

Learning in Games
Abstract: We study the problem of learning a Nash equilibrium (NE) in Markov games which is a cornerstone in multi-agent reinforcement learning (MARL). In particular, we focus on infinite-horizon adversarial team Markov games (ATMGs) in which agents that share a common reward function compete against a single opponent, *the adversary*. These games unify two-player zero-sum Markov games and Markov potential games, resulting in a setting that encompasses both collaboration and competition. Kalogiannis et al. (2023) provided an efficient equilibrium computation algorithm for ATMGs which presumes knowledge of the reward and transition functions and has no sample complexity guarantees. We contribute a learning algorithm that utilizes MARL policy gradient methods with iteration and sample complexity that is polynomial in the approximation error $\epsilon$ and the natural parameters of the ATMG, resolving the main caveats of the solution by (Kalogiannis et al., 2023). It is worth noting that previously, the existence of learning algorithms for NE was known for Markov two-player zero-sum and potential games but not for ATMGs.
    
Seen through the lens of min-max optimization, computing a NE in these games consists a nonconvex--nonconcave saddle-point problem. Min-max optimization has received an extensive study. Nevertheless, the case of nonconvex--nonconcave landscapes remains elusive: in full generality, finding saddle-points is computationally intractable (Daskalakis et al., 2021). We circumvent the aforementioned intractability by developing techniques that exploit the hidden structure of the objective function via a nonconvex--concave reformulation. However, this introduces a challenge of a feasibility set with coupled constraints. We tackle these challenges by establishing novel techniques for optimizing weakly-smooth nonconvex functions, extending the framework of (Devolder et al., 2014).

Learning Equilibria in Adversarial Team Markov Games: A Nonconvex-Hidden-Concave Min-Max Optimization Problem

Abstract

We study the problem of learning a Nash equilibrium (NE) in Markov games which is a cornerstone in multi-agent reinforcement learning (MARL). In particular, we focus on infinite-horizon adversarial team Markov games (ATMGs) in which agents that share a common reward function compete against a single opponent, the adversary . These games unify two-player zero-sum Markov games and Markov potential games, resulting in a setting that encompasses both collaboration and competition. [65] provided an efficient equilibrium computation algorithm for ATMGs which presumes knowledge of the reward and transition functions and has no sample complexity guarantees. We contribute a learning algorithm that utilizes MARL policy gradient methods with iteration and sample complexity that is polynomial in the approximation error ϵ and the natural parameters of the ATMG, resolving the main caveats of the solution by [65]. It is worth noting that previously, the existence of learning algorithms for NE was known for Markov two-player zero-sum and potential games but not for ATMGs.

Seen through the lens of min-max optimization, computing a NE in these games consists a nonconvex-nonconcave saddle-point problem. Min-max optimization has received extensive study. Nevertheless, the case of nonconvex-nonconcave landscapes remains elusive: in full generality, finding saddle-points is computationally intractable [33]. We circumvent the aforementioned intractability by developing techniques that exploit the hidden structure of the objective function via a nonconvex-concave reformulation. However, this introduces the challenge of a feasibility set with coupled constraints. We tackle these challenges by establishing novel techniques for optimizing weakly-smooth nonconvex functions, extending the framework of [35].

1 Introduction

Multi-agent reinforcement learning (MARL) investigates behaviors of multiple interacting agents within a dynamic, shared environment where the actions of each agent not only impact their individual rewards but also the overall state of the system. MARL has introduced several practical techniques that have justifiably captured public interest in recent years, particularly in skill-intensive games like starcraft, go, chess, and poker [12, 97, 101, 79, 16, 15, 14, 86], where its empirical methods have

achieved super-human performance. More recently, MARL methods combined with large language models has excelled in the game of Diplomacy [6]. Despite these practical achievements, theoretical understanding of MARL has lagged behind its empirical successes.

Markov games (MGs) [95] is a rigorous and versatile mathematical structure that MARL employs to systematically formalize the strategic interactions in the dynamic settings [71]. These games extend Markov decision processes (MDPs) [88] to multiple agents, each making decisions and receiving rewards independently as the environment evolves. The joint decisions of the agents influence both individual rewards and the transition of the environment. MARL in general is occupied with leading the multi-agent system to a favorable outcome. Through the lens of game theory, the notion of a 'favorable outcome' is formally defined through concepts like a Nash equilibrium and a (coarse) correlated equilibrium. Although computing Nash equilibria is generally computationally intractable-even in two-player games without states [28, 24]--it becomes tractable in fully cooperative settings like Markov potential games [114, 68] and is also tractable in competitive scenarios such as two-player zero-sum Markov games [27, 103, 17]. Recent advances [65] also show computational tractability in adversarial team Markov games (ATMGs)-a context that combines both cooperative and competitive dynamics among agents. More specifically, an infinite-horizon adversarial team Markov game (ATMG) is a Markov Game in which n team players, compete against one adversary. Each of the team players receives the same reward and is equal to minus the reward of the adversary. ATMGs generalize both Markov zero-sum and potential games; the former can be viewed as ATMGs with n = 1 , the latter by choosing the adversary to be dummy (having one action).

This paper aims to develop learning methods to approximate Nash equilibria in team Markov games by using only individual rewards and state observations as feedback, addressing the following question and answering one of the main caveats of the solutions provided in [65]:

Is it possible for agents to efficiently learn Nash equilibria in adversarial team Markov games, having only access to trajectory roll-out samples and (almost ⋆ ) no communication, i.e., independently? ( ⋆ )

1.1 Our Contributions

Let us provide some context before stating our main results. An infinite-horizon adversarial team Markov game (ATMG) is characterized by a finite state-space S , n team players, each equipped with a finite action-space A i , i ∈ { 1 , . . . , n } , and one adversary with a finite action-space B . Each of the team players receives the same reward which is equal to minus the reward of the adversary. The adversary's value function is defined as the discounted expected sum of their rewards, where the discount factor is γ ∈ [0 , 1) . An approximate Nash equilibrium is a product distribution over policy space such that no agent can improve their value by unilaterally deviating. We propose a learning algorithm that has both iteration and sample complexity polynomial in the parameters of the Markov Game and returns approximate Nash equilibria.

Theorem 1.1 (Informal Version of Theorem 3.3) . There is a learning algorithm ( ISPNG ) that uses bandit feedback and guarantees convergence to an ϵ -approximate Nash equilibrium in adversarial

team Markov games, the sample and iteration complexities of which are

We deem noteworthy that our algorithm manages to compute a Nash equilibrium in a Markov game, which combines opposing and shared agent interests, by only using a number of iterations and samples that is polynomial in the approximation error and the description of the game. Further, it manages to beat the curse of multi-agents [62]i.e., its iteration and sample complexity depends on ∑ n k =1 |A i | instead of ∏ n k =1 |A i | .

In order to achieve the latter contribution, we acquired convergence guarantees for stochastic projected gradient descent in nonconvex functions when the gradient is Hölder-continuous-a notion of continuity weaker than that of Lipschitz. Finally, we contribute a general result that guarantees convergence to a saddle-point in functions that are nonconvex-hidden-strongly-concave.

1.2 Technical Overview

The problem of computing an approximate Nash equilibrium in an adversarial team Markov game boils down to computing an approximate saddle-point ( x ∗ , y ∗ ) of the adversary's value function V ( x , y ) ; see Definition 2.3. The variables x denote the policies of the team, each member of which aims to individually minimize V . Moreover, y denotes the policy of the adversary who aims to maximize V . The equivalence between saddle-points and equilibria is due to (i) the game being zero-sum between the team and the adversary and (ii) the gradient domination property (see Lemma C.7) that holds per player, and has already been established in prior works [3, 68, 114]. In words, gradient domination in our setting implies that any approximate first-order stationary policy is also an approximate best response for that player.

The problem of computing an approximate saddle-point ( x ∗ , y ∗ ) of the objective V ( x , y ) poses computational challenges due to its nonconvex-nonconcave nature. Previous work [65] showed that one can compute an approximate saddle-point ( x ∗ , y ∗ ) of V , by first obtaining an approximate stationary point x ∗ of Φ( x ) = max y V ( x , y ) through a Moreau envelope argument and then extending it to ( x ∗ , y ∗ ) . The proof of extendibility uses involved arguments that utilize the Lagrange multipliers of a carefully chosen nonlinear program (for the stationary point x ∗ ), while the computation of y ∗ requires solving another linear program. It is worth noting that the aforementioned linear program presumes access to the full description of the reward function and the transition model of the underlying Markov game when the team plays policy x ∗ . This fact prevents the possibility of casting this approach into a learning algorithm.

Our proposed (learning) algorithm bypasses the requirement for knowledge of the reward function and the transition model, and works under the bandit feedback framework. The first idea behind our algorithm is to consider the adversary's value function as a function F of the adversary's state-action visitation measure λ , F ( x , λ ) := V ( x , y ) , and the addition of a regularizing term -ν 2 ∥ λ ∥ 2 ( ν can be thought of as a small positive scalar). As a result, the max function of the regularized value function, Φ ν ( x ) := max λ ∈ Λ( x ) { F ( x , λ ) -ν 2 ∥ λ ∥ 2 } , is differentiable, where Λ( x ) ⊆ ∆ |S||B| denotes the feasibility set of λ and depends on x . Effectively, different policies, x , for the team induce a different single agent Markov decision process for the adversary. The addition of the regularizer allows us to apply Danskin's theorem on a function with a unique maximizer circumventing the necessity of solving a linear program; one only needs to approach that unique solution. To the best of our knowledge, this is the first work introducing a function of λ as a regularizing term.

By reformulating the regularized value function using state-action visitation measure λ , the problem boils down to learning an approximate saddle-point of a nonconvex-strongly-concave function with coupled constraints . Coupled constraints are a type of constraints that cannot be expressed as a Cartesian product (the main well-studied setting in min-max optimization [63]), i.e. , the feasibility set Λ( x ) , depends on x . The first challenge towards handling the coupled constraints is to argue that ∇ Φ ν is Hölder-continuous which is a notion of continuity weaker than Lipschitz continuity (see Definition 2.1). Specifically, in Theorem 3.2, we show that Φ ν ( x ) is weakly-smooth, or equivalently, ∇ Φ ν is Hölder-continuous. It seems unlikely that we could use Moreau envelope techniques to prove convergence of stochastic projected gradient descent on a weakly-smooth function. The next

step of our proof is to transfer the weakly-smooth nonconvex optimization problem into a smooth optimization problem with inexact gradient oracles, extending the techniques from [35] to nonconvex and constrained settings. Since we only allow each player to observe the reward they received and not the action chosen by the other players (including the adversary), one last challenge we have to deal with is the inability to estimate the state-action visitation measure λ of the adversary, making the gradient inexact when computing ∇ Φ ν ( x ) in both deterministic and stochastic settings.

2 Preliminaries

Starting, we will introduce the notation conventions we use and split the rest of the preliminaries into two subsections. Section 2.1 provides necessary definitions whereas Section 2.2 deals with the preliminaries of (adversarial team) Markov games and the notion of Nash equilibrium.

Notation. We denote [ n ] := { 1 , . . . , n } . We use superscripts to denote the (discrete) time index, and subscripts to index the players. We use boldface for vectors and matrices; scalars will be denoted by lightface variables. We define ∥ · ∥ 2 , ∥ · ∥ 1 , ∥ · ∥ ∞ to be the ℓ 2 -norm, the ℓ 1 -norm and the ℓ ∞ norm respectively. The simplex of probability vectors supported on a finite set A is noted as ∆( A ) . Unless specified otherwise, we denote ∥ · ∥ 2 by ∥ · ∥ . Diam X denotes the diameter of a compact set X in ℓ 2 -distance. For simplicity in the exposition, we may sometimes use the O ( · ) notation to suppress dependencies that are polynomial in the natural parameters of the problem and ˜ O ( · ) to further hide logarithmic factors; precise statements are given in the Appendix. For the convenience of the reader, a comprehensive overview of our notation is given in Table 1.

2.1 Basic Definitions and Facts

We commence this subsection by introducing a number of concepts and statements of mathematical analysis and optimization. We define Hölder continuity and the notion of a stationary point in constrained minimization and min-max optimization.

The notion of Hölder continuity of the gradient is a weaker notion of Lipschitz gradient continuity.

Definition 2.1 ( p -Hölder continuous gradient) . A function ϕ : R d → R is said to have a ( ℓ p , p ) -Hölder continuous gradient if for every z , z ′ ∈ R d , it holds that:

When p = 1 , we retrieve the definition of an ℓ -smooth function.

Throughout, following standard conventions, we will refer to functions for which the gradient is p -Hölder continuous with a p < 1 as weakly-smooth . We state the notions of first-order stationarity relevant to our work.

Definition 2.2 ( ϵ -FOSP) . In the context of the constrained minimization problem min z ∈Z ϕ ( z ) , a point z ∈ Z is said to be an ϵ -approximate stationary point if,

Similarly, we will define an ϵ -approximate saddle-point for the constrained min-max optimization problem min X max Y f ( x , y ) .

Definition 2.3 ( ϵ -SP) . Let a function f : X × Y → R . A point ( x , y ) ∈ X × Y is said to be an ϵ -approximate saddle-point (or ϵ -FOSP for the min-max problem) if,

2.2 Adversarial Team Markov Games

An adversarial team Markov game is the Markov game extension of normal-form adversarial team games [98]. The game takes place in an infinite-horizon discounted setting where a team of identicallyinterested players compete against one adversarial player, the adversary . We can formally define an adversarial team Markov game as a tuple Γ( S , [ n +1] , A , B , r, P , γ, ρ ) , where:

S is the finite set of states, or state-space , with cardinality S := |S| ;

[ n +1] is the set of players, with the first n players belonging to the team and the last one being the adversary;

A = × n i =1 A i is the finite set of the team's joint actions (or, team's action-space ), while A i is the i -th player's action-space ; respectively B is the adversary's action-space; further, A := max i ∈ [ n ] |A i | and B := |B| ;

r : S × A × B → [0 , 1] is the adversary's reward function;

P : S × A × B → ∆( S ) is transition probability function;

γ ∈ [0 , 1) is the discount factor;

ρ ∈ ∆( S ) is the initial state distribution. We assume that ρ is of full-support, ρ ( s ) > 0 , ∀ s ∈ S .

Every team player i ∈ [ n ] gets the same reward and the sum of team players' rewards are equal to the adversary's loss, i.e. , ∑ n i =1 r i ( s, a , b ) = -r ( s, a , b ) .

2.2.1 Policies, Value Function, and Visitation Measures

In this part, we describe policy classes, the value function, and the state-action visitation measures. All of these notions are indispensable for our analysis.

Policy Definitions. For any agent i , a stationary policy π i is defined as a mapping from any given state to a probability distribution over possible actions, where π i : S ∋ s ↦→ π i ( ·| s ) ∈ ∆( A i ) . A policy π i is described as deterministic when, for any state, it selects a particular action with probability of 1 . To simplify, we denote the policy spaces for the team and the adversary as Π team : S → ∆( A ) and Π adv : S → ∆( B ) , respectively. Additionally, the combined policy space for all participants can be represented as Π : S → ∆( A ) × ∆( B ) .

Direct Policy Parametrization. In the context of our work, we assume the strategy of direct policy representation for all players. Specifically, for each player i within the set [ n ] , the policy space X i is defined as ∆( A i ) S , with π i = x i , such that the probability of choosing action a in state s , x i,s,a , equals π i ( a | s ) . By the usual game-theoretic convention, π -i denotes the policy of all agents apart from i . For the adversary, Y is set as ∆( B ) S , with π adv = y , so that y s,a = π adv ( a | s ) .

Having defined policies, we can introduce some standard shortcut notations such as r ( s, x , y ) := E ( a b ) ∼ ( x , y ) [ r ( s, a , b )] , and the vectors r ( x ) ∈ R |S|×|B| , r ( x , y ) ∈ R |S| with r ( x ) := [ E a ∼ x [ r ( s, a , b )]] s,b and r ( x , y ) := [ E ( a ,b ) ∼ x [ r ( s, a , b )] ] s . Further, we define P ( s ′ | s, x , y ) as P ( s ′ | s, x , y ) := E ( a ,b ) ∼ ( x , y ) [ P ( s ′ | s, a , b )] and the vector P ( s, x , b ) ∈ ∆( S ) with P ( s, x , y ) := [ E ( a ,b ) ∼ ( x , y ) [ P ( s ′ | s, a , b )] ] s ′ .

The Value Function. The value function V s , for a given state s ∈ S , is defined as the adversary's expected total discounted reward over time under a combined policy ( π team , π adv ) from the policy space Π , with x = π team being the aggregation of policies ( π 1 , . . . , π n ) . Formally, this is represented as

where the expected value is calculated over the distribution of trajectories generated by the policies x and y . If the initial state is instead sampled from a distribution ρ , the value function is expressed as V ρ ( x , y ) = E s ∼ ρ [ V s ( x , y )] .

Visitation Measures. The important quantity of state-action visitation measures, or the expected discounted sum of visitations of a state-action pair.

Definition 2.4 (State-Action Visit. Measure) . For any initial distribution ρ ∈ ∆( S ) , transition matrix P , a team policy x , and a policy y ∈ Y , we define the station-action visitation measure of the adversary λ ( y ; x ) as follows:

Where λ s,b ( y ; x ) denotes the ( s, b ) th entry of λ ( y ; x ) .

As we will further discuss in the appendix (Appendix C.1), the correspondence between y and λ is '1-1' for a fixed team policy x . This property is crucial for our contributions.

Reformulation of the Value Function. A key property of the value function V ρ is that it can be rewritten as a concave function of the state-action visitation measure:

Definition 2.5 ( ϵ -NE) . A product policy ( x ∗ , y ∗ ) ∈ X × Y is called an ϵ -approximate Nash equilibrium for an ϵ ≥ 0 , when

2.2.2 The Gradient and Visitation Measure Estimators.

An essential element that led to the development of policy gradient methods is the policy gradient theorem [104]. Notably, it has enabled the design of finite-sample gradient estimators. This technique fits well into the MARL independent learning protocol [27]. After all agents have proposed their policy, the MDP is run to acquire batches of trajectories from which all agents will observe the chain's state and their individual reward. These samples are utilized to estimate gradients.

The team agents implement a batch version of the REINFORCE estimator whose definition is deferred to the Appendix C.6.1. As for the estimators that the adversary utilizes, we define the state-action visitation measure estimator and their gradient estimator closely following [113].

Definition 2.6 (State-Action Visitation Measure Estimator) . Let e s,b be the standard basis for the ( s, b ) th entry. Let τ = ( s 0 , b 0 , s 1 , b 1 , · · · , s H -1 , b H -1 ) denote a trajectory with length H sampled under initial distribution ρ and policy y We define the estimator for λ ( y ; x ) with the trajectory τ as the following

By applying policy gradient theorem [104] along with the chain-rule, the gradient estimator for a value-function that is nonlinear in λ ( y ; x ) , is computed by the following estimator [113].

Definition 2.7 (Gradient Estimator) . Let τ = ( s 0 , b 0 , s 1 , b 1 , · · · , s H -1 , b H -1 ) denote a trajectory with length H sampled under initial distribution ρ and policy y . Let F ( λ ( y )) be the value function of the MDP w.r.t. λ ( y ) and u := ∇ λ F ( λ ( y )) . The estimator for gradient ∇ y F ( λ ( y )) using the sampled trajectory τ is defined as

Sufficient Exploration. A standard, while rather naive, technique of bounding the variance of the REINFORCE gradient estimator is using ζ -greedy policy parametrization. Effectively, every action in a player's dispose is played with a probability of at least ζ . For our convenience, we ensure sufficient exploration by a ζ -truncated simplex approach. Moreover, for a given feasibility set X , we denote X ζ to be the ζ -truncated feasibility set.

3 Main Results

We present our main results in two different subsections. In Section 3.1 we manage to attain guarantees for convergence to an approximate stationary-point to constrained nonconvex optimization with an stochastic inexact gradient oracle- we do so by extending previous results of [35]. While in Section 3.2, we apply the latter results along with RL techniques in order to design the first learning algorithm that computes a Nash equilibrium in ATMGs.

3.1 Stochastic Weakly-Smooth Nonconvex Optimization with Inexact Gradients

In this subsection we prove that projected gradient descent with a stohcastic inexact gradient oracle converges to an ϵ -FOSP in nonconvex functions with Hölder continuous gradients. We will use this key result in subsequent sections. We begin by defining the inexact gradient oracle and its stochastic version.

Definition 3.1 (Inexact Gradient Oracle) . Let a differentiable function ϕ ( z ) and its gradient ∇ ϕ ( z ) . We call the vector-valued function g ( z ) a ϑ -inexact gradient oracle if,

Further, given a random variable ξ in some sample space Ξ , we define a stochastic inexact gradient oracle G : Z× Ξ → R d . We assume that the expected value of this oracle will be equal to a ϑ -inexact gradient oracle g ( z ) . Additionally to being unbiased (with respect to a ϑ -inexact gradient oracle), we assume its variance to be bounded.

Assumption 3.1 (Unbiased and Bounded Variance) . For a variance parameter σ 2 > 0 , the gradient oracle G , satisfies

Following, we consider the simple update rule of Mini-Batch Inexact Stochastic Projected Gradient Descent , with a batch size M > 0 and ˆ g t = 1 M ∑ M j =1 G ( z t , ξ t j ) ,

We can now state our convergence Theorem for (Inexact Stoch-PGD) whose proof we defer to the appendix.

Theorem 3.1 (Convergence to ϵ -FOSP; Formally in Theorem B.1) . Let ϕ : Z → R be a Lipschitz continuous function with ( ℓ p , p ) -Hölder continuous gradient and a desired accuracy ϵ . Also, let a stochastic inexact first-order oracle G satisfying Assumption 3.1. The update rule (Inexact Stoch-PGD) , with a step-size η = O ( ϵ 1 -p p ) , computes an ϵ -approximate stationary point after T = O ( ϵ -1+ p p ) iterations.

3.2 Learning Nash Equilibria in Adversarial Team Markov Games

In this subsection we state our contributed Algorithm 1, or ISPNG, which converges to an ϵ -NE for any ATMG, Γ , with an iteration and sample complexity that scales polynomially with 1 /ϵ and the parameters of Γ . To simply describe the algorithm, the team players initialize their policies and then the following two steps are repeated for T iterations:

the adversary approximately maximizes a regularized version of their value function, V ν ρ ( x , y ) := r ( x ) ⊤ λ ( y ; x ) -ν 2 ∥ λ ( y ; x ) ∥ 2 , using Algorithm 2, and then

every agent independently performs a gradient descent step on the value function.

During this process, all agents use only bandit feedback information in order to estimate the gradients of the value function. We remark that the learning dynamics remain uncoupled. The only instance of communication between agents is the fact that the team and the adversary take turns when updating their policies. During their turn, the adversary approximately best-responds.

Of particular interest is the sub-routine of Algorithm 2, VIS-REG-PG. It is effectively a directly parameterized policy gradient method for an objective function that is concave in the state-action visitation measure λ ( y ; x ) ∈ R |S||B| . The objective function is merely the original value function plus a quadratic term, -ν 2 ∥ λ ( y ; x ) ∥ 2 . We remind the reader that due to the existence of this introduced regularizer, the utility of the adversary u = ∇ λ ( y ; x ) F ν ρ ( x , y ) = r ( x ) -ν λ ( y ; x ) . In order to estimate a gradient, the adversary needs to collect a number of trajectories, τ = ( s 0 , b 0 , s 1 , . . . , s H -1 , b H -1 , s H ) , each of length H . Notably, the adversary only uses the empirical state-action visitation measure for the purpose of gradient estimation of the regularized function.

Algorithm 1 Independent Stochastic Policy-Nested-Gradient (ISPNG)

Input: Accuracy ϵ > 0

1: Based on ϵ , set stepsize η x , T x iterations, batch size M , truncation parameter ζ x , and inner-loop accuracy ϵ y > 0 . ▷ see Theorem C.3

3: for t ← 1 , 2 , . . . , T x do

4: y ( t ) ← VIS-REG-PG ( x ( t -1) , ϵ y )

5: ˆ g ( t ) i ← REINFORCE ( x ( t -1) , y ( t ) ; M )

7: end for

9: x ∗ ← x ( t ⋆ )

10: y ∗ ← y ( t ⋆ +1)

Algorithm 2 Visitation-Regularized Policy Gradient Algorithm (VIS-REG-PG)

Input: An MDP, a joint strategy of the team x , and a desired accuracy ϵ > 0 .

1: Based on ϵ , set batch size K , sample traj. length H , stepsize η y , truncation parameter ζ y and regularization coeff. ν . ▷ see Theorem C.3

2: y (0) ( s, b ) ← 1 |B| , ∀ ( s, b ) ∈ S × B .

3: for Epoch t ← 0 , 1 , . . . , T y do

4: Independently sample K trajectories, K ( t ) , of length H under policy y ( t ) .

6: u ← r ( x ) -ν ˆ λ ( t ) .

7: ˆ g ( t ) y ← 1 K ∑ τ ∈K ( t ) ˜ g ( τ | y ( t ) ; u ) .

8: y ( t +1) ← Proj Y ζy ( y ( t ) + η y ˆ g ( t ) y ) .

9: end for

3.3 Analyzing Independent Stochastic Policy-Nested-Gradient

Algorithm 1, or ISPNG, is an instance of a nested-loop algorithm. As we have already informally stated, ISPNG runs gradient descent on the regularized max function Φ ν ( x ) = max λ ∈ Λ( x ) { r ( x ) ⊤ λ -ν 2 ∥ λ ∥ 2 } for some parameter ν . This function has Hölder-continuous gradient and, as such, the convergence proof is underpinned by Theorem 3.1. Formally we state that:

Theorem 3.2 (Grad. Contunuity of Reg-Max Function) . Let function Φ ν ( x ) be the maximum function of the regularized value function of an ATMG, with regularization coefficient ν > 0 . It is the case that, (i) Φ ν is differentiable, (ii) ∇ x Φ ν is (1 / 2 , ℓ 1 / 2 ) -Hölder continuous, i.e,

ISPNG manages to run gradient descent on function Φ ν though the agents can never observe the exact gradient of Φ ν . This is not only due to the randomness of gradient estimators but mainly because they cannot observe the adversary's actions and thus do not know the gradient w.r.t the regularizing term. Fortunately, the regularization coefficient plays a second role in bounding the inexactness error of the gradient estimates. For that reason, parameter ν admits a careful tuning.

Finally, the differentiability of Φ ν and the per-player gradient domination property of the V ρ implies that an ϵ -FOSP x ∗ and the corresponding best-response for the regularized value function, y ∗ , constitute an ϵ -NE, leading to the main Theorem of this subsection:

▷ for all agents i ∈ [ n ]

▷

for all agents

i

∈

[

n

]

▷

for all agents

i

∈

[

n

]

▷ pick the best iterate

▷ ˜ g as in Definition 2.7.

Theorem 3.3 (Main Result; Formally in Theorem C.3) . Given a desired accuracy ϵ > 0 , Algorithm 1 outputs a joint policy ( x ∗ , y ∗ ) for which it holds that,

with a number of iterations and a number of samples that are poly ( 1 ϵ , n, |S| , ∑ i |A i | + |B| , D m , 1 1 -γ , 1 min s ρ ( s ) ) . By D m we denote the mismatch coefficient D m := ∥ ∥ ∥ d x , y ρ ρ ∥ ∥ ∥ (Definition C.2).

4 Minimax in Nonconvex-Hidden-Strongly-Concave Functions

Finally, we would state a more general result compared to that of Theorem 3.3. We consider the general min-max nonconvex-nonconcave optimization problem, min x ∈X max y ∈Y f ( x , y ) , when an additional structural assumption holds, i.e., when f is nonconvex-hidden-strongly-concave. In particular, function f admits a reformulation of the form,

where function H is a nonconvex-strongly-concave function defined on X × U . The sets X and U are closed and convex, while c ( · ; x ) : Y → U is an invertible mapping parametrized by x . Moreover, we will denote U ( x ) := { u | u = c ( y ; x ) , ∀ y ∈ Y} . We further assume that the mapping c and its inverse are Lipschitz-continuous. Specifically,

Assumption 4.1. For the mapping c and its inverse, c -1 , it holds that

If this is the case, the maximizer u ⋆ ( x ) := argmax u ∈U ( x ) H ( x , u ) , is Hölder continuous w.r.t. x as stated by the following Theorem.

Theorem 4.1 (Formally in Theorem D.2) . Let function f ( x , y ) be nonconvex-hidden-stronglyconcave with a modulus of ν > 0 . Let also function H be a L H -Lipschitz continuous and ℓ H -smooth nonconvex-strongly-concave reformulation of f with an invertible mapping c for which Assumption 4.1 holds. Then,

where, L ⋆ = O ( 1 ν ) .

Theorem 4.2 (Convergence to an ϵ -SP; Formally in Theorem D.3) . Let f be a nonconvex-hiddenstrongly-concave function obeying to the same assumptions as f in Theorem 4.1 and ϵ > 0 . Further assume a maximization oracle with O ( νϵ 2 ) -accuracy. There exists an algorithm that computes an ϵ -approximate saddle-point ( x ∗ , y ∗ ) by making T = O ( 1 ν 2 ϵ 3 ) calls to the maximization oracle. Also, the maximization oracle can be implemented by stochastic gradient ascent with iteration complexity T ′ = ˜ O ( 1 ν 3 ϵ 2 ) , and stepsize η y = O ( ν 2 ϵ 2 ) .

5 Conclusion, Future Work, and Limitations

Conclusions We expanded stochastic gradient techniques to be able to compute a stationary point in constrained optimization of nonconvex with weakly-smooth functions. We applied that result to design the first learning algorithm that computes an ϵ -approximate Nash equilibrium in adversarial team Markov games using a finite number of samples and iterations that scale polynomially with 1 /ϵ and the natural parameters of the game.

Future Work We believe that some questions that require further investigation are the following: (i) Is it possible to extend the techniques of [34] to establish convergence guarantees of stochastic gradient descent on nonconvex functions with Hölder-continuous gradient without batch-sampling of the gradient? (ii) Can we design a two-timescale gradient descent-ascent scheme for ATMGs that converges to a Nash equilibrium with best-iterate guarantees? (iii) Can we utilize some variancereduction techniques to achieve a better sample complexity for learning an ϵ -NE in ATMGs?

Limitations The main limitations of our work are (i) the notion of independent learning as presented is weaker than the one presented in [27] i.e. , our algorithm has an 'inner loop', (ii) the fact that we did not present an example for which the function Φ ν fails to be smooth; hence, it is unclear if we can prove the smoothness of this function and achieve tighter analysis. The first item can be addressed in future work by developing a two-timescale algorithmic approach. As for the second item, we remark that even if it is the case that Φ ν is smooth for ATMGs, our provided convergence rates would be straightforwardly improved without any qualitative modification of the algorithm. Also, we would like to highlight that this discussion is related to Remark 2.

References

P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1, 2004.

J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research , 22(98):1-76, 2021.

E. Altman. Constrained Markov decision processes . Routledge, 2021.

Q. Bai, A. S. Bedi, M. Agarwal, A. Koppel, and V. Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 3682-3689, 2022.

A. Bakhtin, N. Brown, E. Dinan, G. Farina, C. Flaherty, D. Fried, A. Goff, J. Gray, H. Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science , 378(6624):1067-1074, 2022.

N. Basilico, A. Celli, G. D. Nittis, and N. Gatti. Team-maxmin equilibrium: Efficiency bounds and algorithms. In S. Singh and S. Markovitch, editors, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017 , pages 356-362. AAAI Press, 2017.

P. Bernhard and A. Rapaport. On a theorem of danskin with an application to a theorem of von neumann-sion. Nonlinear Analysis: Theory, Methods & Applications , 24(8):1163-1181, 1995.

L. Bisi, L. Sabbioni, E. Vittori, M. Papini, and M. Restelli. Risk-averse trust region optimization for reward-volatility reduction. arXiv preprint arXiv:1912.03193 , 2019.

J. Bolte, T. P. Nguyen, J. Peypouquet, and B. Suter. From error bounds to the complexity of first-order descent methods for convex functions, 2016.

C. Borgs, J. T. Chayes, N. Immorlica, A. T. Kalai, V. S. Mirrokni, and C. H. Papadimitriou. The myth of the folk theorem. Games Econ. Behav. , 70(1):34-43, 2010.

M. Bowling, N. Burch, M. Johanson, and O. Tammelin. Heads-up limit hold'em poker is solved. Science , 347(6218):145-149, 2015.

K. Brantley, M. Dudik, T. Lykouris, S. Miryoosefi, M. Simchowitz, A. Slivkins, and W. Sun. Constrained episodic reinforcement learning in concave-convex and knapsack settings. Advances in Neural Information Processing Systems , 33:16315-16326, 2020.

N. Brown, A. Bakhtin, A. Lerer, and Q. Gong. Combining deep reinforcement learning and search for imperfect-information games. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020 , 2020.

N. Brown and T. Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science , 359(6374):418-424, 2018.

N. Brown and T. Sandholm. Superhuman ai for multiplayer poker. Science , 365(6456):885890, 2019.

Y. Cai, H. Luo, C.-Y. Wei, and W. Zheng. Uncoupled and convergent learning in two-player zero-sum markov games. In ICML 2023 Workshop The Many Facets of Preference-Based Learning , 2023.

Y. Cai, A. Oikonomou, and W. Zheng. Finite-time last-iterate convergence for learning in multi-player games. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022 , 2022.

V. Campos, A. Trott, C. Xiong, R. Socher, X. Giró-i Nieto, and J. Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning , pages 1317-1327. PMLR, 2020.

L. Carminati, F. Cacciamani, M. Ciccone, and N. Gatti. A marriage between adversarial team games and 2-player games: Enabling abstractions, no-regret learning, and subgame solving. In International Conference on Machine Learning, ICML 2022 , volume 162 of Proceedings of Machine Learning Research , pages 2638-2657. PMLR, 2022.

A. Celli and N. Gatti. Computational results for extensive-form adversarial team games. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

S. Cen, Y. Wei, and Y. Chi. Fast policy extragradient methods for competitive games with entropy regularization. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , pages 2795227964, 2021.

D. Chen, Q. Zhang, and T. T. Doan. Convergence and price of anarchy guarantees of the softmax policy gradient in markov potential games. In Decision Awareness in Reinforcement Learning Workshop at ICML 2022 , 2022.

X. Chen, X. Deng, and S. Teng. Settling the complexity of computing two-player nash equilibria. J. ACM , 56(3):14:1-14:57, 2009.

Y. Chow, A. Tamar, S. Mannor, and M. Pavone. Risk-sensitive and robust decision-making: a cvar optimization approach. Advances in neural information processing systems , 28, 2015.

F. C. Chu and J. Y. Halpern. On the np-completeness of finding an optimal strategy in games with common payoffs. Int. J. Game Theory , 30(1):99-106, 2001.

C. Daskalakis, D. J. Foster, and N. Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in neural information processing systems , 33:5527-5540, 2020.

C. Daskalakis, P. W. Goldberg, and C. H. Papadimitriou. The complexity of computing a nash equilibrium. SIAM J. Comput. , 39(1):195-259, 2009.

C. Daskalakis, N. Golowich, and K. Zhang. The complexity of markov equilibrium in stochastic games. CoRR , abs/2204.03991, 2022.

C. Daskalakis, A. Ilyas, V. Syrgkanis, and H. Zeng. Training gans with optimism. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

C. Daskalakis and I. Panageas. The limit points of (optimistic) gradient descent in min-max optimization. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 9256-9266, 2018.

C. Daskalakis and I. Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In A. Blum, editor, 10th Innovations in Theoretical Computer Science Conference, ITCS 2019, January 10-12, 2019, San Diego, California, USA , volume 124 of LIPIcs , pages 27:1-27:18. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2019.

C. Daskalakis, S. Skoulakis, and M. Zampetakis. The complexity of constrained min-max optimization. In S. Khuller and V. V. Williams, editors, STOC '21: 53rd Annual ACM SIGACT Symposium on Theory of Computing, Virtual Event, Italy, June 21-25, 2021 , pages 1466-1478. ACM, 2021.

D. Davis and D. Drusvyatskiy. Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization , 29(1):207-239, 2019.

O. Devolder, F. Glineur, and Y. Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming , 146:37-75, 2014.

J. Diakonikolas, C. Daskalakis, and M. I. Jordan. Efficient methods for structured nonconvexnonconcave min-max optimization, 2021.

D. Ding, C.-Y. Wei, K. Zhang, and M. R. Jovanovi´ c. Independent policy gradient for largescale markov potential games: Sharper rates, function approximation, and game-agnostic convergence. arXiv preprint arXiv:2202.04129 , 2022.

D. Drusvyatskiy and A. S. Lewis. Error bounds, quadratic growth, and linear convergence of proximal methods. Mathematics of Operations Research , 43(3):919-948, 2018.

S. Emmons, C. Oesterheld, A. Critch, V. Conitzer, and S. Russell. For learning in symmetric teams, local optima are global nash equilibria. In International Conference on Machine Learning, ICML 2022 , volume 162 of Proceedings of Machine Learning Research , pages 5924-5943. PMLR, 2022.

L. Erez, T. Lancewicki, U. Sherman, T. Koren, and Y. Mansour. Regret minimization and convergence to equilibria in general-sum markov games. In International Conference on Machine Learning , pages 9343-9373. PMLR, 2023.

B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018.

G. Farina, A. Celli, N. Gatti, and T. Sandholm. Ex ante coordination and collusion in zero-sum multi-player extensive-form games. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018 , pages 9661-9671, 2018.

I. Fatkhullin, N. He, and Y. Hu. Stochastic optimization under hidden convexity, 2023.

T. Fiez, L. Ratliff, E. Mazumdar, E. Faulkner, and A. Narang. Global convergence to local minmax equilibrium in classes of nonconvex zero-sum games. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 29049-29063. Curran Associates, Inc., 2021.

L. Flokas, E.-V. Vlatakis-Gkaragkounis, and G. Piliouras. Solving min-max optimization with hidden structure via gradient descent ascent, 2021.

R. Fox, S. M. McAleer, W. Overman, and I. Panageas. Independent natural policy gradient always converges in markov potential games. In International Conference on Artificial Intelligence and Statistics, AISTATS 2022 , volume 151 of Proceedings of Machine Learning Research , pages 4414-4425. PMLR, 2022.

J. Garcıa and F. Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.

U. Ghai, Z. Lu, and E. Hazan. Non-convex online learning via algorithmic equivalence. Advances in Neural Information Processing Systems , 35:22161-22172, 2022.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014.

E. Gorbunov, A. B. Taylor, and G. Gidel. Last-iterate convergence of optimistic gradient method for monotone variational inequalities. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , 2022.

K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07507 , 2016.

T. Groves. Incentives in teams. Econometrica , 41(4):617-631, 1973.

K. A. Hansen, T. D. Hansen, P. B. Miltersen, and T. B. Sørensen. Approximability and parameterized complexity of minmax values. In International Workshop on Internet and Network Economics , pages 684-695. Springer, 2008.

S. Hansen, W. Dabney, A. Barreto, T. Van de Wiele, D. Warde-Farley, and V. Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint arXiv:1906.05030 , 2019.

S. Hart and A. Mas-Colell. Uncoupled dynamics do not lead to nash equilibrium. American Economic Review , 93(5):1830-1836, 2003.

E. Hazan, S. Kakade, K. Singh, and A. Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning , pages 2681-2691. PMLR, 2019.

S. He, Y. Jiang, H. Zhang, J. Shao, and X. Ji. Wasserstein unsupervised reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6884-6892, 2022.

Y.-C. Ho and K.-C. Chu. Team decision theory and information structures in optimal control problems-part i. IEEE Transactions on Automatic Control , 17(1):15-22, 1972.

J. Hu and M. P. Wellman. Multiagent reinforcement learning: Theoretical framework and an algorithm. In Proceedings of the Fifteenth International Conference on Machine Learning , ICML '98, page 242-250, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc.

J. Hu and M. P. Wellman. Nash q-learning for general-sum stochastic games. J. Mach. Learn. Res. , 4:1039-1069, 2003.

C. Jin, Q. Liu, Y . Wang, and T. Yu. V-learning-a simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint arXiv:2110.14555 , 2021.

C. Jin, P. Netrapalli, and M. Jordan. What is local optimality in nonconvex-nonconcave minimax optimization? In International Conference on Machine Learning , pages 4880-4889. PMLR, 2020.

Y. Jin, V. Muthukumar, and A. Sidford. The complexity of infinite-horizon general-sum stochastic games. arXiv preprint arXiv:2204.04186 , 2022.

F. Kalogiannis, I. Anagnostides, I. Panageas, E.-V. Vlatakis-Gkaragkounis, V. Chatziafratis, and S. A. Stavroulakis. Efficiently computing nash equilibria in adversarial team markov games. In The Eleventh International Conference on Learning Representations , 2023.

F. Kalogiannis, E.-V. Vlatakis-Gkaragkounis, and I. Panageas. Teamwork makes von neumann work: Min-max optimization in two-team zero-sum games. ICLR , 2023.

H. Karimi, J. Nutini, and M. Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I 16 , pages 795-811. Springer, 2016.

S. Leonardos, W. Overman, I. Panageas, and G. Piliouras. Global convergence of multi-agent policy gradient in markov potential games. arXiv preprint arXiv:2106.01969 , 2021.

Q. Lin, Z. Lu, and L. Xiao. An accelerated proximal coordinate gradient method. Advances in Neural Information Processing Systems , 27, 2014.

T. Lin, C. Jin, and M. Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In International Conference on Machine Learning , pages 6083-6093. PMLR, 2020.

M. L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994 , pages 157-163. Elsevier, 1994.

H. Liu and P. Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning , pages 6736-6747. PMLR, 2021.

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

C. Maheshwari, M. Wu, D. Pai, and S. Sastry. Independent and decentralized learning in markov potential games, 2022.

J. Marschak. Elements for a theory of teams. Management Science , 1(2):127-137, 1955.

P. Mertikopoulos, B. Lecouat, H. Zenati, C. Foo, V. Chandrasekhar, and G. Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

A. Mladenovic, I. Sakos, G. Gidel, and G. Piliouras. Generalized natural gradient flows in hidden convex-concave games and gans. In International Conference on Learning Representations , 2021.

A. Mokhtari, A. E. Ozdaglar, and S. Pattathil. A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: Proximal point approach. In S. Chiappa and R. Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy] , volume 108 of Proceedings of Machine Learning Research , pages 1497-1507. PMLR, 2020.

M. Moravˇ cík, M. Schmid, N. Burch, V. Lisý, D. Morrill, N. Bard, T. Davis, K. Waugh, M. Johanson, and M. Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science , 356(6337):508-513, 2017.

M. Mutti, R. De Santi, P. De Bartolomeis, and M. Restelli. Challenging common assumptions in convex reinforcement learning. Advances in Neural Information Processing Systems , 35:4489-4502, 2022.

A. Nemirovski. Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1):229-251, 2004.

Y. Nesterov. Universal gradient methods for convex optimization problems. Mathematical Programming , 152(1):381-404, 2015.

M. Nouiehed, M. Sanjabi, T. Huang, J. D. Lee, and M. Razaviyayn. Solving a class of nonconvex min-max games using iterative first order methods. Advances in Neural Information Processing Systems , 32, 2019.

K. A. Oikonomidis, E. Laude, P. Latafat, A. Themelis, and P. Patrinos. Adaptive proximal gradient methods are universal without approximation. arXiv preprint arXiv:2402.06271 , 2024.

F. Orabona. Normalized gradients for all. arXiv preprint arXiv:2308.05621 , 2023.

J. Perolat, B. de Vylder, D. Hennes, E. Tarassov, F. Strub, V. de Boer, P. Muller, J. T. Connor, N. Burch, T. Anthony, S. McAleer, R. Elie, S. H. Cen, Z. Wang, A. Gruslys, A. Malysheva, M. Khan, S. Ozair, F. Timbers, T. Pohlen, T. Eccles, M. Rowland, M. Lanctot, J.-B. Lespiau, B. Piot, S. Omidshafiei, E. Lockhart, L. Sifre, N. Beauguerlange, R. Munos, D. Silver, S. Singh, D. Hassabis, and K. Tuyls. Mastering the game of stratego with model-free multiagent reinforcement learning, 2022.

M. Piccione and A. Rubinstein. On the interpretation of decision problems with imperfect recall. Games and Economic Behavior , 20(1):3-24, 1997.

M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

R. Radner. Team Decision Problems. The Annals of Mathematical Statistics , 33(3):857 - 881, 1962.

I. Sakos, E.-V. Vlatakis-Gkaragkounis, P. Mertikopoulos, and G. Piliouras. Exploiting hidden structures in non-convex games for convergence to nash equilibrium. Advances in Neural Information Processing Systems , 36, 2024.

M. Sayin, K. Zhang, D. Leslie, T. Basar, and A. Ozdaglar. Decentralized q-learning in zerosum markov games. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 18320-18334. Curran Associates, Inc., 2021.

M. O. Sayin, F. Parise, and A. Ozdaglar. Fictitious play in zero-sum stochastic games. arXiv preprint arXiv:2010.04223 , 2020.

J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015 , volume 37 of JMLR Workshop and Conference Proceedings , pages 1889-1897. JMLR.org, 2015.

L. Schulman and U. V. Vazirani. The duality gap for two-team zero-sum games. In 8th Innovations in Theoretical Computer Science Conference (ITCS 2017) . Schloss DagstuhlLeibniz-Zentrum fuer Informatik, 2017.

L. S. Shapley. Stochastic games. Proceedings of the national academy of sciences , 39(10):10951100, 1953.

A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657 , 2019.

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. P. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis. Mastering the game of go without human knowledge. Nat. , 550(7676):354-359, 2017.

B. V. Stengel and D. Koller. Team-maxmin equilibria. Games and Economic Behavior , 21(1-2):309-321, 1997.

A. Tamar, Y. Chow, M. Ghavamzadeh, and S. Mannor. Policy gradient for coherent risk measures. Advances in neural information processing systems , 28, 2015.

A. Tamar and S. Mannor. Variance adjusted actor critic algorithms. arXiv preprint arXiv:1310.3697 , 2013.

O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, Ç. Gülçehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. P. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nat. , 575(7782):350-354, 2019.

J. von Neumann and O. Morgenstern. Theory of Games and Economic Behavior (60th Anniversary Commemorative Edition) . Princeton University Press, 2007.

C.-Y. Wei, C.-W. Lee, M. Zhang, and H. Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive markov games. In M. Belkin and S. Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 4259-4299. PMLR, 15-19 Aug 2021.

R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8:229-256, 1992.

J. Yang, N. Kiyavash, and N. He. Global convergence and variance-reduced optimization for a class of nonconvex-nonconcave minimax problems. arXiv preprint arXiv:2002.09621 , 2020.

M. Yashtini. On the global convergence rate of the gradient descent method for functions with hölder continuous gradients. Optimization letters , 10:1361-1370, 2016.

T. Yu, Y. Tian, J. Zhang, and S. Sra. Provably efficient algorithms for multi-objective competitive rl. In International Conference on Machine Learning , pages 12167-12176. PMLR, 2021.

T. Zahavy, B. O'Donoghue, G. Desjardins, and S. Singh. Reward is enough for convex mdps. Advances in Neural Information Processing Systems , 34:25746-25759, 2021.

T. Zahavy, Y. Schroecker, F. Behbahani, K. Baumli, S. Flennerhag, S. Hou, and S. Singh. Discovering policies with domino: Diversity optimization maintaining near optimality. arXiv preprint arXiv:2205.13521 , 2022.

B. H. Zhang, G. Farina, and T. Sandholm. Team belief DAG form: A concise representation for team-correlated game-theoretic decision making. CoRR , abs/2202.00789, 2022.

B. H. Zhang and T. Sandholm. Team correlated equilibria in zero-sum extensive-form games via tree decompositions. CoRR , abs/2109.05284, 2021.

J. Zhang, A. Koppel, A. S. Bedi, C. Szepesvari, and M. Wang. Variational policy gradient method for reinforcement learning with general utilities. Advances in Neural Information Processing Systems , 33:4572-4583, 2020.

J. Zhang, C. Ni, C. Szepesvari, M. Wang, et al. On the convergence and sample efficiency of variance-reduced policy gradient method. Advances in Neural Information Processing Systems , 34:2228-2240, 2021.

R. Zhang, Z. Ren, and N. Li. Gradient play in stochastic games: stationary points, convergence, and sample complexity. arXiv preprint arXiv:2106.00198 , 2021.

S. Zhang, B. Liu, and S. Whiteson. Mean-variance policy iteration for risk-averse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10905-10913, 2021.

Y. Zhang and B. An. Computing team-maxmin equilibria in zero-sum multiplayer extensiveform games. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020 , pages 2318-2325. AAAI Press, 2020.

Y. Zhang and B. An. Converging to team-maxmin equilibria in zero-sum multiplayer games. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020 , volume 119 of Proceedings of Machine Learning Research , pages 11033-11043. PMLR, 2020.

Appendix

Table of Contents

A Further Related Work

We accommodate this section to mention a brief collection of related literature in the fields of team games, reinforcement learning, and optimization. The literature is vast and we can only manage to mention some representative works.

A.1 Team Games

Research on team games has been a major focus in economic and group decision theory for decades [75, 53, 89, 59]. A key modern reference is [98], which introduced the team-maxmin equilibrium (TME) for normal-form games, where the team's strategy maximizes their minimal expected payoff against any adversary response. Despite their optimality, TMEs are computationally intractable even for 3 -player team games [54, 11]. Recently, practical algorithms have been developed for multiplayer games [117, 116, 7]. Team equilibria are also relevant to two-player zero-sum games with imperfect recall [87].

Due to TME's intractability, TMECor , a relaxed equilibrium concept involving a correlation device , has been studied [42, 21, 7, 117, 111, 110, 20]. TMECor permits correlated strategies but can be impractical in certain scenarios [98]. TMECor is also NP -hard for imperfect-information extensiveform games (EFGs) [26], although fixed-parameter-tractable ( FPT ) algorithms have been developed for specific EFG classes [111, 110].

The computational aspects of standard Nash equilibrium (NE) in adversarial team games are not well-understood, even in normal-form games. Von Neumann's minimax theorem [102] does not apply to team games, rendering traditional methods ineffective. [94] characterized the duality gap between teams, while in [66] it was shown that standard no-regret learning dynamics, such as gradient descent and optimistic Hedge, may fail to converge to mixed NE in binary-action adversarial team games.

A.2 Reinforcement Learning

Multiagent RL Nash equilibrium computation has been central in multiagent RL. Notable algorithms, such as Nash-Q [60, 61], guarantee convergence to Nash equilibria only under strict game conditions. The behavior of independent policy gradient methods [93] remains poorly understood. The impossibility result by the authors of [56] precludes universal convergence to Nash equilibria even in normal-form games, aligning with the computational intractability ( PPAD -completeness) of Nash equilibria in two-player general-sum games [28, 24]. Surprisingly, recent work shows similar hardness in turn-based stochastic games, making (stationary) CCEs intractable [29, 64].

Thus, research has focused on specific game classes, like Markov potential games [68, 37, 114, 23, 74, 46] or two-player zero-sum Markov games [27, 103, 91, 22, 92]. As noted, adversarial Markov team games can unify and extend these settings. Identifying multi-agent settings where Nash equilibria are efficiently computable is a key open problem (see, e.g. , [27]). Recent guarantees for convergence to Nash equilibria have been found in symmetric games, including symmetric team games [39]. Additionally, weaker solution concepts, relaxing either Markovian or stationarity properties, have gained attention [29, 62].

Convex RL Maximizing a value function regularized by a term that is strongly-concave with respect to the state-action visitation measure is an instance of a convex RL problem. In that sense, our work is also related to that strain of literature. Convex RL [108, 112] is a framework that generalizes standard MDP problems by considering the optimization of an objective function that is convex (or concave) in the state (or state-action) visitation measures that the agent's policies induce. The value function of standard RL has an objective function linear to that measure. Common well-known problems that are unified below the lens of convex are (i) 'pure-exploration' RL [57], where the agent maximizes the entropy of the state visitation measure, (ii) imitation learning [1], where an agent minimizes the distance of the state visitation measure their policy induces and the one induced by an expert, (iii) risk-averse RL [47] where the agent optimizes an objective function that is sensitive to the tail behavior of the agent and not merely their expected behavior [100, 99, 25, 9, 115, 80], (iv) constrained RL [4], where an agent optimizes their value function while making sure to satisfy a number of constraints that are dependent on their state-action visitation measure [5, 107, 13, 2], (v) diverse skills discovery, where the goal is to drive learning agents to acquire a diverse set of emergent skills [19, 41, 52, 55, 58, 72, 96, 109].

A.3 Optimization

Min-max Optimization Min-max optimization studies problems of the form min x ∈X max y ∈Y f ( x , y ) . When the objective function f is convex in x and concave in y , the corresponding variational inequality (VI) is monotone, and a wide range of algorithms have been proposed for computing an approximate saddle-point - see, e.g. , [81, 69].

It is also known that standard Gradient Descent/Ascent (GDA) exhibits time-averaged convergence while the actual trajectory of iterates might cycle [30, 31]. Methods like Extra Gradient or techniques such as optimism are used to ensure convergence [30, 31, 32, 76, 78, 18, 51].

For more general objectives, we know how to compute approximate saddle-points when the weak Minty Property is satisfied [36] and for functions where one (or both) side satisfies the PŁcondition [83, 44, 105]. On the negative side, we know that the problem in its full generality (nonconvexnonconcave landscape with coupled linear constraints) is computationally intractable [33].

Hidden-Convex Optimization This nascent field of optimization [43, 48] considers nonconvex objectives that can be reformulated, through a change of variables, into a convex objective. Further, in the context of game theory, the notion of hidden-monotonicity has made its appearance in [45] and the subsequent works [77, 90].

Weakly-Smooth Optimization The majority of references that we encounter for weakly-smooth minimization assume convexity and concern the unconstrained setting. We mention the important references of [35, 82] while also more recent works [106, 85, 84].

B Nonconvex Weakly-Smooth Constrained Optimization

In this section we prove that stochastic projected gradient descent with an stochastic inexact oracle converges to an ϵ -FOSP in functions with Hölder continuous gradient. We complement this section with the proof of folklore lemmas of constrained optimization that show that the 'gradient mapping' (Definition B.1) is an appropriate surrogate of stationarity also for the family of functions we consider.

Definition B.1 (Gradient Mapping) . We define the gradient mapping and stochastic gradient mapping, r η and ˆ r η , to be:

B.1 Auxiliary Lemmas

In general, demonstrating that the gradient mapping is an adequate surrogate of stationarity in differentiable constraint optimization relies on the Lipschitz continuity of the function. We make sure that this is the case when the gradient is only Hölder continuous with p < 1 .

Lemma B.1 (Inexact-Gradient Mapping as a Stationarity Surrogate) . If ∥ ∥ r η ( z ) ∥ ∥ ≤ ϵ for some z ∈ Z , it holds that:

where z + := Proj Z ( z -η g ( z )) .

Proof. In (Inexact Stoch-PGD), ∥ g ( z ) -∇ ϕ ( z ) ∥ ≤ ϑ, ∀ z ∈ Z . Since z + := Proj Z ( z -η g ( z )) , it holds that

Due to the optimality condition, we have

where N Z ( z ) is the normal cone of Z at z , N Z ( z ) := { v | ⟨ v , z ′ -z ⟩ ≤ 0 , ∀ z ′ ∈ Z} . From the latter, we can conclude that

Now, we bound ∥ ∥ ∥ z + -z + 1 η ∇ ϕ ( z ) -1 η ∇ ϕ ( z + ) ∥ ∥ ∥ ,

Therefore we have

The latter display implies the statement of the lemma.

We immediately have the following corollary,

Corollary B.1. For any z ∈ Z , denote z + := Proj Z ( z -η g ( z )) . E [∥ ∥ r η ( z ) ∥ ∥ ] ≤ ϵ implies that

B.2 Stochastic PGD with Inexact Gradients

The folklore proof of gradient descent for nonconvex functions relies on the Lipschitz continuity of the gradient to prove convergence to a first-order stationary point. When the gradient are not Lipschitz continuous but continuous in the weaker notion of Hölder continuity implies the following fact that we will eventually use to prove a 'descent lemma'.

Fact B.1. Let a function ϕ : Z → R with ( p, ℓ p ) -Hölder continuous gradient. Then, it is the case that for all z , z ′ ,

Following [35], we discuss functions with Hölder-continuous gradient (see Definition 2.1) through the framework of inexact oracle. We show that the answer ( ϕ ( z ) , ∇ ϕ ( z )) of an exact oracle for a nonconvex function satisfying Hölder gradient continuity can be translated into some 'inexact' information for a smooth function. Parameters δ, ℓ ′ in Proposition B.1 can be treated as 'inexactness' parameters and will be chosen as appropriate parameters of the exponent p of Hölder continuity.

Proposition B.1. For given δ , ℓ p , and a tuning of ℓ ′ := ℓ 2 1+ p p δ 1 -p 1+ p , it holds that for any x , x ′ ,

Proof. We let χ := ∥ x -x ′ ∥ . By choosing the optimal ℓ ′ we can verify that

Where in the inequality we use the fact that ( 1 -p 2(1+ p ) ) 1 -p 1+ p ≤ 1 for 0 ≤ p ≤ 1 . Setting ℓ ′ = ℓ 2 1+ p p δ 1 -p 1+ p yields the desired inequality.

Now, we can use Proposition B.1 as in place of the 'descent-lemma' to Theorem C.3 to prove convergence to an ϵ -FOSP.

Theorem B.1. Let ϕ : Z → R be a ( p, ℓ p ) -weakly smooth nonconvex function. Further, assume a stochastic inexact gradient oracle ˆ g . I.e., it holds that E [ˆ g ( z ) -g ( z )] = 0 and E [ ∥ ˆ g ( z ) -g ( z ) ∥ 2 ] ≤ σ 2 M for some g : Z → Z ∗ where ∥ g ( z ) -∇ ϕ ( z ) ∥ ≤ ϑ, ∀ z ∈ Z . Implementing T updates of the form (Inexact Stoch-PGD) using ˆ g and a stepsize η = 1 2 ℓ ′ guarantees that:

We postpone the proof to state a corollary that might help the reader gain some intuition on how the iteration complexity scales with p .

Corollary B.2. Let ϕ , ˆ g , the update rule of (Inexact Stoch-PGD) as in Theorem B.1, and stepsize η = ( ϵ 1 -p 2 3 -2 p · ℓ p ) 1 p . For t ∗ drawn uniformly at random from [1 , . . . , T ] , it holds that:

where r ∗ η := r t ∗ η . Furthermore, by setting the parameters as T ≥ 8 1+ p p ℓ 1 p p ( E [ ϕ ( z 0 )] -ϕ ∗ ) ϵ 1+ p p , δ ≤ ( ϵ 8 ) 1+ p p ℓ 1 p p ,

ϑ ≤ ϵ 8 , and M ≥ 9 σ 2 2 ϵ 2 , it is guaranteed that there will exist a t ⋆ ∈ { 0 , . . . , T -1 } such that E [ r η ( z t ⋆ )] ≤ ϵ.

Proof. For the first claim,

Where the last inequality follows from Theorem B.1 and the fact that E [ ∥ ˆ g ( z ) -g ( z ) ∥ 2 ] ≤ σ 2 M . By setting the parameters as in the corollary, we have E [ r η ( z t ⋆ )] ≤ ϵ.

Remark 1. With the same parameters we choose in Corollary B.2, Lemma B.1 guarantees that for any p ∈ (0 , 1] , r η ( z ) is a sufficient surrogate for stationarity. In particular, ∥ ∥ r η ( z ) ∥ ∥ ≤ ϵ implies that

Finally, we state one more auxiliary claim before proceeding to the proof of Theorem C.3. Claim B.1. Consider an iterate of (Inexact Stoch-PGD), z t . Also, define z + = Proj Z ( z t -η g ( z )) , where g is the inexact-gradient oracle. It is the case that,

Proof. The proof follows easily from arguments we have already used,

Proof of Theorem B.1

Proof. Since ∥ g ( z ) -∇ ϕ ( z ) ∥ ≤ ϑ , from the weakly-smooth condition, we have

Where

(1) is because of Proposition B.1;

in (2), we plug-in the definition of ˆ r t η ;

(3) uses the fact that -〈 ˆ g ( z t ) , ˆ r t η 〉 ≤ -1 η ∥ ∥ ˆ r t η ∥ ∥ 2 ;

(4) is due to Young's inequality;

in (5), we plug-in the error bound on the inexact-gradient oracle ∥∇ ϕ ( z ) -g ( z ) ∥ 2 ≤ ϑ .

Continuing we have

Summing for t = 0 , . . . , T -1 ,

This is equivalent to

Taking expectations, we have

By setting η ← 1 2 ℓ ′ , it holds that

This completes the proof.

C Adversarial Team Markov Games

In this section we the formal proofs of our claims regarding ATMGs. Before proceeding, let us provide a roadmap of the current section:

Beginning, in Table 1 we offer a concise summary of our ATMG-related notation.

We proceed to present a number of crucial facts regarding MDPs in Appendix C.1. In particular, facts regarding the state-action visitation measure.

In Appendix C.3, we demonstrate that the regularized value function has a unique maximizer that changes in Hölder-continuous way w.r.t. to team policies x . This leads to the Höldercontinuity of the gradient of the regularized maximum function Φ ν (see Theorem C.2).

Having established the latter, in Section 3.2 we invoke the results on gradient descent for nonconvex functions with Hölder continuous gradient to get our main theorem regarding ϵ -NE learning in ATMGs (Theorem C.3).

The tuning of the parameters of Theorem C.3 is supported by (i) Appendix C.5, where we get precise guarantees for maximizing the regularizing value function w.r.t. the adversary's policy y (Theorem C.4) (ii) Appendix C.6, where we define and analyze the gradient estimators used by the agents of the MG.

C.1 Further Background on Markov Decision Processes

We need additional preliminaries on Markov decision processes (MDPs). Specifically, we will discuss the properties of the (discounted) state and state-action visitation measure . These measures represent the 'discounted' expected amount of time the Markov chain-induced by the players' fixed policies-spends at state s (respectively, at a state action pair ( s, b ) ) starting from initial state s ′ . Each visit is weighted by a discount factor γ h , where h is the visit time. Notably, in [3] it is defined as a probability measure, meaning that for an initial state distribution ρ , the discounted state visitation distribution sums to 1. For convenience, we will use the unnormalized definition from [88, Chapter 6.10], which sums to 1 1 -γ . This is why we refer to it as a measure instead of a distribution .

Definition C.1 (State Visit. Measure) . Given an initial state distribution ρ ∈ ∆( S ) and a stationary joint policy π ∈ Π , we define the state visitation frequency d π s as follows:

Additionally, expanding the definition, we define d π ρ ( s ) = E s ∼ ρ [ d π s ( s )] .

For convenience, the expression d x , y ρ ( s ) is utilized to represent the state visitation measure resulting from the policies ( x , y ) ∈ X × Y .

Fact C.1. Let MDP, M ( S , B , P , r, γ ) . Let a policy π ∈ ∆( B ) |S| . For the corresponding state-action visitation measure λ ∈ R S × B and the state visitation measure d π ρ ∈ R S , it holds that,

A quantity that is important in contemporary RL literature is that of the mismatch coefficient which we formally define here.

Definition C.2 (Distribution Mismatch Coefficient) . Let ρ ∈ ∆( S ) be a full-support distribution over states, and Π be the joint set of policies. We define the distribution mismatch coefficient D as

where d π ρ ρ denotes element-wise division.

The following theorem that relates policies and visitation measures is essential to our analysis. Theorem C.1 ([88, Theorem 6.9.1]) . Consider an adversarial Markov game Γ policy x , and a fixed team

(ii) Any feasible state-action visitation measure λ defines a feasible y ∈ Y ; namely,

Further, for any such y ∈ Y it holds that λ s,b ( y ; x ) = λ ( s, b ) , ∀ ( s, b ) ∈ S × B , where λ ( y ; x ) is the induced discounted state-action measure.

An implication of the latter theorem is the fact that λ ( · ; x ) is a '1-1' mapping between policies and visitation measures. Following, we see that this mapping is also Lipschitz-continuous and smooth (see Lemmas C.1 to C.3).

Lemma C.1. For any initial distribution ρ ∈ ∆( S ) , function V ρ is L -Lipschitz and ℓ -smooth with L := √ ∑ n i =1 |A i | + |B| (1 -γ ) 2 and ℓ := 2 γ ( ∑ n i =1 |A i | + |B| ) (1 -γ ) 3 , in other words

Proof. The proof follows from Lemma 4.4 in [68].

Lemma C.2. Let λ ∈ R |S||B| be the state-action visitation measure for the adversary, then λ is L λ -Lipschitz continuous and ℓ λ -smooth w.r.t to policy ( x , y ) . Specifically, we have

and

Proof. Each λ s,b can be considered as a value function for the given state s and the reward function is r ( a ′ , b ′ ) = ( b = b ′ ) . Then by applying Lemma C.1, we have

From Equation (6) we get

This implies

Similarly, from Equation (7), we have

Thus

Where ∥·∥ F denotes the Frobenius norm of the matrix. Finally, we have

LemmaC.3. Consider λ inv ( λ ( y ; x )) := λ s,b ( y ; x ) ∑ b ′ λ s,b ′ ( y ; x ) , which is a function that maps the adversary's state-action visitation measure λ ( y ; x ) ∈ Λ( x ) to the adversary's policy y ∈ Y . For any fixed team policy x , λ inv is L λ inv -Lipschitz continuous with respect λ where L λ inv = max s ∈S 2 ρ ( s )(1 -γ ) . Specifically, it holds that

Proof. Take the partial derivative of λ inv ( λ ) , we have

This implies the Lipschitz continuity.

The Regularized Value Function.

Lemma C.4. Function V ν ρ ( x , y ) := V ν ρ ( x , y ) -ν 2 ∥ λ ( y ; x ) ∥ 2 is L ν -Lipschitz continuous and ℓ ν -smooth, where L ν := L + νL λ 2(1 -γ ) and ℓ ν := ℓ + νℓ λ 2(1 -γ ) + νL 2 λ 2 .

Proof. For Lipschitz continuity, we have

For smoothness, denote the Jacobian matrix of λ ( y ; x ) w.r.t to ( x , y ) by J λ ( x , y ) , it holds that

Where in (8) we used the fact that ∥ J λ ( x ′ , y ′ ) ∥ ≤ L λ . We conclude that

Finally, we compute the Lipschitz continuity parameter of the reward vector that we already used in our previous claims.

Lemma C.5. Let r ( x ) be the reward function for the adversary when the team is playing policy x . Then it holds that where L r = √ S ( ∑ n i =1 |A i | + B ) .

Proof.

Where (9) follows from Claim D.9. in [65].

C.2 Auxiliary Lemmas

Bounding the stationarity error on the truncated simplex. The ζ -truncated simplex, ∆ m,ζ is defined a the set of all probability vectors with no entry smaller than ζ > 0 . More formally, for a given dimension m and a 0 < ζ ≤ 1 m , the ζ -truncated simplex is defined to be

Lemma C.6. [40, Lemma 15] Let ∆ m,ζ be the ζ -truncated m -simplex. If 0 ≤ ζ ≤ 1 2 m , then for all x ∈ ∆ m , there exists a x ζ ∈ ∆ m,ζ such that ∥ x -x ζ ∥ ≤ 2 ζm .

Proposition C.1 (Stationarity on the trunc. simplex) . Let an L f -Lipschitz continuous differentiable function f : ∆ m → R . Also, let an ϵ -approximate stationary point x ζ when the feasibility set is the truncated simplex ∆ m,ζ such that

Then, x ζ is an ( ϵ +2 ζmL f ) -stationary point when the entire simplex is considered, i.e,

Proof. Consider x ′ ζ ∈ ∆ m,ζ such that ∥ ∥ ∥ x -x ′ ζ ∥ ∥ ∥ ≤ 2 ζm , such point exists due to Lemma C.6, we have

Where in the last inequality we use the fact that for all x ′ ζ ∈ ∆ m,ζ , we have 〈 -∇ f ( x ζ ) , x ′ ζ -x ζ 〉 ≤ ϵ and ∥∇ f ( x ζ ) ∥ ≤ L f .

From stationarity to optimality.

Lemma C.7 (Gradient Domination) . Let a single-agent MDP with action-space A and directlyparametrized policy x ∈ ∆ ζ ( A ) |S| . Then it holds that

Proof. The proof follows easily from Proposition C.1 and the gradient domination property [3, Lemma 4.1].

C.3 Continuity of the maximizers

We begin this section by firstly introducing a proposition which we will leverage in Lemma C.8.

Proposition C.2. Consider the inequality of the form αλ 2 ≤ βλχ + γχ where λ and χ are variables and α, β, γ are coefficients. Under the constraints that α, β, γ, λ, χ ≥ 0 , there is no solution of the form λ ≤ cχ for any finite constant c ≥ 0 .

Proof. Solving the quadratic inequality for λ gives:

We search for a positive constant c such that λ ≤ βχ + √ χ (4 αγ + β 2 χ ) 2 α ≤ cχ, or equivalently,

By observing that when α, β, γ are all positive constants and as χ → 0 , c →∞ , we conclude that no such finite constant c exists.

We first define the maximizer for the regularized function r ( x ) ⊤ λ -ν 2 ∥ λ ∥ 2 . Since the function is strongly-concave w.r.t. λ , the maximizing λ is unique. Specifically we denote

Now we are ready to show an important lemma regarding to λ ⋆ ( x ) .

Lemma C.8 (Continuity of the max. of reg. functions) . For any adversarial Markov game Γ , λ ⋆ ( x ) defined in (10) is (1 / 2 , L ⋆ ) -Hölder continuous, specifically

where L ⋆ := 2( n ) 1 / 4 ν (1 -γ ) 3 / 2 |S| 1 / 2 ( ∑ n k =1 |A k | + |B| ) 3 4 .

Proof. Consider team policies, x 1 , x 2 . It holds true for the unique maximizers λ ⋆ ( x 1 ) , λ ⋆ ( x 2 ) of the adversary's regularized value function that,

where Λ( x ) is the set of feasible state-action visitation measures of the adversary given team's policy x . To bound the distance between the two vectors λ ⋆ ( x 1 ) , λ ⋆ ( x 2 ) , we observe that for any measure λ ∈ Λ( x 1 ) ∪ Λ( x 2 ) , there exist a measure λ 1 ∈ Λ( x 1 ) that shares the same adversary's policy y as

in λ . It then follows from Lemma C.2 that ∥ ∥ λ 1 -λ ∥ ∥ ≤ L λ ∥ x 1 -x 2 ∥ . Therefore we have for all λ ∈ Λ( x 1 ) ,

Where the last inequality follows from (11) and the fact that ∥ r ( x ) -ν λ ⋆ ( x ) ∥ ≤ √ |S||B| ( 1 + ν 1 -γ ) for any x ∈ X . Similarly, it also holds that for all λ ∈ Λ( x 1 ) ,

Plugging in

Adding the two inequalities results in

On the other hand, since r ( x ) ⊤ λ -ν 2 ∥ λ ∥ 2 is ν -strongly concave in λ , we have for all λ 1 ∈ Λ( x 1 ) .

We again use the fact that for every λ ∈ Λ , it holds that there exists λ 1 ∈ Λ( x 1 ) s.t. ∥ λ -λ 1 ∥ ≤ L λ ∥ x 1 -x 2 ∥ . Therefore it holds that for any λ ∈ Λ ,

Rearranging, we have

We bound Ω 1 , Ω 2 , and Ω 3 separately.

For Ω 1 , since ∥ λ -λ 1 ∥ ≤ L λ ∥ x 1 -x 2 ∥ , we have

Where we use the fact that ∥ ∥ λ -λ ⋆ ( x 1 ) ∥ ∥ ≤ ∥ ∥ λ ∥ ∥ + ∥ λ ⋆ ( x 1 ) ∥ and ∥ λ ∥ ≤ ∥ λ ∥ 1 = 1 1 -γ .

For Ω 2 , only the second term is possibly non-negative, it holds that

Resulting in

For Ω 3 :

Finally, by putting the bounds of Ω 1 , Ω 2 , Ω 3 and setting L ′ := 8 ν 1 -γ √ |S||B| L λ , we have for all λ ∈ Λ ,

Concluding, we plug λ ← λ ⋆ ( x 2 ) in (15), resulting

Adding (14) and (16), we have

Rearranging we get

By setting λ = ∥ λ ⋆ ( x 2 ) -λ ⋆ ( x 1 ) ∥ and χ = ∥ x 1 -x 2 ∥ , we consider the inequality νλ 2 ≤ L r λχ + L ′′ χ with coefficients ν, L r , L ′′ ≥ 0 and variables λ and χ . Choosing α ← ν , β ← L r , and γ ← L ′′ , then Proposition C.2 implies that λ ⋆ ( x ) is not Lipschitz-continuous w.r.t the team policy x . Hence, we consider a solution of the form 0 ≤ λ ≤ βχ + √ χ (4 αγ + β 2 χ ) 2 α ≤ cχ p , where 1 2 -p ≥ 0 . We choose p = 1 2 since it yields the best convergence rate from Theorem B.1. Solving the above inequality with p = 1 2 gives that

if χ = 0 , the inequality holds trivially;

if χ > 0 , we have

Since χ = ∥ x 1 -x 2 ∥ ≤ √ n |S| Diam X i = √ 2 n |S| , by plugging in the coefficients, we have

By setting L ⋆ = c , we conclude that

We are now ready to show that Φ ν ( x ) is weakly-smooth.

Theorem C.2 (Hölder Continuous Max Value Func.) . Let function Φ ν ( x ) be the maximum function of the regularized value function, Φ ν ( x ) := max y ∈Y V ν ρ ( x , y ) . It is the case that,

Φ ν is differentiable,

∇ x Φ ν is (1 / 2 , ℓ 1 / 2 ) -Hölder continuous, i.e,

Proof. Since Φ ν ( x ) has a unique maximizer λ ∈ Λ( x ) , by applying Danskin's Theorem [8] and the ' 1 -1 ' correspondence between λ and y (Theorem C.1), we have

Where in the last inequality we used Lemma C.8 and the fact that ∥ x -x ′ ∥ ≤ √ 2 n |S| . Plugging in the coefficients in Lemma C.8, Lemma C.3, and Lemma C.4, it yields that

C.4 Analysis of ISPNG: Proof of Theorem 3.3

In this part we show that Algorithm 1 converges to an ϵ -NE. Essentially, Algorithm 1 implements projected gradient descent on the regularized maximum function Φ ν : X → R with a stochastic ϑ -inexact gradient oracle. Function Φ ν is Hölder-continuous (see Theorem C.2) and as such we can invoke Theorem C.3 to prove convergence to an ϵ -FOSP.

The inexactness of the gradient oracle, ϑ , is the sum of two error sources:

the fact that the adversary can only approximately maximize the regularized value function V ν ρ ( x , y ) -the iteration and sample complexity of maximizing V ν ρ ( x , · ) is provided in Theorem C.4;

the exact estimation of ∇ Φ ν requires estimation of the adversary's policy y -as we assume that the agents do not observe each other's actions this is impossible. Nevertheless, in Lemma C.9 it is proven that the inexactness error is bounded and controlled through the regularizer's coefficient ν .

After quantifying ϑ as the function of the latter two terms, the optimality gap of Algorithm 2 and a term that scales with O ( ν ) , we can tune the rest of the parameters accordingly.

The resulting ϵ -FOSP, thanks to the gradient domination property (Lemma C.7), corresponds to an ϵ -NE.

Bounding the error of the inexact gradient. Following, we prove that the inexactness error of the gradient oracle is bounded by a function of the controllable parameter ν .

Lemma C.9 (Inexact gradient) . Let V ν ρ ( x , y ) := V ρ ( x , y ) -ν 2 ∑ s ∥ d x , y ( s ) y ( s ) ∥ 2 , y ν ( x ) := argmax y { V ν ρ ( x , y ) } , and g ( x ) := ∇ x V ( x , y ν ( x )) . Then, it holds that

Proof. We observe that

Where in the last inequality we use the fact that ∥ λ ∥ ≤ 1 1 -γ and ∇ x λ ( y ; x ) ≤ L λ .

Learning an ϵ -NE. We can now compile the intermediate statements to guarantee that ISPNG computes an ϵ -NE for any desired accuracy ϵ > 0 within a finite number of iterations and samples.

Theorem C.3. Consider an adversarial team Markov game Γ and Algorithm 1, ISPNG , with an outer-loop parameter tuning of:

Also, let the tuning of the inner-loop subroutine Algorithm 2 ( VIS-REG-PG ) be:

It is the case that the output of the algorithm, ( x ∗ , y ∗ ) , will be an ϵ -NE in expectation. Specifically, we have

Proof. Let x ∗ , y ∗ be the final output of the algorithm, from Lemma C.7, we have

Where (17) follows from Lemma C.9. As for the first term of (17), let us define y ⋆ ( x ) = argmax y ∈Y V ν ρ ( x , y ) and x + = Proj X ( x t ∗ -1 -η x ∇ V ρ ( x t ∗ -1 , y ⋆ ( x t ∗ -1 )) ) . Then it holds that,

Where

Equation (18) is due to ∥ ∥ ∇ V ν ρ ( x , y ) ∥ ∥ ≤ L ν ;

Equation (19) follows from the fact that V ν ρ ( x , y ) is ℓ ν -smooth.

By choosing parameters specified above and combining Corollaries B.1 and B.2, Theorem C.4, Lemma C.12, and Claim B.1, we have the desired result. On the other hand, since

Where in (20) we use the fact that ∥ λ ∥ 2 ≤ 1 (1 -γ ) 2 . Combining Theorem C.4, Lemma C.12, and choosing parameters specified above gives the desired result.

C.5 Visitation-Regularized Policy Gradient Analysis

In this section, we consider the direct parameterization for the policy of the adversary. For any policy y ∈ Y , for any state s ∈ S and any action b ∈ B , we have

Where y s,b denotes ( s, b ) th entry of the policy vector y . In this section, we mainly focus on solving the following policy optimization problem:

Where λ ( y ; x ) is the state-action visitation measure under policy y as in Definition 2.4. r ( x ) is the induced pay-off vector for the adversary when the team is playing according to strategy x and ν is the regularization coefficient. Then by policy gradient theorem [113], denote F ν ( λ ( y ; x )) = V ν ρ ( x , y ) we have

Given the direct parameterization, we can show the following lemmas:

LemmaC.10. For any adversarial policy y and state-action pair ( s, b ) , we have ||∇ y log y ( b | s ) || ≤ 1 ζ , ||∇ 2 y log y ( b | s ) || ≤ 1 ζ 2 , and ||∇ y F ν ( λ ( y ; x )) || ≤ 1 (1 -γ ) 2 ζ + ν (1 -γ ) 3 ζ for any fixed x .

Proof. By direct parameterization y ( b | s ) = y s,b , we have

Where e s,b denotes the standard basis for the ( s, b ) th entry. Similarly, we have

Where diag( · ) denotes the standard diagonal matrix. For the policy gradient, we show that

Where (24) is due to (22).

Before we proceed to show the convergence towards global optimality for (21), we first define the notion of Moreau envelope and the proximal point.

Definition C.3 (Moreau Envelope and Proximal Point) . For any y ∈ Y ζ , we use F ν 1 /β ( λ ( y ; x )) to denote the Moreau envelope of function F ν ( λ ( y ; x )) such that

Moreover, we define the proximal point ˆ y 1 /β of Moreau envelope as following:

Now we proceed to show the following lemma:

Lemma C.11. When running Algorithm 2, for any t ≥ 0 , we have

Where C 3 = √ |S||B| 6 H (1 -γ ) 3 ζ .

Proof.

Where (25) follows from Lemma 3.2 in [34]. For the first part of (26), we have

Where (27) follows from Lemma C.4. By setting η y , β such that 2 η y ℓ ν ≤ η y β 2 , and η 2 y ℓ 2 ν 1 -η y β ≤ η y β 2 , we have

For the third part in (26), we have

Where

(29) follows from Lemma C.4;

(30) is due to Lemma C.14.

Combine (26), (28), and (30), we have

We then show the result for convergence to optimality for (21).

Where y ⋆ ζ is the unique maximizer for the optimization problem (21) .

Proof. From Theorem 1 in [43], by setting β = 4 ℓ ν , α ≤ 2 η y ℓ ν , and η y ≤ 2 9 ℓ ν . Then for any z ∈ Y ζ , we have

Where (31) is due to Lemma C.11. By setting s = η y β 2 , we have (1 + s )(1 -η y β ) ≤ 1 -η y β 2 , 1 + s ≤ 2 , and 1 + 1 s ≤ 3 η y β . From Theorem 1 in [43], we get

Define Λ t := E [ F ν ( λ ( y ⋆ ζ ; x )) -F ν 1 /β ( λ ( y ( t ) ; x )) ] , by setting ( 3 L 2 λ inv α 2 2 η y -(1 -α ) αν 2 ) ≤ 0 , we have

Summing over T iterations, and denote E [ ∥ ∥ ∥ ∇ y F ν ( λ ( y ( t ) ; x )) -ˆ g ( t ) y ) ∥ ∥ ∥ 2 ∣ ∣ y ( t ) ] = σ 2 , we get

By setting H = 2 log(1 /νϵ ) 1 -γ , α ≤ min { 2 η y ℓ ν , νη y 2 L 2 λ inv } , and η y = 2 9 ℓ ν , νϵ 10 ℓ ν σ 2 L 2 λ inv , after

iterations, we get Λ T ≤ ϵ. Where σ 2 = C 1 K + C 2 · γ 2 H , C 1 = 57 (1 -γ ) 6 ζ 2 , C 2 = 126 H 2 (1 -γ ) 6 ζ 2 , C 3 = √ |S||B| 6 H (1 -γ ) 3 ζ . Since F ν ( λ ( y ; x )) is smooth with respect to the state-action visitation measure λ ( y ; x ) . We have

Therefore

Define y ⋆ ∈ Y such that y ⋆ = argmax y ∈Y { r ( x ) ⊤ λ ( y ; x ) -ν 2 ∥ λ ( y ; x ) ∥ 2 } . We bound the distance between the the optimal y ⋆ and y ( T ) from Algorithm 2.

Lemma C.12. For any y ∈ Y ζ , if E [ F ν ( λ ( y ⋆ ζ ; x )) -F ν ( λ ( y ; x )) ] ≤ ϵ, then we have

Proof. Since F ν ( λ ( y ; x )) is ν -strongly concave with respect to λ ( y ; x ) , we have

Where (32) holds because λ ( y ⋆ ζ ; x ) is the optimal solution for F ν ( λ ( y ; x )) for any y ∈ Y . Therefore

From the definition of λ ( y ⋆ ζ ; x ) , it holds that for all y ζ ∈ Y ζ , we have 〈 -∇ λ F ν ( λ ( y ⋆ ζ ; x )) , λ ( y ⋆ ζ ; x ) -λ ( y ζ ; x ) 〉 ≤ 0 . Combine with Lemma C.6 and consider y ⋆ ∈ Y , we have

Where

in (33) y ζ ∈ Y ζ is chosen such that ∥ y ⋆ -y ζ ∥ ≤ 2 ζ |B| according to C.6;

(34) holds because ∥∇ λ F ν ( λ ) ∥ ≤ 2 1 -γ and λ ( y ; x ) is L λ -continuous.

Since F ν ( λ ) is ν -strongly concave w.r.t λ , we have

Thus we conclude that

C.6 Regarding the Gradient and Visitation Estimators

In this subsection we will quantify the bias and variance of the gradient and state-action visitation estimators used in Algorithms 1 and 2. In particular, REINFORCE:

the gradient estimator for team agents is implemented by sampling a trajectory with horizon length, H , that is drawn from a geometric distribution for the team, and

while, the state-action visitation estimators that the adversary uses come from sampled trajectories of a fixed horizon length H .

In the former case, the estimator is unbiased while in the second case the bias decays exponentially in H .

C.6.1 REINFORCE for Vanilla Policy Gradient

In the present work, the team agents only need to implement a batch version of REINFORCE [104]. That is, they get estimates ˆ g ( t ) k = 1 M ∑ M j =1 ˜ g ( t ) k , where:

with each H j is a random variable following a geometric distribution with parameter (1 -γ ) .

Although the authors of [27] use ζ -greedy parametrization in order to bound the variance of the estimator, policies drawn from the ζ -truncated simplex imply the same inequality needed to bound the variance. Hence, we invoke the corresponding lemma.

Lemma C.13 ([27, Lemma 2]) . When Equation (REINFORCE) is implemented with H following a geometric distribution with a parameter 1 -γ , and agent k selects policies from the ζ -truncated simplex on each state, it is the case that the gradient estimates satisfy:

C.6.2 Gradient Estimation for Visitation-Regularized Policy Gradient

In this subsection we will describe (i) a state-action visitation estimator with bounded bias and variance and (ii) a gradient estimator of the regularized value function whose bias and variance are also bounded.

Bounding the variance of the a gradient estimator with a deterministic choice of H was significantly less demanding than doing so with a randomized choice. This comes at the cost with a non-zero bias that nevertheless decays exponentially in H . For any policy of the adversary y ∈ Y , we introduce the H -horizon truncated state-action visitation measure

Where λ H,s,b ( y ; x ) denotes the ( s, b ) th entry of λ H ( y ; x ) . For any reward vector r , we have

C.6.3 Controlling the Estimation Bias and Variance

In this subsection we will present a detailed analysis regarding estimators defined in Definition 2.6, Definition 2.7, and the ones used in Algorithm 2. Particularly, in Lemma C.14 we bound the bias of aforementioned estimators. This bias is inevitable for our analysis due to the fact we are

sampling trajectories of a finite length H over an infinite horizon. Proceeding to Lemma C.16 and Lemma C.17, we bound the variance of the state-action distribution measure estimator ˆ λ ( t ) and the gradient estimator ˆ g ( t ) y w.r.t their biased means. Finally, in Lemma C.18, we bound the distance between the gradient estimator ˆ g ( t ) y and the actual gradient ∇ y F ν ( λ ( y ( t ) ; x )) .

Lemma C.14 (Bounded Bias of the Estimators) . For any adversary's policy y ∈ Y . We let τ = ( s 0 , b 0 , s 1 , b 1 , · · · , s H -1 , b H -1 ) be an H -length trajectory sampled from y , then we have E τ ∼ y [ ˜ λ ( τ | y ) ] = λ H ( y ; x ) and E τ ∼ y [˜ g ( τ | y ; r )] = [ ∇ y λ H ( y ; x )] ⊤ r . This implies that in Algorithm 2, E [ ˆ λ ( t ) ] = λ H ( y ( t ) ; x ) and E [ ˆ g ( t ) y ] = [ ∇ y λ H ( y ( t ) ; x ) ] ⊤ r ( t ) . Moreover, we have:

Proof. From the definition, we have

Therefore,

Then it holds that

Similarly, we have

For the first part in the above inequality, we have

For the second part in (36), we have

Combining (36), (38), and (39), we get the result.

Before we proceed to analyze the variance of the estimators, we first show the Lipschitz continuity of the gradient estimator.

Lemma C.15. Let τ = { s 0 , b 0 , s 1 , b 1 , · · · , s H -1 , b H -1 } be an arbitrary H -length trajectory. The gradient estimator satisfies

For any policy y , for any reward vectors r 1 and r 2 ,

For any policies y 1 and y 2 , for any reward vectors r ,

Proof.

Where

(40) follows from (22);

(41) is because of (23).

Now we analyze the variance of the estimators in the algorithm, we start with showing the following lemma.

Lemma C.16 (Bounded Var. of Visit. Estimator) . For ˆ λ ( t ) in Algorithm 2, the variance is bounded. It holds that

Where λ H ( y ( t ) ; x ) is the truncated state-action visitation measure for λ ( y ( t ) ; x ) defined in (35)

Proof. It holds that

Where:

(42) is due to E [ ˆ λ ( t ) ] = λ H ( y ( t ) ; x ) and the fact that trajectories τ ∈ K ( t ) are independently sampled;

(43) is because the variance is bounded by the second moment;

(44) is because ∥ ˜ λ ( τ | y ( t ) ∥ ≤ 1 1 -γ .

Now we analyze the variance of gradient estimator ˆ g ( t ) y by providing the following lemma: Lemma C.17 (Bounded Var. of Grad. Estimator) . For ˆ g ( t ) y in Algorithm 2, we have

Where (45) is due to Cauchy-Schwarz inequality. For the first part in (45), we have

Where:

(46) is due to Cauchy-Schwarz inequality;

(47) follows from Lemma C.15;

(48) follows the same proof as in (37);

(49) is because of Lemma C.16.

For the second part in (45), it holds that,

Where

(50) is due to Lemma C.14 and the fact that trajectories τ are independently sampled;

(51) is because variance is bounded by second moment;

(52) follows from (22).

Finally for the last part in (45), we have

Where

(54) is due to (22) and (37);

(55) is because of Lemma C.16.

Combine (45), (49), (53) and (56), we get

After bounding the variance of ˆ g ( t ) y in Algorithm 2, we can prove the following lemma

Lemma C.18 (Bounded Dist. with Actual Grad.) . Consider y ( t ) and ˆ g ( t ) y in Algorithm 2, it holds that

Proof. Let r ⋆ = ∇ λ F ν ( λ H ( y ( t ) ; x ) ) = r ( x ) -ν λ H ( y ( t ) ; x ) .

Notice that the first part in (57) is bounded in Lemma C.17 and the second part is bounded in (56). For the last part, observe that

Where

Where (58) is follows from Cauchy-Schwarz inequality. For the first part, we have

Where (59) is because of (37). Since

We have

∥ ∥ ∥ ∥ [ ∇ y λ H ( y ( t ) ; x ) ] ⊤ ( ∇ λ F ν ( λ H ( y ( t ) ; x )) -∇ λ F ν ( λ ( y ( t ) ; x )) ) ∥ ∥ ∥ ∥ 2 ≤ ν 2 (1 -γ ) 6 ζ 2 · γ 2 H . (60) For the second part in (58), it holds that

Combine (58), (60), and (61) we get

Now combine Lemma C.17, (56), (57), and (62), we get

D Nonconvex-Hidden-Strongly-Concave Optimization

In this section we generalize our results to the more general setting of any constrained min-max optimization problem of the form min x ∈X max y ∈Y when f is nonconvex-hidden-strongly-concave. In particular:

In Theorem D.2 we prove the differentiability and Hölder continuity of the max function Φ( x ) = max y ∈Y f ( x , y ) by utilizing the Hölder continuity of the maximizers w.r.t. to x (Theorem D.1).

Finally, in Theorem D.3 we prove that Algorithm 3 (SGDMAX) [70, Algorithm 4] with an appropriate tuning converges to an ϵ -SP for nonconvex-hidden-concave functions.

We begin by stating the assumptions we make.

Assumption D.1. Let f be a function defined on X × Y where X and Y are compact convex sets. L -Lipschitz continuous and ℓ -smooth.

Assumption D.2. Let c be a '1-1' mapping between Y and a compact convex set U parameterized by x ∈ X . Further, we assume that c and its inverse c -1 are L c - and L c -1 -Lipschitz continuous.

Assumption D.3. Let H be a nonconvex-strongly-concave reformulation of f (as in Assumption D.1) for a mapping c (as in Assumption D.2). We assume H to be L H -Lipschitz continuous and ℓ H -smooth.

Moving on, we can show that the maximizers u ⋆ ( · ) are Hölder continuous w.r.t. to x .

Theorem D.1 (Continuity of the maximizers) . Let a function nonconvex-nonconcave function f, c, H as in Assumptions D.1 to D.3. We define u ⋆ ( x ) := argmax u ∈U ( x ) H ( x , u ) , then it is the case that

Proof. Consider any x 1 , x 2 ∈ X , since u ⋆ is the maximizer, it holds that

We now consider u that belong to the set U = U ( x 1 ) ∪ U ( x 2 ) . We observe that due to the Lipschitz mapping, for every u ∈ U , there exist a u 1 ∈ U ( x 1 ) such that ∥ u -u 1 ∥ ≤ L c ∥ x 1 -x 2 ∥ . Similar argument holds for u 2 ∈ U ( x 2 ) . Therefore, from previous two inequalities, we have

Where in the above inequalities we used the fact that ∇ H ( x , u ) ≤ L H . We plug in u ← u ⋆ ( x 2 ) and u ← u ⋆ ( x 1 ) accordingly,

Adding the two inequalities results in,

Since H ( x , · ) is ν -strongly concave in u for all x , it holds that

We again consider feasibility set u ∈ U . Since for every u ∈ U , there exists u 1 ∈ U ( x 1 ) s.t. ∥ u -u 1 ∥ ≤ L c ∥ x 1 -x 2 ∥ . We have

We rearrange the latter display into

We bound Ω 1 , Ω 2 , and Ω 3 separately.

For Ω 1 , we have

For Ω 2 , it holds that

For Ω 3 , since the first term is always non-positive, we only need to bound the second term:

Combining Ω 1 , Ω 2 , and Ω 3 , we conclude that

Plugging in u ← u ⋆ ( x 2 ) in (64) and combine it with (63), we get

Where L ′′ = (1 + 2 ℓ H ) L c Diam U +2 L c L H .

Similarly to Lemma C.8, we can set λ = ∥ u ⋆ ( x 2 ) -u ⋆ ( x 1 ) ∥ and χ = ∥ x 1 -x 2 ∥ and consider the inequality νλ 2 ≤ ℓ H λχ + L ′′ χ . We aim to find the solution of the form ℓ H χ + √ χ (4 νL ′′ + ℓ 2 H χ ) 2 ν ≤ c √ χ. By setting L ⋆ = c and solve for c gives

Finally, we show that Φ is differentiable and Hölder-continuous.

Theorem D.2. Let function Φ be Φ( x ) := max u ∈U ( x ) { H ( x , u ) } . Its gradient ∇ Φ is (1 / 2 , ℓ 1 / 2 ) -Hölder continuous,

where ℓ 1 / 2 := ( (1 + L c -1 ) √ Diam X + L c -1 L ⋆ ) ℓ.

Proof.

Where

in (65) we invoke the Lipschitz continuity of function c -1 ( · );

(66) follows from Theorem D.1.

Following, SGDMAX is presented where we assume a stochastic gradient oracle G = ( G x , Gy ) : X × Y × Ξ → R d that is unbiased and has a bounded variance:

Algorithm 3 SGDMAX

Input: Initialization x (0) , stepsize η x , T x iterations, batch size M , oracle accuracy ζ .

1:

for

t

1

,

2

, . . . , T

do

5: end for

Finally, we can state the theorem of convergence to an ϵ -approximate saddle-point.

Theorem D.3. Let a function f as the one in Theorem D.1. For a desired accuracy ϵ > 0 , Algorithm 3, ( SGDMAX ) with a tuning of T x = O ( ℓ 2 1 / 2 ϵ 3 ) , η x , a max -oracle accuracy ζ = O ( νϵ 2 ℓ 2 ) , and a batch size of M = max { 1 , 9 σ 2 2 ϵ 2 } guarantees that there exists a t ∗ ∈ [ T ] , such that,

Further, the max -oracle , of accuracy ζ , can be implemented by T y = ˜ O ( L L 2 c ν + Lσ 2 L 4 c + ν 2 1 ζ ) iterations of stochastic projected gradient ascent with a step size η y = min { 2 9 L , L 2 c νζ 10 Lσ 2 } .

Proof. The proof follows easily from the proof of projected gradient ascent in hidden-stronglyconcave function found [43, Theorem 6] and Theorems B.1 and D.2.

Remark 2. It has been shown that when a function f enjoys a hidden-strongly-concave reformulation, it satisfies global the Proximal-PŁ condition (or equivalently, global KŁ condition) [43, 67]. While the equivalence between global KŁ condition and quadratic growth condition has been proven [10, 38] when f is concave, to the authors' best knowledge, this equivalence still remains unclear when f is nonconcave. This means that we cannot use [83] to prove the smoothness of the maximum function when the feasibility set of the maximizing variable is constrained.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As claimed in the abstraction and introduction, our main contribution is the design and analysis of a learning algorithm for Adversarial Team Markov Games with polynomial iteration and sample complexity (in the parameters of the underlying Markov Game); this is captured by Theorem 3.3.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our work are captured in conclusion and future work for investigation. Moreover, all necessary assumptions have been properly cited.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proof of the main result and auxiliary claims and lemmas can be found in the appendix and are properly referenced. All the assumptions have been properly defined in the Preliminaries section and section 3, 4, e.g., see Assumption 4.1.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https:// nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper is of theoretical nature about learning Nash equilibria in Markov Games. The authors believe that the paper is aligned with NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is of theoretical nature about learning Nash equilibria in Markov Games. The authors do not foresee any societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies policy gradient methods for computing a Nash equilibrium in adversarial team Markov games. The authors employ an occupancy measure-based regularization to deal with this non-convex minimax optimization problem. They develop a policy gradient method that allows all team members to take policy gradient steps independently, while the adversary takes regularized policy gradient steps. The authors prove that this new method finds a near Nash equilibrium in polynomial iteration/sample complexities, under some mild conditions. The authors further show the tractability of computing a near saddle point for a class of structured non-convex minimax optimization problems.\n\nSTRENGTHS:\n- The authors propose a policy gradient method to find a near Nash equilibrium of adversarial team Markov games, with polynomial iteration/sample complexities. This policy gradient method appear to be new for this problem. \n\n- A new regularization technique is introduced for the adversarial player, which avoids using linear programming in the existing method.\n\nWEAKNESSES:\n- The paper has several writing issues: (i) Introduction with two paragraphs is discouraged for NeurIPS; (ii) Technical overview is verbose, and not all the content is original to this paper; (iii) The preliminary section is cumbersome and contains confusing notation for occupancy measures; (iv) The connection between stochastic gradient and team problems in Section 3 reads ambiguous; (v) The results in Section 4 are not directly related to adversarial team games.\n\n- The motivation for using the policy gradient method to solve adversarial team Markov games is not explained from the perspective of application. It is hard to evaluate the broad impact of this work.\n\n- The motivation for using regularization based on occupancy measures is not well presented. The authors implement regularization directly in Algorithm 1 without explaining the reasons behind it.\n\n- It is not clear why bandit feedback is important in adversarial team Markov games since the proposed policy gradient method assumes simulated game-play, which is not a typical online learning algorithm. \n\n- The proposed policy gradient method assumes policy coordination between team and adversary, which often is not the case in practice. \n\n- Except for regularization for the adversarial player, other techniques are quite standard for policy gradient methods. The novelty of the proposed method is limited or the effectiveness of regularization should be emphasized. \n\n- The polynomial iteration/sample complexities of the proposed policy gradient method are highly sub-optimal in terms of rates and problem dependence.\n\n- The proposed policy gradient method is limited to problems with small state/action spaces, leading dimension scalability issue. \n\n- Conditions for Theorem 3.3 to hold are not explained, hiding limitations of the proposed policy gradient method. Also, the proof of Theorem 3.3 in Appendix should be presented in a more readable way. \n\n- There are no experimental justifications of the proposed policy gradient method.\n\nQUESTIONS:\nHere are other questions for improvement.\n\n- What rewards do team take? It is not clear from Section 2.2.\n\n- What is reformulation in line 207? a confusing inter product notation. \n\n- unclear notation: $\\Delta(\\mathcal{A})$, $\\boldsymbol{r}(\\boldsymbol{x})$, $\\hat{\\boldsymbol{g}}_i$, $\\mathcal{K}^{(t)}$, $\\boldsymbol{x}^{(T)}$, etc.\n\n- Why do you need $\\zeta$-greedy policy? How does policy gradient projection work in Algorithms 1-2? \n\n- wrong index: $k\\in[n]$ lines 3-4 of Algorithm 1.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides multi-agent RL policy gradient method with polynomial guarantees (iteration and sample complexity) for the adversarial team Markov games problem setting.\n\nSTRENGTHS:\nThis paper addresses the main open question from https://openreview.net/forum?id=mjzm6btqgV, that is, developing a policy gradient learning algorithm for the adversarial team Markov games problem. The paper is well-written.\n\nWEAKNESSES:\nAfter the first pass of this work, I did not have many weaknesses for this work as it *made good progress* from the recent ICLR 2023 work https://openreview.net/forum?id=mjzm6btqgV.\n\nBut after the following point:\n\n- Theorem 3.3, the main result, the polynomial dependence of both sample complexity and iterations on the **all** parameters is huge! Even with $\\gamma=0.9$, sample complexity scales as $10^{93}$. The number of atoms in the current universe scales as approx $10^{84}$. *This is disregarding other factors such as states, actions, etc into sample complexity!* Whereas, most practical MARL works like Diplomacy [6], Starcraft (https://deepmind.google/discover/blog/alphastar-mastering-the-real-time-strategy-game-starcraft-ii/), etc, beat human players at a reasonable training time (approx hours/days?). Moreover, ICLR 2023 work https://openreview.net/forum?id=mjzm6btqgV sample complexity scales as $10^{32}$ (all parameters included in this)\n\nI just can't get over what side I should choose: happy for such learning algorithm for ATMG problem or unsatisfied with the tools used with such huge *polynomial (but almost exponential with 10s of states, for example)* suboptimal guarantees.\n\nTo err on the side of this work and ML community progress, I am open to hearing any defense for this. Towards this, I am looking forward to your reply to the below two points. \n\n- Maybe one way is to show hardness results on some finite state-action ATMGs?\n- Please highlight some technical innovations in this work compared to the zero-sum MG sample complexity analysis. Are there points of improvement to be made?\n\nFuture work statement from line 329:\n\n> variance-reduction techniques to achieve a better sample complexity\n\nIs there any conjecture as to how much VR methods save in terms of sample complexity?\n\nAs I am in a dilemma, my current score reflects that. I'll update it after the authors-reviewers discussion period. Thanks!\n\nQUESTIONS:\nna\n\nLIMITATIONS:\nna", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis papers addresses learning equilibria in adversarial team Markov games, where a team of agents sharing a common reward function competes against a single adversary. Previously, [65] addressed this problem for the model-based case with no sample complexity guarantees. On the other hand, this paper presents a learning algorithm for model-free cases with bandit feedback and provide polynomial sample complexity guarantees.\n\nSTRENGTHS:\n- The paper is well-organized. We can see the effort to make the complex ideas accessible for the reader.\n- There is a comprehensive literature review citing more than a hundred papers.\n\nWEAKNESSES:\n- Algorithm descriptions, i.e., Algorithms 1 and 2, are confusing. For example, Algorithm 1 uses VIS-REG-PG as a function. However, VIS-REG-PG has not been presented as a function getting x as an input in Algorithm 2. The function REINFORCE has not been provided explicitly. Although REINFORCE is well-known, its explicit description in the paper could improve the paper's completeness.\n- Though the learning dynamics are claimed to be uncoupled, the agents including the adversary coordinate in collecting samples by playing according to some fixed strategy for some predetermined batch sizes. This contradicts with the adversarial nature of the adversary. Collecting samples is essential for model-free and bandit-feedback scenarios, which is a major contribution of the paper compared to [65].\n- The bounds on the number of gradient updates and the total sample complexity in the main result, Theorem 3.3, depend on (possibly) very large constant terms since they are proportional to $\\frac{1}{(1-\\gamma)^{57}\\epsilon^{10}}$ and $\\frac{1}{(1-\\gamma)^{93}\\epsilon^{16}}$, respectively. This may not be feasible for many practical applications, e.g., when the discount factor is close to $1$ and we want small approximation error $\\epsilon\\approx 0$. This is a limitation because another major contribution of this paper compared to [65] is the sample complexity bounds.\n- No numerical example is provided. Since the bounds appear large, numerical examples can give a better idea whether the algorithm presented is feasible for practical applications or not.\n\nQUESTIONS:\n- What are the practical applications of adversarial team Markov games where learning equilibrium via the algorithm presented can be of interest? For example, step 3 in Algorithm 1 uses REINFORCE with the batch size of M where all agents, including the adversary, keep playing according to fix strategy $(x^{t-1},y^t)$ so that the team members can collect samples and estimate $\\hat{g}_i^t$. This contradicts with the adversarial nature of the adversary. Therefore, are we considering a simulated environment where we coordinate the team members and the adversary to learn some equilibrium of the underlying game? \n- Adversarial team Markov games reduce to team Markov games if the adversary has a single action, and two-agent zero-sum Markov games if the team is a singleton. Can we say something similar for the algorithm presented? In other words, if the adversary is a non-strategic player with a single action, does the algorithm presented reduce to a known learning algorithm for team Markov games (or Markov potential games)? If the team has a single player, does the algorithm reduce to a known learning algorithm for two-agent zero-sum Markov games? If the answer is no, what are the differences? \n- Can the authors rewrite the algorithm descriptions? For example, what is the function VIS-REG-PG? Is it the steps from 2 to 6 or the entire for loop in Algorithm 2? This is confusing since Algorithm 1 and 2 have different epoch numbers $T_x$ and $T_y$.\n- Can the authors provide numerical examples? A comparison with the algorithm of [65] would also be very helpful to observe the impact of model-free and bandit-feedback cases on the convergence.\n\nLIMITATIONS:\nThe limitations of the paper has not been discussed separately in an explicit way. The paper (1) requires the coordination of agents including the adversary in playing the same strategy for sampling trajectories in the estimation of some parameters important for the team members though this contradicts with the adversarial nature of the adversary, and (2) provides possibly very large bounds on the number of gradient updates and the sample complexities. (1) is important for addressing the model-free and bandit-feedback cases. (2) is essentially like asymptotic guarantees from a practical perspective. Those are the major limitations since addressing model-free and bandit-feedback cases, and providing sample complexity bounds are the main contributions of this paper compared to [65].", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper investigates the identification of NE in ATMGs. The authors explore the underlying landscape, leveraging optimization theory to effectively solve this problem.

STRENGTHS:
Theorem 3.1 stands out with significant contribution. Writing is very clear.

WEAKNESSES:
The other theorems, in my view, merely adapt the MARL problem to a specific instance of an optimization problem (with an observation of hidden structure). The paper does not leverage the special structure of Markov Games (or Team Markov Games). Nonetheless, I appreciate the paper's clarity in writing and its theoretical contributions.

I have read the paper thoroughly, and while it presents a significant contribution, I believe it might not be the best fit for NeurIPS. The authors address a very classical optimization problem and apply their findings to an Adversarial Team Markov Game. Although Theorem 3.1 is particularly impressive, the subsequent theorems follow from it primarily due to the compactness and finite state and action space of MARL. This paper, lacking experimental results, seems more suitable for an optimization journal. I know several MG theory papers are already accepted to NeurIPS, even without any experiment. To be clear, I do not want to discourage authors about the fit for the venue.

QUESTIONS:
check weaknesses

LIMITATIONS:
check weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of independent policy gradient learning in adversarial team markov games (ATMGs). In such games, a team of agents with identical reward functions aim to compete against a single adversary whose reward function is the negation of the team's. The paper consider the setting where the MG has unknown transitions and reward functions, and the players can only obtain information about it by sampling trajectories and observing the resulting states and rewards. The main result (Theorem 3.3) gives an efficient algorithm which produces a policy which is an epsilon-Nash equilibrium of the ATMG, in which the players act independently, and which requires a number of samples which is polynomial in the number of states, actions, horizon, 1/epsilon, and distribution mismatch coefficient.\n\nSTRENGTHS:\n- Previously it was only known how to compute a NE in an ATMG in the presence of known transitions and rewards ([65] in the paper). Although the algorithm in [65] is a gradient descent-style algorithm, the present paper makes a convincing argument that the existing approach is insufficient (essentially because one cannot solve a certain linear program exactly).\n\n- As such, the paper introduces several new ideas. One key idea is the regularization of the adversary's value function by the squared l_2 norm of their *state visitation distribution* (denoted by lambda in the paper). This ensures that the adversary's value function is strongly convex as a function of their visitation distribution, which facilities Danskin theorem-type arguments.\n\n- As a result of the above idea, the \"max-function\" of the team's value function (i.e., the regularized value the team receives if the adversary best-responds) turns out to only be Holder-continuous (not Lipschitz continuous, as is typically the case). The paper then proves a new convergence result for gradient descent on nonconvex Holder-continuous functions (which seems to use similar ideas to previous work [35] in the convex case, but is nice to have written out).\n\nWEAKNESSES:\n(A) One weakness is that the algorithm has an \"inner loop\", in which the adversary must run several steps for each step of the team. This is in contrast to, e.g., [27], which has a stronger notion of independent learning (i.e., without an inner loop).\n\n(B) It would be nice to have some more description about why Algorithm 2 works: in particular, it's not immediately clear how to interpret the updates to r(x) (ie subtracting nu * \\hat lambda) in line 4. Since r(x) does not depend on y, this doesn't seem exactly like what GD is doing?\n\n(C) To support the paper's claim that Moreau envelope techniques do not apply, it would be nice to have some result giving an example where Phi^nu(x) is *not* weakly convex (as typically in such situations one would expect Moreau envelope techniques to work).\n\n(D) The paper claims prominently (e.g. in the abstract) it solves \"the main open question from [65]\". I could not find any reference in [65] to the open question which the paper claims to solve. The paper should be updated to either say where specifically that open question is stated, or else rephrase the claim (which is misleading if [65] does not state this explicitly as an open problem).\n\nQUESTIONS:\nSee \"weaknesses\" above.\n\nLIMITATIONS:\nThe paper should be updated to address (A) + (D) in the weaknesses section above (both of which can be fixed by changing a couple of sentences).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-09-08"}