Title: Posterior Sampling for Competitive RL: Function Approximation and Partial Observation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Markov game

Partial observation

Function approximation

Posterior sampling

Reinforcement Learning
Abstract: This paper investigates posterior sampling algorithms for competitive reinforcement learning (RL) in the context of general function approximations. Focusing on zero-sum Markov games (MGs) under two critical settings, namely self-play and adversarial learning, we first propose the self-play and adversarial generalized eluder coefficient (GEC) as complexity measures for function approximation, capturing the exploration-exploitation trade-off in MGs. Based on self-play GEC, we propose a model-based self-play posterior sampling method to control both players to learn Nash equilibrium, which can successfully handle the partial observability of states. Furthermore, we identify a set of partially observable MG models fitting MG learning with the adversarial policies of the opponent. Incorporating the adversarial GEC, we propose a model-based posterior sampling method for learning adversarial MG with potential partial observability. We further provide low regret bounds for proposed algorithms that can scale sublinearly with the proposed GEC and the number of episodes $T$. To the best of our knowledge, we for the first time develop generic model-based posterior sampling algorithms for competitive RL that can be applied to a majority of tractable zero-sum MG classes in both fully observable and partially observable MGs with self-play and adversarial learning.

Posterior Sampling for Competitive RL: Function Approximation and Partial Observation

Abstract

This paper investigates posterior sampling algorithms for competitive reinforcement learning (RL) in the context of general function approximations. Focusing on zero-sum Markov games (MGs) under two critical settings, namely self-play and adversarial learning, we first propose the self-play and adversarial generalized eluder coefficient (GEC) as complexity measures for function approximation, capturing the exploration-exploitation trade-off in MGs. Based on self-play GEC, we propose a model-based self-play posterior sampling method to control both players to learn Nash equilibrium, which can successfully handle the partial observability of states. Furthermore, we identify a set of partially observable MG models fitting MGlearning with the adversarial policies of the opponent. Incorporating the adversarial GEC, we propose a model-based posterior sampling method for learning adversarial MG with potential partial observability. We further provide low regret bounds for proposed algorithms that can scale sublinearly with the proposed GEC and the number of episodes T . To the best of our knowledge, we for the first time develop generic model-based posterior sampling algorithms for competitive RL that can be applied to a majority of tractable zero-sum MG classes in both fully observable and partially observable MGs with self-play and adversarial learning.

1 Introduction

Multi-agent reinforcement learning (MARL) tackles sequential decision-making problems where multiple players simultaneously interact with the shared environment, affecting each other's behavior in a coupled manner. Under a competitive reinforcement learning (RL) setting, the goal of each player is to maximize ( resp. minimize) her own cumulative gains ( resp. losses) in the presence of other agents. Recent years have been tremendous practical successes of MARL in a variety of application domains, such as autonomous driving [56], Go [58], StarCraft [65], Dota2 [8] and Poker [10]. These successes are attributed to advanced MARL algorithms that can coordinate multiple players by exploiting potentially partial observations of the latent states and employ powerful function approximators (neural networks in particular), which empower us to tackle practical problems with large state spaces.

Apart from the empirical success, there is a growing body of literature on establishing theoretical guarantees for Markov games (MGs) [57] - a standard framework for describing the dynamics of competitive RL. In particular, [71, 16, 33, 28, 77] extend the works in single-agent reinforcement

learning (RL) with function approximation [29, 60, 67, 34, 5, 11, 19, 31, 21] by developing sampleefficient algorithms that are capable to solve two-player zero-sum MGs with function approximation. In addition, as opposed to the aforementioned literature on MGs assuming the state of players is fully observable, the recent work [41] analyze Markov games under partial observability [41], i.e., the complete information about underlying states is lacking. However, most of the existing works are built upon the principle of 'optimism in the face of uncertainty' (OFU) [38] for exploration. Furthermore, from a practical perspective, achieving optimism often requires explicit construction of bonus functions, which are often designed in a model-specific fashion and computationally challenging to implement.

Another promising strand of exploration techniques is based on posterior sampling, which is shown by previous works on bandits [12] and RL [48] to perform better than OFU-based algorithms. Meanwhile, posterior sampling methods, unlike OFU-based algorithms [31, 21] that need to solve complex optimization problems to achieve optimism, can be efficiently implemented by ensemble approximations [48, 44, 17, 46] and stochastic gradient Langevin dynamics (SGLD) [70]. Despite the superiority of posterior sampling, its theoretical understanding in MARL remains limited. The only exception is [73], which proposes a model-free posterior sampling algorithm for zero-sum MGs with general function approximation. However, [73] cannot capture some common tractable competitive RL models with a model-based nature, such as linear mixture MGs [16] and low witness rank MGs [28]. Moreover, their result is restricted to the fully observable MGs without handling the partial observability of the players' states. Therefore, we raise the following question:

Can we design provably sample-efficient posterior sampling algorithms for competitive RL with even partial observations under general function approximation?

Concretely, the above question poses three major challenges. First, despite the success of the OFU principle in partially observable Markov games (POMGs), it remains elusive how to incorporate the partial observations into the posterior sampling framework under a MARL setting with provably efficient exploration. Second, it is also unclear whether there is a generic function approximation condition that can cover more known classes in both full and partial observable MARL and is meanwhile compatible with the posterior sampling framework. Third, with the partial observation and function approximation, it is challenging to explore how we can solve MGs under the setups of selfplay, where all players can be coordinated together, and adversarial learning, where the opponents' policies are adversarial and uncontrollable by the learner. Our work takes an initial step towards tackling such challenges by concentrating on the typical competitive RL scenario, the two-player zero-sum MG, and proposing statistically efficient posterior sampling algorithms under function approximation that can solve both self-play and adversarial MGs with full and partial observations.

Contributions. Our contributions are four-fold: (1) We first propose the two generalized eluder coefficient (GEC) as the complexity measure for the competitive RL with function approximation, namely self-play GEC and adversarial GEC, that captures the exploration-exploitation tradeoff in many existing MGs, including linear MGs, linear mixture MGs, weakly revealing POMGs, decodable POMGs. The proposed measures also generalize the recently developed GEC condition [83] from single-agent RL to MARL, suitably adjusting the exploration policy particularly for the adversarial setting. (2) Incorporating the proposed self-play GEC for general function approximation, we propose a model-based posterior sampling algorithm with self-play to learn the Nash equilibrium (NE), which successfully handles the partial observability of states along with a full observable setup by carefully designed likelihood functions. (3) We identify POMG models aligned with the form of the adversarial GEC, which fit MG learning with adversarially-varying policies of the opponent. We further propose a model-based posterior sampling algorithm for adversarial learning with general function approximation. (4) Weprove regret bounds for our proposed algorithms that scale sublinearly with the number of episodes T , the corresponding GEC d GEC , and a quantity measuring the coverage of the optimal model by the initial model sampling distribution. To the best of our knowledge, we present the first model-based posterior sampling approaches to sample-efficiently learn MGs with function approximation, handling partial observability in both self-play and adversarial settings.

Related Works. There is a large body of literature studying MGs, especially zero-sum MGs. In the self-play setting, many papers have focused on solving approximate NE in tabular zero-sum MGs [6, 7, 71, 51, 43], zero-sum MGs with linear function approximation [71, 16], zero-sum MGs with low-rank structures [50, 79, 47], and zero-sum MGs with general function approximation [33, 28, 73]. On the other hand, there are also several recent papers focusing on the adversarial setting

[71, 61, 33, 28] aim to learn the Nash value under the setting of unrevealed opponent's policies, where the adversarial policies of the opponent are unobservable. In addition, another line of adversarial MGs concentrates on a revealed policy setting, where the opponent's full policy can be observed, leading to efficiently learning a sublinear regret comparing against the best policy in hindsight. Particularly, [42] and [77] develop efficient algorithms in tabular and function approximation settings, respectively. Our approach focuses on the unrevealed policy setting, which is considered to be a more practical setup. There are also works studying MGs from various aspects [59, 32, 45, 84, 35, 20, 52, 9, 82, 18, 72, 74], such as multi-player general-sum MGs, reward-free MGs, MGs with delayed feedback, and offline MGs, which are beyond the scope of our work. Most of the aforementioned works follow the OFU principle and differ from our posterior sampling methods. The recent work [73] proposes a model-free posterior sampling algorithm for two-player zero-sum MGs but is limited to the self-play setting with fully observable states. Moreover, their work requires a strong Bellman completeness assumption that is restrictive compared to only requiring realizability in our work, mainly due to the monotonicity, meaning that adding a new function to the function class may violate it. Many modelbased models like linear mixture MGs [16] and low witness rank MGs [28] are not Bellman-complete, so they cannot be captured by [73]. Without the completeness assumption, our model-based posterior sampling approaches can solve a rich class of tractable MGs, including linear mixture MGs, low witness rank MGs [28], and even POMGs, tackling both self-play and adversarial learning settings.

Our work is related to a line of research on posterior sampling methods in RL. For single-agent RL, most existing works such as [54] analyze the Bayesian regret bound. There are also some works [4, 53, 76] focusing on the frequentist (worst-case) regret bound. Our work is more closely related to the recently developed feel-good Thompson sampling technique proposed by [81] for the frequentist regret bound, and its extension to single-agent RL [19, 2, 3] and two-player zero-sum MGs [73].

Our work is also closely related to the line of research on function approximation in RL. Such a line of works proposes algorithms for efficient policy learning under diverse function approximation classes, spanning from linear Markov decision processes (MDPs) [34], linear mixture MDPs [5, 85] to nonlinear and general function classes, including, for instance, generalized linear MDPs [69], kernel and neural function classes [75], bounded eluder dimension [49, 68], Bellman rank [29], witness rank [60], bellman eluder dimension [31], bilinear [21], decision-estimation coefficient [24, 14], decoupling coefficient [19], admissible Bellman characterization [15], and GEC [83] classes.

The research on partial observability in RL [27] is closely related to our work. The works [36, 30] show that learning history-dependent policies generally can cause an exponential sample complexity. Thus, many recent works focus on analyzing tractable subclasses of partially observable Markov decision processes (POMDPs), which includes weakly revealing POMDPs [30, 39], observable POMDPs [26, 25], decodable POMDPs [22, 23], low-rank POMDPs[66], regular PSR [78], PObilinear class [63], latent MDP with sufficient tests [37], B-stable PSR [13], well-conditioned PSR [40], POMDPs with deterministic transition kernels [30, 62], and GEC [83]. Nevertheless, in contrast to our work which focuses on the two-player competitive setting with partial observation, these papers merely consider the single-agent setting. The recent research [41] further generalizes weakly revealing POMDPs to its multi-agent counterpart, weakly revealing POMGs, in a general-sum multiplayer setting based on the OFU principle. But when specialized to the two-player case, our work proposes a general function class that can subsume the class of weakly revealing POMGs as a special case. It would be intriguing to generalize our framework to the general-sum settings in the future.

Notations. We denote by KL( P || Q ) = E x ∼ P [log(d P ( x ) / d Q ( x ))] the KL divergence and D 2 He ( P, Q ) = 1 / 2 · E x ∼ P ( √ d Q ( x ) / d P ( x ) -1) 2 the Hellinger distance. We denote by ∆ X the set of all distributions over X and Unif( X ) the uniform distribution over X . We let x ∧ y be min { x, y } .

2 Problem Setup

We introduce the basic concept of the two-player zero-sum Markov game (MG), function approximation, and the new complexity conditions for function approximation. Concretely, we study two typical classes of MGs, i.e., fully observable MGs and partially observable MGs, as defined below.

Fully Observable Markov Game. We consider an episodic two-player zero-sum fully observable Markov game (FOMG 2 ) specified by a tuple ( S , A , B , P , r, H ) , where S is the state space, A and

B are the action spaces of Players 1 and 2 respectively, H is the length of the episode. We denote by P := { P h } H h =1 the transition kernel with P h ( s ′ | s, a, b ) specifying the probability (density) of transitioning from state s to state s ′ given Players 1 and 2's actions a ∈ A and b ∈ B at step h . We denote the reward function as r = { r h } H h =1 with r h : S × A × B ↦→ [0 , 1] being the reward received by players at step h . We define π = { π h } H h =1 and ν = { ν h } H h =1 as Markovian policies for Players 1 and 2, i.e., π h ( a | s ) and ν h ( b | s ) are the probability of taking action a and b conditioned on the current state s at step h . Without loss of generality, we assume the initial state s 1 is fixed for each episode. We consider a realistic setting where the transition kernel P is unknown and thereby needs to be approximated using the collected data.

Partially Observable Markov Game. This paper further studies an episodic zero-sum partially observable Markov game (POMG), which is distinct from the FOMG setup in that the state s is not directly observable. In particular, a POMG is represented by a tuple ( S , A , B , O , P , O , µ, r, H ) , where S , A , B , H , and P are similarly the state and action spaces, the episode length, and the transition kernel. Here µ 1 ( · ) denotes the initial state distribution. We denote by O := { O h } H h =1 the emission kernel so that O h ( o | s ) is the probability of having a partial observation o ∈ O at state s with O being the observation space. Since we only have an observation o of a state, the reward function is defined as r := { r h } H h =1 with r h ( o, a, b ) ∈ [0 , 1] depending on actions a, b and the observation o , and the policies for players are defined as π = { π h } H h =1 and ν = { ν h } H h =1 , where π h ( a h | τ h -1 , o h ) and ν h ( b h | τ h -1 , o h ) is viewed as the probability of taking actions a h and b h depending on all histories ( τ h -1 , o h ) . Here we let τ h := ( o 1 , a 1 , b 1 . . . , o h , a h , b h ) . Then, in contrast to FOMGs, the policies in POMGs are history-dependent , defined on all prior observations and actions rather than the current state s . We define P π,ν h ( τ h ) := ∫ S h µ 1 ( s 1 ) ∏ h -1 h ′ =1 [ O h ′ ( o h ′ | s h ′ ) π h ′ ( b h ′ | τ h ′ -1 , o h ′ ) ν h ′ ( b h ′ | τ h ′ -1 , o h ′ ) P h ′ ( s h ′ +1 | s h ′ , a h ′ , b h ′ )] O h ( o h | s h )d s 1: h , which is the joint distribution of τ h under the policy pair ( π, ν ) . Removing policies in P π,ν h , we define the function P h ( τ h ) := ∫ S h µ 1 ( s 1 ) ∏ h -1 h ′ =1 [ O h ′ ( o h ′ | s h ′ ) P h ′ ( s h ′ +1 | s h ′ , a h ′ , b h ′ )] O h ( o h | s h )d s 1: h . We assume that the parameters θ := ( µ 1 , P , O ) are unknown and thus P h is unknown as well, which should be approximated in algorithms via online interactions.

Online Interaction with the Environment. In POMGs, at step h of episode t of the interaction, players take actions a t h ∼ π t h ( ·| τ t h -1 , o t h ) and b t h ∼ ν t h ( ·| τ t h -1 , b t h -1 , o t h ) depending on their action and observation histories, receiving a reward r h ( o t h , a t h , b t h ) and transitions from the latent state s t h to s t h +1 ∼ P h ( · | s t h , a t h , b t h ) with an observation o t h +1 ∼ O h ( ·| s t h ) generated. When the underlying state s t h is observable and the policies become Markovian, we have actions a t h ∼ π t h ( s t h ) and b t h ∼ ν t h ( s t h ) and the reward r h ( s t h , a t h , b t h ) . Then, it reduces to the interaction process under the FOMG setting.

Value Function, Best Response, and Nash Equilibrium. To characterize the learning objective and the performance of the algorithms, we define the value function as the expected cumulative rewards under the policy pair ( π, ν ) starting from the initial step h = 1 . For FOMGs, we define the value function as V π,ν := E [ ∑ H h =1 r h ( s h , a h , b h ) | s 1 , π, ν, P ] , where the expectation is taken over all the randomness induced by π , ν , and P . For POMG, we define the value function as V π,ν := E [ ∑ H h =1 r h ( o h , a h , b h ) | π, ν, θ ] , with the expectation taken for π , ν , and θ .

Our work studies the competitive setting of RL, where Player 1 ( max-player ) aims to maximize the value function V π,ν while Player 2 ( min-player ) aims to minimize it. With the defined value function, given a policy pair ( π, ν ) , we define their best responses respectively as br ( π ) ∈ arg min ν V π,ν and br ( ν ) ∈ arg max π V π,ν . Then, we say a policy pair ( π ∗ , ν ∗ ) is a Nash equilibrium (NE) if

Thus, it always holds that π ∗ = br ( ν ∗ ) and ν ∗ = br ( π ∗ ) . For abbreviation, we denote V ∗ = V π ∗ ,ν ∗ , V π, ∗ = min ν V π,ν , and V ∗ ,ν = max π V π,ν , which implies V ∗ = V π ∗ , ∗ = V ∗ ,ν ∗ for NE ( π ∗ , ν ∗ ) . Moreover, we define the policy pair ( π, ν ) as an ε -approximate NE if it satisfies V ∗ ,ν -V π, ∗ ≤ ε .

Function Approximation. Since the environment is unknown to players, the model-based RL setting requires us to learn the true model of the environment, f ∗ , via (general) function approximation. We use the functions f lying in a general model function class F to approximate the environment. We make a standard realizability assumption on the relationship between the model class and the true model.

Assumption 1 (Realizability) . For a model class F , the true model f ∗ satisfies f ∗ ∈ F .

In our work, the true model f ∗ represents the transition kernel P for the FOMG and θ for the POMG. For any f ∈ F , we let P f and θ f = ( µ f , P f , O f ) be the models under the approximation function f and V π,ν f the value function associated with f . For POMGs, we denote P π,ν f,h and P f,h as P π,ν h and P h under the model f .

MGswith Self-Play and Adversarial Learning. Our work investigates two important MG setups for competitive RL, which are the self-play setting and the adversarial setting. In the self-play setting, the learner can control both players together to execute the proposed algorithms to learn an approximate NE. Therefore, our objective is to design sample-efficient algorithms to generate a sequence of policy pairs { ( π t , ν t ) } T t =1 in T episodes such that the following regret can be minimized,

In the adversarial setting, we can no longer coordinate both players, and only single player is controllable. Under such a circumstance, the opponent plays arbitrary and even adversarial policies. Wlog, suppose that the main player is the max-player with the policies { π t } T t =1 generated by a carefully designed algorithm and the opponent is min-player with arbitrary policies { ν t } T t =1 . The objective of the algorithm is to learn policies { π t } T t =1 to maximize the overall cumulative rewards in the presence of an adversary. To measure the performance of algorithms, we define the following regret for the adversarial setting by comparing the learned value against the Nash value, i.e.,

3 Model-Based Posterior Sampling for the Self-Play Setting

We propose algorithms aiming to generate a sequence of policy pairs { ( π t , ν t ) } T t =1 by controlling the learning process of both players such that the regret Reg sp ( T ) is small. Such a regret can be decomposed into two parts, namely ∑ T t =1 [ V ∗ f ∗ -V π t , ∗ f ∗ ] and ∑ T t =1 [ V ∗ ,ν t f ∗ -V ∗ f ∗ ] , which inspires our to design algorithms for learning { π t } T t =1 and { ν t } T t =1 separately by targeting at minimizing these two parts respectively. Due to the symmetric structure of such a game learning problem, we propose the algorithm to learn { π t } T t =1 as summarized in Algorithm 1. The algorithm for learning { ν t } T t =1 can be proposed in a symmetric way in Algorithm 3, which is deferred to Appendix A. Our proposed algorithm features an integration of the model-based posterior sampling and the exploiter-guided self-play in a multi-agent learning scenario. In Algorithm 1, Player 1 is the main player, while Player 2 is called the exploiter, who assists the learning of the main player by exploiting her weakness.

Posterior Sampling for the Main Player. The posterior sampling constructs a posterior distribution p t ( ·| Z t -1 ) over the function class F each round based on collected data and a pre-specified prior distribution p 0 ( · ) , where Z t -1 denotes the random history up to the end of the ( t -1) -th episode. For ease of notation, hereafter, we omit Z t -1 in the posterior distribution. Most recent literature shows that adding an optimism term in the posterior distribution can lead to sample-efficient RL algorithms. Thereby, we define the distribution p t ( · ) over the function class F for the main player as in Line 3 of Algorithm 1, which is proportional to p 0 ( f ) exp[ γV ∗ f + ∑ t -1 τ =1 ∑ H h =1 L τ h ( f )] . Here, γV ∗ f serves as the optimism term, and L τ h ( f ) is the likelihood function built upon the pre-collected data. Such a construction of p t ( · ) indicates that we will assign a higher probability (density) to a function f , which results in higher values of the combination of the optimism term and the likelihood function. We sample a model f t from the distribution p t ( · ) over the model class and learn the policy π t for the main player such that ( π t , ν t ) is the NE of the value function under the model f t in Line 4, where ν t is a dummy policy and only used in our theoretical analysis.

Posterior Sampling for the Exploiter. The exploiter aims to track the best response of π t to assist learning a low regret. The best response of π t generated by the exploiter is nevertheless based on a value function under a different model than f t . Specifically, for the exploiter, we define the posterior sampling distribution q t ( · ) using an optimism term -γV π t , ∗ f and the summation of likelihood functions, i.e., ∑ t -1 τ =1 ∑ H h =1 L τ h ( f ) , along with a prior distribution q 0 ( · ) , in Line 5 of Algorithm 1. The negative term -γV π t , ∗ f favors a model with a low value and is thus optimistic from the exploiter's perspective but pessimistic for the main player. We then sample a model f t from q t ( · ) and compute the best response of π t , denoted as ν t , under the model f t as in Line 7.

Algorithm 1 Model-Based Posterior Sampling for Self-Play (Max-Player)

1: Input: Model class F , prior distributions p 0 and q 0 , γ 1 , and γ 2 .

2: for t = 1 , . . . , T do

4: Compute π t by letting ( π t , ν t ) be the NE of V π,ν f t

3: Draw a model f t ∼ p t ( · ) with defining p t ( f ) ∝ p 0 ( f ) exp[ γ 1 V ∗ f + ∑ t -1 τ =1 ∑ H h =1 L τ h ( f )]

5: Draw a model f t ∼ q t ( · ) with defining q t ( f ) ∝ q 0 ( f ) exp[ -γ 2 V π t , ∗ f + ∑ t -1 τ =1 ∑ H h =1 L τ h ( f )]

6: Compute ν t by letting ν t be the best response of π t w.r.t. V π,ν f t

7: Collect data D t by executing the joint exploration policy σ t

8: Define the likelihood functions { L t h ( f ) } H h =1 using the collected data D t

9: end for

10: Return: ( π 1 , . . . , π T ) .

Data Sampling and Likelihood Function. With the learned π t and ν t , we define a joint exploration policy σ t in Line 7 of Algorithm 1, by executing which we can collect a dataset D t . We are able to further construct the likelihood functions { L t h ( f ) } H h =1 in Line 8 using D t . Different game settings require specifying diverse exploration policies σ t and likelihood functions { L t h ( f ) } H h =1 . Particularly, for the game classes mainly discussed in this work, we set σ t = ( π t , ν t ) for both FOMGs and POMGs. In FOMGs, we let D t = { ( s t h , a t h , b t h , s t h +1 ) } H h =1 , where for each h ∈ [ H ] , the data point ( s t h , a t h , b t h , s t h +1 ) is collected by executing σ t to the h -th step of the game. The corresponding likelihood function is defined using the transition kernel as

Furthermore, under the POMG setting, we let the dataset be D t = { τ t h } H h =1 , where the data point τ t h = ( o t 1 , a t 1 , b t 1 . . . , o t h , a t h , b t h ) is collected by executing σ t to the h -th step of the game for each h ∈ [ H ] . We further define the associated likelihood function as

Such a construction of the likelihood function in a log-likelihood form can result in learning a model f well approximating the true model f ∗ measured via the Hellinger distance.

3.1 Regret Analysis for the Self-play Setting

Our regret analysis is based on a novel structural complexity condition for multi-agent RL and a quantity to measure how the well the prior distributions cover the optimal model f ∗ . We first define the following condition for the self-play setting.

Definition 1 (Self-Play GEC) . For any sequences of functions f t , g t ∈ F , suppose that a pair of policies ( π t , ν t ) satisfies: (a) π t = argmax π min ν V π,ν f t and ν t = argmin ν V π t ,ν g t , or (b) ν t = argmin ν max π V π,ν f t and π t = argmax π V π,ν t g t . Denoting the joint exploration policy as σ t depending on f t and g t , for any ρ ∈ { f, g } and ( π t , ν t ) following (a) and (b) , the self-play GEC d GEC is defined as the minimal constant d satisfying

Our definition of self-play GEC is inspired by [83] for the single-agent RL. Then, it shares an analogous meaning to the single-agent GEC. Here ( σ τ , h ) implies running the joint exploration policy σ τ to step h to collect a data point ξ τ h . The LHS of the inequality is viewed as the prediction error and the RHS is the training error defined on a loss function ℓ plus a burn-in error 2 H ( dHT ) 1 2 + ϵHT that is non-dominating when ϵ is small. The loss function ℓ and ϵ can be problem-specific. We determine ℓ ( f, ξ h ) for FOMGs with ξ h = ( s h , a h ) and POMGs with ξ h = τ h respectively as

such that E ( σ,h ) [ ℓ ( f, ξ h )] = D 2 He ( P σ f ∗ ,h , P σ f,h ) for POMGs. The intuition for GEC is that if hypotheses have a small training error on a well-explored dataset, then the out-of-sample prediction error is also small, which characterizes the hardness of environment exploration.

Since the posterior sampling steps in our algorithms depend on the initial distributions p 0 and q 0 , we define the following quantity to measure how well the prior distributions p 0 and q 0 cover the optimal model f ∗ ∈ F , which is also a multi-agent generalization of its single-agent version [2, 83].

Definition 2 (Prior around True Model) . Given β > 0 and any distribution p 0 ∈ ∆ F , we define

where F ( ε ) := { f ∈ F : sup h,s,a,b KL 1 2 ( P f ∗ ,h ( · | s, a, b ) ∥ P f,h ( · | s, a, b )) ≤ ε } for FOMGs and F ( ε ) := { f ∈ F : sup π,ν KL 1 2 ( P π,ν f ∗ ,H ∥ P π,ν f,H ) ≤ ε } for POMGs.

When the model class F is a finite space, if let p 0 = Unif( F ) , we simply know that ω ( β, p 0 ) ≤ log |F| where |F| is the cardinality of F . Furthermore, for an infinite function class F , the term log |F| can be substituted by a quantity having logarithmic dependence on the covering number of the function class F . With the multi-agent GEC condition and the definition of ω , we have the following regret bound for both FOMGs and POMGs.

Proposition 1. Letting η = 1 / 2 , γ 1 = 2 √ ω (4 HT,p 0 ) T/d GEC , γ 2 = 2 √ ω (4 HT,q 0 ) T/d GEC , ϵ = 1 / √ HT in Definition 1, when T ≥ max { 4 H 2 ω (4 HT,p 0 ) /d GEC , 4 H 2 ω (4 HT,q 0 ) /d GEC , d GEC /H } , under both FOMG and POMG settings, Algorithm 1 admits the following regret bound,

This proposition gives the upper bound E [Reg sp 1 ( T )] following the updating rules in Algorithm 1 when the max-player is the main player. As Algorithm 3 is symmetric to Algorithm 1, we obtain the following regret bound of E [Reg sp 2 ( T )] for Algorithm 3 when the min-player is the main player.

Proposition 2. Under the same parameter settings as Proposition 1, Algorithm 3 admits the following regret bound,

Combining the results of Propositions 1 and 2, due to Reg sp ( T ) = Reg sp 1 ( T ) + Reg sp 2 ( T ) , we obtain the following overall regret when running Algorithms 1 and 3 together.

Theorem 1. Under the settings of Propositions 1 and 2, executing both Algorithms 1 and 3 leads to

The above results indicate that the proposed posterior sampling self-play algorithms (Algorithms 1 and 3) separately admit a sublinear dependence on GEC d GEC , the number of learning episodes T , as well as ω (4 HT,p 0 ) and ω (4 HT,q 0 ) for both FOMG and POMG settings. They lead to the same overall regret bound combining Propositions 1 and 2. In particular, when F is finite with p 0 = q 0 = Unif( F ) , Algorithms 1 and 3 admit regrets of O ( √ d GEC HT · log |F| ) . The quantity ω can be associated with the log-covering number if F is infinite. Please see Appendix C for analysis.

4 Posterior Sampling for the Adversarial Setting

Without loss of generality, we assume that the max-player is the main agent and the min-player is the opponent. Under this setting, the goal of the main player is to maximize her cumulative rewards as much as possible, comparing against the value under the NE, i.e., V ∗ f ∗ . We develop a novel algorithm for this setting as summarized in Algorithm 2. In our algorithm, the opponent's policy is assumed to be arbitrary and is also not revealed to the main player. The only information about the opponent is the current state or the partial observation of her state as well as the actions taken.

We adopt the optimistic posterior sampling approach for the main player with defining an optimism term as γV ∗ f motivated by the above learning target, and the likelihood function L t h ( f ) with L t h ( f ) := η log P f,h ( s t h +1 | s t h , a t h , b t h ) in (1) for FOMGs and L t h ( f ) = η log P f,h ( τ t h ) in (2) for POMGs

Algorithm 2 Model-Based Posterior Sampling with Adversarial Opponent

1: Input: Model class F , prior distributions p 0 , and γ .

2: for t = 1 , . . . , T do

3: Draw a model f t ∼ p t ( · ) with defining p t ( f ) ∝ p 0 ( f ) exp[ γV ∗ f + ∑ t -1 τ =1 ∑ H h =1 L τ h ( f )]

4: Compute π t by letting ( π t , ν t ) be NE of V π,ν f t

5: Opponent picks an arbitrary policy ν t

6: Collect a trajectory D t by executing the joint exploration policy σ t

7: Define the likelihood functions { L t h ( f ) } H h =1 using the collected data D t

8: end for

9: Return: ( π 1 , . . . , π T ) .

respectively. The policy π t learned by the main player is from computing the NE of the value function under the current model f t sampled from the posterior distribution p t . In addition, the joint exploration policy is set to be σ t = ( π t , ν t ) where ν t is the potentially adversarial policy played by the opponent. Thus, we can collect the data defined as D t = { ( s t h , a t h , b t h , s t h +1 ) } H h =1 and D t = { τ t h } H h =1 with τ t h = ( o t 1 , a t 1 , b t 1 . . . , o t h , a t h , b t h ) for FOMGs and POMGs respectively, collected by executing σ t to the h -th step of the game for each h ∈ [ H ] .

Remark 1. In Algorithm 2, we define the joint exploration policy σ t = ( π t , ν t ) , which is the key to the success of the algorithm design under the adversarial setting, especially for POMGs. Under the single-agent setting, the prior work [83] sets the exploration policy for a range of partially observable models subsumed by the PSR model as π t 1: h -1 ◦ h Unif( A ) , i.e., running π t for steps 1 to h -1 and then sampling the data at step h by enforcing a uniform policy. Such an exploration scheme fails to work when facing an uncontrollable opponent who does not play a uniform policy at step h . Theoretically, we prove that employing policies ( π t 1: h , ν t 1: h ) for exploration without the uniform policy, the self-play and adversarial GEC conditions in Definitions 1 and 3 are still satisfied for a class of POMGs including weakly revealing and decodable POMGs. This eventually leads to a unified adversarial learning algorithm for both FOMGs and POMGs.

4.1 Regret Analysis for the Adversarial Setting

Before demonstrating our regret analysis, we first define a multi-agent GEC fitting the adversarial learning scenario. Considering that the opponent's policy is uncontrollable during the learning, we let { ν t } T t =1 be arbitrary, which is clearly distinguished from self-play GEC defined in Definition 1.

Definition 3 (Adversarial GEC) . For any sequence of functions { f t } T t =1 with f t ∈ F and any sequence of the opponent's policies { ν t } T t =1 , suppose that the main player's policies { µ t } T t =1 are generated via µ t = argmax π min ν V π,ν f t . Denoting the joint exploration policy as { σ t } T t =1 depending on { f t } T t =1 , the adversarial GEC d GEC is defined as the minimal constant d satisfying

Our regret analysis for Algorithm 2 also depends on the quantity ω ( β, p 0 ) that characterizes the coverage of the prior distribution p 0 on the true model f ∗ . Then, we have the following regret bound.

Theorem 2. Letting η = 1 2 , γ = 2 √ ω (4 HT,p 0 ) T/d GEC , ϵ = 1 / √ HT in Definition 3, when T ≥ max { 4 H 2 ω (4 HT,p 0 ) /d GEC , d GEC /H } , under both FOMG and POMG settings, Algorithm 2 admits the following regret bound,

The above result indicates that we can achieve a meaningful regret bound by a posterior sampling algorithm with general function approximation, even when the opponent's policy is adversarial and her full policies ν t are not revealed. This regret has a sublinear dependence on d GEC , the number of episodes T , as well as ω (4 HT,p 0 ) . Similarly, when F is finite, Algorithm 2 admits a regret of O ( √ d GEC HT · log |F| ) . The term log |F| can be the log-covering number of F if it is infinite.

5 Theoretical Analysis

This section presents several examples of tractable MG classes captured by the self-play and adversarial GEC, the discussion of the quantity ω ( β, p 0 ) , the proof sketches of main theorems, and the discussion of limitations.

Examples. We call the class with a low d GEC the low self-play GEC class and low adversarial GEC class . Next, we analyze the relation between the proposed classes and the following MG classes. We also propose a new decodable POMG class generalized from the single-agent POMDP. We note that except for the linear MG, the other classes cannot be analyzed by the recent posterior sampling work [73]. We defer detailed definitions and proofs to Appendix B.

Linear MG. This FOMG class admits a linear structure of the reward and transition by feature vectors ϕ ( s, a, b ) ∈ R d as r h ( s, a, b ) = w ⊤ h ϕ ( s, a, b ) and P h ( s ′ | s, a, b ) = θ h ( s ′ ) ⊤ ϕ ( s, a, b ) [71]. We then prove that ' linear MG ⊂ low self-play/adversarial GEC " with d GEC = ˜ O ( H 3 d ) .

Linear Mixture MG. This FOMG class admits a different type of the linear structure for the transition [16] as P h ( s ′ | s, a, b ) = θ ⊤ h ϕ ( s, a, b, s ′ ) with ϕ ( s, a, b, s ′ ) ∈ R d . We prove that ' linear mixture MG ⊂ low self-play/adversarial GEC " with d GEC = ˜ O ( H 3 d ) .

LowSelf-Play Witness Rank. [28] defines this FOMG class for self-play by supposing that an inner product of specific vectors in R d defined on current models can lower bound witnessed model misfit and upper bound the Bellman error with a coefficient κ wit , which generalizes linear/linear mixture MGs. We can prove ' low self-play witness rank ⊂ low self-play GEC " with d GEC = ˜ O ( H 3 d/κ 2 wit ) .

α -Weakly Revealing POMG. This POMG class assumes min h σ S ( O h ) ≥ α where O h ∈ R |O|×|S| is the matrix by O h ( ·|· ) and σ S is the S -th singular value [41]. We prove that ' α -weakly revealing POMG ⊂ low self-play/adversarial GEC " with d GEC = ˜ O ( H 3 |O| 3 |A| 2 |B| 2 |S| 2 /α 2 ) .

Decodable POMG. We propose decodable POMGs by generalizing decodable POMDPs [23, 22], assuming that an unknown decoder ϕ h recovers states from observations, i.e., ϕ h ( o ) = s . We can prove ' decodable POMG ⊂ low self-play/adversarial GEC " with d GEC = ˜ O ( H 3 |O| 3 |A| 2 |B| 2 ) .

Discussion of ω ( β, p 0 ) . We briefly discuss the upper bound of the quantity ω ( β, p 0 ) for FOMGs and POMGs. We refer readers to Appendix C for more detailed proofs. For FOMGs, according to Lemma 2 of [2], when F is finite, p 0 = Unif( F ) , then ω ( β, p 0 ) ≤ log |F| by its definition. When F is infinite, it shows that under mild conditions, there exists a prior p 0 over F , B ≥ log(6 B 2 /ϵ ) , and ν = ϵ/ (6 log(6 B 2 /ϵ )) such that ω ( β, p 0 ) ≤ βϵ + log( N ( ϵ 6 log( B/ν ) )) , where N ( ϵ ) is the ϵ -covering number w.r.t. the distance d ( f, f ′ ) := sup s,a,b,h | D 2 He ( P f,h ( · | s, a, b ) , P f ∗ ,h ( · | s, a, b )) -D 2 He ( P f ′ ,h ( · | s, a, b ) , P f ∗ ,h ( · | s, a, b )) | . Since we have | D 2 He ( P, R ) -D 2 He ( Q,R ) | ≤ √ 2 2 ∥ P -Q ∥ 1 for any distributions P, Q , and R , the covering number w.r.t. the distance d can connect to the more common covering number w.r.t. the ℓ 1 distance. Thus, the upper bound of ω ( β, p 0 ) can be calculated for different cases. Additionally, for POMGs, inspired by [2], our work proves that under similar conditions, ω ( β, p 0 ) with finite and infinite F admit the same bounds as those for FOMGs. The difference is that the covering number is w.r.t. the distance d ( f, f ′ ) = sup π,ν | D 2 He ( P π,ν f,H , P π,ν f ∗ ,H ) -D 2 He ( P π,ν f ′ ,H , P π,ν f ∗ ,H ) | , which further connects to the ℓ 1 distance defined as d 1 ( f, f ′ ) := sup π,ν ∥ P π,ν f,H -P π,ν f ′ ,H ∥ 1 . Such a covering number under ℓ 1 distance is further analyzed in [78]. Our work gives the first detailed proof for the upper bound of ω ( β, p 0 ) under the partially observable setting, which is thus of independent interest.

Next, we outline our proof sketches. Detailed proofs are deferred to Appendices D, E, and F.

Proof Sketch of Theorem 1. To prove Theorem 1, we only need to combine the result in Propositions 1 and 2 via E [Reg sp ( T )] = E [Reg sp 1 ( T )+Reg sp 2 ( T )] . Wethus first give a proof sketch for Proposition 1. We decompose Reg sp 1 ( T ) = Term(i) + Term(ii) where

Intuitively, E [ Term(i) ] is the main player's regret incurred Line 4 of Algorithm 1 and E [ Term(ii) ] is the exploiter's regret incurred by Line 6. We further show

where ∆ V ∗ f t := V π t ,ν t f t -V ∗ f ∗ and ∆ V π t , ∗ f t = V π t ,ν t f t -V π t , ∗ f ∗ are associated with the optimism terms in posterior distributions. The inequality above for Term(i) is due to Line 4 such that V π t ,ν t f t = min ν V π t ,ν f t ≤ V π t ,ν t f t . By Definition 1 for self-play GEC, we obtain that ∑ T t =1 ( V π t ,ν t f t , 1 -V π t ,ν t f ∗ ) and ∑ T t =1 ( V π t ,ν t f ∗ -V π t ,ν t f t , 1 ) can be bounded by

t

where ρ t is chosen as f or f t respectively. By Lemma 11 and Lemma 12, we prove that for both FOMGs and POMGs, the accumulation of the losses ℓ ( f t , ξ ι h ) in (3) connects to the likelihood function L t h defined in (1) and (2). Thus, we obtain E [ Term(i) ] ≤ ∑ T t =1 E Z t -1 E f t ∼ p t {-γ 1 ∆ V ∗ f t -∑ H h =1 ∑ t -1 ι =1 [ L t h ( f t ) -L t h ( f ∗ )] + log p t ( f t ) p 0 ( f t ) } + 2 H ( d GEC HT ) 1 2 + ϵHT and E [ Term(ii) ] has a similar bound based on q t , where Z t -1 is the randomness history. By Lemma 10, the posterior distributions p t and q t following Lines 3 and 5 of Algorithm 1 can minimize the above upper bounds for E [ Term(i) ] and E [ Term(ii) ] . Therefore, we can relax p t and q t to be distributions defined around the true model f ∗ to enlarge above bounds. When T is sufficiently large and η = 1 / 2 , we have

Choosing proper values for ϵ , γ 1 , and γ 2 , we obtain the bound for E [Reg sp 1 ( T )] in Theorem 1 via Reg sp 1 ( T ) = Term(i) + Term(ii). In addition, we can prove the bound of E [Reg sp 2 ( T )] in a symmetric manner. Finally, combining E [Reg sp 1 ( T )] and E [Reg sp 2 ( T )] gives the result in Theorem 1.

Proof Sketch of Theorem 2. Under the adversarial setting, the policy of the opponent ν t is not generated by the algorithm, which could be arbitrarily time-varying. We decompose Reg adv ( T ) = ∑ T t =1 ∆ V ∗ f t + ∑ T t =1 [ V ∗ f t -V π t ,ν t f ∗ ] where ∆ V ∗ f t := V ∗ f ∗ -V ∗ f t relates to optimism. Since ( π t , ν t ) is NE of V π,ν f t as in Line 3 of Algorithm 2, we have V ∗ f t = min ν V π t ,ν f t ≤ V π t ,ν t f t , which leads to

We can bound ∑ T t =1 [ V π t ,ν t f t -V π t ,ν t f ∗ ] via adversarial GEC in Definition 3 by

Connecting the loss ℓ ( f t , ξ ι h ) to the likelihood function L t h defined in (1) and (2) via Lemmas 11 and 12, we obtain E [Reg adv ( T )] ≤ ∑ T t =1 E Z t -1 E f t ∼ p t { γ ∑ T t =1 ∆ V ∗ f t -∑ H h =1 ∑ t -1 ι =1 [ L t h ( f t ) -L t h ( f ∗ )] + log p t ( f t ) p 0 ( f t ) } +2 H ( d GEC HT ) 1 2 + ϵHT . Lemma 10 shows p t in Line 3 of Algorithm 2 can minimize this bound. Thus, relaxing p t to be distribution defined around the true model f ∗ , with sufficiently large T and η = 1 / 2 , we have

Choosing proper values for ϵ and γ , we eventually obtain the bound for E [Reg adv ( T )] in Theorem 2.

Discussion of Limitations. Our work has studied several but a limited number of tractable MG classes in both FOMGs and POMGs. It is interesting to further define new MG classes by generalizing their single-agent counterparts and explore the relation between these MG classes and low selfplay/adversarial GEC classes. It is also intriguing to generalize our method to general-sum settings.

Acknowledgments and Disclosure of Funding

The authors would like to thank the anonymous reviewers for their valuable comments. The authors would like to thank Wei Xiong for the helpful discussions. Shuang Qiu and Tong Zhang acknowledge the funding supported by the GRF 16310222.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Alekh Agarwal and Tong Zhang. Model-based rl with optimistic posterior sampling: Structural conditions and sample complexity. arXiv preprint arXiv:2206.07659 , 2022.

Alekh Agarwal and Tong Zhang. Non-linear reinforcement learning in large action spaces: Structural conditions and sample-efficiency of posterior sampling. arXiv preprint arXiv:2203.08248 , 2022.

Shipra Agrawal and Randy Jia. Posterior sampling for reinforcement learning: worst-case regret bounds. arXiv preprint arXiv:1705.07041 , 2017.

Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning , pages 463-474. PMLR, 2020.

Yu Bai and Chi Jin. Provable self-play algorithms for competitive reinforcement learning. In International conference on machine learning , pages 551-560. PMLR, 2020.

Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play. Advances in neural information processing systems , 33:2159-2170, 2020.

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019.

Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. arXiv preprint arXiv:2305.09659 , 2023.

Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science , 365(6456):885-890, 2019.

Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning , pages 1283-1294. PMLR, 2020.

Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. Advances in neural information processing systems , 24, 2011.

Fan Chen, Yu Bai, and Song Mei. Partially observable rl with b-stability: Unified structural condition and sharp sample-efficient algorithms, 2022.

Fan Chen, Song Mei, and Yu Bai. Unified algorithms for rl with decision-estimation coefficients: No-regret, pac, and reward-free learning. arXiv preprint arXiv:2209.11745 , 2022.

Zixiang Chen, Chris Junchi Li, Angela Yuan, Quanquan Gu, and Michael I Jordan. A general framework for sample-efficient function approximation in reinforcement learning. arXiv preprint arXiv:2209.15634 , 2022.

Zixiang Chen, Dongruo Zhou, and Quanquan Gu. Almost optimal algorithms for two-player zero-sum linear mixture markov games. In International Conference on Algorithmic Learning Theory , pages 227-261. PMLR, 2022.

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems , 31, 2018.

Qiwen Cui and Simon S Du. When is offline two-player zero-sum markov game solvable? arXiv preprint arXiv:2201.03522 , 2022.

Christoph Dann, Mehryar Mohri, Tong Zhang, and Julian Zimmert. A provably efficient model-free posterior sampling method for episodic reinforcement learning. Advances in Neural Information Processing Systems , 34:12040-12051, 2021.

Constantinos Daskalakis, Noah Golowich, and Kaiqing Zhang. The complexity of markov equilibrium in stochastic games. arXiv preprint arXiv:2204.03991 , 2022.

Simon Du, Sham Kakade, Jason Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, and Ruosong Wang. Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning , pages 2826-2836. PMLR, 2021.

Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning , pages 1665-1674. PMLR, 2019.

Yonathan Efroni, Chi Jin, Akshay Krishnamurthy, and Sobhan Miryoosefi. Provable reinforcement learning with a short-term memory. In International Conference on Machine Learning , pages 5832-5850. PMLR, 2022.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

Noah Golowich, Ankur Moitra, and Dhruv Rohatgi. Learning in observable pomdps, without computationally intractable oracles. arXiv preprint arXiv:2206.03446 , 2022.

Noah Golowich, Ankur Moitra, and Dhruv Rohatgi. Planning in observable pomdps in quasipolynomial time. arXiv preprint arXiv:2201.04735 , 2022.

Zhaohan Daniel Guo, Shayan Doroudi, and Emma Brunskill. A pac rl algorithm for episodic pomdps. In Artificial Intelligence and Statistics , pages 510-518. PMLR, 2016.

Baihe Huang, Jason D Lee, Zhaoran Wang, and Zhuoran Yang. Towards general function approximation in zero-sum markov games. arXiv preprint arXiv:2107.14702 , 2021.

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning , pages 1704-1713. PMLR, 2017.

Chi Jin, Sham Kakade, Akshay Krishnamurthy, and Qinghua Liu. Sample-efficient reinforcement learning of undercomplete pomdps. Advances in Neural Information Processing Systems , 33:18530-18539, 2020.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems , 34:13406-13418, 2021.

Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning-a simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint arXiv:2110.14555 , 2021.

Chi Jin, Qinghua Liu, and Tiancheng Yu. The power of exploiter: Provable multi-agent rl in large state spaces. arXiv preprint arXiv:2106.03352 , 2021.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Yujia Jin, Vidya Muthukumar, and Aaron Sidford. The complexity of infinite-horizon generalsum stochastic games. arXiv preprint arXiv:2204.04186 , 2022.

Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Pac reinforcement learning with rich observations. Advances in Neural Information Processing Systems , 29, 2016.

Jeongyeol Kwon, Yonathan Efroni, Constantine Caramanis, and Shie Mannor. Rl for latent mdps: Regret guarantees and a lower bound. Advances in Neural Information Processing Systems , 34:24523-24534, 2021.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Qinghua Liu, Alan Chung, Csaba Szepesvári, and Chi Jin. When is partially observable reinforcement learning not scary? In Conference on Learning Theory , pages 5175-5220. PMLR, 2022.

Qinghua Liu, Praneeth Netrapalli, Csaba Szepesvari, and Chi Jin. Optimistic mle-a generic model-based algorithm for partially observable sequential decision making. arXiv preprint arXiv:2209.14997 , 2022.

Qinghua Liu, Csaba Szepesvári, and Chi Jin. Sample-efficient reinforcement learning of partially observable markov games. arXiv preprint arXiv:2206.01315 , 2022.

Qinghua Liu, Yuanhao Wang, and Chi Jin. Learning markov games with adversarial opponents: Efficient algorithms and fundamental limits. arXiv preprint arXiv:2203.06803 , 2022.

Qinghua Liu, Tiancheng Yu, Yu Bai, and Chi Jin. A sharp analysis of model-based reinforcement learning with self-play. In International Conference on Machine Learning , pages 7001-7010. PMLR, 2021.

Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling. Advances in neural information processing systems , 30, 2017.

Weichao Mao and Tamer Ba¸ sar. Provably efficient reinforcement learning in decentralized general-sum Markov games. arXiv preprint arXiv:2110.05682 , 2021.

Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning , pages 1101-1112. PMLR, 2020.

Chengzhuo Ni, Yuda Song, Xuezhou Zhang, Chi Jin, and Mengdi Wang. Representation learning for general-sum low-rank markov games. arXiv preprint arXiv:2210.16976 , 2022.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016.

Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems , 27, 2014.

Shuang Qiu, Lingxiao Wang, Chenjia Bai, Zhuoran Yang, and Zhaoran Wang. Contrastive ucb: Provably efficient contrastive self-supervised learning in online reinforcement learning. In International Conference on Machine Learning , pages 18168-18210. PMLR, 2022.

Shuang Qiu, Xiaohan Wei, Jieping Ye, Zhaoran Wang, and Zhuoran Yang. Provably efficient fictitious play policy optimization for zero-sum markov games with structured transitions. In International Conference on Machine Learning , pages 8715-8725. PMLR, 2021.

Shuang Qiu, Jieping Ye, Zhaoran Wang, and Zhuoran Yang. On reward-free rl with kernel and neural function approximations: Single-agent mdp and markov game. In International Conference on Machine Learning , pages 8737-8747. PMLR, 2021.

Daniel Russo. Worst-case regret bounds for exploration via randomized value functions. Advances in Neural Information Processing Systems , 32, 2019.

Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research , 39(4):1221-1243, 2014.

Igal Sason and Sergio Verdú. f -divergence inequalities. IEEE Transactions on Information Theory , 62(11):5973-6006, 2016.

Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295 , 2016.

Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences , 39(10):1095-1100, 1953.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, and Marc Lanctot. Mastering the game of Go with deep neural networks and tree search. Nature , 529(7587):484-489, 2016.

Ziang Song, Song Mei, and Yu Bai. When can we learn general-sum Markov games with a large number of players sample-efficiently? arXiv preprint arXiv:2110.04184 , 2021.

Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model-based rl in contextual decision processes: Pac bounds and exponential improvements over model-free approaches. In Conference on learning theory , pages 2898-2933. PMLR, 2019.

Yi Tian, Yuanhao Wang, Tiancheng Yu, and Suvrit Sra. Online learning in unknown markov games. In International conference on machine learning , pages 10279-10288. PMLR, 2021.

Masatoshi Uehara, Ayush Sekhari, Jason D Lee, Nathan Kallus, and Wen Sun. Computationally efficient pac rl in pomdps with latent determinism and conditional embeddings. arXiv preprint arXiv:2206.12081 , 2022.

Masatoshi Uehara, Ayush Sekhari, Jason D Lee, Nathan Kallus, and Wen Sun. Provably efficient reinforcement learning in partially observable dynamical systems. arXiv preprint arXiv:2206.12020 , 2022.

Ramon Van Handel. Probability in high dimension. Technical report, PRINCETON UNIV NJ, 2014.

Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. StarCraft II: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782 , 2017.

Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Represent to control partially observed systems: Representation learning with provable sample efficiency. In The Eleventh International Conference on Learning Representations .

Ruosong Wang, Ruslan Salakhutdinov, and Lin F Yang. Provably efficient reinforcement learning with general value function approximation. arXiv preprint arXiv:2005.10804 , 2020.

Ruosong Wang, Russ R Salakhutdinov, and Lin Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems , 33:6123-6135, 2020.

Yining Wang, Ruosong Wang, Simon S Du, and Akshay Krishnamurthy. Optimism in reinforcement learning with generalized linear function approximation. arXiv preprint arXiv:1912.04136 , 2019.

Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 681-688. Citeseer, 2011.

Qiaomin Xie, Yudong Chen, Zhaoran Wang, and Zhuoran Yang. Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. In Conference on learning theory , pages 3674-3682. PMLR, 2020.

Wei Xiong, Han Zhong, Chengshuai Shi, Cong Shen, Liwei Wang, and Tong Zhang. Nearly minimax optimal offline reinforcement learning with linear function approximation: Singleagent mdp and markov game. arXiv preprint arXiv:2205.15512 , 2022.

Wei Xiong, Han Zhong, Chengshuai Shi, Cong Shen, and Tong Zhang. A self-play posterior sampling algorithm for zero-sum markov games. In International Conference on Machine Learning , pages 24496-24523. PMLR, 2022.

Yunchang Yang, Han Zhong, Tianhao Wu, Bin Liu, Liwei Wang, and Simon S Du. A reduction-based framework for sequential decision making with delayed feedback. arXiv preprint arXiv:2302.01477 , 2023.

Zhuoran Yang, Chi Jin, Zhaoran Wang, Mengdi Wang, and Michael I Jordan. On function approximation in reinforcement learning: Optimism in the face of large state spaces. arXiv preprint arXiv:2011.04622 , 2020.

Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics , pages 1954-1964. PMLR, 2020.

Wenhao Zhan, Jason D Lee, and Zhuoran Yang. Decentralized optimistic hyperpolicy mirror descent: Provably no-regret learning in markov games. arXiv preprint arXiv:2206.01588 , 2022.

Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D Lee. Pac reinforcement learning for predictive state representations. arXiv preprint arXiv:2207.05738 , 2022.

Tianjun Zhang, Tongzheng Ren, Mengjiao Yang, Joseph Gonzalez, Dale Schuurmans, and Bo Dai. Making linear mdps practical via contrastive representation learning. In International Conference on Machine Learning , pages 26447-26466. PMLR, 2022.

Tong Zhang. From ε -entropy to kl-entropy: Analysis of minimum information complexity density estimation. 2006.

Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science , 4(2):834-857, 2022.

Han Zhong, Wei Xiong, Jiyuan Tan, Liwei Wang, Tong Zhang, Zhaoran Wang, and Zhuoran Yang. Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. arXiv preprint arXiv:2202.07511 , 2022.

Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. Gec: A unified framework for interactive decision making in mdp, pomdp, and beyond. arXiv preprint arXiv:2211.01962 , 2022.

Han Zhong, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Can reinforcement learning find stackelberg-nash equilibria in general-sum markov games with myopically rational followers? Journal of Machine Learning Research , 24(35):1-52, 2023.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR, 2021.

Contents

Appendix

A Omitted Algorithm in the Main Text

Algorithm 3 Model-Based Posterior Sampling for Self-Play (Min-Player)

1: Input: Model class F , prior distributions p 0 and q 0 , γ 1 , and γ 2 .

2: for t = 1 , . . . , T do

4: Compute ν t by letting ( π t , ν t ) be NE of V π,ν f t

3: Draw a model f t ∼ p t ( · ) with defining p t ( f ) ∝ p 0 ( f ) exp[ -γ 1 V ∗ f + ∑ t -1 ι =1 ∑ H h =1 L τ h ( f )]

5: Draw a model f t ∼ q t ( · ) with defining q t ( f ) ∝ q 0 ( f ) exp[ γ 2 V ∗ ,ν t f + ∑ t -1 ι =1 ∑ H h =1 L τ h ( f )]

6: Compute π t by letting π t be the best response of ν t w.r.t. V π,ν f t

7: Collect data D t by executing the joint exploration policy σ t

8: Define the likelihood functions { L t h ( f ) } H h =1 using the collected data D t

9: end for

10: Return: ( ν 1 , . . . , ν T ) .

B Computation of Self-Play and Adversarial GEC

In this section, we present the computation of the self-play and adversarial GEC for different FOMG and POMG classes, including linear MGs, linear mixture MGs, low-witness-rank MGs, weakly revealing POMGs, and a novel POMG class, dubbed decodable POMGs, newly proposed by this work.

Note that the GEC d GEC essentially depends on ϵ as shown in Definition 1 and Definition 3 such that it should be expressed as d ϵ GEC . For convenience, we omit such a dependence in the notation. In our main theorems, we set ϵ = 1 / √ HT , and according to our derivation in this section, d GEC has an O (log(1 /ϵ )) = O ( · log( HT )) factor which only scales logarithmically in T .

B.1 Linear MG

In this subsection, we show that the linear MG is in the class of MGs with low self-play and adversarial GEC and further calculate d GEC for the linear MG.

Definition 4 (Linear Markov Game [71]) . The linear MG is a FOMG admitting the following linear structures on the reward function and transition kernel,

where there exist a known feature map ϕ : S × A × B ↦→ R d and unknown coefficients w h ∈ R d and θ h ( s ′ ) ∈ R d that should be learned in algorithms. We assume that the feature map and the coefficients satisfy ∥ ϕ ( s, a, b ) ∥ 2 ≤ 1 , ∫ S ∥ θ h ( s ) ∥ 2 d s ≤ √ d , and ∥ w h ∥ 2 ≤ √ d .

Then, d GEC for the linear MG can be calculated in the following proposition. We can show that both low self-play and adversarial GEC classes subsume the class of linear MGs.

Proposition 3 (Linear MG ⊂ Low Self-Play/Adversarial GEC) . For linear MGs with ϕ ( s, a, b ) ∈ R d as defined in Definition 4, and for ϵ > 0 , when T is sufficiently large, choosing ℓ ( f, ξ h ) = D 2 He ( P f,h ( · | ξ h ) , P f ∗ ,h ( · | ξ h ) ) as in (3) with ξ h = ( s h , a h , b h ) , we have linear MG ⊂ MGwith low self-play and adversarial GEC

with d GEC satisfying

Proof. Since the value function is defined as V π,ν f,h ( s h ) := E π,ν, P f [ ∑ H h ′ = h r h ′ ( s h ′ , a h ′ , b h ′ ) | s h ] at step h with denoting V π,ν f = V π,ν f, 1 ( s 1 ) and we also have a relation V π,ν f =

E π,ν, P f ′ [ ∑ H h =1 ( V π,ν f,h ( s h ) -V π,ν f,h +1 ( s h +1 ))] for any f ′ using the telescoping summation as well as V π,ν f,H +1 ( · ) = 0 , we thus can decompose the value difference as follows,

t

t

t

t

where we define the following Bellman residual in expectation as

The Bellman residual for the linear MG can be linearly represented, whose form is

where we let

such that we have ∥ q h ( f, π, ν ) ∥ 2 ≤ 2 dH according to the definition of the linear MG. Therefore, the value difference term can also be rewritten as

where ( s t h , a t h , b t h ) can be viewed as sampled data following π t , ν t , and P f ∗ . For simplicity, we denote ϕ t h := E π t ,ν t , P f ∗ ϕ ( s t h , a t h , b t h ) , q t h := q h ( f t , π t , ν t ) , and ξ t h := ( s t h , a t h , b t h ) , and define Φ t h := λ I d + ∑ t ι =1 ϕ ι h ( ϕ ι h ) ⊤ with λ > 0 and φ t h := ∥ ϕ t h ∥ (Φ t -1 h ) -1 . Note that we have

where the term ⟨ ϕ ι h , q t h ⟩ 2 can be bounded as

where the first inequality uses V π,ν f,h +1 ( s ) ≤ H and the second inequality uses ∥ P -Q ∥ 2 1 ≤ 8 D 2 He ( P, Q ) . Moreover, using a ⊤ b ≤ ∥ a ∥ A ∥ b ∥ A -1 , we also have

where 1 {·} denotes an indicator function.

Plugging (5) into (7) and taking a summation from t = 1 to T , we obtain

where the second inequality uses a + b ≤ √ a + b and the last inequality is by Cauchy-Schwarz. According to the elliptical potential lemma (Lemma 15), we have the following bound,

where the first inequality uses Lemma 15, and the last inequality uses log det Φ t h ≤ d log tr(Φ T h ) d ≤ d log λd + T d . We also note that ϕ t h cannot exceed 1 too many times, as detailed in Lemma 16. By Lemma 16 and the fact that 1 { φ t h ≥ 1 } ≤ 1 , we obtain

Combining (6), (7), (8),(9), and (10), we obtain

which further leads to

by x ∧ y ≤ √ xy and setting ϵ = λHd where λ > 0 and ℓ ( f, ξ h ) := D 2 He ( P f,h ( · | ξ h ) , P f ∗ ,h ( · | ξ h ) ) with ξ h = ( s h , a h , b h ) . Thus, we obtain that when T is sufficiently large, we have

Note that the above results hold for arbitrary policy pair ( ν t , π t ) . This implies that linear MGs with the dimension d is subsumed by the class of MGs with self-play and adversarial GEC d GEC = 16 H 3 d log(1 + HT ϵ ) . This completes the proof.

B.2 Linear Mixture MG

In this subsection, we discuss the linear mixture MG setting, where the transition kernel is a linear combination of some basis transition kernels and thus admits a linear structure. Specifically, we have the following definition of the linear mixture MG.

Definition 5 (Linear Mixture Markov Game [16]) . There exists a d -dimensional feature map ϕ : S × A × B × S ↦→ R d and an unknown coefficient θ h ∈ R d , such that the transition of the Markov game can be represented by

where ∥ θ h ∥ 2 ≤ B and ∥ ∫ S ϕ ( s, a, b, s ′ ) V ( s ′ )d s ′ ∥ 2 ≤ H for any bounded function V : S ↦→ [0 , H ] .

In the context of linear mixture MGs, both self-play and adversarial GEC can be calculated.

Proposition 4 (Linear Mixture MG ⊂ Low Self-play/Adversarial GEC) . For linear mixture MGs with ϕ ( s, a, b, s ′ ) ∈ R d defined as in Definition 4, and for ϵ > 0 , when T is sufficiently large, choosing ℓ ( f, ξ h ) = D 2 ( P f,h ( · | ξ h ) , P f ∗ ,h ( · | ξ h ) ) as in (3) with ξ h = ( s h , a h , b h ) , we have

He linear mixture MG ⊂ MGwith low self-play and adversarial GEC

with d GEC satisfying

Proof. The proof of GEC bound in a linear mixture Markov game is similar to the one in the linear Markov game case. We first define the Bellman residual in expectation as

One can derive the following property of E π,ν f,h by precisely the same derivation as in (4), which is

We next show that in a linear mixture Markov game, the Bellman residual E π,ν f,h can be linearly represented, whose form is

where we define ϕ ( s h , a h , b h , f, π, ν ) := E π,ν, P f ∗ ∫ S ϕ ( s h , a h , b h , s ′ ) V π,ν f,h +1 ( s ′ )d s ′ and q h ( f ) := θ f,h -θ f ∗ ,h . Note that ∥ ϕ ( s h , a h , b h , f, π, ν ) ∥ ≤ H and ∥ q h ( f ) ∥ ≤ 2 B by Definition 5.

We next define ϕ t h := ϕ ( s t h , a t h , b t h , f t , π t , ν t ) , q t h := q h ( f t ) , Φ t h := λ I d + 1 H 2 ∑ t ι =1 ϕ ι h ( ϕ ι h ) ⊤ and φ t h := ∥ ϕ t h ∥ (Φ t -1 h ) -1 . Since we have

where the first inequality is due to V π ι ,ν ι f ι ,h +1 ( s ) ≤ H and the second inequality is due to ∥ P -Q ∥ 2 2 ≤ 8 D 2 He ( P, Q ) . By utilizing ∥ q h ( f ) ∥ ≤ 2 B and (12), we further obtain an upper bound of ∥ q t h ∥ Φ t -1 h as follows,

The rest of the proof is the same as the proof in the linear Markov game, from (5) to (11), except with a scaling factor. We provide the final result here as follows,

Note that here ( π t , ν t ) can be any arbitrary policy pair, including the ones required as in Definitions 1 and 3. Moreover, we will see that the above result satisfies the definitions of both self-play and adversarial GEC when using the inequality x ∧ y ≤ √ xy and choosing λ = ϵH/B 2 and ℓ ( f, ξ h ) = D 2 He ( P f,h ( · | ξ h ) , P f ∗ ,h ( · | ξ h ) ) . Therefore, we have

for T sufficiently large, which shows that linear mixture MGs with the dimension d is subsumed by MGs with self-play and adversarial GEC d GEC = 16 H 3 d log(1 + TB 2 dϵH ) . This completes the proof.

B.3 Low-Witness-Rank MG

Witness rank is a complexity measure that absorbs a wide range of settings in single-agent RL[60]. The work [28] further studies the witness rank under the MG setting with self-play. In this subsection, we show that MGs with low self-play GEC subsumes the class of MGs with a low witness rank in the

self-play setting. Since the witness rank under the adversarial setting was not investigated in prior works, we here only analyze the self-play setting.

Before presenting our main proposition, we first show the definitions of witnessed model misfit and witness rank in MGs with self-play [28].

Definition 6 (Witnessed model misfit in Markov game) . Suppose that there exists a discriminator class U with u ∈ U : S × A × B × S ↦→ [0 , H ] . The witnessed model misfit in Markov games W ( f, g, ρ, h ) for the models f, g, ρ ∈ F at step h is characterized by

where ˘ π := argmax π min ν V π,ν f and ˘ ν := argmin ν V ˘ π,ν g with f, g ∈ F .

The definition of (˘ π, ˘ ν ) matches the requirement for the policy pair in the self-play GEC, i.e., condition (1) in Definition 1. We note that condition (2) in our definition of self-play GEC is designed for Algorithm 3, which is thus a symmetric condition to (1) . We define the Bellman residual as

Then, we have the following definition of witness rank for MGs with self-play.

Definition 7 (Witness Rank for Self-Play) . There exist two functions W h : F × F ↦→ R d and X h : F ↦→ R d and constant κ > 0 such that

where ˘ π := argmax π min ν V π,ν f and ˘ ν := argmin ν V ˘ π,ν g with f, g ∈ F , ∥ W h ( · ) ∥ 2 ≤ HB for a constant B , and ∥ X h ( · ) ∥ 2 ≤ 1 for any step h . Then, the witness rank of MGs with self-play is defined as the dimension d of the range spaces for functions W and X .

We show that in the self-play setting, d GEC is bounded for MGs with a low witness rank. It is worth noting that the proof of GEC bound for MGs with a low witness rank generalizes the proof for linear MGs in Appendix B.1.

Proposition 5 (Low Self-Play Witness Rank ⊂ Low Self-play GEC) . For a Markov game with witness rank d as defined in Definition 7, when T sufficiently large, choosing ℓ ( f, ξ h ) = D 2 ( P f,h ( · | ξ h ) , P f ∗ ,h ( · | ξ h ) ) as in (3) with ξ h = ( s h , a h , b h ) , we have

MGwith low self-play witness rank ⊂ MGwith low self-play GEC

with d GEC satisfying

Proof. We first give a bound of the model misfit in the Hellinger distance. By 0 ≤ u ≤ H , we have

where ρ can be f or g as in the definition of self-play GEC. By the definition of W ( f, g, ρ, h ) and Jensen's inequality, we further get

where we use sup v E x f ( x, v ) ≤ E x sup v f ( x, v ) and the definition of (˘ π, ˘ ν ) . Note that following our definition of self-play GEC, if we choose f = f t and g = g t in the model misfit, then correspondingly we have ˘ π = π t and ˘ ν = ν t as in the condition (1) of Definition 1.

By the definition of E π,ν ρ,h , we can decompose the value difference as

Hereafter we define W t h := W h ( f t , g t ) , X t h := X h ( ρ t ) , Σ t h := λ I + ∑ t ι =1 X ι h ( X ι h ) ⊤ , and φ t h := ∥ X t h ∥ 2 (Σ t h ) -1 . Using the definition of W h and X h and E π t ,ν t ρ t ,h ≤ H , we obtain

where 1 {·} denotes the indicator function. For Term(ii), we can apply Lemma 16 to obtain

It remains to bound Term(i). We first calculate the sum of ∥ W t h ∥ 2 Σ t h , which is

where the first inequality uses ∥ W t h ∥ ≤ HB , and the second inequality uses the definition of witness rank and (13). We also let ξ ι h := ( s ι h , a ι h , b ι h ) . We next use Lemma 15 to obtain

Finally, by using the Cauchy-Schwarz inequality, we obtain the bound for Term(i) as

where the first inequality is by Cauchy-Schwarz, and the last inequality is by √ a + b ≤ √ a + √ b . We further invoke √ ab ≤ a 4 + b to obtain that

Finally, we combine the bounds for term(i) and term(ii) to obtain that for T sufficiently large,

where we also use the inequality x ∧ y ≤ √ xy . Then, we can see that the above results satisfy Definition 1 with choosing λ = 4 ϵκ 2 wit HB 2 . Thus, we conclude that under the self-play setting, we have

This concludes the proof.

B.4 Weakly Revealing POMG

In this section, we consider a weakly revealing POMG. Recall that { O h } H h =1 stands for the emission kernels, which can be viewed in the form of matrices O h ∈ R |O|×|S| , with the ( o, s ) -th element being O h ( o | s ) for step h . Similarly, the transition matrix P h,a,b ∈ R |S|×|S| is such that the ( s ′ , s ) -th element is P h ( s ′ | s, a, b ) .

Here we consider undercomplete POMGs where there are more observations than hidden states, |O| ≥ |S| . Formally, it can be defined as an α -revealing POMG as follows:

Definition 8 ( α -Revealing POMG [41]) . A POMG is α -revealing if it satisfies that there exists α > 0 such that min h σ S ( O h ) ≥ α , where σ S ( · ) denotes the S -th singular value of a matrix with S = |S| .

The α -revealing condition measures how hard it is to recover states from observations. Before presenting the GEC bound for an α -revealing POMG, we first define some notations. We define τ h : h ′ := ( o i , a i , b i ) h ′ i = h to be a set of observation-action tuples from step h to h ′ . For simplicity, we denote τ 1: h as τ h . We define q 0 := O 1 µ 1 ∈ R |O| , where µ 1 ∈ R |S| is a vector formed by the distribution of the initial state. We define q ( τ h ) := [Pr ( o 1: h , o h +1 | do ( a 1: h , b 1: h ) ) ] o ∈O ∈ R |O| when τ h = ( o i , a i , b i ) h i =1 . Here do ( a 1: h , b 1: h ) means executing actions a h ′ and b h ′ at step h ′ . In other words, Pr ( o 1: h , o h +1 | do ( a 1: h , b 1: h ) ) describes the probability of receiving observations o 1: h given the actions fixed as ( a 1: h -1 , b 1: h -1 ) . We define ¯ q ( τ h ) := q ( τ h ) / P ( τ h ) , which is the concatenation of the probabilities under condition τ h . We have ¯ q ( τ h ) = [Pr( o | τ 1: h )] o ∈O , where Pr( o | τ 1: h ) means the probability of receiving o h +1 = o under the condition that the previous trajectory is τ h .

Based on the above definitions, we then derive the GEC bound for an α -revealing POMG in the following proposition.

Proposition 6 ( α -Revealing POMG ⊂ Low Self-Play/Adversarial GEC) . For an α -revealing POMG defined in Definition 8, with the loss ℓ ( f, ξ h ) defined in (3) for POMGs, we have

α -revealing POMG ⊂ MGwith low self-play and adversarial GEC

with d GEC satisfying

For our proof of this proposition, we prove several supporting lemmas presented in Appendix B.4.1.

Proof. For ease of notation, we let S = |S| , A = |A| , B = |B| , and O = |O| in our proof. We define the following observed operator representation:

for 1 ≤ h ≤ H . One can verify that the previously defined M h ( o, a, b ) has the following property,

Specifically, P ( τ H ) = M H ( o H , a H , b H ) . . . M 1 ( o 1 , a 1 , b 1 ) q 0 . For simplicity, we define

We also rewrite M H : h +1 ( o h +1: H , a h +1: H , b h +1: H ) as m ( o h +1: H , a h +1: H , b h +1: H ) to emphasize that M H : h +1 ( o h +1: H , a h +1: H , b h +1: H ) is a vector since M H ( o H , a H , b H ) is a vector.

We start by using Lemma 5 to decompose the regret as follows

Combine this with the fact that V π t ,ν t f t -V π t ,ν t f ∗ ≤ H ,we have

where we define

and M t h and q t 0 correspond to the model f t sampled in time step t .

We next focus on ∑ τ h ∥ ( M t h ( o h , a h , b h ) -M h ( o h , a h , b h )) q ( τ h -1 ) ∥ 1 σ t ( τ h ) . By the definition of ¯ q and q , we obtain

where ˜ e j denotes the j -th standard basis vector in R O , and σ t ( τ h : h ′ ; τ h -1 ) := π t ( τ h : h ′ ; τ h -1 ) · ν t ( τ h : h ′ ; τ h -1 ) , where π t ( τ h : h ′ ; τ h -1 ) = ∏ h ′ h ′′ = h π t h ′′ ( a h ′′ | τ h ′′ -1 , o h ′′ ) , ν t ( τ h : h ′ ; τ h -1 ) = ∏ h ′ h ′′ = h ν t h ′′ ( b h ′′ | τ h ′′ -1 , o h ′′ ) . For simplicity, we define w h,t,j,o,a,b := ( ˜ e ⊤ j ( M t h ( o, a, b ) -M h ( o, a, b )) O h -1 ) ⊤ and x τ h -1 ,σ,o,a,b := O † h -1 ¯ q ( τ h -1 ) σ ( o, a, b ; τ h -1 ) . Then, we have

Next, we will apply the ℓ 2 eluder technique [13, 83] in Lemma 17 to derive the upper bound. We first analyze the upper bounds of ∑ O j =1 ∥ w h,t,j,o,a,b ∥ 2 and E τ h -1 ∼ P π t ,ν t f ∗ ,h -1 ∥ x τ h -1 ,σ t ,o,a,b ∥ 2 2 for the usage of the ℓ 2 eluder technique. According to Lemma 7, for any ( o, a, b ) ∈ O × A × B , we have

According to Lemma 17, setting R = α √ S and R 2 x R 2 w ≤ 4 A 2 B 2 O 2 S 2 α 2 in the new ℓ 2 eluder technique, we can obtain

where ς = 2log ( 1 + 4 A 2 B 2 O 2 S 2 α 2 ) . By Jensen's inequality, the last term above is further relaxed as { O 2 ABς [ OABα 2 S T + T ∑ t =1 t -1 ∑ ι =1 E τ h -1 ∼ P π ι ,ν ι f ∗ ,h -1 ( ∑ o h ,a h ,b h ∥ ( M t h ( o h , a h , b h ) -M h ( o h , a h , b h ) ) ¯ q ( τ h -1 ) ∥ 1 · σ ι ( o h , a h , b h ; τ h -1 ) ) 2 ] } 1 2 .

Then, invoking Lemma 6, we obtain

Thus, by invoking (14) and (15), we get the regret bound by

Since the above derivation is for any policy pair ( π t , ν t ) , we can show that α -revealing POMG is subsumed by the classes of both self-play and adversarial GEC class with a bounded d GEC . We finally prove

where ς = 2log ( 1 + 4 A 2 B 2 O 2 S 2 α 2 ) . This completes the proof.

B.4.1 Lemmas for Proof of Proposition 6

Here, we present and prove all the lemmas used for the proof of Proposition 6. All the lemmas presented here follow the notations of Proposition 6.

Lemma 3. In an α -revealing POMG, for any policy pair ( π, ν ) , any vector x ∈ R O , and any τ h -1 , we have

Proof. We first obtain

where e i ∈ R S denotes the standard basis vector whose i -th element is 1 and 0 for others. Note that ∑ τ h : H | m ( τ h : H ) O h e i | σ ( τ h : H ; τ h -1 ) = ∑ τ h : H Pr σ ( τ h : H | τ 1: h -1 , s i ) = 1 . Thus we obtain

which completes the proof.

Lemma 4. In an α -revealing POMG, for any policy pair ( π, ν ) and any x ∈ R O , we have

Proof. We first obtain

where ˜ e j is the j -th standard basis vector in R O and e i is the i -th standard basis vector in R S . Note that we have

Combining the above result, we obtain

which completes the proof.

Lemma 5. The value difference in a POMG can be decomposed as

Proof. Denoting r ( τ H ) as the sum of rewards from h = 1 to H on τ H , we obtain

Using the triangle inequality, we further obtain

where m ( τ h +1: H ) = M H : h +1 ( τ h +1: H ) and q ( τ h -1 ) = M h -1:1 ( τ h -1 ) q 0 . Note that Lemma 3 shows that

and

Thus we obtain

which completes the proof.

The next lemma gives the bound of training error by Hellinger distances.

Lemma 6. For any model f t and policy σ ι , we have

where ≲ omits absolute constants.

Proof. By using ( a + b ) 2 ≤ 2 a 2 +2 b 2 and the triangle inequality, we decompose the LHS into two parts to bound them separately. We have

For Term(i), we equivalently rewrite this term as

We first consider the case when h < H . According to the definitions of ¯ q t h and ¯ q h , we obtain

where Pr π ι ,ν ι f t ( τ h -1 , o h ) denotes the probability of ( τ h -1 , o h ) if the actions are taken following ( π ι , ν ι ) and the observations follow the omission and transition process in the model f t . The first inequality is by the fact that the L 1 difference of two marginal distributions is no more than the L 1 difference of two uniformed distributions, and the second inequality is due to ∥ P -Q ∥ 2 1 ≤ 8 D 2 He ( P, Q ) . Similarly, when h = H , Term(i) can be bounded as

For Term(ii), Lemma 4 is used to give an upper bound. Specifically, we obtain that

Combining the above results completes the proof of this lemma.

Lemma 7. For any ( o, a, b ) ∈ O × A × B , we have

Proof. We prove the former statement first. By noting that the O † h -1 ¯ q ( τ h -1 ) = [Pr( s i | τ h -1 )] S i =1 and σ t ( o h , a h , b h ; τ h -1 ) ≤ 1 , we obtain

Therefore the first statement in Lemma 7 is proven. To prove the second statement, we write

where the second inequality is due to ∥ · ∥ 2 ≤ ∥ · ∥ 1 , and ˜ e i stands for the i -th standard basis vector in R O . Then we apply Lemma 4 to obtain

where the second inequality is by ∥ Ax ∥ 1 ≤ ∥ A ∥ 1 ∥ x ∥ 1 . The proof is completed.

B.5 Decodable POMG

In this section, we propose a new class of POMGs, dubbed decodable POMG, by generalizing decodable POMDPs [23, 22] from the single-agent setting to the multi-agent setting.

Definition 9 (Decodable POMG) . We say a POMG is a decodable POMG if an unknown decoder function ϕ h exists, which recovers the state at step h from the observation at step h . We have for any 1 ≤ h ≤ H that

Given the decoder, we can define the transition from observation to observation as follows,

Our next proposition will show that the class of decodable POMGs is subsumed by the class of MGs with low self-play and adversarial GEC.

Proposition 7 (Decodable POMG ⊂ Low Self-Play/Adversarial GEC) . For a decodable POMG as defined in Definition 9, with the loss ℓ ( f, ξ h ) defined in Definition 3 for POMGs, we have

Decodable POMG ⊂ POMG with low self-play and adversarial GEC

with d GEC satisfying

Proof. The proof of this proposition is largely the same as the proof of Proposition 6, except that the coefficient before ∥ x ∥ 1 in Lemma 3 and 4 is changed, from √ S α to 1 , as is proved in Lemma 8 and Lemma 9. Following the proof of Proposition 6 and using Lemmas 8 and 9 yields our result.

B.5.1 Lemmas for Poof of Proposition 7

We present the lemmas for Proposition 7. These lemmas are analogous to Lemma 3 and Lemma 4. For convenience in analysis, we define some notations as follows.

Similar to the case of α -revealing POMG in Appendix B.4, we define q 0 , τ h : h ′ , q ( τ h ) and ¯ q ( τ h ) . We now define a different observable operator as follows,

where e o ∈ R O is the basis vector with only the o -th entry being 1 . With these definitions, one can show that

Wedefine M h ′ : h ( o h : h ′ , a h : h ′ , b h : h ′ ) := M h ′ ( o h ′ , a h ′ , b h ′ ) M h ′ ( o h ′ -1 , a h ′ -1 , b h ′ -1 ) . . . M h ( o h , a h , b h ) and rewrite M H : h +1 ( o h +1: H , a h +1: H , b h +1: H ) as m ( o h +1: H , a h +1: H , b h +1: H ) to emphasize that m ( o h +1: H , a h +1: H , b h +1: H ) is a vector since M H ( o H , a H , b H ) is a vector.

Lemma 8. For any x ∈ R O , any policy pair ( π, ν ) , and any τ h -1 , we have

Proof. We can first bound the LHS as

where ˜ e i is the i -th basis vector of the space R O . Since ∑ τ h +1: H Pr( τ h +1: H | o i ) σ ( τ h : H ; τ h -1 ) ≤ 1 , we further obtain

This completes the proof.

Lemma 9. For any x ∈ R O , policy pair ( π, ν ) and τ h -1 , we have

Proof. We can first show that

Since ∑ o ∈O ,a ∈A ,b ∈B Pr( o i | o, a, b ) σ ( o, a, b ; τ h -1 ) ≤ 1 , we further obtain

This concludes the proof.

C Computation of ω ( β, p 0 )

In this section, we discuss the upper bound of the quantity ω ( β, p 0 ) for both FOMGs and POMGs. For FOMGs, the analysis can be adopted from Lemma 2 in [2]. For POMGs, we give the detailed analysis to show the bound of ω ( β, p 0 ) , which is the first proof for the partially observable setting.

C.1 Fully Observable Markov Game

We can adopt the result from Lemma 2 in [2] to give a bound for ω ( β, p 0 ) in the context of fully observable Markov games.

Proposition 8 (Lemma 2 of [2]) . When F is finite, p 0 is a uniform distribution over F , then ω ( β, p 0 ) ≤ log |F| . When F is infinite, suppose a transition kernel P 0 satisfies P f ∗ ≪ P 0 and ∥ ∥ ∥ d P f ∗ d P 0 ∥ ∥ ∥ ∞ ≤ B , then for ϵ ≤ 2 / 3 and B ≥ log(6 B 2 /ϵ ) , there exists a prior p 0 on F such that

where ν = ϵ/ (6 log(6 B 2 /ϵ )) and N ( ϵ ) stands for the ϵ -covering number w.r.t. the distance

To apply Proposition 8 in FOMGs, we can further show that | D 2 He ( P, R ) -D 2 He ( Q,R ) | ≤ √ 2 D 2 He ( P, R ) | ≤ √ 2 2 ∥ P -Q ∥ 1 for distributions P, Q, R , so that the covering number under the distance d can be bounded by the covering number w.r.t. the ℓ 1 distance denoted as N 1 ( ϵ ) , i.e.,

where N 1 ( ϵ ) is defined w.r.t. the distance

The covering number under the ℓ 1 distance is more common and is readily applicable to many problems.

Taking linear mixture MGs for an instance (defined in Definition 5), we calculate ω (4 HT,p 0 ) , which is the term quantifying the coverage of the initial sampling distribution in our theorems. We first obtain that

where the first inequality is by Cauchy-Schwarz, and the second inequality is due to ∫ s ′ ∈S ∥ ϕ ( s ′ , s, a, b ) ∥ 2 d s ′ ≤ 1 . Since the model space is { θ : sup h ∥ θ h ∥ 2 ≤ B } , under the ℓ 2 distance measure of

according to (16) and the covering number for an Euclidean ball. Combining this result (17) with Proposition 8 together, we can show that there exists a prior distribution p 0 with a sufficiently large T such that

C.2 Partially Observable Markov Game

In this subsection, we prove the bound of ω ( β, p 0 ) for the partially observable setting, inspired by the proof of Proposition 8 (Lemma 2 in [2]).

Proposition 9. When F is finite, p 0 is a uniform distribution over F , then ω ( β, p 0 ) ≤ log |F| . When F is infinite, suppose a distribution P 0 satisfies that for any policy pair ( π, ν ) , P π,ν f ∗ ,H ≪ P π,ν 0 and ∥ d P π,ν f ∗ ,H / d P π,ν 0 ∥ ∞ ≤ B , where B ≥ 1 . Then for ϵ ≤ 2 / 3 and B ≥ log(6 B 2 /ϵ ) , there exists a prior p 0 on F such that

where ν = ϵ/ (6 log(6 B 2 /ϵ )) and N ( · ) is the covering number w.r.t. the distance

The assumption in the theorem above covers the case when S is finite, where we choose P 0 to be uniform on S regardless of the policy. We also note that

which does not depend on the joint policy σ = ( π, ν ) .

To apply the above proposition in POMGs, by the relation between different distances: | D 2 He ( P, R ) -D 2 He ( Q,R ) | ≤ √ 2 D 2 He ( P, R ) | ≤ √ 2 2 ∥ P -Q ∥ 1 for distributions P, Q, R , we can show the covering number under the distance d can be bounded by the covering number w.r.t. the ℓ 1 distance denoted as N 1 ( ϵ ) , i.e.,

where N 1 ( ϵ ) is defined w.r.t. the distance

Such a covering number under ℓ 1 distance is analyzed in the work [78], generalizing whose results gives that POMGs with different structures admit a log-covering number log N 1 ( ϵ ) = ploy( |O| , |A| , |B| , |S| , H, log(1 /ϵ )) . We refer readers to [78] for detailed calculation of logcovering numbers under d 1 ( f, f ′ ) . Therefore, we can eventually show that ω (4 HT,p 0 ) = ploy( |O| , |A| , |B| , |S| , H, log( HT )) . Next, we show the detailed proof for Proposition 9.

we have

Proof. When F is finite and p 0 is the uniform distribution on F , the proof is straightforward as we have ω ( β, p 0 ) ≤ βε +log |F| for any ε ≥ 0 and a uniform distribution p 0 . Setting ε to approach 0 + completes the proof.

When F is infinite, we start by setting up a γ -covering C ( γ ) ⊂ F w.r.t. the distance d ( f, f ′ ) = sup π,ν ∣ ∣ ∣ D 2 He ( P π,ν f,H , P π,ν f ∗ ,H ) -D 2 He ( P π,ν f ′ ,H , P π,ν f ∗ ,H ) ∣ ∣ ∣ , where γ > 0 is a variable to be specified. Since f ∗ is covered, an ˜ f ∈ C ( γ ) satisfies d ( ˜ f, f ∗ ) = sup π,ν D 2 He ( P π,ν ˜ f,H , P π,ν f ∗ ,H ) ≤ γ . We further define C ν ( γ ) := { ν P 0 + (1 -ν ) P f | f ∈ C ( γ ) } and also P f ′ := ν P 0 + (1 -ν ) P ˜ f . We note that sup π,ν ∥ d P π,ν f ∗ ,H d P π,ν f ′ ,H ∥ ∞ ≤ B ν . Then, we obtain

where the first inequality uses Jensen's inequality and the second inequality is by sup π,ν D 2 He ( P π,ν ˜ f,H , P π,ν f ∗ ,H ) ≤ γ and 0 ≤ ∫ √ d P d Q ≤ 1 . To connect to the definition of F ( ε ) , we further invoke Theorem 9 from [55] and obtain

for any policy pair ( π, ν ) , where ζ ( b ) ≤ max { 1 , b log b (1 - √ b ) 2 } for b > 1 . Plugging the above inequalities together, we obtain

It remains to find a proper choice of γ and ν to obtain ζ ( B/ν )( γ + ν ) ≤ ε . We choose ν = ε/ ( 6 log(6 B 2 /ε ) ) and γ = ε/ ( 6 log( B/ν ) ) . Given the condition B ≥ log(6 B 2 /ε ) , we have

Given ε ≤ 2 / 3 and B ≥ 1 , we obtain so that

namely B/ν ≥ 9 . We note that when b > 9 , it holds that

Thus we have ζ ( B/ν ) ≤ 3 log( B/ν ) . Finally, we obtain

where the first inequality uses (19), the first equation uses the choice of γ , and the second inequality uses ζ ( B/ν ) ≤ 3 log( B/ν ) . Choosing p 0 to be a uniform distribution on C ν ( γ ) completes the proof.

D Technical Lemmas for Main Theorems

In this section, we first provide several important supporting lemmas used in the proofs of Theorems 1 and 2. We then present detailed proofs for these lemmas.

D.1 Lemmas

Lemma 10. Let υ be any probability distribution over f ∈ F where F is an arbitrary set. Then, E f ∼ υ ( · ) [ G ( f ) + log υ ( f )] is minimized at υ ( f ) ∝ exp( -G ( f )) .

Proof. This lemma is a corollary of Gibbs variational principle. For the detailed proof of this lemma, we refer the readers to the proof of Lemma 4.10 in [64]. This completes the proof.

The above lemma states that υ ( f ) in the above-described form solves the minimization problem min υ ∈ ∆( F ) E f ∼ υ [ G ( f ) + log υ ( f )] , which helps to understand the design of the posterior sampling steps in our proposed algorithms. This lemma is also used in the proofs of the following two lemmas.

The following two lemmas provide the upper bounds for the expectation of the Hellinger distance by the likelihood functions defined in (1) and (2) for FOMGs and POMGs respectively.

Lemma 11. Under the FOMG setting, for any t ≥ 1 , let Z t be the system randomness history up to the t -th episode, p t ( ·| Z t -1 ) be any posterior distribution over the function class F with p 0 ( · ) denoting an initial distribution, and ( π t , ν t ) be any Markovian policy pair for Player 1 and Player 2 depending on f t ∼ p t . Suppose that ( s t h , a t h , b t h , s t h +1 ) is a data point sampled independently by executing the policy pair ( π t , ν t ) to the h -th step of the t -th episode. If we define L t h ( f ) := η log P f,h ( s t h +1 | s t h , a t h , b t h ) with η = 1 / 2 as in (1) , we have the following relation

where E ( π t ,ν t ,h ) denotes taking an expectation over the data ( s t h , a t h , b t h ) sampled following the policy pair ( π t , ν t ) and the true model P f ∗ up to the h -th step at any t .

Proof. Please see Appendix D.2 for a detailed proof.

Lemma 12. Under the POMG setting, for any t ≥ 1 , let Z t be the system randomness history up to the t -th episode, p t ( ·| Z t -1 ) be any posterior distribution over the function class F with p 0 ( · ) denoting an initial distribution, and ( π t , ν t ) be any general history-dependent policy pair for Player 1 and Player 2 depending on f t ∼ p t . Suppose that τ t h = ( o t 1 , a t 1 , b t 1 . . . , o t h , a t h , b t h ) is a data point sampled independently by executing the policy pair ( π t , ν t ) to the h -th step of the t -th episode. If we define L t h ( f ) := η log P f ( τ t h ) with η = 1 / 2 as in (2) , we have the following relation

where P π,ν f,h denotes the distribution for τ h = ( o 1 , a 1 , b 1 , . . . , o h , a h , b h ) under the model θ f and the policy pair ( π, ν ) up to the h -th step.

Proof. Please see Appendix D.3 for a detailed proof.

The next lemma shows that when the model is sufficiently close to f ∗ with the distances employed in Definition 2, the following value differences are small enough under both FOMG and POMG settings.

Lemma 13. If the model f satisfies sup h,s,a,b KL 1 2 ( P f ∗ ,h ( · | s, a, b ) ∥ P f,h ( · | s, a, b )) ≤ ε for FOMGs and sup π,ν KL 1 2 ( P π,ν f ∗ ,H ∥ P π,ν f,H ) ≤ ε for POMGs, we have that their corresponding value function satisfies

Proof. Please see Appendix D.4 for detailed proof.

Finally, we show that when the model is sufficiently close to f ∗ , we will obtain that the following likelihood function difference is small under both FOMG and POMG settings.

Lemma 14. If the model f satisfies sup h,s,a,b KL 1 2 ( P f ∗ ,h ( · | s, a, b ) ∥ P f,h ( · | s, a, b )) ≤ ε for FOMGs and sup π,ν KL 1 2 ( P π,ν f ∗ ,H ∥ P π,ν f,H ) ≤ ε for POMGs, we have that their corresponding likelihood function defined in (1) and (2) satisfies

where the expectation is taken with respect to the randomness in L t h .

Proof. Please see Appendix D.5 for detailed proof.

D.2 Proof of Lemma 11

Proof. The proof of Lemma 11 can be viewed as a multi-agent generalization of the proof for Lemma E.5 in [83]. We start our proof by first considering the following equality

Next, we lower bound RHS of the above equality. We define

where E ( π ι ,ν ι , P f ∗ ,h ) denotes taking an expectation over the data ( s ι h , a ι h , b ι h , s ι h +1 ) sampled following the policy pair ( π ι , ν ι ) and the true model P f ∗ to the h -th step with s ι h +1 ∼ P f ∗ ,h ( ·| s ι h , a ι h , b ι h ) at round ι . Then, we will show that E Z t -1 [exp( ∑ H h =1 ∑ t -1 ι =1 ˜ L ι h ( f ))] = 1 by induction, following from [80]. Suppose that for any k , we have at k -1 that E Z k -1 [exp( ∑ H h =1 ∑ k -1 ι =1 ˜ L ι h ( f ))] = 1 . Then, at k , we have

where the second equality uses the fact that the data is sampled independently, the third equality is due to E ( π k ,ν k , P f ∗ ,h ) exp( ˜ L k h ( f )) = E ( π k ,ν k , P f ∗ ,h ) exp( L k h ( f )) / E ( π k ,ν k , P f ∗ ,h ) [exp( L k h ( f ))] = 1 by the definition of ˜ L k h ( f ) , and the last equality is by E Z k -1 [exp( ∑ H h =1 ∑ k -1 ι =1 ˜ L ι h ( f ))] = 1 in the above assumption. Moreover, for k = 1 , we have a trivial result that E Z 1 [exp( ∑ H h =1 ˜ L 1 h ( f ))] =

E f 1 ∼ p 1 ∏ H h =1 E ( π 1 ,ν 1 , P f ∗ ,h ) exp( ˜ L 1 h ( f )) = 1 . Consequently, we conclude that for any k , the above equality holds. Then, when k = t -1 , we have

Furthermore, we have

where the last inequality is by Jensen's inequality and the last equality is due to (20). For the first equality, we use the fact that the following distribution is the minimizer

according to Lemma 10, such that plugging in the above distribution leads to the first equality. Thus, according to the definitions of ˜ L t h and L t h , we have

Moreover, to further lower bound the RHS of the above inequality, by the inequality that log x ≤ x -1 and the setting η = 1 2 , we have

where the last equality is by D 2 He ( P, Q ) = 1 2 ∫ ( √ d P ( x ) -√ d Q ( x )) 2 = 1 -∫ √ d P ( x )d Q ( x ) . Here E ( π ι ,ν ι ,h ) denotes taking expectation over ( s ι h , a ι h , b ι h ) sampled following the policy pair ( π ι , ν ι ) and the true model P f ∗ to the h -th step but without the next state s ι h +1 generated by P f ∗ ,h ( ·| s ι h , a ι h , b ι h ) at round ι . In the sequel, combining the above results, we have

This completes the proof.

D.3 Proof of Lemma 12

Proof. The proof of Lemma 12 is similar to the proof of Lemma 11. We will give a brief description of the main steps for our proof of Lemma 12. We start our proof by considering the following equality

Next, we lower bound RHS of the above equality. We define

where E ( π ι ,ν ι ,h ) denotes taking an expectation over the data τ ι h sampled following the policy pair ( π ι , ν ι ) and the true model θ f ∗ to the h -th step at round ι . Then, we can show that

following a similar derivation as (20) in the proof of Lemma 11. With setting η = 1 2 , this result further leads to

In the sequel, combining the above results, we have

This completes the proof.

D.4 Proof of Lemma 13

Proof. We first prove the upper bound of | V π,ν f ∗ -V π,ν f | for any ( π, ν ) and f under FOMG and POMG settings separately.

For FOMGs, we let V π,ν f ∗ = V π,ν f ∗ and V π,ν f = V π,ν f . Then, according to the Bellman equation that Q π,ν f,h ( s, a, b ) = r h ( s, a, b ) + ⟨ P f,h ( ·| s, a, b ) , V π,ν f,h +1 ( · ) ⟩ and also V π,ν f,h ( s, a, b ) = E a ∼ π h ( ·| s ) ,b ∼ ν h ( ·| s ) [ Q π,ν f,h ( s, a, b )] , we have

where the third inequality is by Pinsker's inequality and the last inequality is by the assumption of this lemma. On the other hand, we can show that the above result also holds for POMGs. Then, for this setting, we have

To unify our results, we enlarge the above bound by a factor of H and eventually obtain that

for both FOMGs and POMGs. Moreover, by the properties of the operators min , max , and maxmin , we have

This concludes the proof of this lemma.

D.5 Proof of Lemma 14

Proof. We first prove the upper bound under the FOMG setting. By (1), we know that

which further leads to

where we use the definition of KL divergence in the first inequality and the last inequality is by the condition that sup h,s,a,b KL 1 2 ( P f ∗ ,h ( · | s, a, b ) ∥ P f,h ( · | s, a, b )) ≤ ε .

We next prove the upper bound under the POMG setting. According to (2), we have

which further leads to

where we use the definition of KL divergence and the relation between P f,h and P π,ν f,h . Now we consider to lower bound KL( P π t ,ν t f ∗ ,H ∥ P π t ,ν t f,H ) . We have

where Pr π t ,ν t f denote the probability under the model f and the policy pair π t , ν t , the third equality is by the definition of KL divergence, and the inequality is by the non-negativity of KL divergence. Therefore, combining the above results and the condition sup π,ν KL 1 2 ( P π,ν f ∗ ,H ∥ P π,ν f,H ) ≤ ε , we obtain

This concludes the proof of this lemma.

E Proofs for Algorithms 1 and 3

In this section, we provide the detailed proof for Theorem 1. In particular, our proof is compatible with both the FOMG and the POMG settings. Thus, we unify both setups in a single theorem and present their proof together in this section.

To characterize the value difference under models f and f ∗ we define the following terms, which are

In addition, we define the difference of likelihood functions at step h of time t as

Then, the updating rules of the posterior distribution in Algorithm 1 have the following equivalent forms

and

since here adding or subtracting terms irrelevant to f , i.e., V π, ∗ f ∗ , V ∗ f ∗ , and L t h ( f ∗ ) , within the power of all exponential terms will not change the posterior distribution of f .

To learning a sublinear upper bound for the expected value of the total regret, i.e., E [Reg sp ( T )] = E ∑ T t =1 [ V ∗ ,ν t f ∗ -V π t , ∗ f ∗ ] for the self-play setting, we need to execute both Algorithm 1 and 3, which are two symmetric algorithms. We can decompose Reg sp ( T ) as

and where we define

In fact, executing Algorithm 1 leads to a low regret upper bound for E [Reg sp 1 ( T )] while running Algorithm 3 incurs a low regret upper bound for E [Reg sp 2 ( T )] . Since the two algorithms are symmetric, the derivation of their respective regret bounds is thus similar. In the following subsections, we present the proofs of Proposition 1, Proposition 2, and Theorem 1 sequentially.

E.1 Proof of Proposition 1

Proof. We start our proof by first decomposing the regret as follows

Our goal is to give the upper bound for the expected value of the total regret, which is E [Reg sp 1 ( T )] . Thus, we need to derive the upper bounds of E [ Term(i) ] and E [ Term(ii) ] separately. Intuitively, according to our analysis below, E [ Term(i) ] can be viewed as the regret incurred by the updating rules for the main player in Line 3 and Line 4 of Algorithm 1, and E [ Term(ii) ] is associated with the exploiter's updating rules in Line 5 and Line 6 of Algorithm 1,

Bound E [ Term(i) ] . To bound E [ Term(i) ] , we give the following decomposition

where according to Algorithm 1, we have ( π t , ν t ) = argmax π argmin ν V π,ν f t , i.e., ( π t , ν t ) is the NE of V π,ν f t . Thus, the second equality is by (22), and the inequality is due to

According to the condition (1) in Definition 1 for self-play GEC and the updating rules in Algorithm 1, setting the exploration policy pair as σ t = ( π t , ν t ) , and ρ t = f t for Definition 1, we have

where the second inequality is due to √ xy ≤ 1 γ 1 x 2 + γ 1 4 y 2 . Combining the above results, we have

We need to further bound the RHS of the above equality. Note that for FOMGs and POMGs, we have different definitions of ℓ ( f, ξ h ) . Specifically, according to our definition of ℓ ( f, ξ ι h ) , for FOMGs, we define

and for POMGs, we have the relation

On the other hand, for the two MG settings, we also define L t h ( f ) as in (1) and (2). According to Lemmas 11 and 12, unifying their results, we can obtain

which further gives

where ∆ L t h ( f ) := L t h ( f ) -L t h ( f ∗ ) is defined in (23). Note that in the above analysis, we slightly abuse the notation of π t and ν t , which denote the Markovian policies for FOMG and historydependent policies for POMGs. Therefore, according to Lemma 10, we obtain that the distribution

minimizes the last term in the above inequality. As discussed in (24), this distribution is an equivalent form of the posterior distribution updating rule for p t adopted in Line 3 of Algorithm 1. This result implies that

By Definition 2, further letting ˜ p := p 0 ( f ) · 1 ( f ∈ F ( ε )) /p 0 ( F ( ε )) , we have

where the second inequality is by Lemma 13 and Lemma 14 as well as ε ≤ 1 , and the last inequality is by our setting that η = 1 2 and Hγ 1 ≤ T . Therefore, by combining the above results, we obtain

where we set HT > d GEC , ϵ = 1 / √ HT for self-play GEC in Definition 1, and γ 1 = 2 √ ω (4 HT,p 0 ) T/d GEC . Since we set ϵ = 1 / √ HT , the self-play GEC d GEC is defined associated with ϵ = 1 / √ HT by Definition 1. However, our analysis in Appendix B shows that this setting only introduces log T factors in d GEC in the worst case.

Bound E [ Term(ii) ] . Next, we consider to bound E [ Term(ii) ] . We start by decomposing Term(ii) as follows,

where the second equality is by (21) and the fact that ν t = argmin ν V π t ,ν f t .

In addition, according to Definition 1 and the updating rules in Algorithm 1, setting σ t = ( π t , ν t ) , ρ t = f t , we have

where the second inequality is due to √ xy ≤ 1 γ 2 x 2 + γ 2 4 y 2 . Thus, we obtain that

We further bound the RHS of the above equality. By the definitions of ℓ ( f, ξ h ) for FOMGs and POMGs and also the definitions of L t h ( f ) as in (1) and (2), according to Lemmas 11 and 12, we can obtain

where the expectation for f t ∼ p t exists due to that π t is computed based on f t , and also according to Lemma 10 and (25), the last equality can be achieved by that the updating rule for the distribution q t in Algorithm 1, i.e., equivalently

can minimize E f ∼ q [ γ 2 ∆ V π t , ∗ f -∑ H h =1 ∑ t -1 ι =1 ∆ L ι h ( f ) + log q ( f ) q 0 ( f ) ] .

By Definition 2, further letting ˜ q := q 0 ( f ) · 1 ( f ∈ F ( ε )) /q 0 ( F ( ε )) with ε ≤ 1 , we have

where the second inequality is by Lemma 13 and Lemma 14 as well as ε ≤ 1 , and the last inequality is by our setting that η = 1 2 and Hγ 2 ≤ T . Therefore, combining the above results, we have

where we set HT > d GEC , ϵ = 1 / √ HT for self-play GEC in Definition 1, and γ 2 = 2 √ ω (4 HT,q 0 ) T/d GEC .

Combining Results. Finally, combining the results in (26) and (27), we eventually obtain

This completes the proof.

E.2 Proof of Proposition 2

Proof. Due to the symmetry of Algorithms 1 and 3, we can similarly bound E [Reg sp 2 ( T )] in the way of bounding E [Reg sp 2 ( T )] . Therefore, in this subsection, we only present the main steps for the proof. Specifically, by Algorithm 3, we have a decomposition as

Bound E [ Term(iii) ] . To bound E [ Term(iii) ] , we give the following decomposition

where according to Algorithm 3, we have ( π t , ν t ) = argmax π argmin ν V π,ν f t , which thus leads to

According to the condition (2) in Definition 1 for self-play GEC and the updating rules in Algorithm 3, setting the exploration policy pair as σ t = ( π t , ν t ) , we have

Weneed to further bound the RHS of the above equality. For FOMGs and POMGs, by their definitions of ℓ ( f, ξ h ) as well as L t h ( f ) , according to Lemmas 11 and 12, we obtain

where ∆ L t h ( f ) := L t h ( f ) -L t h ( f ∗ ) is defined in (23). According to Lemma 10, we obtain that the distribution p t defined in Algorithm 3, equivalently

minimizes the last term in the above inequality. This result implies that

By Definition 2, further letting ˜ p := p 0 ( f ) · 1 ( f ∈ F ( ε )) /p 0 ( F ( ε )) , we have

where the second inequality is by Lemma 13 and Lemma 14 as well as ε ≤ 1 , and the last inequality is by our setting that η = 1 2 and Hγ 1 ≤ T . Therefore, by combining the above results, we obtain

where we set HT > d GEC , ϵ = 1 / HT for self-play GEC in Definition 1, and γ 1 = 2 √ ω (4 HT,p 0 ) T/d GEC .

Bound E [ Term(iv) ] . Next, we consider to bound E [ Term(iv) ] as follows,

where the second equality is by the fact that π t = argmax π V π,ν t f t according to the updating rule. By Definition 1 and the updating rules in Algorithm 3, we have

By the definitions of ℓ ( f, ξ h ) for FOMGs and POMGs and also the definitions of L t h ( f ) as in (1) and (2), we have

where the expectation for f t ∼ p t exists due to that ν t is computed based on f t , and also according to Lemma 10, the last equality can be achieved by that the updating rule for the distribution q t in Algorithm 3, i.e., equivalently

can minimize E f ∼ q [ -γ 2 ∆ V ∗ π t ,ν t f -∑ H h =1 ∑ t -1 ι =1 ∆ L ι h ( f ) + log q ( f ) q 0 ( f ) ] . By Definition 2, further letting ˜ q := q 0 ( f ) · 1 ( f ∈ F ( ε )) /q 0 ( F ( ε )) with ε ≤ 1 , we have

where the second inequality is by Lemma 13 and Lemma 14 as well as ε ≤ 1 , and the last inequality is by our setting that η = 1 2 and Hγ 2 ≤ T . Therefore, combining the above results, we have

where we set HT > d GEC , ϵ = 1 / √ HT for self-play GEC in Definition 1, and γ 2 = 2 √ ω (4 HT,q 0 ) T/d GEC .

Combining Results. Finally, combining the results in (28) and (29), we eventually obtain

This completes the proof.

E.3 Proof of Theorem 1

Proof. The proof of Theorem 1 is immediate by the relation that Reg sp ( T ) = Reg sp 1 ( T )+Reg sp 2 ( T ) . Thus, according to the above proofs of Proposition 1 and Proposition 2, under the conditions in both propositions, we have that

This completes the proof.

F Proofs for Algorithm 2

In this section, we provide the detailed proof for Theorem 2 and unify the proofs for both the FOMG and POMG settings together.

We recall that the value difference and the likelihood function difference are already defined in (22) and (23), which are

Then, similar to (24), the updating rules of posterior distribution in Algorithm 2 have the following equivalent form as

since adding or subtracting terms irrelevant to f , i.e., V ∗ f ∗ and L t h ( f ∗ ) , within the power of all exponential terms will not change the posterior distribution of f .

F.1 Proof of Theorem 2

Proof. To bound the expected value of the adversarial regret Reg adv ( T ) , i.e., E [Reg adv ( T )] , we first decompose the regret Reg adv w as follows

Furthermore, for the term ∑ T t =1 [ V ∗ f t -V π t ,ν t f ∗ ] , we have

where the above result is by ( π t , ν t ) = argmax π argmin ν V π,ν f t , i.e., ( π t , ν t ) is the NE of V π,ν f t , according to Algorithm 2, and also due the relation that

Thus, we have

According to Definition 3 and the updating rules in Algorithm 2, setting σ t exp = ( π t , ν t ) , and ℓ ( f t , ξ ι h ) = D 2 He ( P f t ,h ( ·| s ι h , a ι h , b ι h ) , P f ∗ ,h ( ·| s ι h , a ι h , b ι h )) for Definition 1, we have

where the second inequality is due to √ xy ≤ 1 γ x 2 + γ 4 y 2 . Combining the above results, we have

For FOMGs and POMGs, we have different definitions of ℓ ( f, ξ h ) . According to (3), for FOMGs, we define

and for POMGs, the definition of ℓ ensures

For different MG settings, we also define L t h ( f ) as in (1) and (2) which are

Then, combining Lemmas 11 and 12, we know that the following result holds for both FOMGs and POMGs,

which further leads to

where ∆ L ι h ( f ) = L ι h ( f ) -L t h ( f ∗ ) as in (23). Therefore, according to Lemma 10 and (30), we know that the distribution

minimizes the last term in the above inequality, which is the posterior distribution updating rule for p t adopted in Algorithm 2. This result implies that

For the adversarial setting, we define F ( ε ) as in Definition 2 similar to the definition of F ( ε ) in the self-play setting. Further letting ˜ p := p 0 ( f ) · 1 ( f ∈ F ( ε )) /p 0 ( F ( ε )) with ε ≤ 1 and ω be associated with F ( ε ) , we have

where the second inequality is by Lemma 13 and Lemma 14 as well as ε ≤ 1 , and the last inequality is by our setting that η = 1 2 and γ ≤ T . Therefore, by combining the above results, we obtain

where we set HT > d GEC , ϵ = 1 / √ HT for adversarial GEC in Definition 3, and γ = 2 √ ω (4 HT,p 0 ) T/d GEC . Note that since we set ϵ = 1 / √ HT , the adversarial GEC d GEC is now calculated associated with ϵ = 1 / √ HT by Definition 3. Further by our analysis in Appendix B, we know that this setting only introduces log T factors in d GEC in the worst case. This concludes the proof.

G Other Supporting Lemmas

Lemma 15 (Elliptical Potential Lemma in [1], Lemma 11) . Suppose { ϕ t } t ≥ 0 is a sequence in R d satisfying ∥ ϕ t ∥ 2 ≤ 1 , and Λ 0 is a positive definite d × d matrix. Let Λ t = Λ 0 + ∑ t ι =1 ϕ ι ϕ ⊤ ι . Then, the following inequalities hold

Lemma 16 (Estimation of Elliptical Potential [38]) . Given λ > 0 and { ϕ t } t ≥ 0 with ∥ ϕ t ∥ 2 ≤ 1 , denoting Λ t = λ I + ∑ t ι =1 ϕ ι ϕ ⊤ ι , then ϕ ⊤ ι (Λ ι -1 ) -1 ϕ ι is upper bounded by

Lemma 17 ( ℓ 2 Eluder Technique [13, 83]) . Suppose that { w t,j ∈ R d } ( t,j ) ∈ [ T ] × [ J ] , { x t,i ∈ R d ( t,i ) ∈ [ T ] × [ I ] } ( t,i ) , and distributions { p t ∈ ∆ [ I ] } [ T ] satisfy

for any 1 ≤ t ≤ T . Then, for R > 0 , we have

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a zero-sum Markov game with unknown dynamics in the case of full and partial observations. The authors propose algorithms for finding a Nash equilibrium in the games, in which, at each iteration, virtual games with dynamics sampled from certain distributions are solved. The paper's main result is theoretical and consists of estimates for the rate of the algorithms' convergence.\n\nSTRENGTHS:\nThe paper is aimed at solving an important problem. It is well structured and written in clear mathematical language.\n\nWEAKNESSES:\nUnfortunately, as a non-specialist, it is rather difficult for me to assess the significance of the obtained theoretical results. It is not entirely clear what useful conclusion the reader can draw from them concerning practical methods for solving zero-sum Markov games. The proposed algorithms seem very abstract since it is difficult to calculate the indicated distributions in experimental tasks. If I'm wrong and this is feasible, an example confirming this would significantly strengthen the paper.\n\nOne more thing I doubt is the fact that in the algorithms, the authors apparently assume that a Nash equilibrium in Markov games exists. This is true if we consider games with an infinite horizon, but I'm not sure if this is true for games with a fixed number of steps $H$. Usually, in such games, it is assumed that the policy depends not only on the state but also on the step number. Otherwise, the existence of a Nash equilibrium is not obvious to me, and I would welcome a reference to this fact.\n\nThe paper also contains typos:\n\n178 - $V^*$ instead of $V_1^*$.\n189 - $P^{\\pi,\\nu}_f$ instead of $P^{\\pi,\\nu}_h$\n246 - “begin” instead of “bening”\n\nQUESTIONS:\nOn line 151, the authors introduce the reward function $r_h(o,a,b)$. It seems a bit exotic that it depends on an observation $o$ rather than on a state $s$. Is it important for the obtained results?\n\nLIMITATIONS:\nThe paper does not have potentially negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper investigates posterior sampling algorithms for competitive reinforcement learning (RL) with general function approximations in zero-sum Markov games (MGs). It introduces complexity measures for function approximation and proposes model-based self-play and adversarial posterior sampling methods to learn Nash equilibrium in partially observable states. The algorithms provide low regret bounds and can be applied to various tractable zero-sum MG classes in both fully observable and partially observable settings.

STRENGTHS:
I think this work really pushes the Multiagent+PORL community research efforts further by answering:

> can we design a generic posterior sampling algorithm for MGRL in the context of function approximation?

The main contribution of model-based posterior sampling algorithm equipped with rigorous analyses is worthy for publication at NeurIPS.

WEAKNESSES:
I have only minor weaknesses for this work as follows:

1. For the self-play, Algorithm 1 and 2 seems repetitive. One can essentially combine them for presentation.
2. More discussion needs to be added comparing between Self-play and Adversarial results (Thm 1 and 2). For example, the self-play considered here is the zero-sum, hence the player 2 is the worst possible adversary. With this notion, comparing these two results will be helpful for future directions. It will be curious to see if the main player in Self-play (Thm 1 result) can handle the adversary in the Alg 3.

QUESTIONS:
na

LIMITATIONS:
na
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper focuses on posterior sampling for competitive reinforcement learning, aiming to propose a model-based self-play posterior sampling method to approximate Nash equilibrium in the case of self-play and adversarial learning. The theoretical analysis indicates that the proposed method achieves a low regret bound that can scale sublinearly converge.

STRENGTHS:
1. This paper is well-organized and gives a throughout survey of the related work.
2. this paper seems to be a solid theoretical work, though I'm not entirely sure of that.

WEAKNESSES:
1. Too long a supplementary, so that the reviewer may miss some details

QUESTIONS:
line 56: "... partial observations into the posterior sampling framework under a MARL ...", so the question is what is the difficulty of using posterior sampling in POMDP?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper investigates posterior sampling algorithms for competitive RL in the context of general function approximations. The authors propose the self-play and adversarial generalized eluder coefficient (GEC) as complexity measures for function approximation, capturing the exploration-exploitation trade-off in MGs. They further provide low regret bounds for proposed algorithms that can scale sublinearly with the proposed GEC and the number of episodes T.

STRENGTHS:
1. Two new generalized eluder coefficients are proposed as the complexity measure for the competitive RL with function approximation.
2. The authors also propose a novel model-based posterior sampling algorithm with self-play to learn the Nash equilibrium with provable regret bounds.
3. The technical contribution of the paper looks solid.

WEAKNESSES:
Currently, readers are hard to follow the technical results in the main paper. It will be better if the author could include some explanatory parts for explaining the technical details intuitively, so as to highlight their technical contribution.

QUESTIONS:
Please refer to the Weaknesses section.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26