SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shuang Qiu*
masqiu@ust.hk
Ziyu Dai*
zd2365@cims.nyu.edu
Northwestern University, Shaoranwang@gmail.com
zhuoran.yang@yale.edu
This paper investigates posterior sampling algorithms for competitive reinforcement learning (RL) in the context of general function approximations. Focusing on zero-sum Markov games (MGs) under two critical settings, namely self-play and adversarial learning, we first propose the self-play and adversarial generalized eluder coefficient (GEC) as complexity measures for function approximation, capturing the exploration-exploitation trade-off in MGs. Based on self-play GEC, we propose a model-based self-play posterior sampling method to control both players to learn Nash equilibrium, which can successfully handle the partial observability of states. Furthermore, we identify a set of partially observable MG models fitting MG learning with the adversarial policies of the opponent. Incorporating the adversarial GEC, we propose a model-based posterior sampling method for learning adversarial MG with potential partial observability. We further provide low regret bounds for proposed algorithms that can scale sublinearly with the proposed GEC and the number of episodes  $T$ . To the best of our knowledge, we for the first time develop generic model-based posterior sampling algorithms for competitive RL that can be applied to a majority of tractable zero-sum MG classes in both fully observable and partially observable MGs with self-play and adversarial learning.
Multi-agent reinforcement learning (MARL) tackles sequential decision-making problems where multiple players simultaneously interact with the shared environment, affecting each other's behavior in a coupled manner. Under a competitive reinforcement learning (RL) setting, the goal of each player is to maximize (resp. minimize) her own cumulative gains (resp. losses) in the presence of other agents. Recent years have been tremendous practical successes of MARL in a variety of application domains, such as autonomous driving [56], Go [58], StarCraft [65], Dota2 [8] and Poker [10]. These successes are attributed to advanced MARL algorithms that can coordinate multiple players by exploiting potentially partial observations of the latent states and employ powerful function approximators (neural networks in particular), which empower us to tackle practical problems with large state spaces.
Apart from the empirical success, there is a growing body of literature on establishing theoretical guarantees for Markov games (MGs) [57] – a standard framework for describing the dynamics of competitive RL. In particular, [71, 16, 33, 28, 77] extend the works in single-agent reinforcement
learning (RL) with function approximation [29, 60, 67, 34, 5, 11, 19, 31, 21] by developing sample-efficient algorithms that are capable to solve two-player zero-sum MGs with function approximation. In addition, as opposed to the aforementioned literature on MGs assuming the state of players is fully observable, the recent work [41] analyze Markov games under partial observability [41], i.e., the complete information about underlying states is lacking. However, most of the existing works are built upon the principle of "optimism in the face of uncertainty" (OFU) [38] for exploration. Furthermore, from a practical perspective, achieving optimism often requires explicit construction of bonus functions, which are often designed in a model-specific fashion and computationally challenging to implement.
Another promising strand of exploration techniques is based on posterior sampling, which is shown by previous works on bandits [12] and RL [48] to perform better than OFU-based algorithms. Meanwhile, posterior sampling methods, unlike OFU-based algorithms [31, 21] that need to solve complex optimization problems to achieve optimism, can be efficiently implemented by ensemble approximations [48, 44, 17, 46] and stochastic gradient Langevin dynamics (SGLD) [70]. Despite the superiority of posterior sampling, its theoretical understanding in MARL remains limited. The only exception is [73], which proposes a model-free posterior sampling algorithm for zero-sum MGs with general function approximation. However, [73] cannot capture some common tractable competitive RL models with a model-based nature, such as linear mixture MGs [16] and low witness rank MGs [28]. Moreover, their result is restricted to the fully observable MGs without handling the partial observability of the players' states. Therefore, we raise the following question:
Can we design provably sample-efficient posterior sampling algorithms for competitive RL with even partial observations under general function approximation?
Concretely, the above question poses three major challenges. First, despite the success of the OFU principle in partially observable Markov games (POMGs), it remains elusive how to incorporate the partial observations into the posterior sampling framework under a MARL setting with provably efficient exploration. Second, it is also unclear whether there is a generic function approximation condition that can cover more known classes in both full and partial observable MARL and is meanwhile compatible with the posterior sampling framework. Third, with the partial observation and function approximation, it is challenging to explore how we can solve MGs under the setups of self-play, where all players can be coordinated together, and adversarial learning, where the opponents' policies are adversarial and uncontrollable by the learner. Our work takes an initial step towards tackling such challenges by concentrating on the typical competitive RL scenario, the two-player zero-sum MG, and proposing statistically efficient posterior sampling algorithms under function approximation that can solve both self-play and adversarial MGs with full and partial observations.
Contributions. Our contributions are four-fold: (1) We first propose the two generalized eluder coefficient (GEC) as the complexity measure for the competitive RL with function approximation, namely self-play GEC and adversarial GEC, that captures the exploration-exploitation tradeoff in many existing MGs, including linear MGs, linear mixture MGs, weakly revealing POMGs, decodable POMGs. The proposed measures also generalize the recently developed GEC condition [83] from single-agent RL to MARL, suitably adjusting the exploration policy particularly for the adversarial setting. (2) Incorporating the proposed self-play GEC for general function approximation, we propose a model-based posterior sampling algorithm with self-play to learn the Nash equilibrium (NE), which successfully handles the partial observability of states along with a full observable setup by carefully designed likelihood functions. (3) We identify POMG models aligned with the form of the adversarial GEC, which fit MG learning with adversially-varying policies of the opponent. We further propose a model-based posterior sampling algorithm for adversarial learning with general function approximation. (4) We prove regret bounds for our proposed algorithms that scale sublinearly with the number of episodes  $T$ , the corresponding GEC  $d_{\mathrm{GEC}}$ , and a quantity measuring the coverage of the optimal model by the initial model sampling distribution. To the best of our knowledge, we present the first model-based posterior sampling approaches to sample-efficiently learn MGs with function approximation, handling partial observability in both self-play and adversarial settings.
Related Works. There is a large body of literature studying MGs, especially zero-sum MGs. In the self-play setting, many papers have focused on solving approximate NE in tabular zero-sum MGs [6, 7, 71, 51, 43], zero-sum MGs with linear function approximation [71, 16], zero-sum MGs with low-rank structures [50, 79, 47], and zero-sum MGs with general function approximation [33, 28, 73]. On the other hand, there are also several recent papers focusing on the adversarial setting
[71, 61, 33, 28] aim to learn the Nash value under the setting of unrevealed opponent's policies, where the adversarial policies of the opponent are unobservable. In addition, another line of adversarial MGs concentrates on a revealed policy setting, where the opponent's full policy can be observed, leading to efficiently learning a sublinear regret comparing against the best policy in hindsight. Particularly, [42] and [77] develop efficient algorithms in tabular and function approximation settings, respectively. Our approach focuses on the unrevealed policy setting, which is considered to be a more practical setup. There are also works studying MGs from various aspects [59, 32, 45, 84, 35, 20, 52, 9, 82, 18, 72, 74], such as multi-player general-sum MGs, reward-free MGs, MGs with delayed feedback, and offline MGs, which are beyond the scope of our work. Most of the aforementioned works follow the OFU principle and differ from our posterior sampling methods. The recent work [73] proposes a model-free posterior sampling algorithm for two-player zero-sum MGs but is limited to the self-play setting with fully observable states. Moreover, their work requires a strong Bellman completeness assumption that is restrictive compared to only requiring realizability in our work, mainly due to the monotonicity, meaning that adding a new function to the function class may violate it. Many model-based models like linear mixture MGs [16] and low witness rank MGs [28] are not Bellman-complete, so they cannot be captured by [73]. Without the completeness assumption, our model-based posterior sampling approaches can solve a rich class of tractable MGs, including linear mixture MGs, low witness rank MGs [28], and even POMGs, tackling both self-play and adversarial learning settings.
Our work is related to a line of research on posterior sampling methods in RL. For single-agent RL, most existing works such as [54] analyze the Bayesian regret bound. There are also some works [4, 53, 76] focusing on the frequentist (worst-case) regret bound. Our work is more closely related to the recently developed feel-good Thompson sampling technique proposed by [81] for the frequentist regret bound, and its extension to single-agent RL [19, 2, 3] and two-player zero-sum MGs [73].
Our work is also closely related to the line of research on function approximation in RL. Such a line of works proposes algorithms for efficient policy learning under diverse function approximation classes, spanning from linear Markov decision processes (MDPs) [34], linear mixture MDPs [5, 85] to nonlinear and general function classes, including, for instance, generalized linear MDPs [69], kernel and neural function classes [75], bounded eluder dimension [49, 68], Bellman rank [29], witness rank [60], bellman eluder dimension [31], bilinear [21], decision-estimation coefficient [24, 14], decoupling coefficient [19], admissible Bellman characterization [15], and GEC [83] classes.
The research on partial observability in RL [27] is closely related to our work. The works [36, 30] show that learning history-dependent policies generally can cause an exponential sample complexity. Thus, many recent works focus on analyzing tractable subclasses of partially observable Markov decision processes (POMDPs), which includes weakly revealing POMDPs [30, 39], observable POMDPs [26, 25], decodable POMDPs [22, 23], low-rank POMDPs[66], regular PSR [78], PO-bilinear class [63], latent MDP with sufficient tests [37], B-stable PSR [13], well-conditioned PSR [40], POMDPs with deterministic transition kernels [30, 62], and GEC [83]. Nevertheless, in contrast to our work which focuses on the two-player competitive setting with partial observation, these papers merely consider the single-agent setting. The recent research [41] further generalizes weakly revealing POMDPs to its multi-agent counterpart, weakly revealing POMGs, in a general-sum multiplayer setting based on the OFU principle. But when specialized to the two-player case, our work proposes a general function class that can subsume the class of weakly revealing POMGs as a special case. It would be intriguing to generalize our framework to the general-sum settings in the future.
Notations. We denote by  $\mathrm{KL}(P||Q) = \mathbb{E}_{x\sim P}[\log (\mathrm{d}P(x) / \mathrm{d}Q(x))]$  the KL divergence and  $D_{\mathrm{He}}^2 (P,Q) = 1 / 2\cdot \mathbb{E}_{x\sim P}(\sqrt{\mathrm{d}Q(x) / \mathrm{d}P(x)} -1)^2$  the Hellinger distance. We denote by  $\Delta_{\mathcal{X}}$  the set of all distributions over  $\mathcal{X}$  and Unif  $(\mathcal{X})$  the uniform distribution over  $\mathcal{X}$ . We let  $x\wedge y$  be  $\min \{x,y\}$ .
We introduce the basic concept of the two-player zero-sum Markov game (MG), function approximation, and the new complexity conditions for function approximation. Concretely, we study two typical classes of MGs, i.e., fully observable MGs and partially observable MGs, as defined below.
Fully Observable Markov Game. We consider an episodic two-player zero-sum fully observable Markov game (FOMG $^2$ ) specified by a tuple  $(S, \mathcal{A}, \mathcal{B}, \mathbb{P}, r, H)$ , where  $S$  is the state space,  $\mathcal{A}$  and
${}^{2}$  FOMG is typically referred to as MG by most literature. We adopt FOMG to differentiate it from POMG.
$\mathcal{B}$  are the action spaces of Players 1 and 2 respectively,  $H$  is the length of the episode. We denote by  $\mathbb{P} := \{\mathbb{P}_h\}_{h=1}^H$  the transition kernel with  $\mathbb{P}_h(s'|s,a,b)$  specifying the probability (density) of transitioning from state  $s$  to state  $s'$  given Players 1 and 2's actions  $a \in \mathcal{A}$  and  $b \in \mathcal{B}$  at step  $h$ . We denote the reward function as  $r = \{r_h\}_{h=1}^H$  with  $r_h: S \times \mathcal{A} \times \mathcal{B} \mapsto [0,1]$  being the reward received by players at step  $h$ . We define  $\pi = \{\pi_h\}_{h=1}^H$  and  $\nu = \{\nu_h\}_{h=1}^H$  as Markovian policies for Players 1 and 2, i.e.,  $\pi_h(a|s)$  and  $\nu_h(b|s)$  are the probability of taking action  $a$  and  $b$  conditioned on the current state  $s$  at step  $h$ . Without loss of generality, we assume the initial state  $s_1$  is fixed for each episode. We consider a realistic setting where the transition kernel  $\mathbb{P}$  is unknown and thereby needs to be approximated using the collected data.
Partially Observable Markov Game. This paper further studies an episodic zero-sum partially observable Markov game (POMG), which is distinct from the FOMG setup in that the state  $s$  is not directly observable. In particular, a POMG is represented by a tuple  $(\mathcal{S},\mathcal{A},\mathcal{B},\mathcal{O},\mathbb{P},\mathbb{O},\mu ,r,H)$  where  $\mathcal{S}$ ,  $\mathcal{A}$ ,  $\mathcal{B}$ ,  $H$ , and  $\mathbb{P}$  are similarly the state and action spaces, the episode length, and the transition kernel. Here  $\mu_{1}(\cdot)$  denotes the initial state distribution. We denote by  $\mathbb{O} := \{\mathbb{O}_h\}_{h = 1}^H$  the emission kernel so that  $\mathbb{O}_h(o|s)$  is the probability of having a partial observation  $o\in \mathcal{O}$  at state  $s$  with  $\mathcal{O}$  being the observation space. Since we only have an observation  $o$  of a state, the reward function is defined as  $r\coloneqq \{r_h\}_{h = 1}^H$  with  $r_h(o,a,b)\in [0,1]$  depending on actions  $a,b$  and the observation  $o$ , and the policies for players are defined as  $\pi = \{\pi_h\}_{h = 1}^H$  and  $\nu = \{\nu_h\}_{h = 1}^H$ , where  $\pi_h(a_h|\tau_{h - 1},o_h)$  and  $\nu_{h}(b_{h}|\tau_{h - 1},o_{h})$  is viewed as the probability of taking actions  $a_{h}$  and  $b_{h}$  depending on all histories  $(\tau_{h - 1},o_h)$ . Here we let  $\tau_{h}\coloneqq (o_{1},a_{1},b_{1}\dots,o_{h},a_{h},b_{h})$ . Then, in contrast to FOMGs, the policies in POMGs are history-dependent, defined on all prior observations and actions rather than the current state  $s$ . We define  $\mathbf{P}_h^{\pi ,\nu}(\tau_h)\coloneqq \int_{\mathcal{S}^h}\mu_1(s_1)\prod_{h' = 1}^{h - 1}[\mathbb{O}_{h'}(o_{h'}|s_{h'})\pi_{h'}(b_{h'}|\tau_{h' - 1},o_{h'})\nu_{h'}(b_{h'}|\tau_{h' - 1},o_{h'})\mathbb{P}_{h'}(s_{h' + 1}|s_{h'},a_{h'},b_{h'})]\mathbb{O}_h(o_h|s_h)\mathrm{d}s_{1:h}$  which is the joint distribution of  $\tau_h$  under the policy pair  $(\pi ,\nu)$ . Removing policies in  $\mathbf{P}_h^{\pi ,\nu}$ , we define the function  $\mathbf{P}_h(\tau_h)\coloneqq \int_{\mathcal{S}^h}\mu_1(s_1)\prod_{h' = 1}^{h - 1}[\mathbb{O}_{h'}(o_{h'}|s_{h'})\mathbb{P}_{h'}(s_{h' + 1}|s_{h'},a_{h'},b_{h'})]\mathbb{O}_h(o_h|s_h)\mathrm{d}s_{1:h}$ . We assume that the parameters  $\theta \coloneqq (\mu_1,\mathbb{P},\mathbb{O})$  are unknown and thus  $\mathbf{P}_h$  is unknown as well, which should be approximated in algorithms via online interactions.
Online Interaction with the Environment. In POMGs, at step  $h$  of episode  $t$  of the interaction, players take actions  $a_h^t \sim \pi_h^t (\cdot |\tau_{h - 1}^t,o_h^t)$  and  $b_h^t \sim \nu_h^t (\cdot |\tau_{h - 1}^t,b_{h - 1}^t,o_h^t)$  depending on their action and observation histories, receiving a reward  $r_h(o_h^t,a_h^t,b_h^t)$  and transitions from the latent state  $s_h^t$  to  $s_{h + 1}^t \sim \mathbb{P}_h(\cdot |s_h^t,a_h^t,b_h^t)$  with an observation  $o_{h + 1}^{t} \sim \mathbb{O}_{h}(\cdot |s_{h}^{t})$  generated. When the underlying state  $s_h^t$  is observable and the policies become Markovian, we have actions  $a_h^t \sim \pi_h^t (s_h^t)$  and  $b_{h}^{t} \sim \nu_{h}^{t}(s_{h}^{t})$  and the reward  $r_h(s_h^t,a_h^t,b_h^t)$ . Then, it reduces to the interaction process under the FOMG setting.
Value Function, Best Response, and Nash Equilibrium. To characterize the learning objective and the performance of the algorithms, we define the value function as the expected cumulative rewards under the policy pair  $(\pi ,\nu)$  starting from the initial step  $h = 1$  . For FOMGs, we define the value function as  $V^{\pi ,\nu}\coloneqq \mathbb{E}[\sum_{h = 1}^{H}r_h(s_h,a_h,b_h)\mid s_1,\pi ,\nu ,\mathbb{P}]$  , where the expectation is taken over all the randomness induced by  $\pi$ $\nu$  , and  $\mathbb{P}$  . For POMG, we define the value function as  $V^{\pi ,\nu}\coloneqq \mathbb{E}[\sum_{h = 1}^{H}r_{h}(o_{h},a_{h},b_{h})|\pi ,\nu ,\theta ]$  , with the expectation taken for  $\pi ,\nu$  , and  $\theta$
Our work studies the competitive setting of RL, where Player 1 (max-player) aims to maximize the value function  $V^{\pi, \nu}$  while Player 2 (min-player) aims to minimize it. With the defined value function, given a policy pair  $(\pi, \nu)$ , we define their best responses respectively as  $\mathbf{br}(\pi) \in \arg \min_{\nu} V^{\pi, \nu}$  and  $\mathbf{br}(\nu) \in \arg \max_{\pi} V^{\pi, \nu}$ . Then, we say a policy pair  $(\pi^{*}, \nu^{*})$  is a Nash equilibrium (NE) if
$$
V ^ {\pi^ {*}, \nu^ {*}} = \max  _ {\pi} \min  _ {\nu} V ^ {\pi , \nu} = \min  _ {\nu} \max  _ {\pi} V ^ {\pi , \nu}.
$$
Thus, it always holds that  $\pi^{*} = \operatorname{br}(\nu^{*})$  and  $\nu^{*} = \operatorname{br}(\pi^{*})$ . For abbreviation, we denote  $V^{*} = V^{\pi^{*},\nu^{*}}$ ,  $V^{\pi ,*} = \min_{\nu}V^{\pi ,\nu}$ , and  $V^{*,\nu} = \max_{\pi}V^{\pi ,\nu}$ , which implies  $V^{*} = V^{\pi^{*},*} = V^{*,\nu^{*}}$  for NE  $(\pi^{*},\nu^{*})$ . Moreover, we define the policy pair  $(\pi ,\nu)$  as an  $\varepsilon$ -approximate NE if it satisfies  $V^{*,\nu} - V^{\pi ,*}\leq \varepsilon$ .
Function Approximation. Since the environment is unknown to players, the model-based RL setting requires us to learn the true model of the environment,  $f^{*}$ , via (general) function approximation. We use the functions  $f$  lying in a general model function class  $\mathcal{F}$  to approximate the environment. We make a standard realizability assumption on the relationship between the model class and the true model.
Assumption 1 (Realizability). For a model class  $\mathcal{F}$ , the true model  $f^{*}$  satisfies  $f^{*} \in \mathcal{F}$ .
In our work, the true model  $f^{*}$  represents the transition kernel  $\mathbb{P}$  for the FOMG and  $\theta$  for the POMG. For any  $f \in \mathcal{F}$ , we let  $\mathbb{P}_f$  and  $\theta_f = (\mu_f, \mathbb{P}_f, \mathbb{O}_f)$  be the models under the approximation function  $f$  and  $V_{f}^{\pi,\nu}$  the value function associated with  $f$ . For POMGs, we denote  $\mathbf{P}_{f,h}^{\pi,\nu}$  and  $\mathbf{P}_{f,h}$  as  $\mathbf{P}_h^{\pi,\nu}$  and  $\mathbf{P}_h$  under the model  $f$ .
MGs with Self-Play and Adversarial Learning. Our work investigates two important MG setups for competitive RL, which are the self-play setting and the adversarial setting. In the self-play setting, the learner can control both players together to execute the proposed algorithms to learn an approximate NE. Therefore, our objective is to design sample-efficient algorithms to generate a sequence of policy pairs  $\{(\pi^t,\nu^t)\}_{t = 1}^T$  in  $T$  episodes such that the following regret can be minimized,
$$
\operatorname {R e g} ^ {\mathrm {s p}} (T) := \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {*}} ^ {*}, ^ {\nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, *} \right].
$$
In the adversarial setting, we can no longer coordinate both players, and only single player is controllable. Under such a circumstance, the opponent plays arbitrary and even adversarial policies. Wlog, suppose that the main player is the max-player with the policies  $\{\pi^t\}_{t=1}^T$  generated by a carefully designed algorithm and the opponent is min-player with arbitrary policies  $\{\nu^t\}_{t=1}^T$ . The objective of the algorithm is to learn policies  $\{\pi^t\}_{t=1}^T$  to maximize the overall cumulative rewards in the presence of an adversary. To measure the performance of algorithms, we define the following regret for the adversarial setting by comparing the learned value against the Nash value, i.e.,
$$
\mathrm {R e g} ^ {\mathrm {a d v}} (T) = \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {*}} ^ {*} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}} \right].
$$
3 Model-Based Posterior Sampling for the Self-Play Setting
We propose algorithms aiming to generate a sequence of policy pairs  $\{(\pi^t,\nu^t)\}_{t = 1}^T$  by controlling the learning process of both players such that the regret  $\mathrm{Reg}^{\mathrm{sp}}(T)$  is small. Such a regret can be decomposed into two parts, namely  $\sum_{t = 1}^{T}[V_{f^{*}}^{*} - V_{f^{*}}^{\pi^{t},*}]$  and  $\sum_{t = 1}^{T}[V_{f^{*}}^{*,\nu^{t}} - V_{f^{*}}^{*}]$ , which inspires our to design algorithms for learning  $\{\pi^t\}_{t = 1}^T$  and  $\{\nu^t\}_{t = 1}^T$  separately by targeting at minimizing these two parts respectively. Due to the symmetric structure of such a game learning problem, we propose the algorithm to learn  $\{\pi^t\}_{t = 1}^T$  as summarized in Algorithm 1. The algorithm for learning  $\{\nu^t\}_{t = 1}^T$  can be proposed in a symmetric way in Algorithm 3, which is deferred to Appendix A. Our proposed algorithm features an integration of the model-based posterior sampling and the exploiter-guided self-play in a multi-agent learning scenario. In Algorithm 1, Player 1 is the main player, while Player 2 is called the exploiter, who assists the learning of the main player by exploiting her weakness.