SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Posterior Sampling for the Main Player. The posterior sampling constructs a posterior distribution  $p^t (\cdot |Z^{t - 1})$  over the function class  $\mathcal{F}$  each round based on collected data and a pre-specified prior distribution  $p^0 (\cdot)$ , where  $Z^{t - 1}$  denotes the random history up to the end of the  $(t - 1)$ -th episode. For ease of notation, hereafter, we omit  $Z^{t - 1}$  in the posterior distribution. Most recent literature shows that adding an optimism term in the posterior distribution can lead to sample-efficient RL algorithms. Thereby, we define the distribution  $p^t (\cdot)$  over the function class  $\mathcal{F}$  for the main player as in Line 3 of Algorithm 1, which is proportional to  $p^0 (f)\exp [\gamma V_f^* +\sum_{\tau = 1}^{t - 1}\sum_{h = 1}^H L_h^\tau (f)]$ . Here,  $\gamma V_f^*$  serves as the optimism term, and  $L_h^\tau (f)$  is the likelihood function built upon the pre-collected data. Such a construction of  $p^t (\cdot)$  indicates that we will assign a higher probability (density) to a function  $f$ , which results in higher values of the combination of the optimism term and the likelihood function. We sample a model  $\overline{f}^t$  from the distribution  $p^t (\cdot)$  over the model class and learn the policy  $\pi^t$  for the main player such that  $(\pi^t,\overline{\nu}^t)$  is the NE of the value function under the model  $\overline{f}^t$  in Line 4, where  $\overline{\nu}^t$  is a dummy policy and only used in our theoretical analysis.
Posterior Sampling for the Exploiter. The exploiter aims to track the best response of  $\pi^t$  to assist learning a low regret. The best response of  $\pi^t$  generated by the exploiter is nevertheless based on a value function under a different model than  $\overline{f}^t$ . Specifically, for the exploiter, we define the posterior sampling distribution  $q^t(\cdot)$  using an optimism term  $-\gamma V_f^{\pi^t,*}$  and the summation of likelihood functions, i.e.,  $\sum_{\tau=1}^{t-1} \sum_{h=1}^{H} L_h^\tau(f)$ , along with a prior distribution  $q^0(\cdot)$ , in Line 5 of Algorithm 1. The negative term  $-\gamma V_f^{\pi^t,*}$  favors a model with a low value and is thus optimistic from the exploiter's perspective but pessimistic for the main player. We then sample a model  $\underline{f}^t$  from  $q^t(\cdot)$  and compute the best response of  $\pi^t$ , denoted as  $\underline{\nu}^t$ , under the model  $\underline{f}^t$  as in Line 7.
Algorithm 1 Model-Based Posterior Sampling for Self-Play (Max-Player)
1: Input: Model class  $\mathcal{F}$ , prior distributions  $p^0$  and  $q^0$ ,  $\gamma_1$ , and  $\gamma_2$ .  
2: for  $t = 1, \ldots, T$  do  
3: Draw a model  $\overline{f}^t \sim p^t(\cdot)$  with defining  $p^t(f) \propto p^0(f) \exp[\gamma_1 V_f^* + \sum_{\tau=1}^{t-1} \sum_{h=1}^H L_h^\tau(f)]$   
4: Compute  $\pi^t$  by letting  $(\pi^t, \overline{\nu}^t)$  be the NE of  $V_{\overline{f}^t}^{\pi,\nu}$   
5: Draw a model  $\underline{f}^t \sim q^t(\cdot)$  with defining  $q^t(f) \propto q^0(f) \exp[-\gamma_2 V_f^{\pi^t,*} + \sum_{\tau=1}^{t-1} \sum_{h=1}^H L_h^\tau(f)]$   
6: Compute  $\underline{\nu}^t$  by letting  $\underline{\nu}^t$  be the best response of  $\pi^t$  w.r.t.  $V_{\underline{f}^t}^{\pi,\nu}$   
7: Collect data  $\mathcal{D}^t$  by executing the joint exploration policy  $\sigma^t$   
8: Define the likelihood functions  $\{L_h^t(f)\}_{h=1}^H$  using the collected data  $\mathcal{D}^t$   
9: end for  
10: Return:  $(\pi^1, \ldots, \pi^T)$ .
Data Sampling and Likelihood Function. With the learned  $\pi^t$  and  $\underline{\nu}^t$ , we define a joint exploration policy  $\sigma^t$  in Line 7 of Algorithm 1, by executing which we can collect a dataset  $\mathcal{D}^t$ . We are able to further construct the likelihood functions  $\{L_h^t (f)\}_{h = 1}^H$  in Line 8 using  $\mathcal{D}^t$ . Different game settings require specifying diverse exploration policies  $\sigma^t$  and likelihood functions  $\{L_h^t (f)\}_{h = 1}^H$ . Particularly, for the game classes mainly discussed in this work, we set  $\sigma^t = (\pi^t,\underline{\nu}^t)$  for both FOMGs and POMGs. In FOMGs, we let  $\mathcal{D}^t = \{(s_h^t,a_h^t,b_h^t,s_{h + 1}^t)\}_{h = 1}^H$ , where for each  $h\in [H]$ , the data point  $(s_h^t,a_h^t,b_h^t,s_{h + 1}^t)$  is collected by executing  $\sigma^t$  to the  $h$ -th step of the game. The corresponding likelihood function is defined using the transition kernel as
$$
L _ {h} ^ {t} (f) = \eta \log \mathbb {P} _ {f, h} \left(s _ {h + 1} ^ {t} \mid s _ {h} ^ {t}, a _ {h} ^ {t}, b _ {h} ^ {t}\right). \tag {1}
$$
Furthermore, under the POMG setting, we let the dataset be  $\mathcal{D}^t = \{\tau_h^t\}_{h=1}^H$ , where the data point  $\tau_h^t = (a_1^t, a_1^t, b_1^t, \ldots, a_h^t, a_h^t, b_h^t)$  is collected by executing  $\sigma^t$  to the  $h$ -th step of the game for each  $h \in [H]$ . We further define the associated likelihood function as
$$
L _ {h} ^ {t} (f) = \eta \log \mathbf {P} _ {f, h} \left(\tau_ {h} ^ {t}\right). \tag {2}
$$
Such a construction of the likelihood function in a log-likelihood form can result in learning a model  $f$  well approximating the true model  $f^{*}$  measured via the Hellinger distance.
3.1 Regret Analysis for the Self-play Setting
Our regret analysis is based on a novel structural complexity condition for multi-agent RL and a quantity to measure how the well the prior distributions cover the optimal model  $f^{*}$ . We first define the following condition for the self-play setting.
Definition 1 (Self-Play GEC). For any sequences of functions  $f^t, g^t \in \mathcal{F}$ , suppose that a pair of policies  $(\pi^t, \nu^t)$  satisfies: (a)  $\pi^t = \operatorname{argmax}_{\pi} \min_{\nu} V_{f^t}^{\pi, \nu}$  and  $\nu^t = \operatorname{argmin}_{\nu} V_{g^t}^{\pi^t, \nu}$ , or (b)  $\nu^t = \operatorname{argmin}_{\nu} \max_{\pi} V_{f^t}^{\pi, \nu}$  and  $\pi^t = \operatorname{argmax}_{\pi} V_{g^t}^{\pi, \nu^t}$ . Denoting the joint exploration policy as  $\sigma^t$  depending on  $f^t$  and  $g^t$ , for any  $\rho \in \{f, g\}$  and  $(\pi^t, \nu^t)$  following (a) and (b), the self-play GEC  $d_{\mathrm{GEC}}$  is defined as the minimal constant  $d$  satisfying
$$
\left| \sum_ {t = 1} ^ {T} \left(V _ {\rho^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}}\right) \right| \leq \left[ d \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sum_ {\tau = 1} ^ {t - 1} \mathbb {E} _ {\left(\sigma^ {\tau}, h\right)} \ell \left(\rho^ {t}, \xi_ {h} ^ {\tau}\right)\right) \right] ^ {\frac {1}{2}} + 2 H (d H T) ^ {\frac {1}{2}} + \epsilon H T.
$$
Our definition of self-play GEC is inspired by [83] for the single-agent RL. Then, it shares an analogous meaning to the single-agent GEC. Here  $(\sigma^{\tau}, h)$  implies running the joint exploration policy  $\sigma^{\tau}$  to step  $h$  to collect a data point  $\xi_h^\tau$ . The LHS of the inequality is viewed as the prediction error and the RHS is the training error defined on a loss function  $\ell$  plus a burn-in error  $2H(dHT)^{\frac{1}{2}} + \epsilon HT$  that is non-dominating when  $\epsilon$  is small. The loss function  $\ell$  and  $\epsilon$  can be problem-specific. We determine  $\ell(f, \xi_h)$  for FOMGs with  $\xi_h = (s_h, a_h)$  and POMGs with  $\xi_h = \tau_h$  respectively as
$$
\text {F O M G :} D _ {\mathrm {H e}} ^ {2} \left(\mathbb {P} _ {f, h} (\cdot | \xi_ {h}), \mathbb {P} _ {f ^ {*}, h} (\cdot | \xi_ {h})\right), \quad \text {P O M G :} 1 / 2 \cdot \left(\sqrt {\mathbf {P} _ {f , h} (\xi_ {h}) / \mathbf {P} _ {f ^ {*} , h} (\xi_ {h})} - 1\right) ^ {2}, \tag {3}
$$
such that  $\mathbb{E}_{(\sigma,h)}[\ell(f,\xi_h)] = D_{\mathrm{He}}^2(\mathbf{P}_{f^*,h}^\sigma, \mathbf{P}_{f,h}^\sigma)$  for POMGs. The intuition for GEC is that if hypotheses have a small training error on a well-explored dataset, then the out-of-sample prediction error is also small, which characterizes the hardness of environment exploration.
Since the posterior sampling steps in our algorithms depend on the initial distributions  $p^0$  and  $q^0$ , we define the following quantity to measure how well the prior distributions  $p^0$  and  $q^0$  cover the optimal model  $f^{*} \in \mathcal{F}$ , which is also a multi-agent generalization of its single-agent version [2, 83].
Definition 2 (Prior around True Model). Given  $\beta >0$  and any distribution  $p^0\in \Delta_{\mathcal{F}}$ , we define
$$
\omega (\beta , p ^ {0}) = \inf  _ {\varepsilon > 0} \left\{\beta \varepsilon - \ln p ^ {0} [ \mathcal {F} (\varepsilon) ] \right\},
$$
where  $\mathcal{F}(\varepsilon) := \{f \in \mathcal{F} : \sup_{h,s,a,b} \mathrm{KL}^{\frac{1}{2}}(\mathbb{P}_{f^{*},h}(\cdot | s,a,b) \| \mathbb{P}_{f,h}(\cdot | s,a,b)) \leq \varepsilon\}$  for FOMGs and  $\mathcal{F}(\varepsilon) := \{f \in \mathcal{F} : \sup_{\pi,\nu} \mathrm{KL}^{\frac{1}{2}}(\mathbf{P}_{f^{*},H}^{\pi,\nu}) \| \mathbf{P}_{f,H}^{\pi,\nu}) \leq \varepsilon\}$  for POMGs.
When the model class  $\mathcal{F}$  is a finite space, if let  $p^0 = \mathrm{Unif}(\mathcal{F})$ , we simply know that  $\omega (\beta ,p^{0})\leq \log |\mathcal{F}|$  where  $|\mathcal{F}|$  is the cardinality of  $\mathcal{F}$ . Furthermore, for an infinite function class  $\mathcal{F}$ , the term  $\log |\mathcal{F}|$  can be substituted by a quantity having logarithmic dependence on the covering number of the function class  $\mathcal{F}$ . With the multi-agent GEC condition and the definition of  $\omega$ , we have the following regret bound for both FOMGs and POMGs.
Proposition 1. Letting  $\eta = 1/2$ ,  $\gamma_1 = 2\sqrt{\omega(4HT, p^0)T / d_{\mathrm{GEC}}}$ ,  $\gamma_2 = 2\sqrt{\omega(4HT, q^0)T / d_{\mathrm{GEC}}}$ ,  $\epsilon = 1/\sqrt{HT}$  in Definition 1, when  $T \geq \max\{4H^2\omega(4HT, p^0) / d_{\mathrm{GEC}}, 4H^2\omega(4HT, q^0) / d_{\mathrm{GEC}}, d_{\mathrm{GEC}} / H\}$ , under both FOMG and POMG settings, Algorithm 1 admits the following regret bound,
$$
\mathbb {E} \left[ \operatorname {R e g} _ {1} ^ {\mathrm {s p}} (T) \right] := \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(V _ {f ^ {*}} ^ {*} - V _ {f ^ {*}} ^ {\pi^ {t}, *}\right) \right] \leq 6 \sqrt {d _ {\mathrm {G E C}} H T \cdot \left[ \omega \left(4 H T , p ^ {0}\right) + \omega \left(4 H T , q ^ {0}\right) \right]}.
$$
This proposition gives the upper bound  $\mathbb{E}[\mathrm{Reg}_1^{\mathrm{sp}}(T)]$  following the updating rules in Algorithm 1 when the max-player is the main player. As Algorithm 3 is symmetric to Algorithm 1, we obtain the following regret bound of  $\mathbb{E}[\mathrm{Reg}_2^{\mathrm{sp}}(T)]$  for Algorithm 3 when the min-player is the main player.
Proposition 2. Under the same parameter settings as Proposition 1, Algorithm 3 admits the following regret bound,
$$
\mathbb {E} \left[ \operatorname {R e g} _ {2} ^ {\mathrm {s p}} (T) \right] := \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(V _ {f ^ {*}} ^ {*, \nu^ {t}} - V _ {f ^ {*}} ^ {*}\right) \right] \leq 6 \sqrt {d _ {\mathrm {G E C}} H T \cdot \left[ \omega \left(4 H T , p ^ {0}\right) + \omega \left(4 H T , q ^ {0}\right) \right]}.
$$
Combining the results of Propositions 1 and 2, due to  $\mathrm{Reg}^{\mathrm{sp}}(T) = \mathrm{Reg}_1^{\mathrm{sp}}(T) + \mathrm{Reg}_2^{\mathrm{sp}}(T)$ , we obtain the following overall regret when running Algorithms 1 and 3 together.
Theorem 1. Under the settings of Propositions 1 and 2, executing both Algorithms 1 and 3 leads to
$$
\mathbb {E} \left[ \operatorname {R e g} ^ {\mathrm {s p}} (T) \right] \leq 1 2 \sqrt {d _ {\mathrm {G E C}} H T \cdot \left[ \omega \left(4 H T , p ^ {0}\right) + \omega \left(4 H T , q ^ {0}\right) \right]}.
$$
The above results indicate that the proposed posterior sampling self-play algorithms (Algorithms 1 and 3) separately admit a sublinear dependence on GEC  $d_{\mathrm{GEC}}$ , the number of learning episodes  $T$ , as well as  $\omega(4HT, p^0)$  and  $\omega(4HT, q^0)$  for both FOMG and POMG settings. They lead to the same overall regret bound combining Propositions 1 and 2. In particular, when  $\mathcal{F}$  is finite with  $p^0 = q^0 = \mathrm{Unif}(\mathcal{F})$ , Algorithms 1 and 3 admit regrets of  $O(\sqrt{d_{\mathrm{GEC}} HT \cdot \log |\mathcal{F}|})$ . The quantity  $\omega$  can be associated with the log-covering number if  $\mathcal{F}$  is infinite. Please see Appendix C for analysis.
4 Posterior Sampling for the Adversarial Setting
Without loss of generality, we assume that the max-player is the main agent and the min-player is the opponent. Under this setting, the goal of the main player is to maximize her cumulative rewards as much as possible, comparing against the value under the NE, i.e.,  $V_{f^*}^*$ . We develop a novel algorithm for this setting as summarized in Algorithm 2. In our algorithm, the opponent's policy is assumed to be arbitrary and is also not revealed to the main player. The only information about the opponent is the current state or the partial observation of her state as well as the actions taken.
We adopt the optimistic posterior sampling approach for the main player with defining an optimism term as  $\gamma V_{f}^{*}$  motivated by the above learning target, and the likelihood function  $L_{h}^{t}(f)$  with  $L_{h}^{t}(f):= \eta \log \mathbb{P}_{f,h}(s_{h + 1}^{t}|s_{h}^{t},a_{h}^{t},b_{h}^{t})$  in (1) for FOMGs and  $L_{h}^{t}(f) = \eta \log \mathbf{P}_{f,h}(\tau_{h}^{t})$  in (2) for POMGs
Algorithm 2 Model-Based Posterior Sampling with Adversarial Opponent
1: Input: Model class  $\mathcal{F}$ , prior distributions  $p^0$ , and  $\gamma$ .  
2: for  $t = 1, \dots, T$  do  
3: Draw a model  $f^t \sim p^t(\cdot)$  with defining  $p^t(f) \propto p^0(f) \exp[\gamma V_f^* + \sum_{\tau=1}^{t-1} \sum_{h=1}^{H} L_h^\tau(f)]$   
4: Compute  $\pi^t$  by letting  $(\pi^t, \overline{\nu}^t)$  be NE of  $V_{f^t}^{\pi,\nu}$   
5: Opponent picks an arbitrary policy  $\nu^t$   
6: Collect a trajectory  $\mathcal{D}^t$  by executing the joint exploration policy  $\sigma^t$   
7: Define the likelihood functions  $\{L_h^t(f)\}_{h=1}^H$  using the collected data  $\mathcal{D}^t$   
8: end for  
9: Return:  $(\pi^1, \dots, \pi^T)$ .
respectively. The policy  $\pi^t$  learned by the main player is from computing the NE of the value function under the current model  $f^t$  sampled from the posterior distribution  $p^t$ . In addition, the joint exploration policy is set to be  $\sigma^t = (\pi^t,\nu^t)$  where  $\nu^t$  is the potentially adversarial policy played by the opponent. Thus, we can collect the data defined as  $\mathcal{D}^t = \{(s_h^t,a_h^t,b_h^t,s_{h + 1}^t)\}_{h = 1}^H$  and  $\mathcal{D}^t = \{\tau_h^t\}_{h = 1}^H$  with  $\tau_h^t = (o_1^t,o_1^t,b_1^t\dots ,o_h^t,o_h^t,b_h^t)$  for FOMGs and POMGs respectively, collected by executing  $\sigma^t$  to the  $h$ -th step of the game for each  $h\in [H]$ .
Remark 1. In Algorithm 2, we define the joint exploration policy  $\sigma^t = (\pi^t,\nu^t)$ , which is the key to the success of the algorithm design under the adversarial setting, especially for POMGs. Under the single-agent setting, the prior work [83] sets the exploration policy for a range of partially observable models subsumed by the PSR model as  $\pi_{1:h-1}^t\circ_h\mathrm{Unif}(\mathcal{A})$ , i.e., running  $\pi^t$  for steps 1 to  $h-1$  and then sampling the data at step  $h$  by enforcing a uniform policy. Such an exploration scheme fails to work when facing an uncontrollable opponent who does not play a uniform policy at step  $h$ . Theoretically, we prove that employing policies  $(\pi_{1:h}^t,\nu_{1:h}^t)$  for exploration without the uniform policy, the self-play and adversarial GEC conditions in Definitions 1 and 3 are still satisfied for a class of POMGs including weakly revealing and decodable POMGs. This eventually leads to a unified adversarial learning algorithm for both FOMGs and POMGs.
4.1 Regret Analysis for the Adversarial Setting
Before demonstrating our regret analysis, we first define a multi-agent GEC fitting the adversarial learning scenario. Considering that the opponent's policy is uncontrollable during the learning, we let  $\{\nu^t\}_{t=1}^T$  be arbitrary, which is clearly distinguished from self-play GEC defined in Definition 1.
Definition 3 (Adversarial GEC). For any sequence of functions  $\{f^t\}_{t=1}^T$  with  $f^t \in \mathcal{F}$  and any sequence of the opponent's policies  $\{\nu^t\}_{t=1}^T$ , suppose that the main player's policies  $\{\mu^t\}_{t=1}^T$  are generated via  $\mu^t = \operatorname{argmax}_{\pi} \min_{\nu} V_{f^t}^{\pi, \nu}$ . Denoting the joint exploration policy as  $\{\sigma^t\}_{t=1}^T$  depending on  $\{f^t\}_{t=1}^T$ , the adversarial GEC  $d_{\mathrm{GEC}}$  is defined as the minimal constant  $d$  satisfying
$$
\sum_ {t = 1} ^ {T} \left(V _ {f ^ {t}} ^ {\pi^ {t}, \nu^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, \nu^ {t}}\right) \leq \left[ d \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \left(\sum_ {\tau = 1} ^ {t - 1} \mathbb {E} _ {(\sigma^ {\tau}, h)} \ell (f ^ {t}, \xi_ {h} ^ {\tau})\right) \right] ^ {\frac {1}{2}} + 2 H (d H T) ^ {\frac {1}{2}} + \epsilon H T.
$$
Our regret analysis for Algorithm 2 also depends on the quantity  $\omega (\beta ,p^0)$  that characterizes the coverage of the prior distribution  $p^0$  on the true model  $f^{*}$ . Then, we have the following regret bound.
Theorem 2. Letting  $\eta = \frac{1}{2}$ ,  $\gamma = 2\sqrt{\omega(4HT, p^0)T / d_{\mathrm{GEC}}}$ ,  $\epsilon = 1 / \sqrt{HT}$  in Definition 3, when  $T \geq \max \{4H^2\omega(4HT, p^0) / d_{\mathrm{GEC}}, d_{\mathrm{GEC}} / H\}$ , under both FOMG and POMG settings, Algorithm 2 admits the following regret bound,
$$
\mathbb {E} [ \mathrm {R e g} ^ {\mathrm {a d v}} (T) ] \leq 4 \sqrt {d _ {\mathrm {G E C}} H T \cdot \omega (4 H T , p ^ {0})}.
$$
The above result indicates that we can achieve a meaningful regret bound by a posterior sampling algorithm with general function approximation, even when the opponent's policy is adversarial and her full policies  $\nu^t$  are not revealed. This regret has a sublinear dependence on  $d_{\mathrm{GEC}}$ , the number of episodes  $T$ , as well as  $\omega(4HT, p^0)$ . Similarly, when  $\mathcal{F}$  is finite, Algorithm 2 admits a regret of  $O(\sqrt{d_{\mathrm{GEC}} HT \cdot \log |\mathcal{F}|})$ . The term  $\log |\mathcal{F}|$  can be the log-covering number of  $\mathcal{F}$  if it is infinite.
This section presents several examples of tractable MG classes captured by the self-play and adversarial GEC, the discussion of the quantity  $\omega (\beta ,p^0)$ , the proof sketches of main theorems, and the discussion of limitations.
Examples. We call the class with a low  $d_{\mathrm{GEC}}$  the low self-play GEC class and low adversarial GEC class. Next, we analyze the relation between the proposed classes and the following MG classes. We also propose a new decodable POMG class generalized from the single-agent POMDP. We note that except for the linear MG, the other classes cannot be analyzed by the recent posterior sampling work [73]. We defer detailed definitions and proofs to Appendix B.
- Linear MG. This FOMG class admits a linear structure of the reward and transition by feature vectors  $\phi(s, a, b) \in \mathbb{R}^d$  as  $r_h(s, a, b) = \mathbf{w}_h^\top \phi(s, a, b)$  and  $\mathbb{P}_h(s'|s, a, b) = \pmb{\theta}_h(s')^\top \pmb{\phi}(s, a, b)$  [71]. We then prove that "linear MG  $\subset$  low self-play/adversarial GEC" with  $d_{\mathrm{GEC}} = \widetilde{O}(H^3 d)$ .
- Linear Mixture MG. This FOMG class admits a different type of the linear structure for the transition [16] as  $\mathbb{P}_h(s'|s,a,b) = \pmb{\theta}_h^\top \pmb{\phi}(s,a,b,s')$  with  $\phi(s,a,b,s') \in \mathbb{R}^d$ . We prove that "linear mixture  $MG \subset$  low self-play/adversarial GEC" with  $d_{\mathrm{GEC}} = \widetilde{O}(H^3 d)$ .
- Low Self-Play Witness Rank. [28] defines this FOMG class for self-play by supposing that an inner product of specific vectors in  $\mathbb{R}^d$  defined on current models can lower bound witnessed model misfit and upper bound the Bellman error with a coefficient  $\kappa_{\mathrm{wt}}$ , which generalizes linear/linear mixture MGs. We can prove "low self-play witness rank  $\subset$  low self-play GEC" with  $d_{\mathrm{GEC}} = \widetilde{O}(H^3 d / \kappa_{\mathrm{wt}}^2)$ .
-  $\alpha$ -Weakly Revealing POMG. This POMG class assumes  $\min_h \sigma_S(\mathbb{O}_h) \geq \alpha$  where  $\mathbb{O}_h \in \mathbb{R}^{|\mathcal{O}| \times |\mathcal{S}|}$  is the matrix by  $\mathbb{O}_h(\cdot|\cdot)$  and  $\sigma_S$  is the  $S$ -th singular value [41]. We prove that "  $\alpha$ -weakly revealing POMG  $\subset$  low self-play/adversarial GEC" with  $d_{\mathrm{GEC}} = \widetilde{O}(H^3|\mathcal{O}|^3|\mathcal{A}|^2|\mathcal{B}|^2|\mathcal{S}|^2/\alpha^2)$ .
- Decodable POMG. We propose decodable POMGs by generalizing decodable POMDPs [23, 22], assuming that an unknown decoder  $\phi_h$  recovers states from observations, i.e.,  $\phi_h(o) = s$ . We can prove "decodable  $POMG \subset$  low self-play/adversarial  $GEC$  with  $d_{\mathrm{GEC}} = \widetilde{O}(H^3 |\mathcal{O}|^3 |\mathcal{A}|^2 |\mathcal{B}|^2)$ .
Discussion of  $\omega (\beta ,p^0)$ . We briefly discuss the upper bound of the quantity  $\omega (\beta ,p^0)$  for FOMGs and POMGs. We refer readers to Appendix C for more detailed proofs. For FOMGs, according to Lemma 2 of [2], when  $\mathcal{F}$  is finite,  $p^0 = \mathrm{Unif}(\mathcal{F})$ , then  $\omega (\beta ,p^0)\leq \log |\mathcal{F}|$  by its definition. When  $\mathcal{F}$  is infinite, it shows that under mild conditions, there exists a prior  $p^0$  over  $\mathcal{F}$ ,  $B\geq \log (6B^{2} / \epsilon)$ , and  $\nu = \epsilon /(6\log (6B^{2} / \epsilon))$  such that  $\omega (\beta ,p^0)\leq \beta \epsilon +\log (\mathcal{N}(\frac{\epsilon}{6\log(B / \nu)}))$ , where  $\mathcal{N}(\epsilon)$  is the  $\epsilon$ -covering number w.r.t. the distance  $d(f,f')\coloneqq \sup_{s,a,b,h}|D_{\mathrm{He}}^2 (\mathbb{P}_{f,h}(\cdot |s,a,b),\mathbb{P}_{f^*,h}(\cdot |s,a,b)) - D_{\mathrm{He}}^2 (\mathbb{P}_{f',h}(\cdot |s,a,b),\mathbb{P}_{f^*,h}(\cdot |s,a,b))|$ . Since we have  $|D_{\mathrm{He}}^{2}(P,R) - D_{\mathrm{He}}^{2}(Q,R)|\leq \frac{\sqrt{2}}{2}\| P - Q\| _1$  for any distributions  $P,Q$ , and  $R$ , the covering number w.r.t. the distance  $d$  can connect to the more common covering number w.r.t. the  $\ell_1$  distance. Thus, the upper bound of  $\omega (\beta ,p^0)$  can be calculated for different cases. Additionally, for POMGs, inspired by [2], our work proves that under similar conditions,  $\omega (\beta ,p^0)$  with finite and infinite  $\mathcal{F}$  admit the same bounds as those for FOMGs. The difference is that the covering number is w.r.t. the distance  $d(f,f') = \sup_{\pi ,\nu}|D_{\mathrm{He}}^{2}(\mathbf{P}_{f,H}^{\pi ,\nu},\mathbf{P}_{f^{*},H}^{\pi ,\nu}) - D_{\mathrm{He}}^{2}(\mathbf{P}_{f',H}^{\pi ,\nu},\mathbf{P}_{f^{*},H}^{\pi ,\nu})|$ , which further connects to the  $\ell_1$  distance defined as  $d_{1}(f,f')\coloneqq \sup_{\pi ,\nu}\| \mathbf{P}_{f,H}^{\pi ,\nu} - \mathbf{P}_{f',H}^{\pi ,\nu}\| _1$ . Such a covering number under  $\ell_1$  distance is further analyzed in [78]. Our work gives the first detailed proof for the upper bound of  $\omega (\beta ,p^0)$  under the partially observable setting, which is thus of independent interest.
Next, we outline our proof sketches. Detailed proofs are deferred to Appendices D, E, and F.
Proof Sketch of Theorem 1. To prove Theorem 1, we only need to combine the result in Propositions 1 and 2 via  $\mathbb{E}[\mathrm{Reg}^{\mathrm{sp}}(T)] = \mathbb{E}[\mathrm{Reg}_1^{\mathrm{sp}}(T) + \mathrm{Reg}_2^{\mathrm{sp}}(T)]$ . We thus first give a proof sketch for Proposition 1. We decompose  $\mathrm{Reg}_1^{\mathrm{sp}}(T) = \mathrm{Term}(\mathrm{i}) + \mathrm{Term}(\mathrm{ii})$  where
$$
\operatorname {T e r m} (\mathrm {i}) = \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {*}} ^ {*} - V _ {f ^ {*}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} \right], \quad \operatorname {T e r m} (\mathrm {i i}) = \sum_ {t = 1} ^ {T} \left[ V _ {f ^ {*}} ^ {\pi^ {t}, \underline {{\nu}} ^ {t}} - V _ {f ^ {*}} ^ {\pi^ {t}, *} \right].
$$
Intuitively,  $\mathbb{E}[\mathrm{Term(i)}]$  is the main player's regret incurred Line 4 of Algorithm 1 and  $\mathbb{E}[\mathrm{Term(ii)}]$  is the exploiter's regret incurred by Line 6. We further show