SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma C.2. There exists absolute constant  $c_2$  such that under the same condition as Theorem C.1, we have that with probability at least  $1 - \delta$ : for all  $t \in [T]$
4The exploration policy set is problem-dependent and can be simply  $\{\pi^t\}$  for many settings.
-  $(r^{\star},p^{\star})\in \mathcal{B}^{t}$
-  $\sum_{t=1}^{T} \max_{p \in \mathcal{B}_{\mathcal{P}}^{t}} d_{\mathrm{TV}}(\mathbb{P}_{p}^{\pi^{t}}, \mathbb{P}_{p^{\star}}^{\pi^{t}}) \leq \xi(d_{\mathcal{P}}, T, c_{2}\beta_{\mathcal{P}}, |\Pi_{\exp}|)$ ,
-  $\sum_{i < t}|\sigma (r^t (\tau^i) - r^t (\tau^0)) - \sigma (r^\star (\tau^i) - r^\star (\tau^0))|^2\leq \mathcal{O}(\beta_\mathcal{R}).$
Proof. For the first two statements, see the proof of Theorem 3.2 in Liu et al. [2022a]. For the third bullet point, see Liu et al. [2022a, Proposition B.2], which implies that
$$
\mathrm {L H S} \leq \mathcal {O} \left(\beta_ {\mathcal {R}} + \ln (| \mathcal {R} | T / \delta)\right) = \mathcal {O} (\beta).
$$
Proof of Theorem C.1. By using the first relation in Lemma C.2 and the definition of  $(\pi^t,r^t,p^t)$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left[ V _ {r ^ {\star}, p ^ {\star}} ^ {\star} - V _ {r ^ {\star}, p ^ {\star}} ^ {\pi^ {t}} \right] \\ = \sum_ {t = 1} ^ {T} \left[ V _ {r ^ {\star}, p ^ {\star}} ^ {\star} - r ^ {\star} \left(\tau^ {0}\right) + r ^ {\star} \left(\tau^ {0}\right) - V _ {r ^ {\star}, p ^ {\star}} ^ {\pi^ {t}} \right] \\ \leq \sum_ {t = 1} ^ {T} \left[ V _ {r ^ {t}, p ^ {t}} ^ {\pi^ {t}} - r ^ {t} \left(\tau^ {0}\right) + r ^ {\star} \left(\tau^ {0}\right) - V _ {r ^ {\star}, p ^ {\star}} ^ {\pi^ {t}} \right] \\ = \sum_ {t = 1} ^ {T} \left[ V _ {r ^ {t}, p ^ {t}} ^ {\pi^ {t}} - V _ {r ^ {t}, p ^ {\star}} ^ {\pi^ {t}} \right] + \sum_ {t = 1} ^ {T} \left[ V _ {r ^ {t}, p ^ {\star}} ^ {\pi^ {t}} - r ^ {t} \left(\tau^ {0}\right) + r ^ {\star} \left(\tau^ {0}\right) - V _ {r ^ {\star}, p ^ {\star}} ^ {\pi^ {t}} \right]. \\ \end{array}
$$
We can control the first term by the second inequality in Lemma lem:omle-pref
$$
\sum_ {t = 1} ^ {T} \left[ V _ {r ^ {t}, p ^ {t}} ^ {\pi^ {t}} - V _ {r ^ {t}, p ^ {\star}} ^ {\pi^ {t}} \right] \leq 2 \sum_ {t = 1} ^ {T} d _ {\mathrm {T V}} \left(\mathbb {P} _ {p ^ {t}} ^ {\pi^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\pi^ {t}}\right) \leq 2 H \xi \left(d _ {\mathcal {P}}, T, c _ {2} \beta_ {\mathcal {P}}, | \Pi_ {\exp} |\right).
$$
For the second term, by Azuma-Hoeffding inequality and by combining the second inequality in Lemma C.2 with the regret guarantee for eluder dimension,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} [ V _ {r ^ {t}, p ^ {\star}} ^ {\pi^ {t}} - r ^ {t} (\tau^ {0}) + r ^ {\star} (\tau^ {0}) - V _ {r ^ {\star}, p ^ {\star}} ^ {\pi^ {t}} ] \\ \leq \sum_ {t = 1} ^ {T} | [ (r ^ {t} (\tau^ {t}) - r ^ {t} (\tau^ {0})) ] - [ r ^ {\star} (\tau^ {t}) - r ^ {\star} (\tau^ {0}) ] | + \mathcal {O} (H \sqrt {T \ln (1 / \delta)}) \\ \leq \mathcal {O} \left(H \alpha^ {- 1} \sqrt {d _ {\mathcal {R}} T \beta_ {\mathcal {R}}}\right). \\ \end{array}
$$
D OMLE with Perturbed Reward
D.1 Algorithm details and theoretical guarantees
In this section, we modify the optimistic MLE (OMLE) algorithm [Liu et al., 2022a] to deal with unknown reward functions. The adapted algorithm can then be used with Algorithm 1 as described in Example 3. OMLE is a model-based algorithm that requires a model class  $\mathcal{P}$  in addition to the reward class  $\mathcal{R}$ . On a high level, OMLE maintains a joint confidence set  $\mathcal{B}$  in  $\mathcal{R} \times \mathcal{P}$ . In the  $t$ -th iteration, the algorithm follows the following steps:
- Optimistic planning: Find the policy-model pair  $(\pi, r, p)$  that maximizes the value function  $V_{r,p}^{\pi}$  which is the expected cumulative reward the leaner will receive if she follows policy  $\pi$  in a model with transition  $p$  and reward  $r$ ;
- Data collection: Construct an exploration policy set  $\Pi_{\mathrm{exp}}(\pi^t)$  and collect trajectories by running all policies in  $\Pi_{\mathrm{exp}}(\pi^t)$ ;
- Confidence set update: Update the confidence set using the updated log-likelihood.
The main modification we make is the confidence set of  $r$ , since the original OMLE algorithm assumes that  $r$  is known. The pseudocode of our adapted OMLE algorithm is provided in Algorithm 3.
Algorithm 3 Optimistic MLE with  $\epsilon^{\prime}$ -Perturbed Reward Feedback
1:  $\mathcal{B}^1\gets \mathcal{R}\times \mathcal{P}$
2: Execute an arbitrary policy to collect trajectory  $\tau^0$
3: for  $t = 1, \dots, T$  do
4: Compute  $(\pi^t,r^t,p^t) = \arg \max_{\pi ,(r,p)\in B^t}V_{r,p}^\pi$
5: Execute  $\pi^t$  to collect a trajectory  $\tau$ , receive reward  $\hat{r}$ , add  $(\tau, \hat{r})$  into  $\mathcal{D}_{\mathrm{rwd}}$
6: for each  $\pi \in \Pi_{\mathrm{exp}}(\pi^t)$  do
7: Execute  $\pi$  to collect a trajectory  $\tau$ , add  $(\pi, \tau)$  into  $\mathcal{D}_{\text{trans}}$
8: Update
$$
\mathcal {B} ^ {t + 1} \leftarrow \left\{(r, p) \in \mathcal {R} \times \mathcal {P}: \max  _ {(\tau , \hat {r}) \in \mathcal {D} _ {\mathrm {r w d}}} | r (\tau) - \hat {r} | \leq \epsilon^ {\prime} \right.
$$
$$
\text{and}\mathcal{L}(p,\mathcal{D}_{\text{trans}}) > \max_{p^{\prime}\in \mathcal{P}}\mathcal{L}(p^{\prime},\mathcal{D}_{\text{trans}}) - \beta_{\mathcal{P}}\big\}
$$
In Line 8, the log-likelihood function is defined as
$$
\mathcal {L} (p, \mathcal {D} _ {\text {t r a n s}}) := \sum_ {(\pi , \tau) \in \mathcal {D} _ {\text {t r a n s}}} \ln \mathbb {P} _ {p} ^ {\pi} (\tau), \tag {2}
$$
where  $\mathbb{P}_p^\pi (\tau)$  denotes the probability of observing trajectory  $\tau$  in a model with transition function  $p$  under policy  $\pi$ . Other algorithmic parameters  $T$  and  $\beta_{\mathcal{P}}$  are chosen in the statement of Theorem D.1.
To obtain sample efficiency guarantees for OMLE, the following assumption is needed.
Assumption 3 (Generalized eluder-type condition). There exists a real number  $d_{\mathcal{P}} \in \mathbb{R}^{+}$  and a function  $\zeta$  such that: for any  $(T, \Delta) \in \mathbb{N} \times \mathbb{R}^{+}$ , transitions  $\{p^t\}_{t \in [T]}$  and policies  $\{\pi^t\}_{t \in [T]}$ , we have
$$
\forall t \in [ T ], \sum_ {\tau = 1} ^ {t - 1} \sum_ {\pi \in \Pi_ {\exp} (\pi^ {\tau})} d _ {\mathrm {T V}} ^ {2} \left(\mathbb {P} _ {p ^ {t}} ^ {\pi}, \mathbb {P} _ {p ^ {\star}} ^ {\pi}\right) \leq \Delta \Rightarrow \sum_ {t = 1} ^ {T} d _ {\mathrm {T V}} \left(\mathbb {P} _ {p ^ {t}} ^ {\pi^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\pi^ {t}}\right) \leq \xi \left(d _ {\mathcal {P}}, T, \Delta , | \Pi_ {\exp} |\right).
$$
Here  $\mathbb{P}_p^\pi$  is the distribution of trajectories under model  $p$  and policy  $\pi$ , while  $|\Pi_{\mathrm{exp}}| := \max_{\pi} |\Pi_{\mathrm{exp}}(\pi)|$  is the largest possible number of exploration policies.
Assumption 3 shares a similar intuition with the pigeon-hole principle and the elliptical potential lemma, which play important roles in the analysis of tabular MDP and linear bandits respectively. In particular, the  $\xi$  function measures the worst-case growth rate of the cumulative error and is the central quantity characterizing the hardness of the problem. Liu et al. [2022a] prove that a wide range of RL problems satisfy Assumption 3 with moderate  $d_{\mathcal{P}}$ ,  $|\Pi_{\mathrm{exp}}|$  and  $\xi = \widetilde{\mathcal{O}} (\sqrt{d_{\mathcal{P}}\Delta|\Pi_{\mathrm{exp}}|K})$ , including tabular MDPs, factored MDPs, observable POMDPs and decodable POMDPs (see Liu et al. [2022a] for more details).
Theorem D.1. Suppose Assumption 1 and 3 hold. There exists absolute constant  $c_{1}, c_{2} > 0$  such that for any  $(T, \delta) \in \mathbb{N} \times (0, 1]$ , if we choose  $\beta_{\mathcal{P}} = c_{1}\ln(|\mathcal{P}||\Pi_{\exp}|T / \delta)$  in Algorithm 3, then with probability at least  $1 - \delta$ , we have that for all  $t \in [T]$ ,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left[ V ^ {\star} - V ^ {\pi^ {t}} \right] \leq 2 H \xi \left(d _ {\mathcal {P}}, T, c _ {2} \beta_ {\mathcal {P}}, | \Pi_ {\exp} |\right) \\ + \mathcal {O} \left(\min  _ {\omega > 0} \left\{T \sqrt {d _ {\mathcal {R}} (\omega)} \epsilon^ {\prime} + \min  \left\{d _ {\mathcal {R}} (\omega), T \right\} H + T \omega \right\}\right) + \mathcal {O} (H \sqrt {T \ln \delta^ {- 1}}) \\ \end{array}
$$
where  $d_{\mathcal{R}} = \dim_{\mathrm{E}}(\mathcal{R},\omega)$
5The exploration policy set is problem-dependent and can be simply  $\{\pi^t\}$  for many settings.
For problems that satisfy  $\xi = \widetilde{\mathcal{O}} (\sqrt{d_{\mathcal{P}}\beta_{\mathcal{P}}|\Pi_{\mathrm{exp}}|T})$ , Theorem D.1 implies a sample complexity of
$$
\widetilde {\mathcal {O}} \left(\frac {H ^ {2} d _ {\mathcal {P}} | \Pi_ {\mathrm {e x p}} | ^ {2} \ln | \mathcal {P} |}{\epsilon^ {2}} + \frac {H d _ {\mathcal {R}} | \Pi_ {\mathrm {e x p}} |}{\epsilon}\right).
$$
for learning an  $\mathcal{O}(\epsilon)$ -optimal policy, if we have  $\omega = \epsilon$  and  $\epsilon' = \epsilon / \sqrt{d_{\mathcal{R}}}$ . The sample complexity for specific tractable problems can be found in Appendix D.2.
D.2 Examples satisfying generalized eluder-type condition
In this section, we present several canonical examples that satisfy the generalized eluder-type condition with  $\xi = \tilde{\mathcal{O}} (\sqrt{d_{\mathcal{P}}\beta_{\mathcal{P}}|\Pi_{\mathrm{exp}}|T})$ . More examples can be found in Liu et al. [2022a].
Example 6 (Finite-precision Factored MDPs). In factored MDPs, each state  $s$  consists of  $m$  factors denoted by  $(s[1],\dots ,s[m])\in \mathcal{X}^m$ . The transition structure is also factored as
$$
\mathbb {P} _ {h} \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right) = \prod_ {i = 1} ^ {m} \mathbb {P} ^ {i} \left(s _ {h + 1} [ i ] \mid s _ {h} \left[ \mathrm {p a} _ {i} \right], a _ {h}\right),
$$
where  $\mathrm{pa}_i\subseteq [m]$  is the parent set of  $i$ . The reward function is similarly factored:
$$
r _ {h} (s) := \sum_ {i = 1} ^ {m} r _ {h} ^ {i} (s [ i ]).
$$
Define  $B \coloneqq \sum_{i=1}^{m} |\mathcal{X}|^{\mathrm{pa}_i}$ . Factored MDPs satisfy (with  $|\Pi_{\mathrm{exp}}| = 1$  and  $d_{\mathcal{P}} = m^2 |\mathcal{A}|^2 B^2 \mathrm{poly}(H)$ )
$$
\xi \left(d _ {\mathcal {P}}, T, \Delta , \left| \Pi_ {\exp} \right|\right) \leq \sqrt {d _ {\mathcal {P}} \Delta T} + A B ^ {2} \operatorname {p o l y} (H).
$$
Moreover  $\ln |\mathcal{P}|\leq mbAB$ ,  $\ln |\mathcal{R}|\leq mb|\mathcal{X}|$ , where  $b$  is the number of bits needed to specify each entry of  $\mathbb{P}(s_{h + 1}||s_h,a_h)$  or  $r_h(s)^6$ . Therefore Theorem D.1 implies a sample complexity of
$$
\operatorname {p o l y} (H) \cdot \widetilde {\mathcal {O}} \left(\frac {b m ^ {3} | \mathcal {A} | ^ {3} B ^ {3}}{\epsilon^ {2}} + \frac {b m ^ {2} | \mathcal {X} |}{\alpha^ {2} \epsilon^ {2}}\right).
$$
To proceed, we define partially observable Markov decision process (POMDPs).
Definition 6. In a POMDP, states are hidden from the learner and only observations emitted by states can be observed. Formally, at each step  $h \in [H]$ , the learner observes  $o_h \sim \mathbb{O}_h(\cdot \mid s_h)$  where  $s_h$  is the current state and  $\mathbb{O}_h(\cdot \mid s_h) \in \Delta_A$  is the observation distribution conditioning on the current state being  $s_h$ . Then the learner takes action  $a_h$  and the environment transitions to  $s_{h+1} \sim \mathbb{P}_h(\cdot \mid s_h, a_h)$ .
Liu et al. [2022a] prove that the following subclasses of POMDPs satisfy Assumption 3 with moderate  $d_{\mathcal{P}}$  and  $|\Pi_{\mathrm{exp}}|$ .
Example 7 ( $\alpha$ -observable POMDPs). We say a POMDP is  $\alpha$ -observable if for every  $\mu, \mu' \in \Delta_{S}$ ,
$$
\min _ {h} \| \mathbb {E} _ {s \sim \mu} [ \mathbb {O} _ {h} (\cdot | s) ] - \mathbb {E} _ {s \sim \mu^ {\prime}} [ \mathbb {O} _ {h} (\cdot | s) ] \| _ {1} \geq \alpha \| \mu - \mu^ {\prime} \| _ {1}.
$$
Intuitively, the above relation implies that different state distributions will induce different observation distributions, and the parameter  $\alpha$  measures the amount of information preserved after mapping states to observations. It is proved that  $\alpha$ -observable POMDPs satisfy Condition 3 with  $\Pi_{\mathrm{exp}}(\pi) = \{\pi\}$  and  $d_{\mathcal{P}} = \mathrm{poly}(S, A, \alpha^{-1}, H)$  [Liu et al., 2022a].
For simplicity of notations, let  $u(h) = \max \{1, h - m + 1\}$ .
Example 8 (m-step decodable POMDPs). We say a POMDP is  $m$ -step decodable if there exists a set of decoder functions  $\{\phi_h\}_{h \in [H]}$  such that for every  $h \in [H]$ ,  $s_h = \phi_h((o, a)_{u(h):h-1}, o_h)$ . In other words, the current state can be uniquely identified from the most recent  $m$ -step observations and actions. It is proved that  $\alpha$ -observable POMDPs satisfy Condition 3 with  $|\Pi_{\mathrm{exp}}| = A^m$  and  $d_{\mathcal{P}} = \mathrm{poly}(L, A^m, H)$  where  $L = \max_h \mathrm{rank}(\mathbb{P}_h)$  denotes the rank of the transition matrices  $\{\mathbb{P}_h\}_{h \in [H]} \subseteq \mathbb{R}^{SA \times S}$  [Liu et al., 2022a].
<sup>6</sup>We can deal with continuous model classes if we use the bracketing number instead of cardinality in Theorem D.1
D.3 Proof of Theorem D.1
We first define some useful notations. Denote by  $\mathcal{D}_{\mathrm{rwd}}^t$ ,  $\mathcal{D}_{\mathrm{trans}}^t$  the reward, transition dataset at the end of the  $t$ -th iteration. We further denote
$$
\mathcal{B}_{\mathcal{R}}^{t}\gets \big\{r\in \mathcal{R}:\max_{(\tau ,\hat{r})\in \mathcal{D}_{\mathrm{rd}}^{t - 1}}|r(\tau) - \hat{r} |\leq \epsilon^{\prime}\big\} ,
$$
$$
\mathcal {B} _ {\mathcal {P}} ^ {t} \leftarrow \big \{p \in \mathcal {P}: \mathcal {L} (p, \mathcal {D} _ {\mathrm {t r a n s}} ^ {t - 1}) > \max  _ {p ^ {\prime} \in \mathcal {P}} \mathcal {L} (p ^ {\prime}, \mathcal {D} _ {\mathrm {t r a n s}} ^ {t - 1}) - \beta_ {\mathcal {P}} \big \}.
$$
By the definition of  $\mathcal{B}^t$ , we have that  $\mathcal{B}^t = \mathcal{B}_R^t \times \mathcal{B}_P^t$ . Denote by  $(\tau^t, \hat{r}^t)$  the trajectory-reward pair added into  $\mathcal{D}_{\mathrm{rwd}}$  in the  $t$ -th iteration of Algorithm 3.
By the definition of  $\mathcal{B}_{\mathcal{R}}^t$  and the fact that all reward feedback is at most  $\epsilon^\prime$ -corrupted, we directly have that the confidence set  $\mathcal{B}_{\mathcal{R}}^t$  always contain the groundtruth reward function.
Lemma D.2. For all  $t\in [T]$ $r^{\star}\in \mathcal{B}_{\mathcal{R}}^{t}$
Moreover, according to Liu et al. [2022a], the transition confidence set  $\mathcal{B}_{\mathcal{P}}^{t}$  always satisfies the following properties.
Lemma D.3 (Liu et al. [2022a]). There exists absolute constant  $c_2$  such that under Assumption 3 and the same choice of  $\beta_{\mathcal{P}}$  as in Theorem D.1, we have that with probability at least  $1 - \delta$ :
-  $p^{\star} \in \mathcal{B}_{\mathcal{P}}^{t}$ , for all  $t \in [T]$
-  $\sum_{t=1}^{T} \max_{p \in \mathcal{B}_{\mathcal{P}}^{t}} d_{\mathrm{TV}}(\mathbb{P}_{p}^{\pi^{t}}, \mathbb{P}_{p^{\star}}^{\pi^{t}}) \leq \xi(d_{\mathcal{P}}, T, c_{2}\beta_{\mathcal{P}}, |\Pi_{\exp}|)$ .
The first relation states that the transition confidence set contains the groundtruth transition model with high probability. And the second one states that if we use an arbitrary model  $p \in \mathcal{B}_{\mathcal{P}}^{t}$  to predict the transition dynamics under policy  $\pi^t$ , then the cumulative prediction error over  $T$  iterations is upper bounded by function  $\xi$ .
Proof of Theorem D.1. In the following proof, we will assume the two relations in Lemma D.3 hold. We have that
$$
\begin{array}{l} \sum_ {t} \left(V _ {r ^ {\star}, p ^ {\star}} ^ {\star} - V _ {r ^ {\star}, p ^ {\star}} ^ {\pi^ {t}}\right) \\ \leq \sum_ {t} \left(V _ {r ^ {t}, p ^ {t}} ^ {\pi^ {t}} - V _ {r ^ {\star}, p ^ {\star}} ^ {\pi^ {t}}\right) \\ \leq 2 H \sum_ {t} d _ {\mathrm {T V}} \left(\mathbb {P} _ {p ^ {t}} ^ {\pi^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\pi^ {t}}\right) + \sum_ {t} \left(V _ {r ^ {t}, p ^ {\star}} ^ {\pi^ {t}} - V _ {r ^ {\star}, p ^ {\star}} ^ {\pi^ {t}}\right) \\ \leq 2 H \sum_ {t} d _ {\mathrm {T V}} \left(\mathbb {P} _ {p ^ {t}} ^ {\pi^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\pi^ {t}}\right) + \sum_ {t} \left| r ^ {t} \left(\tau^ {t}\right) - r ^ {\star} \left(\tau^ {t}\right) \right| + \mathcal {O} \left(H \sqrt {T \ln (1 / \delta)}\right) \\ \leq 2 H \xi \left(d _ {\mathcal {P}}, T, c _ {2} \beta_ {\mathcal {P}}, | \Pi_ {\exp} |\right) + \sum_ {t} \left| r ^ {t} \left(\tau^ {t}\right) - r ^ {\star} \left(\tau^ {t}\right) \right| + \mathcal {O} \left(H \sqrt {T \ln (1 / \delta)}\right) \\ \leq 2 H \xi (d _ {\mathcal {P}}, T, c _ {2} \beta_ {\mathcal {P}}, | \Pi_ {\exp} |) + \mathcal {O} (T \sqrt {d _ {\mathcal {R}}} \epsilon^ {\prime} + d _ {\mathcal {R}}) + \mathcal {O} (H \sqrt {T \ln (1 / \delta)}), \\ \end{array}
$$
where the first inequality uses the definition of  $(\pi^t, r^t, p^t)$  and the relation  $(r^t, p^t) \in \mathcal{B}^t$ , the third one holds with probability at least  $1 - \delta$  by Azuma-Hoeffding inequality, the fourth one uses the second relation in Lemma D.3, and the last one invokes the standard regret guarantee for eluder dimension (e.g., Russo and Van Roy [2013]) where  $d_{\mathcal{R}} = \dim_{\mathrm{E}}(\mathcal{R}, \epsilon'/2)$ .
E Additional details and proofs for Section 4.2
E.1 Algorithm details
In Algorithm 4, we describe how to learn an approximate von Neumann winner via running two copies of an arbitrary adversarial MDP algorithm. Specifically, we maintain two algorithm instances  $\mathcal{A}^{(1)}$  and  $\mathcal{A}^{(2)}$ . In the  $k$ -th iteration, we first sample two trajectories  $(s_{1:H}^{(1)}, a_{1:H}^{(1)})$  and  $(s_{1:H}^{(2)}, a_{1:H}^{(2)})$
without reward by executing  $\pi_k^{(1)}$  and  $\pi_k^{(2)}$ , the two output policies of  $\mathcal{A}^{(1)}$  and  $\mathcal{A}^{(2)}$ , respectively. Then we input  $(s_H^{(1)}, s_H^{(2)})$  into the comparison oracle and get a binary feedback  $y$ . After that, we augment  $(s_{1:H-1}^{(1)}, a_{1:H-1}^{(1)})$  with zero reward at the first  $H-2$  steps and reward  $y$  at step  $H-1$ , which we feed into  $\mathcal{A}^{(1)}$ . Similarly, we create feedback for  $\mathcal{A}^{(2)}$  by using  $(s_{1:H-1}^{(2)}, a_{1:H-1}^{(2)})$  and reward  $1-y$ . The final output policy  $\bar{\pi}^{(1)}$  is a uniform mixture of all the policies  $\mathcal{A}^{(1)}$  has produced during  $K$  iterations.
Algorithm 4 Learning von Neumann Winner via Adversarial MDP Algorithms
Initialize two algorithm instances  $\mathcal{A}^{(1)},\mathcal{A}^{(2)}$  for adversarial MDPs with horizon length  $H = 1,\dots ,K$  do   
Receive  $\pi_k^{(1)}$  from  $\mathcal{A}^{(1)}$  and  $\pi_k^{(2)}$  from  $\mathcal{A}^{(2)}$    
Sample  $(s_{1:H}^{(1)},a_{1:H}^{(1)})\sim \pi_k^{(1)}$  and  $(s_{1:H}^{(2)},a_{1:H}^{(2)})\sim \pi_k^{(2)}$    
Query comparison oracle  $y\sim \mathrm{Ber}(M(s_H^{(1)},s_H^{(2)}))$    
Return feedback  $(s_1^{(1)},a_1^{(1)},0,\ldots ,s_{H - 2}^{(1)},a_{H - 2}^{(1)},0,s_{H - 1}^{(1)},a_{H - 1}^{(1)},y)$  to  $\mathcal{A}_1$  and  $(s_1^{(2)},a_1^{(2)},0,\ldots ,s_{H - 2}^{(2)},a_{H - 2}^{(2)},0,s_{H - 1}^{(2)},a_{H - 1}^{(2)},1 - y)$  to  $\mathcal{A}_2$  , respectively   
Output average policy mixture  $\bar{\pi}^{(1)}$  where  $\bar{\pi}^{(1)}\coloneqq \mathrm{Unif}(\{\pi_k^{(1)}\}_{k\in [K]})$
Converting the output policy to a Markov policy. Note that the output policy  $\bar{\pi}$  is a general non-Markov policy. However, we can convert it to a Markov policy in a sample-efficient manner for tabular MDPs through a simple procedure: execute  $\bar{\pi}$  for  $N$  episodes, then compute the empirical policy
$$
\hat {\pi} _ {h} (a | s) := \frac {J _ {h} (s , a)}{J _ {h} (s)},
$$
where  $J_{h}(s,a)$  and  $J_{h}(s)$  denote the visitation counters at state-action pair  $(s,a)$  and state  $s$  at step  $h$ , respectively. The following lemma claims that the resulted Markov policy  $\hat{\pi}$  is also an approximate restricted Nash equilibrium.
Lemma E.1. If  $\bar{\pi}$  is an  $\epsilon$ -approximate von Neumann winner, then  $\hat{\pi}$  is a  $2\epsilon$ -approximate von Neumann winner with probability at least  $1 - \delta$ , provided that  $N = \widetilde{\Omega}\left(\frac{SA}{\epsilon^2}\right)$ .
E.2 Proof of Theorem 11
Proof. Define  $U(\pi, \pi') \coloneqq \mathbb{E}_{s_H \sim \pi, s_H' \sim \pi'} M[s_H, s_H']$ . The AMDP regret of  $\mathcal{A}^{(1)}$  gives
$$
\begin{array}{l} \sum_ {k} U \left(\pi_ {k} ^ {(1)}, \pi_ {k} ^ {(2)}\right) \geq \max  _ {\pi : \operatorname {M a r k o v}} \sum_ {k} U \left(\pi , \pi_ {k} ^ {(2)}\right) - \beta K ^ {1 - c} \\ = \max  _ {\pi : \text {g e n e r a l}} \sum_ {k} U \left(\pi , \pi_ {k} ^ {(2)}\right) - \beta K ^ {1 - c} \\ \end{array}
$$
where the second inequality uses the fact that there always exists a Markov best response because only the state distribution at the final step matters in the definition of  $U$ .
Similarly the regret of  $\mathcal{A}^{(2)}$  gives
$$
\sum_ {k} \left(1 - U \left(\pi_ {k} ^ {(1)}, \pi_ {k} ^ {(2)}\right)\right) \geq \max  _ {\pi} \sum_ {k} \left[ 1 - U \left(\pi_ {k} ^ {(1)}, \pi\right) \right] - \beta K ^ {1 - c}.
$$
Summing the two inequalities gives
$$
\min _ {\pi^ {\prime}} U \left(\bar {\pi} ^ {(1)}, \pi^ {\prime}\right) \geq \max _ {\pi} U (\pi , \bar {\pi} ^ {(2)}) - 2 \beta K ^ {1 - c}.
$$
$$
\mathrm {D G a p} \left(\bar {\pi} ^ {(1)}, \bar {\pi} ^ {(2)}\right) \leq 2 \beta / K ^ {c}.
$$
Set  $\hat{\pi}_h(\cdot |s) = \mathrm{Unif}(\mathcal{A})$  if  $J_{h}(s) = 0$ , i.e. if state  $s$  is never visited at step  $h$ .
By the symmetry of preference function, we further have
$$
\begin{array}{l} \mathrm {D G a p} (\bar {\pi} ^ {(1)}, \bar {\pi} ^ {(1)}) = \max _ {\pi^ {\prime}} U (\pi^ {\prime}, \bar {\pi} _ {1}) - \min _ {\pi^ {\prime}} U (\bar {\pi} _ {1}, \pi^ {\prime}) = 2 \max _ {\pi^ {\prime}} U (\pi^ {\prime}, \bar {\pi} _ {1}) - 1 \\ = 2 \left(\max  _ {\pi^ {\prime}} U \left(\pi^ {\prime}, \bar {\pi} _ {1}\right) - U \left(\bar {\pi} _ {1}, \bar {\pi} _ {1}\right)\right) \\ \leq 2 \mathrm {D G a p} (\bar {\pi} ^ {(1)}, \bar {\pi} ^ {(2)}) \leq 4 \beta / K ^ {c}. \\ \end{array}
$$
E.3 Details for adversarial linear MDPs.
To apply the regret guarantees from existing works on adversarial linear MDP [e.g., Sherman et al., 2023], we need to show the constructed reward signal in Algorithm 4 is linearly realizable. Since the reward signal is zero for the first  $H - 2$  steps, we only need to consider step  $H - 1$ . Recall the definition of linear MDPs requires that there exist feature mappings  $\phi$  and  $\psi$  such that  $\mathbb{P}_h(s' \mid s, a) = \langle \phi_h(s, a), \psi(s') \rangle$ . By the bilinear structure of transition, the conditional expectation of reward at step  $H - 1$  in the  $k$ -th iteration can be written as
$$
\begin{array}{l} \mathbb {E} [ y \mid s _ {H - 1} ^ {(1)}, a _ {H - 1} ^ {(1)} ] = \mathbb {E} [ M (s _ {H} ^ {(1)}, s _ {H} ^ {(2)}) \mid s _ {H} ^ {(1)} \sim \mathbb {P} _ {h} (\cdot \mid s _ {H - 1} ^ {(1)}, a _ {H - 1} ^ {(1)}), s _ {H} ^ {(2)} \sim \pi_ {k} ^ {(2)} ] \\ = \left\langle \phi_ {H - 1} (s _ {H - 1} ^ {(1)}, a _ {H - 1} ^ {(1)}), \sum_ {s \in \mathcal {S}} \psi_ {H - 1} (s) \mathbb {E} [ M (s, s _ {H} ^ {(2)}) \mid s _ {H} ^ {(2)} \sim \pi_ {k} ^ {(2)} ] \right\rangle . \\ \end{array}
$$
Therefore, the reward function constructed for  $\mathcal{A}^{(1)}$  is a linear in the feature mapping  $\phi$ . Similarly, we can show the reward function constructed for  $\mathcal{A}^{(2)}$  is also linear.
E.4 Proof of Lemma E.1
Proof. Let  $\iota = c\ln (SAHN / \delta)$  where  $c$  is a large absolute constant. By standard concentration, with probability at least  $1 - \delta$ , we have that for all  $s$ , if  $\mathbb{P}^{\bar{\pi}}(s_h = s) \geq \iota / N$ , then  $J_{h}(s) \geq N\mathbb{P}^{\bar{\pi}}(s_{h} = s) / 2$ . With slight abuse of notation, we define  $\bar{\pi}_h(\cdot \mid s_h) = \sum_{a_{1:h-1}, s_{1:h-1}} \bar{\pi}_h(\cdot \mid a_{1:h-1}, s_{1:h})$ . Therefore, we have