SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Example 1 (Tabular MDPs). Our first example is tabular MDPs whose state and action spaces are finite and small. In this case  $d_{\overline{\mathcal{R}}} = \widetilde{\mathcal{O}}(|S||\mathcal{A}|)$ . The UCBVI-BF algorithm, proposed by Azar et al. [2017], is a model-based tabular RL algorithm which uses upper-confidence bound value iteration with Bernstein-Freedman bonuses. UCBVI-BF has sample complexity  $\mathcal{C}(\epsilon, \delta) = \mathcal{O}\left(H^3 |\mathcal{S}||\mathcal{A}| / \epsilon^2\right)$  and is  $\mathcal{O}(\epsilon / H)$  robust due to Lemma B.4.
Proposition 5. Algorithm 1 with UCBVI-BF learns an  $\epsilon$ -optimal policy of a tabular MDP from preference feedback using  $\widetilde{\mathcal{O}}\left(\frac{H^3|\mathcal{S}||\mathcal{A}|}{\epsilon^2}\right)$  episodes of interaction with the environment and  $\widetilde{\mathcal{O}}\left(\frac{H^2|\mathcal{S}|^2|\mathcal{A}|^2}{\alpha^2\epsilon^2}\right)$  queries to the comparison oracle. The algorithm is computationally efficient.
Example 2 (Low Bellman-eluder dimension). Bellman-eluder dimension [Jin et al., 2021] is a complexity measure for function approximation RL with a Q-value function class  $\mathcal{F}$ . It can be
shown that a large class of RL problems, including tabular MDPs, linear MDPs, reactive POMDPs and low Bellman rank problems, all have small Bellman-eluder dimension. Furthermore, Jin et al. [2021] designed an algorithm, named GOLF, which (i) first constructs a confidence set for the optimal Q-functions by including all the candidates with small temporal difference loss, (ii) then optimistically picks Q-estimate from the confidence set and executes its greedy policy, and (iii) repeats (i) and (ii) using the newly collected data. Assuming that  $\mathcal{F}$  satisfies realizability and completeness property, GOLF is  $g(\epsilon) = \Theta\left(\frac{\epsilon}{\sqrt{d_{\mathrm{BE}}H^2}}\right)$ -robust with sample complexity  $\mathcal{C}(\epsilon,\delta) = \widetilde{\mathcal{O}}\left(\frac{d_{\mathrm{BE}}H^4\ln|\mathcal{F}|}{\epsilon^2}\right)$  where  $d_{\mathrm{BE}}$  is the Bellman-eluder dimension of the problem. By applying Theorem 4, we immediately have the following result.
Proposition 6. Algorithm 1 with GOLF [Jin et al., 2021] learns an  $\epsilon$ -optimal policy of RL problems with Bellman-Eluder dimension  $d_{\mathrm{BE}}$  in  $\widetilde{\mathcal{O}}\left(\frac{d_{\mathrm{BE}}H^4\ln|\mathcal{F}|}{\epsilon^2}\right)$  episodes of interaction with the environment and  $\widetilde{\mathcal{O}}\left(\frac{d_{\mathrm{BE}}d_{\mathcal{R}}^2H^2}{\alpha^2\epsilon^2}\right)$  queries to the comparison oracle.
Trajectory-based preferences When the reward function is trajectory-based, we can instantiate P2R with the OMLE algorithm [Liu et al., 2022a] to solve any model-based RLHF of low generalized eluder dimension. In brief, OMLE is an optimism-based algorithm that maintains a model confidence set. This set comprises model candidates that demonstrate high log-likelihood on previously collected data. And in each iteration, the algorithm chooses a model estimate optimistically and executes its greedy policy to collect new data.
Example 3 (Low generalized eluder dimension). Generalized eluder dimension [Liu et al., 2022a] is a complexity measure for function approximation RL with a transition function class  $\mathcal{P}$ . In Appendix D.1, we show that a simple adaptation of OMLE is  $g(\epsilon) = \Theta\left(\frac{\epsilon}{\sqrt{d_{\mathcal{R}}}}\right)$ -robust with sample complexity  $\mathcal{C}(\epsilon, \delta) = \widetilde{\mathcal{O}}\left(\frac{H^2d_{\mathcal{P}}|\Pi_{\mathrm{exp}}|^2\ln|\mathcal{P}|}{\epsilon^2} + \frac{Hd_{\mathcal{R}}|\Pi_{\mathrm{exp}}|}{\epsilon}\right)$ , where  $d_{\mathcal{P}}$  denotes the generalized eluder dimension of  $\mathcal{P}$ ,  $|\Pi_{\mathrm{exp}}|$  is a parameter in the OMLE algorithm, and  $d_{\mathcal{R}} = \dim_{\mathrm{E}}(\mathcal{R}, \epsilon)$ . Plugging it back into Theorem 4, we obtain the following result.
Proposition 7. Algorithm 1 with OMLE learns an  $\epsilon$ -optimal policy of RL problems with low generalized eluder dimension in  $\widetilde{\mathcal{O}}\left(\frac{H^2d_{\mathcal{P}}|\Pi_{\mathrm{exp}}|^2\ln|\mathcal{P}|}{\epsilon^2} + \frac{Hd_{\mathcal{R}}|\Pi_{\mathrm{exp}}|}{\epsilon}\right)$  episodes of interaction with the environment and  $\widetilde{\mathcal{O}}\left(\frac{d_{\mathcal{R}}d_{\mathcal{R}}^2}{\alpha^2\epsilon^2}\right)$  queries to the comparison oracle.
Liu et al. [2022a] prove that a wide range of model-based reinforcement learning problems have a low generalized eluder dimension  $d_{\mathcal{P}}$  and only require a mild  $|\Pi_{\mathrm{exp}}|$  to run the OMLE algorithm. Examples of such problems include tabular MDPs, factored MDPs, observable POMDPs, and decodable POMDPs. For a formal definition of generalized eluder dimension and more details on the aforementioned bound and examples, we refer interested readers to Appendix D.1 or Liu et al. [2022a]. Finally, we remark that it is possible to apply the P2R framework in other settings with different complexity measures, such as DEC [Foster et al., 2021] or GEC [Zhong et al., 2022], by making some minor modifications to the corresponding algorithms to ensure robustness.
3.3 Extension to  $K$ -wise comparison
In this subsection, we briefly discuss how our results can be extended to  $K$ -wise comparison assuming the following Plackett-Luce (PL) model [Luce, 1959, Plackett, 1975] of  $K$  item preferences.
Definition 4 (Plackett-Luce model). The oracle takes in  $K$  trajectories  $\tau_{1},\ldots ,\tau_{K}$  and outputs a permutation  $\phi :[K]\to [K]$  with probability  $\mathbb{P}(\phi) = \prod_{k = 1}^{K}\frac{\exp(\eta\cdot r(\tau_{\phi^{-1}(k)}))}{\sum_{t = k}^{K}\exp(\eta\cdot r(\tau_{\phi^{-1}(t)}))}$ .
Note that when  $K = 2$ , the above PL model reduces to a pair-wise trajectory-type comparison oracle (Definition 1) with  $\sigma(x) = \exp(\eta x) / (\exp(\eta x) + 1)$  which satisfies Assumption 2 with  $\alpha = \Theta(\eta \exp(-\eta H))$ . The PL model satisfies the following useful property, which is a corollary of its internal consistency [Hunter, 2004].
Property 8 (Hunter [2004, p396]). For any disjoint sets  $\{i_1,j_1\},\ldots ,\{i_k,j_k\} \subseteq [K]$ , the following pair-wise comparisons are mutually independent:  $\mathbf{1}(\phi (i_1) > \phi (j_1)),\ldots ,\mathbf{1}(\phi (i_k) > \phi (j_k))$  where  $\phi$  is a permutation sampled from  $PL(\tau_{1},\dots,\tau_{K})$ . Moreover,  $\mathbf{1}(\phi (i_m) < \phi (j_m))\sim \mathrm{Ber}(\sigma (r(\tau_{i_m}) - r(\tau_{j_m})))$ , where  $\sigma (x) = \exp (\eta x) / (\exp (\eta x) + 1)$ .
This property enables "batch querying" the preferences on  $\lfloor K / 2\rfloor$  pairs of  $(\tau_{1},\tau_{2})$  in parallel, which returns  $\lfloor K / 2\rfloor$  independent pairwise comparisons outcomes. This would enable us to reduce the number of queries by a factor of  $\Omega (K)$  for small  $K$  in both Algorithm 1 and 3.
Theorem 9 (P2R with  $K$ -wise comparison). Suppose  $\mathcal{A}$  is an  $g(\epsilon)$ -robust RL algorithm with sample complexity  $\mathcal{C}(\epsilon, \delta)$ , and assume the same conditions and the same choice of parameters as in Theorem 4. By running  $\mathcal{A}$  with the interface in Algorithm 1, we can learn an  $\epsilon$ -optimal policy using  $\mathcal{C}(\epsilon, \delta)$  samples and  $\widetilde{\mathcal{O}}\left(\frac{d_{\overline{R}}^2}{\alpha^2 g(\epsilon)^2 \min\{K, m\}}\right)$  queries to the  $K$ -wise comparison oracle with probability  $1 - 2\delta$ .
Theorem 9 is a direct consequence of Theorem 4: If  $K \geq 2m$ , we can obtain  $m$  independent comparisons between two trajectories by a single query to the  $K$ -wise comparison oracle and therefore reduce the overall query complexity in Theorem 4 by a factor of  $m$ ; otherwise, we can get  $m$  independent comparisons by making  $\mathcal{O}(m / K)$  queries to the  $K$ -wise comparison oracle, which reduces the overall query complexity by a factor of  $K$ .
However, the above parallelization benefits of using  $K$ -wise comparison might be an artifact of the PL model: it seems improbable that the same human evaluator would independently rank  $\lfloor K / 2 \rfloor$  copies of item  $A$  and item  $B$ . It remains an interesting problem to develop  $K$ -wise comparison models more suitable to RLHF.
4 RLHF From General Preferences
The utility-based approach imposes strong assumptions on human preferences. Not only is the matrix  $M[\tau, \tau']$  in (1) assumed to be exactly realizable by  $\sigma(r(\tau) - r(\tau'))$ , but also  $\sigma$  is assumed to be known and have a gradient lower bound. Moreover, the utility-based approach assumes that transitivity: if  $\operatorname*{Pr}[\tau_1 \succ \tau_2] \geq 0.5$ ,  $\operatorname*{Pr}[\tau_2 \succ \tau_3] \geq 0.5$ , then  $\operatorname*{Pr}[\tau_1 \succ \tau_3] \geq 0.5$ . However, experiments have shown that human preferences can be intransitive [Tversky, 1969]. These limitations of the utility-based approach motivate us to consider general preferences.
A general preference may not be realizable by a utility model, so we cannot define the optimal policy in the usual sense. Instead, we follow Dudík et al. [2015] and consider an alternative solution concept, the von Neumann winner.
Definition 5.  $\pi^{\star}$  is the von Neumann winner policy if  $(\pi^{\star},\pi^{\star})$  is a symmetric Nash equilibrium of the constant-sum game:  $\max_{\pi}\min_{\pi '}\mathbb{E}_{\tau \sim \pi ,\tau '\sim \pi '}M[\tau ,\tau ']$ .
The duality gap of the game is defined as
$$
\operatorname {D G a p} \left(\pi_ {1}, \pi_ {2}\right) := \max  _ {\pi} \mathbb {E} _ {\tau \sim \pi , \tau^ {\prime} \sim \pi_ {2}} M \left[ \tau , \tau^ {\prime} \right] - \min  _ {\pi} \mathbb {E} _ {\tau \sim \pi_ {1}, \tau^ {\prime} \sim \pi} M \left[ \tau , \tau^ {\prime} \right].
$$
We say that  $\pi$  is an  $\epsilon$ -approximate von Neumann winner if the duality gap of  $(\pi, \pi)$  is at most  $\epsilon$ . The von Neumann winner has been studied under the name of maximal lotteries in the context of social choice theory [Kreweras, 1965, Fishburn, 1984]. The von Neumann winner is a natural generalization of the optimal policy concept for non-utility based preferences. It is known that
- Intuitively, the von Neumann winner  $\pi^{\star}$  is a randomized policy that "beats" any other policy  $\pi'$  in the sense that  $\mathbb{E}_{\tau \sim \pi^{\star}, \tau' \sim \pi'} M[\tau, \tau'] \geq 1/2$ ;
- If the utility-based preference model holds and the transitions are deterministic, the von Neumann winner is the optimal policy;
- The von Neumann winner is the only solution concept that satisfies population-consistency and composition-consistency in social choice theory [Brandl et al., 2016].
Finding the von Neumann winner seems prima facie quite different from standard RL tasks. However, in this section we will show how finding the von Neumann winner can be reduced to finding the restricted Nash equilibrium in a type of Markov games. For preferences based on the final state, we can further reduce the problem to RL in adversarial MDP.
4.1 A reduction to Markov games
Factorized and independent Markov games (FI-MG). Consider a two-player zero-sum Markov games with state space  $S = S^{(1)} \times S^{(2)}$ , action space  $\mathcal{A}^{(1)}$  and  $\mathcal{A}^{(2)}$  for each player respectively,
transition kernel  $\{\mathbb{P}_h\}_{h\in [H]}$  and reward function  $r$ . We say the Markov game is factorized and independent if the transition kernel is factorized:
$$
\mathbb {P} _ {h} (s _ {h + 1} \mid s _ {h}, a _ {h}) = \mathbb {P} _ {h} (s _ {h + 1} ^ {(1)} \mid s _ {h} ^ {(1)}, a _ {h} ^ {(1)}) \times \mathbb {P} _ {h} (s _ {h + 1} ^ {(2)} \mid s _ {h} ^ {(2)}, a _ {h} ^ {(2)}),
$$
where  $s_h = (s_h^{(1)}, s_h^{(2)})$ ,  $s_{h+1} = (s_{h+1}^{(1)}, s_{h+1}^{(2)})$ ,  $a_h = (a_h^{(1)}, a_h^{(2)}) \in \mathcal{A}^{(1)} \times \mathcal{A}^{(2)}$ .
The above definition implies that the Markov game can be partitioned into two MDPs, where the transition dynamics are controlled separately by each player, and are completely independent of each other. The only source of correlation between the two MDPs arises from the reward function, which is permitted to depend on the joint trajectory from both MDPs. Building on the above factorization structure, we define partial trajectory  $\tau_{i,h} \coloneqq (s_1^{(i)}, a_1^{(i)}, \ldots, s_h^{(i)})$  that consists of states of the  $i$ -th MDP factor and actions of the  $i$ -th player. Furthermore, we define a restricted policy class  $\Pi_i$  that contains all policies that map a partial trajectory to a distribution in  $\Delta_{A_i}$ , i.e.,
$$
\Pi_ {i} := \left\{\left\{\pi_ {h} \right\} _ {h \in [ H ]}: \pi_ {h} \in \left(\left(S ^ {(i)} \times \mathcal {A} _ {i}\right) ^ {h - 1} \times S ^ {(i)} \rightarrow \Delta_ {\mathcal {A} _ {i}}\right)\right\} \text {f o r} i \in [ 2 ].
$$
And the goal is to learn a restricted Nash equilibrium  $(\mu^{\star},\nu^{\star})\in \Pi_{1}\times \Pi_{2}$  such that
$$
\mu^ {\star} \in \arg \max  _ {\mu \in \Pi_ {1}} \mathbb {E} _ {\tau \sim \mu , \tau^ {\prime} \sim \nu^ {\star}} [ r (\tau , \tau^ {\prime}) ] \quad \text {a n d} \quad \nu^ {\star} \in \arg \min  _ {\nu \in \Pi_ {2}} \mathbb {E} _ {\tau \sim \mu^ {\star}, \tau^ {\prime} \sim \nu} [ r (\tau , \tau^ {\prime}) ].
$$
Finding von Neumann winner via learning restricted Nash. We claim that finding an approximate von Neumann winner can be reduced to learning an approximate restricted Nash equilibrium in a FI-MG. The reduction is straightforward: we simply create a Markov game that consists of two independent copies of the original MDP and control the dynamics in the  $i$ -th copy by the  $i$ -th player's actions. Such construction is clearly factorized and independent. Moreover, the restricted policy class  $\Pi_{i}$  is equivalent to the universal policy class in the original MDP. We further define the reward function as  $r(\tau, \tau') = M[\tau, \tau']$  where  $M$  is the general preference function. By definition 5, we immediately obtain the following equivalence relation.
Proposition 10. If  $(\mu^{\star},\nu^{\star})$  is a restricted Nash equilibrium of the above FI-MG, then both  $\mu^{\star}$  and  $\nu^{\star}$  are von Neumann winner in the original problem.
The problem we are faced with now is how to learn restricted Nash equilibria in FI-MG. In the following sections, we present two approaches that leverage existing RL algorithms to solve this problem: (i) when the preference function depends solely on the final states of the two input trajectories, each player can independently execute an adversarial MDP algorithm; (ii) for general preference functions, a straightforward adaptation of the OMLE algorithm is sufficient under certain eluder-type conditions.
4.2 Learning from final-state-based preferences via adversarial MDPs
In this section, we consider a special case where the preference depends solely on the final states of the two input trajectories, i.e.,  $M(\tau, \tau') = M(s_H, s_H')$ . Given the previous equivalence relation between von Neumann winner and restricted Nash in FI-MG, one natural idea is to apply no-regret learning algorithms, as it is well-known that running two copies of no-regret online learning algorithms against each other can be used to compute Nash equilibria in zero-sum normal-form games. Since this paper focuses on sequential decision making, we need no-regret learning algorithms for adversarial MDPs, which we define below.
Adversarial MDPs. In the adversarial MDP problem, the algorithm interacts with a series of MDPs with the same unknown transition but adversarially chosen rewards for each episode. Formally, there exists an unknown groundtruth transition function  $\mathbb{P} = \{\mathbb{P}_h\}_{h=1}^H$ . At the beginning of the  $k$ -th episode, the algorithm chooses a policy  $\pi^k$  and then the adversary picks a reward function  $r^k = \{r_h^k\}_{h=1}^H$ . After that, the algorithm observes a trajectory  $\tau^k = (s_1^k, a_1^k, y_1^k, \ldots, s_H^k, a_H^k, y_H^k)$  sampled from executing policy  $\pi^k$  in the MDP parameterized by  $\mathbb{P}$  and  $r^k$ , where  $\mathbb{E}[y_h^k \mid s_h^k, a_h^k] = r_h^k(s_h^k, a_h^k)$ . We define the regret of an adversarial MDP algorithm  $\mathcal{A}$  to be the gap between the algorithm's expected payoff and the best payoff achievable by the best fixed Markov policy:
$$
\mathrm {R e g r e t} _ {K} (\mathcal {A}) := \max _ {\pi \in \Pi_ {\mathrm {M a r k o v}}} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi} \left[ \sum_ {h = 1} ^ {H} r _ {h} ^ {k} (s _ {h}, a _ {h}) \right] - \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi_ {k}} \left[ \sum_ {h = 1} ^ {H} r _ {h} ^ {k} (s _ {h}, a _ {h}) \right].
$$
Now we explain how to learn a von Neumann winner via running adversarial MDP algorithms. We simply create two copies of the original MDP and instantiate two adversarial MDP algorithms  $\mathcal{A}_1$  and
$\mathcal{A}_2$  to control each of them separately. To execute  $\mathcal{A}_1$  and  $\mathcal{A}_2$ , we need to provide reward feedback to them in each  $k$ -th episode. Denote by  $s_H^{k,(1)}$  and  $s_H^{k,(2)}$  the final states  $\mathcal{A}_1$  and  $\mathcal{A}_2$  observe in the  $k$ -th episode. We will feed  $y^k \sim \mathrm{Ber}(M(s_H^{k,(1)}, s_H^{k,(2)}))$  into  $\mathcal{A}_1$  and  $1 - y^k$  into  $\mathcal{A}_2$  as their reward at step  $H - 1$ , respectively. And all other steps have zero reward feedback. The formal pseudocode is provided in Algorithm 4 (Appendix E). The following theorem states that as long as the invoked adversarial MDP algorithm has sublinear regret, the above scheme learns an approximate von Neumann winner in a sample-efficient manner.
Theorem 11. Suppose  $\operatorname{Regret}_K(\mathcal{A}) \leq \beta K^{1 - c}$  with probability at least  $1 - \delta$  for some  $c \in (0,1)$ . Then Algorithm 4 with  $K = (4\beta/\epsilon)^{1/c}$  outputs an  $\epsilon$ -approximate von Neumann winner with probability at least  $1 - 2\delta$ .
In order to demonstrate the applicability of Theorem 11, we offer two examples where sublinear regret can be achieved in adversarial MDPs via computationally efficient algorithms. The first one is adversarial tabular MDPs where the number of states and actions are finite, i.e.,  $|S|, |\mathcal{A}| \leq +\infty$ .
Example 4 (adversarial tabular MDPs). Jin et al. [2019] proposed an algorithm  $\mathcal{A}$  with  $\mathrm{Regret}_K(\mathcal{A}) \leq \widetilde{\mathcal{O}}(\sqrt{|S|^2|\mathcal{A}|H^3K})$ . Plugging it into Theorem 11 leads to  $K = \widetilde{\mathcal{O}}(|S|^2|\mathcal{A}|H^3/\epsilon^2)$  sample complexity and query complexity for learning  $\epsilon$ -approximate von Neumann winner.
The second example is adversarial linear MDPs where the number of states and actions can be infinitely large while the transition and reward functions admit special linear structure. See Sherman et al. [2023] for the precise formulation of the adversarial linear MDP problem.
Example 5 (adversarial linear MDPs). Sherman et al. [2023] proposed an algorithm  $\mathcal{A}$  with  $\mathrm{Regret}_K(\mathcal{A}) \leq \widetilde{\mathcal{O}}(dH^2 K^{6/7})$  for online learning in adversarial linear MDPs. $^3$  Combining it with Theorem 11 leads to  $K = \widetilde{\mathcal{O}}(d^7 H^{14} / \epsilon^7)$  sample complexity and query complexity for learning  $\epsilon$ -approximate restricted Nash equilibria.
4.3 Learning from trajectory-based preferences via OMLE_equilibrium
In this section, we consider the more general case where the preference  $M[\tau, \tau']$  is allowed to depend arbitrarily on the two input trajectories. Similar to the utility-based setting, we assume that the learner is provided with a preference class  $\mathcal{M} \subseteq ((\mathcal{S} \times \mathcal{A})^H \times (\mathcal{S} \times \mathcal{A})^H \to [0,1])$  and transition function class  $\mathcal{P}$  a priori, which contains the groundtruth preference and transition we are interacting with. Previously, we have established the reduction from learning the von Neumann winner to learning restricted Nash in FI-MG. In addition, learning restricted Nash in FI-MG is in fact a special case of learning Nash equilibrium in partially observable Markov games (POMGs). As a result, we can directly adapt the existing OMLE algorithm for learning Nash in POMGs [Liu et al., 2022b] to our setting, with only minor modifications required to learn the von Neumann winner. We defer the algorithmic details for this approach (Algorithm 5) to Appendix F, and present only the theoretical guarantee here.
Theorem 12. Suppose Assumption 3 holds. There exist absolute constant  $c_{1}$  and  $c_{2}$  such that for any  $(T,\delta)\in \mathbb{N}\times (0,1]$  if we choose  $\beta_{\mathcal{M}} = c_1\ln (|\mathcal{M}|T / \delta)$  and  $\beta_{\mathcal{P}} = c_1\ln (|\mathcal{M}|T / \delta)$  in Algorithm 5, then with probability at least  $1 - \delta$  the duality gap of the output policy of Algorithm 5 is at most
$$
w h e r e d _ {\mathcal {M}} = \dim_ {\mathrm {E}} (\mathcal {M}, 1 / T). \frac {4 \xi (d _ {\mathcal {P}} , T , c _ {2} \beta_ {\mathcal {P}} , | \Pi_ {\exp} |)}{T} + c _ {2} \sqrt {\frac {d _ {\mathcal {M}} \beta_ {\mathcal {M}}}{T}},
$$
It has been proven that a wide range of RL problems admit a regret formulation of  $\xi = \widetilde{\mathcal{O}} (\sqrt{d_{\mathcal{P}}\beta|\Pi_{\mathrm{exp}}|T})$  with mild  $d_{\mathcal{P}}$  and  $|\Pi_{\mathrm{exp}}|$  [Liu et al., 2022a]. These problems include, but are not limited to, tabular MDPs, factored MDPs, linear kernel MDPs, observable POMDPs, and decodable POMDPs. For more details, please refer to Appendix D.1 or Liu et al. [2022a]. For problems that satisfy  $\xi = \widetilde{\mathcal{O}} (\sqrt{d_{\mathcal{P}}\beta_{\mathcal{P}}|\Pi_{\mathrm{exp}}|T})$ , Theorem D.1 implies a sample complexity of
$$
\widetilde {\mathcal {O}} \left(\frac {d _ {\mathcal {P}} | \Pi_ {\mathrm {e x p}} | \ln | \mathcal {P} |}{\epsilon^ {2}} + \frac {d _ {\mathcal {M}} \cdot \ln | \mathcal {M} |}{\epsilon^ {2}}\right).
$$
The sample complexity for specific tractable problems can be derived by plugging their precise formulation of  $\xi$  (provided in Appendix D.1) into the above bound.
$^{3}$ Sherman et al. [2023] requires the adversarial reward function to be linear in the feature mapping of linear MDPs. In Appendix E, we show the reward signal we constructed in Algorithm 4 satisfies such requirement.
This work was partial supported by National Science Foundation Grant NSF-IIS-2107304 and Office of Naval Research Grant N00014-22-1-2253.