SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Josh Abramson, Arun Ahuja, Federico Carnevale, Petko Georgiev, Alex Goldin, Alden Hung, Jessica Landon, Jirka Lhotka, Timothy Lillicrap, Alistair Muldal, et al. Improving multimodal interactive agents with reinforcement learning from human feedback. arXiv preprint arXiv:2211.11602, 2022.
Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing dueling bandits to cardinal bandits. In International Conference on Machine Learning, pages 856-864. PMLR, 2014.
Riku Arakawa, Sosuke Kobayashi, Yuya Unno, Yuta Tsuboi, and Shin-ichi Maeda. Dqntamer: Human-in-the-loop reinforcement learning with intractable feedback. arXiv preprint arXiv:1810.11748, 2018.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263-272. PMLR, 2017.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
Viktor Bengs, Robert Busa-Fekete, Adil El Mesaoudi-Paul, and Eyke Hüllermeier. Preference-based online learning with dueling bandits: A survey. The Journal of Machine Learning Research, 22(1): 278-385, 2021.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
Florian Brandl, Felix Brandt, and Hans Georg Seedig. Consistent probabilistic social choice. *Econometrica*, 84(5):1839-1880, 2016.
Róbert Busa-Fekete, Balázs Szörenyi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm. Machine learning, 97:327-351, 2014.
Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In International Conference on Machine Learning, pages 3773-3793. PMLR, 2022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Zihan Ding, Yuanpei Chen, Allen Z Ren, Shixiang Shane Gu, Hao Dong, and Chi Jin. Learning a universal human prior for dexterous manipulation from human preference. arXiv preprint arXiv:2304.04602, 2023.
Miroslav Dudík, Katja Hofmann, Robert E Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Conference on Learning Theory, pages 563-587. PMLR, 2015.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
Peter C Fishburn. Probabilistic social choice based on simple voting comparisons. The Review of Economic Studies, 51(4):683-692, 1984.
Masour Zoghi, Shimon Whiteson, Remi Munos, and Maarten Rijke. Relative upper confidence bound for the k-armed dueling bandit problem. In International conference on machine learning, pages 10-18. PMLR, 2014.
A Proofs of Impossibility Results
Proof of Lemma 2. Consider two link functions  $\sigma_{1}(x) := \frac{1}{1 + \exp(-x)}$  and
$$
\sigma_ {2} (x) := \frac {1}{2} + \alpha_ {1} x + \alpha_ {2} (x - \alpha_ {3}) \cdot I [ | x | > \alpha_ {3} ],
$$
where  $\alpha_{1} = 1$ ,  $\alpha_{2} = -0.484$ ,  $\alpha_{3} = 0.3$  Consider an MDP with  $H = 2$  and initial state  $s_0$ . Suppose that there are three terminal states  $s_1, s_2, s_3$ , where we observe that the trajectory preferences only depend on the terminal state in the following way:
$$
\Pr \left[ s _ {1} \succ s _ {2} \right] = 0. 7, \Pr \left[ s _ {2} \succ s _ {3} \right] = 0. 8, \Pr \left[ s _ {1} \succ s _ {3} \right] = 0. 9 0 3.
$$
This can be explained by both  $\sigma_{1}$  with  $r^{(1)} = \{s_0:0,s_1:0.847,s_2:0,s_3: - 1.386\}$ , and  $\sigma_{2}$  with  $r^{(2)} = \{s_0:0,s_1:0.2,s_2:0,s_3: - 0.3\}$ .
Suppose that state  $s_0$  has two actions  $a$  and  $b$ , which leads to distributions  $\{0.39 : s_1, 0.61 : s_3\}$  and  $\{1 : s_2\}$  respectively. Under  $r^{(1)}$  and  $r^{(2)}$ , the optimal action would be  $a$  and  $b$  respectively. Therefore without knowledge of the link function, it is impossible to identify the optimal policy even with perfect knowledge of the transition and comparison probabilities.
Proof of Lemma 3. Similarly, consider an MDP with  $H = 2$  and initial state  $s_0$ . Suppose that there are three terminal states  $s_1, s_2, s_3$ , where we observe that
$$
\operatorname * {P r} \left[ s _ {1} \succ s _ {2} \right] = 1, \operatorname * {P r} \left[ s _ {2} \succ s _ {3} \right] = 1, \operatorname * {P r} \left[ s _ {1} \succ s _ {3} \right] = 1.
$$
Meanwhile the state  $s_0$  has two actions  $a$  and  $b$ , leading to distributions  $\{0.5 : s_1, 0.5 : s_3\}$  and  $\{1 : s_2\}$  respectively. In this case, both  $r^{(1)} = \{s_0 : 0, s_1 : 0.5, s_2 : 0, s_3 : -1\}$  and  $r^{(2)} = \{s_0 : 0, s_1 : 1, s_2 : 0, s_3 : -0.5\}$  fits the comparison results perfectly. However under  $r^{(1)}$ , the optimal action is  $b$  while under  $r^{(2)}$ , the optimal action is  $a$ .
B Proofs for P2R
B.1 Properties of eluder dimension
We begin by proving two properties of the eluder dimension of  $\overline{\mathcal{R}}$ .
Lemma B.1. Consider  $\mathcal{R}_{\mathrm{linear}} \coloneqq \{\theta^\top x\}$ , where  $x \in \mathcal{X} \subseteq \mathbb{R}^d$ ,  $\| \theta \|_2 \leq \gamma$ . Then there exists absolute constant  $C$  such that
$$
\dim_ {\mathrm {E}} (\overline {{\mathcal {R}}} _ {\text {l i n e a r}}, \epsilon) \leq C (d + 1) \ln \left(1 + \frac {\gamma + H}{\epsilon}\right) + 1.
$$
$$
\theta^ {\top} x + c = [ \theta , c ] ^ {\top} [ x, 1 ].
$$
Therefore  $\overline{\mathcal{R}}_{\mathrm{linear}}$  can be seen as a  $(d + 1)$ -dimensional linear function class with parameter norm  $\| [\theta ,c]\| _2\leq \gamma +H$ . The statement is then a direct corollary of Russo and Van Roy [2013, Proposition 6].
Lemma B.2. For a general function class  $\mathcal{R}$  with domain  $\mathcal{X}$  and range  $[0, H]$ , for  $\epsilon < 1$ ,
$$
\dim_ {\mathrm {E}} (\overline {{\mathcal {R}}}, \epsilon) \leq \mathcal {O} (H \dim_ {\mathrm {E}} (\mathcal {R}, \epsilon / 2) ^ {1. 5} / \epsilon).
$$
Proof. Suppose that there exists a sequence
$$
\left\{r _ {i}, c _ {i}, x _ {i}, y _ {i} \right\} _ {i \in [ m ]},
$$
where  $r_i \in \mathcal{R}$ ,  $c_i \in [-H, 0]$ ,  $x_i \in \mathcal{X}$  and  $y_i \in \mathbb{R}$ , such that for all  $i \in [m]$
$$
\left| r _ {i} \left(x _ {i}\right) + c _ {i} - y _ {i} \right| \geq \epsilon , \quad \sum_ {j <   i} \left| r _ {i} \left(x _ {j}\right) + c _ {i} - y _ {j} \right| ^ {2} \leq \epsilon^ {2}.
$$
By definition,  $\dim_{\mathrm{E}}(\overline{\mathcal{R}},\epsilon)$  is the largest  $m$  so that such a sequence exists. By the pigeon-hole principle, there exists a subset  $I\subseteq [m]$  of size at least  $k = \frac{m}{\lceil H / \epsilon_0\rceil}$  and  $\bar{c}\in [0,H - \epsilon_0]$  such that  $\forall i\in I,c_i\in [\bar{c},\bar{c} +\epsilon_0]$ . Denote the subsequence indexed by  $I$  as  $\{r_i',c_i',x_i',y_i'\}_{i\in [k]}$ . Define  $\widetilde{y}_i\coloneqq y_i' - \bar{c}$ . Now consider the sequence  $\{r_i',x_i',\widetilde{y}_i\}_{i\in [k]}$ . By definition  $\forall i\in [k]$
$$
\left| r _ {i} ^ {\prime} \left(x _ {i} ^ {\prime}\right) - \widetilde {y} _ {i} \right| \geq \epsilon - \epsilon_ {0}.
$$
$$
\begin{array}{l} \sum_ {j <   i} \left| r _ {i} ^ {\prime} (x _ {j}) - \widetilde {y} _ {j} \right| ^ {2} = \sum_ {j <   i} \left| r _ {i} ^ {\prime} (x _ {j}) - \left(y _ {j} ^ {\prime} - c _ {j} ^ {\prime}\right) \right| ^ {2} + 2 \sum_ {j <   i} (\bar {c} - c _ {j} ^ {\prime}) \left(r _ {i} ^ {\prime} (x _ {j}) - \left(y _ {j} ^ {\prime} - c _ {j} ^ {\prime}\right)\right) + \sum_ {j <   i} (\bar {c} - c _ {j} ^ {\prime}) ^ {2} \\ \leq \epsilon^ {2} + 2 \sqrt {k} \epsilon_ {0} \epsilon + k \epsilon_ {0} ^ {2}. \\ \end{array}
$$
We can choose  $\epsilon_0\coloneqq \left(\frac{H\epsilon^2}{16m}\right)^{1 / 3}$  so that  $\epsilon_0\leq \epsilon /(4\sqrt{k})$  . Then we can guarantee
$$
\left| r _ {i} ^ {\prime} (x _ {i}) - \widetilde {y} _ {i} \right| \geq 0. 5 \epsilon , \sum_ {j <   i} \left| r _ {i} ^ {\prime} (x _ {j}) - \widetilde {y} _ {j} \right| ^ {2} \leq 2 \epsilon^ {2}.
$$
By Jin et al. [2021, Proposition 43],
$$
k \leq \left(1 + \frac {2 \epsilon^ {2}}{(0 . 5 \epsilon) ^ {2}}\right) \dim_ {\mathrm {E}} (\mathcal {F}, 0. 5 \epsilon).
$$
$$
\frac {m}{\lceil H / \epsilon_ {0} \rceil} \leq 9 \mathrm {d i m} _ {\mathrm {E}} (\mathcal {R}, 0. 5 \epsilon),
$$
$$
\dim_ {\mathrm {E}} (\overline {{\mathcal {R}}}, \epsilon) \leq \frac {2 1 6 H}{\epsilon} \cdot \dim_ {\mathrm {E}} (\mathcal {R}, 0. 5 \epsilon) ^ {1. 5}.
$$
B.2 Properties related to robustness
Lemma B.3 (Robustness to perturbation). Any RL algorithm  $\mathcal{A}$  with sample complexity  $\mathcal{C}(\epsilon, \delta)$  can be converted to an algorithm  $\mathcal{A}'$  that is  $\frac{1}{\mathcal{C}(\epsilon, \delta)}$ -robust with sample complexity  $\mathcal{C}(\epsilon, \delta/3)$ .
Proof. Consider the following modification of  $\mathcal{A}$ : instead of using reward  $r$  directly, we project  $r$  to  $+2H$  and  $-2H$  unbiasedly; that is, the algorithm receives the binarized rewards
$$
b (r) := \left\{2 H: \frac {1}{2} + \frac {r}{4 H}, - 2 H: \frac {1}{2} - \frac {r}{4 H} \right\}.
$$
By the definition of sample complexity, when using samples of  $b(r^{\star})$ ,  $\mathcal{A}$  outputs a policy  $\pi_0$  for  $r^{\star}$  that is  $\epsilon$ -optimal with probability  $1 - \delta$  in  $K := \mathcal{C}(\epsilon, \delta)$  episodes. Denote the trajectories generated by running  $\mathcal{A}$  on  $b(r^{\star})$  by  $\tau_1, \dots, \tau_K$ . Now suppose that for  $\tau_k$ , the reward label is perturbed from  $b(r^{\star}(\tau_k))$  to  $b(r_k')$  with  $|r_k' - r^{\star}(\tau_k)| \leq \epsilon' := (\mathcal{C}(\epsilon, \delta))^{-1}$ ; denote the output policy of  $\mathcal{A}$  by  $\pi'$ . It can be shown that
$$
\left| \ln \left(\frac {b \left(r ^ {\star} \left(\tau_ {k}\right)\right) = 2 H}{b \left(r _ {k} ^ {\prime}\right) = 2 H}\right) \right| \leq \ln \left(1 + \frac {\epsilon^ {\prime}}{H}\right).
$$
Therefore the total density ratio
$$
\sup _ {\vec {r} \in \{2 H, - 2 H \} ^ {K}} \left| \sum_ {k = 1} ^ {K} \ln \left(\frac {b (r ^ {\star} (\tau_ {1})) , \cdots , b (r ^ {\star} (\tau_ {K})) = \vec {r}}{b (r _ {1} ^ {\prime}) , \cdots , b (r _ {K} ^ {\prime}) = \vec {r}}\right) \right| \leq K \ln \left(1 + \frac {\epsilon^ {\prime}}{H}\right) \leq \frac {K \epsilon^ {\prime}}{H} \leq 1.
$$
It follows that the density ratio between  $\pi$  and  $\hat{\pi}$  is also bounded by  $e$ . Therefore the probability that  $\hat{\pi}$  is not  $\epsilon$ -optimal is at most  $3\delta$ . Rescaling  $\delta$  proves the lemma.
Lemma B.4. Any tabular RL algorithm  $\mathcal{A}$  with sample complexity  $\mathcal{C}(\epsilon, \delta)$  is  $\epsilon/(4H)$  robust with sample complexity  $\mathcal{C}(\epsilon/2, \delta)$ .
Proof. Suppose that  $\mathcal{A}$  is run on perturbed rewards, where rewards for trajectory  $\tau$  is changed by  $\varepsilon(\tau)$ . By definition, using  $\mathcal{C}(\epsilon/2, \delta)$  samples and with probability  $1 - \delta$ , it outputs an  $\epsilon/2$ -optimal policy  $\hat{\pi}$  with respect to the perturbed reward function  $r + \varepsilon$ , where  $\| \varepsilon \|_{\infty} \leq \epsilon/4H$ . Denote the value function of policy  $\pi$  with respect to reward  $r$  by  $V^{\pi, r}$ . Further denote the optimal value function with respect to  $r$  by  $\pi^{\star}$ . It holds that for any policy  $\pi$ ,
$$
\left| V ^ {\pi , r} - V ^ {\pi + \varepsilon , r} \right| \leq \epsilon / 4.
$$
$$
\begin{array}{l} V ^ {\pi^ {\star}, r} - V ^ {\hat {\pi}, r} \leq \epsilon / 2 + V ^ {\pi^ {\star}, r + \varepsilon} - V ^ {\hat {\pi}, r + \varepsilon} \\ \leq \epsilon / 2 + \epsilon / 2 = \epsilon . \\ \end{array}
$$
In other words  $\hat{\pi}$  is indeed an  $\epsilon$ -optimal policy with respect to the unperturbed rewards  $r$ .
B.3 Proof of Theorem 4
Lemma B.5. With  $m = \Theta\left(\frac{\ln(1 / \delta')}{\alpha^2\epsilon'^2}\right)$ , for each  $\tau$  such that the comparison oracle is queried, with probability  $1 - \delta'$ ,
$$
\left| \hat {r} (\tau) - \left(r ^ {\star} (\tau) - r ^ {\star} \left(\tau_ {0}\right)\right) \right| \leq \epsilon^ {\prime}.
$$
Proof. Suppose that the comparison oracle is queried for  $\tau$  and the average outcome is  $\bar{o}$ . By Hoeffding bound, with probability  $1 - \delta'$ ,
$$
\left| \bar {o} - \sigma \left(r ^ {\star} (\tau) - r ^ {\star} \left(\tau_ {0}\right)\right) \right| \leq \sqrt {\frac {\ln \left(2 / \delta^ {\prime}\right)}{m}}.
$$
Since  $\hat{r} (\tau) = \mathrm{argmin}_{x\in [-H,H]}|\sigma (x) - \bar{\sigma} |$
$$
| \sigma (\hat {r} (\tau)) - \bar {o} | \leq | \sigma (r ^ {\star} (\tau) - r ^ {\star} (\tau_ {0})) - \bar {o} | \leq \sqrt {\frac {\ln (2 / \delta^ {\prime})}{m}}.
$$
$$
\left| \sigma (\hat {r} (\tau)) - \sigma (r ^ {\star} (\tau) - r ^ {\star} (\tau_ {0})) \right| \leq 2 \sqrt {\frac {\ln (2 / \delta^ {\prime})}{m}}.
$$
By Assumption 2,
$$
\left| \hat {r} (\tau) - \left(r ^ {\star} (\tau) - r ^ {\star} \left(\tau_ {0}\right)\right) \right| \leq \frac {1}{\alpha} \cdot \left| \sigma \left(\hat {r} (\tau)\right) - \sigma \left(r ^ {\star} (\tau) - r ^ {\star} \left(\tau_ {0}\right)\right) \right| \leq \frac {2}{\alpha} \sqrt {\frac {\ln \left(2 / \delta^ {\prime}\right)}{m}} \leq \epsilon^ {\prime}.
$$
Lemma B.6. Set  $m = \Theta \left(\frac{d\ln(d / \delta)}{\epsilon_0^2\alpha^2}\right)$  and  $\beta = \frac{\epsilon_0^2}{4}$ . With probability  $1 - \delta$ , the number of samples on which the comparison oracle is queried is at most  $\dim_{\mathrm{E}}(\overline{\mathcal{R}},\epsilon_0)$ .
Proof. Define  $\widetilde{r}^{\star}(\tau) \coloneqq r^{\star}(\tau) - r^{\star}(\tau_0)$ ,  $\widetilde{r}(\tau) \coloneqq r(\tau) - r(\tau_0)$ .
When the comparison oracle is queried,  $\max_{r,r^{\prime}\in \mathcal{B}_r}([r(\tau) - r(\tau_0)] - [r^{\prime}(\tau) - r^{\prime}(\tau_0)]) > 2\epsilon_0$  which means that either  $|\widetilde{r} (\tau) - \widetilde{r}^{\star}(\tau)| > \epsilon_{0}$  or  $|\widetilde{r} '(\tau) - \widetilde{r}^{\star}(\tau)| > \epsilon_{0}$ . Suppose that there are  $K$  trajectories which require querying comparison oracle. Suppose that the dataset is composed of
$$
\mathcal {D} = \left\{\left(\hat {r} _ {1}, \tau_ {1}\right), \dots , \left(\hat {r} _ {K}, \tau_ {K}\right) \right\},
$$
and  $\widetilde{r}_1, \ldots, \widetilde{r}_K \in \overline{\mathcal{R}}$  are the functions that satisfy  $|\widetilde{r}_k(\tau_k) - \widetilde{r}^\star(\tau_k)| > \epsilon_0$ . We now verify that  $(\widetilde{r}_1, \tau_1, \widetilde{r}_2, \tau_2, \dots, \widetilde{r}_K, \tau_K)$  is an eluder sequence (with respect to function class  $\overline{\mathcal{R}}$ ).
The confidence set condition implies
$$
\sum_ {k <   i} \left(\widetilde {r} _ {i} (\tau_ {k}) - \hat {r} _ {k}\right) ^ {2} \leq \beta .
$$
With probability  $1 - \delta, \forall k \leq K \land 2d$ ,  $|\hat{r}_k - \widetilde{r}^\star(\tau_k)| \leq \frac{\epsilon_0}{4\sqrt{d}}$  (by Lemma B.5). Then for any  $i \leq k$
$$
\begin{array}{l} \sum_ {k \leq i} \left(\widetilde {r} _ {i} (\tau_ {k}) - \widetilde {r} ^ {\star} (\tau_ {k})\right) ^ {2} \leq \sum_ {k \leq i} \left(\widetilde {r} _ {i} (\tau_ {k}) - \hat {r} _ {k}\right) ^ {2} + \sum_ {k \leq i} \left(\widetilde {r} _ {i} (\tau_ {k}) - \hat {r} _ {k} + \widetilde {r} _ {i} (\tau_ {k}) - \widetilde {r} ^ {\star} (\tau_ {k})\right) \cdot \left(\hat {r} _ {k} - \widetilde {r} ^ {\star} (\tau_ {k})\right) \\ \leq \beta + 2 \sqrt {K \beta} \cdot \frac {\epsilon_ {0}}{4 \sqrt {d}} + K \left(\frac {\epsilon_ {0}}{4 \sqrt {d}}\right) ^ {2} \leq \epsilon_ {0} ^ {2}, \\ \end{array}
$$
as long as  $K \leq 2d$ . In other words, with probability  $1 - \delta$ ,  $(\widetilde{r}_1, \tau_1, \widetilde{r}_2, \tau_2, \dots, \widetilde{r}_{K \wedge 2d}, \tau_{K \wedge 2d})$  is an eluder sequence, which by Definition 2 cannot have length more than  $d \coloneqq \dim_{\mathrm{E}}(\overline{\mathcal{R}}, \epsilon_0)$ . It follows that  $K \leq \dim_{\mathrm{E}}(\overline{\mathcal{R}}, \epsilon_0)$ .
Lemma B.7. With probability  $1 - \delta$ ,  $r^{\star} \in \mathcal{B}_r$  throughout the execution of Algorithm 1.
Proof. By Lemma B.5 and Lemma B.6, with probability  $1 - \delta$ , at every step
$$
\sum_ {(\hat {r}, \tau) \in \mathcal {D}} \left(r ^ {\star} (\tau) - r ^ {\star} (\tau_ {0}) - \hat {r}\right) ^ {2} \leq d \cdot \left(\frac {\epsilon_ {0}}{4 \sqrt {d}}\right) ^ {2} \leq \frac {\epsilon_ {0} ^ {2}}{4} = \beta .
$$
Lemma B.8. With probability  $1 - \delta$ , for each  $\tau$  in Line 3 of Algorithm 1, the returned reward  $\hat{r}$  satisfies
$$
\left| \hat {r} - \left(r ^ {\star} (\tau) - r ^ {\star} (\tau_ {0})\right) \right| \leq 2 \epsilon_ {0}.
$$
Proof. We already know that this is true for  $\tau$  such that the comparison oracle is queried. However, if it is not queried, then
$$
\max _ {r, r ^ {\prime} \in \mathcal {B} _ {r}} (r (\tau) - r (\tau_ {0}) - (r ^ {\prime} (\tau) - r ^ {\prime} (\tau_ {0})) <   2 \epsilon_ {0}.
$$
Since  $r^{\star} \in \mathcal{B}_r$  (by Lemma B.7), this immediately implies  $|\hat{r} - (r^{\star}(\tau) - r^{\star}(\tau_0))| \leq 2\epsilon_0$ .
Proof of Theorem 4. Choose  $\epsilon_0\coloneqq g(\epsilon) / 2,\beta = \frac{\epsilon_0^2}{4}$  and  $m = \Theta (\frac{d_{\overline{\mathcal{R}}}\ln(d_{\overline{\mathcal{R}}} / \delta)}{\epsilon_0^2\alpha^2})$
By Lemma B.8 (rescaling  $\delta$ ), with probability  $1 - \delta$ , the reward returned by the reward interface is  $g(\epsilon)$ -close to  $\widetilde{r}^{\star} := r^{\star} - r^{\star}(\tau_0)$  throughout the execution of the algorithm. By the definition of sample complexity, with probability  $1 - \delta$ , the policy returned by  $\mathcal{A}$  is  $\epsilon$ -optimal for  $\widetilde{r}^{\star}$ , which implies that it is also  $\epsilon$ -optimal for  $r^{\star}$ . The number of samples (episodes) is bounded by  $\mathcal{C}(\epsilon, \delta)$ . Finally by Lemma B.6, the number of queries to the comparison oracle is at most
$$
\dim_ {\mathrm {E}} (\overline {{\mathcal {R}}}, \epsilon_ {0}) \cdot m \leq \widetilde {\mathcal {O}} \left(\frac {d _ {\overline {{\mathcal {R}}}} ^ {2}}{g ^ {2} (\epsilon) \alpha^ {2}}\right).
$$
C P-OMLE: Improved query complexity via white-box modification
While the P2R interface provides a clean and effortless recipe for modifying standard RL algorithms to work with preference feedback, it incurs a large query cost to the comparison oracle, e.g., cubic dependence on  $d_{\overline{\mathcal{R}}}$  in Example 3. We believe that this disadvantage is caused by the black-box nature of P2R, and that better query complexities can be achieved by modifying standard RL algorithms in a white-box manner and specialized analysis. In this section, we take OMLE [Liu et al., 2022a] as an example and introduce a standalone adaptation to trajectory preference feedback with improved query complexity. We expect other optimism-based algorithms (e.g., UCBVI-BF and GOLF) can be modified in a similar white-box manner to achieve better query complexity.
The details of the Preference-based OMLE algorithm (P-OMLE) is provided in Algorithm 2. Compared to OMLE with reward feedback (Algorithm 3), the only difference is that now the reward confidence set is computed using preference feedback directly using log-likelihood. Denote by  $V_{r,p}^{\pi}$  the expected cumulative reward the leaner will receive if she follows policy  $\pi$  in model  $(p,r)$ . In the  $t$ -th iteration, the algorithm follows the following steps:
- Optimistic planning: Find the policy-model pair  $(\pi, r, p)$  that maximizes the value function  $V_{r,p}^{\pi}$ ;
- Data collection: Construct an exploration policy set  $\Pi_{\exp}(\pi^t)$  and collect trajectories by running all policies in  $\Pi_{\exp}(\pi^t)$ ;
- Confidence set update: Update the confidence set using the updated log-likelihood:
$$
\mathcal {L} (r, \mathcal {D} _ {\mathrm {r w d}}) := \sum_ {(\tau , \tau^ {\prime}, y) \in \mathcal {D} _ {\mathrm {r w d}}} \ln \sigma (r (\tau) - r (\tau^ {\prime}), y), \quad \mathcal {L} (p, \mathcal {D} _ {\mathrm {t r a n s}}) := \sum_ {(\pi , \tau) \in \mathcal {D} _ {\mathrm {t r a n s}}} \ln \mathbb {P} _ {p} ^ {\pi} (\tau).
$$
Algorithm 2 Preference-based OMLE (P-OMLE)
1:  $\mathcal{B}^1\gets \mathcal{R}\times \mathcal{P}$
2: execute an arbitrary policy to collect trajectory  $\tau^0$
3: for  $t = 1, \dots, T$  do
4: compute  $(\pi^t,r^t,p^t) = \arg \max_{\pi ,(r,p)\in \mathcal{B}^t}[V_{r,p}^\pi -r(\tau^0)]$
5: execute  $\pi^t$  to collect a trajectory  $\tau$
6: invoke comparison oracle on  $\tau$  and  $\tau^0$  to get  $y$ , add  $(\tau, \tau^0, y)$  into  $\mathcal{D}_{\tau \mathrm{wd}}$
7: for each  $\pi \in \Pi_{\mathrm{exp}}(\pi^t)$  do
8: execute  $\pi$  to collect a trajectory  $\tau$ , add  $(\pi, \tau)$  into  $\mathcal{D}_{\text{trans}}$
9: update
$$
\mathcal {B} ^ {t + 1} \leftarrow \left\{(r, p) \in \mathcal {R} \times \mathcal {P}: \mathcal {L} \left(r, \mathcal {D} _ {\mathrm {r w d}}\right) > \max  _ {r ^ {\prime} \in \mathcal {R}} \mathcal {L} \left(r ^ {\prime}, \mathcal {D} _ {\mathrm {r w d}}\right) - \beta_ {\mathcal {R}} \right.
$$
$$
\left. \text {a n d} \mathcal {L} \left(p, \mathcal {D} _ {\text {t r a n s}}\right) > \max  _ {p ^ {\prime} \in \mathcal {P}} \mathcal {L} \left(p ^ {\prime}, \mathcal {D} _ {\text {t r a n s}}\right) - \beta_ {\mathcal {P}} \right\}
$$
We have the follwoing guarantee for Algorithm 2.
Theorem C.1. Suppose Assumption, 1, 2 and 3 hold. There exists an absolute constant  $c_{1}, c_{2} > 0$  such that for any  $(T, \delta) \in \mathbb{N} \times (0, 1]$  if we choose  $\beta_{\mathcal{R}} = c_{1}\ln(|\mathcal{R}|T / \delta)$  and  $\beta_{\mathcal{P}} = c_{1}\ln(|\mathcal{M}|T / \delta)$  in Algorithm 2, then with probability at least  $1 - \delta$ , we have that for all  $t \in [T]$ ,
$$
\sum_ {t = 1} ^ {T} \left[ V ^ {\star} - V ^ {\pi^ {t}} \right] \leq 2 H \xi \left(d _ {\mathcal {P}}, T, c _ {2} \beta_ {\mathcal {P}}, \left| \Pi_ {\exp} \right|\right) + \mathcal {O} \left(H \alpha^ {- 1} \sqrt {d _ {\overline {{\mathcal {R}}}} T \beta_ {\mathcal {R}}}\right)
$$
where  $d_{\overline{\mathcal{R}}} = \dim_{\mathrm{E}}(\overline{\mathcal{R}},1 / \sqrt{T})$
For problems that satisfy  $\xi = \widetilde{\mathcal{O}} (\sqrt{d_{\mathcal{P}}\beta_{\mathcal{P}}|\Pi_{\mathrm{exp}}|T})$ , Theorem C.1 implies both the sample complexity and the query complexity for learning an  $\epsilon$ -optimal policy is upper bounded by:
$$
\widetilde {\mathcal {O}} \left(\frac {H ^ {2} d _ {\mathcal {P}} \left| \Pi_ {\exp} \right| ^ {2} \ln | \mathcal {P} |}{\epsilon^ {2}} + \frac {H ^ {2} d _ {\overline {{\mathcal {R}}}} \ln | \mathcal {R} |}{\alpha^ {2} \epsilon^ {2}}\right).
$$
Compared to the result derived through the P2R interface (Example 3), the sample complexity here is basically the same while the query complexity has improved dependence on  $d_{\overline{\mathcal{R}}}$  (from cubic to linear). Nonetheless, the query complexity here additionally depends on the complexity of learning the transition model, which is not the case in Example 3.
C.1 Proofs for P-OMLE
The proof of Theorem C.1 largely follows from that of Theorem D.1. We first introduce several useful notations. Denote by  $\mathcal{B}_{\mathcal{M}}^t, \mathcal{B}_{\mathcal{P}}^t$  the preference, transition confidence set in the  $t$ -th iteration, which satisfy  $\mathcal{B}^t = \mathcal{B}_{\mathcal{P}}^t \times \mathcal{B}_{\mathcal{M}}^t$ . Denote the groundtruth transition and preference by  $p^\star$  and  $M^\star$ . Denote the trajectories generated when running Algorithm 2 by  $\tau^1, \dots, \tau^T$ .
Similar to Lemma D.3, we have that the confidence set satisfies the following properties.