SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {s} \mathbb {P} ^ {\bar {\pi}} (s _ {h} = s) \cdot \| \hat {\pi} _ {h} (\cdot | s) - \bar {\pi} _ {h} (\cdot | s) \| _ {1} \leq \frac {S \iota}{N} + \sum_ {s} \mathbb {P} ^ {\bar {\pi}} (s _ {h} = s) \cdot \sqrt {\frac {\iota A}{N \mathbb {P} ^ {\bar {\pi}} (s _ {h} = s)}} \\ \leq \frac {S \iota}{N} + \sqrt {\frac {\iota S A}{N}} \leq \frac {\epsilon}{2 H}. \\ \end{array}
$$
Given a Markov policy  $\pi_{1}$  and general policy  $\pi_{2}$ , we define
$$
q _ {h} ^ {\pi_ {1}, \pi_ {2}} (s _ {h}, a _ {h}) := \mathbb {E} _ {s _ {H} \sim \pi_ {1} | s _ {h}, a _ {h}, s _ {H} ^ {\prime} \sim \pi_ {2}} \left[ M (s _ {H}, s _ {H} ^ {\prime}) \right].
$$
It follows that for any policy  $\pi$
$$
\begin{array}{l} | U (\hat {\pi}, \pi) - U (\bar {\pi}, \pi) | = \left| \sum_ {h = 1} ^ {H} \mathbb {E} _ {\bar {\pi}} \left[ \langle \hat {\pi} _ {h} (\cdot \mid s _ {h}) - \bar {\pi} _ {h} (\cdot \mid s _ {h}), q _ {h} ^ {\hat {\pi}, \pi} (s _ {h}, \cdot) \rangle \right] \right| \\ \leq \sum_ {h = 1} ^ {H} \sum_ {s} \mathbb {P} ^ {\bar {\pi}} (s _ {h} = s) \cdot \| \hat {\pi} _ {h} (\cdot | s) - \bar {\pi} _ {h} (\cdot | s) \| _ {1} \leq \epsilon / 2. \\ \end{array}
$$
$$
\mathrm {D G a p} (\hat {\pi}, \hat {\pi}) = 2 \max _ {\pi^ {\prime}} U (\pi^ {\prime}, \hat {\pi}) - 1 \leq 2 \max _ {\pi^ {\prime}} U (\pi^ {\prime}, \bar {\pi}) - 1 + \epsilon \leq 2 \epsilon .
$$
F Additional details and proofs for Section 4.3
F.1 Algorithm details
To state the algorithm in a more compact way, we first introduce several notations. We denote the expected winning times of policy  $\pi$  against policy  $\pi'$  in a RLHF instance with transition  $p$  and preference  $M$  by
$$
V _ {p, M} ^ {\pi , \pi^ {\prime}} := \mathbb {E} \left[ M (\tau , \tau^ {\prime}) \mid \tau \sim \mathbb {P} _ {p} ^ {\pi}, \tau^ {\prime} \sim \mathbb {P} _ {p} ^ {\pi^ {\prime}} \right].
$$
Furthermore, we denote the best-response value against policy  $\pi$  as
$$
V _ {p, M} ^ {\pi , \dagger} = \min _ {\pi^ {\prime}} V _ {p, M} ^ {\pi , \pi^ {\prime}},
$$
and the minimax value as
$$
V _ {p, M} ^ {\star} = \max _ {\pi} \min _ {\pi^ {\prime}} V _ {p, M} ^ {\pi , \pi^ {\prime}}.
$$
Algorithm 5 Learning von Neumann winner via Optimistic MLE
1:  $\mathcal{B}^1\gets \mathcal{P}\times \mathcal{M}$
2: execute an arbitrary policy to collect trajectory  $\tau^0$
3: for  $t = 1, \dots, T$  do
4: compute optimistic von Neumann winner  $(\overline{\pi}^t,\overline{M}^t,\overline{p}^t) = \arg \max_{\pi ,(p,M)\in \mathcal{B}^t}V_{p,M}^{\pi ,\dagger}$
5: compute optimistic best-response  $(\underline{\pi}^t,\underline{M}^t,\underline{p}^t) = \arg \min_{\pi ', (p,M)\in \mathcal{B}^t}V_{p,M}^{\overline{\pi}^t,\pi '}$
6: sample  $\overline{\tau}^t\sim \overline{\pi}^t$  and  $\underline{\tau}^t\sim \underline{\pi}^t$
7: invoke comparison oracle on  $(\overline{\tau}^t, \underline{\tau}^t)$  to get  $y^t$ , add  $(\overline{\tau}^t, \underline{\tau}^t, y^t)$  into  $\mathcal{D}_{\mathrm{pref}}$
8: for each  $\pi \in (\Pi_{\exp}(\overline{\pi}^t) \cup \Pi_{\exp}(\underline{\pi}^t))$  do
9: execute  $\pi$  to collect a trajectory  $\tau$ , add  $(\pi, \tau)$  into  $\mathcal{D}_{\text{trans}}$
10: update
$$
\mathcal {B} ^ {t + 1} \leftarrow \left\{(p, M) \in \mathcal {P} \times \mathcal {M}: \mathcal {L} (M, \mathcal {D} _ {\text {p r e f}}) > \max  _ {M ^ {\prime} \in \mathcal {M}} \mathcal {L} \left(M ^ {\prime}, \mathcal {D} _ {\text {p r e f}}\right) - \beta_ {\mathcal {M}} \right.
$$
$$
\text{and}\mathcal{L}(p,\mathcal{D}_{\text{trans}}) > \max_{p^{\prime}\in \mathcal{P}}\mathcal{L}(p^{\prime},\mathcal{D}_{\text{trans}}) - \beta_{\mathcal{P}}\Big\}
$$
11: output  $\pi^{\mathrm{out}} = \mathrm{Unif}(\{\overline{\pi}^t\}_{t\in [T]})$
We provide the pseudocode of learning von Neumann winner via optimistic MLE in Algorithm 5. In each iteration  $t \in [T]$ , the algorithm performs the following three key steps:
- Optimistic planning: Compute the most optimistic von Neumann winner  $\overline{\pi}^t$  by picking the most optimistic model-preference candidate  $(M, p)$  in the current confidence set  $\mathcal{B}^t$ . Then compute the most optimistic best-response to  $\overline{\pi}^t$ , denoted as  $\underline{\pi}^t$ .
- Data collection: Sample two trajectories  $\overline{\tau}^t$  and  $\underline{\tau}^t$  from  $\overline{\pi}^t$  and  $\underline{\pi}^t$ . Then input them into the comparison oracle to get feedback  $y^t$ , which is added into the preference dataset  $\mathcal{D}_{\mathrm{pref}}$ . And similar to the standard OMLE, we also execute policies from the exploration policy set that is constructed by using  $\overline{\pi}^t$  and  $\underline{\pi}^t$ , and add the collected data into the transition dataset  $\mathcal{D}_{\mathrm{trans}}$ .
- Confidence set update: Update the confidence set using the updated log-likelihood, which is the same as Algorithm 3 except that we replace the utility-based preference therein by general preference
$$
\mathcal {L} (M, \mathcal {D} _ {\mathrm {p r e f}}) := \sum_ {(\tau , \tau^ {\prime}, y) \in \mathcal {D} _ {\mathrm {p r e f}}} \ln \left(y M (\tau , \tau^ {\prime}) + (1 - y) (1 - M (\tau , \tau^ {\prime}))\right).
$$
F.2 Proof of Theorem 12
We first introduce several useful notations. Denote by  $\mathcal{B}_\mathcal{M}^t$ ,  $\mathcal{B}_\mathcal{P}^t$  the preference, transition confidence set in the  $t$ -th iteration, which satisfy  $\mathcal{B}^t = \mathcal{B}_\mathcal{P}^t \times \mathcal{B}_\mathcal{M}^t$ . Denote the groundtruth transition and preference by  $p^\star$  and  $M^\star$ . To prove Theorem 12, it suffices to bound
$$
\sum_ {t} \left(V _ {p ^ {\star}, M ^ {\star}} ^ {\star} - V _ {p ^ {\star}, M ^ {\star}} ^ {\overline {{\pi}} ^ {t}, \dagger}\right).
$$
Similar to the proof of Theorem D.1, we first state several key properties of the MLE confidence set  $\mathcal{B}^t$ , which are trivial extensions of the confidence set properties in Liu et al. [2022a].
Lemma F.1 (Liu et al. [2022a]). Under the same condition as Theorem 12, we have that with probability at least  $1 - \delta$ : for all  $t \in [T]$
-  $(p^{\star}, M^{\star}) \in \mathcal{B}^{t}$ ,
-  $\sum_{t=1}^{T} \max_{p \in \mathcal{B}_{\mathcal{P}}^{t}} \left(d_{\mathrm{TV}}(\mathbb{P}_{p}^{\overline{\pi}^{t}}, \mathbb{P}_{p^{\star}}^{\overline{\pi}^{t}}) + d_{\mathrm{TV}}(\mathbb{P}_{p}^{\frac{\pi}{t}}, \mathbb{P}_{p^{\star}}^{\frac{\pi}{t}})\right) \leq \xi(d_{\mathcal{P}}, T, c_{2}\beta_{\mathcal{P}}, |\Pi_{\exp}|)$ ,
$\cdot \max_{M\in \mathcal{B}_{\mathcal{M}}^t}\sum_{i <   t}|M(\overline{\tau}^i,\underline{\tau}^i) - M^\star (\overline{\tau}^i,\underline{\tau}^i)|^2\leq \mathcal{O}(\beta_{\mathcal{M}}).$
The first relation states that confidence set  $\mathcal{B}^t$  contains the groundtruth transition-preference model with high probability. The second one resembles the second relation in Lemma D.3. The third one states that any preference model  $M$  in confidence set  $\mathcal{B}_{\mathcal{M}}^t$  can well predict the preference over the previously collected trajectory pairs.
By using the first relation in Lemma F.1 and the definition of  $\overline{\pi}^t$  and  $\underline{\pi}^t$ , we have
$$
\begin{array}{l} \sum_ {t} \left(V _ {p ^ {\star}, M ^ {\star}} ^ {\star} - V _ {p ^ {\star}, M ^ {\star}} ^ {\overline {{\pi}} ^ {t}, \dagger}\right) \\ \leq \sum_ {t} \left(V _ {\overline {{p}} ^ {t}, \overline {{M}} ^ {t}} ^ {\overline {{\pi}} ^ {t}, \uparrow} - V _ {\underline {{p}} ^ {t}, \underline {{M}} ^ {t}} ^ {\overline {{\pi}} ^ {t}, \underline {{\pi}} ^ {t}}\right) \\ \leq \sum_ {t} \left(V _ {\overline {{p}} ^ {t}, \overline {{M}} ^ {t}} ^ {\overline {{\pi}} ^ {t}, \underline {{\pi}} ^ {t}} - V _ {\underline {{p}} ^ {t}, \underline {{M}} ^ {t}} ^ {\overline {{\pi}} ^ {t}, \underline {{\pi}} ^ {t}}\right) \\ \leq 2 \sum_ {t} \left(d _ {\mathrm {T V}} \left(\mathbb {P} _ {\overline {{p}} ^ {t}} ^ {\overline {{\pi}} ^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\overline {{\pi}} ^ {t}}\right) + d _ {\mathrm {T V}} \left(\mathbb {P} _ {\overline {{p}} ^ {t}} ^ {\underline {{\pi}} ^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\underline {{\pi}} ^ {t}}\right) + d _ {\mathrm {T V}} \left(\mathbb {P} _ {\underline {{p}} ^ {t}} ^ {\overline {{\pi}} ^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\overline {{\pi}} ^ {t}}\right) + d _ {\mathrm {T V}} \left(\mathbb {P} _ {\underline {{p}} ^ {t}} ^ {\underline {{\pi}} ^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\underline {{\pi}} ^ {t}}\right)\right) \\ + \sum_ {t} \left(V _ {p ^ {\star}, \underline {{M}} ^ {t}} ^ {\overline {{\pi}} ^ {t}, \underline {{\pi}} ^ {t}} - V _ {p ^ {\star}, \underline {{M}} ^ {t}} ^ {\overline {{\pi}} ^ {t}, \underline {{\pi}} ^ {t}}\right) \\ \leq 2 \sum_ {t} \left(d _ {\mathrm {T V}} \left(\mathbb {P} _ {\overline {{p}} ^ {t}} ^ {\overline {{\pi}} ^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\overline {{\pi}} ^ {t}}\right) + d _ {\mathrm {T V}} \left(\mathbb {P} _ {\overline {{p}} ^ {t}} ^ {\overline {{\pi}} ^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\overline {{\pi}} ^ {t}}\right) + d _ {\mathrm {T V}} \left(\mathbb {P} _ {\underline {{p}} ^ {t}} ^ {\overline {{\pi}} ^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\overline {{\pi}} ^ {t}}\right) + d _ {\mathrm {T V}} \left(\mathbb {P} _ {\underline {{p}} ^ {t}} ^ {\overline {{\pi}} ^ {t}}, \mathbb {P}  ^ {\overline {{\pi}} ^ {t}}\right)\right) \\ + \sum_ {t} \left(\overline {{M}} ^ {t} (\bar {\tau} ^ {t}, \underline {{\tau}} ^ {t}) - \underline {{M}} ^ {t} (\bar {\tau} ^ {t}, \underline {{\tau}} ^ {t})\right) + \mathcal {O} \left(\sqrt {T \ln (1 / \delta)}\right), \\ \end{array}
$$
By the second relation in Lemma F.1 and Definition 3, we have
$$
\sum_ {t} \left(d _ {\mathrm {T V}} (\mathbb {P} _ {\overline {{p}} ^ {t}} ^ {\overline {{\pi}} ^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\overline {{\pi}} ^ {t}}) + d _ {\mathrm {T V}} (\mathbb {P} _ {\overline {{p}} ^ {t}} ^ {\overline {{\pi}} ^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\overline {{\pi}} ^ {t}}) + d _ {\mathrm {T V}} (\mathbb {P} _ {\underline {{p}} ^ {t}} ^ {\overline {{\pi}} ^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\overline {{\pi}} ^ {t}}) + d _ {\mathrm {T V}} (\mathbb {P} _ {\underline {{p}} ^ {t}} ^ {\overline {{\pi}} ^ {t}}, \mathbb {P} _ {p ^ {\star}} ^ {\overline {\pi} ^ {t}})\right) \leq 4 \xi (d _ {\mathcal {P}}, T, c \beta_ {\mathcal {P}}, | \Pi_ {\exp} |),
$$
Combining the third relation in Lemma F.1 with the regret bound of eluder dimension (e.g., Lemma 2 in Russo and Van Roy [2013]), we have
$$
\sum_ {t} \left(\overline {{M}} ^ {t} (\overline {{\tau}} ^ {t}, \underline {{\tau}} ^ {t}) - \underline {{M}} ^ {t} (\overline {{\tau}} ^ {t}, \underline {{\tau}} ^ {t})\right) \leq \mathcal {O} (\sqrt {d _ {\mathcal {M}} \beta_ {\mathcal {M}} T}).
$$
Putting all pieces together, we have
$$
\sum_ {t} \left(V _ {p ^ {\star}, M ^ {\star}} ^ {\star} - V _ {p ^ {\star}, M ^ {\star}} ^ {\overline {{\pi}} ^ {t}, \dagger}\right) \leq 4 \xi (d _ {\mathcal {P}}, T, c \beta_ {\mathcal {P}}, | \Pi_ {\mathrm {e x p}} |) + \mathcal {O} (\sqrt {d _ {\mathcal {M}} \beta_ {\mathcal {M}} T}).
$$