SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. For convenience, we assume that the offline dataset used in policy learning is independent of the dataset used in  $V$ -Learning. This can be easily achieved by splitting the original dataset  $\mathcal{D}$  uniformly at random into two datasets of equal size. Also, for analysis, we define
$$
\tilde {L} ^ {\mathrm {M L E}} (\pi) = \mathbb {E} _ {(s, a, g) \sim \mu} \left[ \frac {\hat {U} (s , a ; g) _ {+}}{\alpha} \log \pi (a | s, g) \right], \quad \forall \pi \in \Pi .
$$
Now we condition on the high probability event in Lemma 3, which we denote by  $\mathcal{E}_1$ . Then by Lemma 1, we have  $\left\| \frac{\hat{U}_{+}}{\alpha} - \frac{U_{\alpha + }^{*}}{\alpha}\right\|_{2,\mu} \leq \epsilon_U$ . By Cauchy-Schwarz inequality, we can obtain that for
any policy  $\pi$
$$
\begin{array}{l} \left| L _ {\alpha} ^ {\mathrm {M L E}} (\pi) - \tilde {L} ^ {\mathrm {M L E}} (\pi) \right| \leq \mathbb {E} _ {(s, a, g) \sim \mu} \left[ \left| \frac {U _ {\alpha} ^ {*} (s , a ; g) _ {+} - \hat {U} (s , a ; g) _ {+}}{\alpha} \right| \mid \log \pi (a | s, g) \right| \\ \leq \left\| \frac {\hat {U} _ {+}}{\alpha} - \frac {U _ {\alpha +} ^ {*}}{\alpha} \right\| _ {2, \mu} \| \log \pi (a | s, g) \| _ {2, \mu} \\ \leq \epsilon_ {U} \log (1 / \tau). \\ \end{array}
$$
Also, for any fixed  $\pi \in \Pi$ , since  $\mathbb{E}_{(s,a,g)\sim \mu}[\hat{L}^{\mathrm{MLE}}(\pi)] = \tilde{L}^{\mathrm{MLE}}(\pi)$ , we can obtain by Bernstein's inequality that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} | \tilde {L} ^ {\mathrm {M L E}} (\pi) - \hat {L} ^ {\mathrm {M L E}} (\pi) | \\ \leq O \left(\sqrt {\frac {\operatorname {V a r} _ {\mu} \left(\frac {\hat {U} (s , a ; g) _ {+}}{\alpha} \log \pi (a | s , g)\right) \log (1 / \delta)}{N}} + \frac {\| \hat {U} _ {+} / \alpha \| _ {\infty} \log (1 / \tau)}{N} \log (1 / \delta)\right). \\ \end{array}
$$
Since  $U_{\alpha}^{*}(s,a;g)_{+} / \alpha = d_{\alpha}^{*}(s,a;g) / \mu (s,a;g)\leq C_{\alpha}^{*}$ , we have  $\| U_{\alpha +}^{*} / \alpha \|_{2,\mu}\leq C_{\alpha}^{*}$  and thus  $\| \hat{U}_{+} / \alpha \|_{2,\mu}\leq \| U_{\alpha +}^{*} / \alpha \|_{2,\mu} + \epsilon_{U}\leq C_{\alpha}^{*} + \epsilon_{U}\leq O(C_{\alpha}^{*})$ . Also,
$$
\begin{array}{l} \operatorname {V a r} _ {\mu} \left(\frac {\hat {U} (s , a ; g) _ {+}}{\alpha} \log \pi (a | s, g)\right) \leq \mathbb {E} _ {\mu} \left[ \left(\frac {\hat {U} (s , a ; g) _ {+}}{\alpha}\right) ^ {2} \log^ {2} \pi (a | s, g) \right] \\ \leq \mathbb {E} _ {\mu} \left[ \left(\frac {\hat {U} (s , a ; g) _ {+}}{\alpha}\right) ^ {2} \right] \log^ {2} (1 / \tau) \\ \leq O \left(\left(C _ {\alpha} ^ {*}\right) ^ {2} \log^ {2} (1 / \tau)\right). \\ \end{array}
$$
Applying a union bound over all  $\pi \in \Pi$ , it holds that with probability at least  $1 - \delta$ , for all  $\pi \in \Pi$
$$
\begin{array}{l} | \tilde {L} ^ {\mathrm {M L E}} (\pi) - \hat {L} ^ {\mathrm {M L E}} (\pi) | \\ \leq O \left(C _ {\alpha} ^ {*} \log (1 / \tau) \sqrt {\frac {\log (| \Pi | / \delta)}{N}} + \frac {V _ {\max} \log (1 / \tau)}{\alpha N} \log (| \Pi | / \delta)\right) \\ = \epsilon_ {\Pi} \log (1 / \tau). \\ \end{array}
$$
We denote the above event by  $\mathcal{E}_2$ . When  $\mathcal{E}_1$  and  $\mathcal{E}_2$  hold simultaneously, we have by the triangle inequality that
$$
\left| L _ {\alpha} ^ {\mathrm {M L E}} (\pi) - \hat {L} ^ {\mathrm {M L E}} (\pi) \right| \leq \left| L _ {\alpha} ^ {\mathrm {M L E}} (\pi) - \tilde {L} ^ {\mathrm {M L E}} (\pi) \right| + \left| \tilde {L} ^ {\mathrm {M L E}} (\pi) - \hat {L} ^ {\mathrm {M L E}} (\pi) \right| \leq \left(\epsilon_ {U} + \epsilon_ {\Pi}\right) \log (1 / \tau).
$$
$$
\begin{array}{l} \mathbb {P} (\neg (\mathcal {E} _ {1} \cap \mathcal {E} _ {2})) = \mathbb {P} (\neg \mathcal {E} _ {1} \cup \neg \mathcal {E} _ {2}) \leq \mathbb {P} (\neg \mathcal {E} _ {1}) + \mathbb {P} (\neg \mathcal {E} _ {2}) \\ \leq \delta + \mathbb {P} (\mathcal {E} _ {1}) \mathbb {P} (\neg \mathcal {E} _ {2} | \mathcal {E} _ {1}) + \mathbb {P} (\neg \mathcal {E} _ {1}) \mathbb {P} (\neg \mathcal {E} _ {2} | \neg \mathcal {E} _ {1}) \\ \leq \delta + 1 \times \delta + \delta \times 1 \leq 3 \delta . \\ \end{array}
$$
The conclusion holds by rescaling  $\delta$ .
Lemma 9 (Closeness of MLE objective of  $\pi_{\alpha}^{*}$  and  $\hat{\pi}$ ). Under Assumption 4, with probability at least  $1 - \delta$
$$
L _ {\alpha} ^ {M L E} \left(\pi_ {*} ^ {*}\right) - L _ {\alpha} ^ {M L E} (\hat {\pi}) \leq O \left(\epsilon_ {s t a t} ^ {M L E}\right).
$$
Proof. We condition on the high probability event in Lemma 8. Note that
$$
L _ {\alpha} ^ {\mathrm {M L E}} (\pi_ {\alpha} ^ {*}) - L _ {\alpha} ^ {\mathrm {M L E}} (\hat {\pi}) = \underbrace {L _ {\alpha} ^ {\mathrm {M L E}} (\pi_ {\alpha} ^ {*}) - \hat {L} ^ {\mathrm {M L E}} (\pi_ {\alpha} ^ {*})} _ {(1)} + \underbrace {\hat {L} ^ {\mathrm {M L E}} (\pi_ {\alpha} ^ {*}) - \hat {L} ^ {\mathrm {M L E}} (\hat {\pi})} _ {(2)} + \underbrace {\hat {L} ^ {\mathrm {M L E}} (\hat {\pi}) - L _ {\alpha} ^ {\mathrm {M L E}} (\hat {\pi})} _ {(3)}.
$$
(1), (3)  $\leq O(\epsilon_{\mathrm{stat}}^{\mathrm{MLE}})$  by Lemma 8 and (2)  $\leq 0$  by the optimality of  $\hat{\pi}$  in empirical MLE objective, which completes the proof.
Proof of Lemma 2. We condition on the high probability event in Lemma 8. Note that
$$
\begin{array}{l} L _ {\alpha} ^ {\mathrm {M L E}} (\pi) = \mathbb {E} _ {(s, a, g) \sim \mu} \left[ g _ {*} ^ {\prime} (r (s; g) + \gamma \mathcal {T} V _ {\alpha} ^ {*} (s, a; g) - V _ {\alpha} ^ {*} (s; g)) _ {+} \log \pi (a | s, g) \right] \\ = \mathbb {E} _ {(s, a, g) \sim d _ {c s} ^ {*}} [ \log \pi (a | s, g) ]. \\ \end{array}
$$
$$
L _ {\alpha , \mathrm {R e l}} ^ {\mathrm {M L E}} (\pi) = L _ {\alpha} ^ {\mathrm {M L E}} (\pi) - \mathbb {E} _ {(s, a, g) \sim d _ {\alpha} ^ {*}} [ \log \pi_ {\alpha} ^ {*} (a | s, g) ] = \mathbb {E} _ {(s, a, g) \sim d _ {\alpha} ^ {*}} \left[ \log \frac {\pi (a | s , g)}{\pi_ {\alpha} ^ {*} (a | s , g)} \right],
$$
and note that  $L_{\alpha, \mathrm{Rel}}^{\mathrm{MLE}}$  is a constant shift of  $L_{\alpha}^{\mathrm{MLE}}$ . For any  $\pi \in \Pi$ , we further define  $r_{\pi} = \pi / \pi_{\alpha}^{*}$ . By Assumption 5,  $r_{\pi}(a|s, g) \in [\tau, 1 / \tau]$ . Let  $\tilde{L}_{\alpha, \mathrm{Rel}}^{\mathrm{MLE}}(r_{\pi}) = \mathbb{E}_{(s, a, g) \sim d_{\alpha}^{*}}[\log r_{\pi}(a|s, g)]$  which is  $2\tau^{2}$ -strongly concave w.r.t.  $\| \cdot \|_{2, d_{\alpha}^{*}}$  when  $r_{\pi}(a|s, g) \in [\tau, 1 / \tau]$ . Since  $\pi_{\alpha}^{*}$  is the maximizer of  $L_{\alpha}^{\mathrm{MLE}}$  and thus the maximizer of  $L_{\alpha, \mathrm{Rel}}^{\mathrm{MLE}}$ , we have that  $r_{\pi_{\alpha}^{*}}$  is the maximizer of  $\tilde{L}_{\alpha, \mathrm{Rel}}^{\mathrm{MLE}}$ . By strong concavity and the optimality of  $r_{\pi_{\alpha}^{*}}$ , we have
$$
\begin{array}{l} \tau^ {2} \| r _ {\pi_ {\alpha} ^ {*}} - r _ {\hat {\pi}} \| _ {2, d _ {*} ^ {*}} ^ {2} \leq \tilde {L} _ {\alpha , \operatorname {R e l}} ^ {\mathrm {M L E}} \left(r _ {\pi_ {\alpha} ^ {*}}\right) - \tilde {L} _ {\alpha , \operatorname {R e l}} ^ {\mathrm {M L E}} \left(r _ {\hat {\pi}}\right) \\ = L _ {\alpha , \text {R e l}} ^ {\text {M L E}} \left(\pi_ {\alpha} ^ {*}\right) - L _ {\alpha , \text {R e l}} ^ {\text {M L E}} (\hat {\pi}) = L _ {\alpha} ^ {\text {M L E}} \left(\pi_ {\alpha} ^ {*}\right) - L _ {\alpha} ^ {\text {M L E}} (\hat {\pi}) \leq O \left(\epsilon_ {\text {s t a t}} ^ {\text {M L E}}\right), \\ \end{array}
$$
where the last inequality holds by Lemma 9. Note that
$$
\begin{array}{l} \left\| r _ {\pi_ {\alpha} ^ {*}} - r _ {\hat {\pi}} \right\| _ {2, d _ {\alpha} ^ {*}} ^ {2} = \mathbb {E} _ {(s, a, g) \sim d _ {\alpha} ^ {*}} \left[ \left(r _ {\hat {\pi}} (a | s, g) - r _ {\pi_ {\alpha} ^ {*}} (a | s, g)\right) ^ {2} \right] \\ = \mathbb {E} _ {(s, a, g) \sim d _ {\alpha} ^ {*}} \left[ \left(\frac {\hat {\pi} (a | s , g)}{\pi_ {\alpha} ^ {*} (a | s , g)} - 1\right) ^ {2} \right] \\ \gtrsim \mathbb {E} _ {(s, g) \sim d _ {\alpha} ^ {*}} \left[ \left\| \hat {\pi} (\cdot | s, g) - \pi_ {\alpha} ^ {*} (\cdot | s, g) \right\| _ {\mathsf {T V}} ^ {2} \right] \\ \geq \left(\mathbb {E} _ {(s, g) \sim d _ {\alpha} ^ {*}} \left[ \| \hat {\pi} (\cdot | s, g) - \pi_ {\alpha} ^ {*} (\cdot | s, g) \| _ {\mathsf {T V}} \right]\right) ^ {2} \\ \end{array}
$$
where the last inequality holds since TV distance is upper bounded by  $\chi^2$  distance. Therefore, we can finally obtain that
$$
\mathbb {E} _ {(s, g) \sim d _ {\alpha} ^ {*}} \left[ \| \hat {\pi} (\cdot | s, g) - \pi_ {\alpha} ^ {*} (\cdot | s, g) \| _ {\mathsf {T V}} \right] \leq O \left(\sqrt {\epsilon_ {\mathrm {s t a t}} ^ {\mathrm {M L E}} / \tau^ {2}}\right).
$$
Proof of Theorem 1. We condition on the high probability event in Lemma 8. By performance difference lemma [Agarwal et al., 2019], we have
$$
\begin{array}{l} J \left(\pi_ {\alpha} ^ {*}\right) - J (\hat {\pi}) = \mathbb {E} _ {(s, a, g) \sim d _ {\alpha} ^ {*}} \left[ A ^ {\hat {\pi}} (s, a; g) \right] \\ = \mathbb {E} _ {(s, g) \sim d _ {\mathrm {s}} ^ {*}} \left[ \mathbb {E} _ {a \sim \pi_ {\mathrm {s}} ^ {*} (\cdot | s, g)} A ^ {\hat {\pi}} (s, a; g) - \mathbb {E} _ {a \sim \hat {\pi} (\cdot | s, g)} A ^ {\hat {\pi}} (s, a; g) \right] \\ \lesssim V _ {\max } \mathbb {E} _ {(s, g) \sim d _ {\alpha} ^ {*}} \left[ \| \pi_ {\alpha} ^ {*} (\cdot | s, g) - \hat {\pi} (\cdot | s, g) \| _ {1} \right] \\ \lesssim V _ {\max } \mathbb {E} _ {(s, g) \sim d _ {\alpha} ^ {*}} [ \| \pi_ {\alpha} ^ {*} (\cdot | s, g) - \hat {\pi} (\cdot | s, g) \| _ {\mathsf {T V}} ] \\ \lesssim V _ {\mathrm {m a x}} \sqrt {\frac {\epsilon_ {\mathrm {s t a t}} ^ {\mathrm {M L E}}}{\tau^ {2}}}, \\ \end{array}
$$
where the last inequality holds by Lemma 2.
D Statistical Rate of the Suboptimality in Different Settings
In this section, we analyze the statistical rate of the suboptimality of the output policy  $\hat{\pi}$  by Algorithm 1 in different settings.
D.1 Deterministic Settings
Proof of Theorem 2. By Theorem 1 and Proposition 3.1,
$$
\begin{array}{l} J (\pi^ {*}) - J (\hat {\pi}) \\ = J \left(\pi^ {*}\right) - J \left(\pi_ {\alpha} ^ {*}\right) + J \left(\pi_ {\alpha} ^ {*}\right) - J (\hat {\pi}) \\ \lesssim \alpha (C _ {\alpha} ^ {*}) ^ {2} + V _ {\max } \sqrt {\frac {\epsilon_ {\mathrm {s t a t}} ^ {\mathrm {M L E}}}{\tau^ {2}}} \\ \lesssim \alpha (C _ {\alpha} ^ {*}) ^ {2} + \frac {V _ {\operatorname* {m a x}} \sqrt {\log (1 / \tau)}}{\tau} \sqrt {\sqrt {\epsilon_ {\mathrm {s t a t}} / \alpha^ {2}} + C _ {\alpha} ^ {*} \sqrt {\frac {\log (| \Pi | / \delta)}{N}} + \frac {V _ {\operatorname* {m a x}}}{\alpha N} \log (| \Pi | / \delta)}. \\ \end{array}
$$
Since  $\epsilon_{\mathrm{stat}} \asymp V_{\mathrm{max}}^2 \sqrt{\frac{\log(|\mathcal{V}| / \delta)}{N}}$ , we can further obtain that
$$
\begin{array}{l} J (\pi^ {*}) - J (\hat {\pi}) \\ \lesssim \alpha (C _ {\alpha} ^ {*}) ^ {2} + \frac {V _ {\max } \sqrt {\log (1 / \tau)}}{\tau} \sqrt {\frac {V _ {\max }}{\alpha} \left(\frac {\log (| \mathcal {V} | / \delta)}{N}\right) ^ {1 / 4}} + C _ {\alpha} ^ {*} \sqrt {\frac {\log (| \Pi | / \delta)}{N}} + \frac {V _ {\max }}{\alpha N} \log (| \Pi | / \delta) \\ \lesssim \alpha (C _ {\alpha} ^ {*}) ^ {2} + \frac {V _ {\operatorname* {m a x}} \sqrt {\log (1 / \tau)}}{\tau} \sqrt {\frac {V _ {\operatorname* {m a x}} C _ {\alpha} ^ {*}}{\alpha N ^ {1 / 4}} \log (| \mathcal {V} | | \Pi | / \delta)} \\ \lesssim \left(\frac {V _ {\operatorname* {m a x}} ^ {3} (C _ {\alpha} ^ {*}) ^ {3} \log (1 / \tau) \log (| \mathcal {V} | | \Pi | / \delta)}{\tau^ {2} N ^ {1 / 4}}\right) ^ {1 / 3} \\ \end{array}
$$
where  $\alpha \asymp \left(\frac{V_{\mathrm{max}}^3\log(1 / \tau)\log(|\mathcal{V}||\Pi| / \delta)}{\tau^2(C_\alpha^*)^3N^{1 / 4}}\right)^{1 / 3}$ .
D.2 Stochastic Settings
Theorem 3 (Statistical rate of the suboptimality in stochastic settings). Under Assumptions 1 to 5, with probability at least  $1 - \delta$ , the output policy  $\hat{\pi}$  by Algorithm 1 (with the choice of Algorithm 3 for  $V$ -learning in stochastic settings) satisfies
$$
J (\pi^ {*}) - J (\hat {\pi}) \lesssim \left(\frac {V _ {\operatorname* {m a x}} ^ {3} (C _ {\alpha} ^ {*}) ^ {3} \log (1 / \tau) \log (| \mathcal {V} | | \mathcal {P} | | \Pi | / \delta)}{\tau^ {2} N ^ {1 / 4}}\right) ^ {1 / 3}
$$
if we choose  $\alpha \asymp \left(\frac{V_{\mathrm{max}}^3\log(1 / \tau)\log(|\mathcal{V}||\mathcal{P}||\Pi| / \delta)}{\tau^2(C_\alpha^*)^3N^{1 / 4}}\right)^{1 / 3}$  and assume  $N = N_0$
Proof. The proof is identical to the proof of Theorem 2 except that we replace  $\epsilon_{\mathrm{stat}}$  with  $\epsilon_{\mathrm{stat}}^{\mathrm{stochastic}}$ .