SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since the all-policy concentrability assumption is strong and can often be violated in practice, recent algorithms requiring only partial data coverage assumptions are developed based on single-policy  $\ell_{\infty}$  concentrability (Rashidinejad et al., 2021; Zhan et al., 2022; Rashidinejad et al., 2022; Chen and Jiang, 2022; Ozdaglar et al., 2022). However, This could still be restrictive even if only single-policy concentrability is required since the  $\ell_{\infty}$  concentrability is an upper bound of density ratios over all state-action pairs.
Milder versions of  $\ell_{\infty}$  concentrability have been studied in both all-policy concentrability framework (Xie and Jiang, 2020, 2021; Feng et al., 2019; Uehara et al., 2020) or and single-policy concentrability (Uehara and Sun, 2021; Xie et al., 2021; Song et al., 2022; Cheng et al., 2022). However, these works based on milder versions of  $\ell_{\infty}$  single-policy concentrability are either computationally intractable (Uehara and Sun, 2021; Xie et al., 2021) or suffer a suboptimal statistical rate (Cheng et al., 2022).
Our work uses  $\ell_{2}$  concentrability version, which also appears in Xie and Jiang (2020). In particular, they also use weighted average Bellman error in their algorithm. However, their algorithm requires all-policy concentrability assumptions and thus cannot deal with partial dataset coverage. To the best of our knowledge among previous works, only Uehara and Sun (2021) used  $\ell_{2}$  single-policy concentrability to characterize data coverage, but their algorithm is designed for model-based settings and are computationally intractable. Another closely related work is Uehara et al. (2020), which also uses weighted average Bellman error. However, their algorithm is in the off-policy evaluation (OPE) framework, and they use  $\ell_{\infty}$  concentrability version to characterize dataset coverage.
A.2 Conservative offline reinforcement learning
To address partial dataset coverage in offline RL, a line of recent applied works studies conservative algorithms, which can be divided into several categories.
The first category enforces the learned policy to be close to the behavior policy (or equivalently, dataset), which ensures that candidate policies not well covered by the dataset are eliminated. This can be accomplished by either adding constraints explicitly (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Jaques et al., 2019; Siegel et al., 2020; Ghasemipour et al., 2020; Fujimoto and Gu, 2021), implicitly (Peng et al., 2019; Nair et al., 2020), or by importance sampling with bounded ratio (Swaminathan and Joachims, 2015; Liu et al., 2019b; Nachum et al., 2019b; Zhang et al., 2020c,b; Lee et al., 2021).
The second category consists of model-based methods such as adversarial model learning (Rigter et al., 2022), learning pessimistic models (Kidambi et al., 2020; Guo et al., 2022), using model ensembles to form penalties (Yu et al., 2020), or combining model and values (Yu et al., 2021).
The last category aims to learn conservative values such as fitted Q-iteration using conservative update (Liu et al., 2020), conservative Q-learning (CQL) (Kumar et al., 2020), critic regularization (Kostrikov et al., 2021), and subtracting penalties (RezaEIFar et al., 2022).
On the theoretical side, many works use some form of uncertainty quantification to design to ensure pessimism (Yin and Wang, 2021; Kumar et al., 2021; Uehara et al., 2021; Yin et al., 2022; Zhang et al., 2022; Yan et al., 2022; Shi and Chi, 2022; Wang et al., 2022). Except for uncertainty
quantification, in linear function approximation settings, Zanette et al. (2021) uses value function perturbation combined with the actor-critic method. Recent advances in the theory of pessimistic algorithms include MIS (Zhan et al., 2022; Chen and Jiang, 2022) and adversarially trained actor-critic (ATAC) (Cheng et al., 2022). In particular, our algorithm is built based on MIS combined with the actor-critic method.
B Proof of Proposition 2.1
The first part of the proposition is easy to see:
$$
(C _ {\ell_ {2}} ^ {\pi}) ^ {2} = \| w ^ {\pi} \| _ {2, \mu} ^ {2} = \mathbb {E} _ {\mu} [ (w ^ {\pi} (s, a)) ^ {2} ] \leq \mathbb {E} _ {\mu} \left[ C _ {\ell_ {\infty}} ^ {\pi} \frac {d ^ {\pi} (s , a)}{\mu (s , a)} \right] = C _ {\ell_ {\infty}} ^ {\pi} \mathbb {E} _ {\mu} \left[ \frac {d ^ {\pi} (s , a)}{\mu (s , a)} \right] = C _ {\ell_ {\infty}} ^ {\pi},
$$
$$
C _ {\ell_ {2}} ^ {\pi} = \| w ^ {\pi} \| _ {2, \mu} \leq \| w ^ {\pi} \| _ {\infty} = C _ {\infty} ^ {\pi}.
$$
For the second part, consider the case where for a fixed policy  $\pi$  and for any  $s\in S,a\in \mathcal{A}$ , there exists  $f\in \mathcal{F}$  such that  $(f - \mathcal{T}^{\pi}f)(s,a)$  is non-zero only at  $(s,a)$  and is zero otherwise. The Bellman-consistent concentrability defines  $C_{\mathrm{Bellman}}^{\pi}$  to be the smallest constant that
$$
\begin{array}{l} \max _ {f \in \mathcal {F}} \frac {\| f - \mathcal {T} ^ {\pi} f \| _ {2 , d ^ {\pi}} ^ {2}}{\| f - \mathcal {T} ^ {\pi} f \| _ {2 , \mu} ^ {2}} \leq C _ {\mathrm {B e l l m a n}} ^ {\pi} \\ \Longrightarrow \max  _ {s, a} \frac {(f - \mathcal {T} ^ {\pi} f) ^ {2} (s , a) d ^ {\pi} (s , a)}{(f - \mathcal {T} ^ {\pi} f) ^ {2} (s , a) \mu (s , a)} \leq C _ {\mathrm {B e l l m a n}} ^ {\pi} \\ \Rightarrow \max  _ {s, a} \frac {d ^ {\pi} (s , a)}{\mu (s , a)} \leq C _ {\text {B e l l m a n}} ^ {\pi}, \\ \end{array}
$$
which makes it equal to the  $\ell_{\infty}$  variant. On the other hand, the  $\ell_{2}$  variant only requires the average importance weights to be bounded:
$$
\sum_ {s, a} \left(\frac {d ^ {\pi} (s , a)}{\mu (s , a)}\right) ^ {2} \mu (s, a) = \mathbb {E} _ {d ^ {\pi}} \left[ \frac {d ^ {\pi} (s , a)}{\mu (s , a)} \right] \leq \left(C _ {\ell_ {2}} ^ {\pi}\right) ^ {2}.
$$
C Theoretical Analysis of the Main Theorem
In this section, we provide theoretical proof of our main theorem (Theorem 1). We first present two key lemmas in Appendix C.1 and then prove the main theorem in Appendix C.3. For convenience, we always assume Assumptions 1 to 3 hold.
C.1 Key lemmas
The first lemma shows that with high probability, the population version of our weighted average Bellman error regularizer is close to the empirical version.
Lemma 2 (Concentration of the empirical regularizer). With probability at least  $1 - \delta$ , for any  $f \in \mathcal{F}$ ,  $\pi \in \Pi$ , we have
$$
\left| \mathcal {E} _ {\mu} (\pi , f) - \mathcal {E} _ {\mathcal {D}} (\pi , f) \right| \leq \epsilon_ {\text {s t a t}}.
$$
Proof. We condition on the high probability event in Lemma 4. For any  $f \in \mathcal{F}$  and  $\pi \in \Pi$ , define  $w_{\pi, f}^{*} = \arg \max_{w \in \mathcal{W}} \mathcal{E}_{\mu}(\pi, f) = \arg \max_{w \in \mathcal{W}} |\mathbb{E}_{\mu}[w(s, a)(f - \mathcal{T}^{\pi}f)(s, a)]|$  and define  $\hat{w}_{\pi, f} =$
$\arg \max_{w\in \mathcal{W}}\mathcal{E}_{\mathcal{D}}(\pi ,f) = \arg \max_{w\in \mathcal{W}}\left|\frac{1}{N}\sum_{(s,a,r,s')\in \mathcal{D}}w(s,a)(f(s,a) - r - \gamma f(s',\pi))\right|.$  Then
$$
\begin{array}{l} \mathcal {E} _ {\mu} (\pi , f) - \mathcal {E} _ {\mathcal {D}} (\pi , f) \\ = \left| \mathbb {E} _ {\mu} \left[ w _ {\pi , f} ^ {*} (s, a) \left(f - \mathcal {T} ^ {\pi} f\right) (s, a) \right] \right| - \left| \frac {1}{N} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} \hat {w} _ {\pi , f} (s, a) \left(f (s, a) - r - \gamma f \left(s ^ {\prime}, \pi\right)\right) \right| \\ = \left| \mathbb {E} _ {\mu} \left[ w _ {\pi , f} ^ {*} (s, a) \left(f - \mathcal {T} ^ {\pi} f\right) (s, a) \right] \right| - \left| \mathbb {E} _ {\mu} \left[ \hat {w} _ {\pi , f} (s, a) \left(f - \mathcal {T} ^ {\pi} f\right) (s, a) \right] \right| \\ + \left| \mathbb {E} _ {\mu} \left[ \hat {w} _ {\pi , f} (s, a) \left(f - \mathcal {T} ^ {\pi} f\right) (s, a) \right] \right| - \left| \frac {1}{N} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} \hat {w} _ {\pi , f} (s, a) \left(f (s, a) - r - \gamma f \left(s ^ {\prime}, \pi\right)\right) \right| \\ \geq 0 - \epsilon_ {\text {s t a t}} = - \epsilon_ {\text {s t a t}}, \\ \end{array}
$$
where the inequality holds by the optimality of  $w_{\pi ,f}^{*}$  and Lemma 4. Similarly,
$$
\begin{array}{l} \mathcal {E} _ {\mu} (\pi , f) - \mathcal {E} _ {\mathcal {D}} (\pi , f) \\ = \left| \mathbb {E} _ {\mu} \left[ w _ {\pi , f} ^ {*} (s, a) \left(f - \mathcal {T} ^ {\pi} f\right) (s, a) \right] \right| - \left| \frac {1}{N} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} w _ {\pi , f} ^ {*} (s, a) \left(f (s, a) - r - \gamma f \left(s ^ {\prime}, \pi\right)\right) \right| \\ + \left| \frac {1}{N} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} w _ {\pi , f} ^ {*} (s, a) \left(f (s, a) - r - \gamma f \left(s ^ {\prime}, \pi\right)\right) \right| \\ - \left| \frac {1}{N} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} \hat {w} _ {\pi , f} (s, a) \left(f (s, a) - r - \gamma f \left(s ^ {\prime}, \pi\right)\right) \right| \\ \end{array}
$$
$$
\leq \epsilon_ {\text {s t a t}} + 0 = \epsilon_ {\text {s t a t}},
$$
where the inequality holds by the optimality of  $\hat{w}_{\pi ,f}$  and Lemma 4.
The next lemma provides a high-probability upper bound of the empirical weighted average Bellman error of  $f_{\pi}$  w.r.t.  $\pi$ , where  $f_{\pi}$  is the (approximate)  $Q$ -function of  $\pi$ .
Lemma 3 (Empirical weighted average Bellman error of approximate  $Q$  function). With probability at least  $1 - \delta$ , for any  $\pi \in \Pi$ , we have
$$
\mathcal {E} _ {\mathcal {D}} (\pi , f _ {\pi}) \leq C _ {\ell_ {2}} ^ {\star} \sqrt {\epsilon_ {\mathcal {F}}} + \epsilon_ {s t a t}.
$$
where  $f_{\pi} = \arg \min_{f\in \mathcal{F}}\max_{admissible\nu}\| f - \mathcal{T}^{\pi}f\|_{2,\nu}^{2}$
Proof. We condition on the high probability event in Lemma 2. Since
$$
\begin{array}{l} \mathcal {E} _ {\mu} (\pi , f _ {\pi}) = \max  _ {w \in \mathcal {W}} | \mathbb {E} _ {\mu} [ w (s, a) (f _ {\pi} - \mathcal {T} ^ {\pi} f _ {\pi}) (s, a) ] | \\ \leq \max _ {w \in \mathcal {W}} \| w \| _ {2, \mu} \| f _ {\pi} - \mathcal {T} ^ {\pi} f _ {\pi} \| _ {2, \mu} \\ \leq C _ {\ell_ {2}} ^ {\star} \sqrt {\epsilon_ {F}}, \\ \end{array}
$$
where the first inequality is by Cauchy-Schwarz inequality and the second inequality is by the definition of  $f_{\pi}$  and Assumption 1, we can immediately obtain that
$$
\mathcal {E} _ {\mathcal {D}} (\pi , f _ {\pi}) \leq \mathcal {E} _ {\mu} (\pi , f _ {\pi}) + \epsilon_ {\text {s t a t}} = C _ {\ell_ {2}} ^ {\star} \sqrt {\epsilon_ {\mathcal {F}}} + \epsilon_ {\text {s t a t}}.
$$
C.2 Complementary lemmas
We provide two complementary lemmas in this section, which are both high-probability concentration inequalities.
Lemma 4 (Concentration of weighted average Bellman error). With probability at least  $1 - \delta$ , for any  $f \in \mathcal{F}$ ,  $\pi \in \Pi$  and  $w \in \mathcal{W}$ , we have
$$
\begin{array}{l} \left| \left| \mathbb {E} _ {\mu} \left[ (f - \mathcal {T} ^ {\pi} f) w \right] \right| - \left| \frac {1}{N} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} w (s, a) \left(f (s, a) - r - \gamma f \left(s ^ {\prime}, \pi\right)\right) \right| \right| \\ \leq O \left(V _ {\max } C _ {\ell_ {2}} ^ {\star} \sqrt {\frac {\log (| \mathcal {F} | | \Pi | | \mathcal {W} | / \delta)}{N}} + \frac {V _ {\max } B _ {w} \log (| \mathcal {F} | | \Pi | | \mathcal {W} | / \delta)}{N}\right) \triangleq \epsilon_ {s t a t}. \\ \end{array}
$$
Proof. It suffices to bound
$$
\left| \mathbb {E} _ {\mu} [ (f - \mathcal {T} ^ {\pi} f) w ] - \frac {1}{N} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} w (s, a) (f (s, a) - r - \gamma f (s ^ {\prime}, \pi)) \right|
$$
for any fixed  $f\in \mathcal{F},\pi \in \Pi ,w\in \mathcal{W}$  , and a union bound and the triangle inequality conclude.
$$
\begin{array}{l} \mathbb {E} _ {\mu} \left[ \frac {1}{N} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} (f (s, a) - r - \gamma f (s ^ {\prime}, \pi)) w (s, a) \right] \\ = \mathbb {E} _ {(s, a) \sim \mu , r \sim r (s, a), s ^ {\prime} \sim P (\cdot | s, a)} [ (f (s, a) - r - \gamma f (s ^ {\prime}, \pi)) w (s, a) ] \\ = \mathbb {E} _ {(s, a) \sim \mu} \mathbb {E} _ {r \sim r (s, a), s ^ {\prime} \sim P (\cdot | s, a)} \left[ \left(f (s, a) - r - \gamma f \left(s ^ {\prime}, \pi\right)\right) w (s, a) | s, a \right] \\ = \mathbb {E} _ {(s, a) \sim \mu} \left[ \left(f (s, a) - \mathcal {T} ^ {\pi} f (s, a)\right) w (s, a) \right]. \\ \end{array}
$$
Then by Bernstein's inequality, we have that with probability at least  $1 - \delta$
$$
\begin{array}{l} \left| \mathbb {E} _ {\mu} [ (f - \mathcal {T} ^ {\pi} f) w ] - \frac {1}{N} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} (f (s, a) - r - \gamma f (s ^ {\prime}, \pi)) w (s, a) \right| \\ \leq O \left(\sqrt {\frac {\operatorname {V a r} _ {\mu} [ (f - \mathcal {T} ^ {\pi} f) w ] \log (1 / \delta)}{N}} + \frac {V _ {\max} B _ {w} \log (1 / \delta)}{N}\right). \\ \end{array}
$$
$$
\operatorname {V a r} _ {\mu} \left[ (f - \mathcal {T} ^ {\pi} f) w \right] \leq \mathbb {E} _ {\mu} \left[ (f - \mathcal {T} ^ {\pi} f) ^ {2} w ^ {2} \right] \leq O \left(V _ {\max } ^ {2} \| w \| _ {2, \mu} ^ {2}\right) \leq O \left(V _ {\max } ^ {2} \left(C _ {\ell_ {2}} ^ {\star}\right) ^ {2}\right),
$$
we can obtain that
$$
\begin{array}{l} \left| \mathbb {E} _ {\mu} [ (f - \mathcal {T} ^ {\pi} f) w ] - \frac {1}{N} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} (f (s, a) - r - \gamma f (s ^ {\prime}, \pi)) w (s, a) \right| \\ \leq O \left(V _ {\max } C _ {\ell_ {2}} ^ {\star} \sqrt {\frac {\log (1 / \delta)}{N}} + \frac {V _ {\max } B _ {w} \log (1 / \delta)}{N}\right), \\ \end{array}
$$
which implies the result.
Lemma 5 (Concentration of the actor's objective). With probability at least  $1 - \delta$ , for any  $f \in \mathcal{F}$ ,  $\pi \in \Pi$ , we have
$$
\left| \mathcal {L} _ {\mu} (\pi , f) - \mathcal {L} _ {\mathcal {D}} (\pi , f) \right| \leq O \left(V _ {\max } \sqrt {\frac {\log (| \mathcal {F} | | \Pi | / \delta)}{N}}\right) \leq \epsilon_ {s t a t}
$$
where  $\epsilon_{stat}$  is defined in Lemma 4.
Proof. Note that  $\mathbb{E}_{\mu}[\mathcal{L}_{\mathcal{D}}(\pi, f)] = \mathcal{L}_{\mu}(\pi, f)$  and  $|f(s, \pi) - f(s, a)| \leq O(V_{\max})$ . Applying a Hoeffding's inequality for any fixed  $f$ ,  $\pi$  and a union bound over all  $f \in \mathcal{F}$ ,  $\pi \in \Pi$ , we can obtain that with probability at least  $1 - \delta$ , it holds that
$$
\left| \mathcal {L} _ {\mu} (\pi , f) - \mathcal {L} _ {\mathcal {D}} (\pi , f) \right| \leq O \left(V _ {\max } \sqrt {\frac {\log (| \mathcal {F} | | \Pi | / \delta)}{N}}\right).
$$
for all  $f\in \mathcal{F}$  and  $\pi \in \Pi$
Now we are able to prove our main theorem equipped with lemmas in previous sections.
Proof of Theorem 1. By Lemma 1 and the definition of  $\bar{\pi}$ , we have
$$
\begin{array}{l} J (\pi) - J (\bar {\pi}) = \frac {1}{K} \sum_ {k = 1} ^ {K} (J (\pi) - J (\pi_ {k})) \\ = \frac {1}{K} \sum_ {k = 1} ^ {K} (\underbrace {\mathbb {E} _ {\mu} [ (f _ {k} - \mathcal {T} ^ {\pi_ {k}} f _ {k}) (s , a) ]} _ {(a)} + \underbrace {\mathbb {E} _ {\pi} [ (\mathcal {T} ^ {\pi_ {k}} f _ {k} - f _ {k}) (s , a) ]} _ {(b)}) \\ + \underbrace {\mathbb {E} _ {\pi} [ f _ {k} (s , \pi) - f _ {k} (s , \pi_ {k}) ]} _ {(c)} + \underbrace {\mathcal {L} _ {\mu} (\pi_ {k} , f _ {k}) - \mathcal {L} _ {\mu} (\pi_ {k} , Q ^ {\pi_ {k}})} _ {(d)}. \\ \end{array}
$$
Now we condition on the high probability event in Lemma 2 and Lemma 5 simultaneously and rescale  $\delta$  by  $1/2$  and apply a union bound. Note that  $d^{\pi} / \mu \in \mathcal{W}$ , which implies that  $(a) + (b) \leq 2\mathcal{E}_{\mu}(\pi_k, f_k) \leq 2\mathcal{E}_{\mathcal{D}}(\pi_k, f_k) + 2\epsilon_{\mathrm{stat}}$ , where the last inequality holds by Lemma 2. By Lemma 13 of Cheng et al. (2022), we can obtain that
$$
\left| \mathcal {L} _ {\mu} \left(\pi_ {k}, Q ^ {\pi_ {k}}\right) - \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {\pi_ {k}}\right) \right| \leq \left\| f _ {\pi_ {k}} - \mathcal {T} ^ {\pi_ {k}} f _ {\pi_ {k}} \right\| _ {2, \mu} + \left\| f _ {\pi_ {k}} - \mathcal {T} ^ {\pi_ {k}} f _ {\pi_ {k}} \right\| _ {2, d ^ {\pi_ {k}}} \leq O (\sqrt {\epsilon_ {\mathcal {F}}}),
$$
where  $f_{\pi_k} \triangleq \arg \min_{f \in \mathcal{F}} \max_{\mathrm{admissible} \, \nu} \| f - \mathcal{T}^\pi f \|_{2,\nu}^2$ . Also, by Lemma 5, we have
$$
\left| \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {k}\right) - \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) \right| + \left| \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {\pi_ {k}}\right) - \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, f _ {\pi_ {k}}\right) \right| \leq O \left(\epsilon_ {\text {s t a t}}\right).
$$
$$
\begin{array}{l} (a) + (b) + (d) \leq \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {k}\right) + 2 \mathcal {E} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) + 2 \epsilon_ {\text {s t a t}} - \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {\pi_ {k}}\right) + O (\sqrt {\epsilon_ {\mathcal {F}}}) \\ \leq \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) + 2 \mathcal {E} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) + O \left(\epsilon_ {\text {s t a t}}\right) - \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, f _ {\pi_ {k}}\right) + O \left(\sqrt {\epsilon_ {\mathcal {F}}}\right) \\ \leq \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, f _ {\pi_ {k}}\right) + 2 \mathcal {E} _ {\mathcal {D}} \left(\pi_ {k}, f _ {\pi_ {k}}\right) + O \left(\epsilon_ {\text {s t a t}}\right) - \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, f _ {\pi_ {k}}\right) + O \left(\sqrt {\epsilon_ {\mathcal {F}}}\right) \\ \leq O \left(\epsilon_ {\text {s t a t}} + C _ {\ell_ {2}} ^ {\star} \sqrt {\epsilon_ {\mathcal {F}}}\right). \\ \end{array}
$$
where the third inequality holds by the optimality of  $f_{k}$ , and the last inequality holds by Lemma 3. Therefore,
$$
J (\pi) - J (\bar {\pi}) \leq O \left(\epsilon_ {\text {s t a t}} + C _ {\ell_ {2}} ^ {\star} \sqrt {\epsilon_ {\mathcal {F}}}\right) + \epsilon_ {\text {o p t}} ^ {\pi}.
$$
D Analysis of Robust Policy Improvement
Proof of Theorem 2. By Lemma 1 and the definition of  $\bar{\pi}$ , we have
$$
\begin{array}{l} J (\mu) - J (\bar {\pi}) = \frac {1}{K} \sum_ {k = 1} ^ {K} (J (\mu) - J (\pi_ {k})) \\ = \frac {1}{K} \sum_ {k = 1} ^ {K} (\underbrace {\mathbb {E} _ {\mu} [ (f _ {k} - \mathcal {T} ^ {\pi_ {k}} f _ {k}) (s , a) ]} _ {(a)} + \underbrace {\mathbb {E} _ {\mu} [ (\mathcal {T} ^ {\pi_ {k}} f _ {k} - f _ {k}) (s , a) ]} _ {(b)}) \\ + \underbrace {\mathbb {E} _ {\pi} [ f _ {k} (s , \mu) - f _ {k} (s , \pi_ {k}) ]} _ {(c)} + \underbrace {\mathcal {L} _ {\mu} (\pi_ {k} , f _ {k}) - \mathcal {L} _ {\mu} (\pi_ {k} , Q ^ {\pi_ {k}})} _ {(d)}. \\ \end{array}
$$
Now we condition on the high probability event in Lemma 2 and Lemma 5 simultaneously and rescale  $\delta$  by  $1/2$  and apply a union bound. Note that  $(a) + (b) = 0$ . By Lemma 13 of Cheng et al. (2022), we can obtain that
$$
\left| \mathcal {L} _ {\mu} \left(\pi_ {k}, Q ^ {\pi_ {k}}\right) - \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {\pi_ {k}}\right) \right| \leq \| f _ {\pi_ {k}} - \mathcal {T} ^ {\pi_ {k}} f _ {\pi_ {k}} \| _ {2, \mu} + \| f _ {\pi_ {k}} - \mathcal {T} ^ {\pi_ {k}} f _ {\pi_ {k}} \| _ {2, d ^ {\pi_ {k}}} \leq O (\sqrt {\epsilon_ {\mathcal {F}}}), \tag {9}
$$
$$
\left| \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {k}\right) - \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) \right| + \left| \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {\pi_ {k}}\right) - \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, f _ {\pi_ {k}}\right) \right| \leq O \left(\epsilon_ {\text {s t a t}}\right). \tag {10}
$$
$$
\begin{array}{l} \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {k}\right) - \mathcal {L} _ {\mu} \left(\pi_ {k}, Q ^ {\pi_ {k}}\right) \\ \leq \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {k}\right) + \beta \mathcal {E} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) - \mathcal {L} _ {\mu} \left(\pi_ {k}, Q ^ {\pi_ {k}}\right) \quad \left(\mathcal {E} _ {\mathcal {D}} (\cdot , \cdot) \geq 0\right) \\ \leq \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {k}\right) + \beta \mathcal {E} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) - \mathcal {L} _ {\mu} \left(\pi_ {k}, Q ^ {\pi_ {k}}\right) - \beta \mathcal {E} _ {\mathcal {D}} \left(\pi_ {k}, f _ {\pi_ {k}}\right) \\ + \beta C _ {\ell_ {2}} ^ {\star} \sqrt {\epsilon_ {\mathcal {F}}} + \beta \epsilon_ {\text {s t a t}} \tag {Lemma3} \\ \end{array}
$$
$$
\begin{array}{l} \lesssim \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) + \beta \mathcal {E} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) - \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, f _ {\pi_ {k}}\right) - \beta \mathcal {E} _ {\mathcal {D}} \left(\pi_ {k}, f _ {\pi_ {k}}\right) \\ + (\beta + 1) \left(\epsilon_ {\text {s t a t}} + C _ {\ell_ {2}} ^ {\star} \sqrt {\epsilon_ {\mathcal {F}}}\right) \tag {(9), (10)} \\ \leq (\beta + 1) \left(\epsilon_ {\text {s t a t}} + C _ {\ell_ {2}} ^ {\star} \sqrt {\epsilon_ {\mathcal {F}}}\right). \quad \left(\text {O p t i m a l i t y} f _ {k}\right) \\ \end{array}
$$
$$
J (\mu) - J (\bar {\pi}) \lesssim (\beta + 1) \left(\epsilon_ {\mathrm {s t a t}} + C _ {\ell_ {2}} ^ {\star} \sqrt {\epsilon_ {\mathcal {F}}}\right) + \epsilon_ {\mathrm {o p t}} ^ {\pi}.
$$
E Analysis of the Suboptimality of ATAC
In this section, we prove Proposition 4.1 in Appendix E.1. For convenience, we use  $\operatorname{Bern}(p)$  to denote a Bernoulli variable with parameter  $p$ , and use  $\operatorname{Bin}(n,p)$  to denote a binomial variable with parameters  $n$  and  $p$ .
Construction of a two-arm bandit example. Assume there are two arms  $a_1, a_2$ . Assume  $\beta = \tilde{\Theta}(N^\alpha)$  where  $\alpha > \frac{1}{2}$ . Note that this is a more general case than  $\beta = \Theta(N^{2/3})$  as stated in the proposition. Let  $\Delta = \min\{\beta/N, 1/10\}$ . Assume the reward of the first arm is deterministic, i.e.,  $r(a_1) = 1/2 + \Delta$ , and  $r(a_2) = \operatorname{Bern}(1/2)$ . Let  $\Pi = \{\pi_1, \pi_2\}$ , where  $\pi_1(a_1) = 1$ ,  $\pi_1(a_2) = 0$ , and  $\pi_2(a_1) = 0$ ,  $\pi_2(a_2) = 1$ . Also, let  $\mathcal{F} = \{f_1, f_2\}$ , where  $f_1(a_1) = 1/2 + \Delta$ ,  $f_1(a_2) = 1/2$  and  $f_2(a_1) = 1/2 + \Delta$ ,  $f_2(a_2) = 1/2 + 2\Delta$ . Finally, the dataset distribution  $\mu$  satisfies  $\mu(a_2) = \frac{1}{N\Delta^2}$ ,  $\mu(a_1) = 1 - \mu(a_2)$ . Note that in bandit settings, for any policy  $\pi$  and any function  $f$ , we have  $Q^\pi = T^\pi f = r$ . Therefore, the example above satisfies both completeness and realizability assumptions.
Proof of the suboptimality of ATAC. The ATAC algorithm is simplified to
$$
\hat {\pi} \in \operatorname {a r g} \max  _ {\pi \in \Pi} \mathbb {E} _ {\mu} \left[ f ^ {\pi} (\pi) - f ^ {\pi} (a) \right],
$$
$$
\text {s . t .} f ^ {\pi} \in \arg \min  _ {f \in \mathcal {F}} \mathbb {E} _ {\mu} [ f (\pi) - f (a) ] + \beta \mathbb {E} _ {\mu} [ ((f - r) (a)) ^ {2} ],
$$
in bandit settings.
For convenience, we assume that  $\mathbb{P}_{\mu}(a = a_1) = \mathbb{P}_{\mathcal{D}}(a = a_1)$ . Note that by anti-concentration of the binomial distribution (Lemma 6), we have that with constant probability, it holds that  $\hat{r}(a_2) \geq \frac{1}{2} + \frac{2}{\sqrt{N\mu(a_2)}} = \frac{1}{2} + 2\Delta$ .
Conditioned on this event, we compute  $f^{\pi_1}$  and  $f^{\pi_2}$  separately. We have
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {D}} \left(\pi_ {1}, f _ {1}\right) = \mathbb {E} _ {\mu} \left[ f _ {1} \left(\pi_ {1}\right) - f _ {1} (a) \right] = f _ {1} \left(\pi_ {1}\right) - \mu \left(a _ {1}\right) f _ {1} \left(a _ {1}\right) - \mu \left(a _ {2}\right) f _ {1} \left(a _ {2}\right) \\ = \mu \left(a _ {2}\right) \left(f _ {1} \left(a _ {1}\right) - f _ {1} \left(a _ {2}\right)\right) = \frac {1}{N \Delta}, \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {D}} \left(\pi_ {1}, f _ {2}\right) = \mathbb {E} _ {\mu} \left[ f _ {2} \left(\pi_ {1}\right) - f _ {2} (a) \right] = f _ {2} \left(\pi_ {1}\right) - \mu \left(a _ {1}\right) f _ {2} \left(a _ {1}\right) - \mu \left(a _ {2}\right) f _ {2} \left(a _ {2}\right) \\ = \mu \left(a _ {2}\right) \left(f _ {2} \left(a _ {1}\right) - f _ {2} \left(a _ {2}\right)\right) = - \frac {1}{N \Delta}, \\ \end{array}
$$