SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4.1 Identification of Soft Q-functions
Consider an  $L^2$ -space  $\mathcal{H}$  where the inner product is defined as  $\langle h_1, h_2 \rangle = \mathbb{E}_{(s,a) \sim P_b} [h_1(s,a)h_2(s,a)]$ . Then we define two operators and a key function:
$$
P _ {\alpha} ^ {\star}: \mathcal {H} \ni f \mapsto \mathbb {E} _ {s ^ {\prime} \sim P (s, a), a ^ {\prime} \sim \pi_ {\alpha} ^ {\star}} [ f (s ^ {\prime}, a ^ {\prime}) \mid (s, a) = \cdot ] \in \mathcal {H},
$$
$$
\left\{P _ {\alpha} ^ {\star} \right\} ^ {\top}: \mathcal {H} \ni f \mapsto \int P (\cdot | s, a) \pi_ {\alpha} ^ {\star} (\cdot | \cdot) f (s, a) d (s, a) \in \mathcal {H},
$$
$$
l _ {\alpha} ^ {\star} (s, a) := \left\{ \begin{array}{l l} \frac {(I - \gamma \{P _ {\alpha} ^ {\star} \} ^ {\top}) ^ {- 1} (P _ {b} (s , a) q _ {\alpha} ^ {\star} (s , a))}{P _ {b} (s , a)} & (s, a) \in (\mathcal {S} \times \mathcal {A}) _ {b}, \\ 0 & (s, a) \neq (\mathcal {S} \times \mathcal {A}) _ {b}. \end{array} \right.
$$
These satisfy a key adjoint property, which we leverage to show  $(q_{\alpha}^{\star},l_{\alpha}^{\star})$  is a saddle point of  $L_{\alpha}(q,l)$
Lemma 1.  $\forall q\in \mathcal{H}$  , we have  $\langle l_\alpha^\star ,(I - \gamma P_\alpha^\star)q\rangle_{\mathcal{H}} = \langle q_\alpha^\star ,q\rangle_{\mathcal{H}}$
Our first assumption ensures that  $l_{\alpha}^{\star}$  exists.
Assumption 1. Suppose  $\| d_{\pi_{\alpha}^{\star},P_b} / P_b\|_{\infty} < \infty$ . Note the infinity norm  $\| \cdot \|_{\infty}$  is over  $S\times \mathcal{A}$ .
Proposition 1. Under Assumption 1, we have  $\| l_{\alpha}^{\star}\|_{\infty} < \infty$
Proposition 1 is immediate noting that  $(I - \gamma \{P_{\alpha}^{\star}\}^{\top})^{-1}(P_{b}(\cdot)q_{\alpha}^{\star}(\cdot)) = \sum_{t = 0}^{\infty}\gamma^{t}(\{P_{\alpha}^{\star}\}^{\top})^{t}(P_{b}q_{\alpha}^{\star})$  and recalling the discounted occupancy measure under  $\pi_{\alpha}^{\star}$  with initial distribution  $\mu_0$  is written as  $d_{\pi_{\alpha}^{\star},\mu_0} = (1 - \gamma)(I - \gamma \{P_\alpha^\star \}^\top)^{-1}(\mu_0)$ . Hence,  $\| l_{\alpha}^{\star}\|_{\infty}\leq (1 - \gamma)^{-1}R_{\max}\| d_{\pi_{\alpha ,P_b}^{\star}} / P_b\|_{\infty}$ . Note that  $\| d_{\pi_{\alpha ,P_b}^{\star}} / P_b\|_{\infty}$  crucially differs with the standard density-ratio-based concentrability coefficient  $\| d_{\pi_{\alpha}^{\star},\mu_0} / P_b\|_{\infty}$  in offline RL. Unlike  $\| d_{\pi_{\alpha}^{\star},P_b} / P_b\|_{\infty}$ , the value of  $\| d_{\pi_{\alpha}^{\star},\mu_0} / P_b\|_{\infty}$  can be infinite when the initial distribution  $\mu_0$  is not covered by offline data  $P_b$  as the practical motivating example is explained in the footnote in Section 1 and Example 2.
Our next assumption ensures  $q_{\alpha}^{\star} \geq 0$ , which also guarantees that  $l_{\alpha}^{\star} \geq 0$ .
Assumption 2. Suppose  $\alpha \log \| \pi_{\alpha}^{*} / \pi_{b}\|_{\infty}\leq R_{\min}$
Assumption 2 can be satisfied by rescaling reward (i.e., rescaling  $R_{\mathrm{min}}$ ) as long as  $\| \pi_{\alpha}^{*} / \pi_{b}\|_{\infty}$  is finite. Hence, it is very mild. Putting Lemma 1 together with our assumptions we have the following.
Lemma 2. Suppose Assumptions 1 and 2 hold. Then,  $(q_{\alpha}^{\star},l_{\alpha}^{\star})$  is a saddle point of  $L_{\alpha}(q,l)$  over  $q\in \mathcal{H},l\in \mathcal{H}$ , i.e.,  $L_{\alpha}(q,l_{\alpha}^{\star})\geq L_{\alpha}(q_{\alpha}^{\star},l_{\alpha}^{\star})\geq L_{\alpha}(q_{\alpha}^{\star},l)\forall q\in \mathcal{H},\forall l\in \mathcal{H}$ .
Recall that a point  $(\tilde{q},\tilde{l})$  is a saddle point if and only if the strong duality holds, and  $\tilde{q}\in$  arg  $\min_{q\in \mathcal{H}}\sup_{l\in \mathcal{H}}L_{\alpha}(q,l),\tilde{l}\in$  arg  $\max_{l\in \mathcal{H}}\inf_{q\in \mathcal{H}}L_{\alpha}(q,l)$  using the general characterization (Bertsekas, 2009). Hence, Lemma 2 ensures  $q_{\alpha}^{\star}\in$  arg  $\min_{q\in \mathcal{H}}\max_{l\in \mathcal{H}}L_{\alpha}(q,l)$
Next, we consider the constrained optimization problem when we use function classes  $\mathcal{Q} \subset \mathcal{H}, \mathcal{L} \subset \mathcal{H}$ . As long as the saddle point is included in  $(\mathcal{Q}, \mathcal{L})$ , we can prove that  $q_{\alpha}^{\star}$  is a unique minimaxer.
Lemma 3. Suppose Assumptions 1 and 2 hold,  $q_{\alpha}^{\star} \in \mathcal{Q}$ , and  $l_{\alpha}^{\star} \in \mathcal{L}$ . Then, we have that  $q_{\alpha}^{\star} = \arg \min_{q \in \mathcal{Q}} \sup_{l \in \mathcal{L}} L_{\alpha}(q, l)$  on the support  $(\mathcal{S} \times \mathcal{A})_b$ .
This establishes that realizability  $(q_{\alpha}^{\star}\in \mathcal{Q},l_{\alpha}^{\star}\in \mathcal{L})$  is sufficient to identify  $q_{\alpha}^{\star}$  on the offline data distribution. At a high level,  $q_{\alpha}^{\star}\in \arg \min_{q\in \mathcal{Q}}\sup_{l\in \mathcal{L}}L_{\alpha}(q,l)$  is established through the invariance of saddle points, i.e., saddle points over original sets remain saddle points over restricted sets. Its uniqueness is verified by the strong convexity in  $q$  of  $L_{\alpha}(q,l)$  induced by  $\mathbb{E}_{(s,a)\sim P_b}[q^2 (s,a)]$
3We use the notation  $\cdot^{\top}$  because  $P_{\alpha}^{\star \top}$  is interpreted as the adjoint operator in the non-weighted  $L^2$  -space.
4.2  $L^2$ -convergence Rate for Soft Q-estimators
Based on the population-level results in Section 4.1, we give a finite-sample error analysis of  $\hat{q}_{\alpha}$
Assumption 3 (Realizability of soft Q-function). Suppose  $q_{\alpha}^{\star} \in \mathcal{Q}$  and  $\|q\|_{\infty} \leq B_{\mathcal{Q}} \forall q \in \mathcal{Q}$ .
Assumption 4 (Realizability of Lagrange multiplier). Suppose  $l_{\alpha}^{\star}\in \mathcal{L}$  and  $\| l\|_{\infty}\leq B_{\mathcal{L}}\forall l\in \mathcal{L}$
It is natural to set  $B_{\mathcal{Q}} = (1 - \gamma)^{-1}R_{\max}$  and  $B_{\mathcal{L}} = (1 - \gamma)^{-1}R_{\max}\| d_{\pi_{\alpha}^{*},P_b} / P_b\|_{\infty}$ , but letting these be arbitrary offers further flexibility to our results.
Theorem 1 ( $L^2$ -convergence of soft Q-estimators). Suppose Assumptions 1, 2, 3, and 4 hold. Then, with probability  $1 - \delta$ , the  $L^2$ -error  $\| \hat{q}_{\alpha} - q_{\alpha}^{\star}\|_2$  is upper-bounded by
$$
c \left(\mathcal {B} _ {\mathcal {Q}} ^ {2} + \mathcal {B} _ {\mathcal {Q}} \mathcal {B} _ {\mathcal {L}} \left\{\alpha + \ln (| \mathcal {A} |) \right\}\right) ^ {1 / 2} \left(\ln (| \mathcal {Q} | | \mathcal {L} | / \delta) / n\right) ^ {1 / 4}.
$$
Our result is significant as it relies on realizability-type conditions rather than Bellman closedness. Since the majority of existing works focus on non-regularized Q-functions, we postpone the comparison to these existing works to the next section. Note when  $\mathcal{Q}$  and  $\mathcal{L}$  are infinite, we can easily replace  $|\mathcal{Q}|$ ,  $|\mathcal{L}|$  with their  $L^{\infty}$ -covering numbers following Uehara et al. (2021). Details are given in the appendix.
4.3  $L^2$ -convergence Rate for  $Q^{\star}$ -functions
Next, we give analogous finite-sample error analysis of  $\hat{q}_0$  leveraging the same reasoning.
Assumption 5 (Realizability of  $Q^{\star}$ -functions). Suppose  $q^{\star} \in \mathcal{Q}$  and  $\| q \|_{\infty} \leq B_{\mathcal{Q}} \forall q \in \mathcal{Q}$ .
Next, we define the Lagrange multiplier:
$$
\left\{P ^ {\star} \right\} ^ {\top}: \mathcal {H} \ni f \mapsto \int P (\cdot | s, a) \pi^ {\star} (\cdot | \cdot) f (s, a) d \mu (s, a) \in \mathcal {H},
$$
$$
l ^ {\star} := \left\{\left(I - \gamma \left\{P ^ {\star} \right\} ^ {\top}\right) ^ {- 1} \left(q ^ {\star} P _ {\pi_ {b}}\right) \right\} / P _ {\pi_ {b}}.
$$
While  $l^{\star}$  involves the density ratio, this is always well-defined as long as  $\| d_{\pi^{\star},P_b} / P_b\|_{\infty} < \infty$ . Then, it can be similarly established that  $(q^{\star},l^{\star})$  is a saddle point of  $L_0(q,l)$  over  $q\in \mathcal{H},l\in \mathcal{H}$  as we show in Lemma 2. We lastly require its realizability.
Assumption 6 (Realizability of Lagrange multiplier). Suppose  $\| d_{\pi^{\star},P_b} / P_b\|_{\infty} < \infty$  and  $l^{\star}\in \mathcal{L}$  Further suppose  $\| l\|_{\infty}\leq B_{\mathcal{L}}\forall l\in \mathcal{L}$
Theorem 2 ( $L^2$ -convergence of Q-estimators). Suppose Assumptions 5 and 6 hold. Then, with probability  $1 - \delta$ , the  $L^2$ -error  $\| \hat{q}_0 - q^\star \|_2$  is upper-bounded by
$$
c \left(\mathcal {B} _ {\mathcal {Q}} ^ {2} + \mathcal {B} _ {\mathcal {L}} \mathcal {B} _ {\mathcal {Q}}\right) ^ {1 / 2} (\ln (| \mathcal {Q} | | \mathcal {L} | / \delta) / n) ^ {1 / 4}.
$$
To the best of our knowledge, this is the first guarantee on  $L^2$  errors for learning  $q^{*}$  using general function approximation without relying on Bellman completeness. This is highly nontrivial, and we have carefully crafted our algorithm to obtain this guarantee. Existing results are often specific to particular models, such as linear models (Shi et al., 2022a), or they require Bellman completeness (Chen and Jiang, 2019; Chen and Qi, 2022), or they are limited to offline policy evaluation scenarios (Huang and Jiang, 2022) (i.e., cases involving linear Bellman operators, but nonlinear Bellman operators). Actually, it seems that even under the assumption of Bellman completeness, obtaining an L2 guarantee without strong coverage assumptions remains unclear. A detailed comparison among these different approaches is presented in Section A.
5 Finite Sample Guarantee of MSQP
In this section, we present our primary sample complexity guarantee for our MSQP algorithm under the assumptions of realizability of  $q_{\alpha}^{\star}$  and  $l_{\alpha}^{\star}$  and partial coverage. We first show the learned policy  $\hat{\pi}_{\alpha}$  can compete with  $\pi_{\alpha}^{\star}$ . Finally we show  $\hat{\pi}_{\alpha}$  can compete with  $\pi^{\star}$  by selecting  $\alpha$  properly.
We first introduce the flattened behavior policy  $\pi_b^\diamond$ , which is uniform on the support of  $\pi_b$ . We use it as a technical device to define a model-free concentrability coefficient following Xie et al. (2021).
Definition 1 (Model-free concentrability coefficient). Define
$$
C _ {\mathcal {Q}, d _ {\pi_ {\alpha} ^ {\star}}, \mu_ {0}} := \sup _ {q \in \mathcal {Q}} \frac {\mathbb {E} _ {s \sim d _ {\pi_ {\alpha} ^ {\star}} , \mu_ {0} , a \sim \pi_ {b} ^ {\diamond} (a | s)} [ \| q (s , a) - q _ {\alpha} ^ {\star} (s , a) \| _ {2} ^ {2} ]}{\mathbb {E} _ {(s , a) \sim P _ {b}} [ \| q (s , a) - q _ {\alpha} ^ {\star} (s , a) \| _ {2} ^ {2} ]}
$$
where  $\pi_b^\diamond (\cdot \mid s) = \left\{ \begin{array}{ll}0 & \pi_b(\cdot \mid s) = 0\\ 1 / |\{a\in \mathcal{A}\mid \pi_b(a\mid s) > 0\} | & \pi_b(\cdot \mid s) > 0 \end{array} \right.$  is the flattened behavior policy.
Clearly,  $C_{\mathcal{Q},d_{\pi_{\hat{\alpha}}^*,\mu_0}}$  is smaller than density-ratio-based concentrability coefficient, in other words,
$$
C _ {\mathcal {Q}, d _ {\pi_ {\alpha} ^ {\star}, \mu_ {0}}} \leq \max  _ {(s, a)} \frac {d _ {\pi_ {\alpha} ^ {\star} , \mu_ {0}} (s) \pi_ {b} ^ {\diamond} (a \mid s)}{P _ {b} (s) \pi_ {b} (a \mid s)}.
$$
Here, we always have  $\| \pi_b^\diamond /\pi_b\| < \infty$  even if  $\pi_b(a\mid s)$  is 0 for some  $(s,a)$ . In the special case where  $\pi_b(a\mid s)\geq 1 / C'$  for any  $(s,a)$ , we have  $C_{\mathcal{Q},d_{\pi_\alpha^\star ,\mu_0}}\leq C'\| d_{\pi_\alpha^\star ,\mu_0} / P_b\|_\infty$ . The coefficient  $C_{\mathcal{Q},d_{\pi_\alpha^\star ,\mu_0}}$  is a refined concentrability coefficient, which adapts to a function class  $\mathcal{Q}$ . For example, in linear MDPs, it reduces to a relative condition number as follows. Similar properties are obtained in related works (Xie et al., 2021; Uehara and Sun, 2021).
Example 1 (Linear MDPs). A linear MDP is one such that, for a known feature vector  $\phi : S \times \mathcal{A} \to \mathbb{R}^d$ , the true density satisfies  $P(s' \mid s, a) = \langle \mu^\star(s'), \phi(s, a) \rangle$  for some  $\mu^\star : S \to \mathbb{R}^d$  and the reward function satisfies  $\mathbb{E}[r \mid s, a] = \langle \theta_r, \phi(s, a) \rangle$  for some  $\theta_r \in \mathbb{R}^d$ .
In linear MDPs,  $q_{\alpha}^{\star}$  is clearly linear in  $\phi(s,a)$ . Hence, the natural function class is  $\mathcal{Q} = \{\langle \theta, \phi(s,a) \rangle \mid \| \theta \| \leq B\}$  for a certain  $B \in \mathbb{R}^{+}$ . Then, we have
$$
C _ {\mathcal {Q}, d _ {\pi_ {\alpha} ^ {\star}, \mu_ {0}}} = \sup _ {x \neq 0} \frac {x ^ {\top} \mathbb {E} _ {s \sim d _ {\pi_ {\alpha} ^ {\star} , \mu_ {0}} , a \sim \pi_ {b} ^ {\diamond} (a | s)} [ \phi (s , a) \phi (s , a) ^ {\top} ] x}{x ^ {\top} \mathbb {E} _ {(s , a) \sim P _ {b}} [ \phi (s , a) \phi (s , a) ^ {\top} ] x}.
$$
We are now prepared to present our main result, which states that given the realizability of the soft Q-function  $q_{\alpha}^{\star}$  and Lagrange multiplier  $l_{\alpha}^{\star}$ , it is possible to compete with  $\pi_{\alpha}^{\star}$  under the coverage condition  $C_{Q,d_{\pi_{\alpha}^{\star},\mu_0}} < \infty$ ,  $\|d_{\pi_{\alpha}^{\star},P_b}/P_b\|_{\infty} < \infty$ .
Theorem 3 ( $\hat{\pi}_{\alpha}$  can compete with  $\pi_{\alpha}^{\star}$ ). Fix  $\alpha > 0$ . Suppose Assumptions 1, 2, 3, and 4 hold. With probability  $1 - \delta$ , the regret  $J(\pi_{\alpha}^{\star}) - J(\hat{\pi}_{\alpha})$  is upper-bounded by
$$
n ^ {- 1 / 4} \mathrm {P o l y} \left(| \mathcal {A} |, \mathcal {B} _ {\mathcal {Q}}, \mathcal {B} _ {\mathcal {L}}, C _ {\mathcal {Q}, d _ {\pi_ {\alpha} ^ {\star}}, \mu_ {0}}, \ln \left(\frac {| \mathcal {Q} | | \mathcal {L} |}{\delta}\right), \frac {1}{1 - \gamma}, R _ {\max }\right)
$$
The proof mainly consists of two steps: (1) obtaining  $L^2$ -errors of  $\hat{q}_{\alpha}$  as previously demonstrated in Theorem 1, (2) translating this error into the error of  $\hat{\pi}_{\alpha}$ . In the second step, the Lipschitz continuity of the softmax function plays a crucial role. If there is no regularization  $(\alpha = 0)$  and the greedy policy of  $q_0^\star$  is utilized, the second step does not proceed (without any further additional assumptions).
Our ultimate goal is to compete with  $\pi^{\star}$ . Theorem 3 serves as the primary foundation for this goal. The remaining task is to analyze the approximation error  $J(\pi^{\star}) - J(\pi_{\alpha}^{\star})$ . Fortunately, this term can be controlled through  $\alpha$  and the density ratio between  $\pi^{\star}$  and  $\pi_{b}$ . Then, by properly controlling  $\alpha$ , we can obtain the following sample complexity result.
Theorem 4 (PAC guarantee of  $\hat{\pi}_{\alpha}$ ). Fix any  $\epsilon > 0$ . Suppose Assumptions 1, 2, 3, and 4 hold for  $\alpha = c / n^{1/8}$  and  $\| \pi_0^\star / \pi_b \|_\infty \leq C_0$ ,  $C_{\mathcal{Q}, d_{\pi_\alpha^\star, \mu_0}} < \infty$ . Then, if  $n$  is at least
$$
\epsilon^ {- 8} \mathrm {P o l y} (| \mathcal {A} |, \mathcal {B} _ {\mathcal {Q}}, \mathcal {B} _ {\mathcal {L}}, C _ {\mathcal {Q}, d _ {\pi_ {\alpha} ^ {\star}}, \mu_ {0}}, \ln (| \mathcal {Q} | | \mathcal {L} | / \delta), (1 - \gamma) ^ {- 1}, \ln (C _ {0}), R _ {\max}),
$$
with probability at least  $1 - \delta$ , we can ensure  $J(\pi^{\star}) - J(\hat{\pi}_{\alpha}) \leq \epsilon$ .
In summary, the realizability of  $q_{\alpha}^{\star}, l_{\alpha}^{\star}$ , per-step coverage  $\| \pi_0^\star /\pi_b\|_\infty < \infty$  and partial coverage  $C_{\mathcal{Q},d_{\pi_{\alpha}^{\star}},\mu_0} < \infty$ ,  $\| d_{\pi_{\alpha}^{\star},P_b} / P_b\|_\infty < \infty$  are sufficient to compete with  $\pi^{\star}$ . This is a novel and attractive result. Firstly, if we solely use the naive FQI or Bellman residual minimization, existing PAC results require the global coverage  $\| d_{\pi ,\mu_0} / P_b\|_\infty < \infty$  for any possible policy  $\pi$  (Munos and Szepesvári, 2008; Antos et al., 2008). Our result only requires coverage under a single policy  $\pi_{\alpha}^{\star}$  (near-optimal policy). Secondly, we only require the realizability of two functions, and we do not necessitate
realizability-type conditions for all policies in the policy class or Bellman completeness, unlike existing works with partial coverage (Xie et al., 2021; Jiang and Huang, 2020).
The most similar result is Zhan et al. (2022). However, our guarantee possesses a certain advantage over their guarantee as follows. They demonstrate the realizability of certain functions  $\tilde{w}_{\alpha}^{\star}, v_{\alpha}^{\star}$  and partial coverage  $\| d_{\tilde{\pi}_{\alpha}, \mu_0} / P_b\| < \infty$  are sufficient conditions in offline RL, where  $\tilde{w}_{\alpha}^{\star} = d_{\tilde{\pi}_{\alpha}, \mu_0} / P_b$  ( $\tilde{\pi}_{\alpha}$  is a certain regularized optimal policy, but fundamentally distinct from  $\pi_{\alpha}^{\star}$ ) and  $v_{\alpha}^{\star}$  is a near-optimal regularized value function parameterized by  $\alpha$ . Here, we have  $\tilde{w}_0^{\star} = w^{\star}, v_0^{\star} = v^{\star}$ . Our guarantee has a similar flavor in the sense that it roughly illustrates realizability and partial coverage are sufficient conditions. However, the meanings of realizability and partial coverage are significantly different. In particular, by employing our algorithm, we can ensure PAC guarantees under the boundedness of the refined concentrability coefficient  $C_{\mathcal{Q}, d_{\pi_{\alpha}^{\star}, \mu_0}} < \infty$  (and  $\|d_{\pi_{\alpha}^{\star}, P_b} / P_b\|_{\infty}$  through  $\mathcal{B}_{\mathcal{L}}$ ). As a result, the  $L^{\infty}$ -norm of the density-ratio-based concentrability coefficient  $\|d_{\pi_{\alpha}^{\star}, \mu_0} / P_b\|_{\infty}$  can even be infinite. More specifically, we can permit situations where  $\max_s \mu_0(s) / P_b(s) = \infty$  as we will see the practical example soon. Conversely, Zhan et al. (2022) excludes this possibility since the algorithm explicitly estimates the density ratio  $\tilde{w}_{\alpha}^{\star}$ .
Example 2 (Contextual bandit under external validity). We consider the contextual bandit setting where we want to optimize  $J(\pi) = \mathbb{E}_{s \sim \mu_0, a \sim \pi(s), r \sim P_r(s, a)}[r]$  using offline data  $s \sim P_b, a \sim \pi_b(s), r \sim P(s, a)$ . This is the simplest RL setting with  $\gamma = 0$ . Here, note  $\mu_0$  could be different from  $P_b$ . This case often happens in practice as discussed in the literature on causal inference related to external validity (Pearl and Baireinboim, 2014; Dahabreh et al., 2019; Uehara et al., 2020), which refers to the shift between the target population and the offline data. Here, our PAC guarantee does not require that  $\mu_0(s)$  is covered by  $P_b(s)$  in terms of the density ratio as long as the relative condition number is upper-bounded when we use linear models. On the other hand, Zhan et al. (2022) excludes this possibility.
Despite the aforementioned advantage of our approach, unfortunately, our sample complexity of  $O(1 / \epsilon^8)$  is slower compared to that of  $O(1 / \epsilon^6)$  in Zhan et al. (2022). In the following, we demonstrate that MQP, which is a special version of MSQP when  $\alpha \to 0$ , can achieve a faster rate of  $O(1 / \epsilon^2)$ .
6 Finite Sample Guarantee of MQP
In this section, building upon the convergence result of  $\hat{q}_0$ , we demonstrate the finite sample guarantee of our MQP algorithm under partial coverage. We first introduce the soft margin.
Assumption 7 (Soft margin). For any  $a' \in \mathcal{A}$ , there exists  $t_0 \in \mathbb{R}^+, \beta \in (0, \infty]$  such that
$$
\mathbb {P} _ {s \sim d _ {\pi^ {\star}, \mu_ {0}}} \left(0 <   | q ^ {\star} (s, \pi^ {\star} (s)) - q ^ {\star} (s, a ^ {\prime}) | <   t\right) \leq \left(t / t _ {0}\right) ^ {\beta}
$$
for any  $t > 0$ . Here, we use the convention  $x^{\infty} = 0$  if  $0 < x < 1$  and  $x^{\infty} = \infty$  if  $x > 1$ .
In the extreme case, if there exists a gap in  $q^{\star}$  (also known as a hard margin) so that the best action is always better than the second-best by some lower bounded amount, then the soft margin is satisfied with  $\beta = \infty$ . Thus, the soft margin is more general than the gap condition used in Simchowitz and Jamieson (2019); Wu et al. (2022). Crucially, a gap generally does not exist in continuous state spaces unless Q-functions are discontinuous or one action is always option, or a gap involves a large  $t_0$  constant in discrete state spaces with bad dependence on the number of states. In contrast, a soft margin with some  $\beta > 0$  generally holds (see, e.g., lemma 4 in Hu et al., 2021). The soft margin is widely used in the literature on classification, decision making, and RL (Audibert and Tsybakov, 2007; Perchet and Rigollet, 2013; Luedtke and Chambaz, 2020; Hu et al., 2021, 2022).
Theorem 5 (PAC guarantee of  $\hat{\pi}_0$ ). Suppose Assumptions 5, 6, and 7 hold and  $\| \pi^{\star} / \pi_b\|_{\infty}\leq C_0$ . Fix any  $\epsilon >0$ . Then, if  $n$  is at least
$$
\left\{\frac {| \mathcal {A} |}{\epsilon} \right\} ^ {\frac {4 + 2 \beta}{\beta}} \operatorname {P o l y} \left(t _ {0} ^ {- 1}, | \mathcal {A} |, \mathcal {B} _ {\mathcal {Q}}, \mathcal {B} _ {\mathcal {L}}, C _ {\mathcal {Q}, d _ {\pi^ {\star}}, \mu_ {0}}, \ln \left(\frac {| \mathcal {Q} | | \mathcal {L} |}{\delta}\right), (1 - \gamma) ^ {- 1}, \ln \left(C _ {0}\right), R _ {\max }\right)
$$
with probability at least  $1 - \delta$ , we can ensure  $J(\pi^{\star}) - J(\hat{\pi}_0) \leq \epsilon$ .
The proof mainly consists of two steps: (1) obtaining  $L^2$ -errors of  $\hat{q}_0$  as demonstrated in Theorem 2, (2) translating this error into the error of  $\hat{\pi}_0$ . In the second step, the soft margin plays a crucial role.
These theorems indicate that the realizability of the  $Q$ -function  $q^{\star}$  and Lagrange multiplier  $l^{\star}$ , and the soft margin are sufficient for the PAC guarantee under partial coverage  $C_{\mathcal{Q},d_{\pi^{*}},\mu_{0}} < \infty$
$\| d_{\pi^{\star},P_b} / P_b\|_{\infty} < \infty$ . Our algorithm is agnostic to  $\beta$  and operates under any value of  $\beta$ . In particular, when there is a gap  $(\beta = \infty)$ , we can achieve sample complexity of  $O(1 / \epsilon^{2})$ . In comparison to Theorem 3, although we additionally use the soft margin, the realizability in Theorem 5 is more appealing since it is imposed on the standard Q-function  $q^{\star}$ . The closest guarantee to our work can be found in Chen and Jiang (2022), which demonstrates that the existence of the gap in  $q^{\star}$ , the realizability of  $q^{\star}$ ,  $w^{\star} \coloneqq d_{\pi^{\star},\mu_0} / P_{\pi_b}$ , and partial coverage  $\| w^{\star}\|_{\infty} < \infty$  are sufficient conditions. A similar comparison is made in Ozdaglar et al. (2023). In comparison to their work, we use the soft margin, which is significantly less stringent.
We propose two value-based algorithms, MSQP and MQP, that operate under realizability of certain functions and partial coverage (i.e., single-policy-coverage). Notably, our guarantee does not require Bellman completeness and uniform-type realizability over the policy class. While guarantees with similar flavors are obtained in Zhan et al. (2022); Chen and Jiang (2022), MSQP can potentially relax the density-ratio-based partial coverage regarding the initial distribution as opposed to Zhan et al. (2022), and MQP can operate under the soft-margin, which is less stringent than the hard margin imposed in Chen and Jiang (2022). Moreover, both algorithms work on Q-functions, which are more commonly used in practice.
This material is based upon work supported by the National Science Foundation under Grant Nos. IIS 1846210, CCF 2002272, IIS 2107304, CIF 2212262, IIS 2144994, IIS 2154711.
JDL acknowledges support of the ARO under MURI Award W911NF-11-1-0304, the Sloan Research Fellowship, ONR Young Investigator Award, and NSF CAREER Award 2144994.