SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 3 (Boundedness on  $\Omega$ ). There exists a non-negative constant  $1 \leq \mathcal{U}_{\infty}^{\tau} < \infty$ , the function  $\tau(s, a) \in [0, \mathcal{U}_{\infty}^{\tau}]$ ,  $\forall \tau \in \Omega$ ,  $s \in S$ ,  $a \in \mathcal{A}$ .
Theorem 4.1. Under Assumptions 1-3 and denote supremum of  $\mu$ -weighted  $L_{2}$  norm of  $\Omega$ , i.e.,  $\sup_{\tau \in \Omega} \| \tau(s, a) \|_{L_{2}(\mu)}$ , as  $\mathcal{U}_{2}^{\tau}$ . Let  $\widehat{\pi}$  be the output of solving (6) when we set  $\varepsilon_{n} = \tilde{\mathcal{O}}(n^{-1/2} \mathcal{U}_{2}^{\tau} (\sqrt{\ln\{\mathrm{Vol}(\Theta)/\delta\}} + \mathcal{U}_{\infty}^{\tau} \sqrt{\varepsilon_{\mathcal{Q}}})$  and  $\widetilde{\sigma}_{n} = \tilde{\mathcal{O}}(n^{-1/2} \mathcal{U}_{2}^{\tau} L \sqrt{\ln\{\mathrm{Vol}(\Theta)/\delta\}} + M(\mathcal{U}_{2}^{\tau} - 1)^{2})$ , then for any policy  $\pi \in \Pi$  and some constant  $\mathcal{U}_{2}^{\star} \in [1, \mathcal{U}_{2}^{\tau})$ , w.p.  $\geq 1 - \delta$ ,
$$
\begin{array}{l} J (\pi) - J (\widehat {\pi}) \leq \frac {1}{1 - \gamma} \widetilde {\mathcal {O}} \left(\underbrace {\mathcal {U} _ {2} ^ {\star} \mathfrak {C} _ {\bar {V} , L} \sqrt {\frac {\ln \{\mathrm {V o l} (\Theta) / \delta \}}{n M}}} _ {\epsilon_ {\sigma}} + \underbrace {\sqrt {\frac {\mathfrak {C} _ {\mathcal {U} _ {\infty} ^ {\tau}}}{M}} \max  \{(\varepsilon_ {\mathcal {Q}}) ^ {1 / 2} , (\varepsilon_ {\mathcal {Q}}) ^ {3 / 4} \}} _ {\epsilon_ {m i s}}\right) \\ + \min  _ {\left\{\rho : \| \underline {{\mu}} \| _ {L _ {2} (\mu)} \leq \mathcal {U} _ {2} ^ {*} \right\}} \left\{ \right.\mathbb {E} _ {(d _ {\pi} - \rho) ^ {+}} \big [ \underbrace {\mathbb {1} _ {\mu = 0} (\mathbb {I} - \gamma \mathrm {P} ^ {\pi}) \Delta_ {\overline {{q ^ {\pi}}} - \underline {{q}} ^ {\pi}} (s , a)} _ {\epsilon_ {o f f}} + \underbrace {\mathbb {1} _ {\mu > 0} \mathfrak {C} _ {\bar {V} , \gamma} \sqrt {\frac {\ln \{\mathrm {V o l} (\Theta) / \delta \}}{n}}} _ {\epsilon_ {b}} \big ] \big \} \left. \right)\left. \right). \\ \end{array}
$$
Here  $\Delta_{\overline{q^{\pi}} -\underline{q}^{\pi}}(s,a) = \overline{q^{\pi}} (s,a) - \underline{q}^{\pi}(s,a)$  for  $\overline{q^{\pi}}\coloneqq \arg \max_{q\in \mathcal{Q}_{\varepsilon_n}}q(s^0,\pi)$  and  $\underline{q}^{\pi}\coloneqq \arg \min_{q\in \mathcal{Q}_{\varepsilon_n}}q(s^0,\pi)$ . For Pollard's pseudo-dimensions  $D_{\Omega},D_{\mathcal{Q}},D_{\Pi}$ ,  $\mathrm{Vol}(\Theta) = (e^{D}\max \{D_{\Omega},D_{\mathcal{Q}},D_{\Pi}\} +1)^{3}(\{1\vee L\} \mathcal{U}_{2}^{\tau})^{2D}$  with the effective pseudo dimension  $D = D_{\Omega}+$ $D_{\mathcal{Q}} + D_{\Pi}$ , where  $L$  is Lipschitz constant of  $M$ -strongly convex function  $\mathbb{D}(\cdot)$ . Moreover,  $\mathfrak{C}_x$  and  $\tilde{\mathcal{O}}$  denote constant terms depending on  $x$ , and big-Oh notation ignoring high-order terms, respectively.
In the upper bound of Theorem 4.1, we split the regret into four different parts: the on-support intrinsic uncertainty  $\epsilon_{\sigma}$ , the on-support bias  $\epsilon_{b}$ , the violation of realizability  $\epsilon_{\mathrm{mis}}$ , and the off-support extrapolation error  $\epsilon_{\mathrm{off}}$ . Recall that we require  $q^{\pi} \in \mathcal{Q}$  as in Assumption 1, in fact, we can further relax the condition to requiring  $q^{\pi}$  to be in the linear hull of  $\mathcal{Q}$  [48], which is more robust to the realizability error  $\epsilon_{\mathrm{mis}}$ . In the following, we focus on investigating the roles of the on-support and off-support error terms in the regret bound.
On-support errors: bias and uncertainty tradeoff. The on-support error consists of two terms:  $\epsilon_{b}$  and  $\epsilon_{\sigma}$ . The on-support uncertainty deviation,  $\epsilon_{\sigma}$ , is scaled by a weighted  $L_{2}$ -based concentrability coefficient  $\mathcal{U}_2^{\star} \coloneqq \|\rho/\mu\|_{L_2(\mu)}$ , which measures the distribution mismatch between the implicit exploratory data distribution and the baseline data distribution  $\mu$ . Meanwhile,  $\epsilon_{b}$  depends on the probability mass of  $(d_{\pi} - \rho)^{+}\mathbb{1}_{\mu > 0}$ , and represents the bias weighted by the probability mass difference between  $d_{\pi}$  and  $\rho$  in the support region of  $\mu$ . In general, a small value of  $\mathcal{U}_2^{\star}$  necessitates choice of the distribution  $\rho$  to be closer to  $\mu$  which reduces  $\epsilon_{\sigma}$ , reducing  $\epsilon_{\sigma}$  but potentially increasing the on-support bias  $\epsilon_{b}$  due to the possible mismatch between  $d_{\pi}$  and  $\rho$ . Consequently, within the on-support region, there is a trade-off between  $\epsilon_{\sigma}$  and  $\epsilon_{b}$ , which is adjusted through  $\mathcal{U}_2^{\star}$ .
Off-support error: enhanced model extrapolation. One of our main algorithmic contributions is that the off-support extrapolation error  $\epsilon_{\mathrm{off}}$  can be minimized by selecting the best possible  $\rho$  without worrying about balancing the error trade-off, unlike the on-support scenario. This desirable property is essential for allowing the model to harness its extrapolation capabilities to minimize  $\epsilon_{\mathrm{off}}$ , while simultaneously achieving a good on-support estimation error. As a result, the model attains a small regret. Recall the bi-level formulation; at the lower level, (7) addresses uncertainty arising from the distributional shift using  $L_{2}(\mu)$  control rather than  $L_{\infty}$  control. This plays an important role in enhancing the power of the model extrapolation. In particular, Specifically, there exists an implicit exploratory data distribution  $\rho$  with on-support behavior  $(\rho \mathbb{1}_{\mu >0})$  close to  $\mu$ , such that  $\| \rho /\mu \|_{L_2(\mu)}$  is small. On the other hand, its off-support behavior  $(\rho \mathbb{1}_{\mu = 0})$  can be arbitrarily flexible, ensuring that  $d_{\pi}\mathbb{1}_{\mu = 0}$  is close to  $\rho \mathbb{1}_{\mu = 0}$ . Consequently,  $(d_{\pi} - \rho)^{+}\mathbb{1}_{\mu = 0}$  is small, as is  $\epsilon_{\mathrm{off}}$ .
When a dataset with partial coverage, as indicated in [49], it is necessary to provide a guarantee: learn the policy with "best efforts" which is competitive to any policy as long as it is covered. Before we state the near-optimal regret guarantee of our algorithm, we formally define a notion of covered policies according to a newly-defined concentrability coefficient.
Definition 4.1 ( $\mathcal{U}_2^\tau$ -covered policy class). Let  $\Pi(\mathcal{U}_2^\tau)$  denote the  $\mathcal{U}_2^\tau$ -covered policy class of  $\mu$  for  $\mathcal{U}_2^\tau \geq 1$ , defined as
$$
\Pi (\mathcal {U} _ {2} ^ {\tau}) := \left\{\pi \in \Pi : \left\| \frac {d _ {\pi} (s , a) \mathbb {1} _ {\mu (s , a) > 0}}{\mu (s , a)} \right\| _ {L _ {2} (\mu)} \leq \mathcal {U} _ {2} ^ {\tau} a n d \sup  _ {s, a} \frac {d _ {\pi} (s , a) \mathbb {1} _ {\mu (s , a) = 0}}{\mu (s , a)} <   + \infty \right\}.
$$
Note that this mixture density ratio concentrability coefficient is always bounded by the  $L_{\infty}$ -based concentrability coefficient. Thus such single-policy concentrability assumption in terms of the mixture density ratio is weaker than the standard  $L_{\infty}$  density ratio-based assumption.
Corollary 4.1 (Near-optimal regret). Under Assumptions 1-3 with  $\varepsilon_{\mathcal{Q}} \in [0,1)$ , and we set  $\varepsilon_{n}, \widetilde{\sigma}_{n}$  as in Theorem 4.1, then for any good comparator policy  $\pi^{\diamond} \in \Pi(\mathcal{U}_2^\tau)$  (not necessary the optimal policy  $\pi^*$ ), w.p.  $\geq 1 - \delta$ , the output policy  $\widehat{\pi}$  of (6) satisfies
$$
J (\pi^ {\diamond}) - J (\widehat {\pi}) \leq \frac {1}{1 - \gamma} \widetilde {\mathcal {O}} \left(\mathcal {U} _ {2} ^ {\star} (\bar {V} + L) \sqrt {\frac {\ln \{\mathrm {V o l} (\Theta) / \delta \}}{n M}} + \sqrt {(1 + \mathcal {U} _ {\infty} ^ {\tau} + \mathcal {U} _ {\infty} ^ {\tau} / M) \varepsilon_ {\mathcal {Q}}}\right).
$$
A close prior result to Corollary 4.1 is that of [9], which develops a pessimistic algorithm based on a nontrivial performance gap condition. Their regret guarantees only hold if the data covers the optimal policy  $\pi^{*}$ , in particular, requiring a bounded  $L_{\infty}$  single-policy concentrability with respect to  $\pi^{*}$ . In comparison, our guarantee can still provide a meaningful guarantee even when  $\pi^{*}$  is not covered by data. In the following, we include the sample complexity of our algorithm when  $\varepsilon_{\mathcal{Q}} = 0$ .
Corollary 4.2 (Polynomial sample complexity). Under the conditions in Corollary 4.1, the output policy  $\widehat{\pi}$  of solving (6) satisfies  $J(\pi^{\diamond}) - J(\widehat{\pi}) \leq \varepsilon$  w.p.  $\geq 1 - \delta$ , if
$$
n = \mathcal {O} \Bigg (\Big (\frac {(\mathcal {U} _ {2} ^ {\star} (\bar {V} + L) / \sqrt {M}) ^ {2}}{\varepsilon^ {2} (1 - \gamma) ^ {2}} + \frac {(\mathcal {U} _ {2} ^ {\tau} \bar {V} ^ {2} (\bar {V} + L) / M) ^ {0 . 6 7}}{\varepsilon^ {1 . 3 3} (1 - \gamma) ^ {1 . 3 3}} + \frac {\mathcal {U} _ {\infty} ^ {\tau} (\bar {V} + L)}{\varepsilon (1 - \gamma)} \Big) \ln \frac {\mathrm {V o l} (\Theta)}{\delta} \Bigg).
$$
The sample complexity consists of three terms corresponding to the slow rate  $\mathcal{O}(n^{-1/2})$  and the two faster rate  $\mathcal{O}(n^{-1})$  and  $\mathcal{O}(n^{-3/4})$  terms in Corollary 4.1. When  $\mathcal{U}_2^\tau$  and  $\mathcal{U}_\infty^\tau$  are not too much larger than  $\mathcal{U}_2^\star$ , the fast rate terms are dominated, and the sample complexity is of order  $\mathcal{O}(1/\varepsilon^2)$ , which is much faster than  $\mathcal{O}(1/\varepsilon^6)$  in the close work of [59]. It is worth noting that even in exploratory settings where the global coverage assumption holds, our sample complexity rate matches the fast rate in popular offline RL frameworks with general function approximation [8, 54, 12].
In addition to the near-optimal regret guarantee, in safety-critical applications, an offline RL algorithm should consistently improve upon the baseline (behavior) policies that collected the data [19, 26]. Our algorithm also achieves this improvement guarantee with respect to the baseline policy.
Theorem 4.2 (Baseline policy improvement). Under Assumptions 1-3 with  $\varepsilon_{\mathcal{Q}} = 0$  and set  $\varepsilon_{n}, \widetilde{\sigma}_{n}$  as in Theorem 4.1. Suppose  $1 \in \Omega$  and the baseline policy  $\pi_{b} \in \Pi$  such that  $d_{\pi_{b}} = \mu$ , then the regret  $(1 - \gamma)(J(\pi_{b}) - J(\widehat{\pi}))$  for the output policy  $\widehat{\pi}$  of solving (6), w.p.  $\geq 1 - \delta$ , is upper bounded by
$$
\mathcal {O} \left(\sqrt {\frac {(\bar {V} + L) ^ {2} \ln \{\mathrm {V o l} (\Theta) / \delta \}}{n M}} + \sqrt {\frac {(\bar {V} ^ {3} + \bar {V} ^ {2} L)}{M}} \left(\frac {\ln \{\mathrm {V o l} (\Theta) / \delta \}}{n}\right) ^ {\frac {3}{4}} + \frac {(\bar {V} + L) \ln \{\mathrm {V o l} (\Theta) / \delta \}}{n}\right).
$$
The aforementioned information-theoretic results enhance the understanding of the developed algorithm, in terms of the function approximation and coverage conditions, sample complexity, horizon dependency, and bound tightness. In practice, although the information-theoretic algorithm offers a feasible solution to the problem, it is not yet tractable and computationally efficient due to the need to solve constrained optimization. In the following section, we develop a practical algorithm as a computationally efficient counterpart for the information-theoretic algorithm.
5 Penalized Adversarial Estimation Algorithm
Although the information-theoretic algorithm offers a feasible solution to the problem, it is not yet tractable and computationally efficient due to the need to solve constrained optimization. In this section, we develop an adversarial estimation proximal-mapping algorithm that still adheres to the pessimism principle, but through penalization. Specifically, the adversarial estimation loss is constructed as follows:  $\max_{\tau} \mathcal{L}(q, \tau, \pi, c^*, \lambda)$  for solving
$$
q \left(s ^ {0}, \pi\right) + \frac {1}{(1 - \gamma) n} \left\{c ^ {*} \left| \sum_ {i = 1} ^ {n} \tau \left(s _ {i}, a _ {i}\right) \left(q \left(s _ {i}, a _ {i}\right) - r _ {i} - \gamma q \left(s _ {i} ^ {\prime}, \pi\right)\right) \right| - \lambda \sum_ {i = 1} ^ {n} \mathbb {D} \left(\tau \left(s _ {i}, a _ {i}\right)\right) \right\}.
$$
We observe that the inner minimization for solving  $q$  is relatively straightforward, as we can obtain a closed-form global solver using the maximum mean discrepancy principle [20, 44]. In contrast, optimizing  $\tau_{\psi}$  is more involved, often requiring a sufficiently expressive non-linear function approximation class, e.g., neural networks. However, concavity typically does not hold for such a class of functions [21]. From this perspective, our problem can be viewed as solving a non-concave maximization problem, conditional on the solved global optimizer  $\bar{q} \coloneqq \arg \min_q \mathcal{L}(q,\tau ,\pi ,c^*,\lambda)$ . At each iteration, we propose to update  $\tau$  by solving the proximal mapping [37] using the Euclidean distance to reduce the computational burden. As a result, the pre-iteration computation is quite low.
Algorithm 1 Adversarial proximal-mapping algorithm
1: Input observed data  $\mathcal{D}_{1:n} = \{(s_i, a_i, r_i, s_i')\}_{i=1}^n$  and parameters  $q^0, \tau^0, \pi^0, c^*, \lambda$  and  $\zeta$ .
2: For  $k = 1$  to  $\bar{K}$ :
3: Update  $\tau^k$  and  $q^k$  by solving  $\max_{\tau} \min_{q} \mathcal{L}(q, \tau, \pi^{k-1}, c^*, \lambda)$
4: Update  $\pi^k$  by solving  $\pi^k (\cdot |s) = \operatorname *{argmax}_{\pi \in \Pi}\zeta \left\langle q^k (\cdot ,s),\pi (\cdot |s)\right\rangle -D_{\mathrm{NegEntropy}}\left(\pi (\cdot |s),\pi^k (\cdot |s)\right)$ .
5: Return the policy  $\widehat{\pi}$ , which randomly selects a policy from the set  $\{\pi^k\}_{k=1}^K$ .
Once  $q$  and  $\tau$  are solved, we apply mirror descent in terms of the negative entropy  $D_{\mathrm{NegEntropy}}$  [5]. That is, given a stochastic gradient direction of  $\pi$  we solve the prox-mapping in each iteration as outlined in step 4 of Algorithm 1. A detailed version of Algorithm 1 with extended discussions on convergence and complexity is provided in Appendix. In the following, we establish the regret guarantee for the policy output by Algorithm 1.
Theorem 5.1. Under Assumptions 1-3 with  $\varepsilon_{\mathcal{Q}} = 0$ , we properly choose  $\lambda = \lambda (\mathcal{U}_2^\tau)$ , i.e.,  $\lambda$  well depends on  $\mathcal{U}_2^\tau$ , and  $c^* = \widetilde{\mathcal{O}}\big(\sqrt{n\bar{V} / (\lambda L\mathcal{U}_2^\tau\ln\{\mathrm{Vol}(\Theta^\dagger) / \delta\})}\big)$ . After running  $\bar{K} \geq \log |\mathcal{A}|$  rounds of Algorithm 1 with the stepsize  $\zeta = \sqrt{\log|\mathcal{A}| / (2\bar{V}\bar{K})}$ , for any policy  $\pi \in \Pi$ , the output policy  $\widehat{\pi}$  of the algorithm, w.p  $\geq 1 - \delta$ , satisfies,
$$
\begin{array}{l} J (\pi) - J (\widehat {\pi}) \leq \frac {1}{1 - \gamma} \widetilde {\mathcal {O}} \left(\sqrt [ 4 ]{\frac {(\mathcal {U} _ {2} ^ {\star}) ^ {2} \mathfrak {C} _ {\bar {V} , \lambda , L} ^ {1} \ln \{\mathrm {V o l} (\Theta^ {\dagger}) / \delta \}}{n}} + \sqrt {\frac {\bar {V} \log | \mathcal {A} |}{\bar {K}}} \right. \\ \left. + \frac {1}{\bar {K}} \sum_ {k = 1} ^ {\bar {K}} \min  _ {\rho_ {k} \in \Delta_ {\mathcal {U} _ {2} ^ {\star}}} \mathbb {E} _ {(d _ {\pi} - \rho_ {k}) ^ {+}} \left[ \mathbb {1} _ {\mu = 0} \left(\mathcal {B} ^ {\pi^ {k}} q ^ {k} (s, a) - q ^ {k} (s, a)\right) + \mathbb {1} _ {\mu > 0} \sqrt {\frac {\mathfrak {C} _ {\bar {V} , \lambda , L} ^ {2} \ln \{\operatorname {V o l} (\Theta^ {\dagger}) / \delta \}}{n}} \right]\right), \\ \end{array}
$$
where  $\Delta_{\mathcal{U}_2^\star} \coloneqq \{\rho_k : \| \frac{\rho_k}{\mu} \|_{L_2(\mu)} < \mathcal{U}_2^\star\}$ ,  $\mathfrak{C}_{V,\lambda,L}^1$ ,  $\mathfrak{C}_{V,\lambda,L}^2$  are some constant terms, and the function class complexity  $\mathrm{Vol}(\Theta^\dagger) = (e^D \max \{D_\Omega, D_\mathcal{Q}, D_\Pi\} + 1)^3 (\{1 \vee L\} \mathcal{U}_2^\tau)^{2D}$  for  $D = D_\Omega + D_\mathcal{Q} + D_\Pi$ .
Trajectory-adaptive exploratory data distribution. Similar to Theorem 4.1, the penalized algorithm also exhibits a desirable extrapolation property for minimizing extrapolation error while simultaneously preserving small on-support estimation errors. This is achieved through adaptations of the implicit exploratory data distributions,  $\rho_{k}$  for  $k\in [\bar{K} ]$  . In contrast to the information-theoretic algorithm, the automatic splitting by  $\rho_{k}$  now depends on the optimization trajectory. At each iteration  $k$  , the penalized algorithm allows each implicit exploratory data distribution  $\rho_{k}$  to adapt to the comparator policy  $\pi$  . This results in a more flexible adaptation than the one in the information-theoretic algorithm, either for balancing the trade-off between on-support bias and uncertainty incurred by the distributional mismatch between  $d_{\pi}$  and  $\rho_{k}$  , or for selecting the best implicit exploratory to minimize model extrapolation error.
**Optimization error.** Blessed by the reparametrization in the proximal-mapping policy update, which projects the mixture policies into the parametric space  $\Pi_{\omega}$ , the complexity of the restricted policy class is independent of the class of  $\mathcal{Q}$  and the horizon optimization trajectory  $\bar{K}$ . As a result, the optimization error  $\mathcal{O}(\sqrt{\bar{V}\log|\mathcal{A}| / \bar{K}})$  can be reduced arbitrarily by increasing the maximum number of iterations,  $\bar{K}$ , without sacrificing overall regret to balance statistical error and optimization error. This allows for the construction of tight regret bounds. This distinguishes our algorithm from API-style algorithms, which do not possess a policy class that is independent of  $\mathcal{Q}$  [2, 43, 53].
5.1 An Application to Linear MDPs with Refined Concentrability Coefficient
In this section, we conduct a case study in linear MDPs with insufficient data coverage. The concept of the linear MDP is initially developed in the fully exploratory setting [55]. Let  $\phi : S \times \mathcal{A} \to \mathbb{R}^d$  be a  $d$ -dimensional feature mapping. We assume throughout that these feature mappings are normalized, such that  $\| \phi(s, a) \|_{L_2} \leq 1$  uniformly for all  $(s, a) \in S \times \mathcal{A}$ . We focus on action-value functions that are linear in  $\phi$  and consider families of the following form:  $\mathcal{Q}_{\theta} := \{(s, a) \mapsto \langle \phi(s, a), \theta \rangle \mid \| \theta \|_{L_2} \leq c_{\theta}\}$ , where  $c_{\theta} \in [0, \overline{V}]$ . For stochastic policies, we consider the soft-max policy class  $\Pi_{\omega} := \{\pi_{\omega}(a | s) \propto e^{\langle \phi(s, a), \omega \rangle} \mid \| \omega \|_{L_2} \leq c_{\omega}\}$ , where  $c_{\omega} \in (0, \infty)$ . Note that the softmax policy class is consistent with the implicit policy class produced by the mirror descent updates with negative entropy in Algorithm 1, where the exponentiated gradient update rule is applied in each iteration. For the importance-weight class, we also consider the following form:  $\Omega_{\psi} := \{(s, a) \mapsto \langle \phi(s, a), \psi \rangle \mid \| \psi \|_{L_2} \leq c_{\psi}\}$  where  $c_{\psi} \in (0, \infty)$ . To simplify the analysis, we assume the realizability condition for  $\mathcal{Q}_{\theta}$  is exactly met. In this linear MDP setting, we further refine the density ratio to a relative condition number to characterize partial coverage. This concept is recently introduced in the policy gradient literature [1] and is consistently upper-bounded by the  $L_{\infty}$ -based density ratio concentrability coefficient.
Definition 5.1 (Relative condition number). For any policy  $\pi \in \Pi_{\omega}$  and behavior policy  $\pi_b$  such that  $d_{\pi_b} = \mu$ , the relative condition number is defined as  $\iota(d_{\pi}, \mu) = \sup_{x \in \mathbb{R}^d} \frac{x^T \mathbb{E}_{d_{\pi}}[\phi(s, a) \phi(s, a)^\top] x}{x^\top \mathbb{E}_{\mu}[\phi(s, a) \phi(s, a)^\top] x}$ .
Assumption 4 (Bounded relative condition number). For any  $\pi \in \Pi_{\omega}$ ,  $\iota(d_{\pi}, \mu) < \infty$ .
Intuitively, this implies that as long as a high-quality comparator policy exists, which only visits the subspace defined by the feature mapping  $\phi$  and is covered by the offline data, our algorithm can effectively compete against it [49]. This partial coverage assumption, in terms of the relative condition number, is considerably weaker than density ratio-based assumptions. In the following, we present our main near-optimal guarantee in linear MDPs. In addition, we design and conduct numerical experiments to empirically validate Theorem 5.2 in terms of the regret rate of convergence.
Theorem 5.2. Under Assumption 4, if we set properly choose  $\lambda = \lambda(c_{\psi})$  and  $c^* = \widetilde{\mathcal{O}}\left(\sqrt[4]{n / d} \ln \{(1 + e\sqrt{n}(1 \vee L) V c_{\psi} c_{\omega}) / \delta\}\right)$ , and suppose  $\widehat{\pi}^{lr}$  is returned by Algorithm 1 with linear function approxiamiton after running  $\bar{K} \gg \log |\mathcal{A}|$  rounds, then for any policy in  $\pi \in \Pi_{\omega}(\mathcal{U}_2^{lr})$  for  $\mathcal{U}_2^{lr} \geq 1$ , w.p.  $\geq 1 - \delta$ ,  $J(\pi) - J(\widehat{\pi}^{lr})$  is bounded by
$$
\widetilde {\mathcal {O}} \Bigg (\frac {\sqrt {\min  \{\kappa^ {2} c _ {\psi} ^ {2} \left\{\mathcal {U} _ {2} ^ {l r} \right\} d ^ {2} , \iota (d _ {\pi} , \mu) d \}}}{1 - \gamma} \sqrt [ 4 ]{\frac {\mathfrak {C} _ {\bar {V} , \lambda , L} d \ln \{(1 + e \sqrt {n} (1 \vee L) \bar {V} c _ {\psi} c _ {\omega}) / \delta \}}{n}} \Bigg),
$$
where  $\kappa = \text{trace}(\mathbb{E}_{\mu}[\phi(s, a)\phi(s, a)^\top])$  and  $c_{\psi}\{\mathcal{U}_2^{lr}\} = \sup_{\{\psi : \| \phi(s, a)^\top \psi \|_{L_2(\mu)} = \mathcal{U}_2^{lr}\}} \| \psi \|_{L_\infty}$ .
To the best of our knowledge, this is the first result PAC guarantees for an offline model-free RL algorithm in linear MDPs, requiring only realizability and single-policy concentrability. The regret bound we obtain is at least linear and, at best, sub-linear with respect to the feature dimension  $d$ . Our approach demonstrates a sample complexity improvement in terms of feature dimension compared to prior work by [22], with a complexity of  $\mathcal{O}(d^{1/2})$  versus  $\mathcal{O}(d)$ . It is worth noting that [22] only establishes results that compete with the optimal policy, and when specialized to linear MDPs, assumes the offline data has global coverage. Another previous study by [53] achieves a similar sub-linear rate in  $d$  as our approach; however, their algorithm is computationally intractable, relying on a much stronger Bellman-completeness assumption and requiring a small action space.
In this section, we evaluate the performance of our practical algorithm by comparing to the model-free offline RL baselines including CQL [25], BEAR [24], BCQ [18], OptiDICE [27], ATAC [10], IQL [23], and TD3+BC [17]. We also compete with a popular model-based approach COMBO [57].
Synthetic data. We consider two synthetic environments: a synthetic CartPole environment from the OpenAI Gym [7] and a simulated environment. Detailed discussions on the experimental designs are deferred to the Appendix. In both settings, following [48], we first learn a sub-optimal policy using DQN [34] and then apply softmax to its  $q$ -function, divided by a temperature parameter  $\alpha$  to set the action probabilities to define a behavior policy  $\pi_b$ .
A smaller  $\alpha$  implies  $\pi_b$  is less explored, and thus the support of  $\mu = d_{\pi_b}$  is relatively small. We vary different values of  $\alpha$  for evaluating the algorithm performance in "low", "medium" and "relatively high" offline data exploration scenarios. We use  $\gamma = 0.95$  with the sample-size  $n = 1500$  in all experiments. Tuning parameter selection is an open problem in offline policy optimization. Fortunately, Theorem 5.2 suggests an offline selection rule for hyperparameters  $\lambda$  and  $c^*$ . In the following