SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since we do not have direct access to the dataset distribution  $\mu$ , our algorithm instead solves an empirical version of (2), which can be formalized as
$$
\hat {\pi} \in \arg \max  _ {\pi \in \Pi} \mathcal {L} _ {\mathcal {D}} (\pi , f ^ {\pi}), \quad \text {s . t .} f ^ {\pi} \in \arg \min  _ {f \in \mathcal {F}} \mathcal {L} _ {\mathcal {D}} (\pi , f) + \beta \mathcal {E} _ {\mathcal {D}} (\pi , f), \tag {5}
$$
$$
\mathcal {L} _ {\mathcal {D}} (\pi , f) = \mathbb {E} _ {\mathcal {D}} [ f (s, \pi) - f (s, a) ], \tag {6}
$$
$$
\left. \mathcal {E} _ {\mathcal {D}} (\pi , f) = \max  _ {w \in \mathcal {W}} \left| \mathbb {E} _ {\mathcal {D}} \left[ w (s, a) \left(f (s, a) - r - \gamma f \left(s ^ {\prime}, \pi\right)\right) \right] \right|. \right. \tag {7}
$$
Algorithm 1 Actor-Critic Regularized by Average Bellman error (A-CRAB)
1: Input: Dataset  $\mathcal{D} = \{(s_i, a_i, r_i, s_i')\}_{i=1}^N$ , value function class  $\mathcal{F}$ , importance weight function class  $\mathcal{W}$ , no-regret policy optimization oracle PO (Definition 2).
2: Initialization:  $\pi_1$  : uniform policy,  $\beta$  : hyperparameter.
3: for  $k = 1,2,\ldots ,K$  do
4:  $f_{k}\gets \arg \min_{f\in \mathcal{F}}\mathcal{L}_{\mathcal{D}}(\pi_{k},f) + \beta \mathcal{E}_{\mathcal{D}}(\pi_{k},f)$ , where  $\mathcal{L}_{\mathcal{D}}$  and  $\mathcal{E}_{\mathcal{D}}$  are defined in (6), (7)
5:  $\pi_{k + 1}\gets \mathsf{PO}(\pi_k,f_k,\mathcal{D}).$
6: end for
7: Output:  $\bar{\pi} = \mathsf{Unif}\left(\{\pi_k\}_{k=1}^K\right)$ .
Similar to Cheng et al. (2022), we view program (5) as a Stackelberg game and solve it using a no-regret oracle as shown in Algorithm 1. At each step  $k$ , the critic minimizes the objective defined by (7) w.r.t.  $\pi_k$ , and  $\pi_{k+1}$  is generated by a no-regret policy optimization oracle, given below.
Definition 2 (No-regret policy optimization oracle). An algorithm  $PO$  is defined as a no-regret policy optimization oracle if for any (adversarial) sequence of functions  $f_{1}, f_{2}, \ldots, f_{K} \in \mathcal{F}$  where  $f_{k}: S \times \mathcal{A} \to [0, V_{\max}], \forall k \in [K]$ , the policy sequence  $\pi_{1}, \pi_{2}, \ldots, \pi_{K}$  produced by  $PO$  satisfies that for any policy  $\pi \in \Pi$ , it holds that  $\epsilon_{opt}^{\pi} \triangleq \frac{1}{K} \sum_{k=1}^{K} \mathbb{E}_{\pi}[f_{k}(s, \pi) - f_{k}(s, \pi_{k})] = o(1)$ .
Among the well-known instances of the above no-regret policy optimization oracle is natural policy gradient (Kakade, 2001) of the form  $\pi_{k + 1}(a|s)\propto \pi_k(a|s)\exp (\eta f_k(s,a))$  with  $\eta = \sqrt{\frac{\log|\mathcal{A}|}{2V_{\max}^2K}}$  (Even-Dar et al., 2009; Agarwal et al., 2021; Xie et al., 2021; Cheng et al., 2022). A detailed discussion of the above policy optimization oracle can be found in Cheng et al. (2022). Utilizing the no-regret oracle in solving the Stackelberg optimization problem in (5) yields Algorithm 1.
A remark on critic's optimization problem. In our algorithm, for any given  $\pi$ , the critic needs to solve a  $\min_{f\in \mathcal{F}}\max_{w\in \mathcal{W}}$  optimization problem, whereas in ATAC (Cheng et al., 2022), the critic needs to solve a  $\min_{f\in \mathcal{F}}\max_{g\in \mathcal{F}}$  problem. Since we only assume single-policy realizability for the class  $\mathcal{W}$  (Assumption 3) but assume all-policy realizability for  $\mathcal{F}$  (Assumption 1) (and Cheng et al. (2022) even requires the Bellman-completeness assumption over  $\mathcal{F}$  which is much stronger), in general, the cardinality of  $\mathcal{W}$  could be much smaller than  $\mathcal{F}$ , which makes the optimization region of the critic's optimization problem in our algorithm  $\mathcal{F} \times \mathcal{W}$  smaller than  $\mathcal{F} \times \mathcal{F}$  in ATAC.
In this section, we show the theoretical guarantee of our main algorithm (Algorithm 1), which is statistically optimal in terms of  $N$ .
4.1 Performance guarantee of the A-Crab algorithm
We first formally present our main theorem, which provides a theoretical guarantee of our A-Crab algorithm (Algorithm 1). A proof sketch is provided in Section 5 and the complete proof is deferred to Appendix C.3.
Theorem 1 (Main theorem). Under Assumptions 1 and 2 and let  $\pi \in \Pi$  be any policy satisfying Assumption 3, then with probability at least  $1 - \delta$ ,
$$
J (\pi) - J (\bar {\pi}) \leq O \left(\epsilon_ {s t a t} + C _ {\ell_ {2}} ^ {\star} \sqrt {\epsilon_ {\mathcal {F}}}\right) + \epsilon_ {o p t} ^ {\pi},
$$
where  $\epsilon_{stat} \asymp V_{\max} C_{\ell_2}^{\star} \sqrt{\frac{\log(|\mathcal{F}| |\Pi| |\mathcal{W}| / \delta)}{N}} + \frac{V_{\max} B_w \log(|\mathcal{F}| |\Pi| |\mathcal{W}| / \delta)}{N}$ , and  $\bar{\pi}$  is returned by Algorithm 1 with the choice of  $\beta = 2$ .
Below we discuss the advantages of our approach as shown in the above theorem.
Optimal statistical rate and computational efficiency. When  $\epsilon_{\mathcal{F}} = 0$  (i.e., there is no model misspecification), and when  $\pi = \pi^{\star}$  is one of the optimal policies, the output policy  $\bar{\pi}$  achieves  $O(1 / \sqrt{N})$  suboptimality rate which is optimal in  $N$  dependence (as long as  $K$  is large enough). This improves the  $O(1 / N^{1 / 3})$  rate of the previous algorithm (Cheng et al., 2022). Note that the algorithm of Xie et al. (2021) can also achieve the optimal  $O(1 / \sqrt{N})$  rate but their algorithm involves hard constraints of squared  $\ell_2$  Bellman error and thus is computationally intractable. Cheng et al. (2022) convert the hard constraints to a regularizer, making the algorithm computationally tractable while degenerating the statistical rate. Our algorithm is both statistically optimal and computationally efficient, which improves upon both Xie et al. (2021); Cheng et al. (2022) simultaneously.
Competing with any policy. Another advantage of our algorithm is that it can compete with any policy  $\pi \in \Pi$  as long as  $w^{\pi} = d^{\pi} / \mu$  is contained in  $\mathcal{W}$ . In particular, the importance ratio of the behavior policy  $w^{\mu} = d^{\mu} / \mu = \mu / \mu \equiv 1$  is always contained in  $\mathcal{W}$ , which implies that our algorithm satisfies robust policy improvement (see Theorem 2 for details).
Robustness to model misspecification. Theorem 1 also shows that our algorithm is robust to model misspecification on realizability assumption. Note that our algorithm does not need a completeness assumption, while Xie et al. (2021); Cheng et al. (2022) both require the (approximate) completeness assumption.
Removal of the completeness assumption on  $\mathcal{F}$ . Compared to our algorithm, Cheng et al. (2022) additionally need a completeness assumption on  $\mathcal{F}$ , which requires that for any  $f\in \mathcal{F}$  and  $\pi \in \Pi$ , it approximately holds that  $\mathcal{T}^{\pi}f\in \mathcal{F}$ . They need this completeness assumption because they use the estimator  $\mathbb{E}_{\mathcal{D}}\left[(f(s,a) - r - \gamma f(s',\pi))^2\right] - \min_{g\in \mathcal{F}}\mathbb{E}_{\mathcal{D}}\left[(g(s,a) - r - \gamma f(s',\pi))^2\right]$  to address the over-estimation issue caused by their squared  $\ell_2$  Bellman error regularizer, and to make this estimator accurate, they need  $\min_{g\in \mathcal{F}}\mathbb{E}_{\mathcal{D}}\left[(g(s,a) - r - \gamma f(s',\pi))^2\right]$  to be small, which can be implied by the (approximate) completeness assumption. In our algorithm, thanks to the nice property of the weighted average Bellman error regularizer which can be estimated by a simple and unbiased estimator, we can get rid of this strong assumption.
4.2 A-Crab for robust policy improvement
Robust policy improvement (RPI) refers to the property of an offline RL algorithm that the learned policy (almost) always improves upon the behavior policy used to collect data over a wide range of the choice of some specific hyperparameters (in this paper, the hyperparameter is  $\beta$ ) (Cheng et al., 2022). Similar to ATAC in Cheng et al. (2022), our A-Crab also enjoys the RPI property. Theorem 2 implies that as long as  $\beta = o(\sqrt{N})$ , our algorithm can learn a policy with vanishing suboptimality compared to the behavior policy with high probability. The proof is deferred to Appendix D.
Theorem 2 (Robust policy improvement). Under Assumptions 1 and 2, with probability at least  $1 - \delta$ ,
$$
J (\mu) - J (\bar {\pi}) \lesssim (\beta + 1) \left(\epsilon_ {s t a t} + C _ {\ell_ {2}} ^ {\star} \sqrt {\epsilon_ {\mathcal {F}}}\right) + \epsilon_ {o p t} ^ {\pi},
$$
where  $\epsilon_{stat} \asymp V_{\max} C_{\ell_2}^{\star} \sqrt{\frac{\log(|\mathcal{F}| |\Pi| |\mathcal{W}| / \delta)}{N}} + \frac{V_{\max} B_w \log(|\mathcal{F}| |\Pi| |\mathcal{W}| / \delta)}{N}$ , and  $\bar{\pi}$  is returned by Algorithm 1 with the choice of any  $\beta \geq 0$ .
4.3 Suboptimality of squared  $l_{2}$  norm of Bellman error as regularizers
The ATAC algorithm of Cheng et al. (2022) suffers suboptimal statistical rate  $O(1 / N^{1 / 3})$  due to the squared  $\ell_2$  Bellman error regularizer. Intuitively, in Cheng et al. (2022), they use Lemma 1 to
decompose the performance difference and use  $\| f - \mathcal{T}^{\pi}f\|_{2,\mu}$  to upper bound  $\mathbb{E}_{\mu}[(f - \mathcal{T}^{\pi}f)(s,a)]$ , which causes suboptimality since in general the former could be much larger than the latter. To overcome this suboptimal step, in our algorithm, we use a weighted version of  $\mathbb{E}_{\mu}[(f - \mathcal{T}^{\pi}f)(s,a)]$  as our regularizer instead of  $\| f - \mathcal{T}^{\pi}f\|_{2,\mu}$ . Proposition 4.1 shows that ATAC is indeed statistically suboptimal even under their optimal choice of the hyperparameter  $\beta = \Theta (N^{2 / 3})$ . The proof is deferred to Appendix E.1.
Proposition 4.1 (Suboptimality of ATAC). If we change the regularizer s.t.
$$
\mathcal {E} _ {\mu} (\pi , f) = \| f - \mathcal {T} ^ {\pi} f \| _ {2, \mu} ^ {2} \quad a n d \quad \mathcal {E} _ {\mathcal {D}} (\pi , f) = \mathcal {L} (f, f, \pi , \mathcal {D}) - \min _ {g \in \mathcal {F}} \mathcal {L} (g, f, \pi , \mathcal {D}),
$$
where  $\mathcal{L}(g,f,\pi ,\mathcal{D}) = \mathbb{E}_{\mathcal{D}}[(g(s,a) - r - \gamma f(s',\pi))^2 ]$  then even under all policy realizability  $(Q^{\pi}\in \mathcal{F}$  for all  $\pi \in \Pi)$  and completeness assumption  $(\mathcal{T}^{\pi}f\in \mathcal{F}$  for all  $\pi \in \Pi$  and  $f\in \mathcal{F}$  with their optimal choice of  $\beta = \Theta (N^{2 / 3})$  (Cheng et al., 2022), there exists an instance s.t. the suboptimality of the returned policy of (5) (i.e., the output policy by ATAC) is  $\Omega (1 / N^{1 / 3})$  with at least constant probability.
5 Proof Sketch
We provide a proof sketch of our main theorem (Theorem 1) in this section. The key lemma of the proof is presented in Lemma 1.
Lemma 1 (Performance difference decomposition, Lemma 12 in Cheng et al. (2022)). For any  $\pi, \hat{\pi} \in \Pi$ , and any  $f: S \times \mathcal{A} \to \mathbb{R}$ , we can decompose  $J(\pi) - J(\hat{\pi})$  as
$$
\mathbb {E} _ {\mu} [ (f - \mathcal {T} ^ {\hat {\pi}} f) (s, a) ] + \mathbb {E} _ {\pi} [ (\mathcal {T} ^ {\hat {\pi}} f - f) (s, a) ] + \mathbb {E} _ {\pi} [ f (s, \pi) - f (s, \hat {\pi}) ] + \mathcal {L} _ {\mu} (\hat {\pi}, f) - \mathcal {L} _ {\mu} (\hat {\pi}, Q ^ {\hat {\pi}}).
$$
Note that the first two terms in the RHS of the decomposition are average Bellman errors of  $f$  and  $\hat{\pi}$  w.r.t.  $\mu$  and  $d^{\pi}$ . Based on this lemma, we can directly use the average Bellman error as the regularizer instead of the squared Bellman error, which could be much larger and causes suboptimality.
Proof sketch of Theorem 1. For simplicity, we assume realizability of  $Q^{\pi}$ , i.e.,  $\epsilon_{\mathcal{F}} = 0$ . By Lemma 1 and the definition of  $\bar{\pi}$ , we have  $J(\pi) - J(\bar{\pi}) = \frac{1}{K} \sum_{k=1}^{K} (J(\pi) - J(\pi_k))$ , which equals to
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 1} ^ {K} (\underbrace {\mathbb {E} _ {\mu} [ (f _ {k} - \mathcal {T} ^ {\pi_ {k}} f _ {k}) (s , a) ]} _ {(a)} + \underbrace {\mathbb {E} _ {\pi} [ (\mathcal {T} ^ {\pi_ {k}} f _ {k} - f _ {k}) (s , a) ]} _ {(b)}) \\ + \underbrace {\mathbb {E} _ {\pi} [ f _ {k} (s , \pi) - f _ {k} (s , \pi_ {k}) ]} _ {(c)} + \underbrace {\mathcal {L} _ {\mu} (\pi_ {k} , f _ {k}) - \mathcal {L} _ {\mu} (\pi_ {k} , Q ^ {\pi_ {k}})} _ {(d)}. \\ \end{array}
$$
By the concentration argument, with high probability, we have  $\mathcal{E}_{\mu}(\pi ,f) = \mathcal{E}_{\mathcal{D}}(\pi ,f)\pm \epsilon_{\mathrm{stat}}$  and  $\mathcal{L}_{\mu}(\pi ,f) = \mathcal{L}_{\mathcal{D}}(\pi ,f)\pm \epsilon_{\mathrm{stat}}$  for all  $\pi \in \Pi$  and  $f\in \mathcal{F}$ . Combining the fact that  $d^{\pi} / \mu \in \mathcal{W}$ , one can show that  $(a) + (b)\leq 2\mathcal{E}_{\mu}(\pi_k,f_k)\leq 2\mathcal{E}_{\mathcal{D}}(\pi_k,f_k) + O(\epsilon_{\mathrm{stat}})$ . Therefore,
$$
\begin{array}{l} (a) + (b) + (d) \leq \mathcal {L} _ {\mu} \left(\pi_ {k}, f _ {k}\right) + 2 \mathcal {E} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) + O \left(\epsilon_ {\text {s t a t}}\right) - \mathcal {L} _ {\mu} \left(\pi_ {k}, Q ^ {\pi_ {k}}\right) \\ \leq \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) + 2 \mathcal {E} _ {\mathcal {D}} \left(\pi_ {k}, f _ {k}\right) + O \left(\epsilon_ {\text {s t a t}}\right) - \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, Q ^ {\pi_ {k}}\right) \\ \leq \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, Q ^ {\pi_ {k}}\right) + 2 \mathcal {E} _ {\mathcal {D}} \left(\pi_ {k}, Q ^ {\pi_ {k}}\right) + O \left(\epsilon_ {\text {s t a t}}\right) - \mathcal {L} _ {\mathcal {D}} \left(\pi_ {k}, Q ^ {\pi_ {k}}\right) \\ \leq O \left(\epsilon_ {\text {s t a t}}\right) + 2 \mathcal {E} _ {\mu} \left(\pi_ {k}, Q ^ {\pi_ {k}}\right) = O \left(\epsilon_ {\text {s t a t}}\right), \\ \end{array}
$$
where the third inequality holds by the optimality of  $f_{k}$ , and the last equality holds since the Bellman error of  $Q^{\pi}$  w.r.t.  $\pi$  is 0. Therefore, with high probability,
$$
J (\pi) - J (\bar {\pi}) \leq O \left(\epsilon_ {\mathrm {s t a t}}\right) + \epsilon_ {\mathrm {o p t}} ^ {\pi}.
$$
In this section, we conduct experiments of our proposed A-Crab algorithm (Algorithm 1) using a selection of the Mujoco datasets (v2) from D4RL offline RL benchmark (Fu et al., 2020). In particular, we compare the performances of A-Crab and ATAC, since ATAC is the state-of-the-art algorithm on a range of continuous control tasks (Cheng et al., 2022).
A more practical version of weighted average Bellman error. Recall the definition of our proposed weighted average Bellman error regularizer
$$
\mathcal {E} _ {\mathcal {D}} (\pi , f) = \max  _ {w \in \mathcal {W}} | \mathbb {E} _ {\mathcal {D}} [ w (s, a) (f (s, a) - r - \gamma f (s ^ {\prime}, \pi)) ] |.
$$
Since the calculation of  $\mathcal{E}_{\mathcal{D}}(\pi, f)$  requires solving an optimization problem w.r.t. importance weights  $w$ , for computational efficiency, we choose  $\mathcal{W} = [0, C_{\infty}]^{S \times A}$  as in Hong et al. (2023), and thus
$$
\mathcal {E} _ {\mathcal {D}} ^ {\mathrm {a p p}} (\pi , f) = C _ {\infty} \max  \left\{\mathbb {E} _ {\mathcal {D}} \left[ \left(f (s, a) - r - \gamma f \left(s ^ {\prime}, \pi\right)\right) _ {+} \right], \mathbb {E} _ {\mathcal {D}} \left[ \left(r + \gamma f \left(s ^ {\prime}, \pi\right) - f (s, a)\right) _ {+} \right] \right\}, \tag {8}
$$
where  $(\cdot)_{+} = \max \{\cdot, 0\}$  and  $C_{\infty}$  can be viewed as a hyperparameter. We also observed that using a combination of squared Bellman error and our average Bellman error achieves better performance in practice, and we conjecture the reason is that the squared Bellman error regularizer is computationally more efficient and statistically suboptimal, while our average Bellman error regularizer is statistically optimal while computationally less efficient, and thus the combination of these two regularizers can benefit the training procedure.
The practical implementation of our algorithm is nearly identical to (a lightweight version of) ATAC (Cheng et al., 2022), except that we choose
$$
\frac {1}{2} \left(2 \mathcal {E} _ {\mathcal {D}} ^ {\mathrm {a p p}} (\pi , f) + \beta \mathbb {E} _ {\mathcal {D}} [ ((f - \mathcal {T} ^ {\pi} f) (s, a)) ^ {2} ]\right)
$$
as the regularizer, while ATAC uses  $\beta \mathbb{E}_{\mathcal{D}}[(f - \mathcal{T}^{\pi}f)(s,a))^{2}]$ . All hyperparameters are the same as ATAC, including  $\beta$ . For the additional hyperparameter  $C_{\infty}$ , we do a grid search on  $\{1,2,5,10,20,50,100,200\}$ .
Figure 2 compares the performance of ACrab and ATAC during training. It shows that our A-Crab has higher returns and smaller deviations than ATAC in various settings (walker2d-random, halfcheetah-medium-replay, hopper-medium-expert). We provide more details and results in Appendix F. We also observed that in most settings, A-Crab has a smaller variance, which shows that the training procedure of A-Crab is more stable. We provide the choice of  $\beta$  and  $C_{\infty}$  for each setting in Table F.3. Note that we use the same value of  $\beta$  as in ATAC.
(a) walker2d-rand
(b) halfcheetah-med-replay
(c) hopper-med-exp
Figure 2: Comparison of A-Crab and ATAC. For each algorithm, we run 8 copies with random seeds 0-7 and plot the mean and standard deviation. We use the same pre-training method as ATAC for 100 epochs, and the plot starts after pre-training.
We present a new offline RL algorithm called A-Crab (Algorithm 1) that can be combined with general function approximators and handle datasets with partial coverage. A-Crab is an actor-critic method, where the critic finds a relatively pessimistic evaluation of the actor while minimizing an importance-weighted average Bellman error. We prove that A-Crab achieves the optimal statistical rate of  $1 / \sqrt{N}$  converging to the best policy "covered" in the data. Importantly, the notion of coverage here is a weaker  $\ell_2$  variant of the single-policy concentrability, which only requires the average marginalized importance weights over visitations of the target policy to be bounded. Also, A-Crab enjoys robust policy improvement that consistently improves over the data-collection behavior policy. Moreover, we empirically validated the effectiveness of A-Crab in the D4RL benchmark. Interesting avenues for future work include combining A-Crab's offline learning with an online fine-tuning algorithm with a limited trial-and-error budget and developing new measures for single-policy coverage that leverage both the visitation and hypothesis class structures.
This work is partially supported by NSF Grants IIS-1901252 and CCF-1909499. The work was done when HZ was a visiting researcher at Meta.
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. J. Mach. Learn. Res., 22(98): 1-76, 2021.
Andras Antos, Rémi Munos, and Csaba Szepesvari. Fitted Q-iteration in continuous action-space mdps. In Neural Information Processing Systems, 2007.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71 (1):89-129, 2008.
Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. arXiv preprint arXiv:1905.00360, 2019.
Jinglin Chen and Nan Jiang. Offline reinforcement learning under value and density-ratio realizability: The power of gaps. In *The 38th Conference on Uncertainty in Artificial Intelligence*, 2022.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarily trained actor critic for offline reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162, 2022.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdelmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414-419, 2022.
Eyal Even-Dar, Sham M Kakade, and Yishay Mansour. Online markov decision processes. Mathematics of Operations Research, 34(3):726-736, 2009.
Amir Massoud Farahmand, Rémi Munos, and Csaba Szepesvári. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems, 2010.
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romero-Paredes, Mohammadin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-53, 2022.
Yihao Feng, Lihong Li, and Qiang Liu. A kernel loss for solving the Bellman equation. arXiv preprint arXiv:1905.10506, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. EMaQ: Expectedmax Q-learning operator for simple yet effective offline and online RL. arXiv preprint arXiv:2007.11091, 2020.
Kaiyang Guo, Yunfeng Shao, and Yanhui Geng. Model-based offline reinforcement learning with pessimism-modulated dynamics belief. arXiv preprint arXiv:2210.06692, 2022.
Kihyuk Hong, Yuhang Li, and Ambuj Tewari. A primal-dual-critic algorithm for offline constrained reinforcement learning. arXiv preprint arXiv:2306.07818, 2023.
Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34:13626-13640, 2021.
Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pages 2730-2775. PMLR, 2022.
Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. arXiv preprint arXiv:2007.02151, 2020a.
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. GenDICE: Generalized offline estimation of stationary values. In International Conference on Learning Representations, 2020b.
Shantong Zhang, Bo Liu, and Shimon Whiteson. GradientDICE: Rethinking generalized offline estimation of stationary values. arXiv preprint arXiv:2001.11113, 2020c.
Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 5757-5773. PMLR, 2022.
In this section, we review additional related literature not covered in the introductions.
A.1 Dataset Coverage Assumptions
One central challenge of offline RL is the insufficient coverage of the dataset. In RL theory, concentration is often used to characterize dataset coverage (Munos, 2007; Scherrer, 2014). For example, many earlier works require all-policy  $\ell_{\infty}$  concentration (Scherrer, 2014; Liu et al., 2019a; Chen and Jiang, 2019; Jiang, 2019; Wang et al., 2019; Liao et al., 2020; Zhang et al., 2020a). Some works even require the ratio between occupancy probability induced by polices and the dataset distribution to be bounded for every time step (Szepesvári and Munos, 2005; Munos, 2007; Antos et al., 2008; Farahmand et al., 2010; Antos et al., 2007). The work Xie and Jiang (2021) makes stronger assumptions such as requiring lower bound on conditionals  $\mu(a|s)$ .