SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In offline RL, we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, and we want to make these assumptions as harmless as possible. In this work, we propose value-based algorithms for offline RL with PAC guarantees under just partial coverage, specifically, coverage of just a single comparator policy, and realizability of the soft (entropy-regularized) Q-function of the single policy and a related function defined as a saddle point of certain minimax optimization problem. This offers refined and generally more lax conditions for offline RL. We further show an analogous result for vanilla Q-functions under a soft margin condition. To attain these guarantees, we leverage novel minimax learning algorithms and analyses to accurately estimate either soft or vanilla Q-functions with strong  $L^2$ -convergence guarantees. Our algorithms' loss functions arise from casting the estimation problems as nonlinear convex optimization problems and Lagrangifying. Surprisingly we handle partial coverage even without explicitly enforcing pessimism.
In offline Reinforcement Learning (RL), we must learn exclusively from offline data and are unable to actively interact with the environment (Levine et al., 2020). Offline RL has garnered considerable interest in a range of applications where experimentation may be prohibitively costly or risky.
Offline RL is generally based on two types of assumptions: sufficient coverage in the offline data and sufficient function approximation. For instance, classical Fitted-Q-iteration (Antos et al., 2008; Chen and Jiang, 2019) requires (a) full coverage in the offline data,  $\max_{(s,a)}d_{\pi ,\mu_0}(s,a) / P_{\pi_b}(s,a) < \infty$  for any policy  $\pi$  where  $P_{\pi_b}(s,a)$  is the offline data's distribution on the states and actions and  $d_{\pi ,\mu_0}(s,a)$  is the state-action occupancy distribution under a policy  $\pi$  and initial-state distribution  $\mu_0(s)$ ; (b) realizability of the  $Q^{*}$ -function in a hypothesis class; and (c) Bellman completeness, i.e., the Bellman operator applied to any function in the hypothesis class remains in the class. Full coverage (a) and Bellman completeness (c) can be particularly stringent because offline data is often insufficiently exploratory and Bellman completeness significantly restricts transition dynamics.
To overcome these challenges, we here propose algorithms with guarantees under realizability of single functions and refined partial coverage of single policies, and without Bellman completeness. We tackle this by introducing two novel value-based algorithms. The first algorithm, MSQP (mimimax soft-Q-learning with penalization), comprises of two steps: learning soft Q-functions (a.k.a., entropy-
*This work was done at Cornell University
Table 1: Summary of partial-coverage-type guarantees with model-free general function approximation. Here,  $w^{\star} \coloneqq d_{\pi^{\star},\mu_0} / P_b$  where  $d_{\pi^{\star},\mu_0}$  is the occupancy distribution under the optimal policy  $\pi^{\star}$  starting from  $\mu_0$  and  $P_b$  is the distribution over the offline data. A function  $\tilde{w}_{\alpha}^{\star}$  is a regularized marginal density ratio that satisfies  $\tilde{w}_0^\star = w^\star$ . Functions  $q^{\star}, q_{\alpha}^{\star}, q_{\pi}$  are the optimal  $Q^{\star}$ -function, the soft Q-function, and the Q-function under a policy  $\pi$ , respectively. Functions  $v_{\alpha}^{\star}, l_{\alpha}^{\star}$  are Lagrange multipliers of specific minimax optimization problems. The operator  $B^{\pi}$  is a Bellman operator under a policy  $\pi$ . Function classes  $\mathcal{W}, \mathcal{Q}, \mathcal{L}, \mathcal{V}$  consist of functions that map states (and actions) to real numbers. Note the guarantees provided by Jiang and Huang (2020); Xie et al. (2021) are more general than the below in that the output policy can compete with any policy in the policy class  $\Pi$ . For simplicity, we set the comparator policy to be the optimal policy  $\pi^{\star}$  in this table. Note that other studies (Ozdaglar et al., 2023; Rashidinejad et al., 2022; Zhu et al., 2023) proposing model-free general function approximation under partial coverage rely on the completeness-type assumption as in (Xie et al., 2021) or realizability for any  $\pi$  as in Jiang and Huang (2020).
Primary Assumptions
Jiang and Huang (2020)	w* ∈ W, qπ ∈ Q ∀π ∈ Π
Xie et al. (2021)	qπ ∈ Q, BπQ ⊂ Q ∀π ∈ Π
Zhan et al. (2022)	tilde{w}_{\alpha}^* ∈ W, v_{\alpha}^* ∈ V
MSQP	q*_{\alpha} ∈ Q, l*_{\alpha} ∈ L
Chen and Jiang (2022)	Hard margin, w* ∈ W, q* ∈ Q
MQP	Soft margin, q* ∈ Q, l* ∈ L
regularized Q-functions, as defined in Fox et al., 2015; Schulman et al., 2017) from offline data, and using the softmax policies of the learned soft Q-functions. The second algorithm, MQP (mimimax Q-learning with penalization), consists of two steps: learning standard Q-functions from offline data and employing the greedy policy of the learned Q function on the offline data.
Using the above-mentioned two algorithms, we attain PAC guarantees under partial coverage and realizability, yet without Bellman completeness. In particular, in MSQP using soft Q-functions, we ensure strong performance under the realizability of  $q_{\alpha}^{\star}, l_{\alpha}^{\star}$  and the (density-ratio-based) partial coverage  $\max_{(s,a)} d_{\pi_{\alpha}^{\star}, \mu_0}(s,a) / P_b(s,a) < \infty$ . Here  $q_{\alpha}^{\star}$  is a soft Q-function,  $l_{\alpha}^{\star}$  is a function that possesses a certain dual relation to  $q_{\alpha}^{\star}, \pi_{\alpha}^{\star}$  is the soft-max optimal policy, and  $\alpha$  is the temperature parameter for the entropy-regularization. Notably,  $\max_{(s,a)} d_{\pi_{\alpha}^{\star}, \mu_0}(s,a) / P_b(s,a) < \infty$  is significantly less stringent than the uniform coverage in that the coverage is only imposed against a policy  $\pi_{\alpha}^{\star}$ . In MQP using Q-functions, we similarly ensure strong performance under a soft margin, the realizability of  $q^{\star}, l^{\star}$ , and the partial coverage  $\max_{(s,a)} d_{\pi^{\star}, \mu_0}(s,a) / P_b(s,a) < \infty$ . Here  $q^{\star}$  is the vanilla Q-function and  $l^{\star}$  is a function that possesses a certain dual relation to  $q^{\star}$ , and  $\pi^{\star}$  is the usual optimal policy. Note the soft margin is introduced to allow realizability on standard Q-functions rather than soft Q-functions. However, the conditions  $\max_{(s,a)} d_{\pi_{\alpha}^{\star}, \mu_0}(s,a) / P_b(s,a) < \infty$  or  $\max_{(s,a)} d_{\pi^{\star}, \mu_0}(s,a) / P_b(s,a) < \infty$  may still be strong as these marginal density ratios may not exist in large-scale MDPs. For example, this condition is easily violated when the initial distribution  $\mu_0$  is not covered by  $P_b$  (i.e.,  $\max_s \mu_0(s) / P_b(s) = \infty$  where  $P_b(s) := \sum_a P_b(s,a)$ ). Therefore, as an additional innovation, in our algorithms we can further relax these density-ratio-based partial coverage conditions. Specifically, we can demonstrate results under a refined partial coverage, which is adaptive to Q-function classes, even when the initial distribution  $\mu_0$  is not covered by  $P_b$ .<sup>2</sup>
The primary challenge lies in the design of loss functions for effectively learning soft Q-functions and vanilla Q-functions from offline data without Bellman completeness. To tackle this, we devise new minimax loss functions with certain regularization terms to achieve favorable  $L^2$ -convergence rates on the offline data (i.e., in terms of  $\mathbb{E}_{(s,a)}\sim P_b[\{\hat{q}_\alpha -q\} ^2 (s,a)]$  given an estimator  $\hat{q}$ ). This result serves as the key building block for obtaining refined partial coverage under realizability and is of independent interest in its own right. Existing results are often constrained to specific models, such as linear models (Shi et al., 2022a), or they require Bellman completeness (Antos et al., 2008; Chen and Qi, 2022; Chen and Jiang, 2019). In contrast, our guarantee is applicable to any function
Note  $\mu_0$  and  $P_{b}$  could be generally different even in the contextual bandit setting. This important setting is often considered in the literature on external validity/transportability in causal inference, as results of randomized clinical trials cannot be directly transported because covariate distributions in offline data and target data are different (Cole and Stuart, 2010; Pearl and Baireinboim, 2014; Dahabreh et al., 2019).
approximation method, without the need for Bellman completeness. To the best of our knowledge, this is the first guarantee of its kind.
Our work exhibits marked improvements over two closely related studies (Zhan et al., 2022; Chen and Jiang, 2022). Similar to our work, they propose algorithms that operate under the realizability of specific functions and partial coverage, yet without Bellman completeness. Zhan et al. (2022) ensures a PAC guarantee under (a') partial coverage in the offline data  $\max_{(s,a)}d_{\tilde{\pi}_{\alpha}^{\star},\mu_0}(s,a) / P_b(s,a) < \infty$  where  $\tilde{\pi}_{\alpha}^{\star}$  is a specific near-optimal policy under the regularization, which differs from the soft optimal policy, and (b') realizability of  $d_{\tilde{\pi}_{\alpha}^{\star},\mu_0} / P_b$  and the regularized value function. However, unlike MSQP, it is unclear how to refine the abovementioned coverage, i.e., the guarantee could be vacuous when the initial distribution is not covered by offline data. A similar guarantee, but without regularization, is obtained under the additional hard margin (a.k.a., gap) condition in Chen and Jiang (2022). Our soft margin is a strict relaxation of the hard margin, which is important because, unlike the soft margin, the hard margin generally does not hold in continuous state spaces and involves very large constants in discrete state spaces. Lastly, although Chen and Jiang (2022); Zhan et al. (2022) use completely different algorithms and attain guarantees for regularized value-functions and non-regularized value functions, respectively, our guarantee can afford guarantees for regularized and non-regularized value-functions in a unified manner since MQP can be seen as a limit of MSQP when  $\alpha$  goes to 0.
Our contributions are summarized below and in Table 1.
1. We establish that the optimal policy can be learned under partial coverage and realizability of the optimal soft Q-function and its dual. Notably, we abstain from the use of possibly stronger conditions in offline RL, such as full coverage, Bellman completeness, and uniform realizability over the policy class (such as  $q_{\pi} \in \mathcal{Q}$  for any  $\pi$  as in Jiang and Huang, 2020). In particular, while a similar guarantee is provided in Zhan et al. (2022), our partial coverage guarantee has an advantage in that we are able to potentially accommodate scenarios where the initial distribution is not covered by  $P_b$ . This is feasible because our algorithm is value-based in nature, which allows us to leverage the structure of the Q-function classes and refine the coverage condition.
2. We demonstrate that the optimal policy can be learned under partial coverage, realizability of the Q-function and its dual, and a soft margin. While a similar guarantee is obtained in Chen and Jiang (2022), our guarantee has the advantage that the soft margin is significantly less stringent than the hard margin required therein.
We summarize related works as follows. Further related works is discussed in Section A.
Offline RL under partial coverage. There is a growing number of results under partial coverage following the principle of pessimism in offline RL (Yu et al., 2020; Kidambi et al., 2020). In comparison to works that focus on tabular (Rashidinejad et al., 2021; Li et al., 2022; Shi et al., 2022b; Yin and Wang, 2021) or linear models (Jin et al., 2020; Chang et al., 2021; Zhang et al., 2022; Nguyen-Tang et al., 2022; Bai et al., 2022), our emphasis is on general function approximation (Jiang and Huang, 2020; Uehara and Sun, 2021; Xie et al., 2021; Zhan et al., 2022; Zhu et al., 2023; Rashidinejad et al., 2022; Zanette and Wainwright, 2022; Ozdaglar et al., 2023). Among them, we specifically focus on model-free methods. The representative work is summarized in Table 1.
Soft (entropy-regularized) Q-functions. Soft Q-functions are utilized in various contexts in RL (Geist et al., 2019; Neu et al., 2017). They have been shown to improve performance in online RL settings, as demonstrated in Soft Q-Learning (Fox et al., 2015; Schulman et al., 2017) and Soft Actor Critic (Haarnoja et al., 2018). In the field of imitation learning, they play a crucial role in Maximum Entropy IRL (Ziebart et al., 2008, 2010). Furthermore, within the realm of offline RL, these soft Q-functions are utilized to make the learned policy and behavior policy sufficiently similar (Wu et al., 2019; Fakoor et al., 2021). However, to the best of the authors' knowledge, none of these proposals in the context of offline RL have provided sample complexity results under partial coverage.
Lagrangian view of offline RL. In the realm of offline policy evaluation (OPE), Nachum and Dai (2020); Yang et al. (2020); Huang and Jiang (2022) have formulated the problem as a constrained
linear optimization problem. Notably, within the context of policy optimization, Zhan et al. (2022) have proposed estimators for regularized density ratios with  $L^2$ -convergence guarantees, which is a crucial step in obtaining a near-optimal policy. Our work is similarly motivated, but with a key distinction: our target functions are the soft Q-function and Q-function, rather than the regularized density ratio, which presents additional analytical challenges due to the nonlinear constraint.
We consider an infinite-horizon discounted MDP  $\mathcal{M} = \langle S, \mathcal{A}, P, r, \gamma, \mu_0 \rangle$  where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the finite action space,  $\gamma \in [0,1)$  is the discount factor, reward  $r$  is a random variable following  $P_r(\cdot \mid s, a)$  on  $[R_{\min}, R_{\max}]$  ( $R_{\min} \geq 0$ ),  $\mu_0$  is the initial distribution. A policy  $\pi: \mathcal{S} \to \Delta(\mathcal{A})$  is a map from the state to the distribution over actions. We denote the discounted state-action occupancy distribution under a policy  $\pi$  starting from an initial distribution  $\mu_0$  by  $d_{\pi, \mu_0}(s, a)$ . With slight abuse of notation, we denote  $d_{\pi, \mu_0}(s) = \sum_a d_{\pi, \mu_0}(s, a)$ . We define the value under  $\pi$  as  $J(\pi) := \mathbb{E}_{\pi}[\sum_{t=0}^{\infty} \gamma^t \tilde{r}(s_t, a_t)]$  where the expectation is taken under  $\pi$ . We denote the optimal policy  $\arg \max_{\pi} J(\pi)$  by  $\pi^\star$ , and its Q-function  $\mathbb{E}_{\pi^\star}[\sum_t \gamma^t \tilde{r}(s_t, a_t) \mid s_0 = s, a_0 = a]$  by  $q^\star(s, a)$ .
In offline RL, using offline data  $\mathcal{D} = \{(s_i,a_i,r_i,s_i'):i = 1,\ldots ,n\}$ , we search for the policy  $\pi^{\star}$  that maximizes the policy value. We suppose each  $(s_i,a_i,r_i,s_i')$  is sampled i.i.d. from  $s_i\sim P_b,a_i\sim \pi_b(\cdot |s)$ ,  $r_i\sim P_r(\cdot |s_i,a_i)$ ,  $s_i'\sim P(\cdot |s_i,a_i)$ . We denote the sample average of  $f$  by  $\mathbb{E}_n[f(s,a,r,s')] = \frac{1}{n}\sum_{i = 1}^{n}f(s_i,a_i,r_i,s_i')$ , and the expectation of  $f$  with respect to the offline data distribution by  $\mathbb{E}[f(s,a,r,s')]$  (without any scripts). The policy  $\pi_b$  used to collect data is typically referred to as a behavior policy. With slight abuse of notation, we denote  $P_{b}(s,a) = P_{b}(s)\pi_{b}(a\mid s)$ .
Notation. We denote the support of  $P_{b}(\cdot)$  by  $(\mathcal{S}\times \mathcal{A})_b$  , and the  $L^{\infty}$  -norm on  $(\mathcal{S}\times \mathcal{A})_b$  by  $\| \cdot \|_{\infty ,b}$  . The  $L^{\infty}$  -norm on  $(\mathcal{S}\times \mathcal{A})$  is denoted by  $\| \cdot \|_{\infty}$  . We define  $w_{\pi}(s,a) = d_{\pi ,\mu_0}(s,a) / P_b(s,a)$  (if it exists). We define  $\operatorname {sofmax}(h) = \frac{\exp(h(s,a))}{\sum_a\exp(h(s,a))}$  and  $\| h\| _2 = \mathbb{E}_{(s,a)\sim P_b}[h^2 (s,a)]^{1 / 2}$  for  $h:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$  . We denote universal constants by  $c_{1},c_{2},\ldots$  . We use the convention  $a / 0 = \infty$  when  $a\neq 0$  and  $0 / 0 = 0$
In this section, we present two algorithms. The first algorithm aims to estimate the soft optimal policy by first estimating a soft Q-function. The second algorithm estimates the optimal policy after estimating the Q-function.
3.1 Minimax Soft-Q-learning with Penalization
Our ultimate aim is to mimic the optimal policy  $\pi^{\star}$ . As a first step, we begin by finding a policy that maximizes the following regularized objective:  $\arg \max_{\pi} J_{\alpha}(\pi)$  where for  $\alpha > 0$  we define
$$
J _ {\alpha} (\pi) = (1 - \gamma) ^ {- 1} \mathbb {E} _ {(s, a) \sim d _ {\pi , \mu_ {0}}, r \sim P _ {r} (\cdot | s, a)} [ r - \underbrace {\alpha \log \{\pi (a \mid s) / \pi_ {b} (a \mid s) \}} _ {\text {K L p e n a l t y (b e t w e e n} \pi \text {a n d} \pi_ {b})} ]
$$
This objective function is used in a variety of contexts in RL as mentioned in Section 1.1. The optimal policy that maximizes  $J_{\alpha}(\pi)$  with respect to  $\pi$  is
$$
\pi_ {\alpha} ^ {\star} = \operatorname {s o f t m a x} \left(q _ {\alpha} ^ {\star} / \alpha + \log \pi_ {b}\right), \tag {1}
$$
where  $q_{\alpha}^{\star}:S\times \mathcal{A}\to \mathbb{R}$  is the soft Q-function uniquely characterized by the soft Bellman equation:
$$
\forall (s, a); \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ \gamma \Omega_ {\alpha , \pi_ {b}} (q _ {\alpha} ^ {\star}) (s ^ {\prime}) + r - q _ {\alpha} ^ {\star} (s, a) \mid s, a ] = 0,
$$
where  $\Omega_{\alpha, \pi_b} : [\mathcal{S} \times \mathcal{A} \to \mathbb{R}] \to [\mathcal{S} \to \mathbb{R}]$  has  $\Omega_{\alpha, \pi_b}(q)(s) = \alpha \log \sum_{a} \{\exp(q(s, a') / \alpha) \pi_b(a' | s)\}$ . As opposed to the standard objective function with  $\alpha = 0$ , the KL penalty term serves as a regularization term that renders  $\pi_\alpha^\star$  sufficiently proximate to  $\pi_b$ . As  $\alpha$  approaches  $\infty$ , the optimal policy  $\pi_\alpha^\star$  approaches  $\pi_b$ . On the other hand, when  $\alpha = 0$ ,  $\pi_\alpha^\star$  is  $\pi^\star$ . Thus, in order to compete with  $\pi^\star$ , it is necessary to keep  $\alpha$  sufficiently small. We elaborate on this selection procedure in Section 5.
The natural method for offline RL using this formulation involves learning  $q_{\alpha}^{\star}$  from the offline data and plugging it into (1). The question that remains is how to accurately learn  $q_{\alpha}^{\star}$  from the offline data. We consider the following optimization problem:
$$
\arg \min  _ {q \in \mathcal {Q} ^ {\prime}} 0. 5 \mathbb {E} _ {(s, a) \sim P _ {b}} [ q ^ {2} (s, a) ] \tag {2}
$$
Algorithm 1 MSQP (Minimax Soft-Q-learning with Penalization)
1: Require: Parameter  $\alpha \in \mathbb{R}^{+}$ , Models  $\mathcal{Q},\mathcal{L}\subset [S\times \mathcal{A}\to \mathbb{R}^{+}]$ .
2: Estimate  $q_{\alpha}^{\star}$  as follows:
$$
\hat {q} _ {\alpha} \in \underset {q \in \mathcal {Q}} {\arg \min } \underset {l \in \mathcal {L}} {\max } \mathbb {E} _ {n} \left[ q ^ {2} (s, a) / 2 + \left\{\gamma \Omega_ {\alpha , \pi_ {b}} (q) \left(s ^ {\prime}\right) + r - q (s, a) \right\} l (s, a) \right]. \tag {5}
$$
3: Estimate the soft optimal policy:  $\hat{\pi}_{\alpha} = \mathrm{softmax}(\hat{q}_{\alpha} / \alpha + \log \pi_{b})$
Algorithm 2 MQP (Minimax  $Q^{\star}$ -learning with Penalization)
1: Require: Models  $\mathcal{Q},\mathcal{L}\subset [S\times \mathcal{A}\to \mathbb{R}^{+}]$
2: Estimate  $q^{\star}$  as follows:
$$
\hat {q} _ {0} \in \underset {q \in \mathcal {Q}} {\arg \min } \underset {l \in \mathcal {L}} {\max } \mathbb {E} _ {n} \left[ q ^ {2} \left(s, a\right) / 2 + \left\{\gamma \underset {a ^ {\prime}} {\max } q \left(s ^ {\prime}, a ^ {\prime}\right) + r - q (s, a) \right\} l (s, a) \right]. \tag {8}
$$
3: Estimate the optimal policy:  $\hat{\pi}_0(a\mid s) = \arg \max_{a:\pi_b(a|s) > 0}\hat{q}_0(s,a)$ .
where  $\mathcal{Q}'$  consists of all functions  $q: S \times \mathcal{A} \to \mathbb{R}$  satisfying
$$
\mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ \gamma \Omega_ {\alpha , \pi_ {b}} (q) (s ^ {\prime}) + r - q (s, a) \mid s, a ] = 0 \quad \forall (s, a) \in (\mathcal {S} \times \mathcal {A}) _ {b}. \tag {3}
$$
Here, because of the constraint (3), the solution is  $q_{\alpha}^{\star}$ . Furthermore, we use  $q^{2}(s,a)$  in (2) because this choice relaxes the equality in (3) to an inequality  $\leq 0$  as we will demonstrate in Section B. Consequently, the entire optimization problem outlined in (2) and (3) transforms into a convex optimization problem.
Then, using the method of Lagrange multipliers, (2) is transformed into
$$
\min  _ {q} \max  _ {l} L _ {\alpha} (q, l), \quad L _ {\alpha} (q, l) := \mathbb {E} \left[ q ^ {2} (s, a) / 2 + \left\{\gamma \Omega_ {\alpha , \pi_ {b}} (q) \left(s ^ {\prime}\right) + r - q (s, a) \right\} l (s, a) \right]. \tag {4}
$$
Being motivated by the above formulation, our MSQP algorithm, specified in Algorithm 1, approximates this formulation by replacing expectations with sample averages and restricting optimization to function classes with bounded complexity.
Remark 1 (Computation). Although minimax optimization is generally difficult to solve, it is computationally feasible when we choose RKHS or linear function classes for  $\mathcal{L}$ . In this case, we can solve the inner maximization problem analytically in closed form, as the objective function is linear in  $l$ . As a result, the minimax optimization problem reduces to empirical risk minimization.
3.2 Minimax  $Q^{\star}$ -learning with Penalization
Next, we examine a policy learning algorithm utilizing  $Q^{\star}$ -functions. To learn  $Q^{\star}$ , our objective function is derived from the constrained optimization problem:
$$
\arg \min  _ {q \in \mathcal {Q} ^ {\star^ {\prime}}} 0. 5 \mathbb {E} _ {(s, a) \sim P _ {b}} [ q ^ {2} (s, a) ] \tag {6}
$$
where  $\mathcal{Q}^{\star'}$  consists of all functions  $q: S \times \mathcal{A} \to \mathbb{R}$  satisfying
$$
\forall (s, a) \in (\mathcal {S} \times \mathcal {A}) _ {b}; \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ \gamma \max  _ {a ^ {\prime} \in \mathcal {A}} q (s ^ {\prime}, a ^ {\prime}) + r - q (s, a) \mid s, a ] = 0.
$$
Next, again using the method of Lagrange multipliers, (6) is transformed into
$$
\min  _ {q} \max  _ {l} L _ {0} (q, l), L _ {0} (q, l) := \mathbb {E} [ q ^ {2} (s, a) / 2 + \{\gamma \max  _ {a ^ {\prime}} q \left(s ^ {\prime}, a ^ {\prime}\right) + r - q (s, a) \} l (s, a) ]. \tag {7}
$$
Note  $L_{0}$  is the limit of  $L_{\alpha}$  as  $\alpha \to 0$
Our MQP algorithm, specified in Algorithm 2, similarly approximates this formulation by replacing expectations with sample averages and restricting optimization to function classes with bounded complexity. Our final policy is greedy with respect to the learned Q-function but restricting to the support of the offline data in order to avoid exploiting regions not covered by the offline data.
Remark 2 (Prominent differences). There exist several other minimax estimators for  $Q^{\star}$  including BRM (Antos et al., 2008) and MABO (Xie and Jiang, 2020). Although these ensure convergence guarantees in terms of Bellman residual errors, they do not ensure the guarantee in terms of  $L^2$ -errors, which is our focus. Our minimax objective function differs significantly from that of the aforementioned approaches, and its unique design plays a pivotal role in enabling  $L^2$ -rates.
4  $L^2$ -convergence Rates for Soft  $Q$ -functions and  $Q^{\star}$ -functions
To analyze our Q-estimators we first establish conditions that ensure  $q_{\alpha}^{\star} = \arg \min_{q \in \mathcal{Q}} \max_{l \in \mathcal{L}} L_{\alpha}(q, l)$  on the support  $(\mathcal{S} \times \mathcal{A})_b$ . Building on this, we prove  $L^2$ -convergence rates for  $\hat{q}_{\alpha}$  and  $\hat{q}_0$ . These  $L^2$ -convergence guarantees are subsequently translated into performance guarantees of the policies we output in Section 5.