SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proof of Proposition 3.2 is shown in Appendix A.2. According to the above proposition, one can first learn the  $V$  function according to (6), and then use the learned  $V$  function to learn the desired policy by (7). We call the first step  $V$ -learning and the second step policy learning, which will be discussed in detail in Sections 3.1 and 3.2 respectively. Finally, the main algorithm, which we call VP-learning, is presented in Algorithm 1.
Algorithm 1 VP-learning
1: Input: Dataset  $\mathcal{D} = \{(s_i, a_i, r_i, s_i', g_i)\}_{i=1}^N$ ,  $\mathcal{D}_0 = \{(s_{0,i}, g_{0,i})\}_{i=1}^{N_0}$ , value function class  $\mathcal{V}$ , policy class  $\Pi$ , model class  $\mathcal{P}$  for stochastic settings.
2: Obtain  $\hat{U}$  by V-Learning (Algorithm 2 or 3).
3: Obtain  $\hat{\pi}$  by policy learning (Algorithm 4) using learned function  $\hat{U}$ .
4: Output:  $\hat{\pi}$ .
3.1  $V$ -Learning
$$
L _ {\alpha} (V) = \alpha ((1 - \gamma) \mathbb {E} _ {(s, g) \sim (\rho , p (g))} [ V (s; g) ] + \mathbb {E} _ {(s, a, g) \sim \mu} [ \mathbb {1} \{g _ {*} ^ {\prime} (A _ {V} (s, a; g)) \geq 0 \} \bar {g} _ {*} (A _ {V} (s, a; g)) ]). \tag {8}
$$
Then (6) is equivalent to  $\min_{V(s;g)\geq 0}L_{\alpha}(V)$ . A natural estimator of  $L_{\alpha}(V)$  is
$$
\frac {1 - \gamma}{N _ {0}} \sum_ {i = 1} ^ {N _ {0}} \alpha \cdot V \left(s _ {0, i}; g _ {0, i}\right) + \frac {1}{N} \sum_ {i = 1} ^ {N} \alpha \cdot g _ {* +} \left(r _ {i} + \gamma V \left(s _ {i} ^ {\prime}; g _ {i}\right) - V \left(s _ {i}; g _ {i}\right)\right). \tag {9}
$$
However, when the transition kernel is not deterministic, this estimator is biased and will cause an over-estimation issue since  $g_{*}(x) = \alpha f_{*}(x / \alpha) = \frac{\alpha(x / \alpha + 1)^{2}}{2} -\frac{\alpha}{2}$  contains a square operator outside of the Bellman operator (consider estimating  $(\mathbb{E}[X])^2$  using  $\frac{1}{N}\sum X_i^2$ ).
Therefore, we use the original version of Ma et al. [2022c] for  $V$ -learning in deterministic dynamics (Algorithm 2 in Section 3.1.1), and a slightly modified version in stochastic dynamics (Algorithm 3 in Section 3.1.2). For both settings, we assume realizability of  $V_{\alpha}^{*}$  on value function class  $\mathcal{V}$ :
Assumption 2 (Realizability of  $V_{\alpha}^{*}$ ). Assume  $V_{\alpha}^{*} \in \mathcal{V}$ .
3.1.1  $V$ -Learning in Deterministic Dynamics
When the transition kernel  $P$  is deterministic, it holds that  $\mathcal{T}V(s,a;g) = V(s';g)$  where  $P(s'|s,a) = 1$ . In this case, the natural estimator (9) is unbiased and can be directly applied to the  $V$ -learning procedure. The  $V$ -learning algorithm for deterministic dynamic settings is presented in Algorithm 2.
Algorithm 2  $V$ -learning in deterministic dynamics
1: Input: Dataset  $\mathcal{D} = \{(s_i, a_i, r_i, s_i', g_i)\}_{i=1}^N$ ,  $\mathcal{D}_0 = \{(s_{0,i}, g_{0,i})\}_{i=1}^{N_0}$ , value function class  $\mathcal{V}$ .
2:  $V$ -learning by solving  $\hat{V} = \arg \min_{V \in \mathcal{V}} \hat{L}^{(d)}(V)$  where
$$
\hat {L} ^ {(d)} (V) \triangleq \frac {1 - \gamma}{N _ {0}} \sum_ {i = 1} ^ {N _ {0}} \alpha \cdot V \left(s _ {0, i}; g _ {0, i}\right) + \frac {\alpha}{N} \sum_ {i = 1} ^ {N} g _ {* +} \left(r _ {i} + \gamma V \left(s _ {i} ^ {\prime}; g _ {i}\right) - V \left(s _ {i}; g _ {i}\right)\right). \tag {10}
$$
3:  $\hat{U}(s, a; g) \gets r(s; g) + \gamma \hat{V}(s'; g) - \hat{V}(s; g) + \alpha$
4: Output:  $\hat{V},\hat{U}$
Now for any  $V$ , we define  $U_V(s, a; g) = r(s; g) + \gamma \mathcal{T} V(s, a; g) - V(s; g) + \alpha = A_V(s, a; g) + \alpha$  which can be interpreted as the advantage function of  $V$  with an  $\alpha$ -shift. We also denote  $U_{\alpha}^{*} = U_{V_{\alpha}^{*}}$ .
Note that besides the learned  $\hat{V}$  function, Algorithm 2 also outputs a  $\hat{U}$  function. By (7), one can observe that in policy learning, what we indeed need is  $\hat{U}$  instead of  $\hat{V}$ , and thus in the  $V$ -learning procedure we also compute this  $\hat{U}$  function in preparation for policy learning.
One may challenge that  $\hat{U}$  cannot be computed for all  $(s, a; g)$  since we do not have knowledge of all  $r(s; g)$ . However, we only need the value of  $\hat{U}(s_i, a_i; g_i)$  for  $(s_i, a_i; g_i)$  contained in the offline dataset, where  $r_i$  is also contained. Therefore, we can evaluate the value of  $\hat{U}$  at all  $(s, a; g)$  tuples requested in the policy learning algorithm.
Note that Algorithm 2 is equivalent to the first step of Ma et al. [2022c] except for the choice of  $\alpha$  and a clip for the value of  $g_{*}$ . However, the above  $V$ -learning, as well as the original GoFAR algorithm, might suffer the over-estimation issue under stochastic dynamics, and we present algorithms suitable for stochastic dynamics in Section 3.1.2.
3.1.2  $V$ -Learning in Stochastic Dynamics
When the transition kernel is stochastic, one cannot directly use  $V(s'; g)$  to estimate  $\mathcal{T}V(s, a; g)$ . Since  $\mathcal{T}V(s, a; g) = \mathbb{E}_{s' \sim P(\cdot|s, a)}[V(s'; g)]$ , a natural idea is to learn the ground-truth transition kernel  $P^{\star 1}$  first, and then use the learned transition kernel  $\hat{P}$  to estimate  $\mathcal{T}V(s, a; g)$ :  $\hat{\mathcal{T}}V(s, a; g) = \mathbb{E}_{s' \sim \hat{P}(\cdot|s, a)}[V(s'; g)]$ . This is achievable under the following realizability assumption.
Assumption 3 (Realizability of the ground-truth transition model). Assume the ground-truth transition kernel  $P^{\star} \in \mathcal{P}$ .
The algorithm for stochastic dynamic settings is presented in Algorithm 3, where we first learn the transition kernel  $\hat{P}$ , and then plug in the learned transition kernel to learn  $V$  function. Similar to Algorithm 2, we also compute  $\hat{U}$  in  $V$ -learning procedure.
Algorithm 3  $V$ -learning in stochastic dynamics
1: Input: Dataset  $\mathcal{D} = \{(s_i, a_i, r_i, s_i', g_i)\}_{i=1}^N$ ,  $\mathcal{D}_0 = \{(s_{0,i}, g_{0,i})\}_{i=1}^{N_0}$ , value function class  $\mathcal{V}$ , model class  $\mathcal{P}$ .
2: Estimate the transition kernel via maximum likelihood estimation (MLE)
$$
\hat {P} = \max  _ {P \in \mathcal {P}} \frac {1}{N} \sum_ {i = 1} ^ {N} \log P \left(s _ {i} ^ {\prime} \mid s _ {i}, a _ {i}\right) \tag {11}
$$
3:  $V$ -learning using the learned transition kernel:  $\hat{V} = \arg \min_{V\in \mathcal{V}}\hat{L}^{(s)}(V)$  with
$$
\hat {L} ^ {(s)} (V) \triangleq \frac {1 - \gamma}{N _ {0}} \sum_ {i = 1} ^ {N _ {0}} \alpha \cdot V \left(s _ {0, i}; g _ {0, i}\right) + \frac {\alpha}{N} \sum_ {i = 1} ^ {N} g _ {* +} \left(r _ {i} + \gamma \hat {\mathcal {T}} V \left(s _ {i}, a _ {i}; g _ {i}\right) - V \left(s _ {i}; g _ {i}\right)\right). \tag {12}
$$
where  $\hat{\mathcal{T}} V(s,a;;g) = \mathbb{E}_{s'\sim \hat{P} (\cdot |s,a)}[V(s';g)]$
4:  $\hat{U}(s, a; g) \gets r(s; g) + \gamma \hat{T} \hat{V}(s, a; g) - \hat{V}(s; g) + \alpha$
5: Output:  $\hat{V},\hat{U}$
We now derive policy learning, the second step of the VP-learning algorithm. Note that  $\pi_{\alpha}^{*} = \arg \max_{\pi}\mathbb{E}_{(s,a,g)\sim d_{\alpha}^{*}}[\log \pi (a|s,g)]$ . By (7), we also have
$$
d _ {\alpha} ^ {*} (s, a; g) = \mu (s, a; g) g _ {*} ^ {\prime} (r (s; g) + \gamma \mathcal {T} V _ {\alpha} ^ {*} (s, a; g) - V _ {\alpha} ^ {*} (s; g)) _ {+} = \mu (s, a; g) \frac {U _ {\alpha} ^ {*} (s , a ; g) _ {+}}{\alpha}.
$$
For notation convenience, in stochastic settings, we use  $P^{\star}$  to denote the ground-truth transition kernel.
Therefore,  $\pi_{\alpha}^{*} = \arg \max_{\pi} L_{\alpha}^{\mathrm{MLE}}(\pi)$  where  $L_{\alpha}^{\mathrm{MLE}}(\pi) \triangleq \mathbb{E}_{(s,a,g) \sim \mu}\left[\frac{U_{\alpha}^{*}(s,a;g) +}{\alpha} \log \pi(a|s,g)\right]$ . Since we already learned  $\hat{U}$ , which is close to  $U_{\alpha}^{*}$ , we can use the following estimator for  $L_{\alpha}^{\mathrm{MLE}}(\pi)$ :
$$
\hat {L} ^ {\mathrm {M L E}} (\pi) = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {\hat {U} \left(s _ {i} , a _ {i} ; g _ {i}\right) _ {+}}{\alpha} \log \pi \left(a _ {i} \mid s _ {i}, g _ {i}\right). \tag {13}
$$
Algorithm 4 Policy learning
1: Input: Dataset  $\mathcal{D} = \{(s_i, a_i, r_i, s_i', g_i)\}_{i=1}^N$ , policy class  $\Pi$ ,  $\hat{U}$  learned by Algorithm 2 or 3.
2: Policy learning by:
$$
\hat {\pi} = \arg \max  _ {\pi \in \Pi} \hat {L} ^ {\mathrm {M L E}} (\pi) \triangleq \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {\hat {U} \left(s _ {i} , a _ {i} ; g _ {i}\right) _ {+}}{\alpha} \log \pi \left(a _ {i} \mid s _ {i}, g _ {i}\right). \tag {14}
$$
3: Output:  $\hat{\pi}$ .
The policy learning algorithm is presented in Algorithm 4, which can be viewed as a weighted maximum likelihood estimation (MLE) procedure. Finally, we make the following two assumptions on the policy class II.
Assumption 4 (Single-policy realizability). Assume  $\pi_{\alpha}^{*}\in \Pi$
Assumption 5 (Lower bound of policy). For any policy  $\pi \in \Pi$ , we assume that  $\pi(a|s, g) \geq \tau > 0$  for any  $(s, a, g) \in S \times \mathcal{A} \times \mathcal{G}$ .
Remark 1. One may consider Assumption 5 strong if  $\tau$  is a constant independent of  $\alpha$  or  $N$ . However, we allow that  $\tau$  depends on  $\alpha$ . In that case,  $\tau$  can be extremely small, and any policy mixed with a uniform policy with a tiny probability satisfies this assumption. Therefore, Assumption 5 is mild.
In this section, we provide theoretical guarantees of our main algorithm (Algorithm 1). We first show the results for  $V$ -Learning and policy learning in Section 4.1 and Section 4.2 respectively and then combine them to obtain our main theorem in Section 4.3.
4.1 Analysis of  $V$ -Learning
We mainly focus on  $V$ -learning in deterministic dynamics in this section. The analysis for stochastic dynamics is similar and presented in Appendix B.2.
As discussed in Section 3.1.1, although the first step of the algorithm is called  $V$ -learning, the main goal of this step is to estimate  $U_{\alpha}^{*} = U_{V_{\alpha}^{*}}$  accurately. The following lemma provides a theoretical guarantee that the output of  $V$ -learning algorithm  $\hat{U}$  is a good estimator of  $U_{\alpha}^{*}$  in positive parts:
Lemma 1 (Closeness of  $\hat{U}_{+}$  and  $U_{\alpha +}^{*}$ ). Under Assumptions 1 and 2, with probability at least  $1 - \delta$ ,  $\| \hat{U}_{+} - U_{\alpha +}^{*}\|_{2,\mu} \leq O\left(\sqrt{\epsilon_{stat}}\right)$ , where  $\hat{U}$  is the output of Algorithm 2 and  $\epsilon_{stat} \asymp V_{\max}^2 \sqrt{\frac{\log(|\mathcal{V}| / \delta)}{N}}$ .
Proof sketch. By standard concentration argument, it can be shown that the empirical estimator  $\hat{L}^{(d)}$  in Algorithm 2 (which is unbiased in deterministic dynamics) concentrates well on  $L_{\alpha}$  for all  $V \in \mathcal{V}$  (Lemma 3). Therefore, by realizability of  $V_{\alpha}^{*}$ , the value of  $L_{\alpha}$  at  $V_{\alpha}^{*}$  and the learned  $V$ -function  $\hat{V}$  are close (Lemma 4). Finally, one can observe that  $L_{\alpha}$  is "semi-strongly" convex w.r.t.  $U_{V+}$  in  $\|\cdot\|_{2,\mu}$ -norm, and thus we can show that  $\hat{U}_{+}$  and  $U_{\alpha+}^{*}$  are also close.
The complete proof of Lemma 1 is deferred to Appendix B.1. In Appendix B.2, we also show the counterpart of Lemma 1 for stochastic dynamic settings.
4.2 Analysis of Policy Learning
After obtaining an accurate estimator  $\hat{U}_{+}$  of  $U_{\alpha +}^{*}$  in the  $V$  -Learning procedure, i.e.,  $\| \hat{U}_{+} - U_{\alpha +}^{*}\|_{2,\mu}\lesssim$ $\sqrt{\epsilon_{\mathrm{stat}}}$ , we can use  $\hat{U}_{+}$  to perform policy learning and obtain the following guarantee:
Lemma 2 (Closeness of  $\pi_{\alpha}^{*}$  and  $\hat{\pi}$ ). Under Assumptions 4 and 5, with probability at least  $1 - \delta$ , the output policy  $\hat{\pi}$  of Algorithm 4 satisfies
$$
\mathbb {E} _ {s \sim d _ {\alpha} ^ {*}, g \sim p (g)} \| \pi_ {\alpha} ^ {*} (\cdot | s, g) - \hat {\pi} (\cdot | s, g) \| _ {\mathsf {T V}} \leq O \left(\sqrt {\epsilon_ {s t a t} ^ {M L E} / \tau^ {2}}\right),
$$
where  $\epsilon_{stat}^{MLE}$  is defined in Lemma 8.
The proof of Lemma 2 is provided in Appendix C.2. This result shows that the TV distance between the regularized optimal policy  $\pi_{\alpha}^{*}$  and the output policy  $\hat{\pi}$  by Algorithm 4 is small, which translates to a bounded performance difference between these two policies as formalized in Theorem 1.
Theorem 1 (Suboptimality of  $\hat{\pi}$ ). Under Assumptions 4 and 5, with probability at least  $1 - \delta$ , the output policy  $\hat{\pi}$  of Algorithm 4 satisfies  $J(\pi_{\alpha}^{*}) - J(\hat{\pi}) \leq O\left(V_{\max}\sqrt{\epsilon_{stat}^{MLE} / \tau^{2}}\right)$ .
The proof of Theorem 1 is deferred to Appendix C.3.
4.3 Main Theorem: Statistical Rate of Suboptimality
Theorem 1 compares the performance difference between  $\hat{\pi}$  and the regularized optimal policy  $\pi_{\alpha}^{*}$ . Since the ultimate goal is to compare with the optimal policy  $\pi^{*}$ , we also need to combine this result with Proposition 3.1. By carefully choosing the value of  $\alpha$  to balance  $J(\hat{\pi}) - J(\pi_{\alpha}^{*})$  and  $J(\pi) - J(\pi_{\alpha}^{*})$ , we can bound the suboptimality of the policy  $\hat{\pi}$  output by Algorithm 1 compared to the optimal policy  $\pi^{*}$ , leading to the following main result:
Theorem 2 (Statistical rate of suboptimality (in deterministic dynamics)). Under Assumptions 1, 2, 4 and 5, with probability at least  $1 - \delta$ , the output policy  $\hat{\pi}$  by Algorithm 1 (with the choice of Algorithm 2 for  $V$ -learning in deterministic dynamics) satisfies
$$
J (\pi^ {*}) - J (\hat {\pi}) \lesssim \left(\frac {V _ {\max} ^ {3} (C _ {\alpha} ^ {*}) ^ {3} \log (1 / \tau) \log (| \mathcal {V} | | \Pi | / \delta)}{\tau^ {2} N ^ {1 / 4}}\right) ^ {1 / 3}
$$
if we choose  $\alpha \asymp \left(\frac{V_{\mathrm{max}}^3\log(1 / \tau)\log(|\mathcal{V}||\Pi| / \delta)}{\tau^2(C_\alpha^*)^3N^{1 / 4}}\right)^{1 / 3}$  and assume  $N = N_0$
The proof of Theorem 2 is deferred to Appendix D.1. Note that Theorem 2 provides a suboptimality rate of  $O(1 / N^{1 / 12})$  which implies an  $O(1 / \mathrm{poly}(\epsilon))$  sample complexity and thus is statistically efficient. A similar rate can also be obtained in stochastic dynamic settings, and we present the result in Appendix D.2. Note that our rate is slightly worse than the  $O(1 / N^{1 / 6})$  rate in Zhan et al. [2022], and worse than the optimal rate  $O(1 / \sqrt{N})$  in Rashidinejad et al. [2022]. We briefly discuss the intrinsic difficulty to derive an optimal convergence rate. First, we only require a realizability assumption on our function class, while Rashidinejad et al. [2022] requires a much stronger completeness assumption. Second, our optimization procedure is uninterleaved and only requires solving regression problems, while Zhan et al. [2022] and Rashidinejad et al. [2022] require solving minimax problems. Finally, Rashidinejad et al. [2022] assumes that the behavior policy is known and directly computes the policy using the knowledge of behavior policy, while our algorithm uses a more practical method, i.e., MLE, to solve the policy in the policy learning step.
We also compare our theoretical results to Ma et al. [2022c]. Theorem 4.1 of Ma et al. [2022c] provides a finite-sample guarantee for the suboptimality. However, they compare the performance of  $\hat{\pi}$  to  $\pi_{\alpha}^{*}$  (with  $\alpha = 1$ ) instead of  $\pi^{*}$ . Since the performance gap between  $\pi^{*}$  and  $\pi_{\alpha}^{*}$  can be as large as a constant when  $\alpha = 1$ , even zero suboptimality (compared to  $\pi_{\alpha}^{*}$ ) cannot imply that the learned policy has good performance. Moreover, their theoretical analysis assumes that  $V^{*}$  can be learned with zero error, which is unreasonable in practical scenarios. We also note that they only provide a proof for deterministic policy classes, which can be restrictive in practice.
In this section, we provide experimental results of our VP-learning algorithm with different choices of  $\alpha$  under five different environments: FetchReach, FetchPick, FetchPush, FetchSlide, and HandReach [Plappert et al., 2018]. Similar to Ma et al. [2022c], the datasets for the five tasks are from Yang et al. [2022]. All the implementation details of our VP-learning are the same as GoFAR (see dataset details and implementation details in Ma et al. [2022c])², except for the value of  $\alpha$ . Note that our VP-learning algorithm with  $\alpha = 1$  is equivalent to the GoFAR algorithm. Table 1 presents the discounted returns and Table 2 presents the final distances of the policies trained after 100 epochs and evaluated over 10 runs. For each environment and each  $\alpha$ , the result was averaged over 3 random seeds. The best results of each environment are in bold.
Table 1: Discounted return of different choices of  $\alpha$  ,averaged over 3 random seeds.
α\Env	FetchReach	FetchPick	FetchPush	FetchSlide	HandReach
0.01	27.4 ± 0.29	18.5 ± 0.1	18.0 ± 1.8	2.36 ± 1.13	8.72 ± 1.69
0.02	27.4 ± 0.32	18.7 ± 1.8	18.6 ± 2.6	2.40 ± 0.47	7.96 ± 1.27
0.05	27.4 ± 0.32	17.3 ± 1.1	19.3 ± 2.0	3.18 ± 0.90	8.98 ± 3.11
0.1	27.4 ± 0.33	20.3 ± 1.3	20.3 ± 2.5	3.22 ± 0.38	5.28 ± 1.25
0.2	27.4 ± 0.32	20.7 ± 0.9	17.7 ± 2.9	2.25 ± 0.23	2.92 ± 0.98
0.5	27.5 ± 0.29	18.5 ± 0.4	20.1 ± 2.2	3.47 ± 1.08	5.74 ± 2.72
1	27.3 ± 0.34	18.2 ± 1.2	19.6 ± 1.6	2.75 ± 1.84	7.13 ± 3.60
2	27.4 ± 0.29	18.3 ± 0.7	19.6 ± 1.4	1.80 ± 0.66	3.99 ± 1.88
Table 2: Final distance of different choices of  $\alpha$  ,averaged over 3 random seeds.
α\Env	FetchReach	FetchPick	FetchPush	FetchSlide	HandReach
0.01	0.0171 ± 0.0017	0.042 ± 0.004	0.033 ± 0.001	0.1177 ± 0.012	0.0269 ± 0.0049
0.02	0.0168 ± 0.0016	0.045 ± 0.012	0.031 ± 0.002	0.1085 ± 0.010	0.0274 ± 0.0049
0.05	0.0181 ± 0.0011	0.052 ± 0.013	0.032 ± 0.002	0.1061 ± 0.009	0.0270 ± 0.0049
0.1	0.0173 ± 0.0014	0.032 ± 0.010	0.027 ± 0.002	0.1018 ± 0.002	0.0275 ± 0.0043
0.2	0.0172 ± 0.0019	0.031 ± 0.004	0.031 ± 0.003	0.1029 ± 0.010	0.0275 ± 0.0046
0.5	0.0166 ± 0.0011	0.044 ± 0.009	0.031 ± 0.005	0.1017 ± 0.017	0.026826 ± 0.0049
1	0.0175 ± 0.0013	0.043 ± 0.011	0.043 ± 0.012	0.1202 ± 0.019	0.026828 ± 0.0044
2	0.0171 ± 0.0011	0.034 ± 0.005	0.032 ± 0.001	0.1044 ± 0.011	0.0275 ± 0.0045
The empirical results demonstrate the correctness of our theoretical analysis: choosing  $\alpha = 1$  will result in a large suboptimality of  $\pi_{\alpha}^{*}$  and thus the learned policy  $\hat{\pi}$ . Instead, we should carefully choose the value of  $\alpha$  to ensure a vanishing suboptimality. In practice, we can tune the value of  $\alpha$  and typically it is less than one. In our experiments, the best  $\alpha$  ranges over [0.05, 0.5].
In this paper, we theoretically analyze the VP-learning algorithm (Algorithm 1, which is based on the previous empirically successful algorithm in Ma et al. [2022c]) for both single-task and goal-conditioned offline settings. This algorithm can deal with general value function approximation and only requires near minimal assumptions on the dataset (single-policy concentrability) and function class (realizability). We also provide an  $O(1 / N^{1 / 12})$  upper bound of the suboptimality of the policy learned by the algorithm and empirically validate its effectiveness.
As for future directions, one important question is whether we can achieve the optimal suboptimality rate  $\tilde{O}(1/\sqrt{N})$  while keeping the algorithm practical without unreasonably strong assumptions.
2We use the code at https://github.com/JasonMa2016/GoFAR with different values of  $\alpha$  for our experiments.
We thank the anonymous reviewer for catching a technical issue in a previous version of our paper. The work was done when HZ was a visiting researcher at Meta.
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pages 10-4, 2019.
Andras Antos, Rémi Munos, and Csaba Szepesvari. Fitted Q-iteration in continuous action-space mdps. In Neural Information Processing Systems, 2007.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71 (1):89-129, 2008.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07749, 2021.
Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. arXiv preprint arXiv:1905.00360, 2019.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarily trained actor critic for offline reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 3852-3878. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/cheng22b.html.
Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. The complexity of constrained min-max optimization. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 1466-1478, 2021.
Amir Massoud Farahmand, Rémi Munos, and Csaba Szepesvári. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems, 2010.
Yihao Feng, Lihong Li, and Qiang Liu. A kernel loss for solving the Bellman equation. arXiv preprint arXiv:1905.10506, 2019.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088, 2019.
Nan Jiang. On value functions and the agent-environment boundary. arXiv preprint arXiv:1905.13341, 2019.
Nan Jiang and Jiawei Huang. Minimax value interval for off-policy evaluation and policy optimization. Advances in Neural Information Processing Systems, 33, 2020.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pages 1094-8. CiteSeer, 1993.
Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2021.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.