SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Compared with Xie et al. [2021] when the function class is finite. In this case, the analysis of the VS-based algorithms and RO-based algorithms of Xie et al. [2021] give the bounds that in our setting can be translated $^7$  into:  $Hb\sqrt{\max_h\ln(|\mathcal{F}_h||\Pi_h^{all}|)\cdot C_2(\pi) / K}$  and  $Hb\sqrt{C_2(\pi)}\sqrt[3]{\max_h\ln(|\mathcal{F}_h||\Pi_h^{soft}(T)|) / K} + Hb / \sqrt{T}$ , respectively, where  $C_2(\pi) := \max_{h\in [H],\tilde{\pi}\in\Pi^{all},f\in\mathcal{F}}\frac{\|fh-\mathbb{T}_h^{\tilde{\pi}}fh_{+1}\|_2^2}{\|fh-\mathbb{T}_h^{\tilde{\pi}}fh_{+1}\|_2^2,\mu}$ . Instead, our bounds for both the VS-based and RO-based algorithms are  $Hb\sqrt{\max_h\ln(|\mathcal{F}_h||\Pi_h^{soft}(T)|)\cdot\mathcal{C}(\pi;1/\sqrt{K})/K} + Hb / \sqrt{T}$ . We improve upon the results of Xie et al. [2021] on several fronts. First, our diversity measures  $\mathcal{C}(\pi;1/K)$  is always smaller than their measure  $C_2(\pi)$ , since  $\mathcal{C}(\pi;1/\sqrt{K}) \leq \mathcal{C}(\pi;0) \leq C_2(\pi)$ . Second, for the VS-based algorithm,  $\Pi^{soft}(T) \subset \Pi^{all}, \forall T$ , our bound is always tighter. In fact,  $|\Pi^{all}|$  is arbitrarily large that bounds depending on this quantity is vacuous. Third, for the RO-based algorithm, the rates in terms of  $K$  in the bound of Xie et al. [2021] are slower than that in our bound. Specifically, if  $\Pi_h^{soft}(T) = \tilde{\mathcal{O}}_T(1)$ , then these rates are  $K^{-1/3}$  vs  $K^{-1/2}$  (with an optimal choice of  $T = K$  for both bounds). If we consider the worst case that  $\Pi_h^{soft}(T) = \mathcal{O}(T\log|\mathcal{F}_h|)$ , then these rates are  $K^{-1/5}$  vs  $K^{-1/4}$  (with an optimal choice of  $T = K^{2/5}$  and  $T = \sqrt{K}$  in the respective bounds). Finally, our results hold under the general adaptively collected data rather than their independent episode setting. We summarize the bounds in the finite function class cases in Table 1, and give comparisons for the linear model cases in Table 2.
Compared with LCB-based algorithms. When  $\mathcal{F}_h$  is a  $d$ -dimensional linear model with feature maps  $\{\phi_h\}_{h\in [H]}$ , our bounds reduce into  $Hb\sqrt{d\cdot K^{-1}\cdot\mathcal{C}(\pi;1 / \sqrt{K})}$  (Proposition 1), which matches the order of (and potentially tighter than) the bound in Zanette et al. [2021], since  $\mathcal{C}(\pi ;1 / \sqrt{K})$  is always smaller (or at least equal to) than the relative condition number  $\max_h\sup_{x\in \mathbb{R}^d}\frac{x^T\mathbb{E}_\pi[\phi_h(s_h,a_h)\phi_h(s_h,a_h)^T]x}{x^T\mathbb{E}_\mu[\phi_h(s_h,a_h)\phi_h(s_h,a_h)^T]x}$ . Compared with the bound of LCB-based algorithms in Jin et al. [2021b], we improve a factor  $\sqrt{d}$  and holds under the more general Assumption 2.2 which includes low-rank MDPs. In a more refined analysis [Xiong et al., 2023], the LCB-based algorithm obtains the same dependence on  $d$  for low-rank MDPs as our guarantees. However, this improvement relies on a uniform coverage assumption, i.e.,  $\min_{h\in [H]}\lambda_{\min}\left(\mathbb{E}_{(s_h,a_h)\sim d_h^\mu}\left[\phi_h(s_h,a_h)\phi_h(s_h,a_h)^T\right]\right) > 0$ , which we do not require. Di et al. [2023] generalize the results of Xiong et al. [2023] from linear MDPs to MDPs with general function approximation. However, they still rely on a uniform coverage assumption. Finally note that, for VS-based and RO-based algorithms, we provide high-probability bounds for a smoothing version of  $\hat{\pi}$  over the randomization of the algorithms, not for  $\hat{\pi}$  itself.
Compared with model-based PS. Uehara and Sun [2022] consider model-based PS for offline RL, where they obtain the Bayesian sub-optimality bound of  $H^2 \sqrt{C^{\text{Bayes}} \cdot \ln |\mathcal{M}| / K}$  where  $C^{\text{Bayes}}$  is the Bayesian version of a relative condition number and  $\mathcal{M}$  is a finite model class. Two key distinctions are that our method in Algorithm 4 is model-free, and our achieved bound is in the frequentist (i.e., worst-case) nature, which is a stronger result than the Bayesian bound of the same order.
We contributed to the understanding of sample-efficient offline RL in the context of (value) function approximation. We proposed a notion of data diversity that generalizes the previous data coverage measures and importantly expands the class of sample-efficient offline RL. We studied three different algorithms: VS, RO, and PS, where the PS-based algorithm is our novel proposal. We showed that VS, RO, and PS all have same-order guarantees under standard assumptions.
$^{7}$ Xie et al. [2021] consider discounted MDP and a restricted policy class for the comparator class.
Alekh Agarwal and Tong Zhang. Non-linear reinforcement learning in large action spaces: Structural conditions and sample-efficiency of posterior sampling. In Conference on Learning Theory, pages 2776-2814. PMLR, 2022. 2, 3, 7
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. J. Mach. Learn. Res., 22(98): 1-76, 2021. 2, 8, 9
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. In Gábor Lugosi and Hans Ulrich Simon, editors, Learning Theory, 19th Annual Conference on Learning Theory, COLT 2006, Pittsburgh, PA, USA, June 22-25, 2006, Proceedings, volume 4005 of Lecture Notes in Computer Science, pages 574-588. Springer, 2006. 1
Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a meta-algorithm and applications. Theory of Computing, 8(6):121-164, 2012. doi: 10.4086/toc.2012.v008a006.6
Peter L Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. The Annals of Statistics, 33(4):1497-1537, 2005. 35
Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, pages 1042-1051. PMLR, 2019. 2
Jinglin Chen and Nan Jiang. Offline reinforcement learning under value and density-ratio realizability: the power of gaps. In Uncertainty in Artificial Intelligence, pages 378-388. PMLR, 2022. 2, 5, 18
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarily trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3852-3878. PMLR, 2022. 6, 18
Christoph Dann, Mehryar Mohri, Tong Zhang, and Julian Zimmert. A provably efficient model-free posterior sampling method for episodic reinforcement learning. In Advances in Neural Information Processing Systems, 2021. 2, 3, 5, 6, 7, 21, 29, 30, 31, 34, 37
Qiwei Di, Heyang Zhao, Jiafan He, and Quanquan Gu. Pessimistic nonlinear least-squares value iteration for offline reinforcement learning. arXiv preprint arXiv:2310.01380, 2023. 9, 10
Yaqi Duan, Zeyu Jia, and Mengdi Wang. Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning, pages 2701-2709. PMLR, 2020. 4, 9, 17
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005. 1
Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487, 2021. 3
David A. Freedman. On tail probabilities for martingales. The Annals of Probability, 3(1):100-118, 1975. ISSN 00911798. 19
Germano Gabbianelli, Gergely Neu, Nneka Okolo, and Matteo Papini. Offline primal-dual reinforcement learning for linear mdps. arXiv preprint arXiv:2305.12944, 2023. 18
Ruohan Zhan, Zhimei Ren, Susan Athey, and Zhengyuan Zhou. Policy learning with adaptively collected data. Management Science, 2023. 4
Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 2730-2775. PMLR, 02-05 Jul 2022. 5, 9, 18
Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science, 4(2):834-857, 2022. 2, 3, 7
Tong Zhang. Mathematical Analysis of Machine Learning Algorithms. Cambridge University Press, 2023. 5
Yulai Zhao, Zhuoran Yang, Zhaoran Wang, and Jason D Lee. Local optimization achieves global optimality in multi-agent reinforcement learning. arXiv preprint arXiv:2305.04819, 2023. 9
Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. A posterior sampling framework for interactive decision making. arXiv preprint arXiv:2211.01962, 2022. 2, 3
Hanlin Zhu, Paria Rashidinejad, and Jiantao Jiao. Importance weighted actor-critic for optimal conservative offline reinforcement learning. arXiv preprint arXiv:2301.12714, 2023. 18
1 Introduction 1
2 Background and Problem Formulation 3
2.1 Episodic Time-inhomogenous Markov Decision Process 3
2.2 Offline Data Generation 4
2.3 Policy and function classes 4
2.4 Effective sizes of policy and function classes 5
3 Algorithms 6
4 Main Results 7
4.1 Data diversity 7
4.2 Offline learning guarantees 8
Appendices 17
Appendix A Extended Discussion 17
A.1 Linear function classes 17
A.2 Comparison with primal-dual methods for offline RL 18
Appendix B Preparation 19
B.1 Variance condition and Bernstein's inequality 19
B.2 Functional projections for misspecification 20
B.3 Induced MDPs 21
B.4 Error decomposition 21
B.5 Decoupling lemma 22
B.6 Regret of the multiplicative weights algorithm for the actors 22
Appendix C Proof of Theorem 1 24
C.1 Proof of Theorem 1 25
C.2 Proof of Lemma C.1 25
C.3 Proof of Lemma C.2 26
Appendix D Proof of Theorem 2 28
Appendix E Proof of Theorem 3 28
E.1 Generalized form of posterior and Proposition 2 29
E.2 Proof of Theorem 3 30
E.3 Proof of Proposition 2 30
E.3.1 Lower-bounding log-partition function. 31
E.3.2 Upper-bounding log-partition function 37
E.3.3 Proof of Proposition 2 40
Appendix F Proof of Proposition 1 40
Appendix G Support Lemmas 42
Appendix A Extended Discussion
We also extend the discussion of our data diversity in comparison to the existing distribution mismatch measures in the special case where each  $\mathcal{F}_h$  is a linear function in a known feature map.
A.1 Linear function classes
In this section, we consider the linear model cases, where there are known feature maps  $\phi_h: \mathcal{X} \times \mathcal{A} \to \mathbb{R}^d$  and w.l.o.g.  $\max_{h \in [H]} \| \phi(\cdot, \cdot) \|_{\infty} \leq 1$ , such that  $\mathcal{F}_h = \{\phi_h(\cdot, \cdot)^T w : w \in \mathbb{R}^d, \|w\|_2^2 \leq b\}$ . Recall that, in this case, e.g., it follows from [Zanette et al., 2021, Lemma 6], that we have
$$
\log N (\epsilon ; \mathcal {F} _ {h}, \| \cdot \| _ {\infty}) \leq d \log (1 + \frac {2}{\epsilon}),
$$
$$
\log N (\epsilon ; \Pi_ {h} ^ {\text {s o f t}} (T), \| \cdot \| _ {\infty}) \leq d \log \left(1 + \frac {1 6 b T}{\epsilon}\right).
$$
Thus, our bounds from Proposition 1 can simplified as
$$
\begin{array}{l} \tilde {\mathcal {O}} (H b \sqrt {d \log (1 + 1 6 b K T) \cdot \mathcal {C} (\pi ; 1 / \sqrt {K}) / K} + H b \sqrt {T ^ {- 1} \ln \mathrm {V o l} (\mathcal {A}))} \\ = \tilde {\mathcal {O}} \left(H b \sqrt {\frac {\operatorname* {m a x} \{d \mathcal {C} (\pi ; 1 / \sqrt {K}) , \ln \operatorname {V o l} (\mathcal {A}) \}}{K}}\right) \\ \end{array}
$$
where we choose  $T = K$ . To simplify the comparison, we assume that  $d\mathcal{C}(\pi ;1 / \sqrt{K})\geq \ln \operatorname {Vol}(\mathcal{A})$
Let us now compute various notions of data coverage in this linear model case. We first need to define the following quantities (various forms of covariance matrices).
$$
\Sigma_ {h} := \lambda I + \sum_ {k = 1} ^ {K} \phi_ {h} \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \phi_ {h} \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {T},
$$
$$
\Lambda_ {h} := \lambda I + \sum_ {k = 1} ^ {K} \phi_ {h} (s _ {h} ^ {k}, a _ {h} ^ {k}) \phi_ {h} (s _ {h} ^ {k}, a _ {h} ^ {k}) ^ {T} / [ \mathbb {V} _ {h} V _ {h + 1} ^ {\pi} ] (s _ {h} ^ {k}, a _ {h} ^ {k}),
$$
$$
\bar {\phi} _ {h} ^ {\pi} := \mathbb {E} _ {\pi} [ \phi_ {h} (s _ {h}, a _ {h}) ],
$$
$$
\bar {\Sigma} _ {h} := \mathbb {E} _ {\mu} \left[ \phi \left(s _ {h}, a _ {h}\right) \phi \left(s _ {h}, a _ {h}\right) ^ {T} \right].
$$
We define the following distribution mismatch quantities, which were used in the literature.
$$
C _ {p e v i} (\pi) := \max _ {h \in [ H ]} \left(\mathbb {E} _ {\pi} \left[ \| \phi_ {h} (s _ {h}, a _ {h}) \| _ {\Sigma_ {h} ^ {- 1}} \right]\right) ^ {2},
$$
$$
C _ {p e v i - a d v} (\pi) := \max _ {h \in [ H ]} \left(\mathbb {E} _ {\pi} \left[ \| \phi_ {h} (s _ {h}, a _ {h}) \| _ {\Lambda_ {h} ^ {- 1}} \right]\right) ^ {2},
$$
$$
C _ {p a c l e} (\pi) := \max  _ {h \in [ H ]} \| \bar {\phi} _ {h} ^ {\pi} \| _ {\Sigma_ {h}} ^ {2 - 1},
$$
$$
C _ {b c p} (\pi) := \max  _ {h \in [ H ]} \left(\mathbb {E} _ {\pi} \left[ \| \phi_ {h} (s _ {h}, a _ {h}) \| _ {\bar {\Sigma} _ {h}} \right]\right) ^ {2}.
$$
The sub-optimality bounds of various methods are summarized in Table 2. For comparing our data diversity measure with different notions of distribution mismatch, we have
$$
C _ {p e v i} (\pi) \geq C _ {p a c l e} (\pi) \approx \mathcal {C} (\pi ; 0) / K \leq C _ {b c p} (\pi) / K.
$$
where the “ $\approx$ ” denotes that the involved terms scale in the same order and can be implied by Fredman's matrix inequality (see [Duan et al., 2020, Lemma B.5]) (under additional conditions). Note that  $\mathcal{C}(\pi;1/\sqrt{K}) \leq \mathcal{C}(\pi;0)$ , thus our data diversity is the tightest quantity among all that are considered.
Algorithm	Sub-optimality bound
PEVI [Jin et al., 2021b]	Hb√Cpevi(π)·d
PEVI-ADV+ [Xiong et al., 2023]	H√Cpevi-avi(π)·d
PACLE [Zanette et al., 2021]	Hb√Cpackle(π)·d
VC [Xie et al., 2021, Section 3]	Hb√Cbcp(π)·d/K
RO [Xie et al., 2021, Section 4]	Hb√Cbcp(π) √3d/K
Ours (VS, RO, PS)	Hb√C(π; 1/√K)·d/K
Table 2: Sub-optimality bounds when the function class  ${\mathcal{F}}_{h}$  is linear in  ${\phi }_{h} : S \times  \mathcal{A} \rightarrow  {\mathbb{R}}^{d}$  .
Note that the data coverage measure in Xiong et al. [2023], roughly speaking, can be bounded as follows:
$$
C _ {p e v i - a d v} (\pi) \leq b ^ {2} C _ {p e v i} (\pi),
$$
where we use the inequality  $[\mathbb{V}_h V_{h+1}^\pi](s_h^k, a_h^k) \leq b^2$ . Thus the bound of Xiong et al. [2023] in general has a tighter dependence on  $b$  (which implicitly depends on  $H$ ) than all the bounds of all other works considered in Table 2, due to that Xiong et al. [2023] incorporated the variance information into the estimation via the variance-weighted value iteration algorithm. However, obtaining this improved bound in Xiong et al. [2023] relies on a uniform coverage assumption which we do not require.
A.2 Comparison with primal-dual methods for offline RL
As opposed to the value-based methods we considered in our paper, an important alternative approach to offline RL is the primal-dual methods [Zhan et al., 2022, Chen and Jiang, 2022, Rashidinejad et al., 2023, Gabbianelli et al., 2023, Ozdaglar et al., 2023]. However, the guarantees of primal-dual methods use a different set of assumptions than the value-based methods we consider (the former assumes realizability for the ratio between the state-action occupancy density of the target policy and the state-action occupancy density of the behavior policy, except for Gabbianelli et al. [2023] where this realizability assumption is implicitly encoded under a stronger assumption of linear MDP). This makes the results presented in our paper and the results in the primal-dual methods not directly comparable.
Since the work of Gabbianelli et al. [2023] considers linear MDPs, it is more comparable (than the other primal-dual methods we mentioned) to the instantiating of our results to the linear function class. Gabbianelli et al. [2023] consider primal-dual methods for offline RL in both infinite-horizon discounted MDP and average-reward MDP. Our analysis framework for the regularized optimization method in the episodic MDP should work for the infinite-horizon discounted MDP as well, where the regularized optimization achieves the optimal sample complexity of  $\mathcal{O}(\epsilon^{-2})$  while the sample complexity in Gabbianelli et al. [2023] in this setting is  $\mathcal{O}(\epsilon^{-4})$ . However, Gabbianelli et al. [2023] offers a better computational complexity  $(\mathcal{O}(K)$  vs  $\mathcal{O}(K^{7/5}))$  and also works in the average-reward MDP setting which is beyond the episodic MDP setting considered in our work; though our bounds hold for general function approximation that is beyond the strong assumption of linear MDPs.
The concurrent work of Zhu et al. [2023] combined the actor-critic framework with marginalized importance sampling (MIS) for an RO-based algorithm, which also improves the sub-optimal rate of order  $1 / K^{1 / 3}$  by Xie et al. [2021], Cheng et al. [2022] to the optimal rate of order  $1 / \sqrt{K}$ . Instead, we obtain the optimal rate of order  $1 / \sqrt{K}$  with a refined analysis for a standard RO-based algorithm. That is, unlike Zhu et al. [2023], we do not use MIS; consequently, we do not require the realizability assumption for the ratio between the state-action occupancy density of the target policy and that of the behavior policy.
Appendix B Preparation
We now get into more involved parts where we present the proof process and the technical results for obtaining Theorem 1, Theorem 2, and Theorem 3. In order to prove our main results in Section 4, we shall need some old tools and develop some new useful tools. For convenience, we start out with both old and new notations of quantities summarized in Table 3 that we are going to use frequently in our proofs.
Name	Notation	Expression
transition sample	zk	(s^k_h, a^k_h, r^k_h, a^k_{h+1})
transition sample	zh	(sh, ah, rh, sh+1)
Bellman error	E^π(h, fh+1)(sh, ah)	(T^π_h f_{h+1} - f_h)(sh, ah)
TD loss function	l_π(f_h, fh+1; zh)	(f_h(sh, ah) - rh - fh+1(sh+1, π))^2
empirical squared Bellman error (SBE)	L_π(f_h, fh+1)	Σ^K_{k=1} l_π(f_h, fh+1; z^k_h)
empirical bias-adjusted SBE	L_π(f)	Σ^H_{h=1} L_π(f_h, fh+1) - inf_{g\in\mathcal{F}} \sum_{h=1}^{H} L_π(g_h, fh+1)
excess TD loss	ΔL_π(f_h, fh+1; zh)	l_π(f_h, fh+1; zh) - l_π(T^π_h f_{h+1}, fh+1; zh)
-	Eμ[·]	1/K ∑_{k=1}^{K} E_{μ^k} [·]
-	E_k[·](:= E_{μ^k} [·])	E[·|{z^i_h}^i∈[k-1]h∈[H]]
Table 3: A summary of notations and quantities of interest.
The quantity  $l_{\tilde{\pi}}(f_h, f_{h+1}; z_h)$  can be viewed as a temporal difference (TD) loss function defined on data point  $z_h$  conditioned on each  $f_{h+1}$  and  $\tilde{\pi}$ . The quantity  $\mathbb{T}_h^{\tilde{\pi}} f_{h+1}$  can be viewed as the Bellman regression function, where, conditioned on each  $f_{h+1}$  and  $\tilde{\pi}$ , for any  $(s_h, a_h)$ , we have
$$
\mathbb {T} _ {h} ^ {\tilde {\pi}} f _ {h + 1} (s _ {h}, a _ {h}) = \mathbb {E} _ {r _ {h}, s _ {h + 1} | s _ {h}, a _ {h}} [ r _ {h} + f _ {h + 1} (s _ {h + 1}, \tilde {\pi}) ] = \underset {g} {\arg \inf} \mathbb {E} _ {r _ {h}, s _ {h + 1} | s _ {h}, a _ {h}} l _ {\tilde {\pi}} (g, f _ {h + 1}; z _ {h}).
$$
Thus, the quantity  $\Delta L_{\tilde{\pi}}(f_h,f_{h + 1};z_h)$  can be referred to as the excess TD loss, incurred by the predictor  $f_{h}$ , relative to the TD regression function  $\mathbb{T}_h^{\tilde{\pi}}f_{h + 1}$ , on data  $z_{h}$  and conditioned on  $f_{h + 1}$  and  $\tilde{\pi}$ .
B.1 Variance condition and Bernstein's inequality
We also define the  $\sigma$ -algebra  $\mathcal{A}_h^k \coloneqq \sigma(\mathcal{D}_{k-1} \cup \{(s_{h'}^k, a_{h'}^k, r_{h'}^k)\}_{h' \in [h-1]} \cup (s_h^k, a_h^k))$  and denote  $\mathbb{E}_{k,h}[\cdot] \coloneqq \mathbb{E}[\cdot|\mathcal{A}_h^k]$ . The following lemma establishes the variance condition on the excess TD loss, a TD analogous to the variance condition that is widely used in the empirical process theory [Massart, 2000].
Lemma B.1. For any  $\mathcal{A}_h^k$ -measurable policy  $\pi$ , we have
$$
\mathbb {E} _ {k, h} \left[ \Delta L _ {\pi} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right) \right] = \mathcal {E} _ {h} ^ {\pi} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {2},
$$
$$
\mathbb {E} _ {k, h} \left[ \Delta L _ {\pi} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right) ^ {2} \right] \leq 3 6 b ^ {2} \mathcal {E} _ {h} ^ {\pi} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {2}.
$$
Proof of Lemma B.1. The result directly exploits the boundedness of the TD loss function and the squared loss is Lipschitz. In concrete, it is a direct application of Lemma G.1.  $\square$
The following lemma establishes the martingale extension of Bernstein's inequality, typically called Freedman's inequality [Freedman, 1975]. In this lemma, we prove a slightly modified version of the original Freedman's inequality for our own convenience. The proof for this lemma is elementary which we also show here.
Lemma B.2 (Freedman's inequality). Let  $X_{1}, \ldots, X_{T}$  be any sequence of real-valued random variables. Denote  $\mathbb{E}_t[\cdot] = \mathbb{E}[|\boldsymbol{X}_1, \ldots, \boldsymbol{X}_{t-1}]$ . Assume that  $X_{t} \leq R$  for some  $R > 0$  and  $\mathbb{E}_t[X_t] = 0$  for all  $t$ . Define the random variables
$$
S := \sum_ {t = 1} ^ {T} X _ {t}, \quad V := \sum_ {i = 1} ^ {T} \mathbb {E} _ {t} \left[ X _ {t} ^ {2} \right].
$$
Then for any  $\delta > 0$ , with probability at least  $1 - \delta$ , for any  $\lambda \in [0,1 / R]$
$$
S \leq (e - 2) \lambda V + \frac {\ln (1 / \delta)}{\lambda}.
$$
Proof of Lemma B.2. Let us define the following sequence of random variables:  $Z_0 = 1, Z_t = Z_{t-1} \frac{e^{\lambda X_t}}{\mathbb{E}_t [e^{\lambda X_t}]}$ . We have
$$
\mathbb {E} _ {t} [ Z _ {t} ] = \mathbb {E} _ {t} \left[ Z _ {t - 1} \frac {e ^ {\lambda X _ {t}}}{\mathbb {E} _ {t} [ e ^ {\lambda X _ {t}} ]} \right] = \frac {Z _ {t - 1}}{\mathbb {E} _ {t} [ e ^ {\lambda X _ {t}} ]} \mathbb {E} _ {t} [ e ^ {\lambda X _ {t}} ] = Z _ {t - 1}.
$$
$$
\mathbb {E} [ Z _ {T} ] = \mathbb {E} \mathbb {E} _ {T} [ Z _ {T} ] = \mathbb {E} [ Z _ {T - 1} ] = \dots = \mathbb {E} [ Z _ {0} ] = 1.
$$
$$
Z _ {T} = \frac {e ^ {\lambda S}}{\prod_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ e ^ {\lambda X _ {t}} \right]} = \frac {e ^ {\lambda S}}{\sum_ {t = 1} ^ {T} e ^ {\ln \mathbb {E} _ {t} \left[ e ^ {\lambda X _ {t}} \right]}} = \exp \left(\lambda S - \sum_ {t = 1} ^ {T} \ln \mathbb {E} _ {t} \left[ e ^ {\lambda X _ {t}} \right]\right). \tag {3}
$$
Since  $Z_{T} \geq 0$ , it follows from Markov's inequality that, for any  $\delta > 0$ , we have
$$
\Pr \left(Z _ {T} \geq 1 / \delta\right) \leq \delta \mathbb {E} \left[ Z _ {T} \right] = \delta . \tag {4}
$$
We now bound the logarithmic moment generating function  $\ln \mathbb{E}_t[e^{\lambda X_t}]$  using elementary inequalities: For any  $\lambda \in [0,1 / R]$ , we have
$$
\ln \mathbb {E} _ {t} \left[ e ^ {\lambda X _ {t}} \right] \leq \mathbb {E} _ {t} \left[ e ^ {\lambda X _ {t}} \right] - 1 \leq \lambda \mathbb {E} _ {t} \left[ X _ {t} \right] + (e - 2) \lambda^ {2} \mathbb {E} _ {t} \left[ X _ {t} ^ {2} \right], \tag {5}
$$
where the first inequality uses  $\ln z\leq z - 1,\forall z\geq 0$  and the second inequality uses that  $e^z\leq 1 + z + (e - 2)z^2,\forall z\leq 1$  and that  $\lambda X_{t}\leq 1$
Plugging Equation (5) into Equation (3), then all together into Equation (4) complete the proof.
B.2 Functional projections for misspecification
Since Assumption 2.1 and Assumption 2.2 allow misspecification up to some errors  $\xi$  and  $\nu$ , while we are working on the function class  $\mathcal{F}$ , we rely on the following projection operators, Definition 4 and Definition 5, to handle misspecification.
Definition 4 (Projection of action-value functions). For any  $\tilde{\pi} \in \Pi^{soft}(T)$  for some  $T \in \mathbb{N}$ , we define the projection of the state-action value function  $\tilde{\pi}$  onto  $\mathcal{F}$  as