SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Estimating  $d^{\pi}$  for any policy  $\pi$ . In this step, we would like to sample the environment and collect a set of sample trajectories, in a way that allows for reasonable estimation of the occupancy distribution  $d^{\pi}$  induced by any policy  $\pi$ . For this purpose, we invoke the exploration strategy and the accompanying estimation scheme developed in Li et al. (2023). Working forward (i.e., from  $h = 1$  to  $H$ ), this approach collects, for each step  $h$ , a set of  $N$  sample trajectories in order to facilitate estimation of the occupancy distributions, which amounts to a total number of
$$
N H =: K _ {\text {p r e p a r e}} ^ {\text {o n}} = K ^ {\text {o n}} / 3 \tag {14}
$$
sample trajectories collected in this stage. See Algorithm 3 in Appendix D.1 for a precise description of this strategy. Noteworthy, while Algorithm 3 specifies how to estimate  $\widehat{d^{\pi}}$  for any policy  $\pi$ , we won't need to compute it explicitly unless this policy  $\pi$  is encountered during the subsequent steps of the algorithm; in other words,  $\widehat{d^{\pi}}$  should be viewed as a sort of "function handle" that will only be executed when called later.
Estimating  $d^{\mathrm{off}}$  for the historical dataset  $\mathcal{D}^{\mathrm{off}}$ . In addition, we are in need of estimating the occupancy distribution  $d^{\mathrm{off}}$ . Towards this end, we propose the following empirical estimate using the  $K^{\mathrm{off}} / 2$  sample trajectories from  $\mathcal{D}^{\mathrm{off},1}$ :
$$
\widehat {d} _ {h} ^ {\text {o f f}} (s, a) = \frac {2 N _ {h} ^ {\text {o f f}} (s , a)}{K ^ {\text {o f f}}} 1 1 \left(\frac {N _ {h} ^ {\text {o f f}} (s , a)}{K ^ {\text {o f f}}} \geq c _ {\text {o f f}} \left\{\frac {\log \frac {H S A}{\delta}}{K ^ {\text {o f f}}} + \frac {H ^ {4} S ^ {4} A ^ {4} \log \frac {H S A}{\delta}}{N} + \frac {S A}{K ^ {\text {o n}}} \right\}\right) \tag {15}
$$
for all  $(s, a) \in S \times \mathcal{A}$ , where  $c_{\mathrm{off}} > 0$  is some universal constant. Here,  $1 - \delta$  indicates the target success probability, and
$$
N _ {h} ^ {\text {o f f}} (s, a) = \sum_ {k = 1} ^ {K ^ {\text {o f f}} / 2} \mathbb {1} \left(s _ {h} ^ {k, \text {o f f}} = s, a _ {h} ^ {k, \text {o f f}} = a\right), \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}. \tag {16}
$$
In other words,  $\widehat{d}_h^{\mathrm{off}}(s,a)$  is taken to be the empirical visitation frequency of  $(s,a)$  in  $\mathcal{D}^{\mathrm{off},1}$  if  $(s,a)$  is adequately visited, and zero otherwise. The cutoff threshold will be made clear in our analysis.
Stage 2: online exploration. Armed with the above estimates of the occupancy distributions, we can readily proceed to compute the desired exploration policies and sample the environment. We seek to devise two exploration strategies, with one strategy selected to imitate the offline dataset, and the other one employed to explore the insufficiently visited territory. As a preliminary fact, if we have a dataset containing  $K$  independent trajectories — generated independently from a mixture of deterministic policies with occupancy distribution  $d^{\mathrm{b}}$  — then it has been shown previously (see, e.g., Li et al. (2023, Section 3.3)) that the model-based offline approach is able to compute a policy  $\widehat{\pi}$  obeying
$$
V ^ {\star} (\rho) - V ^ {\widehat {\pi}} (\rho) \lesssim H \left[ \sum_ {h} \sum_ {s, a} \frac {d _ {h} ^ {\pi^ {\star}} (s , a)}{1 / H + K ^ {\mathsf {o n}} d _ {h} ^ {\mathsf {b}} (s , a)} \right] ^ {\frac {1}{2}}. \tag {17}
$$
This upper bound in (17) provides a guideline regarding how to design a sample-efficient exploration scheme.
Imitating the offline dataset. The offline dataset  $\mathcal{D}^{\mathrm{off}}$  is most informative when it contains expert data, a scenario when the data distribution resembles the distribution induced by the optimal policy  $\pi^{\star}$ . If this is the case, then it is desirable to find a policy similar to  $\pi^{\mathrm{off}}$  in (6) (the mixed policy generating  $\mathcal{D}^{\mathrm{off}}$ ) and employ it to collect new data, in order to retain and further strengthen the benefits of
such offline data. To do so, we attempt to approximate  $d^{\pi^*}$  by  $\widehat{d}^{\mathrm{off}}$  in (17) when attempting to minimize (17). In fact, we would like to compute a mixture of deterministic policies by (approximately) solving the following optimization problem:
$$
\mu^ {\text {i m i t a t e}} \approx \arg \min  _ {\mu \in \Delta (\Pi)} \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} \max  _ {a \in \mathcal {A}} \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right]}, \tag {18}
$$
which is clearly equivalent to
$$
\mu^ {\text {i m i t a t e}} \approx \arg \min  _ {\mu \in \Delta (\Pi)} \max  _ {\pi : \mathcal {S} \times [ H ] \rightarrow \Delta (\mathcal {A})} \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} \mathbb {E} _ {a \sim \pi_ {h} (\cdot | s)} \left[ \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right]} \right]. \tag {19}
$$
In order to solve this minimax problem (19) (note that its objective function is convex in  $\mu$ ), we resort to the Follow-The-Regularized-Leader (FTRL) strategy from the online learning literature (Shalev-Shwartz, 2012); more specifically, we perform the following updates iteratively for  $t = 1,\dots,T_{\max}$ :
$$
\pi_ {h} ^ {t + 1} (\cdot \mid s) \propto \exp \left(\eta \sum_ {k = 1} ^ {t} \frac {\widehat {d} _ {h} ^ {\mathrm {o f f}} (s , \cdot)}{\frac {1}{K ^ {\mathrm {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu^ {k}} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , \cdot) \right]}\right), \quad \forall s \in \mathcal {S}, \tag {20a}
$$
$$
\mu^ {t + 1} \approx \arg \min  _ {\mu \in \Delta (\Pi)} \sum_ {h = 1} ^ {H} \sum_ {s \in S} \mathbb {E} _ {a \sim \pi_ {h} ^ {t + 1} (\cdot | s)} \left[ \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right]} \right], \tag {20b}
$$
where  $\eta$  denotes the learning rate to be specified later. We shall discuss how to solve the optimization sub-problem (20b) in Appendix C. The output of this step is a mixture of deterministic policies taking the following form:
$$
\pi^ {\text {i m i t a t e}} = \mathbb {E} _ {\pi \sim \mu^ {\text {i m i t a t e}}} [ \pi ] \quad \text {w i t h} \quad \mu^ {\text {i m i t a t e}} = \frac {1}{T _ {\max }} \sum_ {t = 1} ^ {T _ {\max }} \mu^ {t}. \tag {21}
$$
Exploring the unknown environment. In addition to mimicking the behavior of the historical dataset, we shall also attempt to explore the environment in a way that complements pre-collected data. Towards this end, it suffices to invoke the reward-agnostic online exploration scheme proposed in Li et al. (2023), whose precise description will be provided in Algorithm 5 in Appendix D.2 to make the paper self-contained. The resulting policy mixture is denoted by
$$
\pi^ {\text {e x p l o r e}} = \mathbb {E} _ {\pi \sim \mu^ {\text {e x p l o r e}}} [ \pi ], \tag {22}
$$
with  $\mu^{\mathrm{explore}}\in \Delta (\Pi)$  representing the associated weight vector.
With the above two exploration policies (21) and (22) in place, we execute the MDP to obtain sample trajectories as follows:
1) Execute the MDP  $K_{\text{imitate}}^{\text{on}}$  times using policy  $\pi^{\text{imitate}}$  to obtain a dataset containing  $K_{\text{imitate}}^{\text{on}} = K^{\text{on}} / 3$  independent sample trajectories, denoted by  $\mathcal{D}_{\text{imitate}}^{\text{on}}$ ;
2) Execute the MDP  $K_{\text{explore}}^{\text{on}}$  times using policy  $\pi^{\text{explore}}$  to obtain a dataset containing  $K_{\text{explore}}^{\text{on}} = K^{\text{on}} / 3$  independent sample trajectories, denoted by  $\mathcal{D}_{\text{explore}}^{\text{on}}$ .
Stage 3: policy learning via offline RL. Once the above online exploration process is completed, we are positioned to compute a near-optimal policy on the basis of the data in hand. More precisely,
- Let us look at the following dataset
$$
\mathcal {D} = \mathcal {D} ^ {\text {o f f}, 2} \cup \mathcal {D} _ {\text {i m i t a t e}} ^ {\text {o n}} \cup \mathcal {D} _ {\text {e x p l o r e}} ^ {\text {o n}}. \tag {23}
$$
In light of the complicated statistical dependency between  $\mathcal{D}^{\mathrm{off},1}$  and  $\mathcal{D}_{\mathrm{imitate}}^{\mathrm{on}} \cup \mathcal{D}_{\mathrm{explore}}^{\mathrm{on}}$ , we only include the second half  $\mathcal{D}^{\mathrm{off},2}$  of the offline dataset  $\mathcal{D}^{\mathrm{off}}$ , so as to exploit the fact that  $\mathcal{D}^{\mathrm{off},2}$  is statistically independent from  $\mathcal{D}_{\mathrm{imitate}}^{\mathrm{on}} \cup \mathcal{D}_{\mathrm{explore}}^{\mathrm{on}}$ .
- We invoke the pessimistic model-based offline RL algorithm proposed in Li et al. (2022) to compute the final policy estimate  $\widehat{\pi}$ ; see Algorithm 6 in Appendix D.3 for more details.
As it turns out, Algorithm 1 is capable of achieving provable sample efficiency, as demonstrated in the following theorem. Here and below, we recall that  $K = K^{\mathrm{off}} + K^{\mathrm{on}}$ .
Theorem 1. Consider  $\delta \in (0,1)$  and  $\varepsilon \in (0,H]$ . Choose the algorithmic parameters such that
$$
\eta = \sqrt {\frac {\log A}{2 T _ {\max} (K ^ {\mathrm {o n}} H) ^ {2}}} \quad a n d \quad T _ {\max} \geq 2 (K ^ {\mathrm {o n}} H) ^ {2} \log A.
$$
Suppose that
$$
K ^ {\text {o n}} + K ^ {\text {o f f}} \geq c _ {1} \frac {H ^ {3} S C ^ {\star} (\sigma)}{\varepsilon^ {2}} \log^ {2} \frac {K}{\delta} \tag {24a}
$$
$$
K ^ {\text {o n}} \geq c _ {1} \frac {H ^ {3} S A \min  \{H \sigma , 1 \}}{\varepsilon^ {2}} \log \frac {K}{\delta} \tag {24b}
$$
for some large enough constant  $c_1 > 0$ . Then with probability at least  $1 - \delta$ , the policy  $\widehat{\pi}$  returned by Algorithm 1 satisfies
$$
V _ {1} ^ {\star} (\rho) - V ^ {\widehat {\pi}} (\rho) \leq \varepsilon ,
$$
provided that  $K^{\mathrm{on}}$  and  $K^{\mathrm{off}}$  both exceed some polynomial  $\mathsf{poly}(H, S, A, C^{\star}(\sigma), \log \frac{K}{\delta})$  (independent of  $\varepsilon$ ).
The proof is deferred to Appendix E. In a nutshell, Theorem 1 uncovers that our algorithm yields  $\varepsilon$ -accuracy as long as
$$
K ^ {\text {o n}} + K ^ {\text {o f f}} \gtrsim \frac {H ^ {3} S C ^ {\star} (\sigma)}{\varepsilon^ {2}} \log^ {2} \frac {K}{\delta}, \tag {25a}
$$
$$
K ^ {\mathrm {o n}} \gtrsim \frac {H ^ {3} S A \min  \{H \sigma , 1 \}}{\varepsilon^ {2}} \log \frac {K}{\delta}, \tag {25b}
$$
ignoring lower-order terms. Several implications of this result are as follows.
Sample complexity benefits compared with pure online or pure offline RL. To make apparent its advantage compared with both pure offline and online RL, we make comparisons with several most relevant works. Discussions of other related works are deferred to Appendix A.
Sample complexity with balanced online and offline data. For the ease of presentation, let us look at a simple case where  $K^{\mathrm{off}} = K^{\mathrm{on}} = K / 2$ . The sample complexity bound (25) in this case simplifies to
$$
\widetilde {O} \left(\min  _ {\sigma \in [ 0, 1 ]} \left\{\frac {H ^ {3} S A \min  \{H \sigma , 1 \}}{\varepsilon^ {2}} + \frac {H ^ {3} S C ^ {\star} (\sigma)}{\varepsilon^ {2}} \right\}\right) =: \widetilde {O} \left(\min  _ {\sigma \in [ 0, 1 ]} f _ {\text {m i x e d}} (\sigma)\right). \tag {26}
$$
Comparisons with pure online RL. We now look at pure online RL, corresponding to the case where  $K = K^{\mathrm{on}}$  (so that all sample episodes are collected via online exploration). In this case, the minimax-optimal sample complexity for computing an  $\varepsilon$ -optimal policy is known to be (Azar et al., 2017; Li et al., 2023)
$$
\widetilde {O} \left(\frac {H ^ {3} S A}{\varepsilon^ {2}}\right) = \widetilde {O} \left(f _ {\text {m i x e d}} (1)\right) \tag {27}
$$
assuming that  $\varepsilon$  is sufficiently small, which is clearly worse than (26). For instance, if there exists some very small  $\sigma \ll 1 / H$  obeying  $C^\star (\sigma)\lesssim 1$ , then the ratio of (26) to (27) is at most
$$
H \sigma + 1 / A \ll 1, \tag {28}
$$
thus resulting in substantial sample size savings.
Comparisons with pure offline RL. In contrast, in the pure offline case where  $K = K^{\mathrm{off}}$ , the minimax sample complexity is known to be (Li et al., 2022)
$$
\widetilde {O} \left(\frac {H ^ {3} S C ^ {\star} (0)}{\varepsilon^ {2}}\right) = \widetilde {O} \left(f _ {\text {m i x e d}} (0)\right) \tag {29}
$$
for any target accuracy level  $\varepsilon$ , which is apparently larger than (26) in general. In particular, recognizing that  $C^{\star}(0) = \infty$  in the presence of incomplete coverage of the state-action space reachable by  $\pi^{\star}$ , we might harvest enormous sample size benefits (by exploiting the ability of online RL to visit the previously uncovered state-action-step tuples).
Comparison with Wagenmaker and Pacchiano (2022). It is worth noting that Wagenmaker and Pacchiano (2022) also considered policy fine-tuning and proposed a method called FTPedel to tackle linear MDPs. The results therein, however, were mainly instance-dependent, thus making it difficult to compare in general. That being said, we would like to clarify two points:
- Wagenmaker and Pacchiano (2022) imposed all-policy concentrability assumptions, requiring the combined dataset (i.e., the offline and online data altogether) to cover certain feature vectors for all linear softmax policies (see Wagenmaker and Pacchiano (2022, Definition 4.1)). In contrast, our results only assume single-policy (partial) concentrability, which is much weaker than the all-policy counterpart.
- When specializing Wagenmaker and Pacchiano (2022, Corollary 1) to the tabular cases, the sample complexity therein becomes  $\widetilde{O}(H^7 S^2 A^2 / \varepsilon^2)$ , which is much larger than our result.
Miscellaneous properties of the proposed algorithm. In addition to the sample complexity advantages, the proposed hybrid RL enjoys several attributes that could be practically appealing.
Adaptivity to unknown optimal  $\sigma$ . While we have introduced the parameter  $\sigma$  to capture incomplete coverage, our algorithm does not rely on any knowledge of  $\sigma$ . Take the balanced case described around (26) for instance: our algorithm automatically identifies the optimal  $\sigma$  that minimizes the function  $f_{\mathrm{mixed}}(\sigma)$  over all  $\sigma \in [0,1]$ . In other words, Algorithm 1 is able to automatically identify the optimal trade-offs between distribution mismatch and inadequate coverage.
Reward-agnostic data collection. It is noteworthy that the online exploration procedure employed in Algorithm 1 does not require any prior information about the reward function. In other words, it is mainly designed to improve coverage of the state-action space, a property independent from the reward function. In truth, the reward function is only queried at the last step to output the learned policy. This enables us to perform hybrid RL in a reward-agnostic manner, which is particularly intriguing in practice, as there is no shortage of scenarios where the reward functions might be engineered subsequently to meet different objectives.
Strengthening behavior cloning. Another notable feature is that our algorithm does not rely on prior knowledge about the policies generating the offline dataset  $\mathcal{D}^{\mathrm{off}}$ ; in fact, it is capable of finding a mixed exploration policy  $\pi^{\mathrm{imitate}}$  that inherits the advantages of the unknown behavior policy  $\pi^{\mathrm{off}}$ . This could be of particular interest for behavior cloning, where the offline dataset  $\mathcal{D}^{\mathrm{off}}$  is generated by an expert policy, with  $C^{\star} = C^{\star}(0) \approx 1$ , i.e. the expert policy covers the optimal one. In this situation, the supplement of online data collection improves behavior cloning by lowering the statistical error from  $\sqrt{\frac{H^{3}SC^{\star}}{K_{\mathrm{off}}}}$  to  $\sqrt{\frac{H^{3}SC^{\star}}{K_{\mathrm{off}} + K_{\mathrm{on}}}}$ , together with an executable learned policy  $\pi^{\mathrm{imitate}}$ .
Computational complexity. We now take a moment to discuss the computational cost of the proposed algorithm. In Stage 1, we need to first estimate the transition matrices  $\{P_h\}$ , which can be accomplished with runtime  $O(K^{\mathrm{on}})$ . In the ensuing stages, we call Algorithm 3 to estimate  $\widehat{d}^{\pi}$  for each  $\pi$  we encounter. When computing  $\pi^{\mathrm{imitate}}$ , we need to calculate  $\widehat{d}^{\pi}$  for  $T_1 = T_{\max}T_2$  times, where  $T_2$  denotes the number of iterations for calculating  $\mu^{t+1}$  in Eq. (20b); in comparison, the computational cost of estimating  $d^{\pi}$  to yield  $\pi^{\mathrm{explore}}$  in Eq. (22) is much smaller. For each  $\widehat{d}^{\pi}$ , it needs  $O(HS^2A)$  computation. With a slight modification on the target Eq. (19) as follows
$$
\mu^ {\text {i m i t a t e}} \approx
$$
$$
\arg \min  _ {\mu \in \Delta (\Pi)} \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} \max  _ {a \in \mathcal {A}} \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K H} + O \left(\frac {1}{S H}\right) \widehat {d} _ {h} ^ {\text {o f f}} (s , a) + \underset {\pi^ {\prime} \sim \mu^ {\text {e x p l o r e}}} {\mathbb {E}} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right] + \underset {\pi^ {\prime} \sim \mu} {\mathbb {E}} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right]}, \tag {30}
$$
we can find a good enough  $\pi^{\mathrm{imitate}}$  with  $T_{\mathrm{max}} = \widetilde{O}(H^{2}S^{2})$  for  $\eta \asymp \frac{1}{H^{2}S^{2}}$  and  $O(H^{4}S^{4}A^{2})$  Frank-Wolfe updates for  $\alpha \asymp \frac{1}{H^{3}S^{3}A^{2}}$ . These taken collectively lead to the following computational complexity for each stage:  $O(K^{\mathrm{on}} + H^{7}S^{8}A^{3} + K^{\mathrm{off}}H)$  for Stage 1,  $O(H^{7}S^{7}A^{3})$  for Stage 2, and  $O(KH)$  for Stage 3.
We have studied the policy fine-tuning problem of practical interest, where one is allowed to exploit pre-collected historical data to facilitate and improve online RL. We have proposed a three-stage algorithm tailored to the tabular setting, which attains provable sample size savings compared with both pure online RL and pure offline RL algorithms. Our algorithm design has leveraged key insights from recent advances in both model-based offline RL and reward-agnostic online RL.
While the proposed algorithm achieves provable sample efficiency, this cannot be guaranteed unless the sample size already surpasses a fairly large threshold (in other words, the algorithm imposes a high burn-in cost). It would be of great interest to see whether one can achieve sample optimality for the entire  $\varepsilon$ -range. Another issue arises from the computation side: even though the proposed algorithm can be implemented in polynomial time, the computational complexity of the Frank-Wolfe-type subroutine might already be too expensive for solving problems with enormous dimensionality. Can we hope to further accelerate it to make it practically more appealing? Finally, it might also be interesting to study sample-efficient hybrid RL in the presence of low-complexity function approximation, in the hope of further reducing sample complexity.
Y. Chen is supported in part by the Alfred P. Sloan Research Fellowship, the Google Research Scholar Award, the AFOSR grant FA9550-22-1-0198, the ONR grant N00014-22-1-2354, and the NSF grants CCF-2221009 and CCF-1907661. Y. Chi are supported in part by the grants ONR N00014-19-1-2404, NSF CCF-2106778, DMS-2134080 and CNS-2148212. J. D. Lee acknowledges support of the ARO under MURI Award W911NF-11-1-0304, the Sloan Research Fellowship, the NSF grants CCF 2002272, IIS 2107304 and CIF 2212262, the ONR Young Investigator Award, and the NSF CAREER Award 2144994.
Agarwal, A., Kakade, S., Krishnamurthy, A., and Sun, W. (2020). Flambe: Structural complexity and representation learning of low rank mdps. arXiv preprint arXiv:2006.10814.
Alon, N. and Spencer, J. H. (2016). The probabilistic method. John Wiley & Sons.
Auer, P. and Ortner, R. (2006). Logarithmic online regret bounds for undiscounted reinforcement learning. Advances in neural information processing systems, 19.
Azar, M. G., Osband, I., and Munos, R. (2017). Minimax regret bounds for reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 263-272. JMLR.org.
Bai, Y., Xie, T., Jiang, N., and Wang, Y.-X. (2019). Provably efficient  $q$ -learning with low switching cost. In Advances in Neural Information Processing Systems, volume 32, pages 8002-8011.
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. (2019). Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680.
Bertsekas, D. P. (2017). Dynamic programming and optimal control (4th edition). Athena Scientific.
Brafman, R. I. and Tennenholtz, M. (2002). R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3(Oct):213-231.
Brambilla, M., Ferrante, E., Birattari, M., and Dorigo, M. (2013). Swarm robotics: a review from the swarm engineering perspective. Swarm Intelligence, 7(1):1-41.
Bubeck, S. (2015). Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning, 8(3-4):231-357.
Cai, C., Cai, T. T., and Li, H. (2022). Transfer learning for contextual multi-armed bandits. arXiv preprint arXiv:2211.12612.
Zhang, Z., Chen, Y., Lee, J. D., and Du, S. S. (2023). Settling the sample complexity of online reinforcement learning. arXiv preprint arXiv:2307.13586.
Zhang, Z., Du, S., and Ji, X. (2021b). Near optimal reward-free reinforcement learning. In International Conference on Machine Learning, pages 12402-12412. PMLR.
Zhang, Z., Zhou, Y., and Ji, X. (2020b). Model-free reinforcement learning: from clipped pseudo-regret to sample complexity. arXiv preprint arXiv:2006.03864.
Zhou, D., Gu, Q., and Szeptesvari, C. (2021a). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, pages 4532-4576. PMLR.
Zhou, Z., Zhou, Z., Bai, Q., Qiu, L., Blanchet, J., and Glynn, P. (2021b). Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 3331-3339. PMLR.
A Other related works
In this section, we briefly discuss a small set of additional prior works related to the current paper.
(Reward-aware) online RL. In online RL, an agent seeks to find a near-optimal policy by sequentially and adaptively interacting with the unknown environment, without having access to any additional offline dataset. The extensive studies of online RL gravitate around how to optimally trade off exploration against exploitation, for which the principle of optimism in the face of uncertainty plays a crucial role (Auer and Ortner, 2006; Jaksch et al., 2010; Azar et al., 2017; Dann et al., 2017; Jin et al., 2018; Bai et al., 2019; Dong et al., 2019; Zhang et al., 2020b; Menard et al., 2021b; Li et al., 2021b; Zhang et al., 2023). Information-theoretic regret lower bounds have been established by Domingues et al. (2021); Jin et al. (2018), which are shown to be achievable (up to log factor) by the model-based approach for arbitrary sample sizes (Zhang et al., 2023). A further strand of works extended these studies to the case with function approximation, including both linear function approximation (Jin et al., 2020b; Zanette et al., 2020; Zhou et al., 2021a; Li et al., 2021a) and other more general families of function approximation (Du et al., 2021; Jin et al., 2021a; Foster et al., 2021).