SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gen Li* CUHK
Wenhao Zhan* Princeton
Jason D. Lee  
Princeton
Yuejie Chi CMU
Yuxin Chen†  
UPenn
This paper studies tabular reinforcement learning (RL) in the hybrid setting, which assumes access to both an offline dataset and online interactions with the unknown environment. A central question boils down to how to efficiently utilize online data to strengthen and complement the offline dataset and enable effective policy fine-tuning. Leveraging recent advances in reward-agnostic exploration and offline RL, we design a three-stage hybrid RL algorithm that beats the best of both worlds — pure offline RL and pure online RL — in terms of sample complexities. The proposed algorithm does not require any reward information during data collection. Our theory is developed based on a new notion called single-policy partial concentrability, which captures the trade-off between distribution mismatch and miscoverage and guides the interplay between offline and online data.
As reinforcement learning (RL) shows promise in achieving super-human empirical success across diverse fields (e.g., games (Silver et al., 2016; Vinyals et al., 2019; Berner et al., 2019; Mnih et al., 2013), robotics (Brambilla et al., 2013), autonomous driving (Shalev-Shwartz et al., 2016)), theoretical understanding about RL has also been substantially expanded, with the aim of distilling fundamental principles that can inform and guide practice. Among all sorts of theoretical questions being pursued, how to make the best use of data emerges as a question of profound interest for problems with enormous dimensionality.
There are at least two mainstream mechanisms when it comes to data collection: online RL and offline RL. Let us briefly describe their attributes and differences as follows.
Online RL. In this setting, an agent learns how to maximize her cumulative reward through interaction with the unknown environment (by, say, executing a sequence of adaptively chosen actions and utilizing the instantaneous feedback of the environment). Given that all information about the environment is obtained through real-time data collection, the main challenge lies in how to (optimally) manage the trade-off between exploration and exploitation. Towards this, one popular approach advertises the principle of optimism in the face of uncertainty — e.g., employing upper confidence bounds during value estimation to guide exploration — whose effectiveness has been shown for both the tabular case (Auer and Ortner, 2006; Jaksch et al., 2010; Azar et al., 2017; Dann et al., 2017; Jin et al., 2018; Bai et al., 2019; Dong et al., 2019; Zhang et al., 2020b; Menard et al., 2021b; Li et al., 2021b) and the case with function approximation (Jin et al., 2020b; Zanette et al., 2020; Zhou et al., 2021a; Li et al., 2021a; Du et al., 2021; Jin et al., 2021a; Foster et al., 2021; Chen et al., 2022b).
Offline RL. In contrast, offline RL assumes access to a pre-collected dataset, without given permission to perform any further data collection. The feasibility of reliable offline RL depends heavily
on the quality of the dataset at hand. A central challenge stems from the presence of distribution shift: the distribution of the offline dataset might differ significantly from that induced by the target policy. Another common challenge arises from insufficient data coverage: a nontrivial fraction of the state-action pairs might be inadequately visited in the available dataset, thus precluding one from faithfully evaluating many policies based solely on the offline dataset. To circumvent these obstacles, recent works proposed the principle of pessimism in the face of uncertainty, recommending caution when selecting poorly visited actions (Liu et al., 2020; Kumar et al., 2020; Jin et al., 2021b; Rashidinejad et al., 2021; Uehara and Sun, 2021; Li et al., 2022; Yin et al., 2021; Shi et al., 2022; Cai et al., 2022). Without requiring uniform coverage of all policies, the pessimism approach proves effective as long as the so-called single-policy concentrability is satisfied, which only assumes adequate coverage over the part of the state-action space reachable by the desirable policy.
In reality, however, both mechanisms above come with limitations. For instance, even the single-policy concentrability requirement might be too stringent (and hence fragile) for offline RL, as it is not uncommon for the historical dataset to miss a small yet essential part of the state-action space. Pure online RL might also be overly restrictive, given that there might be information from past data that could help initialize online exploration and mitigate the burden of further data collection.
All this motivates the studies of hybrid RL, a scenario where the agent has access to an offline dataset while, in the meantime, (limited) online data collection is permitted as well. Oftentimes, this scenario is practically not only feasible but also appealing: on the one hand, offline data provides useful information for policy pre-training, while further online exploration helps enrich existing data and allows for effective policy fine-tuning. As a matter of fact, multiple empirical works (Rajeswaran et al., 2017; Vecerik et al., 2017; Kalashnikov et al., 2018; Hester et al., 2018; Nair et al., 2018, 2020) indicated that combining online RL with offline datasets outperforms both pure online RL and pure offline RL. Nevertheless, theoretical pursuits about hybrid RL are lagging behind. Two recent works Ross and Bagnell (2012); Xie et al. (2021b) studied a restricted setting, where the agent is aware of a Markovian behavior policy (a policy that generates offline data) and can either execute the behavior policy or any other adaptive choice to draw samples in each episode; in this case, Xie et al. (2021b) proved that under the single-policy concentrability assumption of the offline dataset, having perfect knowledge about the behavior policy does not improve online exploration in the minimax sense. Another strand of works Song et al. (2022); Nakamoto et al. (2023); Wagenmaker and Pacchiano (2022) looked at a more general offline dataset and investigated how to leverage offline data in online exploration. From the sample complexity viewpoint, Wagenmaker and Pacchiano (2022) studied the statistical benefits of hybrid RL in the presence of linear function approximation; the result therein, however, required strong assumptions on data coverage (i.e., all-policy concentrability) and fell short of unveiling provable gains in the tabular case (as we shall elucidate momentarily). In light of such theoretical inadequacy in previous works, this paper is motivated to pursue the following question:
Does hybrid RL allow for improved sample complexity compared to pure online or offline RL in the tabular case?
We deliver an affirmative answer to the above question. Further relaxing the single-policy concentrability assumption, we introduce a relaxed notation called single-policy partial concentrability (to be made precise in Definition 2), which (i) allows the dataset to miss a fraction of the state-action space visited by the optimal policy and (ii) captures the tradeoff between distribution mismatch and lack of coverage. Armed with this notion, our results reveal provable statistical benefits of hybrid RL compared with both pure online and offline RL. The main contributions are summarized below.
A novel three-stage algorithm. We design a new hybrid RL algorithm consisting of three stages. In the first stage, we obtain crude estimation of the occupancy distribution  $d^{\pi}$  w.r.t. any policy  $\pi$  as well as the data distribution  $d^{\mathrm{off}}$  of the offline dataset. The second stage performs online exploration; in particular, we execute one exploration policy to imitate the offline dataset and another one to explore the inadequately visited part of the unknown environment, with both policies computed by approximately solving convex optimization sub-problems. Notably, these two stages do not count on the availability of reward information, and thus operate in a reward-agnostic manner. The final stage then invokes the state-of-the-art offline RL algorithm for policy learning, on the basis of all data we have available (including both online and offline data).
Computationally efficient subroutines. Throughout the first two stages of the algorithm, we need to solve a couple of convex sub-problems with exponentially large dimensions. In order to attain computational efficiency, we design efficient Frank-Wolfe-type paradigms to solve the sub-problems approximately, which run in polynomial time. This plays a crucial role in ensuring computational tractability of the proposed three-stage algorithm.
Improved sample complexity. We characterize the sample complexity of our algorithm (see Theorem 1), which provably improves upon both pure online and offline RL. On the one hand, hybrid RL achieves strictly enhanced performance compared to pure offline RL (assuming the same sample size) when the offline data falls short of covering all state-action pairs reachable by the desired policy. On the other hand, the sample size allocated to online exploration in our algorithm might only need to be proportional to the fraction  $\sigma$  of the state-action space uncovered by the offline dataset, thus resulting in sample size saving in general compared to pure online RL (a case with  $\sigma = 1$ ).
Notation. Let us also introduce several useful notation. For integer  $m > 0$ , we let  $[m]$  represent the set  $\{1, \dots, m\}$ . For any set  $\mathcal{B}$ , we denote by  $\mathcal{B}^c$  its complement. For any policy  $\pi_0$ , we let  $\mathbb{1}_{\pi_0}: \Pi \to \{0, 1\}$  be an indicator function such that  $\mathbb{1}_{\pi_0}(\pi) = 1$  if  $\pi = \pi_0$  and  $\mathbb{1}_{\pi_0}(\pi) = 0$  otherwise. For any finite set  $\mathcal{A}$ , we denote by  $\Delta(\mathcal{A})$  the probability simplex over  $\mathcal{A}$ . Letting  $\mathcal{X} := (S, A, H, \frac{1}{\varepsilon}, \frac{1}{\delta})$ , we use the notation  $f(\mathcal{X}) = O(g(\mathcal{X}))$  or  $f(\mathcal{X}) \lesssim g(\mathcal{X})$  to indicate the existence of a universal constant  $C_1 > 0$  such that  $f \leq C_1g$ , the notation  $f(\mathcal{X}) \gtrsim g(\mathcal{X})$  to indicate that  $g(\mathcal{X}) = O(f(\mathcal{X}))$ , and the notation  $f(\mathcal{X}) \asymp g(\mathcal{X})$  to mean that  $f(\mathcal{X}) \lesssim g(\mathcal{X})$  and  $f(\mathcal{X}) \gtrsim g(\mathcal{X})$  hold simultaneously. The notation  $\widetilde{O}(\cdot)$  is defined in the same way as  $O(\cdot)$  except that it hides logarithmic factors.
2 Preliminaries and problem settings
Episodic finite-horizon MDPs. We study episodic finite-horizon Markov decision processes with  $S$  states,  $A$  actions, and horizon length  $H$ . We use  $\mathcal{M} = (\mathcal{S},\mathcal{A},H,P = \{P_h\}_{h = 1}^H,r = \{r_h\}_{h = 1}^H)$  to represent such an MDP, where  $\mathcal{S} = [S]$  and  $\mathcal{A} = [A]$  represent the state space and the action space, respectively. For each step  $h\in [H]$ , we let  $P_{h}:\mathcal{S}\times \mathcal{A}\to \Delta (\mathcal{S})$  represent the transition probability at this step, such that taking action  $a$  in state  $s$  at step  $h$  yields a transition to the next state drawn from the distribution  $P_{h}(\cdot |s,a)$ ; throughout the paper, we often employ the shorthand notation  $P_{h,s,a}\coloneqq P_h(\cdot |s,a)$ . Another ingredient is the reward function specified by  $r_h:\mathcal{S}\times \mathcal{A}\rightarrow [0,1]$  at step  $h$ ; namely, the agent will receive an immediate reward  $r_h(s,a)$  upon executing action  $a$  in state  $s$  at step  $h$ . It is assumed that the reward function is fully revealed upon completion of online data collection. Additionally, we assume throughout that each episode of the MDP starts from an initial state independently generated from some (unknown) initial state distribution  $\rho \in \Delta (S)$ .
A time-inhomogeneous Markovian policy is often denoted by  $\pi = \{\pi_h\}_{h=1}^H$  with  $\pi_h : \mathcal{S} \to \Delta(\mathcal{A})$ , where  $\pi_h(\cdot | s)$  characterizes the (randomized) action selection probability of the agent in state  $s$  at step  $h$ . If  $\pi$  is a deterministic policy, then we often abuse the noation and let  $\pi_h(s)$  represent the action selected in state  $s$  at step  $h$ . We find it convenient to introduce the following notation:
$$
\Pi := \text {t h e s e t o f a l l d e t e r m i n i s t i c p o l i c i e s}. \tag {1}
$$
We also need to handle mixed deterministic policies (i.e., each realization of the policy is randomly drawn from a mixture of deterministic policies). A mixed deterministic policy  $\pi^{\mathrm{mixed}}$  is denoted by
$$
\pi^ {\text {m i x e d}} = \sum_ {\pi \in \Pi} \mu (\pi) \pi = \mathbb {E} _ {\pi \sim \mu} [ \pi ] \quad \text {f o r s o m e} \mu \in \Delta (\Pi). \tag {2}
$$
Moreover, for any policy  $\pi$ , we define its associated value function (resp. Q-function) as follows, representing the expected cumulative rewards conditioned on an initial state (resp. an initial state-action pair):
$$
V _ {h} ^ {\pi} (s) := \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime}: h \leq h ^ {\prime} \leq H} r _ {h ^ {\prime}} (s, a) \Bigg | s _ {h} = s \right], \quad \forall s \in \mathcal {S};
$$
$$
Q _ {h} ^ {\pi} (s, a) := \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime}: h \leq h ^ {\prime} \leq H} r _ {h ^ {\prime}} (s, a) \Bigg |   s _ {h} = s, a _ {h} = a \right], \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}.
$$
Here, the expectation is over the length- $H$  sample trajectory  $(s_1, a_1, s_2, a_2, \ldots, s_H, a_H)$  when executing policy  $\pi$  in  $\mathcal{M}$ , where  $s_h$  (resp.  $a_h$ ) denotes the state (resp. action) at step  $h$  of this trajectory.
When the initial state is drawn from  $\rho$ , we further augment the notation and denote
$$
V _ {1} ^ {\pi} (\rho) = \mathbb {E} _ {s \sim \rho} \left[ V _ {1} ^ {\pi} (s) \right].
$$
Importantly, there exists at least one deterministic policy, denoted by  $\pi^{\star}$  throughout, that is able to maximize  $V_h^\pi (s)$  and  $Q_{h}^{\pi}(s,a)$  simultaneously for all  $(h,s,a)\in [H]\times S\times \mathcal{A}$ ; namely,
$$
V _ {h} ^ {\star} (s) := V _ {h} ^ {\pi^ {\star}} (s) = \max  _ {\pi} V _ {h} ^ {\pi} (s), \qquad Q _ {h} ^ {\star} (s, a) := Q _ {h} ^ {\pi^ {\star}} (s, a) = \max  _ {\pi} Q _ {h} ^ {\pi} (s, a), \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}.
$$
Moving beyond value functions and Q-functions, we would like to define, for each policy  $\pi$ , the associated state-action occupancy distribution  $d^{\pi} = [d_h^\pi]_{1 \leq h \leq H}$  such that
$$
d _ {h} ^ {\pi} (s, a) := \mathbb {P} \left(s _ {h} = s, a _ {h} = a \mid \pi\right), \quad \forall (s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ];
$$
in other words, this is the probability of the state-action pair  $(s,a)\in S\times \mathcal{A}$  being visited by  $\pi$  at step  $h$ . We shall also overload  $d^{\pi}$  to represent the state occupancy distribution such that
$$
d _ {h} ^ {\pi} (s) := \sum_ {a \in \mathcal {A}} d _ {h} ^ {\pi} (s, a) = \mathbb {P} \left(s _ {h} = s \mid \pi\right), \quad \forall (s, h) \in \mathcal {S} \times [ H ]. \tag {3}
$$
Given that each episode always starts with a state drawn from  $\rho$ , it is easily seen that  $d_1^\pi(s) = \rho(s)$  for any policy  $\pi$  and any  $s \in S$ .
Sampling mechanism. We consider a hybrid RL setting that assumes access to a historical dataset as well as the ability to further explore the environment via real-time sampling, as detailed below.
Offline data. Suppose that we have available a historical dataset (also called an offline dataset)
$$
\mathcal {D} ^ {\text {o f f}} = \left\{\tau^ {k, \text {o f f}} \right\} _ {1 \leq k \leq K ^ {\text {o f f}}}, \tag {4}
$$
containing  $K^{\mathrm{off}}$  sample trajectories each of length  $H$ . Here, the  $k$ -th trajectory in  $\mathcal{D}^{\mathrm{off}}$  is denoted by
$$
\tau^ {k, \text {o f f}} = \left(s _ {1} ^ {k, \text {o f f}}, a _ {1} ^ {k, \text {o f f}}, \dots , s _ {H} ^ {k, \text {o f f}}, a _ {H} ^ {k, \text {o f f}}\right), \tag {5}
$$
where  $s_h^{k,\mathrm{off}}$  and  $a_h^{k,\mathrm{off}}$  indicate respectively the state and action at step  $h$  of this trajectory  $\tau^{k,\mathrm{off}}$ . It is assumed that each trajectory  $\tau^{k,\mathrm{off}}$  is drawn independently using policy  $\pi^{\mathrm{off}}$ , which takes the form of a mixture of deterministic policies
$$
\pi^ {\text {o f f}} = \mathbb {E} _ {\pi \sim \mu^ {\text {o f f}}} [ \pi ] \quad \text {w i t h} \mu^ {\text {o f f}} \in \Delta (\Pi). \tag {6}
$$
Note that the learner only has access to the data samples but not  $\pi^{\mathrm{off}}$ . Throughout the paper, we use  $d^{\mathrm{off}} = \{d_h^{\mathrm{off}}\}_{1\leq h\leq H}$  to represent the occupancy distribution of this offline dataset such that
$$
d _ {h} ^ {\text {o f f}} (s, a) := \mathbb {P} \left(\left(s _ {h} ^ {k, \text {o f f}}, a _ {h} ^ {k, \text {o f f}}\right) = (s, a)\right), \quad \forall (s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ]. \tag {7}
$$
Online exploration. In addition to the offline dataset, the learner is allowed to interact with the unknown environment and collect more data in real time, in the hope of compensating for the insufficiency of the pre-collected data at hand and fine-tuning the policy estimate. More specifically, the learner is able to sample  $K^{\mathrm{on}}$  trajectories sequentially. In each sample trajectory,
- the initial state is generated independently from an (unknown) distribution  $\rho \in \Delta (\mathcal{S})$
- the learner selects a policy to execute the MDP, obtaining a sample trajectory of length  $H$ .
The total number of sample trajectories is thus given by
$$
K = K ^ {\text {o f f}} + K ^ {\text {o n}}. \tag {8}
$$
Concentrability assumptions for the offline dataset. To quantify the quality of the historical dataset, prior offline RL literature introduced the following single-policy concentrability coefficient based on certain density ratio of interest; see, e.g., Rashidinejad et al. (2021); Li et al. (2022).
Definition 1 (Single-policy concentrability). The single-policy concentrability coefficient  $C^{\star}$  of the offline dataset  $\mathcal{D}^{\mathrm{off}}$  is defined as
$$
C ^ {\star} := \max  _ {(s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ]} \frac {d _ {h} ^ {\pi^ {\star}} (s , a)}{d _ {h} ^ {\text {o f f}} (s , a)}. \tag {9}
$$
In words,  $C^{\star}$  employs the  $\ell_{\infty}$ -norm of the density ratio  $d^{\pi^{\star}} / d^{\mathrm{off}}$  to capture the shift of distributions between the occupancy distribution induced by the desired policy  $\pi^{\star}$  and the data distribution at hand. The terminology "single-policy" underscores that Definition 1 only compares the offline data distribution against the one generated by a single policy  $\pi^{\star}$ , which stands in stark contrast to other all-policy concentrability coefficients that are defined to account for all policies simultaneously.
One notable fact about Definition 1 is that: for  $C^{\star}$  to be finite, the historical data distribution needs to cover all state-action-step tuples reachable by  $\pi^{\star}$ . This requirement is, in general, inevitable if only the offline dataset is available; see the minimax lower bounds in Rashidinejad et al. (2021); Li et al. (2022) for more precise justifications. However, a requirement of this kind could be overly stringent for the hybrid setting considered herein, as the issue of incomplete coverage can potentially be overcome with the aid of online data collection. In light of this observation, we generalize Definition 1 to account for the trade-offs between distributional mismatch and partial coverage.
Definition 2 (Single-policy partial concentrability). For any  $\sigma \in [0,1]$ , the single-policy partial concentrability coefficient  $C^{\star}(\sigma)$  of the offline dataset  $\mathcal{D}^{\mathrm{off}}$  is defined as
$$
C ^ {\star} (\sigma) := \min  \left\{\max  _ {1 \leq h \leq H} \max  _ {(s, a) \in \mathcal {G} _ {h}} \frac {d _ {h} ^ {\pi^ {\star}} (s , a)}{d _ {h} ^ {\text {o f f}} (s , a)} \mid \left\{\mathcal {G} _ {h} \right\} _ {1 \leq h \leq H} \in \mathcal {G} (\sigma) \right\}, \tag {10}
$$
$$
\mathcal {G} (\sigma) := \left\{\left\{\mathcal {G} _ {h} \right\} _ {1 \leq h \leq H} \subseteq \mathcal {S} \times \mathcal {A} \mid \frac {1}{H} \sum_ {h = 1} ^ {H} \sum_ {(s, a) \notin \mathcal {G} _ {h}} d _ {h} ^ {\pi^ {*}} (s, a) \leq \sigma \right\}. \tag {11}
$$
In Definition 2, we allow a fraction of the state-action space reachable by  $\pi^{\star}$  to be insufficiently covered (as reflected in the definition of  $\mathcal{G}(\sigma)$  measured by the state-action occupancy distribution) — hence the terminology "partial". Intuitively,  $\mathcal{G}_h$  corresponds to a set of state-action pairs that undergo reasonable distribution shift (so that the corresponding density ratio does not rise above  $C^\star (\sigma)$ ), whereas the total occupancy density of its complement subset  $\mathcal{G}_h^c$  induced by  $\pi^{\star}$  is under control (i.e., no larger than  $\sigma$  when averaged across steps). As a self-evident fact,  $C^\star (\sigma)$  is nonincreasing in  $\sigma$ ; this means that as  $\sigma$  increases, we might incur a less severe distribution shift in a restricted part, at the price of less coverage. In this sense,  $C^\star (\sigma)$  reflects certain tradeoffs between distribution shift and coverage. Clearly,  $C^\star (\sigma)$  reduces to  $C^\star$  in Definition 1 by taking  $\sigma = 0$ .
Goal. Given a historical dataset  $\mathcal{D}^{\mathrm{off}}$  containing  $K^{\mathrm{off}}$  sample trajectories, we would like to design an online exploration scheme, in conjunction with the accompanying policy learning algorithm, so as to achieve desirable policy learning (or policy fine-tuning) in a data-efficient manner. Ideally, we would expect a hybrid RL algorithm to harvest provable statistical benefits compared to both purely online RL and purely offline RL approaches.
In this section, we propose a new algorithm to tackle the hybrid RL setting. Our algorithm design leverages recent ideas developed in offline RL and reward-agnostic exploration to improve sample efficiency. Our algorithm consists of three stages to be described shortly; informally, the first two stages conduct reward-agnostic exploration to imitate and complement the offline dataset, whereas the third stage invokes a sample-optimal offline RL algorithm to compute a near-optimal policy.
In the sequel, we split the offline dataset  $\mathcal{D}^{\mathrm{off}}$  into two halves:
$$
\mathcal {D} ^ {\text {o f f}, 1} \quad \text {a n d} \quad \mathcal {D} ^ {\text {o f f}, 2}, \tag {12}
$$
where  $\mathcal{D}^{\mathrm{off},1}$  (resp.  $\mathcal{D}^{\mathrm{off},2}$ ) consists of the first (resp. last)  $K^{\mathrm{off}} / 2$  independent trajectories from  $\mathcal{D}^{\mathrm{off}}$ . As we shall also see momentarily, online exploration in the proposed algorithm — which collects  $K^{\mathrm{on}}$  trajectories in total — can be divided into three parts, collecting  $K_{\mathrm{prepare}}^{\mathrm{on}}$ ,  $K_{\mathrm{imitate}}^{\mathrm{on}}$  and  $K_{\mathrm{explore}}^{\mathrm{on}}$  sample trajectories, respectively. Throughout this paper, for simplicity we choose
$$
K _ {\text {p r e p a r e}} ^ {\text {o n}} = K _ {\text {i m i t a t e}} ^ {\text {o n}} = K _ {\text {e x p l o r e}} ^ {\text {o n}} = K ^ {\text {o n}} / 3. \tag {13}
$$
3.1 A three-stage algorithm
We now elaborate on the three stages of the proposed algorithm. Due to space limitation, the pseudocode of the complete algorithm is provided in Appendix B.
Stage 1: estimation of the occupancy distributions. As a preparatory step for reward-agnostic exploration, we first attempt to estimate the occupancy distribution induced by any policy as well as the occupancy distribution  $d^{\mathrm{off}}$  associated with the historical dataset, as described below.