SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruiqi Zhang
Department of Statistics  
University of California, Berkeley  
rqzhang@berkeley.edu
Andrea Zanette
Department of EECS University of California, Berkeley zanette@berkeley.edu
In some applications of reinforcement learning, a dataset of pre-collected experience is already available but it is also possible to acquire some additional online data to help improve the quality of the policy. However, it may be preferable to gather additional data with a single, non-reactive exploration policy and avoid the engineering costs associated with switching policies.
In this paper we propose an algorithm with provable guarantees that can leverage an offline dataset to design a single non-reactive policy for exploration. We theoretically analyze the algorithm and measure the quality of the final policy as a function of the local coverage of the original dataset and the amount of additional data collected.
Reinforcement learning (RL) is a general framework for data-driven, sequential decision making [Puterman, 1994, Sutton and Barto, 2018]. In RL, a common goal is to identify a near-optimal policy, and there exist two main paradigms: online and offline RL.
Online RL is effective when the practical cost of a bad decision is low, such as in simulated environments (e.g., [Mnih et al., 2015, Silver et al., 2016]). In online RL, a well designed learning algorithm starts from tabula rasa and implements a sequence of policies with a value that should approach that of an optimal policy. When the cost of making a mistake is high, such as in healthcare [Gottesman et al., 2018] and in self-driving [Kiran et al., 2021], an offline approach is preferred. In offline RL, the agent uses a dataset of pre-collected experience to extract a policy that is as good as possible. In this latter case, the quality of the policy that can be extracted from the dataset is limited by the quality of the dataset.
Many applications, however, fall between these two opposite settings: for example, a company that sells products online has most likely recorded the feedback that it has received from its customers, but can also collect a small amount of additional strategic data in order to improve its recommendation engine. While in principle an online exploration algorithm can be used to collect fresh data, in practice there are a number of practical engineering considerations that require the policy to be
deployed to be non-reactive. We say that a policy is non-reactive, (or passive, memoryless) if it chooses actions only according to the current state of the system. Most online algorithms are, by design, reactive to the data being acquired.
An example of a situation where non-reactive policies may be preferred are those where a human in the loop is required to validate each exploratory policy before they are deployed, to ensure they are of high quality [Dann et al., 2019] and safe [Yang et al., 2021], as well as free of discriminatory content [Koenecke et al., 2020]. Other situations that may warrant non-reactive exploration are those where the interaction with the user occurs through a distributed system with delayed feedback. In recommendation systems, data collection may only take minutes, but policy deployment and updates can span weeks [Afsar et al., 2022]. Similar considerations apply across various RL application domains, including healthcare [Yu et al., 2021], computer networks [Xu et al., 2018], and new material design [Raccuglia et al., 2016]. In all such cases, the engineering effort required to implement a system that handles real-time policy switches may be prohibitive: deploying a single, non-reactive policy is much preferred.
Non-reactive exploration from offline data Most exploration algorithms that we are aware of incorporate policy switches when they interact with the environment [Dann and Brunskill, 2015, Dann et al., 2017, Azar et al., 2017, Jin et al., 2018, Dann et al., 2019, Zanette and Brunskill, 2019, Zhang et al., 2020b]. Implementing a sequence of non-reactive policies is necessary in order to achieve near-optimal regret: the number of policy switches must be at least  $\widetilde{O}\left(H\left|S\right|\left|\mathcal{A}\right|\log \log K\right)$  where  $S,A,H,K$  are the state space, action space, horizon and the total number of episodes, respectively [Qiao et al., 2022]. With no switches, i.e., when a fully non-reactive data collection strategy is implemented, it is information theoretically impossible [Xiao et al., 2022] to identify a good policy using a number of samples polynomial in the size of the state and action space.
However, these fundamental limits apply to the case where the agent learns from tabula rasa. In the more common case where offline data is available, we demonstrate that it is possible to leverage the dataset to design an effective non-reactive exploratory policy. More precisely, an available offline dataset contains information (e.g., transitions) about a certain area of the state-action space, a concept known as partial coverage. A dataset with partial coverage naturally identifies a 'sub-region' of the original MDP—more precisely, a sub-graph—that is relatively well explored. We demonstrate that it is possible to use the dataset to design a non-reactive policy that further explores such sub-region. The additional data collected can be used to learn a near-optimal policy in such sub-region.
In other words, exploration with no policy switches can collect additional information and compete with the best policy that is restricted to an area where the original dataset has sufficient information. The value of such policy can be much higher than the one that can be computed using only the offline dataset, and does not directly depend on a concentrability coefficient [Munos and Szepesvári, 2008, Chen and Jiang, 2019].
Perhaps surprisingly, addressing the problem of reactive exploration in reinforcement learning requires an approach that combines both optimism and pessimism in the face of uncertainty to explore efficiently. While optimism drives exploration, pessimism ensures that the agent explores conservatively, in a way that restricts its exploration effort to a region that it knows how to navigate, and so our paper makes a technical contribution which can be of independent interest.
Contributions To the best of our knowledge, this is the first paper with theoretical rigor that considers the problem of designing an experiment in reinforcement learning for online, passive exploration, using a dataset of pre-collected experience. More precisely, our contributions are as follows:
- We introduce an algorithm that takes as input a dataset, uses it to design and deploy a non-reactive exploratory policy, and then outputs a locally near-optimal policy.
- We introduce the concept of sparsified MDP, which is actively used by our algorithm to design the exploratory policy, as well as to theoretically analyze the quality of the final policy that it finds.
- We rigorously establish a nearly minimax-optimal upper bound for the sample complexity needed to learn a local  $\varepsilon$ -optimal policy using our algorithm.
<sup>1</sup>More rigorously, our sample complexity matches the minimax lower bound when we have some degree of knowledge for the full MDP, see the discussion in Section section 5. The lower bound for the samples needed in reward-free case is proved in [Jin et al., 2020b], but their result applies to the non-homogeneous MDP. The sample complexity on homogeneous MDP should be shaved off by an  $H$  factor.
In this section we discuss some related literature. Our work is related to low-switching algorithms, but unlike those, we focus on the limit case where no-switches are allowed. For more related work about low-switching algorithms, offline RL, task-agnostic RL, and reward-free RL we refer to Appendix F.
Low-switching RL In reinforcement learning, [Bai et al., 2019] first proposed Q-learning with UCB2 exploration, proving an  $O(H^3 |\mathcal{S}||\mathcal{A}|\log K)$  switching cost. This was later improved by a factor of  $H$  by the UCBadvantage algorithm in [Zhang et al., 2020b]. Recently, [Qiao et al., 2022] generalized the policy elimination algorithm from [Cesa-Bianchi et al., 2013] and introduced APEVE, which attains an optimal  $O(H|\mathcal{S}||\mathcal{A}|\log \log K)$  switching cost. The reward-free version of their algorithm (which is not regret minimizing) has an  $O(H|\mathcal{S}||\mathcal{A}|)$  switching cost.
Similar ideas were soon applied in RL with linear function approximation [Gao et al., 2021, Wang et al., 2021, Qiao and Wang, 2022] and general function approximation [Qiao et al., 2023]. Additionally, numerous research efforts have focused on low-adaptivity in other learning domains, such as batchedueling bandits [Agarwal et al., 2022], batched convex optimization [Duchi et al., 2018], linear contextual bandits [Ruan et al., 2021], and deployment-efficient RL [Huang et al., 2022].
Our work was inspired by the problem of non-reactive policy design in linear contextual bandits. Given access to an offline dataset, [Zanette et al., 2021a] proposed an algorithm to output a single exploratory policy, which generates a dataset from which a near-optimal policy can be extracted. However, there are a number of additional challenges which arise in reinforcement learning, including the fact that the state space is only partially explored in the offline dataset. In fact, in reinforcement learning, [Xiao et al., 2022] established an exponential lower bound for any non-adaptive policy learning algorithm starting from tabula rasa.
3 Setup
Throughout this paper, we let  $[n] = \{1,2,\dots,n\}$ . We adopt the big-O notation, where  $\widetilde{O} (\cdot)$  suppresses poly-log factors of the input parameters. We indicate the cardinality of a set  $\mathcal{X}$  with  $|\mathcal{X}|$ .
Markov decision process We consider time-homogeneous episodic Markov decision processes (MDPs). They are defined by a finite state space  $S$ , a finite action space  $\mathcal{A}$ , a transition kernel  $\mathbb{P}$ , a reward function  $r$  and the episodic length  $H$ . The transition probability  $\mathbb{P}(s' \mid s, a)$ , which does not depend on the current time-step  $h \in [H]$ , denotes the probability of transitioning to state  $s' \in S$  when taking action  $a \in \mathcal{A}$  in the current state  $s \in S$ . Typically we denote with  $s_1$  the initial state. For simplicity, we consider deterministic reward functions  $r: S \times \mathcal{A} \to [0,1]$ . A deterministic non-reactive (or memoryless, or passive) policy  $\pi = \{\pi_h\}_{h \in [H]}$  maps a given state to an action.
The value function is defined as the expected cumulated reward. It depends on the state  $s$  under consideration, the transition  $\mathbb{P}$  and reward  $r$  that define the MDP as well as on the policy  $\pi$  being implemented. It is defined as  $V_{h}(s;\mathbb{P},r,\pi) = \mathbb{E}_{\mathbb{P},\pi}[\sum_{i=h}^{H}r(s_{i},a_{i})\mid s_{h} = s]$ , where  $\mathbb{E}_{\mathbb{P},\pi}$  denotes the expectation generated by  $\mathbb{P}$  and policy  $\pi$ . A closely related quantity is the state-action value function, or  $Q$ -function, defined as  $Q_{h}(s,a;\mathbb{P},r,\pi) = \mathbb{E}_{\mathbb{P},\pi}[\sum_{i=h}^{H}r(s_{i},a_{i})\mid s_{h} = s,a_{h} = a]$ . When it is clear from the context, we sometimes omit  $(\mathbb{P},r)$  and simply write them as  $V_{h}^{\pi}(s)$  and  $Q_{h}^{\pi}(s,a)$ . We denote an MDP defined by  $S, A$  and the transition matrix  $\mathbb{P}$  as  $\mathcal{M} = (S, A, \mathbb{P})$ .
3.1 Interaction protocol
Algorithm 1 Design of experiments in reinforcement learning
Input: Offline dataset  $\mathcal{D}$
1: Offline phase: use  $\mathcal{D}$  to compute the exploratory policy  $\pi_{ex}$
2: Online phase: deploy  $\pi_{ex}$  to collect the online dataset  $\mathcal{D}'$
3: Planning phase: receive the reward function  $r$  and use  $\mathcal{D} \cup \mathcal{D}'$  to extract  $\pi_{final}$
Output: Return  $\pi_{final}$
In this paper we assume access to an offline dataset  $\mathcal{D} = \{(s, a, s')\}$  where every state-action  $(s, a)$  is sampled in an i.i.d. fashion from some distribution  $\mu$  and  $s' \sim \mathbb{P}(\cdot \mid s, a)$ , which is common in the offline RL literature [Xie et al., 2021a, Zhan et al., 2022, Rashidinejad et al., 2021, Uehara and Sun, 2021]. We denote  $N(s, a)$  and  $N(s, a, s')$  as the number of  $(s, a)$  and  $(s, a, s')$  samples in the offline dataset  $\mathcal{D}$ , respectively. The interaction protocol considered in this paper consists of three distinct phases, which are displayed in algorithm 1. They are:
- the offline phase, where the learner uses an offline dataset  $\mathcal{D}$  of pre-collected experience to design the non-reactive exploratory policy  $\pi_{ex}$ ;
- the online phase where  $\pi_{ex}$  is deployed to generate the online dataset  $\mathcal{D}'$ ;
- the planning phase where the learner receives a reward function and uses all the data collected to extract a good policy  $\pi_{final}$  with respect to that reward function.
The objective is to minimize the number of online episodic interactions needed to find a policy  $\pi_{final}$  whose value is as high as possible. Moreover, we focus on the reward-free RL setting [Jin et al., 2020a, Kaufmann et al., 2021, Li et al., 2023b], which is more general than reward-aware RL. In the offline and online phase, the data are generated without specific reward signals, and the entire reward information is then given in the planning phase. One of the primary advantages of using reward-free offline data is that it allows for the collection of data without the need for explicit reward signals. This can be particularly beneficial in environments where obtaining reward signals is costly, risky, ethically challenging, or where the reward functions are human-designed.
4 Algorithm: balancing optimism and pessimism for experimental design
In this section we outline our algorithm Reward-Free Non-reactive Policy Design (RF-NPD), which follows the high-level protocol described in algorithm 1. The technical novelty lies almost entirely in the design of the exploratory policy  $\pi_{ex}$ . In order to prepare the reader for the discussion of the algorithm, we first give some intuition in section 4.1 followed by the definition of sparsified MDP in section 4.2, a central concept of this paper, and then describe the implementation of line 1 in the protocol in algorithm 1 in section 4.3. We conclude by presenting the implementation of lines 2 and 3 in the protocol in algorithm 1.
4.1 Intuition
In order to present the main intuition for this paper, in this section we assume that enough transitions are available in the dataset for every edge  $(s,a)\to s^{\prime}$ , namely that the critical condition
$$
N (s, a, s ^ {\prime}) \geq \Phi = \widetilde {\Theta} \left(H ^ {2}\right) \tag {4.1}
$$
holds for all tuples  $(s, a, s') \in S \times \mathcal{A} \times S$  (the precise value for  $\Phi$  will be given later in eq. (5.1)). Such condition is hardly satisfied everywhere in the state-action-state space, but assuming it in this section simplifies the presentation of one of the key ideas of this paper.
The key observation is that when eq. (4.1) holds for all  $(s, a, s')$ , we can use the empirical transition kernel to design an exploration policy  $\pi_{ex}$  to eventually extract a near-optimal policy  $\pi_{final}$  for any desired level of sub-optimality  $\varepsilon$ , despite eq. (4.1) being independent of  $\varepsilon$ . More precisely, let  $\widehat{\mathbb{P}}$  be the empirical transition kernel defined in the usual way  $\widehat{\mathbb{P}}(s' \mid s, a) = N(s, a, s') / N(s, a)$  for any tuple  $(s, a, s')$ . The intuition—which will be verified rigorously in the analysis of the algorithm—is the following:
If eq. (4.1) holds for every  $(s, a, s')$  then  $\widehat{\mathbb{P}}$  can be used to design a non-reactive exploration policy  $\pi_{ex}$  which can be deployed on  $\mathcal{M}$  to find an  $\varepsilon$ -optimal policy  $\pi_{final}$  using  $\asymp \frac{1}{\varepsilon^2}$  samples.
We remark that even if the condition 4.1 holds for all tuples  $(s, a, s')$ , the empirical kernel  $\widehat{\mathbb{P}}$  is not accurate enough to extract an  $\varepsilon$ -optimal policy from the dataset  $\mathcal{D}$  without collecting further data. Indeed, the threshold  $\Phi = \widetilde{\Theta}(H^2)$  on the number of samples is independent of the desired sub-optimality  $\varepsilon > 0$ , while it is well known that at least  $\sim \frac{1}{\varepsilon^2}$  offline samples are needed to find an  $\varepsilon$ -optimal policy. Therefore, directly implementing an offline RL algorithm to use the available
offline dataset  $\mathcal{D}$  does not yield an  $\varepsilon$ -optimal policy. However, the threshold  $\Phi = \widetilde{\Theta}(H^2)$  is sufficient to design a non-reactive exploratory policy  $\pi_{ex}$  that can discover an  $\varepsilon$ -optimal policy  $\pi_{final}$  after collecting  $\sim \frac{1}{\varepsilon^2}$  online data.
4.2 Sparsified MDP
The intuition in the prior section must be modified to work with heterogeneous datasets and dynamics where  $N(s, a, s') \geq \Phi$  may fail to hold everywhere. For example, if  $\mathbb{P}(s' \mid s, a)$  is very small for a certain tuple  $(s, a, s')$ , it is unlikely that the dataset contains  $N(s, a, s') \geq \Phi$  samples for that particular tuple. In a more extreme setting, if the dataset is empty, the critical condition in eq. (4.1) is violated for all tuples  $(s, a, s')$ , and in fact the lower bound of Xiao et al. [2022] states that finding  $\varepsilon$ -optimal policies by exploring with a non-reactive policy is not feasible with  $\sim \frac{1}{\varepsilon^2}$  sample complexity. This suggests that in general it is not possible to output an  $\varepsilon$ -optimal policy using the protocol in algorithm 1.
However, a real-world dataset generally covers at least a portion of the state-action space, and so we expect the condition  $N(s,a,s') \geq \Phi$  to hold somewhere; the sub-region of the MDP where it holds represents the connectivity graph of the sparsified MDP. This is the region that the agent knows how to navigate using the offline dataset  $\mathcal{D}$ , and so it is the one that the agent can explore further using  $\pi_{ex}$ . More precisely, the sparsified MDP is defined to have identical dynamics as the original MDP on the edges  $(s,a) \longrightarrow s'$  that satisfy the critical condition 4.1. When instead the edge  $(s,a) \longrightarrow s'$  fails to satisfy the critical condition 4.1, it is replaced with a transition  $(s,a) \longrightarrow s^{\dagger}$  to an absorbing state  $s^{\dagger}$ .
Definition 4.1 (Sparsified MDP). Let  $s^{\dagger}$  be an absorbing state, i.e., such that  $\mathbb{P}\left(s^{\dagger} \mid s^{\dagger}, a\right) = 1$  and  $r(s^{\dagger}, a) = 0$  for all  $a \in \mathcal{A}$ . The state space in the sparsified MDP  $\mathcal{M}^{\dagger}$  is defined as that of the original MDP with the addition of  $s^{\dagger}$ . The dynamics  $\mathbb{P}^{\dagger}$  of the sparsified MDP are defined as
$$
\mathbb {P} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) = \left\{ \begin{array}{l l} \mathbb {P} \left(s ^ {\prime} \mid s, a\right) & i f N (s, a, s ^ {\prime}) \geq \Phi \\ 0 & i f N (s, a, s ^ {\prime}) <   \Phi , \end{array} \right. \quad \mathbb {P} ^ {\dagger} \left(s ^ {\dagger} \mid s, a\right) = \sum_ {\substack {s ^ {\prime} \neq s ^ {\dagger} \\ N (s, a, s ^ {\prime}) <   \Phi}} \mathbb {P} \left(s ^ {\prime} \mid s, a\right). \tag{4.2}
$$
For any deterministic reward function  $r: \mathcal{S} \times \mathcal{A} \to [0,1]$ , the reward function on the sparsified MDP is defined as  $r^\dagger(s,a) = r(s,a)$ ; for simplicity we only consider deterministic reward functions.
The empirical sparsified MDP  $\widehat{\mathcal{M}}^{\dagger} = (\mathcal{S}\cup \{s^{\dagger}\},\mathcal{A},\widehat{\mathbb{P}}^{\dagger})$  is defined in the same way but by using the empirical transition kernel in eq. (4.2). The empirical sparsified MDP is used by our algorithm to design the exploratory policy, while the (population) sparsified MDP is used for its theoretical analysis. They are two fundamental concepts in this paper. By formulating the sparsified MDP, we restrict the transitions and rewards within the area where we know how to navigate, embodying the principle of pessimism. Various forms of pessimistic regularization have been introduced to address the challenges of partially covered offline data. Examples include a pessimistic MDP [Kidambi et al., 2020] and limiting policies to those covered by offline data [Liu et al., 2020].
4.3 Offline design of experiments
In this section we describe the main sub-component of the algorithm, namely the sub-routine that uses the offline dataset  $\mathcal{D}$  to compute the exploratory policy  $\pi_{ex}$ . The exploratory policy  $\pi_{ex}$  is a mixture of the policies  $\pi^1, \pi^2, \ldots$  produced by a variant of the reward-free exploration algorithm of [Kaufmann et al., 2021, Ménard et al., 2021]. Unlike prior literature, the reward-free algorithm is not interfaced with the real MDP  $\mathcal{M}$ , but rather simulated on the empirical sparsified MDP  $\widehat{\mathcal{M}}^\dagger$ . This avoids interacting with  $\mathcal{M}$  with a reactive policy, but it introduces some bias that must be controlled. The overall procedure is detailed in algorithm 2. To be clear, no real-world samples are collected by algorithm 2; instead we use the word 'virtual samples' to refer to those generated from  $\widehat{\mathcal{M}}^\dagger$ .
At a high level, algorithm 2 implements value iteration using the empirical transition kernel  $\widehat{\mathbb{P}}^{\dagger}$ , with the exploration bonus defined in eq. (4.3) that replaces the reward function. The exploration bonus can be seen as implementing the principle of optimism in the face of uncertainty; however, the possibility of transitioning to an absorbing state with zero reward (due to the use of the absorbing state in the definition of  $\widehat{\mathbb{P}}^{\dagger}$ ) implements the principle of pessimism.
Algorithm 2 RF-UCB  $(\widehat{\mathcal{M}}^{\dagger},K_{ucb},\varepsilon ,\delta)$
Input:  $\delta \in (0,1),\varepsilon >0$  , number of episode  $K_{ucb}$  , MDP  $\widehat{\mathcal{M}}^{\dagger}$    
1: Initialize Counter  $n^1 (s,a) = n^1 (s,a,s') = 0$  for any  $(s,a,s')\in S\times A\times S$    
2: for  $k = 1,2,\dots,K_{ucb}$  do   
3: for  $h = H,H - 1,\ldots ,1$  do   
4: Set  $U_h^k (s,a) = 0$  for any  $(h,s,a)\in [H]\times S\times \mathcal{A}$    
5: for  $(s,a)\in S\times \mathcal{A}$  do   
6: Calculate the empirical uncertainty  $U_h^k (s,a)$  using eq. (4.4) where  $\phi$  is from eq. (4.3)   
7: end for   
8:  $\pi_h^k (s)\coloneqq \arg \max_{a\in \mathcal{A}}U_h^k (s,a),\forall s\in \mathcal{S}$  and  $\pi_h^k (s^\dagger)\coloneqq$  any action.   
9: end for   
10: Set initial state  $s_1^k = s_1$    
11: for  $h = 1,2,\dots,H$  do Sample  $a_h^k\sim \pi_h^k (s_h^k),s_{h + 1}^k\sim \widehat{\mathbb{P}}^\dagger (s_h^k,a_h^k)$    
12: end for   
13:  $n^{k + 1}(s,a) = n^{k}(s,a) + \sum_{h\in [H]}\mathbb{I}[(s,a) = (s_h^k,a_h^k)]$    
14:  $n^{k + 1}(s,a,s') = n^{k}(s,a,s') + \sum_{h\in [H]}\mathbb{I}[(s,a,s') = (s_h^k,a_h^k,s_{h + 1}^k)]$    
15: end for   
Output:  $\pi_{ex} =$  Uniform  $\{\pi^k\}_{k\in [K_{ucb}]}$
This delicate interplay between optimism and pessimism is critical to the success of the overall procedure: while optimism encourages exploration, pessimism ensures that the exploration efforts are directed to the region of the state space that the agent actually knows how to navigate, and prevents the agent from getting 'trapped' in unknown regions. In fact, these latter regions could have combinatorial structures [Xiao et al., 2022] which cannot be explored with non-reactive policies.
More precisely, at the beginning of the  $k$ -th virtual episode in algorithm 2,  $n^k (s,a)$  and  $n^k (s,a,s')$  denote the counters for the number of virtual samples simulated from  $\widehat{\mathcal{M}}^\dagger$  at each  $(s,a)$  and  $(s,a,s')$  tuple. We define the bonus function