SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\phi (x, \delta) = \frac {H}{x} \left[ \log \left(6 H | \mathcal {S} | | \mathcal {A} | / \delta\right) + | \mathcal {S} | \log \left(e \left(1 + x / | \mathcal {S} |\right)\right) \right], \tag {4.3}
$$
which is used to construct the empirical uncertainty function  $U_h^k$ , a quantity that serves as a proxy for the uncertainty of the value of any policy  $\pi$  on the spasified MDP. Specifically, for the  $k$ -th virtual episode, we set  $U_{H + 1}^k (s,a) = 0$  and  $s\in S, a\in \mathcal{A}$ . For  $h\in [H]$ , we further define:
$$
U _ {h} ^ {k} (s, a) = H \min  \left\{1, \phi \left(n ^ {k} (s, a)\right) \right\} + \widehat {\mathbb {P}} ^ {\dagger} (s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} U _ {h + 1} ^ {k} (\cdot , a ^ {\prime})\right). \tag {4.4}
$$
Note that, the above bonus function takes a similar form of the bonus function in [Ménard et al., 2021]. This order of  $O(1 / x)$  is set to achieve the optimal sample complexity, and other works have also investigated into other forms of bonus function [Kaufmann et al., 2021]. Finally, in line 10 through line 12 the current policy  $\pi^k$  —which is the greedy policy with respect to  $U^k$  —is simulated on the empirical reference MDP  $\widehat{\mathcal{M}}^\dagger$ , and the virtual counters are updated. It is crucial to note that the simulation takes place entirely offline, by generating virtual transitions from  $\widehat{\mathcal{M}}^\dagger$ . Upon termination of algorithm 2, the uniform mixture of policies  $\pi^1, \pi^2, \ldots$  form the non-reactive exploration policy  $\pi_{ex}$ , ensuring that the latter has wide 'coverage' over  $\mathcal{M}^\dagger$ .
4.4 Online and planning phase
Algorithm 2 implements line 1 of the procedure in algorithm 1 by finding the exploratory policy  $\pi_{ex}$ . After that, in line 2 of the interaction protocol the online dataset  $\mathcal{D}'$  is generated by deploying  $\pi_{ex}$  on the real MDP  $\mathcal{M}$  to generate  $K_{de}$  trajectories. Conceptually, the online dataset  $\mathcal{D}'$  and the offline dataset  $\mathcal{D}$  identify an updated empirical transition kernel  $\widetilde{\mathbb{P}}$  and its sparsified version $^2$ $\widetilde{\mathbb{P}}^\dagger$ . Finally, in
2For any  $(s,a,s^{\prime})\in S\times \mathcal{A}\times \mathcal{S},$  we define  $\widetilde{\mathbb{P}}^\dagger (s' | s,a) = \frac{m(s,a,s')}{m(s,a)}$  if  $N(s,a,s^{\prime})\geq \Phi$  and  $\widetilde{\mathbb{P}}^\dagger (s' | s,a) = 0$  otherwise. Finally, for any  $(s,a)\in S\times \mathcal{A},$  we have  $\widetilde{\mathbb{P}}^\dagger (s^\dagger |s,a) = \frac{1}{m(s,a)}\sum_{s'\in S,N(s,a,s') <   \Phi}m(s,a,s')$  and for any  $a\in \mathcal{A}$  , we have  $\widetilde{\mathbb{P}} (s^{\dagger}\mid s^{\dagger},a) = 1.$  Here  $N(s,a,s^{\prime})$  is the counter of initial offline data and  $m(\cdot ,\cdot)$  is the counter of online data.
line 3 a reward function  $r$  is received, and the value iteration algorithm (See Appendix E) is invoked with  $r$  as reward function and  $\widetilde{\mathbb{P}}^{\dagger}$  as dynamics, and the near-optimal policy  $\pi_{final}$  is produced. The use of the (updated) empirical sparsified dynamics  $\widetilde{\mathbb{P}}^{\dagger}$  can be seen as incorporating the principle of pessimism under uncertainty due to the presence of the absorbing state.
Our complete algorithm is reported in algorithm 3, and it can be seen as implementing the interaction protocol described in algorithm 1.
Algorithm 3 Reward-Free Non-reactive Policy Design (RF-NPD)
Input: Offline dataset  $\mathcal{D}$ , target suboptimality  $\varepsilon > 0$ , failure tolerance  $\delta \in (0,1]$ .
1: Construct the empirical sparsified MDP  $\widehat{\mathcal{M}}^{\dagger}$ .
2: Offline phase: run RF-UCB( $\widetilde{\mathcal{M}}^\dagger$ ,  $K_{ucb}, \varepsilon, \delta$ ) to obtain the exploratory policy  $\pi_{ex}$ .
3: Online phase: deploy  $\pi_{ex}$  on the MDP  $\mathcal{M}$  for  $K_{de}$  episodes to get the online dataset  $\mathcal{D}'$ .
4: Planning phase: receive the reward function  $r$ , construct  $\widetilde{\mathcal{M}}^{\dagger}$  from the online dataset  $\mathcal{D}'$ , compute  $\pi_{final}$  (which is the optimal policy on  $\widetilde{\mathcal{M}}^{\dagger}$ ) using value iteration (Appendix E).
Output:  $\pi_{final}$
5 Main Result
In this section, we present a performance bound on our algorithm, namely a bound on the suboptimality of the value of the final policy  $\pi_{final}$  when measured on the sparsified MDP  $\mathcal{M}^{\dagger}$ . The sparsified MDP arises because it is generally not possible to directly compete with the optimal policy using a non-reactive data collection strategy and a polynomial number of samples due to the lower bound of Xiao et al. [2022]; more details are given in Appendix C.
In order to state the main result, we let  $K = K_{ucb} = K_{de}$ , where  $K_{ucb}$  and  $K_{de}$  are the number of episodes for the offline simulation and online interaction, respectively. Let  $C$  be some universal constant, and choose the threshold in the definition of sparsified MDP as
$$
\Phi = 6 H ^ {2} \log (1 2 H | \mathcal {S} | ^ {2} | \mathcal {A} | / \delta). \tag {5.1}
$$
Theorem 5.1. For any  $\varepsilon >0$  and  $0 < \delta < 1$ , if we let the number of online episodes be
$$
K = \frac {C H ^ {2} | \mathcal {S} | ^ {2} | \mathcal {A} |}{\varepsilon^ {2}} \mathrm {p o l y l o g} \left(| \mathcal {S} |, | \mathcal {A} |, H, \frac {1}{\varepsilon}, \frac {1}{\delta}\right),
$$
then with probability at least  $1 - \delta$ , for any reward function  $r$ , the final policy  $\pi_{final}$  returned by Algorithm 3 satisfies the bound
$$
\max  _ {\pi \in \Pi} V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) - V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi_ {f i n a l}\right) \leq \varepsilon . \tag {5.2}
$$
The theorem gives a performance guarantee on the value of the policy  $\pi_{final}$ , which depends both on the initial coverage of the offline dataset  $\mathcal{D}$  as well as on the number of samples collected in the online phase. The dependence on the coverage of the offline dataset is implicit through the definition of the (population) sparsified  $\mathcal{M}^{\dagger}$ , which is determined by the counts  $N(\cdot, \cdot)$ .
In order to gain some intuition, we examine some special cases as a function of the coverage of the offline dataset.
Empty dataset Suppose that the offline dataset  $\mathcal{D}$  is empty. Then the sparsified MDP identifies a multi-armed bandit at the initial state  $s_1$ , where any action  $a$  taken from such state gives back the reward  $r(s_1, a)$  and leads to the absorbing state  $s^\dagger$ . In this case, our algorithm essentially designs an allocation strategy  $\pi_{ex}$  that is uniform across all actions at the starting state  $s_1$ . Given enough online samples,  $\pi_{final}$  converges to the action with the highest instantaneous reward on the multi-armed bandit induced by the start state. With no coverage from the offline dataset, the lower bound of Xiao et al. [2022] for non-reactive policies precludes finding an  $\varepsilon$ -optimal policy on the original MDP  $\mathcal{M}$  unless exponentially many samples are collected.
Known connectivity graph On the other extreme, assume that the offline dataset contains enough information everywhere in the state-action space such that the critical condition 4.1 is satisfied for
all  $(s, a, s')$  tuples. Then the sparsified MDP and the real MDP coincide, i.e.,  $\mathcal{M} = \mathcal{M}^{\dagger}$ , and so the final policy  $\pi_{final}$  directly competes with the optimal policy  $\pi^{*}$  for any given reward function in eq. (5.2). More precisely, the policy  $\pi_{final}$  is  $\varepsilon$ -suboptimal on  $\mathcal{M}$  if  $\widetilde{O}(H^{2}|S|^{2}|\mathcal{A}| / \varepsilon^{2})$  trajectories are collected in the online phase, a result that matches the lower bound for reward-free exploration of Jin et al. [2020b] up to log factors. However, we achieve such results with a data collection strategy that is completely passive, one that is computed with the help of an initial offline dataset whose size  $|\mathcal{D}| \approx \Phi \times |S|^{2}|\mathcal{A}| = \widetilde{O}(H^{2}|S|^{2}|\mathcal{A}|)$  need not depend on final accuracy  $\varepsilon$ .
Partial coverage In more typical cases, the offline dataset has only partial coverage over the state-action space and the critical condition 4.1 may be violated in certain state-action-successor states. In this case, the connectivity graph of the sparsified MDP  $\mathcal{M}^{\dagger}$  is a sub-graph of the original MDP  $\mathcal{M}$  augmented with edges towards the absorbing state. The lack of coverage of the original dataset arises through the sparsified MDP in the guarantees that we present in theorem 5.1. In this section, we 'translate' such guarantees into guarantees on  $\mathcal{M}$ , in which case the 'lack of coverage' is naturally represented by the concentrability coefficient
$$
C^{*} = \sup_{s,a}d_{\pi}(s,a) / \mu (s,a),
$$
see for examples the papers [Munos and Szepesvári, 2008, Chen and Jiang, 2019] for background material on the concentrability factor. More precisely, we compute the sample complexity—in terms of online as well as offline samples—required for  $\pi_{final}$  to be  $\varepsilon$ -suboptimal with respect to any comparator policy  $\pi$ , and so in particular with respect to the optimal policy  $\pi_*$  on the "real" MDP  $\mathcal{M}$ . The next corollary is proved in appendix B.3.
Corollary 5.2. Suppose that the offline dataset contains
$$
\widetilde {O} \Big (\frac {H ^ {4} | \mathcal {S} | ^ {2} | \mathcal {A} | C ^ {*}}{\varepsilon} \Big),
$$
samples and that additional
$$
\widetilde {O} \left(\frac {H ^ {3} | \mathcal {S} | ^ {2} | \mathcal {A} |}{\varepsilon^ {2}}\right)
$$
online samples are collected during the online phase. Then with probability at least  $1 - \delta$ , for any reward function  $r$ , the policy  $\pi_{final}$  is  $\epsilon$ -suboptimal with respect to any comparator policy  $\pi$
$$
V _ {1} \left(s _ {1}; \mathbb {P}, r, \pi\right) - V _ {1} \left(s _ {1}; \mathbb {P}, r, \pi_ {\text {f i n a l}}\right) \leq \varepsilon . \tag {5.3}
$$
The online sample size is equivalent to the one that arises in the statement of theorem 5.1 (expressed as number of online trajectories), and does not depend on the concentrability coefficient. The dependence on the offline dataset in theorem 5.1 is implicit in the definition of sparsified MDP; here we have made it explicit using the notion of concentrability.
Corollary 5.2 can be used to compare the achievable guarantees of our procedure with that of an offline algorithm, such as the minimax-optimal procedure detailed in [Xie et al., 2021b]. The procedure described in [Xie et al., 2021b] achieves (5.3) with probability at least  $1 - \delta$  by using
$$
\widetilde {O} \left(\frac {H ^ {3} | \mathcal {S} | C ^ {*}}{\varepsilon^ {2}} + \frac {H ^ {5 . 5} | \mathcal {S} | C ^ {*}}{\varepsilon}\right) \tag {5.4}
$$
offline samples<sup>3</sup>. In terms of offline data, our procedure has a similar dependence on various factors, but it depends on the desired accuracy  $\varepsilon$  through  $\widetilde{O}(1 / \varepsilon)$  as opposed to  $\widetilde{O}(1 / \varepsilon^2)$  which is typical for an offline algorithm. This implies that in the small- $\varepsilon$  regime, if sufficient online samples are collected, one can improve upon a fully offline procedure by collecting a number of additional online samples in a non-reactive way.
Finally, notice that one may improve upon an offline dataset by collecting more data from the distribution  $\mu$ , i.e., without performing experimental design. Compared to this latter case, notice that our online sample complexity does not depend on the concentrability coefficient. Further discussion can be found in appendix B.
3Technically, [Zhan et al., 2022] considers the non-homogeneous setting, and expresses their result in terms of number of trajectories. In obtaining eq. (5.4), we 'removed' an  $H$  factor due to our dynamics being homogeneous, and add it back to express the result in terms of number of samples. However, notice that [Zhan et al., 2022] consider the reward-aware setting, which is simpler than reward-free RL setting that we consider. This should add an additional  $|S|$  factor that is not accounted for in eq. (5.4), see the paper Jin et al. [2020b] for more details.
6 Proof
In this section we prove theorem 5.1, and defer the proofs of the supporting statements to the Appendix A.
Let us define the comparator policy  $\pi_{*}^{\dagger}$  used for the comparison in eq. (5.2) to be the (deterministic) policy with the highest value function on the sparsified MDP:
$$
\pi_{*}^{\dagger}:= \operatorname *{arg  max}_{\pi \in \Pi}V_{1}(s_{1};\mathbb{P}^{\dagger},r^{\dagger},\pi).
$$
We can bound the suboptimality using the triangle inequality as
$$
\begin{array}{l} V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi_ {*} ^ {\dagger}\right) - V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi_ {f i n a l}\right) \leq \left| V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi_ {*} ^ {\dagger}\right) - V _ {1} \left(s _ {1}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi_ {*} ^ {\dagger}\right) \right| \\ + \underbrace {V _ {1} \left(s _ {1} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi_ {*} ^ {\dagger}\right) - V _ {1} \left(s _ {1} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi_ {f i n a l}\right)} _ {\leq 0} + \left| V _ {1} \left(s _ {1} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi_ {f i n a l} - V _ {1} \left(s _ {1} ; \mathbb {P} ^ {\dagger} , r ^ {\dagger} , \pi_ {f i n a l}\right)\right) \right| \\ \leq 2 \sup  _ {\pi \in \Pi , r} \left| V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) - V _ {1} \left(s _ {1}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right|. \\ \end{array}
$$
The middle term after the first inequality is negative due to the optimality of  $\pi_{final}$  on  $\widetilde{\mathbb{P}}^{\dagger}$  and  $r^\dagger$ . It suffices to prove that for any arbitrary policy  $\pi$  and reward function  $r$  the following statement holds with probability at least  $1 - \delta$
$$
\underbrace {\left| V _ {1} \left(s _ {1} ; \mathbb {P} ^ {\dagger} , r ^ {\dagger} , \pi\right) - V _ {1} \left(s _ {1} ; \widetilde {\mathbb {P}} ^ {\dagger} , r ^ {\dagger} , \pi\right) \right|} _ {\text {E s t i m a t i o n e r r o r}} \leq \frac {\varepsilon}{2}. \tag {6.1}
$$
Bounding the estimation error using the population uncertainty function In order to prove eq. (6.1), we first define the population uncertainty function  $X$ , which is a scalar function over the state-action space. It represents the maximum estimation error on the value of any policy when it is evaluated on  $\widetilde{\mathcal{M}}^{\dagger}$  instead of  $\mathcal{M}$ . For any  $(s, a) \in S \times \mathcal{A}$ , the uncertainty function is defined as  $X_{H + 1}(s, a) \coloneqq 0$  and for  $h \in [H]$ ,
$$
X _ {h} (s, a) := \min \left\{H - h + 1; 9 H \phi (m (s, a)) + \left(1 + \frac {1}{H}\right) \sum_ {s ^ {\prime}} \widetilde {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \left(\max _ {a ^ {\prime}} \left\{X _ {h + 1} \left(s ^ {\prime}, a ^ {\prime}\right) \right\}\right) \right\}.
$$
We extend the definition to the absorbing state by letting  $X_{h}(s^{\dagger},a) = 0$  for any  $h\in [H],a\in \mathcal{A}$ . The summation  $\sum_{s'}$  used above is over  $s^\prime \in S\cup \{s^\dagger \}$ , but since  $X_{h}(s^{\dagger},a) = 0$  for any  $h\in [H],a\in \mathcal{A}$ , it is equivalent to that over  $s^\prime \in S$ . Intuitively,  $X_{h}(s,a)$  takes a similar form as Bellman optimality equation. The additional  $(1 + 1 / H)$  factor and additional term  $9H\phi (m(s,a))$  quantify the uncertainty of the true Q function on the sparsified MDP and  $9H\phi (m(s,a))$  will converge to zero when the sample size goes to infinity. This definition of uncertainty function and the following lemma follow closely from the uncertainty function defined in [Ménard et al., 2021].
The next lemma highlights the key property of the uncertainty function  $X$ , namely that for any reward function and any policy  $\pi$ , we can upper bound the estimation error via the uncertainty function at the initial times-step; it is proved in appendix A.2.1.
Lemma 6.1. With probability  $1 - \delta$ , for any reward function  $r$  and any deterministic policy  $\pi$ , it holds that
$$
\left| V _ {1} \left(s _ {1}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) - V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right| \leq \max  _ {a} X _ {1} \left(s _ {1}, a\right) + C \sqrt {\max  _ {a} X _ {1} \left(s _ {1} , a\right)}. \tag {6.2}
$$
The uncertainty function  $X$  contains the inverse number of online samples  $1 / m(s,a)$  through  $\phi (m(s,a))$ , and so lemma 6.1 expresses the estimation error in eq. (6.1) as the maximum expected size of the confidence intervals  $\sup_{\pi}\mathbb{E}_{\widetilde{\mathbb{P}}^{\dagger},(s,a)\sim \pi}\sqrt{1 / m(s,a)}$ , a quantity that directly depends on the number  $m(\cdot ,\cdot)$  of samples collected during the online phase.
Leveraging the exploration mechanics Throughout this section,  $C$  denotes some universal constant and may vary from line to line. Recall that the agent greedily minimizes the empirical uncertainty function  $U$  to compute the exploratory policy  $\pi_{ex}$ . The empirical uncertainty is defined as  $U_{H + 1}^{k}(s,a) = 0$  for any  $k$ ,  $s\in S$ ,  $a\in \mathcal{A}$  and
$$
U _ {h} ^ {k} (s, a) = H \min  \{1, \phi \left(n ^ {k} (s, a)\right) \} + \widehat {\mathbb {P}} ^ {\dagger} (s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} U _ {h + 1} ^ {k} (\cdot , a ^ {\prime})\right), \tag {6.3}
$$
where  $n^k (s,a)$  is the counter of the times we encounter  $(s,a)$  until the beginning of the  $k$ -the virtual episode in the simulation phase. Note that,  $U_h^k (s,a)$  takes a similar form as  $X_{h}(s,a)$ , except that  $U_h^k (s,a)$  depends on the empirical transition probability  $\widehat{\mathbb{P}}^\dagger$  while  $X_{h}(s,a)$  depends on the true transition probability on the sparsified MDP. For the exploration scheme to be effective,  $X$  and  $U$  should be close in value, a concept which is at the core of this work and which we formally state below and prove in appendix A.2.2.
Lemma 6.2 (Bounding uncertainty function with empirical uncertainty functions). With probability at least  $1 - \delta$ , we have for any  $(h, s, a) \in [H] \times S \times \mathcal{A}$ ,
$$
X _ {h} (s, a) \leq \frac {C}{K} \sum_ {k = 1} ^ {K} U _ {h} ^ {k} (s, a).
$$
Notice that  $X_{h}$  is the population uncertainty after the online samples have been collected, while  $U_{h}^{k}$  is the corresponding empirical uncertainty which varies during the planning phase.
Rate of decrease of the estimation error Combining lemmas 6.1 and 6.2 shows that (a function of) the agent's uncertainty estimate  $U$  upper bounds the estimation error in eq. (6.1). In order to conclude, we need to show that  $U$  decreases on average at the rate  $1 / K$ , a statement that we present below and prove in appendix A.2.3.
Lemma 6.3. With probability at least  $1 - \delta$ , we have
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} U _ {1} ^ {k} (s, a) \leq \frac {H ^ {2} | \mathcal {S} | ^ {2} | \mathcal {A} |}{K} \text {p o l y l o g} \left(K, | \mathcal {S} |, | \mathcal {A} |, H, \frac {1}{\varepsilon}, \frac {1}{\delta}\right). \tag {6.4}
$$
Then, for any  $\varepsilon > 0$ , if we take
$$
K := \frac {C H ^ {2} | \mathcal {S} | | \mathcal {A} |}{\varepsilon^ {2}} \left(\iota + | \mathcal {S} |\right) \text {p o l y l o g} \left(| \mathcal {S} |, | \mathcal {A} |, H, \frac {1}{\varepsilon}, \frac {1}{\delta}\right),
$$
then with probability at least  $1 - \delta$ , it holds that
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} U _ {1} ^ {k} (s _ {1}, a) \leq \varepsilon^ {2}.
$$
After combining lemmas 6.1 to 6.3, we see that the estimation error can be bounded as
$$
\begin{array}{l} \left| V _ {1} \left(s _ {1}; \mathbb {P} ^ {\dagger}, r ^ {\dagger}, \pi\right) - V _ {1} \left(s _ {1}; \widetilde {\mathbb {P}} ^ {\dagger}, r ^ {\dagger}, \pi\right) \right| \leq \max  _ {a} X _ {1} \left(s _ {1}, a\right) + C \sqrt {\max  _ {a} X _ {1} \left(s _ {1} , a\right)} \\ \leq C \max _ {a} \left[ \sqrt {\frac {1}{K} \sum_ {k = 1} ^ {K} U _ {1} ^ {k} (s _ {1} , a)} + \frac {1}{K} \sum_ {k = 1} ^ {K} U _ {1} ^ {k} (s _ {1} , a) \right] \\ \leq C \left(\varepsilon + \varepsilon^ {2}\right) \\ \leq C \varepsilon \quad (\text {f o r} 0 <   \varepsilon <   c o n s t) \\ \end{array}
$$
Here, the constant  $C$  may vary between lines. Rescaling the universal constant  $C$  and the failure probability  $\delta$ , we complete the upper bound in equation (6.1) and hence the proof for the main result.
Andrea Zanette. When is realizability sufficient for off-policy reinforcement learning? ICML 2023, 2023.
Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning, pages 7304-7312. PMLR, 2019.
Andrea Zanette and Martin J. Wainwright. Bellman residual orthogonalization for offline reinforcement learning, 2022. URL https://arxiv.org/abs/2203.12786.
Andrea Zanette, Alessandro Lazaric, Mykel J Kochenderfer, and Emma Brunskill. Provably efficient reward-agnostic navigation with linear value iteration. Advances in Neural Information Processing Systems, 33:11756-11766, 2020.
Andrea Zanette, Kefan Dong, Jonathan N Lee, and Emma Brunskill. Design of experiments for stochastic contextual linear bandits. Advances in Neural Information Processing Systems, 34: 22720-22731, 2021a.
Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34:13626-13640, 2021b.
Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason D Lee. Offline reinforcement learning with realizability and single-policy concentrability. arXiv preprint arXiv:2202.04634, 2022.
Ruiqi Zhang, Xuezhou Zhang, Chengzhuo Ni, and Mengdi Wang. Off-policy fitted q-evaluation with differentiable function approximators: Z-estimation and inference theory. In International Conference on Machine Learning, pages 26713-26749. PMLR, 2022.
Xuezhou Zhang, Yuzhe Ma, and Adish Singla. Task-agnostic exploration in reinforcement learning. Advances in Neural Information Processing Systems, 33:11734-11743, 2020a.
Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition. Advances in Neural Information Processing Systems, 33: 15198-15207, 2020b.
Zihan Zhang, Simon Du, and Xiangyang Ji. Near optimal reward-free reinforcement learning. In International Conference on Machine Learning, pages 12402-12412. PMLR, 2021.
1 Introduction 1
2 Related Work 3
3 Setup 3
3.1 Interaction protocol 3
4 Algorithm: balancing optimism and pessimism for experimental design 4
4.1 Intuition 4
4.2 Sparsified MDP 5
4.3 Offline design of experiments 5
4.4 Online and planning phase 6
5 Main Result 7
6 Proof 9
A Proof of the main result 18
A.1 Definitions 18
A.1.1 Sparsified MDP 18
A.1.2 High probability events 19
A.1.3 Uncertainty function and empirical uncertainty function 20
A.2 Proof of the key theorems and lemmas 21
A.2.1 Uncertainty Functions upper bounds the estimation error 21
A.2.2 Proof of lemma 6.2 22
A.2.3 Upper bounding the empirical uncertainty function (lemma 6.3) 23
A.3Omitted proofs 25
A.3.1 Proof for lemma A.4 (high probability event) 25
A.3.2 Proof for lemma A.14 (property of intermediate uncertainty function) 26
A.3.3 Proof for lemma A.15 and lemma A.16 (properties of uncertainty function) 28
A.3.4 Proof for lemma A.17, lemma A.18, lemma A.19, lemma A.20 (properties of bonus function) 29
A.3.5 Proof for lemma A.21 and lemma A.22 (properties of empirical sparsified MDP) 31
B Additional comparisons 33