SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We are now ready to formally present the offline RL problem in episodic RDPs. Assume that we have access to a batch dataset  $\mathcal{D}$  collected through interacting with an unknown (but fixed) episodic RDP  $\mathbf{R}$  using a regular behavior policy  $\pi^{\mathrm{b}}$ . We assume that  $\mathcal{D}$  comprises  $N$  episodes, where the  $k$ -th episode is of the form  $e_{0:H}^{k} = a_{0}^{k}r_{0}^{k}o_{0}^{k}\cdots a_{H}^{k}r_{H}^{k}o_{H}^{k}$ , where  $q_{0}^{k} = q_{0}$  and where, for each  $t \in [H]$ ,
$$
a _ {t} ^ {k} \sim \pi^ {\mathfrak {b}} (q _ {t} ^ {k}), \quad r _ {t} ^ {k} \sim \theta_ {\mathfrak {r}} (q _ {t} ^ {k}, a _ {t} ^ {k}), \quad o _ {t} ^ {k} \sim \theta_ {\mathfrak {o}} (q _ {t} ^ {k}, a _ {t} ^ {k}), \quad q _ {t + 1} ^ {k} = \tau (q _ {t} ^ {k}, a _ {t} ^ {k} o _ {t} ^ {k}).
$$
The goal is to compute a near-optimal policy  $\widehat{\pi}$  using the dataset  $\mathcal{D}$  (and without further exploration). More precisely, for a pre-specified accuracy  $\varepsilon \in (0,H]$ , we aim to find an  $\varepsilon$ -optimal policy  $\widehat{\pi}$ , using the smallest dataset  $\mathcal{D}$  possible. A policy  $\widehat{\pi}$  is  $\varepsilon$ -optimal iff  $\mathbb{E}_{h_0}[V_0^*(h_0) - V_0^{\widehat{\pi}}(h_0)] \leq \varepsilon$ , where  $h_0 = a_{\perp}o_0$ , for some random  $o_0 \in \mathcal{O}$ .
By virtue of Proposition 2, one may expect that it is sufficient to search for regular  $\varepsilon$ -optimal policies, which is indeed the case. In order to learn an  $\varepsilon$ -optimal policy from  $\mathcal{D}$ , some assumption is necessary regarding the policy  $\pi^{\mathrm{b}}$  that was used to collect the episodes. Let  $d_t^{\mathrm{b}} \coloneqq d_t^{\pi^{\mathrm{b}}}$  be the occupancy distribution of  $\pi^{\mathrm{b}}$ . The following assumption requires that the behavior policy assigns a positive probability to all actions, which ensures that  $\pi^{\mathrm{b}}$  explores the entire RDP.
Assumption 1.  $\min_{t\in [H],q\in \mathcal{Q}_t,a\in \mathcal{A}}d_t^{\flat}(q,a) > 0$
Assumption 1 is only needed by Theorem 6, which reconstructs the full unknown RDP. Theorem 8, instead, relies on a weaker assumption that can be expressed with the coefficient introduced in Definition 1.
The second assumption we require concerns the richness of  $\pi^{\mathrm{b}}$  and its capability to allow us to distinguish the various RDP states. This is perfectly captured by notions of distinguishability arising in automata theory, such as in Balle et al. [10]. We apply these concepts in our context, where such discrete distributions are generated from an RDP and a policy. Consider a minimal RDP  $\mathbf{R}$  with states  $\mathcal{Q} = \cup_{t\in [H + 1]}\mathcal{Q}_t$ . Given some policy  $\pi$ , at each timestep  $t\in [H]$ , every RDP state  $q\in \mathcal{Q}_t$  defines a unique probability distribution over the episode suffixes  $\mathcal{E}_{H - t} = (\mathcal{ARO})^{H - t + 1}$ . Then, the states in each  $\mathcal{Q}_t$  can be compared through the probability distributions they induce over  $\mathcal{E}_{H - t}$ . Consider any  $L = \{L_{\ell}\}_{\ell = 0}^{H}$ , where each  $L_{\ell}$  is a metric over  $\Delta (\mathcal{E}_{\ell})$ . We define the  $L$ -distinguishability of  $\mathbf{R}$  and  $\pi$  as the maximum  $\mu_0$  such that, for any  $t\in [H]$  and any two distinct  $q,q'\in \mathcal{Q}_t$ , the probability distributions over suffix traces  $e_{t:H}\in \mathcal{E}_{\ell}$  from the two states satisfy
$$
L _ {H - t} \left(\mathbb {P} \left(e _ {t: H} \mid q _ {t} = q, \pi\right), \mathbb {P} \left(e _ {t: H} \mid q _ {t} = q ^ {\prime}, \pi\right)\right) \geq \mu_ {0}.
$$
We will often omit the remaining length of the episode  $\ell = H - t$  from  $L_{\ell}$  and simply write  $L$ . We consider the  $L_{\infty}^{\mathsf{p}}$ -distinguishability, instantiating the definition above with the metric  $L_{\infty}^{\mathsf{p}}(p_1,p_2) = \max_{u\in [\ell ],e\in \mathcal{E}_u}|p_1(e*) - p_2(e*)|$ , where  $p_i(e*)$  represents the probability of the trace prefix  $e\in \mathcal{E}_u$ , followed by any trace  $e'\in \mathcal{E}_{\ell -u - 1}$ . The  $L_{1}^{\mathsf{p}}$ -distinguishability is defined analogously using  $L_{1}^{\mathsf{p}}(p_{1},p_{2}) = \sum_{u\in [\ell ],e\in \mathcal{E}_{u}}|p_{1}(e*) - p_{2}(e*)|$ . We can now require a positive distinguishability with our second assumption.
Assumption 2. The behavior policy  $\pi^{\flat}$  has  $L_{\infty}^{\mathsf{p}}$  -distinguishability of at least  $\mu_0 > 0$
Finally, in order to capture the mismatch in occupancy measure between the optimal policy and the behavior policy, we introduce a key quantity called single-policy RDP concentrability coefficient, which extends the single-policy concentrability coefficient in MDPs to RDPs:
Definition 1. The single-policy RDP concentrability coefficient of an RDP  $\mathbf{R}$  with episode horizon  $H$  and with respect to a policy  $\pi^{\mathrm{b}}$  is defined as:
$$
C _ {\mathbf {R}} ^ {*} = \max  _ {t \in [ H ], q \in \mathcal {Q} _ {t}, a o \in \mathcal {A O}} \frac {d _ {t} ^ {*} (q , a o)}{d _ {t} ^ {\mathrm {b}} (q , a o)}. \tag {2}
$$
The concentrability coefficient in Definition 1 resembles the notions of concentrability in MDPs (e.g., [14, 15]). It should be stressed, however, that those in MDPs are defined in terms of observation-action pairs  $(o,a)$ , whereas  $C_{\mathbf{R}}^{*}$  is defined in terms of hidden RDP states and actions-observations,  $(q,ao)$ . It is worth remarking that  $C_{\mathbf{R}}^{*}$  could be equivalently defined in terms of state-action pairs  $(q,a)$ . Finally, in the special case where the RDP is Markovian – in which case it coincides with an episodic MDP – we have  $\mathcal{Q} \subseteq \mathcal{O} \times [H + 1]$  and  $C_{\mathbf{R}}^{*}$  coincides with the standard single-policy concentrability coefficient for MDPs in [15]. This fact is shown in the proof of Corollary 17.
3 RegORL: Learning an Episodic RDP
In this section we present an algorithm for learning the transition function of an RDP  $\mathbf{R}$  from a dataset  $\mathcal{D}$  of episodes generated by a regular behavior policy  $\pi^{\mathrm{b}}$ . To simplify the presentation, we treat  $\mathcal{D}$  as a multiset of traces in  $\mathcal{E}_H$ . The learning agent only has access to the non-Markovian traces
Function ADACT-H(D,δ)
Input: Dataset  $\mathcal{D}$  containing  $N$  traces in  $\mathcal{E}_H$ , failure probability  $0 < \delta < 1$   
Output: Set  $\mathcal{Q}$  of RDP states, transition function  $\tau : \mathcal{Q} \times \mathcal{A}\mathcal{O} \to \mathcal{Q}$ $\mathcal{Q}_0 \gets \{q_0\}, \mathcal{X}(q_0) \gets \mathcal{D}$  // initial state  
for  $t = 0, \ldots, H$  do  
 $\mathcal{Q}_{c,t+1} \gets \{qao \mid q \in \mathcal{Q}_t, ao \in \mathcal{AO}\}$  // get candidate states  
foreach  $qao \in \mathcal{Q}_{c,t+1}$  do  $\mathcal{X}(qao) \gets \{e_{t+1:H} \mid aroe_{t+1:H} \in \mathcal{X}(q)\}$  // compute suffixes  
 $q_m a_m o_m \gets \arg \max_{qao \in \mathcal{Q}_{c,t+1}} |\mathcal{X}(qao)|$  // most common candidate  
 $\mathcal{Q}_{t+1} \gets \{q_m a_m o_m\}, \tau(q_m, a_m o_m) = q_m a_m o_m$  // promote candidate  
 $\mathcal{Q}_{c,t+1} \gets \mathcal{Q}_{c,t+1} \setminus \{q_m a_m o_m\}$  // remove from candidate states  
for  $qao \in \mathcal{Q}_{c,t+1}$  do  
Similar  $\left\{\right.$ $q' \in \mathcal{Q}_{t+1}$  | not TESTDISTINCT(t,  $\mathcal{X}(qao), \mathcal{X}(q'), \delta\}$  // confidence test  
if Similar =  $\emptyset$  then  $\mathcal{Q}_{t+1} \cup \{qao\}, \tau(q, ao) = qao$  // promote candidate  
else  $q' \gets$  element in Similar,  $\tau(q, ao) = q'$ ,  $\mathcal{X}(q') \gets \mathcal{X}(q') \cup \mathcal{X}(qao)$  // merge states  
end  
end  
return  $\mathcal{Q}_0 \cup \dots \cup \mathcal{Q}_{H+1}, \tau$   
Function TESTDISTINCT(t,  $\mathcal{X}_1, \mathcal{X}_2, \delta$ )  
return  $L_{\infty}^p(\mathcal{X}_1, \mathcal{X}_2) \geq \sqrt{2\log(8(ARO)^{H-t}/\delta)/\min(|\mathcal{X}_1|, |\mathcal{X}_2|)}$
in  $\mathcal{D}$ , and needs prior knowledge of  $\mathcal{A}$ ,  $\mathcal{R}$  and  $\mathcal{O}$ , but no prior knowledge of  $\pi^{\mathrm{b}}$  and  $\mathbf{R}$ . Our algorithm is an adaptation of ADACT [10] to episodic RDPs, and we thus refer to the algorithm as ADACT-H.
The intuition behind ADACT-H is that due to Proposition 1, two histories  $h_1$  and  $h_2$  should map to the same RDP state if they induce the same probability distribution on suffixes. ADACT-H starts by adding an initial RDP state  $q_0$  to  $\mathcal{Q}_0$ , whose suffixes are the full traces in  $\mathcal{D}$  (line 1). The algorithm then iteratively constructs the state sets  $\mathcal{Q}_1, \ldots, \mathcal{Q}_{H+1}$ . In each iteration  $t \in [H]$ , ADACT-H creates a set of candidate states  $\mathcal{Q}_{c,t+1}$  by extending all states in  $\mathcal{Q}_t$  with symbols in  $\mathcal{AO}$  (line 3). We use  $qao$  to simultaneously refer to a candidate state and its state-symbol prefix  $(q, ao)$ . We associate each candidate state  $qao$  with a multiset of suffixes  $\mathcal{X}(qao)$ , i.e. traces in  $\mathcal{E}_{H-t-1}$ , obtained by selecting all suffixes in  $\mathcal{X}(q)$  that start with action  $a$  and observation  $o$  (line 4).
Next, ADACT-H finds the candidate state whose suffix multiset has maximum cardinality, and promotes this candidate to  $\mathcal{Q}_{t + 1}$  by defining the transition function  $\tau$  accordingly (lines 5-7). The algorithm then iterates over each remaining candidate state  $qao\in \mathcal{Q}_{c,t + 1}$ , comparing the distribution on suffixes in  $\mathcal{X}(qao)$  to those of states in  $\mathcal{Q}_{t + 1}$  (line 9). If the suffix distribution is different from that of each state in  $\mathcal{Q}_{t + 1}$ ,  $qao$  is promoted to  $\mathcal{Q}_{t + 1}$  (line 10), else  $qao$  is merged with a state  $q^{\prime}\in \mathcal{Q}_{t + 1}$  that has a similar suffix distribution (line 11). Finally, ADACT-H returns the set of RDP states  $\mathcal{Q}$  and the associated transition function  $\tau$ . The function TESTDISTINCT compares two multisets  $\mathcal{X}_1$  and  $\mathcal{X}_2$  of traces in  $\mathcal{E}_{H - t - 1}$  using the metric  $L_{\infty}^{p}$ . For  $i\in \{1,2\}$  and each trace  $e\in \mathcal{E}_{H - t - 1}$ , let  $\widehat{p_i} (e) = \sum_{x\in \mathcal{X}_i}\mathbb{I}(x = e) / |\mathcal{X}_i|$  be the empirical estimate of  $p_i$ , as the proportion of elements in  $\mathcal{X}_i$  equal to  $e$ . TESTDISTINCT compares  $L_{\infty}^{p}(\mathcal{X}_1,\mathcal{X}_2)\coloneqq L_{\infty}^{p}(\widehat{p}_1,\widehat{p}_2)$  to a confidence threshold.
Markov transformation. We are now ready to connect the RDP learning phase with the MDP learning phase. RDPs do not respect the Markov property over their observations and rewards, if automaton states remain hidden. However, we can use the reconstructed transition function  $\tau$  returned by ADACT-H, extended over histories as  $\bar{\tau}:\mathcal{H}\to \mathcal{Q}$ , to recover the Markov property. In what follows we formalize the notion of Markov transformation and the properties that its outputs satisfy.
Definition 2. Let  $e_{0:H} \in \mathcal{E}_H$  be an episode collected from an RDP  $\mathbf{R}$  and a policy  $\pi^{\mathrm{b}}$  that is regular in  $\mathbf{R}$ . The Markov transformation of  $e_H$  with respect to  $\mathbf{R}$  is the episode constructed as  $a_0r_0q_1\ldots a_Hr_Hq_{H + 1}$ , where  $q_{t + 1} = \bar{\tau} (h_t)$  and  $h_t = a_0o_0\dots a_ot_t, t \in [H]$ . The Markov transformation of a dataset  $\mathcal{D}$  is the Markov transformation of all the episodes it contains.
A Markov transformation discards all observations from  $\mathcal{D}$  and replaces them with RDP states output by  $\bar{\tau}$ . The dataset so constructed can be seen as generated from an MDP, which we define next.
Definition 3. The episodic MDP associated to an episodic RDP  $\mathbf{R}$  is  $\mathbf{M}_{\mathbf{R}} = \langle \mathcal{Q},\mathcal{A},\mathcal{R},T,\theta_{\mathrm{r}},H\rangle$  where  $T(q,a,q^{\prime}) = \sum_{o\in \mathcal{O}}\mathbb{I}(q^{\prime} = \tau (q,ao))\theta_{o}(q,a,o)$  for each  $(q,a,q^{\prime})\in \mathcal{Q}\times \mathcal{A}\times \mathcal{Q}$ .
The associated MDP in Definition 3 is the decision process that corresponds to the Markov transformation of Definition 2, i.e. any episode produced with the Markov transformation can be equivalently seen as being generated from the associated MDP, in the sense of the following proposition.
Proposition 3. Let  $e_{0:H}$  be an episode sampled from an episodic RDP  $\mathbf{R}$  under a regular policy  $\pi \in \Pi_{\mathbf{R}}$ , with  $\pi(h, a) = \pi_{\mathrm{r}}(\bar{\tau}(h), a)$ . If  $e_{H}'$  is the Markov transformation of  $e_{H}$  with respect to  $\mathbf{R}$ , then  $\mathbb{P}(e_{H}' \mid \mathbf{R}, \pi) = \mathbb{P}(e_{H}' \mid \mathbf{M}_{\mathbf{R}}, \pi_{\mathrm{r}})$ , where  $\mathbf{M}_{\mathbf{R}}$  is the MDP associated to  $\mathbf{R}$ .
Rewards are not affected by the Markov transformation, only observations, implying the following.
Proposition 4. Let  $\pi \in \Pi_{\mathbf{R}}$  be a regular policy in  $\mathbf{R}$  such that  $\pi(h, a) = \pi_{\mathrm{r}}(\bar{\tau}(h), a)$ . Then  $\mathbb{E}[V_{0,\mathbf{R}}^{\pi}] = \mathbb{E}[V_{0,\mathbf{M}_{\mathbf{R}}}^{\pi_{\mathrm{r}}}]$ , where  $V_{0,\mathbf{R}}^{\pi}$  and  $V_{0,\mathbf{M}_{\mathbf{R}}}^{\pi_{\mathrm{r}}}$  are the values in the respective decision process, and  $\mathbb{E}[V_{0,\mathbf{R}}^{*}] = \mathbb{E}[V_{0,\mathbf{M}_{\mathbf{R}}}^{*}]$ , where expectations are with respect to randomness in  $o_0$ .
Corollary 5. Given  $\varepsilon \in (0, H]$ , if  $\pi_{\mathbf{r}}: \mathcal{Q} \to \Delta(\mathcal{A})$  is an  $\varepsilon$ -optimal policy of  $\mathbf{M}_{\mathbf{R}}$ , the MDP associated to some RDP  $\mathbf{R}$ , then,  $\pi(h, a) = \pi_{\mathbf{r}}(\bar{\tau}(h), a)$  is  $\varepsilon$ -optimal in  $\mathbf{R}$ .
In summary, from Proposition 3, if  $\mathcal{D}_{\mathrm{m}}$  is the Markov transformation of a dataset  $\mathcal{D}$  with respect to an RDP  $\mathbf{R}$ , then,  $\mathcal{D}_{\mathrm{m}}$  can be seen as being generated from the associated MDP  $\mathbf{M}_{\mathbf{R}}$ . Hence, any offline RL algorithm for MDPs can be used for learning in  $\mathcal{D}_{\mathrm{m}}$ . Moreover, according to Corollary 5, any solution for  $\mathbf{M}_{\mathbf{R}}$  can be translated via  $\bar{\tau}$  into a policy for the original RDP, with the same guarantees.
Complete algorithm. The complete procedure is illustrated in Algorithm 1. Initially, the input dataset  $\mathcal{D}$  is separated in two halves. The first portion is used for learning the transition function of the unknown RDP with ADACT-H (Section 3). If an upper bound  $\overline{Q}$  on  $|\mathcal{Q}|$  is available, it can optionally be provided to compute a more appropriate failure parameter for ADACT-H. If not available, we adopt the upper bound of  $2(AO)^H$  states, which is valid for any instance, due to histories having finite length. As we will see in Theorem 6, this would only contribute linearly in  $H$  to the required dataset size. The output function computed by ADACT-H is then used to compute a Markov transformation of the second phase, as specified in Definition 2. The resulting dataset, now Markovian, can be passed to a generic offline RL algorithm, which we represent with the function OFFLINERL  $(\mathcal{D},\varepsilon ,\delta)$ . In Appendix D, we instantiate it for a specific state-of-the-art offline RL algorithm.
Algorithm 1: Full procedure (RegORL)
Input: Dataset  $\mathcal{D}$ , accuracy  $\varepsilon \in (0, H]$ , failure probability  $0 < \delta < 1$ , (optionally) upper bound  $\overline{Q}$  on  $|\mathcal{Q}|$   
Output: Policy  $\hat{\pi}: \mathcal{H} \to \Delta(\mathcal{A})$
1  $\mathcal{D}_1, \mathcal{D}_2 \gets$  separate  $\mathcal{D}$  into two datasets of the same size
2  $\mathcal{Q},\tau \gets \mathrm{ADACT - H}(\mathcal{D}_1,\delta /(4AO\overline{Q}))$  , where  $\overline{Q} = 2(AO)^H$  if not provided
3  $\mathcal{D}_2^{\prime}\gets$  Markov transformation of  $\mathcal{D}_2$  with respect to  $\bar{\tau}$  as in Definition 2
4  $\hat{\pi}_{\mathrm{m}}\gets$  OFFLINERL  $(D_2^{\prime},\varepsilon ,\delta /2)$
5 return  $\hat{\pi}: h \mapsto \hat{\pi}_{\mathfrak{m}}(\bar{\tau}(h))$
We now turn to theoretical performance guarantees of RegORL. Our main performance result is a sample complexity bound in Theorem 7, ensuring that, for any accuracy  $\varepsilon \in (0,H]$ , RegORL finds an  $\varepsilon$ -optimal policy. We also report a sample complexity bound for ADACT-H in Theorem 6, and an alternative bound in Theorem 8. In comparison, the sample complexity bound for ADACT [10] is
$$
\widetilde {\mathcal {O}} \left(\frac {Q ^ {4} A ^ {2} O ^ {2} H ^ {5} \log (1 / \delta)}{\varepsilon^ {2}} \max  \left\{\frac {1}{\mu_ {0} ^ {2}}, \frac {H ^ {4} O ^ {2} A ^ {2}}{\varepsilon^ {4}} \right\}\right).
$$
We achieve a tighter bound by using Bernstein's inequalities and exploiting the finiteness of histories.
Theorem 6. Consider a dataset  $\mathcal{D}$  of episodes sampled from an RDP  $\mathbf{R}$  and a regular policy  $\pi^{\mathrm{b}}\in \Pi_{\mathbf{R}}$ . With probability  $1 - \delta$ , the output of ADACT-H  $(\mathcal{D},\delta /(2QAO))$  is the transition function of the minimal RDP equivalent to  $\mathbf{R}$ , provided that  $|\mathcal{D}|\geq N_{\delta}$ , where
$$
N _ {\delta} := \frac {2 1 \log (8 Q A O / \delta)}{d _ {\mathrm {m i n}} ^ {\tt b} \mu_ {0}} \sqrt {H \log (2 A R O)} \in \widetilde {\mathcal {O}} \left(\frac {\sqrt {H}}{d _ {\mathrm {m i n}} ^ {\tt b} \mu_ {0}}\right),
$$
$d_{\min}^{\mathsf{b}} \coloneqq \min \{d_t^{\mathsf{b}}(q,ao) \mid t \in [H], q \in \mathcal{Q}_t, ao \in \mathcal{AO}, d_t^{\mathsf{b}}(q,ao) > 0\}$  is the minimal occupancy distribution, and  $\mu_0$  is the  $L_{\infty}^{\mathsf{p}}$ -distinguishability.
The proof appears in Appendix C.2. Theorem 6 tells us that the sample complexity of ADACT-H, to return a minimal RDP, is inversely proportional to  $\mu_0$ , the  $L_{\infty}^{\mathrm{p}}$ -distinguishability of  $\mathbf{R}$  and  $\pi^{\mathrm{b}}$  and the minimal occupancy  $d_{\mathrm{min}}^{\mathrm{b}}$ . Note that  $d_{\mathrm{min}}^{\mathrm{b}} \leq 1/(QOA)$ . The bound also depends on  $Q$ , the number of RDP states, implicitly through  $d_{\mathrm{min}}^{\mathrm{b}}$  and explicitly via a logarithmic term. In the absence of prior knowledge of  $Q$ , one may use in the argument of Algorithm 1 the worst-case upper bound  $\overline{Q} = 2(AO)^H$ . The sample complexity would then have an additional linear term in  $H$ , since  $\overline{Q}$  is only used in the logarithmic term to set the appropriate value of  $\delta$ . However, this will not impact the value of the  $d_{\mathrm{min}}^{\mathrm{b}}$  term.
Theorem 6 is a sample complexity guarantee for the first phase of the algorithm, which learns  $\tau$ , the structure of the minimal RDP that is equivalent to the underlying RDP. If  $\delta$  is the desired failure probability of the complete algorithm, RegORL executes ADACT-H so that its success probability is at least  $1 - \delta /2$ . This means that with the same probability,  $\mathcal{D}_2^{\prime}$  is an MDP dataset with the properties listed in Section 3. As a consequence, provided that OFFLINERL is some generic  $(\varepsilon ,\delta /2)$ -PAC offline RL algorithm for MDPs, the output of RegORL is an  $\varepsilon$ -optimal policy with probability  $1 - \delta$ .
Theorem 7. Consider a dataset  $\mathcal{D}$  of episodes sampled from an RDP  $\mathbf{R}$  and a regular policy  $\pi^{\mathrm{b}}\in \Pi_{\mathbf{R}}$ . For any  $\varepsilon \in (0,H]$  and  $0 < \delta < 1$ , if OFFLINE is an  $(\varepsilon ,\delta /2)$ -PAC offline algorithm for MDPs with sample complexity  $N_{\mathfrak{m}}$ , then, the output of RegORL  $(\mathcal{D},\varepsilon ,\delta)$  is an  $\varepsilon$ -optimal policy in  $\mathbf{R}$ , with probability at least  $1 - \delta$ , provided that  $|\mathcal{D}|\geq 2\max \{N_{\delta /2},N_{\mathfrak{m}}\}$ .
As we can see, the sample complexity requirement separates for the two phases. While  $N_{\delta /2}$  is due to the RDP learning component, defined in Theorem 6, the quantity  $N_{\mathrm{m}}$  completely depends on the offline RL algorithm for MDPs that is adopted. Among other terms, the performance guarantees of offline algorithms can often be characterized through the single-policy concentrability for MDPs  $C^*$ . However, since states become observations in the associated MDP, due to the properties of Proposition 3,  $C^*$  coincides with  $C_{\mathbf{R}}^*$ , the RDP single-policy concentrability of Definition 1.
In Appendix D, we demonstrate a specific instantiation of RegORL with an off-the-shelf offline RL algorithm from the literature by Li et al. [16]. This yields the following requirement for  $N_{\mathfrak{m}}$ :
$$
N _ {\mathrm {m}} \geq \frac {c H ^ {3} Q C _ {\mathbf {R}} ^ {*} \log \frac {2 H N _ {\mathrm {m}}}{\delta}}{\varepsilon^ {2}},
$$
for a constant  $c > 0$
To eliminate the dependence that Theorem 6 has on  $d_{\mathrm{min}}^{\mathrm{b}}$ , we develop a variant of ADACT-H which does not learn a complete RDP. Rather, it only reconstructs a subset of states that are likely under the behavior policy. The algorithm, which we call ADACT-H-A (with 'A' standing for "approximation"), is defined in Appendix C.3. Theorem 8 is an upper bound on the sample complexity of ADACT-H-A that takes the accuracy  $\varepsilon$  as input and returns the transition function of an  $\varepsilon /2$ -approximate RDP  $\mathbf{R}'$ , whose optimal policy is  $\varepsilon /2$ -optimal for the original RDP  $\mathbf{R}$ . By performing a Markov transformation for  $\mathbf{R}'$  and using an  $(\varepsilon /2,\delta /2)$ -PAC offline algorithm for MDPs, we can compute an  $\varepsilon$ -optimal policy for  $\mathbf{R}$ . The total sample complexity can be combined in the same way as in Theorem 7.
Theorem 8. Consider a dataset  $\mathcal{D}$  of episodes sampled from an RDP  $\mathbf{R}$  and a regular policy  $\pi^{\mathrm{b}}\in \Pi_{\mathbf{R}}$ . With probability  $1 - \delta$ , the output of ADACT-H-A, called with  $\mathcal{D}$ ,  $\delta /(2QAO)$  and  $\varepsilon \in (0,H]$  in input, is the transition function of an  $\varepsilon /2$ -approximate RDP  $\mathbf{R}'$ , provided that  $|\mathcal{D}|\geq N_{\delta}'$ , where
$$
N _ {\delta} ^ {\prime} := \frac {5 0 4 H Q A O C _ {\mathbf {R} ^ {\prime}} ^ {*} \log (1 6 Q A O / \delta)}{\varepsilon \mu_ {0}} \sqrt {H \log (2 A R O)} \in \widetilde {\mathcal {O}} \left(\frac {H ^ {3 / 2} Q A O C _ {\mathbf {R} ^ {\prime}} ^ {*}}{\varepsilon \mu_ {0}}\right).
$$
This theorem does not rely on Assumption 1, because a finite  $C_{\mathbf{R}}^{*}$  suffices.
5 Sample Complexity Lower Bound
The main result of this section is Theorem 9, a sample complexity lower bound for offline RL in RDPs. It shows that the dataset size required by any RL algorithm scales with the relevant parameters.
Figure 2: One episodic RDP instance  $\mathbf{R}_{101,1} \in \mathbb{R}(L, H, \xi, \eta)$ , associated to the parity function  $f_{101}$  with code 101, and the optimal arm  $a_1'$ . The length is  $L = |101| = 3$ , the horizon  $H = 5$ , the noise parameter  $\xi$  and the bandit bonus parameter is  $\eta$ . The transition function only depends on the observations, not the actions. The output distributions are:  $u = \mathrm{unif}\{0,1\}$ ,  $u_+ = \mathrm{unif}\{+, -\}$ ,  $v_\alpha(+) = (1 + \alpha)/2$ ,  $v_\alpha(-) = (1 - \alpha)/2$ . The star denotes any symbol. If the label of a state  $q$  is  $a.d$ , then the observation function is  $\theta_\mathrm{o}(q, a) = d$ . Refer to Appendix E for details.
Theorem 9. For any  $(C_{\mathbf{R}}^{*},H,\varepsilon ,\mu_{0})$  satisfying  $C_\mathbf{R}^*\geq 2$ $H\geq 2$  and  $\varepsilon \le H\mu_0 / 64$  there exists an RDP with horizon  $H$ $L_{1}^{\mathrm{p}}$  -distinguishability  $\mu_0$  and a regular behavior policy  $\pi^{\mathrm{b}}$  with RDP single-policy concentrability  $C_{\mathbf{R}}^{*}$  such that if  $\mathcal{D}$  has been generated using  $\pi^{\mathrm{b}}$  and  $\mathbf{R}$ , and