SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ V _ {0, \mathbf {R}} ^ {\pi} \right] = \sum_ {r _ {1} \dots r _ {H} \in \mathcal {R} ^ {H + 1}} \mathbb {P} \left(r _ {1: H} = r _ {1} \dots r _ {H} \mid \mathbf {R}, \pi\right) \sum_ {i = 1} ^ {H} r _ {i} (10) \\ = \sum_ {e ^ {\prime} \in \mathcal {E} _ {H} ^ {\prime}} \mathbb {P} \left(\phi \left(e _ {H}\right) = e ^ {\prime} \mid \mathbf {R}, \pi\right) \sum_ {i = 1} ^ {H} r _ {i} (11) \\ = \sum_ {e ^ {\prime} \in \mathcal {E} _ {H} ^ {\prime}} \mathbb {P} \left(e _ {H} ^ {\prime} = e ^ {\prime} \mid \mathbf {M} _ {\mathbf {R}}, \pi_ {\mathrm {r}}\right) \sum_ {i = 1} ^ {H} r _ {i} (12) \\ = \mathbb {E} \left[ V _ {0, \mathbf {M} _ {\mathbf {R}}} ^ {\pi_ {r}} \right] (13) \\ \end{array}
$$
For the second part of the statement, let  $\Pi_{\mathbf{R}}$  and  $\Pi_{\mathbf{M}}$  be the regular and the Markov policies in  $\mathbf{R}$  and  $\mathbf{M}_{\mathbf{R}}$ , respectively. Then, using Proposition 2 and the first part of this statement,
$$
V _ {0, \mathbf {R}} ^ {*} = \max  _ {\pi \in \Pi_ {\mathbf {R}}} \mathbb {E} \left[ V _ {0, \mathbf {R}} ^ {\pi} \right] = \max  _ {\pi \in \Pi_ {\mathbf {R}}} \mathbb {E} \left[ V _ {0, \mathbf {M} _ {\mathbf {R}}} ^ {\pi_ {\mathrm {r}}} \right] = \max  _ {\pi_ {\mathrm {r}} \in \Pi_ {\mathbf {M}}} \mathbb {E} \left[ V _ {0, \mathbf {M} _ {\mathbf {R}}} ^ {\pi_ {\mathrm {r}}} \right] = V _ {0, \mathbf {M} _ {\mathbf {R}}} ^ {*} \tag {14}
$$
Corollary 5 is a consequence of the two parts of Proposition 4. Since  $V_{0,\mathbf{R}}^{\pi} = V_{0,\mathbf{M}_{\mathbf{R}}}^{\pi_{\mathrm{r}}}$  and the value achieved by the optimal policy in the respective model is the same,  $\varepsilon$ -optimality of  $\pi$  in  $\mathbf{R}$  implies  $\varepsilon$ -optimality of  $\pi_{\mathrm{r}}$  in  $\mathbf{M}_{\mathbf{M}}$ , and vice versa.
C Sample Complexity of ADACT-H
In this section we prove high-probability upper bounds on the sample complexity of ADACT-H.
We first state Hoeffding's inequality for Bernoulli variables. In what follows we take log to be the natural logarithm.
Lemma 10 (Hoeffding's inequality). Let  $X_{1},\ldots ,X_{N}$  be  $N$  independent random Bernoulli variables with the same expected value  $\mathbb{E}[X_1] = p$ , and let  $\widehat{p}_N = \sum_{i = 1}^N X_i / N$  be an empirical estimate of  $p$ . Then, for any  $\delta \in (0,1)$ ,
$$
\mathbb {P} \left(| \widehat {p} _ {N} - p | \geq \sqrt {\frac {\log (2 / \delta)}{2 N}}\right) \leq \delta . \tag {15}
$$
An alternative to Hoeffding's inequality is the empirical Bernstein inequality, which can be expressed as follows for Bernoulli variables [70, 71].
Lemma 11 (Empirical Bernstein inequality). Let  $X_{1},\ldots ,X_{N}$  be  $N$  independent random Bernoulli variables with the same expected value  $\mathbb{E}[X_1] = p$ , and let  $\widehat{p}_N = \sum_{i = 1}^N X_i / N$  be an empirical estimate of  $p$ . Then, for any  $\delta \in (0,1)$ ,
$$
\mathbb {P} \left(| \widehat {p} _ {N} - p | \geq \sqrt {\frac {2 \widehat {p} \log (4 / \delta)}{N}} + \frac {1 4 \log (4 / \delta)}{3 N}\right) \leq \delta . \tag {16}
$$
If  $X \sim p_{X}$  is a discrete random variable, the entropy of  $X$  is  $H(X) = -\sum_{x \in \mathcal{X}} p_{X}(x) \log p_{X}(x)$ . Further, for  $x \in (0,1)$ , we define the binary entropy function as  $H_{2}(x) = -x \log(x) - (1 - x) \log(1 - x)$ . If  $(X,Y) \sim p_{XY}$  are two discrete variables, the conditional entropy is  $H(Y \mid X) = \sum_{x \in \mathcal{X}} p_{X}(x) H(Y \mid X = x)$ . The mutual information is  $I(X;Y) = I(Y;X) = D_{\mathrm{KL}}(p_{XY} \parallel p_{X} \cdot p_{Y})$ , where  $D_{\mathrm{KL}}$  is the Kullback-Leibler divergence. If  $X,Y,Z$  are three random variables, we write  $X \to Y \to Z$  if the conditional distribution of  $Z$  does not depend on  $X$ , given  $Y$ . With these definitions, we state Fano's inequality, as one can find in Cover and Thomas [72], (2.140).
Theorem 12 (Fano's inequality). Let  $X \to Y \to \hat{X}$ , for  $X, \hat{X} \in \mathcal{X}$  and  $P_e = \mathbb{P}(\hat{X} \neq X)$ . Then,
$$
H _ {2} \left(P _ {e}\right) + P _ {e} \log (| \mathcal {X} | - 1) \geq H (X \mid Y). \tag {17}
$$
In this section we prove Theorem 6, which states a high-probability upper bound on the sample complexity of ADACT-H. The first two lemmas are adaptations of Lemmas 19 and 20 in Balle et al. [10] to the episodic setting.
Lemma 13. For  $t \in [H]$ , let  $\mathcal{X}_1$  and  $\mathcal{X}_2$  be multisets sampled from distributions  $p_1$  and  $p_2$  in  $\Delta(\mathcal{E}_{H - t - 1})$ . If  $p_1 = p_2$ , then TESTDISTINCT(t,  $\mathcal{X}_1, \mathcal{X}_2, \delta$ ) returns False with probability  $1 - \delta$ .
Proof. For each  $i \in \{1,2\}$  and each trace  $e \in \mathcal{E}_{H - t - 1}$ , we can view each episode as a random Bernoulli variable with expected value  $p_i(e)$  that takes value 1 if we observe  $e$ , and 0 otherwise. Let  $\widehat{p}_i(e) = \sum_{x \in \mathcal{X}_i} \mathbb{I}(x = e) / |\mathcal{X}_i|$  be the empirical estimate of  $p_i$ , i.e. the proportion of elements in  $\mathcal{X}_i$  equal to  $e$ . For each  $i \in \{1,2\}$ , each  $u \in [H - t - 1]$  and each prefix  $e_{0:u} \in \mathcal{E}_u$ , Hoeffding's inequality yields
$$
\mathbb {P} \left(| \widehat {p _ {i}} (e _ {0: u} *) - p _ {i} (e _ {0: u} *) | \geq \sqrt {\frac {\log (2 / \delta_ {\mathbf {s}})}{2 | \mathcal {X} _ {i} |}}\right) \leq \delta_ {\mathbf {s}}.
$$
The total number of non-empty prefixes of  $\mathcal{E}_{H - t - 1}$  equals a geometric sum:
$$
(A R O) ^ {1} + \dots + (A R O) ^ {H - t} = \frac {(A R O) ^ {H + 1 - t} - 1}{A R O - 1} - 1 \leq 2 (A R O) ^ {H - t}.
$$
Choosing  $\delta_{\mathfrak{s}} = \delta /4(ARO)^{H - t}$  and taking a union bound implies that the above inequality holds for each  $i\in \{1,2\}$  and each  $e_{0:u}$  simultaneously with probability  $1 - 4(ARO)^{H - t}\delta_{\mathfrak{s}} = 1 - \delta$  implying
$$
\begin{array}{l} L _ {\infty} ^ {\mathsf {p}} (\mathcal {X} _ {1}, \mathcal {X} _ {2}) = \max _ {u, e _ {0: u}} | \widehat {p _ {1}} (e _ {0: u} *) - \widehat {p _ {2}} (e _ {0: u} *) | \leq L _ {\infty} ^ {\mathsf {p}} (p _ {1}, p _ {2}) + \sqrt {\frac {\log (2 / \delta_ {\mathbf {s}})}{2 | \mathcal {X} _ {1} |}} + \sqrt {\frac {\log (2 / \delta_ {\mathbf {s}})}{2 | \mathcal {X} _ {2} |}} \\ \leq 0 + 2 \sqrt {\frac {\log (2 / \delta_ {\mathfrak {s}})}{2 \min (| \mathcal {X} _ {1} | , | \mathcal {X} _ {2} |)}} = \sqrt {\frac {2 \log (8 (A R O) ^ {H - t} / \delta)}{\min (| \mathcal {X} _ {1} | , | \mathcal {X} _ {2} |)}}, \\ \end{array}
$$
which is precisely the condition under which TESTDISTINCT  $(t,\mathcal{X}_1,\mathcal{X}_2,\delta)$  returns False.
Lemma 14. For  $t \in [H]$ , let  $\mathcal{X}_1$  and  $\mathcal{X}_2$  be multisets sampled from distributions  $p_1$  and  $p_2$  in  $\Delta(\mathcal{E}_{H - t - 1})$ . If the  $L_{\infty}^{\mathsf{p}}$ -distinguishability of  $\pi^{\mathsf{b}}$  is  $\mu_0$ , then TESTDISTINCT(  $t, \mathcal{X}_1, \mathcal{X}_2, \delta$ ) returns True with probability  $1 - \delta$  provided that
$$
\min  \left(\left| \mathcal {X} _ {1} \right|, \left| \mathcal {X} _ {2} \right|\right) \geq \frac {8}{\mu_ {0} ^ {2}} \left(\log \left(2 \left(A R O\right) ^ {H - t}\right) + \log (4 / \delta)\right).
$$
Proof. Using the same argument as in the proof of Lemma 13, Hoeffding's inequality yields
$$
\mathbb {P} \left(\left| \widehat {p _ {i}} \left(e _ {0: u} *\right) - p _ {i} \left(e _ {0: u} *\right) \right| > \sqrt {\frac {\log \left(2 / \delta_ {\mathbf {s}}\right)}{2 | \mathcal {X} _ {i} |}}\right) \leq \delta_ {\mathbf {s}},
$$
with the inequality holding simultaneously for  $i \in \{1,2\}$  and each prefix  $e_{0:u}$  with probability  $1 - \delta$  by choosing  $\delta_{\mathbf{s}} = \delta /4(ARO)^{H - t}$ . Choosing  $\mu_0 \geq 4\sqrt{\log(2 / \delta_{\mathbf{s}}) / 2|\mathcal{X}_i|}$  for each  $i \in \{1,2\}$  yields
$$
\left| \mathcal {X} _ {i} \right| \geq \min  \left(\left| \mathcal {X} _ {1} \right|, \left| \mathcal {X} _ {2} \right|\right) \geq \frac {8}{\mu_ {0} ^ {2}} \log \left(2 / \delta_ {\mathfrak {s}}\right) = \frac {8}{\mu_ {0} ^ {2}} \left(\log \left(2 \left(A R O\right) ^ {H - t}\right) + \log \left(4 / \delta\right)\right).
$$
In this case we have
$$
\begin{array}{l} L _ {\infty} ^ {p} \left(\mathcal {X} _ {1}, \mathcal {X} _ {2}\right) = \max  _ {u, e _ {0: u}} | \widehat {p} _ {1} (e) - \widehat {p} _ {2} (e) | \geq L _ {\infty} ^ {p} \left(p _ {1}, p _ {2}\right) - \sqrt {\frac {\log \left(2 / \delta_ {\mathfrak {s}}\right)}{2 \left| \mathcal {X} _ {1} \right|}} - \sqrt {\frac {\log \left(2 / \delta_ {\mathfrak {s}}\right)}{2 \left| \mathcal {X} _ {2} \right|}} \\ \geq \mu_ {0} - \frac {\mu_ {0}}{4} - \frac {\mu_ {0}}{4} = \frac {\mu_ {0}}{2} \geq 2 \sqrt {\frac {\log (2 / \delta_ {\mathsf {s}})}{2 \min (| \mathcal {X} _ {1} | , | \mathcal {X} _ {2} |)}} = \sqrt {\frac {2 \log (8 (A R O) ^ {H - t} / \delta)}{\min (| \mathcal {X} _ {1} | , | \mathcal {X} _ {2} |)}}, \\ \end{array}
$$
which is precisely the condition under which TESTDISTINCT  $(t,\mathcal{X}_1,\mathcal{X}_2,\delta)$  returns True.
We are now ready to prove Theorem 6, which we restate below:
$$
N _ {\delta} := \frac {2 1 \log (8 Q A O / \delta)}{d _ {\mathrm {m i n}} ^ {\mathsf {b}} \mu_ {0}} \sqrt {H \log (2 A R O)} \in \widetilde {\mathcal {O}} \left(\frac {\sqrt {H}}{d _ {\mathrm {m i n}} ^ {\mathsf {b}} \mu_ {0}}\right),
$$
Proof. The proof consists in choosing  $N$  and  $\delta$  such that the condition in Lemma 14 is true with high probability for each application of TESTDISTINCT. Consider an iteration  $t\in [H]$  of ADACT-H. For a candidate state  $qao\in \mathcal{Q}_{\mathrm{c},t + 1}$ , its associated probability is  $d_t^{\mathrm{b}}(q,ao)$  with empirical estimate  $\widehat{p}_t(qao) = |\mathcal{X}(qao)| / N$ , i.e. the proportion of episodes in  $\mathcal{D}$  that are consistent with  $qao$ . We can apply the empirical Bernstein inequality in (16) to show that
$$
\mathbb {P} \left(\left| \widehat {p} _ {t} (q a o) - d _ {t} ^ {\mathrm {b}} (q, a o) \right| \geq \sqrt {\frac {2 \widehat {p} _ {t} (q a o) \ell}{N}} + \frac {1 4 \ell}{3 N} = \frac {\sqrt {2 M \ell} + 1 4 \ell / 3}{N}\right) \leq \delta ,
$$
where  $M = |\mathcal{X}(qao)|$ ,  $\ell = \log (4 / \delta)$ , and  $\delta$  is the failure probability of ADACT-H. To obtain a bound on  $M$  and  $N$ , assume that we can estimate  $d_t^{\tt b}(q,ao)$  with accuracy  $d_{t}^{\tt b}(q,ao) / 2$ , which yields
$$
\frac {d _ {t} ^ {\mathrm {b}} (q , a o)}{2} \geq \frac {\sqrt {2 M \ell} + 1 4 \ell / 3}{N} \tag {18}
$$
$$
\widehat {p} _ {t} (q a o) \geq d _ {t} ^ {\mathrm {b}} (q, a o) - \frac {\sqrt {2 M \ell} + 1 4 \ell / 3}{N} \geq d _ {t} ^ {\mathrm {b}} (q, a o) - \frac {d _ {t} ^ {\mathrm {b}} (q , a o)}{2} = \frac {d _ {t} ^ {\mathrm {b}} (q , a o)}{2}. \tag {19}
$$
Combining these two results, we obtain
$$
M = N \widehat {p} _ {t} (q a o) \geq N d _ {t} ^ {\mathrm {b}} (q, a o) / 2 \geq \frac {N}{2 N} \left(\sqrt {2 M \ell} + 1 4 \ell / 3\right) = \frac {1}{2} \left(\sqrt {2 M \ell} + 1 4 \ell / 3\right). \tag {20}
$$
Solving for  $M$  yields  $M \geq 4\ell$ , which is subsumed by the bound on  $M$  in Lemma 14 since  $\mu_0 < 1$ . Hence the bound on  $M$  in Lemma 14 is sufficient to ensure that we estimate  $d_t^{\tt b}(q,ao)$  with accuracy  $d_t^{\tt b}(q,ao) / 2$ . We can now insert the bound on  $M$  from Lemma 14 into (18) to obtain a bound on  $N$ :
$$
N \geq \frac {2 (\sqrt {2 M \ell} + 1 4 \ell / 3)}{d _ {t} ^ {\mathrm {b}} (q , a o)} \geq \frac {2 \ell}{d _ {t} ^ {\mathrm {b}} (q , a o)} \left(\frac {4}{\mu_ {0}} \sqrt {\frac {(H - t) \log (2 A R O)}{\ell} + 1} + \frac {1 4}{3}\right) \equiv N _ {1}. \tag {21}
$$
To simplify the bound, we can choose any value larger than  $N_{1}$ :
$$
\begin{array}{l} N _ {1} \leq \frac {2 \ell}{d _ {t} ^ {\mathrm {b}} (q , a o)} \left(\frac {4}{\mu_ {0}} \sqrt {H \log (2 A R O) + H \log (2 A R O)} + \frac {1 4}{3 \mu_ {0}} \sqrt {H \log (2 A R O)}\right) \\ <   \frac {2 1 \ell}{d _ {\min } ^ {\mathrm {b}} \mu_ {0}} \sqrt {H \log (2 A R O)} \equiv N _ {0}, \tag {22} \\ \end{array}
$$
where we have used  $d_{t}^{\mathrm{b}}(q,ao)\geq d_{\mathrm{min}}^{\mathrm{b}}$ $\mu_0 < 1,\ell = \log 4 + \log (1 / \delta)\geq 1,H\log (2ARO)\geq \log 4\geq$ $4\sqrt{2} +14 / 3 <   \frac{21}{2}$  . Choosing  $\delta = \delta_0 / 2QAO$  , a union bound implies that accurately estimating  $d_{t}^{\mathrm{b}}(q,ao)$  for each candidate state qao and accurately estimating  $p(e_{0:u}*)$  for each prefix in the multiset  $\mathcal{X}(qao)$  associated with qao occurs with probability  $1 - 2QA O\delta = 1 - \delta_0$  , since there are at most  $QAO$  candidate states. Substituting the expression for  $\delta$  in  $N_{0}$  yields the bound in the theorem.
It remains to show that the resulting RDP is minimal. We show the result by induction. The base case is given by the set  $\mathcal{Q}_0$ , which is clearly minimal since it only contains the initial state  $q_0$ . For  $t \in [H]$ , assume that the algorithm has learned a minimal RDP for sets  $\mathcal{Q}_0, \ldots, \mathcal{Q}_t$ . Let  $\mathcal{Q}_{t+1}$  be the set of states at layer  $t+1$  of a minimal RDP. Due to Proposition 1, each pair of histories that map to a state  $q_{t+1} \in \mathcal{Q}_{t+1}$  generate the same probability distribution over suffixes. Hence by Lemma 13, with high probability TESTDISTINCT(t,  $\mathcal{X}(qao), \mathcal{X}(q'a'o')$ ,  $\delta$ ) returns false for each pair of candidate states  $qao$  and  $q'a'o'$  that map to  $q_{t+1}$ . Consequently, the algorithm
merges  $qao$  and  $q'a'o'$ . On the other hand, by assumption, each pair of histories that map to different states of  $\mathcal{Q}_{t+1}$  have  $L_{\infty}^p$ -distinguishability  $\mu_0$ . Hence by Lemma 14, with high probability TESTDISTINCT(t,  $\mathcal{X}(qao)$ ,  $\mathcal{X}(q'a'o')$ ,  $\delta$ ) returns true for each pair of candidate states  $qao$  and  $q'a'o'$  that map to different states in  $\mathcal{Q}_{t+1}$ . Consequently, the algorithm does not merge  $qao$  and  $q'a'o'$ . It follows that with high probability, ADACT-H will generate exactly the set  $\mathcal{Q}_{t+1}$ , which is that of a minimal RDP.
C.3 Proof of Theorem 8
In this section we prove Theorem 8, which states an alternative upper bound on the sample complexity of ADACT-H. The proof requires an alternative definition of the algorithm, which we call ADACT-H-A (for "approximation").
Function ADACT-H-A(D,  $\delta ,\varepsilon ,\overline{Q},\overline{C})$
Input: Dataset  $\mathcal{D}$  , failure probability  $0 <   \delta <  1$  , accuracy  $\varepsilon$  , upper bounds  $\overline{Q}$  on  $|\mathcal{Q}^{\prime}|$  and  $\overline{C}$  on  $C_{\mathbf{R}}^*$  Output: Set  $\mathcal{Q}'$  of RDP states, transition function  $\tau^{\prime}:\mathcal{Q}^{\prime}\times \mathcal{A}\mathcal{O}\to \mathcal{Q}^{\prime}$  of an approximate RDP  $\mathbf{R}'$    
1  $\mathcal{Q}_0^\prime \gets \{q_0\} ,\mathcal{X}(q_0)\gets \mathcal{D}$  // initial state   
2  $\mathcal{Q}_0^\prime \leftarrow \mathcal{Q}_0^\prime \cup \{q_0^e\} ,\mathcal{X}(q_0^e)\gets \emptyset$  // initial side state   
3 for  $t = 0,\dots ,H$  do   
4  $\mathcal{Q}_{t + 1}^{\prime}\gets \{q_{t + 1}^{e}\}$  // side state   
5 foreach ao  $\in$  AO do  $\tau '(q_t^e,a_o) = q_{t + 1}^e,\mathcal{X}(q_{t + 1}^e)\gets \{e_{t + 1:H}\mid aro_{t + 1:H}\in \mathcal{X}(q_t^e)\}$    
6  $\mathcal{Q}_{c,t + 1}^{\prime}\gets \{qao\mid q\in \mathcal{Q}_t^\prime ,ao\in \mathcal{AO}\}$  // get candidate states   
7 foreach qao  $\in$  Qc,t+1 do X(qao)  $\leftarrow$  {et+1:H | aroet+1:H E X(q)} // compute suffixes   
8 qamom  $\leftarrow$  arg maxqao∈Qc,t+1|X(qao)| // most common candidate   
9 Qt+1 O t+1 U {qmamom}, T(qm, amom) = qamom // promote candidate   
10 Qc,t+1 Cc,t+1 {qmamom} // remove from candidate states   
11 for qao E Qc,t+1 such that |X(qao)/N E (4QAOHC) do Similar  $\leftarrow \{q^{\prime}\in \mathcal{Q}_{t + 1}^{\prime}\mid$  not TESTDISTINCT(t, X(qao), X(q'), δ)} // confidence test if Similar  $= \emptyset$  then  $Q_{t + 1}^{\prime}\gets Q_{t + 1}^{\prime}\cup \{qao\} ,\tau^{\prime}(q,ao) = qao$  // promote candidate else  $q^{\prime}\gets$  element in Similar,  $\tau^{\prime}(q,ao) = q^{\prime},\mathcal{X}(q^{\prime})\gets \mathcal{X}(q^{\prime})\cup \mathcal{X}(qao)$  // merge states if  $|\mathcal{Q}_0^\prime | + \dots +|\mathcal{Q}_{t + 1}^\prime | > \overline{Q}$  then return Failure   
end   
for qao E Qc,t+1 such that |X(qao)/N < E/(4QAOHC) do   
1 | T(q,ao) = qe+1, X(qe+1) - X(qe+1) U X(qao) // merge with side state   
19 end   
20 return Q'U...U Q_H+1, T'
Proof. ADACT-H-A returns the set of RDP states  $\mathcal{Q}'$  and transition function  $\tau'$  of an approximate RDP  $\mathbf{R}'$ , taking as input the accuracy  $\varepsilon$ , an upper bound  $\overline{Q}$  on  $|\mathcal{Q}'|$ , and an upper bound  $\overline{C}$  on the concentrability  $C_{\mathbf{R}'}^*$  of  $\mathbf{R}'$ . If, at any moment, the number of RDP states  $|\mathcal{Q}'|$  exceeds  $\overline{Q}$ , the algorithm returns Failure (line 15). ADACT-H-A defines a sequence of side states  $q_0^{\mathrm{e}}, \ldots, q_{H+1}^{\mathrm{e}}$  (lines 2 and 4), and defines  $\tau'(q_t^{\mathrm{e}}, ao) = q_{t+1}^{\mathrm{e}}$  for each  $t \in [H]$  and  $ao \in \mathcal{A}\mathcal{O}$  (line 5). For each candidate state  $qao \in \mathcal{Q}_{c,t+1}'$  such that  $|\mathcal{X}(qao)| / N \geq \varepsilon / (4\overline{Q}AOH\overline{C})$ , the definition of ADACT-H-A is the same as that of ADACT-H, including the call to TESTDISTINCT (lines 11-14). For each candidate state  $qao \in \mathcal{Q}_{c,t+1}'$  such that  $|\mathcal{X}(qao)| / N < \varepsilon / (4\overline{Q}AOH\overline{C})$ , instead of mapping  $(q, ao)$  to the correct RDP state, ADACT-H-A maps  $(q, ao)$  to the side state  $q_{t+1}^{\mathrm{e}}$  (lines 17-18). Once in  $q_{t+1}^{\mathrm{e}}$ ,  $\mathbf{R}'$  remains in a side state for the rest of the episode. The side states do not satisfy Proposition 1, since the
histories that map to side states may assign different probabilities to suffixes (and TESTDISTINCT is never called).
We define an alternative occupancy measure  $d_t'(q, ao)$  associated with the approximate RDP  $\mathbf{R}'$  and the behavior policy  $\pi^{\mathrm{b}}$ . The new definition is given by  $d_0'(q_0, a_0 o_0) = \theta_{\circ}(q_0, a_0, o_0)$  and
$$
d _ {t} ^ {\prime} (q _ {t}, a _ {t} o _ {t}) = \sum_ {(q, a o) \in \tau^ {\prime - 1} (q _ {t})} d _ {t - 1} ^ {\prime} (q, a o) \cdot \pi^ {\flat} (q _ {t}, a _ {t}) \cdot \theta_ {\mathfrak {o}} (q _ {t}, a _ {t}, o _ {t}), t > 0.
$$
The only difference between  $d_t'$  and  $d_t^b$  is that  $d_t'$  is defined with respect to the transition function  $\tau'$  of the approximate RDP  $\mathbf{R}'$ , instead of the transition function  $\tau$  associated with the original RDP  $\mathbf{R}$ . Note that apart from the side states,  $\mathbf{R}'$  will contain the same states as  $\mathbf{R}$ , as long as the candidate states satisfy the condition on line 11, and  $\tau'$  will be the same as  $\tau$  on those states. Since the states and behavior policy are the same, the  $L_{\infty}^{\mathrm{p}}$ -distinguishability  $\mu_0$  of  $\mathbf{R}'$  will be the same as that of  $\mathbf{R}$ .
First consider each candidate state  $qao \in \mathcal{Q}_{\mathrm{c},t+1}^{\prime}$  such that  $|\mathcal{X}(qao)| / N \geq \varepsilon / (4\overline{Q}AOH\overline{C})$ . In this case, ADACT-H-A calls TESTDISTINCT, so Lemmas 13 and 14 apply to these candidate states. The associated occupancy is  $d_t'(q,ao)$  with empirical estimate  $\widehat{p}_t(qao) = |\mathcal{X}(qao)| / N$ . Hence the empirical Bernstein inequality applies to  $d_t'(q,ao)$  and  $\widehat{p}_t(qao)$ . Just as in the proof of Theorem 6, we choose  $\mathcal{X}(qao)$  large enough to accurately estimate  $d_t'(q,ao)$  within a factor  $d_t'(q,ao) / 2$  with probability  $1 - \delta$ . We thus obtain an alternative upper bound on  $d_t'(q,ao)$  as follows:
$$
d _ {t} ^ {\prime} (q, a o) \geq \frac {| \mathcal {X} (q a o) |}{N} - \frac {d _ {t} ^ {\prime} (q , a o)}{2} \Leftrightarrow \frac {3 d _ {t} ^ {\prime} (q , a o)}{2} \geq \frac {| \mathcal {X} (q a o) |}{N} \geq \frac {\varepsilon}{4 \overline {{Q}} A O H \overline {{C}}}.
$$
From here, we can use the proof of Theorem 6 by substituting  $d_t^\prime$  for  $d_t^{\mathrm{b}}$ , up until the definition of the bound  $N_{1}$  on  $|\mathcal{D}|$  in (21). Inserting the bound on  $d_t^\prime (q,ao)$  into the expression for  $N_{1}$  yields
$$
\begin{array}{l} N _ {1} \leq \frac {2 \ell}{d _ {t} ^ {\prime} (q , a o)} \left(\frac {4}{\mu_ {0}} \sqrt {H \log (2 A R O) + H \log (2 A R O)} + \frac {1 4}{3 \mu_ {0}} \sqrt {H \log (2 A R O)}\right) \\ \leq \frac {1 2 6 \bar {Q} A O H \bar {C} \ell}{\varepsilon \mu_ {0}} \sqrt {H \log (2 A R O)} \equiv N _ {2}. \tag {23} \\ \end{array}
$$
Next, consider each candidate state  $qao \in \mathcal{Q}_{c,t+1}'$  such that  $|\mathcal{X}(qao)| / N < \varepsilon / (4\overline{Q}AOH\overline{C})$ . In this case, we instead choose  $\mathcal{X}(qao)$  large enough to estimate  $d_{t}'(q,ao)$  with accuracy  $\beta$  with probability  $1 - \delta$ . From the empirical Bernstein inequality, estimating  $d_{t}'(q,ao)$  with accuracy  $\beta$  implies
$$
\beta \geq \sqrt {\frac {2 \widehat {p} _ {t} (q a o) \ell}{N}} + \frac {1 4 \ell}{3 N} \Leftrightarrow N \geq \frac {2 \ell}{\beta} \left(\frac {1 4}{3} + \frac {\widehat {p} _ {t} (q a o)}{\beta}\right) \equiv N _ {3}.
$$
Choosing  $\beta = \varepsilon /(4\overline{Q} AOH\overline{C})$  implies  $\widehat{p}_t(qao) < \beta$ , and we can thus simplify  $N_{3}$  as
$$
N _ {3} = \frac {2 \ell}{\beta} \left(\frac {1 4}{3} + \frac {\widehat {p} _ {t} (q a o)}{\beta}\right) <   \frac {1 2 \ell}{\beta} = \frac {4 8 \bar {Q} A O H \bar {C} \ell}{\varepsilon} \equiv N _ {4}. \tag {24}
$$
In addition, this choice of  $\beta$  yields the following bound on  $d_t^\prime (q,ao)$
$$
d _ {t} ^ {\prime} (q, a o) \leq \widehat {p} _ {t} (q a o) + \beta <   \frac {\varepsilon}{4 \overline {{Q}} A O H \overline {{C}}} + \frac {\varepsilon}{4 \overline {{Q}} A O H \overline {{C}}} = \frac {\varepsilon}{2 \overline {{Q}} A O H \overline {{C}}}.
$$
We prove that  $\mathbf{R}'$  is an  $\varepsilon/2$ -approximation of the original RDP  $\mathbf{R}$ . We briefly overload notation by letting  $d_t^*(q, ao)$  refer to the occupancy of an optimal policy  $\pi^*$  with respect to the transition function  $\tau'$  of  $\mathbf{R}'$ . Consider a candidate state  $qao \in \mathcal{Q}_{c,t+1}'$  such that  $|\mathcal{X}(qao)| / N < \varepsilon / (4\overline{Q}AOH\overline{C})$ . The contribution to the expected optimal reward of  $\mathbf{R}$  of all histories that map to  $qao$  is bounded as
$$
d _ {t} ^ {*} (q, a o) (H - t) \leq C _ {\mathbf {R} ^ {\prime}} ^ {*} d _ {t} ^ {\prime} (q, a o) H <   \frac {\varepsilon}{2 \overline {{Q}} A O},
$$
since  $(H - t)$  is the maximum reward obtained during the remaining time steps. Since  $qao$  is mapped to a side state of  $\mathbf{R}'$ , an optimal policy for  $\mathbf{R}'$  may not accurately estimate the expected optimal value for  $qao$ , but the contribution of all such candidate states to the expected optimal value is at most
$$
\sum_ {t \in [ H - 1 ]} \sum_ {q \in \mathcal {Q} _ {t}} \sum_ {a o \in \mathcal {A O}} d _ {t} ^ {*} (q, a o) (H - t) \leq \sum_ {t \in [ H - 1 ]} \sum_ {q \in \mathcal {Q} _ {t}} \sum_ {a o \in \mathcal {A O}} \frac {\varepsilon}{2 \overline {{Q}} A O} \leq \frac {\varepsilon}{2},
$$
since there can be at most  $\overline{Q}AO$  such candidate states. Hence any optimal policy for  $\mathbf{R}'$  is an  $\varepsilon/2$ -optimal policy for  $\mathbf{R}$ , which implies that we can approximate an  $\varepsilon$ -optimal regular policy for the exact RDP  $\mathbf{R}$  by finding an  $\varepsilon/2$ -optimal policy for the approximate RDP  $\mathbf{R}'$ .