SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
| \mathcal {D} | \notin \Omega \left(\frac {H}{\mu_ {0}} + \frac {C _ {\mathbf {R}} ^ {*} H ^ {2}}{\varepsilon^ {2}}\right) \tag {3}
$$
then, for any algorithm  $\mathfrak{A}:\mathcal{D}\mapsto \widehat{\pi}$  returning non-Markov deterministic policies, the probability that  $\widehat{\pi}$  is not  $\varepsilon$ -optimal is at least  $1 / 4$
The proof relies on worst-case RDP instances that carefully combine two-armed bandits with noisy parity functions. This last component allows to capture the difficulty of learning in presence of temporal dependencies. Figure 2 shows an RDP in this class. At the beginning of each episode, the observation causes a transition towards either the bandit component (bottom branch) or the noisy parity function (top branches). Acting optimally in the two parity branches requires to predict the output of a parity function, which depends on some unknown binary code (of length 3, in the example). The first term in Theorem 9 is due to this component, because the code scales linearly with  $H$ , or  $Q$ , while the amount of information revealed about the code is controlled by  $\mu_0$ . The second term is caused by the required optimality in the bandit.
Differently from this lower bound, the parameter  $\mu_0$ , appearing in the upper bounds of Theorems 6 and 8, is a  $L_{\infty}^{\mathsf{p}}$ -distinguishability. However, the two are related, since  $L_{1}^{\mathsf{p}}(q,q^{\prime})\geq L_{\infty}^{\mathsf{p}}(q,q^{\prime})$ . Intuitively, the  $L_{1}^{\mathsf{p}}$ -distinguishability accounts for all the information that is available as differences in episode probabilities. The  $L_{\infty}^{\mathsf{p}}$ -distinguishability, on the other hand, quantifies the maximum the difference in probability associated to specific suffixes. This is the information used by the algorithm and the one appearing in the two upper bounds.
In this paper we propose an algorithm for Offline RL in episodic Regular Decision Processes. Our algorithm exploits automata learning techniques to reduce the problem of RL in RDPs, in which observations and rewards are non-Markovian, into standard offline RL for MDPs. We provide the first high-probability sample complexity guarantees for this setting, as well as a lower bound that shows how its complexity relates to the parameters that characterize the decision process and the behavior policy. We identify the RDP single-policy concentrability as an analogous quantity to the one used for MDPs in the literature. Our sample complexity upper bound depends on the  $L_{\infty}^{\mathsf{p}}$ -distinguishability of the behavior policy. As a future work, we plan to investigate if any milder notion of distinguishability also suffices. This is motivated by our lower bound which only involves the  $L_{1}^{\mathsf{p}}$ -distinguishability over the same policy. Finally, our results have strong implications for online learning in RDPs, which is a relevant setting to be explored.
Roberto Cipollone is partially supported by the EU H2020 project AIPlan4EU (No. 101016442), the ERC-ADG White-Mech (No. 834228), the EU ICT-48 2020 project TAILOR (No. 952215), the PRIN project RIPER (No. 20203FFYLK), and the PNRR MUR project FAIR (No. PE0000013). Anders Jonsson is partially supported by the EU ICT-48 2020 project TAILOR (No. 952215), AGAUR SGR, and the Spanish grant PID2019-108141GB-I00. Alessandro Ronca is partially supported by the ERC project WhiteMech (No. 834228), and the ERC project ARiAT (No. 852769). Mohammad Sadegh Talebi is partially supported by the Independent Research Fund Denmark, grant number 1026-00397B.
[74] Avrim Blum, Merrick L. Furst, Jeffrey C. Jackson, Michael J. Kearns, Yishay Mansour, and Steven Rudich. Weakly learning DNF and characterizing statistical query learning using Fourier analysis. In STOC, pages 253-262, 1994.
[75] Sumegha Garg, Ran Raz, and Avishay Tal. Extractor-based time-space lower bounds for learning. In STOC, pages 990-1002, 2018.
[76] Sumegha Garg, Pravesh K. Kothari, Pengda Liu, and Ran Raz. Memory-sample lower bounds for learning parity with noise. CoRR, abs/2107.02320, 2021.
A Extended Discussion of Related Work 16
A.1 Offline RL in MDPs 16
A.2 Online RL in RDPs 16
A.3 Non-Markov Rewards and Reward Machines 16
A.4 State Representations 17
A.5 Feature MDPs and General RL 17
A.6 Learning PDFA 18
B RDP properties 18
B.1 RDPs and Regular Policies 18
B.2 Markov Transformation 19
C Sample Complexity of ADACT-H 20
C.1 Preliminaries 20
C.2 Proof of Theorem 6 21
C.3 Proof of Theorem 8 23
C.4 Distinguishability Parameters 26
D RegORL with Subsampled VI-LCB 26
E Sample Complexity Lower Bound: Proof of Theorem 9 28
E.1 Learning parity with noise 28
E.2 Class of hard RDP instances 29
E.3 Proof of Theorem 9 30
Some of the related work mentioned in the introduction requires a more extensive discussion, which we provide below.
A.1 Offline RL in MDPs
There is a rich and growing literature on offline RL in MDPs; see, e.g., [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. A closely related line of work is off-policy learning in MDPs [21, 22, 23]; we refer to the recent survey [24] for a discussion on how the various settings are related. For offline RL in MDPs, the papers cited above report learning algorithms with theoretical guarantees on their sample efficiency. The majority of these algorithms are designed based on the pessimism principle. While most literature focuses on tabular MDPs, the case of linear function approximation is discussed in some papers, e.g., [20]. In several settings, the presented algorithms are shown to be minimax optimal. For instance, in the case of tabular episodic MDPs, it is established in [16] that the optimal sample complexity depends on the size of state-space, episode length, as well as some notion of concentrability reflecting the distribution mismatch between the behavior and optimal policy.
A.2 Online RL in RDPs
RDPs have been introduced in [3] as a formalism based on temporal logic. They admit an equivalent formulation in terms of automata, which is favoured in the context of RL. Several algorithms for online RL in RDPs exist [25, 26, 27] but complexity bounds are only given in [26] for the infinite-horizon discounted setting. This work [26] shows the correspondence between RDPs and Probabilistic Deterministic Finite Automata (PDFA), and it introduces the idea of using PDFA-learning techniques to learn RDPs. Their sample complexity bounds are not immediately comparable to ours, due to the different setting. Importantly, this algorithm uses the uniform policy for learning. So, the algorithm might be adapted to our setting only under the assumption that the behaviour policy is uniform. Even in this case, our bounds show an improved dependency on several key quantities. Furthermore, we provide a sample complexity lower bound, whereas their results are limited to showing that a dependency on the quantities occurring in their upper bounds is necessary.
The first RL algorithm for RDPs appears in [25] for the online discounted setting. It is automaton-based, and in particular, it learns the RDP in the form of a Mealy machine. The algorithm is shown in [26] to incur an exponential dependency on the length of the relevant histories. An algorithm that achieves performance guarantees building on the techniques from [26] but also integrating an effective exploration strategy is given in [27]. This work also introduces the idea of seeing the transition function of a PDFA as a Markov abstraction of the histories to be passed to an RL algorithm for MDPs, so as to employ it in a modular manner.
The algorithms in [28, 29, 30, 31] apply to RDPs even though they have not been developed specifically for RDPs. Toro Icarte et al. [28] present an RL algorithm for the subclass of POMDPs that have a finite set of reachable belief states. Such POMDPs can be captured by finite-state automata, and are in fact RDPs. Their algorithm is based on automata learning, but it does not come with an analysis of its performance guarantees. The RL techniques presented in [29, 30] for feature MDPs are in fact applicable to episodic RDPs. The techniques are based on suffix trees, rather than automata. There are cases when the size of the smallest suffix tree is exponential in the horizon, while an automaton of linear size exists—see section below on feature MDPs. Thus their techniques cannot yield optimal bounds for RDPs. Mahmud [31] introduces an RL algorithm for Deterministic Markov Decision Models (MDDs). Such MDDs are also automaton-based, and they are more general than RDPs since the automaton is only required to predict rewards. Thus their RL algorithm applies to RDPs. However, it is an algorithm without guarantees.
A.3 Non-Markov Rewards and Reward Machines
MDPs with non-Markov rewards are a special case of NMDPs, where only rewards are non-Markovian. Namely, observed states satisfy the Markov property, while rewards may depend on the history of past states. The specific kind of non-Markov rewards considered in the literature amount to the subclass of RDPs where the automaton's state is only needed to predict the next reward—while the next observation (i.e., state) can be predicted from the last observation.
Non-Markov rewards can already be found in [2], where the reward function is specified in a temporal logic of the past. More recently, the setting has been revisited with so-called reward machines [44] as well as with temporal logics of the future on finite traces [64, 47]. A reward machine is a finite automaton (or transducer) used to specify a non-Markovian reward function. Reward machines have been introduced in [44] along with an RL algorithm that assumes the reward machine to be known. RL algorithms with unknown reward machine, or equivalently unknown temporal specification, are presented in [65, 45], with no performance guarantees. Reward machines have been generalised so as to predict observations as well [28], which makes them equivalent to RDPs—as mentioned above.
The first performance bounds for RL with reward machines have been recently established in [49]. The work shows regret bounds that take into account the structure provided by a reward machine, and hence improve over the bounds that one would obtain by naively adapting regret bounds for MDPs.
An automaton-based method for dealing with non-Markov sparse rewards is proposed in [48].
A.4 State Representations
State representations are maps from histories to a finite state space. The map defined by the transition function of an RDP is a state representation. The studies on state representations [39, 40, 41, 42] focus on regret bounds for RL given a candidate set of state representations. While in our case the state representations are concretely defined by the class of finite-state automata, in their case they are arbitrary maps. This is a challenging setting, which does not allow for taking advantage of the properties of specific classes of state representations. The regret bounds in [39, 41, 42] are for finite sets of state representations, and they all show a linear dependency on the cardinality of the given set of state representations. In our case, the candidate state representations correspond to the set of automata with at most  $Q = 2(AO)^H$  states and  $AO$  input letters. Such a set contains at least  $Q^{QAO}$  automata—the number of distinct transition functions. Thus, if we could instantiate their bounds in our setting, they would have an exponential dependency on the number  $Q$  of RDP states, and hence a doubly-exponential dependency on the horizon  $H$ . We avoid this dependency, obtaining polynomial bounds in the mentioned quantities.
Nguyen et al. [40] consider the case of a countably infinite set of state representations, and present an algorithm whose regret bound does not show a dependency such as the one discussed above. Instead, they show a dependency on a quantity  $K_{0}$ , which admits several interpretations, including one based on the descriptional complexity of the candidate state representations. Thus, there may be a way to relate  $K_{0}$  to the quantities we use in our bounds. However, the formal relationship between the two, if any, renders highly non-trivial, which prevents one to use their ideas in the case of RDPs. We believe establishing a fomal relationship between their model and RDPs is an interesting, yet challenging, topic for future work. Furthermore, it should be stressed that even if the relationship was clear and one could borrow ideas from [40], the resulting sample complexity bound would have to grow as  $1 / \varepsilon^{3}$  in view of their regret bound scaling as  $T^{2 / 3}$ . In contrast, our bounds achieve an optimal dependency of  $1 / \varepsilon^{2}$  on  $\varepsilon$ .
A.5 Feature MDPs and General RL
Hutter [29] introduces feature MDPs, where histories are mapped to states by a feature map. It relates to our work since the map provided by the transition function of an RDP is a feature map. The concrete feature maps considered in [29] are based on U-Trees [66]. The idea is also revisited in [30] with Prediction Suffix Trees (PSTs) [67, 68]. Both U-Trees and PSTs are suffix trees. There are cases when their size is exponential in the horizon, while an automaton of linear size exists. For instance, in the case of a parity condition over the history. To see this, note that a suffix  $x$  of a bit string  $bx$  does not suffice to establish parity of  $bx$ . In fact, the parity of  $0x$  is different from the parity of  $1x$ . Thus a suffix tree for parity must encode all suffixes, and hence it will have a number of leaves that is exponential in the maximum length of a relevant string—the horizon  $H$  in the case of episodic RL.
Hutter [69] provides several formal characterisations of feature maps. All the characterisations are less restrictive than the one defined by an RDP. In particular, their most restrictive characterisation is given in their Equation (6). It only requires to preserve the ability to predict rewards, not observations—this is also adopted by Mahmud [31]. The states of our automata suffice to predict observations as well. It is unclear whether automata techniques can be used to learn directly abstractions that do not preserve the dynamics entirely.
Majeed and Hutter [55] study convergence of Q-learning when executed over the state space of an MDP that underlies a non-Markov decision processes. Such a state space corresponds to the state space of the RDP, but they do not consider the problem of learning the state space.
Lattimore et al. [43] consider General RL as the problem of RL when we are given a set of candidate NMDPs, rather than assuming the decision process to belong to a fixed class. Similarly to the works on state representations, it does not commit to specific classes of NMDPs, and their bounds have a linear dependency on the number of candidate models. As remarked above, in our setting, it amounts to an exponential dependency on the number of states of the candidate RDPs, and hence a doubly-exponential dependency on the horizon; we avoid such exponential dependencies.
A.6 Learning PDFA
Our algorithms for learning an RDP borrow and improve over techniques for learning Probabilistic-Deterministic Finite Automata (PDFA). The first PAC learning algorithm for acyclic PDFA has been presented in [56], then followed by extensions and variants that can handle PDFA with cycles [57, 58, 10, 59, 60]. All bounds feature some variant of a distinguishability parameter, which we adopt in our bounds, properly adapting it to the offline RL setting. Our algorithm builds upon the state-of-the-art algorithm ADACT [10], and we derive bounds that are a substantial improvement over the ones that can be obtained from a straightforward application of any existing PDFA-learning algorithm to the offline RL setting.
B RDP properties
In this section we prove several properties of RDPs that are stated in Sections 2 and 3.
B.1 RDPs and Regular Policies
In this section, we prove Propositions 1 and 2.
Proof. By induction on  $t$ . For  $t = H$ , all histories in  $\mathcal{H}_H$  generate the empty suffix in  $(\mathcal{ARO})^0$  with probability 1 (the stop symbol is omitted). For  $t < H$ , the probability of generating a suffix  $aroe_{t+2:H}$  is
$$
\begin{array}{l} \mathbb {P} \left(a r o e _ {t + 2: H} \mid h _ {1}, \pi , \mathbf {R}\right) = \pi \left(h _ {1}, a\right) \cdot \mathbb {P} \left(r, o \mid \bar {\tau} \left(h _ {1}\right), a, \mathbf {R}\right) \cdot \mathbb {P} \left(e _ {t + 2: H} \mid h _ {1} a o, \pi , \mathbf {R}\right) \\ = \pi (h _ {2}, a) \cdot \mathbb {P} (r, o \mid \bar {\tau} (h _ {2}), a, \mathbf {R}) \cdot \mathbb {P} \left(e _ {t + 2: H} \mid h _ {2} a o, \pi , \mathbf {R}\right) = \mathbb {P} \left(a r o e _ {t + 2: H} \mid h _ {2}, \pi , \mathbf {R}\right), \\ \end{array}
$$
where we have used the fact that  $\pi$  is regular,  $\bar{\tau}(h_1) = \bar{\tau}(h_2)$ ,  $\bar{\tau}(h_1ao) = \tau(\bar{\tau}(h_1), ao) = \tau(\bar{\tau}(h_2), ao) = \bar{\tau}(h_2ao)$ , and the induction hypothesis.
The following statement appears in the literature [3, Theorem 2], but the authors do not provide a complete proof, so for completeness we prove the statement here.
Proof. Given  $\mathbf{R}$ , consider any optimal policy  $\pi^{*}:\mathcal{H}\to \Delta (\mathcal{A})$ , not necessarily regular. We prove the statement by constructing a policy  $\pi$  and showing by induction on  $t\in [H]$  that  $\pi$  is both optimal and regular. The base case is given by  $t = H$ . In this case, for an arbitrary  $a\in \mathcal{A}$ , define  $\pi (h)\coloneqq \mathbb{1}_a$  for each history  $h\in \mathcal{H}_H$ . Since  $V_{H}^{\pi}(h) = 0$  by definition,  $\pi$  is optimal for each history  $h\in \mathcal{H}_H$ , and regular since it always selects the same action.
For  $t < H$ , we first construct a new policy  $\pi_{\mathbf{c}}$  which is the composition of policies  $\pi^{*}$  and  $\pi$ . Concretely, for each history  $h \in \mathcal{H}_u$  such that  $u \leq t$ ,  $\pi_{\mathbf{c}}(h) = \pi^{*}(h)$  acts according to  $\pi^{*}$ , while for each history  $h \in \mathcal{H}_u$  such that  $u > t$ ,  $\pi_{\mathbf{c}}(h) = \pi(h)$  acts according to  $\pi$ . Clearly,  $\pi_{\mathbf{c}}$  is an optimal policy for  $\mathbf{R}$  since  $\pi^{*}$  is optimal and since by induction,  $\pi$  is optimal for histories in  $\mathcal{H}_u$ ,  $u > t$ .
Consider a pair of histories  $h_1$  and  $h_2$  in  $\mathcal{H}_t$  such that  $\bar{\tau}(h_1) = \bar{\tau}(h_2)$  but  $\pi_{\mathbf{c}}(h_1) \neq \pi_{\mathbf{c}}(h_2)$ . Define  $\pi(h_1) := \pi(h_2) := \pi_{\mathbf{c}}(h_1)$ . Since the value function can be written as an expectation
over suffixes, due to Proposition 1 and the fact that  $\pi$  is regular for histories in  $\mathcal{H}_u$ ,  $u > t$ , we have  $V_{t}^{\pi}(h_{1}) = V_{t}^{\pi}(h_{2})$ . Since  $\pi_{\mathfrak{c}}$  is the same as  $\pi$  for histories in  $\mathcal{H}_u$ ,  $u > t$ , this implies  $V_{t}^{\pi}(h_{1}) = V_{t}^{\pi_{\mathfrak{c}}} (h_{1}) \leq V_{t}^{\pi_{\mathfrak{c}}} (h_{2})$  since  $\pi_{\mathfrak{c}}$  is optimal for  $h_2$ . If we were to instead define  $\pi(h_1) := \pi(h_2) := \pi_{\mathfrak{c}}(h_2)$ , we would obtain  $V_{t}^{\pi_{\mathfrak{c}}} (h_2) \leq V_t^{\pi_c}(h_1)$ . The only possibility is  $V_{t}^{\pi_{\mathfrak{c}}} (h_1) = V_t^{\pi_c} (h_2)$  which is the same value achieved by the policy  $\pi$ . Hence  $\pi$  is optimal for  $h_1$  and  $h_2$ .
We now repeat the same procedure for each pair of histories  $h_1$  and  $h_2$  in  $\mathcal{H}_t$  such that  $\bar{\tau}(h_1) = \bar{\tau}(h_2)$  but  $\pi_c(h_1) \neq \pi_c(h_2)$ . If necessary, we complete the definition of  $\pi$  by copying the action choices of  $\pi_c$ . The resulting policy  $\pi$  is optimal for each history  $h \in \mathcal{H}_t$ , and regular since it makes the same action choices for each pair of histories  $h_1$  and  $h_2$  in  $h \in \mathcal{H}_t$  such that  $\bar{\tau}(h_1) = \bar{\tau}(h_2)$ .
B.2 Markov Transformation
In this section, we verify the properties of the Markov transformation, which is the intermediate step that RegORL uses to recover the Markov property in the original dataset. This transformation has been formalized in Definition 2. We use  $\mathcal{D}$  and  $\mathcal{D}'$  to denote the original and the transformed datasets, respectively.
Proof. For  $t \in [H]$ , let  $e_t \in \mathcal{E}_t = (\mathcal{ARO})^{t+1}$  be an episode prefix in  $\mathbf{R}$ ,  $\phi(e_t) \in \mathcal{E}_t' = (\mathcal{ARQ})^{t+1}$  be its Markov transformation and  $e_t' \in \mathcal{E}_t'$  be an episode of the associated MDP. The function  $\phi: \mathcal{E} \to \mathcal{E}'$  transforms the observations according to  $\bar{\tau}$ , and preserves actions and rewards. The statement says that  $\mathbb{P}(\phi(e_t) \mid \mathbf{R}, \pi) = \mathbb{P}(e_t' \mid \mathbf{M}_{\mathbf{R}}, \pi_{\mathrm{r}})$  (note that  $\phi(e_t)$  and  $e_t'$  are distinct random variables). We prove this by induction. For  $t = 0$ , we recall that the irrelevant quantities  $a_0, r_0$  are constant and,
$$
\begin{array}{l} \mathbb {P} \left(\phi \left(a _ {0} r _ {0} o _ {0}\right) = a _ {0} r _ {0} q \mid \mathbf {R}, \pi\right) = \sum_ {o \in \mathcal {O}} \mathbb {I} \left(\tau \left(q _ {0}, a _ {0} o\right) = q\right) \theta_ {\mathrm {o}} \left(q _ {0}, a _ {0}, o\right) \\ = T \left(q _ {0}, a _ {0}, q\right) \\ = \mathbb {P} \left(e _ {0} ^ {\prime} = a _ {0} r _ {0} q \mid \mathbf {M} _ {\mathbf {R}}, \pi_ {\mathrm {r}}\right) \tag {4} \\ \end{array}
$$
where  $T: \mathcal{Q} \times \mathcal{A} \to \Delta(\mathcal{Q})$  is the transition function of  $\mathbf{M}_{\mathbf{R}}$ , from Definition 3. Due to the role of the dummy action,  $T(q_0, a_0)$  is the initial distribution of the MDP.
For the inductive step, assume that  $\mathbb{P}(\phi(e_{t-1}) \mid \mathbf{R}, \pi) = \mathbb{P}(e_{t-1}' \mid \mathbf{M}_{\mathbf{R}}, \pi_{\mathbf{r}})$ . Then, for any  $e' \in \mathcal{E}_{t-1}'$ ,  $arq \in \mathcal{ARQ}$ , if  $q'$  is the last element of  $e'$ , we have
$$
\begin{array}{l} \mathbb {P} \left(\phi \left(e _ {t}\right) = e ^ {\prime} a r q \mid \mathbf {R}, \pi\right) = \mathbb {P} \left(\phi \left(e _ {t - 1}\right) = e ^ {\prime} \mid \mathbf {R}, \pi\right) \mathbb {P} \left(a _ {t} r _ {t} q _ {t + 1} = a r q \mid \phi \left(e _ {t - 1}\right) = e ^ {\prime}, \mathbf {R}, \pi\right) (5) \\ = \mathbb {P} \left(e _ {t - 1} ^ {\prime} = e ^ {\prime} \mid \mathbf {M} _ {\mathbf {R}}, \pi_ {\mathrm {r}}\right) \mathbb {P} \left(a _ {t} r _ {t} q _ {t + 1} = a r q \mid q _ {t} = q ^ {\prime}, \mathbf {R}, \pi\right) (6) \\ = \mathbb {P} \left(e _ {t - 1} ^ {\prime} = e ^ {\prime} \mid \mathbf {M} _ {\mathbf {R}}, \pi_ {\mathrm {r}}\right) \pi_ {\mathrm {r}} \left(q ^ {\prime}, a\right) \theta_ {\mathrm {r}} \left(q ^ {\prime}, a, r\right) \sum_ {o \in \mathcal {O}} \theta_ {\mathrm {o}} \left(q ^ {\prime}, a, o\right) \mathbb {I} (q = \tau \left(q ^ {\prime}, a o\right)) (7) \\ = \mathbb {P} \left(e _ {t - 1} ^ {\prime} = e ^ {\prime} \mid \mathbf {M} _ {\mathbf {R}}, \pi_ {\mathrm {r}}\right) \pi_ {\mathrm {r}} \left(q ^ {\prime}, a\right) \theta_ {\mathrm {r}} \left(q ^ {\prime}, a, r\right) T \left(q ^ {\prime}, a, q\right) (8) \\ = \mathbb {P} \left(e _ {t} ^ {\prime} = e ^ {\prime} a r q \mid \mathbf {M} _ {\mathbf {R}}, \pi_ {\mathrm {r}}\right) (9) \\ \end{array}
$$
where, in (6), we have used the induction hypothesis and the fact that  $a_{t}r_{t}q_{t + 1}$  are Markov in  $q^{\prime}$  by regularity of the policy.
Thanks to this relation, the values of corresponding policies are also related in the following way.
Proof. The statement is composed of two parts. First, we show that  $\mathbb{E}[V_{0,\mathbf{R}}^{\pi}] = \mathbb{E}[V_{0,\mathbf{M}_{\mathbf{R}}}^{\pi}]$ , which is a direct consequence of Proposition 3. Following the same convention as in the proof of Proposition 3,
we use  $\mathcal{E}_t' = (\mathcal{ARQ})^{t + 1}$  and  $\phi$  for the Markov transformation. Then,