Title: Provably Efficient Offline Reinforcement Learning in Regular Decision Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Offline Reinforcement Learning

Regular Decision Processes

Sample complexity

Automata
Abstract: This paper deals with offline (or batch) Reinforcement Learning (RL) in episodic Regular Decision Processes (RDPs). RDPs are the subclass of Non-Markov Decision Processes where the dependency on the history of past events can be captured by a finite-state automaton. We consider a setting where the automaton that underlies the RDP is unknown, and a learner strives to learn a near-optimal policy using pre-collected data, in the form of non-Markov sequences of observations, without further exploration. We present RegORL, an algorithm that suitably combines automata learning techniques and state-of-the-art algorithms for offline RL in MDPs. RegORL has a modular design allowing one to use any off-the-shelf offline RL algorithm in MDPs. We report a non-asymptotic high-probability sample complexity bound for RegORL to yield an $\varepsilon$-optimal policy, which makes appear a notion of concentrability relevant for RDPs. Furthermore, we present a sample complexity lower bound for offline RL in RDPs. To our best knowledge, this is the first work presenting a provably efficient algorithm for offline learning in RDPs.

Provably Efficient Offline Reinforcement Learning in Regular Decision Processes

Abstract

This paper deals with offline (or batch) Reinforcement Learning (RL) in episodic Regular Decision Processes (RDPs). RDPs are the subclass of Non-Markov Decision Processes where the dependency on the history of past events can be captured by a finite-state automaton. We consider a setting where the automaton that underlies the RDP is unknown, and a learner strives to learn a near-optimal policy using pre-collected data, in the form of non-Markov sequences of observations, without further exploration. We present RegORL , an algorithm that suitably combines automata learning techniques and state-of-the-art algorithms for offline RL in MDPs. RegORL has a modular design allowing one to use any off-the-shelf offline RL algorithm in MDPs. We report a non-asymptotic high-probability sample complexity bound for RegORL to yield an ε -optimal policy, which makes appear a notion of concentrability relevant for RDPs. Furthermore, we present a sample complexity lower bound for offline RL in RDPs. To our best knowledge, this is the first work presenting a provably efficient algorithm for offline learning in RDPs.

1 Introduction

Most reinforcement learning (RL) algorithms hinge on the Markovian assumption, i.e. that the underlying system transitions and rewards are Markovian in some natural notion of (observable) state, and hence, the distribution of future observations depends only on the current state-action of the system. This fundamental assumption allows one to model decision making using the powerful framework of Markov Decision Processes (MDPs) [1]. However, there are many application scenarios where rewards are issued according to temporal conditions over histories (or trajectories), and others where the environment itself evolves in a history-dependent manner. As a result, Markovian approaches may prove unsuitable for modeling such situations. These scenarios can be appropriately modeled as Non-Markovian Decision Processes (NMDPs) [2, 3].

NMDPs describe environments where the distribution on the next observation and reward is a function of the history. In these environments, behaving optimally may also require to take histories into account. For example, a robot may receive a reward for delivering an item only if the item was previously requested, and a self-driving car is more likely to skid and lose control if it previously rained. Also, consider a mobile robot that has to track an object which may disappear from its field of view. The object is likely to be found again in the same place where it was seen last time. This requires the agent to remember, hence to act according to information in its interaction history. In general, an NMDP can show an arbitrary dependency on the history or trace, preventing efficient learning. Consequently, recent research has focused on tractable sub-classes of NMDPs. In Regular

Anders Jonsson

Mohammad Sadegh Talebi

Decision Processes (RDPs) [3], the next observation and reward distributions depend on regular properties of the history, which can be captured by a deterministic finite-state automaton. This determines the existence of a finite state space where states are determined by histories, and where the Markov property is regained.

In this paper, we investigate offline RL in episodic RDPs, where the goal is to find a near-optimal policy using a pre-collected dataset, with minimal possible size, generated by a fixed behavior policy (and without further exploration). Offline RL in MDPs has received extensive attention recently, and provably sample efficient algorithms have been proposed for various settings. Despite the extensive and rich literature on MDPs, comparatively little work exists on offline RL in NMDPs. The scarcity of results may likely be attributed to the difficult nature of the problem rather than the lack of interest.

Partially-Observable Markov Decision Processes (POMDPs) [4] are also NMDPs, and RDPs can be seen as the subclass of POMDPs that enjoy the property of having hidden states determined by the history of observations. This is a key property that allows one to take advantage of a set of planning and learning techniques that do not apply to arbitrary POMDPs. Planning in POMDPs is computationally intractable [5], and two common approaches to solve (and learn) them rely on maintaining either a belief state or a finite history of observations. Maintaining and updating a belief state is worst-case exponential in the size of the original observation space, while the latter approach yields a space whose size is exponential in the history length. State-of-the-art work on offline RL in POMDPs considers restricted classes of POMDPs such as undercomplete POMDPs (e.g., [6, 7]), which cannot be used to model all RDP instances. General POMDPs are only considered under assumptions such as the possibility of reaching every belief state in a few steps [8] or ergodicity [9]. While existing offline RL algorithms for solving POMDPs cannot guarantee provable learning in a generic RDP, the structural properties of RDPs indicate that they can be solved more efficiently using techniques that are carefully tailored to their structure. Exploiting the structure in RDPs is thus key in designing provably sample-efficient learning algorithms.

1.1 Summary of Contributions

We formalize offline RL in RDPs (Section 2), and establish a first, to the best of our knowledge, sample complexity lower bound thereof (Section 5). We introduce an algorithm, called RegORL , that learns ε -optimal policies for any RDP, in the episodic setting. At the core of RegORL , there is a component called ADACT-H, which is a variant of ADACT [10], carefully tailored to episodic RDPs. ADACT-H learns a minimal automaton that underlies the unknown RDP without prior knowledge. The output automaton is further used to derive a Markov abstraction of data to be used by any off-the-shelf algorithm for offline RL in episodic MDPs. We present a sample-complexity bound for ADACT-H to return a minimal underlying automaton with high probability. This bound substantially improves the existing bound for the original ADACT, and can be of independent interest. In view of the modular design of RegORL , the total sample complexity is controlled by twice that of ADACT-H (Theorem 6) and that for the incorporated off-the-shelf algorithm. We also present another variant of ADACT-H, called ADACT-H-A. In contrast to ADACT-H that learns a complete RDP, ADACT-H-A only reconstructs a subset of states that are likely under the behavior policy, in relation to an input accuracy parameter. As such, ADACT-H-A renders more aligned with the practice of RL than ADACT-H. Furthermore, we provide a first lower-bound for offline RL in RDPs that involves relevant parameters for the problem, such as the RDP single-policy concentrability, which extends an analogous notion for MDPs from the literature. Finally, if contrasted to both online learning in RDPs and automata learning, our results suggest possible improvements in sample complexity results for both areas.

1.2 Related Work

Offline RL in MDPs. There is a rich and growing literature on offline RL, and provably sample efficient algorithms have been proposed for various settings of MDPs; see, e.g., [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. For example, in the case of episodic MDPs, it is established that the optimal sample size in offline RL depends on the size of state-space, episode length, as well as some notion of concentrability, reflecting the distribution mismatch between the behavior and optimal policies. A closely related problem is off-policy learning; see, e.g., [21, 22, 23] and the recent survey [24].

Online RL in RDPs. Several algorithms for online RL in RDPs exist [25, 26, 27] but complexity bounds are only given in [26] for the infinite-horizon discounted setting. The sample complexity bounds in [26] are not immediately comparable to ours, due to the different setting. Importantly, the algorithm in [26] uses the uniform policy for learning, and it therefore might be adapted to our setting only under the assumption that the behaviour policy is uniform. Even in this case, our bounds show an improved dependency on several key quantities. Furthermore, we provide a sample complexity lower bound, whereas their results are limited to showing that a dependency on the quantities occurring in their upper bounds is necessary.

The online RL algorithms in [28, 29, 30, 31] have been developed for formalisms that are closely related to RDPs, and such algorithms can be applied to RDPs. However, these algorithms are not proven to be sample efficient.

POMDPs. Every RDP can be seen as a POMDP whose hidden dynamics evolves according to its finite-state automaton. However, RL in POMDPs is a largely open problem. Even for a known POMDP, computing a near-optimal policy is PSPACE-complete [5]. For unknown dynamics, which is the setting considered here, favourable bounds have been obtained for the class of undercomplete POMDPs [6, 7], which does not include all RDPs, or alternatively, under other assumptions such as few-step reachability [8] or ergodicity [9]. This relationship between RDPs and POMDPs can be also seen from the notion of state. In fact, the automaton state of an RDP is an instance of information state, as defined in [32], and of belief, as in classic POMDP literature [33].

PSRs. Predictive State Representations (PSRs) [34, 35, 36, 37] are general descriptions of dynamical systems that capture POMDPs and hence RDPs. There exist polynomial PAC bounds for online RL in PSRs [38]. Nonetheless, these bounds are looser than the one we show here, since they must necessarily consider a wider class of models. Moreover, although a minimum core set for PSRs is similar to a minimal RDP, the bounds feature a number of quantities that are specific to PSRs (e.g., regularity parameter) and do not immediately apply to RDPs.

Other Non-Markovian Settings. Feature MDPs and state representations both share the idea of having a map from histories to a state space. This is analogous to the map determined by the transition function of the automaton underlying an RDP. Algorithmic solutions for feature MDPs are based on suffix trees, and they cannot yield optimal performance in our setting [29, 30]. The automaton of an RDP can be seen as providing one kind of state representation [39, 40, 41, 42]. The existing bounds for state representations show a linear dependency on the number of candidate representations, which is exponential in the number of states in our case. A similar dependency is also observed in [43]. RL with non-Markovian rewards is considered in [44, 45, 46, 47, 48, 49]. The idea of a map from histories to states is also found in [32]. Non-Markovianity is also introduced by logical specifications that the agent is required to satisfy [50, 51, 52, 53, 54]; however, it is resolved a priori from the known specification. The convergence properties of Q-learning over a (known) underlying state space such as the one of an RDP are studied in [55].

Learning PDFA. Our algorithms for learning an RDP borrow and improve over techniques for learning Probabilistic-Deterministic Finite Automata (PDFA) [56, 57, 58, 10, 59, 60]. Our algorithm builds upon the state-of-the-art algorithm ADACT [10], and we derive bounds that are a substantial improvement over the ones that would be obtained from a straightforward application of any existing PDFA-learning algorithm to the offline RL setting.

We provide additional literature review in Appendix A.

2 Preliminaries and Problem Formulation

̸

Notations. Given a set Y , ∆( Y ) denotes the set of probability distributions over Y . For a function f : X → ∆( Y ) , f ( x, y ) is the probability of y given x . Further, we write y ∼ f ( x ) to abbreviate y ∼ f ( x, · ) . For y ∈ Y , we use ✶ y ∈ ∆( Y ) to denote the Kronecker delta defined as ✶ y ( y ) = 1 and ✶ y ( y ′ ) = 0 for each y ′ ∈ Y such that y ′ = y . Given an event E , I ( E ) denotes the indicator function of E , which equals 1 if E is true, and 0 otherwise, e.g. ✶ y ( y ′ ) = I ( y = y ′ ) . For any integer Z ≥ 0 , we let [ Z ] := { 0 , . . . , Z } . Given a set X , for k ∈ N , X k represents the set of sequences of length k whose elements are from X . Also, X ∗ = ∪ ∞ k =0 X k . The notation ˜ O ( · ) hides poly-logarithmic terms.

2.1 Episodic Regular Decision Processes

We first introduce generic episodic decision processes. An episodic decision process is a tuple P = ⟨O , A , R , ¯ T, ¯ R,H ⟩ , where O is a finite set of observations, A is a finite set of actions, R ⊂ [0 , 1] is a finite set of rewards and H ≥ 1 is a finite horizon. As is common in automata theory, we use sequences a m r m o m · · · a n r n o n to denote traces of actions, rewards and observations, and concatenation ARO = { aro : a ∈ A , r ∈ R , o ∈ O} to denote sets of sequences. Let E t = ( ARO ) t +1 be the set of traces of length t + 1 , and let e m : n ∈ E n -m denote a trace from time m to time n , included. A trajectory e 0: T is the full trace generated until time T . We assume that a trajectory e 0: T can be partitioned into episodes e ℓ : ℓ + H ∈ E H of length H +1 , and that the dynamics at time T = k ( H +1)+ t , t ∈ [ H ] , are conditionally independent of the previous episodes and all rewards, i.e. the dynamics only depend on a k ( H +1) o k ( H +1) · · · a T o T . For t ∈ [ H ] , let H t = ( AO ) t +1 denote the relevant part of the trajectory for decision making, and let H = ∪ H t =0 H t . We refer to elements in H as histories , even though they are not complete trajectories. In each episode e 0: H , a 0 = a ⊥ is a dummy action used to initialize the distribution on H 0 . The transition function ¯ T : H×A→ ∆( O ) and the reward function ¯ R : H×A→ ∆( R ) only depend on the history of the current episode. Given P , a generic policy is a function π : ( AO ) ∗ → ∆( A ) that maps trajectories to distributions over actions. The value function V π : [ H ] ×H→ R of a policy π is a mapping that assigns real values to histories. For h ∈ H , it is defined as V π ( H,h ) := 0 and

For brevity, we write V π t ( h ) := V π ( t, h ) . The optimal value function V ∗ is defined as V ∗ t ( h ) := sup π V π t ( h ) , ∀ t ∈ [ H ] , ∀ h ∈ H t , where sup is taken over all policies π : ( AO ) ∗ → ∆( A ) . Any policy achieving V ∗ is called optimal, which we denote by π ∗ ; namely V π ∗ = V ∗ . Solving P amounts to finding π ∗ . In what follows we consider simpler policies of the form π : H → ∆( A ) mapping histories to distributions over actions. Let Π H denote the set of such policies. It can be shown that Π H always contains an optimal policy, i.e. V ∗ t ( h ) := max π ∈ Π H V π t ( h ) , ∀ t ∈ [ H ] , ∀ h ∈ H t . An episodic MDP is an episodic decision process whose dynamics at each timestep t only depends on the last observation and action [1].

Episodic RDPs. Anepisodic Regular Decision Process (RDP) [3, 25] is an episodic decision process R = ⟨O , A , R , ¯ T, ¯ R,H ⟩ described by a finite transducer (Moore machine) ⟨Q , Σ , Ω , τ, θ, q 0 ⟩ , where Q is a finite set of states, Σ = AO is a finite input alphabet composed of actions and observations, Ω is a finite output alphabet, τ : Q × Σ → Q is a transition function, θ : Q → Ω is an output function, and q 0 ∈ Q is a fixed initial state [61, 62, 63]. The output space Ω = Ω o × Ω r consists of a finite set of functions that compute the conditional probabilities of observations and rewards, meaning Ω o ⊂ A → ∆( O ) and Ω r ⊂ A → ∆( R ) . For simplicity, we use two output functions, θ o : Q×A→ ∆( O ) and θ r : Q×A→ ∆( R ) , to denote the individual conditional probabilities. Let τ -1 denote the inverse of τ , i.e. τ -1 ( q ) ⊆ Q × AO is the subset of state-symbol pairs that map to q ∈ Q . In this context, an input symbol is an element of AO . We use A,R,O,Q to denote the cardinality of A , R , O , Q , respectively, and assume A ≥ 2 .

An RDP R implicitly represents a function ¯ τ : H → Q from histories in H to states in Q , recursively defined as ¯ τ ( h 0 ) := τ ( q 0 , a 0 o 0 ) and ¯ τ ( h t ) := τ (¯ τ ( h t -1 ) , a t o t ) . The dynamics and of R are defined as ¯ T ( h, a, o ) = θ o (¯ τ ( h ) , a, o ) and ¯ R ( h, a, r ) = θ r (¯ τ ( h ) , a, r ) , ∀ h ∈ H , ∀ aro ∈ ARO . Episodic RDPs are acyclic, i.e. the states can be partitioned as Q = Q 0 ∪ · · · ∪ Q H +1 , where each Q t +1 is the set of states generated by histories in H t for each t ∈ [ H ] . An RDP is minimal if its Moore machine is minimal. Since there is nothing to predict at time H + 1 , a minimal RDP contains a single state q H +1 in Q H +1 . To ensure that an acyclic RDP R is minimal, we introduce a designated termination observation o ⊥ in O and define τ ( q H +1 , ao ⊥ ) = q H +1 and θ o ( q H +1 , a ) = ✶ o ⊥ for each a ∈ A . Hence, q H +1 is absorbing and the states in Q implicitly count how many steps are left until we observe o ⊥ . Without o ⊥ , a Moore machine could potentially represent all episodes using fewer than H +2 states.

Since the conditional probabilities of observations and rewards are fully determined by the current state-action pair ( q, a ) , an RDP R adheres to the Markov property over its states, but not over the observations . Given a state q t ∈ Q and an action a t ∈ A , the probability of the next transition is

(a) Cookie domain.

?

?

?

(b) Agent's view.

Evidently, in the special case where an RDP is Markovian in both observations and rewards, it reduces to an episodic MDP. More precisely, any episodic MDP with actions A , states O and horizon H can be represented by some episodic RDP with states Q ⊆ O × [ H +1] and inputs AO .

An important class of policies for RDPs are the regular policies. Given an RDP R , a policy π : H → ∆( A ) is called regular if π ( h 1 ) = π ( h 2 ) whenever ¯ τ ( h 1 ) = ¯ τ ( h 2 ) , for all h 1 , h 2 ∈ H . Let Π R denote the set of regular policies for R . Regular policies exhibit powerful properties. First, under a regular policy, suffixes have the same probability of being generated for histories that map to the same RDP state. Second, there exists at least one optimal policy that is regular.

Proposition 1. Consider an RDP R , a regular policy π ∈ Π R and two histories h 1 and h 2 in H t , t ∈ [ H ] , such that ¯ τ ( h 1 ) = ¯ τ ( h 2 ) . For each suffix e t +1: H ∈ E H -t -1 , the probability of generating e t +1: H is the same for h 1 and h 2 , i.e. P ( e t +1: H | h 1 , π, R ) = P ( e t +1: H | h 2 , π, R ) .

Proposition 2. Each RDP R has at least one optimal policy π ∗ ∈ Π R .

Due to Proposition 2, when solving an RDP R , we can restrict our search to the set of regular policies Π R . A regular policy can be compactly defined as π : Q → ∆( A ) , where π ( q 0 ) = ✶ a ⊥ always selects the dummy action a ⊥ , and its value function as V π : [ H ] ×Q→ R .

Next, we define occupancy measures for RDPs. Given a regular policy π : Q → ∆( A ) and t ∈ [ H ] , let d π t ∈ ∆( Q t ×AO ) be the induced probability distribution over states in Q t and input symbols in AO , recursively defined as d π 0 ( q 0 , a 0 o 0 ) := θ o ( q 0 , a 0 , o 0 ) and

We also overload the notation by writing d π t ( q t , a t ) = ∑ o ∈O d π ( q t , a t o ) . Of particular interest is the occupancy distribution d ∗ t := d π ∗ t , associated with an optimal policy π ∗ .

Example 1 (The cookie domain [28]) . The cookie domain (Figure 1a) has three rooms connected by a hallway. The agent (purple triangle) can move in the four cardinal directions. When pressing a button in the orange room, a cookie randomly appears in either the green or the blue room. The agent receives a reward of +1 for eating the cookie and it may then press the button again. This domain is partially observable since the agent can only see what is in the room that it currently occupies (Figure 1b). The cookie domain can be modelled as an episodic RDP with states Q = [ H +1] ×O×U , with U = { u 1 , u 2 , u 3 , u 4 } . The value of U is u 1 when the button has not been pressed yet (or not pressed since the last cookie was eaten). The value is u 2 when the button has been pressed, but the agent has not visited the blue or green room yet. In this state, the environment has a 50% probability of generating the observation of a cookie when the agent enters either room for the first time. If the agent visits the green room and finds no cookie, the value becomes u 3 , meaning that the cookie is in the blue room. The meaning of u 4 is dual to that of u 3 .

2.2 Offline RL in RDPs

We are now ready to formally present the offline RL problem in episodic RDPs. Assume that we have access to a batch dataset D collected through interacting with an unknown (but fixed) episodic RDP R using a regular behavior policy π b . We assume that D comprises N episodes, where the k -th episode is of the form e k 0: H = a k 0 r k 0 o k 0 · · · a k H r k H o k H , where q k 0 = q 0 and where, for each t ∈ [ H ] ,

The goal is to compute a near-optimal policy ̂ π using the dataset D (and without further exploration). More precisely, for a pre-specified accuracy ε ∈ (0 , H ] , we aim to find an ε -optimal policy ̂ π , using the smallest dataset D possible. A policy ̂ π is ε -optimal iff E h 0 [ V ∗ 0 ( h 0 ) -V ̂ π 0 ( h 0 )] ≤ ε , where h 0 = a ⊥ o 0 , for some random o 0 ∈ O .

By virtue of Proposition 2, one may expect that it is sufficient to search for regular ε -optimal policies, which is indeed the case. In order to learn an ε -optimal policy from D , some assumption is necessary regarding the policy π b that was used to collect the episodes. Let d b t := d π b t be the occupancy distribution of π b . The following assumption requires that the behavior policy assigns a positive probability to all actions, which ensures that π b explores the entire RDP.

Assumption 1 is only needed by Theorem 6, which reconstructs the full unknown RDP. Theorem 8, instead, relies on a weaker assumption that can be expressed with the coefficient introduced in Definition 1.

The second assumption we require concerns the richness of π b and its capability to allow us to distinguish the various RDP states. This is perfectly captured by notions of distiguishability arising in automata theory, such as in Balle et al. [10]. We apply these concepts in our context, where such discrete distributions are generated from an RDP and a policy. Consider a minimal RDP R with states Q = ∪ t ∈ [ H +1] Q t . Given some policy π , at each timestep t ∈ [ H ] , every RDP state q ∈ Q t defines a unique probability distribution over the episode suffixes E H -t = ( ARO ) H -t +1 . Then, the states in each Q t can be compared through the probability distributions they induce over E H -t . Consider any L = { L ℓ } H ℓ =0 , where each L ℓ is a metric over ∆( E ℓ ) . We define the L -distinguishability of R and π as the maximum µ 0 such that, for any t ∈ [ H ] and any two distinct q, q ′ ∈ Q t , the probability distributions over suffix traces e t : H ∈ E ℓ from the two states satisfy

We will often omit the remaining length of the episode ℓ = H -t from L ℓ and simply write L . We consider the L p ∞ -distinguishability, instantiating the definition above with the metric L p ∞ ( p 1 , p 2 ) = max u ∈ [ ℓ ] ,e ∈E u | p 1 ( e ∗ ) -p 2 ( e ∗ ) | , where p i ( e ∗ ) represents the probability of the trace prefix e ∈ E u , followed by any trace e ′ ∈ E ℓ -u -1 . The L p 1 -distinguishability is defined analogously using L p 1 ( p 1 , p 2 ) = ∑ u ∈ [ ℓ ] ,e ∈E u | p 1 ( e ∗ ) -p 2 ( e ∗ ) | . We can now require a positive distinguishability with our second assumption.

Assumption 2. The behavior policy π b has L p ∞ -distinguishability of at least µ 0 > 0 .

Finally, in order to capture the mismatch in occupancy measure between the optimal policy and the behavior policy, we introduce a key quantity called single-policy RDP concentrability coefficient , which extends the single-policy concentrability coefficient in MDPs to RDPs:

Definition 1. The single-policy RDP concentrability coefficient of an RDP R with episode horizon H and with respect to a policy π b is defined as:

The concentrability coefficient in Definition 1 resembles the notions of concentrability in MDPs (e.g., [14, 15]). It should be stressed, however, that those in MDPs are defined in terms of observationaction pairs ( o, a ) , whereas C ∗ R is defined in terms of hidden RDP states and actions-observations, ( q, ao ) . It is worth remarking that C ∗ R could be equivalently defined in terms of state-action pairs ( q, a ) . Finally, in the special case where the RDP is Markovian - in which case it coincides with an episodic MDP - we have Q ⊆ O × [ H +1] and C ∗ R coincides with the standard single-policy concentrability coefficient for MDPs in [15]. This fact is shown in the proof of Corollary 17.

3 RegORL : Learning an Episodic RDP

In this section we present an algorithm for learning the transition function of an RDP R from a dataset D of episodes generated by a regular behavior policy π b . To simplify the presentation, we treat D as a multiset of traces in E H . The learning agent only has access to the non-Markovian traces

Function ADACT-H( D , δ )

in D , and needs prior knowledge of A , R and O , but no prior knowledge of π b and R . Our algorithm is an adaptation of ADACT [10] to episodic RDPs, and we thus refer to the algorithm as ADACT-H.

The intuition behind ADACT-H is that due to Proposition 1, two histories h 1 and h 2 should map to the same RDP state if they induce the same probability distribution on suffixes. ADACT-H starts by adding an initial RDP state q 0 to Q 0 , whose suffixes are the full traces in D (line 1). The algorithm then iteratively constructs the state sets Q 1 , . . . , Q H +1 . In each iteration t ∈ [ H ] , ADACT-H creates a set of candidate states Q c ,t +1 by extending all states in Q t with symbols in AO (line 3). We use qao to simultaneously refer to a candidate state and its state-symbol prefix ( q, ao ) . We associate each candidate state qao with a multiset of suffixes X ( qao ) , i.e. traces in E H -t -1 , obtained by selecting all suffixes in X ( q ) that start with action a and observation o (line 4).

Next, ADACT-H finds the candidate state whose suffix multiset has maximum cardinality, and promotes this candidate to Q t +1 by defining the transition function τ accordingly (lines 5-7). The algorithm then iterates over each remaining candidate state qao ∈ Q c ,t +1 , comparing the distribution on suffixes in X ( qao ) to those of states in Q t +1 (line 9). If the suffix distribution is different from that of each state in Q t +1 , qao is promoted to Q t +1 (line 10), else qao is merged with a state q ′ ∈ Q t +1 that has a similar suffix distribution (line 11). Finally, ADACT-H returns the set of RDP states Q and the associated transition function τ . The function TESTDISTINCT compares two multisets X 1 and X 2 of traces in E H -t -1 using the metric L p ∞ . For i ∈ { 1 , 2 } and each trace e ∈ E H -t -1 , let ̂ p i ( e ) = ∑ x ∈X i I ( x = e ) / |X i | be the empirical estimate of p i , as the proportion of elements in X i equal to e . TESTDISTINCT compares L p ∞ ( X 1 , X 2 ) := L p ∞ ( ̂ p 1 , ̂ p 2 ) to a confidence threshold.

Markov transformation. We are now ready to connect the RDP learning phase with the MDP learning phase. RDPs do not respect the Markov property over their observations and rewards, if automaton states remain hidden. However, we can use the reconstructed transition function τ returned by ADACT-H, extended over histories as ¯ τ : H → Q , to recover the Markov property. In what follows we formalize the notion of Markov transformation and the properties that its outputs satisfy.

Definition 2. Let e 0: H ∈ E H be an episode collected from an RDP R and a policy π b that is regular in R . The Markov transformation of e H with respect to R is the episode constructed as a 0 r 0 q 1 . . . a H r H q H +1 , where q t +1 = ¯ τ ( h t ) and h t = a 0 o 0 · · · a t o t , t ∈ [ H ] . The Markov transformation of a dataset D is the Markov transformation of all the episodes it contains.

A Markov transformation discards all observations from D and replaces them with RDP states output by ¯ τ . The dataset so constructed can be seen as generated from an MDP, which we define next.

Definition 3. The episodic MDP associated to an episodic RDP R is M R = ⟨Q , A , R , T, θ r , H ⟩ , where T ( q, a, q ′ ) = ∑ o ∈O I ( q ′ = τ ( q, ao )) θ o ( q, a, o ) for each ( q, a, q ′ ) ∈ Q × A × Q .

The associated MDP in Definition 3 is the decision process that corresponds to the Markov transformation of Definition 2, i.e. any episode produced with the Markov transformation can be equivalently seen as being generated from the associated MDP, in the sense of the following proposition.

Proposition 3. Let e 0: H be an episode sampled from an episodic RDP R under a regular policy π ∈ Π R , with π ( h, a ) = π r (¯ τ ( h ) , a ) . If e ′ H is the Markov transformation of e H with respect to R , then P ( e ′ H | R , π ) = P ( e ′ H | M R , π r ) , where M R is the MDP associated to R .

Rewards are not affected by the Markov transformation, only observations, implying the following.

Proposition 4. Let π ∈ Π R be a regular policy in R such that π ( h, a ) = π r (¯ τ ( h ) , a ) . Then E [ V π 0 , R ] = E [ V π r 0 , M R ] , where V π 0 , R and V π r 0 , M R are the values in the respective decision process, and E [ V ∗ 0 , R ] = E [ V ∗ 0 , M R ] , where expectations are with respect to randomness in o 0 .

Corollary 5. Given ε ∈ (0 , H ] , if π r : Q → ∆( A ) is an ε -optimal policy of M R , the MDP associated to some RDP R , then, π ( h, a ) = π r (¯ τ ( h ) , a ) is ε -optimal in R .

In summary, from Proposition 3, if D m is the Markov transformation of a dataset D with respect to an RDP R , then, D m can be seen as being generated from the associated MDP M R . Hence, any offline RL algorithm for MDPs can be used for learning in D m . Moreover, according to Corollary 5, any solution for M R can be translated via ¯ τ into a policy for the original RDP, with the same guarantees.

Complete algorithm. The complete procedure is illustrated in Algorithm 1. Initially, the input dataset D is separated in two halves. The first portion is used for learning the transition function of the unknown RDP with ADACT-H (Section 3). If an upper bound Q on |Q| is available, it can optionally be provided to compute a more appropriate failure parameter for ADACT-H. If not available, we adopt the upper bound of 2( AO ) H states, which is valid for any instance, due to histories having finite length. As we will see in Theorem 6, this would only contribute linearly in H to the required dataset size. The output function computed by ADACT-H is then used to compute a Markov transformation of the second phase, as specified in Definition 2. The resulting dataset, now Markovian, can be passed to a generic offline RL algorithm, which we represent with the function OFFLINERL ( D , ε, δ ) . In Appendix D, we instantiate it for a specific state-of-the-art offline RL algorithm.

Algorithm 1: Full procedure ( RegORL )

Input: Dataset D , accuracy ε ∈ (0 , H ] , failure probability 0 < δ < 1 , (optionally) upper bound Q on |Q| Output: Policy ˆ π : H → ∆( A )

1 D 1 , D 2 ← separate D into two datasets of the same size

2 Q , τ ← ADACT-H ( D 1 , δ/ (4 AOQ )) , where Q = 2( AO ) H if not provided

3 D ′ 2 ← Markov transformation of D 2 with respect to ¯ τ as in Definition 2

4 ˆ π m ← OFFLINERL ( D ′ 2 , ε, δ/ 2)

5 return ˆ π : h ↦→ ˆ π m (¯ τ ( h ))

4 Theoretical Guarantees

We now turn to theoretical performance guarantees of RegORL . Our main performance result is a sample complexity bound in Theorem 7, ensuring that, for any accuracy ε ∈ (0 , H ] , RegORL finds an ε -optimal policy. We also report a sample complexity bound for ADACT-H in Theorem 6, and an alternative bound in Theorem 8. In comparison, the sample complexity bound for ADACT [10] is

We achieve a tighter bound by using Bernstein's inequalities and exploiting the finiteness of histories.

Theorem 6. Consider a dataset D of episodes sampled from an RDP R and a regular policy π b ∈ Π R . With probability 1 -δ , the output of ADACT-H ( D , δ/ (2 QAO )) is the transition function of the minimal RDP equivalent to R , provided that |D| ≥ N δ , where

d b min := min { d b t ( q, ao ) | t ∈ [ H ] , q ∈ Q t , ao ∈ AO , d b t ( q, ao ) > 0 } is the minimal occupancy distribution, and µ 0 is the L p ∞ -distinguishability.

The proof appears in Appendix C.2. Theorem 6 tells us that the sample complexity of ADACT-H, to return a minimal RDP, is inversely proportional to µ 0 , the L p ∞ -distinguishability of R and π b , and the minimal occupancy d b min . Note that d b min ≤ 1 / ( QOA ) . The bound also depends on Q , the number of RDP states, implicitly through d b min and explicitly via a logarithmic term. In the absence of prior knowledge of Q , one may use in the argument of Algorithm 1 the worst-case upper bound Q = 2( AO ) H . The sample complexity would then have an additional linear term in H , since Q is only used in the logarithmic term to set the appropriate value of δ . However, this will not impact the value of the d b min term.

Theorem 6 is a sample complexity guarantee for the first phase of the algorithm, which learns τ , the structure of the minimal RDP that is equivalent to the underlying RDP. If δ is the desired failure probability of the complete algorithm, RegORL executes ADACT-H so that its success probability is at least 1 -δ/ 2 . This means that with the same probability, D ′ 2 is an MDP dataset with the properties listed in Section 3. As a consequence, provided that OFFLINERL is some generic ( ε, δ/ 2) -PAC offline RL algorithm for MDPs, the output of RegORL is an ε -optimal policy with probability 1 -δ .

Theorem 7. Consider a dataset D of episodes sampled from an RDP R and a regular policy π b ∈ Π R . For any ε ∈ (0 , H ] and 0 < δ < 1 , if OFFLINERL is an ( ε, δ/ 2) -PAC offline algorithm for MDPs with sample complexity N m , then, the output of RegORL ( D , ε, δ ) is an ε -optimal policy in R , with probability at least 1 -δ , provided that |D| ≥ 2 max { N δ/ 2 , N m } .

As we can see, the sample complexity requirement separates for the two phases. While N δ/ 2 is due to the RDP learning component, defined in Theorem 6, the quantity N m completely depends on the offline RL algorithm for MDPs that is adopted. Among other terms, the performance guarantees of offline algorithms can often be characterized through the single-policy concentrability for MDPs C ∗ . However, since states become observations in the associated MDP, due to the properties of Proposition 3, C ∗ coincides with C ∗ R , the RDP single-policy concentrability of Definition 1.

In Appendix D, we demonstrate a specific instantiation of RegORL with an off-the-shelf offline RL algorithm from the literature by Li et al. [16]. This yields the following requirement for N m :

for a constant c > 0 .

To eliminate the dependence that Theorem 6 has on d b min , we develop a variant of ADACT-H which does not learn a complete RDP. Rather, it only reconstructs a subset of states that are likely under the behavior policy. The algorithm, which we call ADACT-H-A (with 'A' standing for 'approximation'), is defined in Appendix C.3. Theorem 8 is an upper bound on the sample complexity of ADACT-H-A that takes the accuracy ε as input and returns the transition function of an ε/ 2 -approximate RDP R ′ , whose optimal policy is ε/ 2 -optimal for the original RDP R . By performing a Markov transformation for R ′ and using an ( ε/ 2 , δ/ 2) -PAC offline algorithm for MDPs, we can compute an ε -optimal policy for R . The total sample complexity can be combined in the same way as in Theorem 7.

Theorem 8. Consider a dataset D of episodes sampled from an RDP R and a regular policy π b ∈ Π R . With probability 1 -δ , the output of ADACT-H -A, called with D , δ/ (2 QAO ) and ε ∈ (0 , H ] in input, is the transition function of an ε/ 2 -approximate RDP R ′ , provided that |D| ≥ N ′ δ , where

This theorem does not rely on Assumption 1, because a finite C ∗ R suffices.

5 Sample Complexity Lower Bound

The main result of this section is Theorem 9, a sample complexity lower bound for offline RL in RDPs. It shows that the dataset size required by any RL algorithm scales with the relevant parameters.

q

01

∗

.u

q

02

∗

.u

q

03

a

′

0

.v

ξ

a

′

1

.v

-

ξ

q

s

∗

.s

q

11

∗

.u

q

12

∗

.u

q

13

a

′

0

.v

-

ξ

a

′

1

.v

ξ

q

b

1

∗

.

+

q

b

2

∗

.

+

q

b

3

a

′

0

.u

+

a

′

1

.v

η

q

+4

q

+5

q

-

4

q

-

5

q

o

⊥

0

1

∗

∗

0

1

0

1

-

+

+

∗

∗

+

-

∗

∗

∗

∗

∗

Theorem 9. For any ( C ∗ R , H, ε, µ 0 ) satisfying C ∗ R ≥ 2 , H ≥ 2 and ε ≤ Hµ 0 / 64 , there exists an RDP with horizon H , L p 1 -distinguishability µ 0 and a regular behavior policy π b with RDP single-policy concentrability C ∗ R , such that if D has been generated using π b and R , and

then, for any algorithm A : D ↦→ ̂ π returning non-Markov deterministic policies, the probability that ̂ π is not ε -optimal is at least 1 / 4 .

The proof relies on worst-case RDP instances that carefully combine two-armed bandits with noisy parity functions. This last component allows to capture the difficulty of learning in presence of temporal dependencies. Figure 2 shows an RDP in this class. At the beginning of each episode, the observation causes a transition towards either the bandit component (bottom branch) or the noisy parity function (top branches). Acting optimally in the two parity branches requires to predict the output of a parity function, which depends on some unknown binary code (of length 3 , in the example). The first term in Theorem 9 is due to this component, because the code scales linearly with H , or Q , while the amount of information revealed about the code is controlled by µ 0 . The second term is caused by the required optimality in the bandit.

Differently from this lower bound, the parameter µ 0 , appearing in the upper bounds of Theorems 6 and 8, is a L p ∞ -distinguishability. However, the two are related, since L p 1 ( q, q ′ ) ≥ L p ∞ ( q, q ′ ) . Intuitively, the L p 1 -distinguishability accounts for all the information that is available as differences in episode probabilities. The L p ∞ -distinguishability, on the other hand, quantifies the maximum the difference in probability associated to specific suffixes. This is the information used by the algorithm and the one appearing in the two upper bounds.

6 Conclusion

In this paper we propose an algorithm for Offline RL in episodic Regular Decision Processes. Our algorithm exploits automata learning techniques to reduce the problem of RL in RDPs, in which observations and rewards are non-Markovian, into standard offline RL for MDPs. We provide the first high-probability sample complexity guarantees for this setting, as well as a lower bound that shows how its complexity relates to the parameters that characterize the decision process and the behavior policy. We identify the RDP single-policy concentrability as an analogous quantity to the one used for MDPs in the literature. Our sample complexity upper bound depends on the L p ∞ -distinguishability of the behavior policy. As a future work, we plan to investigate if any milder notion of distinguishability also suffices. This is motivated by our lower bound which only involves the L p 1 -distinguishability over the same policy. Finally, our results have strong implications for online learning in RDPs, which is a relevant setting to be explored.

Acknowledgments

Roberto Cipollone is partially supported by the EU H2020 project AIPlan4EU (No. 101016442), the ERC-ADG White-Mech (No. 834228), the EU ICT-48 2020 project TAILOR (No. 952215), the PRIN project RIPER (No. 20203FFYLK), and the PNRR MUR project FAIR (No. PE0000013). Anders Jonsson is partially supported by the EU ICT-48 2020 project TAILOR (No. 952215), AGAUR SGR, and the Spanish grant PID2019-108141GB-I00. Alessandro Ronca is partially supported by the ERC project WhiteMech (No. 834228), and the ERC project ARiAT (No. 852769). Mohammad Sadegh Talebi is partially supported by the Independent Research Fund Denmark, grant number 1026-00397B.

References

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . Wiley, 1994.

Fahiem Bacchus, Craig Boutilier, and Adam J. Grove. Rewarding behaviors. In AAAI , pages 1160-1167, 1996.

Ronen I. Brafman and Giuseppe De Giacomo. Regular decision processes: A model for non-markovian domains. In IJCAI , pages 5516-5522, 2019.

Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artif. Intell. , 101(1-2):99-134, 1998.

Christos H. Papadimitriou and John N. Tsitsiklis. The complexity of Markov decision processes. Mathematics of Operations Research , 12(3):441-450, 1987.

Chi Jin, Sham M. Kakade, Akshay Krishnamurthy, and Qinghua Liu. Sample-efficient reinforcement learning of undercomplete POMDPs. In NeurIPS , 2020.

Hongyi Guo, Qi Cai, Yufeng Zhang, Zhuoran Yang, and Zhaoran Wang. Provably efficient offline reinforcement learning for partially observable Markov decision processes. In ICML , pages 8016-8038, 2022.

Zhaohan Daniel Guo, Shayan Doroudi, and Emma Brunskill. A PAC RL algorithm for episodic POMDPs. In AISTATS , pages 510-518, 2016.

Kamyar Azizzadenesheli, Alessandro Lazaric, and Animashree Anandkumar. Reinforcement learning of POMDPs using spectral methods. In COLT , pages 193-256, 2016.

Borja Balle, Jorge Castro, and Ricard Gavaldà. Learning probabilistic automata: A study in state distinguishability. Theor. Comput. Sci. , 473:46-60, 2013.

Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In ICLR , 2022.

Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. In NeurIPS , pages 4065-4078, 2021.

Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In ICML , pages 1042-1051, 2019.

Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sampleefficient offline and online reinforcement learning. In NeurIPS , pages 27395-27407, 2021.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. In NeurIPS , pages 11702-11716, 2021.

Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. CoRR , abs/2204.05275, 2022.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In COLT , pages 2730-2775, 2022.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In ICML , pages 5084-5096, 2021.

Tongzheng Ren, Jialian Li, Bo Dai, Simon S Du, and Sujay Sanghavi. Nearly horizon-free offline reinforcement learning. In NeurIPS , pages 15621-15634, 2021.

Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline RL in low-rank MDPs. In ICLR , 2022.

Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning , pages 2139-2148. PMLR, 2016.

Hamid Reza Maei, Csaba Szepesvári, Shalabh Bhatnagar, and Richard S Sutton. Toward off-policy learning control with function approximation. In ICML , pages 719-726, 2010.

Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient off-policy evaluation in Markov decision processes. The Journal of Machine Learning Research , 21(1):6742-6804, 2020.

Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355 , 2022.

Eden Abadi and Ronen I. Brafman. Learning and solving regular decision processes. In IJCAI , pages 1948-1954, 2020.

Alessandro Ronca and Giuseppe De Giacomo. Efficient PAC reinforcement learning in regular decision processes. In IJCAI , pages 2026-2032, 2021.

Alessandro Ronca, Gabriel Paludo Licks, and Giuseppe De Giacomo. Markov abstractions for PAC reinforcement learning in non-Markov decision processes. In IJCAI , pages 3408-3415, 2022.

Rodrigo Toro Icarte, Ethan Waldie, Toryn Q. Klassen, Richard Anthony Valenzano, Margarita P. Castro, and Sheila A. McIlraith. Learning reward machines for partially observable reinforcement learning. In NeurIPS , pages 15497-15508, 2019.

Marcus Hutter. Feature reinforcement learning: Part I. Unstructured MDPs. J. Artif. Gen. Intell. , 1(1): 3-24, 2009.

Joel Veness, Kee Siong Ng, Marcus Hutter, William T. B. Uther, and David Silver. A Monte-Carlo AIXI approximation. J. Artif. Intell. Res. , 40:95-142, 2011.

M. M. Hassan Mahmud. Constructing states for reinforcement learning. In ICML , pages 727-734, 2010.

Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan. Approximate information state for approximate planning and reinforcement learning in partially observed systems. The Journal of Machine Learning Research , 23(1):483-565, 2022.

Karl Johan Åström. Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis and Applications , 10(1):174-205, 1965.

Satinder Singh, Michael L. Littman, Nicholas K. Jong, David Pardoe, and Peter Stone. Learning predictive state representations. In ICML , pages 712-719, 2003.

Michael R. James and Satinder Singh. Learning and discovery of predictive state representations in dynamical systems with reset. In ICML , 2004.

Michael H. Bowling, Peter McCracken, Michael James, James Neufeld, and Dana F. Wilkinson. Learning predictive state representations using non-blind policies. In ICML , pages 129-136, 2006.

Alex Kulesza, Nan Jiang, and Satinder Singh. Spectral learning of predictive state representations with insufficient statistics. In AAAI , pages 2715-2721, 2015.

Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D. Lee. PAC reinforcement learning for predictive state representations. In ICLR , 2023.

Odalric-Ambrym Maillard, Rémi Munos, and Daniil Ryabko. Selecting the state-representation in reinforcement learning. In NeurIPS , pages 2627-2635, 2011.

Phuong Nguyen, Odalric-Ambrym Maillard, Daniil Ryabko, and Ronald Ortner. Competing with an infinite set of models in reinforcement learning. In AISTATS , pages 463-471, 2013.

Odalric-Ambrym Maillard, Phuong Nguyen, Ronald Ortner, and Daniil Ryabko. Optimal regret bounds for selecting the state representation in reinforcement learning. In ICML , pages 543-551, 2013.

Ronald Ortner, Matteo Pirotta, Alessandro Lazaric, Ronan Fruit, and Odalric-Ambrym Maillard. Regret bounds for learning state representations in reinforcement learning. In NeurIPS , pages 12717-12727, 2019.

Tor Lattimore, Marcus Hutter, and Peter Sunehag. The sample-complexity of general reinforcement learning. In ICML , pages 28-36, 2013.

Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Anthony Valenzano, and Sheila A. McIlraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In ICML , pages 2112-2121, 2018.

Zhe Xu, Ivan Gavran, Yousef Ahmad, Rupak Majumdar, Daniel Neider, Ufuk Topcu, and Bo Wu. Joint inference of reward machines and policies for reinforcement learning. In ICAPS , pages 590-598, 2020.

Giuseppe De Giacomo, Luca Iocchi, Marco Favorito, and Fabio Patrizi. Foundations for restraining bolts: Reinforcement learning with LTLf/LDLf restraining specifications. In ICAPS , pages 128-136, 2019.

Giuseppe De Giacomo, Marco Favorito, Luca Iocchi, Fabio Patrizi, and Alessandro Ronca. Temporal logic monitoring rewards via transducers. In KR , pages 860-870, 2020.

Mohammadhosein Hasanbeig, Natasha Yogananda Jeppu, Alessandro Abate, Tom Melham, and Daniel Kroening. DeepSynth: Automata synthesis for automatic task segmentation in deep reinforcement learning. In AAAI , pages 7647-7656, 2021.

Hippolyte Bourel, Anders Jonsson, Odalric-Ambrym Maillard, and Mohammad Sadegh Talebi. Exploration in reward machines with low regret. In International Conference on Artificial Intelligence and Statistics , pages 4114-4146. PMLR, 2023.

Jie Fu and Ufuk Topcu. Probably approximately correct MDP learning and control with temporal logic constraints. In RSS , 2014.

Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Cautious reinforcement learning with logical constraints. In AAMAS , pages 483-491, 2020.

Lewis Hammond, Alessandro Abate, Julian Gutierrez, and Michael J. Wooldridge. Multi-agent reinforcement learning with temporal logic specifications. In AAMAS , pages 583-592, 2021.

Alper Kamil Bozkurt, Yu Wang, Michael M. Zavlanos, and Miroslav Pajic. Control synthesis from linear temporal logic specifications using model-free reinforcement learning. In ICRA , pages 10349-10355, 2020.

Ernst Moritz Hahn, Mateo Perez, Sven Schewe, Fabio Somenzi, Ashutosh Trivedi, and Dominik Wojtczak. Omega-regular objectives in model-free reinforcement learning. In TACAS , pages 395-412, 2019.

Sultan Javed Majeed and Marcus Hutter. On Q-learning convergence for non-Markov decision processes. In IJCAI , pages 2546-2552, 2018.

Dana Ron, Yoram Singer, and Naftali Tishby. On the learnability and usage of acyclic probabilistic finite automata. J. Comput. Syst. Sci. , 56(2):133-152, 1998.

Alexander Clark and Franck Thollard. PAC-learnability of probabilistic deterministic finite state automata. J. Mach. Learn. Res. , 5:473-497, 2004.

Nick Palmer and Paul W. Goldberg. PAC-learnability of probabilistic deterministic finite state automata in terms of variation distance. Theor. Comput. Sci. , 387(1):18-31, 2007.

Borja Balle. Learning Finite-State Machines: Statistical and Algorithmic Aspects . PhD thesis, Universitat Politècnica de Catalunya, 2013.

Borja Balle, Jorge Castro, and Ricard Gavaldà. Adaptively learning probabilistic deterministic automata from data streams. Machine Learning , 96(1):99-127, 2014.

Edward Forrest Moore. Gedanken-experiments on sequential machines. Automata studies , 34:129-153, 1956.

John E. Hopcroft, Rajeev Motwani, and Jeffrey D. Ullman. Introduction to Automata Theory, Languages, and Computation . Addison-Wesley Longman Publishing Co., Inc., 2006.

Jeffrey Shallit. A second course in formal languages and automata theory . Cambridge University Press, 2008.

Ronen I. Brafman, Giuseppe De Giacomo, and Fabio Patrizi. LTLf/LDLf non-markovian rewards. In AAAI , pages 1771-1778, 2018.

Maor Gaon and Ronen I. Brafman. Reinforcement learning with non-markovian rewards. In AAAI , pages 3980-3987, 2020.

Andrew Kachites McCallum. Reinforcement Learning with Selective Perception and Hidden State . PhD thesis, University of Rochester, 1996.

Jorma Rissanen. A universal data compression system. IEEE Transactions on information theory , 29(5): 656-664, 1983.

Dana Ron, Yoram Singer, and Naftali Tishby. The power of amnesia: Learning probabilistic automata with variable memory length. Mach. Learn. , 25(2-3):117-149, 1996.

Marcus Hutter. Extreme state aggregation beyond Markov decision processes. Theor. Comp. Sci. , pages 73-91, 2016.

Andreas Maurer and Massimiliano Pontil. Empirical Bernstein bounds and sample-variance penalization. In COLT , 2009.

Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying PAC and regret: Uniform PAC bounds for episodic reinforcement learning. In NeurIPS , pages 5713-5723, 2017.

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (2. Ed.) . Wiley, 2006.

Balázs Szörényi. Characterizing statistical query learning: Simplified notions and proofs. In ALT , pages 186-200, 2009.

Avrim Blum, Merrick L. Furst, Jeffrey C. Jackson, Michael J. Kearns, Yishay Mansour, and Steven Rudich. Weakly learning DNF and characterizing statistical query learning using Fourier analysis. In STOC , pages 253-262, 1994.

Sumegha Garg, Ran Raz, and Avishay Tal. Extractor-based time-space lower bounds for learning. In STOC , pages 990-1002, 2018.

Sumegha Garg, Pravesh K. Kothari, Pengda Liu, and Ran Raz. Memory-sample lower bounds for learning parity with noise. CoRR , abs/2107.02320, 2021.

Appendices

A Extended Discussion of Related Work

Some of the related work mentioned in the introduction requires a more extensive discussion, which we provide below.

A.1 Offline RL in MDPs

There is a rich and growing literature on offline RL in MDPs; see, e.g., [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. A closely related line of work is off-policy learning in MDPs [21, 22, 23]; we refer to the recent survey [24] for a discussion on how the various settings are related. For offline RL in MDPs, the papers cited above report learning algorithms with theoretical guarantees on their sample efficiency. The majority of these algorithms are designed based on the pessimism principle . While most literature focuses on tabular MDPs, the case of linear function approximation is discussed in some papers, e.g., [20]. In several settings, the presented algorithms are shown to be minimax optimal. For instance, in the case of tabular episodic MDPs, it is established in [16] that the optimal sample complexity depends on the size of state-space, episode length, as well as some notion of concentrability reflecting the distribution mismatch between the behavior and optimal policy.

A.2 Online RL in RDPs

RDPs have been introduced in [3] as a formalism based on temporal logic. They admit an equivalent formulation in terms of automata, which is favoured in the context of RL. Several algorithms for online RL in RDPs exist [25, 26, 27] but complexity bounds are only given in [26] for the infinitehorizon discounted setting. This work [26] shows the correspondence between RDPs and Probabilistic Deterministic Finite Automata (PDFA), and it introduces the idea of using PDFA-learning techniques to learn RDPs. Their sample complexity bounds are not immediately comparable to ours, due to the different setting. Importantly, this algorithm uses the uniform policy for learning. So, the algorithm might be adapted to our setting only under the assumption that the behaviour policy is uniform. Even in this case, our bounds show an improved dependency on several key quantities. Furthermore, we provide a sample complexity lower bound, whereas their results are limited to showing that a dependency on the quantities occurring in their upper bounds is necessary.

The first RL algorithm for RDPs appears in [25] for the online discounted setting. It is automatonbased, and in particular, it learns the RDP in the form of a Mealy machine. The algorithm is shown in [26] to incur an exponential dependency on the length of the relevant histories. An algorithm that achieves performance guarantees building on the techniques from [26] but also integrating an effective exploration strategy is given in [27]. This work also introduces the idea of seeing the transition function of a PDFA as a Markov abstraction of the histories to be passed to an RL algorithm for MDPs, so as to employ it in a modular manner.

The algorithms in [28, 29, 30, 31] apply to RDPs even though they have not been developed specifically for RDPs. Toro Icarte et al. [28] present an RL algorithm for the subclass of POMDPs that have a finite set of reachable belief states. Such POMDPs can be captured by finite-state automata, and are in fact RDPs. Their algorithm is based on automata learning, but it does not come with an analysis of its performance guarantees. The RL techniques presented in [29, 30] for feature MDPs are in fact applicable to episodic RDPs. The techniques are based on suffix trees, rather than automata. There are cases when the size of the smallest suffix tree is exponential in the horizon, while an automaton of linear size exists-see section below on feature MDPs. Thus their techniques cannot yield optimal bounds for RDPs. Mahmud [31] introduces an RL algorithm for Deterministic Markov Decision Models (MDDs). Such MDDs are also automaton-based, and they are more general than RDPs since the automaton is only required to predict rewards. Thus their RL algorithm applies to RDPs. However, it is an algorithm without guarantees.

A.3 Non-Markov Rewards and Reward Machines

MDPswith non-Markov rewards are a special case of NMDPs, where only rewards are non-Markovian. Namely, observed states satisfy the Markov property, while rewards may depend on the history of past states. The specific kind of non-Markov rewards considered in the literature amount to the subclass of RDPs where the automaton's state is only needed to predict the next reward-while the next observation (i.e., state) can be predicted from the last observation.

Non-Markov rewards can already be found in [2], where the reward function is specified in a temporal logic of the past. More recently, the setting has been revisited with so-called reward machines [44] as well as with temporal logics of the future on finite traces [64, 47]. A reward machine is a finite automaton (or transducer) used to specify a non-Markovian reward function. Reward machines have been introduced in [44] along with an RL algorithm that assumes the reward machine to be known. RL algorithms with unknown reward machine, or equivalently unknown temporal specification, are presented in [65, 45], with no performance guarantees. Reward machines have been generalised so as to predict observations as well [28], which makes them equivalent to RDPs-as mentioned above.

The first performance bounds for RL with reward machines have been recently established in [49]. The work shows regret bounds that take into account the structure provided by a reward machine, and hence improve over the bounds that one would obtain by naively adapting regret bounds for MDPs.

An automaton-based method for dealing with non-Markov sparse rewards is proposed in [48].

A.4 State Representations

State representations are maps from histories to a finite state space. The map defined by the transition function of an RDP is a state representation. The studies on state representations [39, 40, 41, 42] focus on regret bounds for RL given a candidate set of state representations. While in our case the state representations are concretely defined by the class of finite-state automata, in their case they are arbitrary maps. This is a challenging setting, which does not allow for taking advantage of the properties of specific classes of state representations. The regret bounds in [39, 41, 42] are for finite sets of state representations, and they all show a linear dependency on the cardinality of the given set of state representations. In our case, the candidate state representations correspond to the set of automata with at most Q = 2( AO ) H states and AO input letters. Such a set contains at least Q QAO automata-the number of distinct transition functions. Thus, if we could instantiate their bounds in our setting, they would have an exponential dependency on the number Q of RDP states, and hence a doubly-exponential dependency on the horizon H . We avoid this dependency, obtaining polynomial bounds in the mentioned quantities.

Nguyen et al. [40] consider the case of a countably infinite set of state representations, and present an algorithm whose regret bound does not show a dependency such as the one discussed above. Instead, they show a dependency on a quantity K 0 , which admits several interpretations, including one based on the descriptional complexity of the candidate state representations. Thus, there may be a way to relate K 0 to the quantities we use in our bounds. However, the formal relationship between the two, if any, renders highly non-trivial, which prevents one to use their ideas in the case of RDPs. We believe establishing a fomal relationship between their model and RDPs is an interesting, yet challenging, topic for future work. Furthermore, it should be stressed that even if the relationship was clear and one could borrow ideas from [40], the resulting sample complexity bound would have to grow as 1 /ε 3 in view of their regret bound scaling as T 2 / 3 . In contrast, our bounds achieve an optimal dependency of 1 /ε 2 on ε .

A.5 Feature MDPs and General RL

Hutter [29] introduces feature MDPs , where histories are mapped to states by a feature map. It relates to our work since the map provided by the transition function of an RDP is a feature map. The concrete feature maps considered in [29] are based on U-Trees [66]. The idea is also revisited in [30] with Prediction Suffix Trees (PSTs) [67, 68]. Both U-Trees and PSTs are suffix trees. There are cases when their size is exponential in the horizon, while an automaton of linear size exists. For instance, in the case of a parity condition over the history. To see this, note that a suffix x of a bit string bx does not suffice to establish parity of bx . In fact, the parity of 0 x is different from the parity of 1 x . Thus a suffix tree for parity must encode all suffixes, and hence it will have a number of leaves that is exponential in the maximum length of a relevant string-the horizon H in the case of episodic RL.

Hutter [69] provides several formal characterisations of feature maps. All the characterisations are less restrictive than the one defined by an RDP. In particular, their most restrictive characterisation is given in their Equation (6). It only requires to preserve the ability to predict rewards, not observations-this is also adopted by Mahmud [31]. The states of our automata suffice to predict observations as well. It is unclear whether automata techniques can be used to learn directly abstractions that do not preserve the dynamics entirely.

Majeed and Hutter [55] study convergence of Q-learning when executed over the state space of an MDP that underlies a non-Markov decision processes. Such a state space corresponds to the state space of the RDP, but they do not consider the problem of learning the state space.

Lattimore et al. [43] consider General RL as the problem of RL when we are given a set of candidate NMDPs, rather than assuming the decision process to belong to a fixed class. Similarly to the works on state representations, it does not commit to specific classes of NMDPs, and their bounds have a linear dependency on the number of candidate models. As remarked above, in our setting, it amounts to an exponential dependency on the number of states of the candidate RDPs, and hence a doubly-exponential dependency on the horizon; we avoid such exponential dependencies.

A.6 Learning PDFA

Our algorithms for learning an RDP borrow and improve over techniques for learning ProbabilisticDeterministic Finite Automata (PDFA). The first PAC learning algorithm for acyclic PDFA has been presented in [56], then followed by extensions and variants that can handle PDFA with cycles [57, 58, 10, 59, 60]. All bounds feature some variant of a distinguishability parameter , which we adopt in our bounds, properly adapting it to the offline RL setting. Our algorithm builds upon the state-of-the-art algorithm ADACT [10], and we derive bounds that are a substantial improvement over the ones that can be obtained from a straightforward application of any existing PDFA-learning algorithm to the offline RL setting.

B RDP properties

In this section we prove several properties of RDPs that are stated in Sections 2 and 3.

B.1 RDPs and Regular Policies

In this section, we prove Propositions 1 and 2.

Proposition 1. Consider an RDP R , a regular policy π ∈ Π R and two histories h 1 and h 2 in H t , t ∈ [ H ] , such that ¯ τ ( h 1 ) = ¯ τ ( h 2 ) . For each suffix e t +1: H ∈ E H -t -1 , the probability of generating e t +1: H is the same for h 1 and h 2 , i.e. P ( e t +1: H | h 1 , π, R ) = P ( e t +1: H | h 2 , π, R ) .

Proof. By induction on t . For t = H , all histories in H H generate the empty suffix in ( ARO ) 0 with probability 1 (the stop symbol is omitted). For t < H , the probability of generating a suffix aroe t +2: H is

P ( aroe t +2: H | h 1 , π, R ) = π ( h 1 , a ) · P ( r, o | ¯ τ ( h 1 ) , a, R ) · P ( e t +2: H | h 1 ao, π, R ) = π ( h 2 , a ) · P ( r, o | ¯ τ ( h 2 ) , a, R ) · P ( e t +2: H | h 2 ao, π, R ) = P ( aroe t +2: H | h 2 , π, R ) , where we have used the fact that π is regular, ¯ τ ( h 1 ) = ¯ τ ( h 2 ) , ¯ τ ( h 1 ao ) = τ (¯ τ ( h 1 ) , ao ) = τ (¯ τ ( h 2 ) , ao ) = ¯ τ ( h 2 ao ) , and the induction hypothesis.

The following statement appears in the literature [3, Theorem 2], but the authors do not provide a complete proof, so for completeness we prove the statement here.

Proposition 2. Each RDP R has at least one optimal policy π ∗ ∈ Π R .

Proof. Given R , consider any optimal policy π ∗ : H → ∆( A ) , not necessarily regular. We prove the statement by constructing a policy π and showing by induction on t ∈ [ H ] that π is both optimal and regular. The base case is given by t = H . In this case, for an arbitrary a ∈ A , define π ( h ) := ✶ a for each history h ∈ H H . Since V π H ( h ) = 0 by definition, π is optimal for each history h ∈ H H , and regular since it always selects the same action.

For t < H , we first construct a new policy π c which is the composition of policies π ∗ and π . Concretely, for each history h ∈ H u such that u ≤ t , π c ( h ) = π ∗ ( h ) acts according to π ∗ , while for each history h ∈ H u such that u > t , π c ( h ) = π ( h ) acts according to π . Clearly, π c is an optimal policy for R since π ∗ is optimal and since by induction, π is optimal for histories in H u , u > t .

̸

Consider a pair of histories h 1 and h 2 in H t such that ¯ τ ( h 1 ) = ¯ τ ( h 2 ) but π c ( h 1 ) = π c ( h 2 ) . Define π ( h 1 ) := π ( h 2 ) := π c ( h 1 ) . Since the value function can be written as an expectation

over suffixes, due to Proposition 1 and the fact that π is regular for histories in H u , u > t , we have V π t ( h 1 ) = V π t ( h 2 ) . Since π c is the same as π for histories in H u , u > t , this implies V π t ( h 1 ) = V π c t ( h 1 ) ≤ V π c t ( h 2 ) since π c is optimal for h 2 . If we were to instead define π ( h 1 ) := π ( h 2 ) := π c ( h 2 ) , we would obtain V π c t ( h 2 ) ≤ V π c t ( h 1 ) . The only possibility is V π c t ( h 1 ) = V π c t ( h 2 ) , which is the same value achieved by the policy π . Hence π is optimal for h 1 and h 2 .

̸

We now repeat the same procedure for each pair of histories h 1 and h 2 in H t such that ¯ τ ( h 1 ) = ¯ τ ( h 2 ) but π c ( h 1 ) = π c ( h 2 ) . If necessary, we complete the definition of π by copying the action choices of π c . The resulting policy π is optimal for each history h ∈ H t , and regular since it makes the same action choices for each pair of histories h 1 and h 2 in h ∈ H t such that ¯ τ ( h 1 ) = ¯ τ ( h 2 ) .

B.2 Markov Transformation

In this section, we verify the properties of the Markov transformation, which is the intermediate step that RegORL uses to recover the Markov property in the original dataset. This transformation has been formalized in Definition 2. We use D and D ′ to denote the original and the transformed datasets, respectively.

Proposition 3. Let e 0: H be an episode sampled from an episodic RDP R under a regular policy π ∈ Π R , with π ( h, a ) = π r (¯ τ ( h ) , a ) . If e ′ H is the Markov transformation of e H with respect to R , then P ( e ′ H | R , π ) = P ( e ′ H | M R , π r ) , where M R is the MDP associated to R .

Proof. For t ∈ [ H ] , let e t ∈ E t = ( ARO ) t +1 be an episode prefix in R , ϕ ( e t ) ∈ E ′ t = ( ARQ ) t +1 be its Markov transformation and e ′ t ∈ E ′ t be an episode of the associated MDP. The function ϕ : E → E ′ transforms the observations according to ¯ τ , and preserves actions and rewards. The statement says that P ( ϕ ( e t ) | R , π ) = P ( e ′ t | M R , π r ) (note that ϕ ( e t ) and e ′ t are distinct random variables). We prove this by induction. For t = 0 , we recall that the irrelevant quantities a 0 , r 0 are constant and,

where T : Q×A→ ∆( Q ) is the transition function of M R , from Definition 3. Due to the role of the dummy action, T ( q 0 , a 0 ) is the initial distribution of the MDP.

For the inductive step, assume that P ( ϕ ( e t -1 ) | R , π ) = P ( e ′ t -1 | M R , π r ) . Then, for any e ′ ∈ E ′ t -1 , arq ∈ ARQ , if q ′ is the last element of e ′ , we have

where, in (6), we have used the induction hypothesis and the fact that a t r t q t +1 are Markov in q ′ by regularity of the policy.

Thanks to this relation, the values of corresponding policies are also related in the following way.

Proposition 4. Let π ∈ Π R be a regular policy in R such that π ( h, a ) = π r (¯ τ ( h ) , a ) . Then E [ V π 0 , R ] = E [ V π r 0 , M R ] , where V π 0 , R and V π r 0 , M R are the values in the respective decision process, and E [ V ∗ 0 , R ] = E [ V ∗ 0 , M R ] , where expectations are with respect to randomness in o 0 .

Proof. The statement is composed of two parts. First, we show that E [ V π 0 , R ] = E [ V π r 0 , M R ] , which is a direct consequence of Proposition 3. Following the same convention as in the proof of Proposition 3,

we use E ′ t = ( ARQ ) t +1 and ϕ for the Markov transformation. Then,

For the second part of the statement, let Π R and Π M be the regular and the Markov policies in R and M R , respectively. Then, using Proposition 2 and the first part of this statement,

Corollary 5 is a consequence of the two parts of Proposition 4. Since V π 0 , R = V π r 0 , M R and the value achieved by the optimal policy in the respective model is the same, ε -optimality of π in R implies ε -optimality of π r in M M , and vice versa.

C Sample Complexity of ADACT-H

In this section we prove high-probability upper bounds on the sample complexity of ADACT-H.

C.1 Preliminaries

We first state Hoeffding's inequality for Bernoulli variables. In what follows we take log to be the natural logarithm.

Lemma 10 (Hoeffding's inequality) . Let X 1 , . . . , X N be N independent random Bernoulli variables with the same expected value E [ X 1 ] = p , and let ̂ p N = ∑ N i =1 X i /N be an empirical estimate of p . Then, for any δ ∈ (0 , 1) ,

An alternative to Hoeffding's inequality is the empirical Bernstein inequality, which can be expressed as follows for Bernoulli variables [70, 71].

Lemma 11 (Empirical Bernstein inequality) . Let X 1 , . . . , X N be N independent random Bernoulli variables with the same expected value E [ X 1 ] = p , and let ̂ p N = ∑ N i =1 X i /N be an empirical estimate of p . Then, for any δ ∈ (0 , 1) ,

If X ∼ p X is a discrete random variable, the entropy of X is H ( X ) = -∑ x ∈X p X ( x ) log p X ( x ) . Further, for x ∈ (0 , 1) , we define the binary entropy function as H 2 ( x ) = -x log( x ) -(1 -x ) log(1 -x ) . If ( X,Y ) ∼ p XY are two discrete variables, the conditional entropy is H ( Y | X ) = ∑ x ∈X p X ( x ) H ( Y | X = x ) . The mutual information is I ( X ; Y ) = I ( Y ; X ) = D KL ( p XY ∥ p X · p Y ) , where D KL is the Kullback-Leibler divergence. If X,Y,Z are three random variables, we write X → Y → Z if the conditional distribution of Z does not depend on X , given Y . With these definition, we state Fano's inequality, as one can find in Cover and Thomas [72], (2.140).

̸

C.2 Proof of Theorem 6

In this section we prove Theorem 6, which states a high-probability upper bound on the sample complexity of ADACT-H. The first two lemmas are adaptations of Lemmas 19 and 20 in Balle et al. [10] to the episodic setting.

Lemma 13. For t ∈ [ H ] , let X 1 and X 2 be multisets sampled from distributions p 1 and p 2 in ∆( E H -t -1 ) . If p 1 = p 2 , then TESTDISTINCT ( t, X 1 , X 2 , δ ) returns False with probability 1 -δ .

Proof. For each i ∈ { 1 , 2 } and each trace e ∈ E H -t -1 , we can view each episode as a random Bernoulli variable with expected value p i ( e ) that takes value 1 if we observe e , and 0 otherwise. Let ̂ p i ( e ) = ∑ x ∈X i I ( x = e ) / |X i | be the empirical estimate of p i , i.e. the proportion of elements in X i equal to e . For each i ∈ { 1 , 2 } , each u ∈ [ H -t -1] and each prefix e 0: u ∈ E u , Hoeffding's inequality yields

The total number of non-empty prefixes of E H -t -1 equals a geometric sum:

Choosing δ s = δ/ 4( ARO ) H -t and taking a union bound implies that the above inequality holds for each i ∈ { 1 , 2 } and each e 0: u simultaneously with probability 1 -4( ARO ) H -t δ s = 1 -δ , implying

which is precisely the condition under which TESTDISTINCT ( t, X 1 , X 2 , δ ) returns False.

Lemma 14. For t ∈ [ H ] , let X 1 and X 2 be multisets sampled from distributions p 1 and p 2 in ∆( E H -t -1 ) . If the L p ∞ -distinguishability of π b is µ 0 , then TESTDISTINCT ( t, X 1 , X 2 , δ ) returns True with probability 1 -δ provided that

Proof. Using the same argument as in the proof of Lemma 13, Hoeffding's inequality yields

with the inequality holding simultaneously for i ∈ { 1 , 2 } and each prefix e 0: u with probability 1 -δ by choosing δ s = δ/ 4( ARO ) H -t . Choosing µ 0 ≥ 4 √ log(2 /δ s ) / 2 |X i | for each i ∈ { 1 , 2 } yields

In this case we have

which is precisely the condition under which TESTDISTINCT ( t, X 1 , X 2 , δ ) returns True.

We are now ready to prove Theorem 6, which we restate below:

Theorem 6. Consider a dataset D of episodes sampled from an RDP R and a regular policy π b ∈ Π R . With probability 1 -δ , the output of ADACT-H ( D , δ/ (2 QAO )) is the transition function of the minimal RDP equivalent to R , provided that |D| ≥ N δ , where

d b min := min { d b t ( q, ao ) | t ∈ [ H ] , q ∈ Q t , ao ∈ AO , d b t ( q, ao ) > 0 } is the minimal occupancy distribution, and µ 0 is the L p ∞ -distinguishability.

Proof. The proof consists in choosing N and δ such that the condition in Lemma 14 is true with high probability for each application of TESTDISTINCT. Consider an iteration t ∈ [ H ] of ADACT-H. For a candidate state qao ∈ Q c ,t +1 , its associated probability is d b t ( q, ao ) with empirical estimate ̂ p t ( qao ) = |X ( qao ) | /N , i.e. the proportion of episodes in D that are consistent with qao . We can apply the empirical Bernstein inequality in (16) to show that

where M = |X ( qao ) | , ℓ = log(4 /δ ) , and δ is the failure probability of ADACT-H. To obtain a bound on M and N , assume that we can estimate d b t ( q, ao ) with accuracy d b t ( q, ao ) / 2 , which yields

Combining these two results, we obtain

Solving for M yields M ≥ 4 ℓ , which is subsumed by the bound on M in Lemma 14 since µ 0 < 1 . Hence the bound on M in Lemma 14 is sufficient to ensure that we estimate d b t ( q, ao ) with accuracy d b t ( q, ao ) / 2 . We can now insert the bound on M from Lemma 14 into (18) to obtain a bound on N :

To simplify the bound, we can choose any value larger than N 1 :

where we have used d b t ( q, ao ) ≥ d b min , µ 0 < 1 , ℓ = log 4 + log(1 /δ ) ≥ 1 , H log(2 ARO ) ≥ log 4 ≥ 1 and 4 √ 2 + 14 / 3 < 21 2 . Choosing δ = δ 0 / 2 QAO , a union bound implies that accurately estimating d b t ( q, ao ) for each candidate state qao and accurately estimating p ( e 0: u ∗ ) for each prefix in the multiset X ( qao ) associated with qao occurs with probability 1 -2 QAOδ = 1 -δ 0 , since there are at most QAO candidate states. Substituting the expression for δ in N 0 yields the bound in the theorem.

It remains to show that the resulting RDP is minimal. We show the result by induction. The base case is given by the set Q 0 , which is clearly minimal since it only contains the initial state q 0 . For t ∈ [ H ] , assume that the algorithm has learned a minimal RDP for sets Q 0 , . . . , Q t . Let Q t +1 be the set of states at layer t + 1 of a minimal RDP. Due to Proposition 1, each pair of histories that map to a state q t +1 ∈ Q t +1 generate the same probability distribution over suffixes. Hence by Lemma 13, with high probability TESTDISTINCT ( t, X ( qao ) , X ( q ′ a ′ o ′ ) , δ ) returns false for each pair of candidate states qao and q ′ a ′ o ′ that map to q t +1 . Consequently, the algorithm

merges qao and q ′ a ′ o ′ . On the other hand, by assumption, each pair of histories that map to different states of Q t +1 have L p ∞ -distinguishability µ 0 . Hence by Lemma 14, with high probability TESTDISTINCT ( t, X ( qao ) , X ( q ′ a ′ o ′ ) , δ ) returns true for each pair of candidate states qao and q ′ a ′ o ′ that map to different states in Q t +1 . Consequently, the algorithm does not merge qao and q ′ a ′ o ′ . It follows that with high probability, ADACT-H will generate exactly the set Q t +1 , which is that of a minimal RDP.

C.3 Proof of Theorem 8

In this section we prove Theorem 8, which states an alternative upper bound on the sample complexity of ADACT-H. The proof requires an alternative definition of the algorithm, which we call ADACTH-A (for 'approximation').

Theorem 8. Consider a dataset D of episodes sampled from an RDP R and a regular policy π b ∈ Π R . With probability 1 -δ , the output of ADACT-H -A, called with D , δ/ (2 QAO ) and ε ∈ (0 , H ] in input, is the transition function of an ε/ 2 -approximate RDP R ′ , provided that |D| ≥ N ′ δ , where

Function ADACT-H-A( D , δ , ε , Q , C )

Proof. ADACT-H-A returns the set of RDP states Q ′ and transition function τ ′ of an approximate RDP R ′ , taking as input the accuracy ε , an upper bound Q on |Q ′ | , and an upper bound C on the concentrability C ∗ R ′ of R ′ . If, at any moment, the number of RDP states |Q ′ | exceeds Q , the algorithm returns Failure (line 15). ADACT-H-A defines a sequence of side states q e 0 , . . . , q e H +1 (lines 2 and 4), and defines τ ′ ( q e t , ao ) = q e t +1 for each t ∈ [ H ] and ao ∈ AO (line 5). For each candidate state qao ∈ Q ′ c ,t +1 such that |X ( qao ) | /N ≥ ε/ (4 QAOHC ) , the definition of ADACT-H-A is the same as that of ADACT-H, including the call to TESTDISTINCT (lines 11-14). For each candidate state qao ∈ Q ′ c ,t +1 such that |X ( qao ) | /N < ε/ (4 QAOHC ) , instead of mapping ( q, ao ) to the correct RDP state, ADACT-H-A maps ( q, ao ) to the side state q e t +1 (lines 17-18). Once in q e t +1 , R ′ remains in a side state for the rest of the episode. The side states do not satisfy Proposition 1, since the

histories that map to side states may assign different probabilities to suffixes (and TESTDISTINCT is never called).

We define an alternative occupancy measure d ′ t ( q, ao ) associated with the approximate RDP R ′ and the behavior policy π b . The new definition is given by d ′ 0 ( q 0 , a 0 o 0 ) = θ o ( q 0 , a 0 , o 0 ) and

The only difference between d ′ t and d b t is that d ′ t is defined with respect to the transition function τ ′ of the approximate RDP R ′ , instead of the transition function τ associated with the original RDP R . Note that apart from the side states, R ′ will contain the same states as R , as long as the candidate states satisfy the condition on line 11, and τ ′ will be the same as τ on those states. Since the states and behavior policy are the same, the L p ∞ -distingishability µ 0 of R ′ will be the same as that of R .

First consider each candidate state qao ∈ Q ′ c ,t +1 such that |X ( qao ) | /N ≥ ε/ (4 QAOHC ) . In this case, ADACT-H-A calls TESTDISTINCT, so Lemmas 13 and 14 apply to these candidate states. The associated occupancy is d ′ t ( q, ao ) with empirical estimate ̂ p t ( qao ) = |X ( qao ) | /N . Hence the empirical Bernstein inequality applies to d ′ t ( q, ao ) and ̂ p t ( qao ) . Just as in the proof of Theorem 6, we choose X ( qao ) large enough to accurately estimate d ′ t ( q, ao ) within a factor d ′ t ( q, ao ) / 2 with probability 1 -δ . We thus obtain an alternative upper bound on d ′ t ( q, ao ) as follows:

From here, we can use the proof of Theorem 6 by substituting d ′ t for d b t , up until the definition of the bound N 1 on |D| in (21). Inserting the bound on d ′ t ( q, ao ) into the expression for N 1 yields

Next, consider each candidate state qao ∈ Q ′ c ,t +1 such that |X ( qao ) | /N < ε/ (4 QAOHC ) . In this case, we instead choose X ( qao ) large enough to estimate d ′ t ( q, ao ) with accuracy β with probability 1 -δ . From the empirical Bernstein inequality, estimating d ′ t ( q, ao ) with accuracy β implies

Choosing β = ε/ (4 QAOHC ) implies ̂ p t ( qao ) < β , and we can thus simplify N 3 as

In addition, this choice of β yields the following bound on d ′ t ( q, ao ) :

We prove that R ′ is an ε/ 2 -approximation of the original RDP R . We briefly overload notation by letting d ∗ t ( q, ao ) refer to the occupancy of an optimal policy π ∗ with respect to the transition function τ ′ of R ′ . Consider a candidate state qao ∈ Q ′ c ,t +1 such that |X ( qao ) | /N < ε/ (4 QAOHC ) . The contribution to the expected optimal reward of R of all histories that map to qao is bounded as

since ( H -t ) is the maximum reward obtained during the remaining time steps. Since qao is mapped to a side state of R ′ , an optimal policy for R ′ may not accurately estimate the expected optimal value for qao , but the contribution of all such candidate states to the expected optimal value is at most

since there can be at most QAO such candidate states. Hence any optimal policy for R ′ is an ε/ 2 -optimal policy for R , which implies that we can approximate an ε -optimal regular policy for the exact RDP R by finding an ε/ 2 -optimal policy for the approximate RDP R ′ .

It is easy to verify that the bound N 4 in (24) is less than the bound N 2 in (23). Hence a worst-case bound is obtained by assuming that |X ( qao ) | /N ≥ ε/ (4 QAOHC ) for each t ∈ [ H ] and each candidate state qao ∈ Q ′ c ,t +1 , which yields an upper bound N 2 . Note that ADACT-H-A takes as input an upper bound Q on the number of RDP states |Q ′ | of R ′ , as well as an upper bound C of the concentrability coefficient C ∗ R ′ . If the learning agent has no prior knowledge of Q and C , it could start with small estimates of Q and C , and in the case that ADACT-H-A returns Failure or the resulting policy has larger concentrability than C for R ′ , iteratively double the estimates Q and/or C and call the algorithm again. This only increases the computational complexity of ADACT-H-A by a factor O (log QC ∗ R ′ ) , and the resulting upper bounds Q and C do not exceed 2 Q and 2 C ∗ R ′ . Since we already have an estimate Q , in each iteration we can call ADACT-H-A with δ = δ 1 / (2 QAO ) to ensure that the bound N 2 holds for each candidate state simultaneously with probability 1 -δ 1 . Substituting this value of δ in the bound N 2 in (23) and using Q < 2 Q and C < 2 C ∗ R ′ yields the sample complexity bound stated in the theorem.

Lemma 15. The concentrability C ∗ R ′ of the approximate RDP R ′ satisfies

Proof. For each t > 0 , let d ′ t ( q e t ) be the occupancy of the side state q e t in the approximate RDP R ′ . We prove by induction on t that d ′ t ( q e t ) satisfies

The base case is given by t = 1 . In this case, a candidate state ( q 0 , ao ) is mapped to q e 1 if d b t ( q 0 , ao ) = d ′ t ( q 0 , ao ) < ε/ (2 QAOHC ) . Since there can be at most AO = |Q 0 | AO such candidate states, we have

For t > 1 , a candidate state ( q t -1 , ao ) is mapped to q e t if d ′ t ( q t -1 , ao ) < ε/ (2 QAOHC ) . Again, there can be at most | Q t -1 | AO such candidate states. Since all occupancy of q e t -1 is also mapped to q e t , we have

where we have used the induction hypothesis.

Consider a candidate state ( q, ao ) of R at time t . Due to approximation, some histories in ¯ τ -1 ( q ) are mapped to side states in R ′ instead of q , and we can therefore write d b t ( q, ao ) = d ′ t ( q, ao ) + ξ ≤ d ′ t ( q, ao ) + d ′ t ( q e t ) , where ξ is the total occupancy of histories in ¯ τ -1 ( q ) mapped to side states. In turn, this implies

The concentrability of a candidate state ( q, ao ) in the approximate RDP R ′ that is not mapped to a side state (i.e. d ′ t ( q, ao ) ≥ ε/ (6 QAOHC ) ) can now be bounded as

This concludes the proof of the lemma.

C.4 Distinguishability Parameters

As defined in the main text, for t ∈ [ H ] , we consider a metric L over distributions on the remaining part of the episode ∆( E ℓ ) , for ℓ = H -t . Then, the L -distinguishability of an RDP R and a policy π is the maximum µ 0 such that, for any t ∈ [ H ] and any two distinct q, q ′ ∈ Q t , the probability distributions over suffix traces e t : H ∈ E ℓ from the two states satisfy

So, µ 0 is a feature of the RDP and the policy combined and it quantifies the distance between any two distinct states of the RDP with respect to the distributions they induce over the observable quantities. Distinguishability parameters have been first introduced in Ron et al. [56] and later generalized for other metrics. They can be also found in Balle [59] for PDFA learning and in Ronca and De Giacomo [26], Ronca et al. [27] for RDP learning.

According to the definition we adopt, there exists an L -distinguishability for any RDP and policy. However, as stated in Assumption 2, we require µ 0 to be strictly positive. This does not constitute a restriction for the RDP, since it can be always minimized while preserving all conditional probabilities. Though it implies that, in any state, the behavior policy takes with positive probability all actions that are needed to observe episode suffixes that have different probability under the two states. Clearly if this was not the case for two distinct q, q ′ ∈ Q t at some t ∈ [ H ] , P ( e t : H | q t = q, π ) = P ( e t : H | q t = q ′ , π ) and no information would be available for the algorithm to distinguish q and q ′ . Assumption 2 is implied by Assumption 1. However, it becomes necessary for Theorem 8, since this does not rely on Assumption 1.

The metric selected also influences the actual value of the distinguishability parameter. In this paper, we adopt L p ∞ , as it can be seen from the TESTDISTINCT function in the two algorithms. A more standard distance would be L ∞ . According to Eq. (25), an L ∞ -distinguishability of µ 0 implies that for any t ∈ [ H ] and two distinct q, q ′ ∈ Q t ,

This means that some sequence until the end of the episode has a different probability of being generated from the two states. Although similar, the L p ∞ distance, maximizes for the full trace as well as any of its prefixes as

As it has been discussed in Balle [59], Appendix A.5, the prefix L p ∞ metric always upper bounds the L ∞ metric, up to a multiplicative factor, while there are pairs of distributions in which L ∞ is exponentially smaller than L p ∞ with respect to the expected suffix length. This motivates our choice. Moreover, in the specific case of our fixed horizon setting, we have that the L p ∞ -distinguishability is never lower than L ∞ -distinguishability. Note that in the hard instance of Appendix E.2, the two coincide.

The lower bound is stated in terms of the L p 1 -distinguishability of the RDP. While L p ∞ is achieved for one specific trace prefix maximizing the difference in probability, L p 1 takes all traces into account as ∑ u ∈ [ H -t ] ,e ∈E u | P ( e t : H = e ∗ | q t = q ) -P ( e t : H = e ∗ | q t = q ′ ) | . Due to this relation, the L p ∞ -distinguishability always lower bounds the L p 1 -distinguishability in the fixed horizon setting.

D RegORL with Subsampled VI-LCB

In this section we demonstrate the composition of our proposed algorithm with a specific Offline Reinforcement Learning algorithm for MDPs. Specifically, we adopt Subsampled VI-LCB , from Algorithm 3 of Li et al. [16] and report the combined sample complexity of this choice, through a simple application of Theorem 7.

First, we introduce the occupancy distribution and the single-policy conentrability coefficient for MDPs. Let M = ⟨Q , A , R , T, R, H ⟩ be an MDP with states Q , horizon H , transition function T : Q×A→Q and reward function R : Q×A→ ∆( R ) . The state-action occupancy distribution of a policy π : Q → ∆( A ) in M at step t ∈ [ H ] is d π m ,t ( q, a ) = P ( q t = q, a t = a | M , π ) . For our

purposes, it suffices to consider a fixed initial state q 0 . Finally, the MDP single-policy concentrability of a policy π b is [15]:

We can now express the sample complexity of Subsampled VI-LCB .

Theorem 16 (Li et al. [16]) . Let D be a dataset of N m episodes, sampled from an MDP M with a Markov policy π b . For any ε ∈ (0 , H ] and 0 < δ < 1 / 12 , with probability exceeding 1 -δ , the policy ̂ π returned by Subsampled VI-LCB obeys V ∗ 0 ( q 0 ) -V ̂ π 0 ( q 0 ) ≤ ε , as long as:

for a positive constant c .

The analysis in Li et al. [16] of Subsampled VI-LCB assumes that the reward function is deterministic and known. Thus, restricting our attention to this setting, we consider any episodic RDP with history-dependent, deterministic rewards. The reward function can be regarded as known, since it may be easily extracted from the dataset resulting from the Markov transformation of Definition 2.

Corollary 17. Let D be a dataset of N episodes, sampled with a regular policy π b ∈ Π R from an RDP R with deterministic rewards. If Subsampled VI-LCB is the OFFLINERL algorithm in Algorithm 1, then, for any ε ∈ (0 , H ] and 0 < δ < 1 / 12 , with probability exceeding 1 -δ , the output of RegORL ( D , ε, δ ) is an ε -optimal policy of R , as long as

Proof. This corollary follows as a direct application of Theorem 16 to Theorem 6. It only remains to verify that the single-policy concentrability of the MDP underlying the dataset D ′ that Subsampled VI-LCB receives is C ∗ R . The dataset D ′ is generated according to the Markov transformation ¯ τ from Definition 2. We only consider the cases in which ADACT-H succeeds. Let π ∈ Π R be any regular policy and q t , q ′ t the states reached at step t by R and M R , respectively. Then for t > 0 ,

This is valid for any regular policy, and for the optimal and behavior policies in particular. Then,

Similarly to the previous corollary, it is also possible to combine Theorem 16 with Theorem 8. In this case, the sample complexity of Subsampled VI-LCB for learning an ε/ 2 -accurate policy with probability 1 -δ/ 2 would be combined with N ′ δ/ 2 of Theorem 8.

E Sample Complexity Lower Bound: Proof of Theorem 9

In this section, we prove the sample complexity lower bound in Theorem 9. The proof is based on a suitable composition of a two-armed bandit and an instance of the learning problem associated to noisy parity functions. We first describe this latter problem and its sample-complexity lower bound in Appendix E.1. Then, we compose a hard class of RDP instances in Appendix E.2, and prove the final statement in Appendix E.3.

E.1 Learning parity with noise

Let B = { 0 , 1 } and L ∈ N . For any string x ∈ B L , the parity function f x : B L → B is f x ( y ) = ⊕ i ∈ [ L -1] x i y i , where ⊕ is addition modulo 2. For noise parameter ξ ∈ (0 , 0 . 5) , a noisy parity function f x,ξ returns f x ( y ) with probability 0 . 5 + ξ and 1 -f x ( y ) otherwise. Consider the class of parity functions F ( L ) = { f x } x ∈ B L and the class of noisy parity functions F ( L, ξ ) = { f x,ξ } x ∈ B L . Assume that x , y 1 , y 2 , . . . ∼ unif( B L ) are uniformly sampled. The success probability of a streaming algorithm A for F ( L, ξ ) is the probability that A recovers x , given in input a sequence of observations ( y i , f x,ξ ( y i )) i .

Lemma 18. Any streaming algorithm for F ( L, ξ ) with a success probability higher than O (2 -L ) requires at least Ω( L/ξ ) or 2 Ω( L ) input samples ( y i , f x,ξ ( y i )) i .

Proof. Learning in F ( L, ξ ) is the problem of recovering x ∈ 2 B from noisy data ( y i , b i ) , where b i = f x ( y i ) with probability 0 . 5 + ξ , and b i = 1 -f x ( y i ) otherwise. This is the problem of learning in F ( L ) with corruption rate 0 . 5 -ξ . Hence, we focus on the problem of learning noiseless parity first.

The Statistical Query dimension SQDIM ( C , d ) , characterizes the complexity of learning in the class C with respect to the prior distribution d ∈ ∆( C ) . As defined in [73], SQDIM ( C , d ) is the maximum n ∈ N such that there exist distinct f 1 , . . . , f n ∈ C , such that their pairwise correlations with respect to d are between -1 /n and 1 /n . For the class of parity functions, under the uniform distribution over x , SQDIM ( F ( L ) , unif) = 2 L . This was already observed in [74], for a slightly different notion of SQ dimension. However, to verify this, we can consider a natural ordering over binary strings in X , and represent the problem of learning F ( L ) as a matrix M = ( m ij ) ∈ { 1 , -1 } 2 L × 2 L , defined as m ij = ( -1) f x j ( y i ) = ( -1) y i · x j , where scalar product is modulo 2. We have that M is a Hadamard matrix. Then, since every row is orthogonal to the others, and the same is true for columns, every couple of parity functions are uncorrelated under the uniform distribution over x .

Regarding the noisy parity problem, since SQDIM ( F ( L ) , unif) = 2 L , we can apply Corollary 8 of [75] with m = 2 L , to have that the matrix M corresponding to the parity problem is a ( k, l ) -L 2 -extractor with error 2 -r , for k, l, r ∈ Ω( L ) . Since M is a suitable extractor, we can apply Theorem 1 of [76], which considers the problem of learning M with the additional noise parameter ξ . We obtain that, in the streaming setting, any branching program B for F ( L, ξ ) whose depth is at most 2 f 1 ( k,l,r ) and width is at most 2 ckl/ξ has a success probability of at most O (2 -f 1 ( k,l,r ) ) , where c is a suitable constant and f 1 is equation (1) of [76].

Then, if the success probability is not in O (2 -f 1 ( k,l,r ) ) , meaning it is higher, we have that the depth of A exceeds 2 f 1 ( k,l,r ) or the width of A exceeds 2 ckl/ξ . Since k, l, r ∈ Ω( L ) , then, if the success probability is not in O (2 -L ) , the depth of A is 2 Ω( L ) or the width of A is 2 Ω( L 2 ) /ξ . Width and depth refer to the computational model that represents A as a branching program. A branching program is a directed acyclic graph in which internal nodes have one outgoing edge for each possible input sample, that is | B L × B | = 2 L +1 in our problem, and leaves correspond to algorithm decisions. From the required width and depth we know that A has a leaf in layer 2 Ω( L ) or in a layer that contains 2 Ω( L 2 ) /ξ nodes. The former case implies a worst case sample complexity requirement that is exponential in L . For the latter, we observe that in order to reach that width, at least log 2 L +1 2 Ω( L 2 /ξ ) transitions and input samples, are required. This is Ω( L/ξ ) .

q

01

∗

.u

q

02

∗

.u

q

03

a

′

0

.v

ξ

a

′

1

.v

-

ξ

q

s

∗

.s

q

11

∗

.u

q

12

∗

.u

q

13

a

′

0

.v

-

ξ

a

′

1

.v

ξ

q

b

1

∗

.

+

q

b

2

∗

.

+

q

b

3

a

′

0

.v

0

a

′

1

.v

η

q

+4

q

+5

q

-

4

q

-

5

q

o

⊥

0

1

∗

∗

0

1

0

1

-

+

+

∗

∗

+

-

∗

∗

∗

∗

∗

✶

✶

E.2 Class of hard RDP instances

For our main lower bound, we define a class of hard RDP instances. Figure 3 shows one possible instance in this class. We will soon define it formally, but we can observe that its structure is organized in two main paths. The two branches in the top part encode a parity computation according to some hidden code x ∈ B L , so that behaving optimally in that region requires to solve a parity problem (the one of Lemma 18). The bottom part reaches a two-armed bandit whose optimal action is c . The right-most states are winning or losing states that provide a positive and null reward accordingly.

Formally, we define a class of hard RDP instances as R ( L, H, ξ, η ) = { R x,c } x ∈ B L ,c ∈{ 0 , 1 } where R x,c = ⟨Q , AO , Ω , τ, θ, q s , H ⟩ , for Q = { q s , q o ⊥ }∪{ q 0 i , q 1 i , q bi } i =1 ,...,L ∪{ q + ,i , q -,i } i = L +1 ,...,H , A = { a ′ 0 , a ′ 1 } , O = { 0 , 1 , + , -} . Assume L ≥ 1 and H > L . Rewards are zero everywhere, except in the winning states

θ r ( q, a, r ) = ✶ 1 if q = q + i with i > L , ✶ 0 , otherwise (39) where we recall that ✶ x represents the deterministic distribution on x . For observation probabilities, we denote the distributions u ( o ) := unif { 0 , 1 } and

Now define observations as

✶ Finally, the transition function is defined such that ¯ τ ( q s , h L -1 ) = q iL with i = f x ( o 0: L -1 ) , and

In addition, the trnasitions τ ( q 0 i , ao ) and τ ( q 1 i , ao ) , for i < L , are defined according to o ∈ { 0 , 1 } and the parity code x . Namely, τ ( q 0 i , ao ) equals q 1 ,i +1 iff o ⊕ x ( i ) = 1 , and q 0 ,i +1 , otherwise. τ ( q 1 i , ao ) is defined analogously.

E.3 Proof of Theorem 9

Theorem 9. For any ( C ∗ R , H, ε, µ 0 ) satisfying C ∗ R ≥ 2 , H ≥ 2 and ε ≤ Hµ 0 / 64 , there exists an RDP with horizon H , L p 1 -distinguishability µ 0 and a regular behavior policy π b with RDP single-policy concentrability C ∗ R , such that if D has been generated using π b and R , and

then, for any algorithm A : D ↦→ ̂ π returning non-Markov deterministic policies, the probability that ̂ π is not ε -optimal is at least 1 / 4 .

Proof. Denote with π b a regular policy in R and D ∈ D a dataset of episodes of length H , collected from R and the behavior policy π b . For an RDP R , let Π d = A H be the set of deterministic nonMarkov policies and A = D → Π d an offline RL algorithm. For δ < 0 . 5 , we say that an algorithm A is ( ε, δ ) -PAC for the class of RDPs R under φ , if, for every R ∈ R and D ∈ D , if the condition φ ( D , π b ) is verified, then the output policy A ( D ) is ε -optimal in R , with probability 1 -δ . One notable case is that of φ requiring a minimum dataset size.

Since the output of a generic algorithm might be any generic non-Markov deterministic policy, we cannot restrict our attention to regular policies. We expand the value of a policy π : H → A in a RDP R x,c ∈ R ( L, H, ξ, η ) as follows:

where in Eq. (52) we have used the uniform probability over x . Now, for any history-dependent deterministic policy π in episodic RDPs, it is possible to identify an associated regular stochastic policy π r : Q ′ → ∆( A ) , where Q ′ := Q\{ q o ⊥ } and:

∑

∈

h

′

¯

τ

-

1

(

q

)

=

(

q

h

|

′

π

|

)

In other words, π r encodes the probability that π takes action a , given that some history has led to state q . With this convention, we resume from Eq. (53)

For the optimal policy in particular, this becomes:

From the ε -optimality of π = A ( D ) , then,

Now, assume that

P

(

h

P

π

)

=

I

(

π

(

h

′

) =

a

)

(55)

Then, all of the following is true: π r ( q 0 L , a ′ 0 ) ≥ 3 / 4 , π r ( q 1 L , a ′ 1 ) ≥ 3 / 4 , π r ( q bL , c ) ≥ 3 / 4 . This means that, for small ε , any ε -optimal policy must frequently select the optimal action for both the parity problem and the bandit. Let us represent the first two events with B p and the third with B b . Since A is ( ε, δ ) -PAC for R ( L, H, ξ, η ) under φ , the probability of B p ∧ B b is at least 1 -δ , for any D and π b satisfying φ ( D , π b ) .

We proceed to compute the necessary data to satisfy both events with high probability. The dataset D can be partitioned in two subsets D p and D b , containing any episode from D whose initial observation is { 0 , 1 } and +, respectively. The two datasets share no information and D p and D b are mutually independent. To see this, we observe that the sequence a L +1 r L +1 o L +1 . . . o H is independent of

a 0 r 0 o 0 . . . a L given o L , since + or -determines at step L determines the rest of the episode. Also, for any two episodes e H , e ′ H , the sequence a 1 r 1 o 1 . . . o L is independent of a ′ 1 r ′ 1 o ′ 1 . . . o ′ L given o 0 . Since, o 0 ∼ s , that is the starting distribution, the two datasets are independent. Let Q p = { q s , q o ⊥ } ∪ { q 0 i , q 1 i } i =1 ,...,L ∪{ q + ,i , q -,i } i = L +1 ,...,H and Q b = { q s , q o ⊥ } ∪ { q bi } i =1 ,...,L ∪ { q + ,i , q -,i } i = L +1 ,...,H be the reachable states in the two datasets. Then, we consider two separate classes R ( L, H, ξ ) and R ( L, H, η ) as the sets of RDPs in R ( L, H, ξ, η ) , restricted to Q p and Q b , respectively. To do so we construct R r ∈ R ( L, H, ξ ) and R c ∈ R ( L, H, η ) such that the initial observation follows unif( { 0 , 1 } ) in R r and ✶ + in R c . Now, from the independence of the two datasets and the fact that A is ( ε, δ ) -PAC in D , there must exists an algorithm A p : D p ↦→ π p that is (2 ε, δ ) -PAC in R ( L, H, ξ ) under some φ p , and A b : D b ↦→ π b that is (2 ε, δ ) -PAC in R ( L, H, η ) under some φ b . If this was not the case, B p ∧ B b could not be verified in one of the two terms.

We analyze A p first and we show that its requirement φ p is |D p | ∈ Ω( L/ξ ) ∪ 2 Ω( L ) . For a contradiction, assume this is not the case and that |D p | = g ( L, ξ ) ̸∈ Ω( L/ξ ) ∪ 2 Ω( L ) is allowed. Then, we can use A p to solve the noisy parity problem under the streaming setting with g ( L, ξ ) samples (this setting has been introduced in Appendix E.1). We proceed as follows. Consider any noisy parity function f x,ξ with unknown x . Sample a sequence of strings { y i } i ∈ 2 L from the uniform distribution and collect g ( L, ξ ) pairs ( y i , p i ) , sampling p i ∼ f x,ξ ( y i ) . Then, for H > L , compose a dataset of episodes { e i } i . All actions of e i are selected uniformly in { a ′ 0 , a ′ 1 } . The observations o 0: L -1 are y i and o L equals p i if a L = a ′ 0 , 1 -p i , otherwise (0 and 1 take roles of + and -symbols here). Rewards r L +1: H are equal to one if o L = 1 , null otherwise. We obtain that dataset so constructed is equally likely under this procedure than under the uniform policy and the RDP R x ∈ R ( L, H, ξ ) . Since A p is (2 ε, δ ) -PAC for R ( L, H, ξ ) , with probability 1 -δ , the output policy π p satisfies:

where π pr is the stochastic regular policy for π p . This can be seen by our assumption in Eq. (64) and doubling both ε and the sub-optimality gap of Eq. (63), due to the updated probability for the initial observation. Then, for any sequence y ∈ 2 L and associated history h L -1 with o 0: L -1 = y ,

which is the noiseless parity function based on x . This means that it is possible to reconstruct x solely by interacting with π p , without collecting further samples. The solution we have described is a streaming algorithm with sample complexity g ( L, ξ ) . Since this contradicts Lemma 18, we have proven |D p | ∈ Ω( L/ξ ) ∪ 2 Ω( L ) .

We now consider the bandit problem, which is solved by A b . Similarly to the previous case, from the ε -optimality of A b ( D b ) , we obtain the necessary condition: π br ( q bL , c ) ≥ 3 / 4 from Eq. (63). This condition is expressed for the stochastic policy π br , However, we notice that for q bL in particular, the only possible history is h L -1 = a 0 + a 1 . . . + , where all actions must also be deterministic. Then,

implying that π br can only be deterministic for q bL . This means that A b must solve best-arm identification in the two arm bandit at q bL . We can compose a simplified dataset that is relevant for the bandit as:

̸

Since D b can be deterministically reconstructed from D ′ b , we have the following conditional independence: π b ⊥ c | D ′ b , where c ∈ { a ′ 0 , a ′ 1 } is the optimal arm, and π b = A b ( D b ) is the output of the algorithm. Denoting with ˆ c = π b ( h L -1 ) the selected arm, the error probability is P e := P (ˆ c = c ) . Applying Fano's inequality from Theorem 12 to the variables c →D ′ b → ˆ c gives:

where we have used the fact that ˆ c is a Bernoulli variable and the uniform prior over c . Now, assuming C ≥ 2 , we construct the following behavior policy: π b ( q bL , a 0 ) = 1 -1 /C and π b ( q bL , a 1 ) = 1 /C . In the following, we write N b := |D b | and omit the implicit dependency on π b .

Then from Eq. (70), and the fact that A b is (2 ε, δ ) -PAC,

Which means that this must be φ b , the requirement for A b .

Finally, to compose the results from both branches, we observe that |D| = |D p | + |D b | . Also, for any δ ∈ (0 , 0 . 5) , say 1 / 4 , (log 2 -H ( δ )) becomes a positive constant, and we can add both sizes asymptotically:

To relate the parameters to features of the RDP, we observe that the number of states of any RDP in R ( L, H, ξ, η ) is Q ≤ 3 H . Also, the behavior policy is uniform everywhere except in q bL . Assuming C ≥ 2 , the computation of the single-policy concentrability coefficient yields C ∗ R = C , for any c ∈ { a ′ 0 , a ′ 1 } . Next, we compute the L p 1 -distinguishability of any RDP in this class. The L p 1 -distinguishability of a set of states Q is the minimum L 1 distance in distribution between episodes prefixes that are generated starting from any two states in Q . Let us consider the L 1 norm for the pair q 01 and q 11 ,

The L 1 distance of suffixes from q 01 , q 11 that are shorter than L have also a distance of 2 ξ . On the other hand, any shorter prefix has a distance of 0. Since, L p 1 sums all these distances, the minimum across the q 0 i , q 1 i pairs, is attained for q 0 L and q 1 L , which determines µ 0 ≥ 2 ξ . Also, the distance between any other pair of states in the same layer is strictly higher, since they differ deterministically in some reward or observation. Hence, the L p 1 -distinguishability of the entire RDP is µ 0 = 2 ξ . Now, we choose L = H/ 2 , η = 32 ε/H and we assume ε ≤ Hµ 0 / 64 , H ≥ 2 . We can verify that these choices are consistent with the previous assumption min { ξ, η } ≥ 16 ε H -L . Substituting, the final requirement φ for the complete algorithm A is an exponential number of episodes in H or:

Now, for any H,µ 0 , C ∗ R , ε satisfying the previous assumptions, any algorithm cannot be ( ε, 1 / 4) -optimal for the instances in R ( H/ 2 , H, µ 0 , 32 ε/H ) if Eq. (96) is not satisfied.

Note that in our RDP instance, the number of states and the horizon length scale linearly. So, we might equivalently write HO instead of H 2 .

Peer reviews:
comments: SUMMARY:
The paper presents an offline RL algorithm to learn near-optimal policies in a (episodic) Regular Decision Process (RDP). The problem is to learn a policy given a data set. The algorithm is split into two parts. First, is to learn the transition function of the RDP. Then, the problem of off-line learning on RDP is reduced to offline learning in MDP (intuitively, the underlying automata in the RDP is extracted) to generate a near-optimal policy in the RDP. The upper bound provided in this setting improves upon an upper bound of a similar algorithm in the non-episodic variant and a lower bound is also provided.

STRENGTHS:
Good problem. Well-written paper. Contributions and algorithms are well-stated and well-contextualized w.r.t. related work.

WEAKNESSES:
NA.

QUESTIONS:
What may be some challenges in adopting these algorithms in practice?

LIMITATIONS:
NA.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
The authors presents a novel algorithm for offline RL in episodic Regular Decision Processes (RDP), a computationally feasible subclass of Non-Markovian Decision Processes. The presented algorithm have two phases: 1) learning the underlying automata by a novel algorithm, and 2) Markov transformation of the rest of the dataset and use any state-of-the-art algorithm for offline RL in usual MDPs. To obtain a theoretical guarantees, authors presents a notion of concentrability coefficient in the RDP setup. Finally, the lower bounds on sample complexity were presented.

STRENGTHS:
- The first algorithm for offline RL in a challenging setup of Regular Decision Processes with provable guarantees.
- A novel algorithm for automata learning that have a much tighter sample complexity than in the previous work.
- The lower bounds shows that the introduced concentrability coefficient is indeed makes sense from the point of view of learning in RDB.

WEAKNESSES:
- Lack of empirical validation. The empirical experiments even on toy examples should much increase the value of the presented paper.

QUESTIONS:
- What are the limitation to generalize the presented techniques beyond the tabular setting?
- Is it possible to propose direct algorithm that will not depend on reduction to usual MDPs?

LIMITATIONS:
This is a theoretical paper that does not need to address the potential societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents RegORL, an algorithm for offline RL in episodic RDPs. The algorithm combines automata learning techniques with state-of-the-art offline RL algorithms for MDPs. The authors provide a non-asymptotic high-probability sample complexity bound for RegORL, which guarantees the learning of an $\\epsilon$-optimal policy. They also establish a sample complexity lower bound for offline RL in RDPs.\n\nSTRENGTHS:\n1. The paper comprehensively and rigorously explores algorithms for offline RDPs while leveraging automata learning techniques. This contribution carries great significance within the realm of offline RL theory.\n\n2. The structure of this paper is well-organized, resulting in a smooth and coherent reading experience. The comprehensive summary of relevant literature further enhances its overall completeness.\n\nWEAKNESSES:\n1. It would be beneficial to present a comparative analysis of RDP algorithms in tabular form, facilitating a clearer understanding of the different studies in this area.\n\n2.  In what ways does the offline setting present additional difficulties beyond the combination of commonly used techniques from offline RL and the technique used in online RDP research [1]?\n\n\n\n\n\n[1] Alessandro Ronca and Giuseppe De Giacomo. Efficient PAC reinforcement learning in regular decision\n418 processes. In IJCAI, pages 2026–2032, 2021.\n\nQUESTIONS:\n1. Does the claim made by the authors in line 98, stating that computing a near-optimal policy is challenging even for a known POMDP, imply that this difficulty does not exist in the case of RDPs?\n\n2. Can the techniques proposed in this paper be applied to settings where the state or observation space is infinite, such as RDPs with a linear structure?\n\nLIMITATIONS:\nThe suggestions have been claimed in \"Weaknesses\" and \"Questions\".", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of offline reinforcement learning (ORL) of episodic regular decision processes (ERDP) where we wish to find an near-optimal policy for an unknown ERDP with a small dataset of trajectories (collected with a behavioral policy). The authors took a reductive approach to first find the states and transitions of the minimal automata (by a proposed algorithm ADACT-H) underlying the ERDP and then to transform the original dataset into an MDP (using the estimated automata states as MDP states) and lastly to apply any existing ORL algorithm to the resulting MDP. The theoretical results are substantiated with theorems, necessary definitions and assumptions.\n\nSTRENGTHS:\n1. The exposition is easy to follow and the overall logic is coherent and sound.\n1. The presented results support claims of contributions and seem novel.\n1. The problem studied is interesting and I think that the reductive approach in this work involving recent results from automata theory (ADACT which learns a minimal Moore machine) may inspire other RL studies.\n\nWEAKNESSES:\n1. Perhaps the result of a conscious choice of tradeoff in the presentation, it seldomly highlights the key theoretical challenges and the authors' insights in the main text of the paper. Maybe the authors could reclaim some space for this purpose from example 1? (It is not referenced in the main text.)\n1. In a similar vein, I wish that there is more technical discussion on (promising) un/under-explored alternative approaches and implications for related problem settings in closing. This should help contextualize the contributions of the work for a broader audience.\n\nQUESTIONS:\n1. The trajectory length is fixed ($H$) the episodic setting, which implies a fixed length language (unlike general automata which can accept variable length sentences). Furthermore, the automata state space is stratified over tilmestep--making the DFAs acyclic--, I wonder if this makes the full generalization of DFAs a good fit for the episodic setting. \n1. Related to above, could you comment on the relevance of your approach to non-episodic setting? The DFA modeling of non-episodic environments seems more natural.\n1. Is there some interesting interaction between the learned DFA and subsequent ORL? For example, does it matter if the learned DFA is slightly larger than minimal? Is there some kind of tradeoff worth sharing or further study? What are some looseness in the reduction approach you took? \n1. Assumption 1, which asks for all observation to be present in the dataset, seems very restrictive (perhaps impossible for many ERDP). Since the ORL step is over $\\mathcal{Q}\\times\\mathcal{A}$, is it possible to relax the assumption?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the sample complexity of offline reinforcement learning (RL) in environments with non-Markov observation, i.e., partially observable Markov decision processes (POMDPs). In specific, the paper considers the episodic regular decision process (RDP) where the spaces of state, action, and observation are all finite. Given a batch of offline data, the proposed (offline RL) algorithm learns the ''states'' from the history data and converts the problem as offline RL for MDPs. The paper provides sample complexity bound for learning the states, as well as the optimal policy under certain assumptions. The paper also provides lower bound to prove the optimality of the upper bounds.\n\nOverall, the reviewer feels that the proposed algorithm for POMDPs brings new insight to the RL community and the theoretical results are strong enough to be accepted, even though they are built on strong assumptions, for example the distinguishability condition. As claimed by the authors, this is the first RL complexity bound of the setting and approach. Please see more details in the strengths and weakness parts.\n\nSTRENGTHS:\nThe episodic regular decision process is conceptually a very large set of models that can be used to model many realistic environments. The 'state' the proposed algorithm trying to learn is essentially a sufficient statistics of the history for inference of future observations (folloiwing the behavior policy), which is very similar to the predictive stats representation (PSR) but slightly different in key assumptions (and leading to different learning approach). If my understanding is correct, every episodic finite-state POMDP is also an episodic RDP since we can always define the concatenation of the entire history or the belief, i.e., conditional distribution, of the 'hidden state' as the 'state' of the RDP.\n\nWEAKNESSES:\n1. The methodology that we first learn the state and then apply RL algorithm to the transformed data is quite a quite a straightforward approach. And it seems that Assumption 2 is the critical reason that we can eventually obtain the polynomial sample upper bound $\\sqrt{H}log(QAO)/(d\\mu)$ following such a straightforward approach. My biggest concern will be the scale of $\\mu_0$ which is related to the distinguishability. In particular, $\\mu_0$ represents the $L_\\infty$ distance between distributions over $e_{t:H}$, which will be $(OA)^{H-t}$-dimensional vectors with bounded $L_1$ norm. Roughly speaking, it feels that the average $L_\\infty$ distribution will be $(OA)^{-(H-t)}$. Then, unless the maximal distance $\\mu_0$ has a jump in the orders due to some special structure of the RDP, we seem to have exponential dependency on $H$ in the sample complexity. Perhaps more discussion of the scale of $\\mu_0$ with examples would be necessary.\n2. Assumption 1 is also a very strong assumption and greatly reduces the difficulty of learning the states and optimal policy. I am wondering if we can still obtain the current strong results by adding pessimism (e.g., as in [18]) into the current algorithm and removing Assumption 1.\n3. More technical discussion about the (reason for) improvement compared with the previous results [10, 27] in the main paper would be helpful for readers to understand the contribution of the current paper better.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}