An Analysis of Abstracted Model-Based Reinforcement Learning
Year: 2021

Abstract:
Many methods for Model-based Reinforcement Learning (MBRL) provide guaran1 tees for the accuracy of the Markov decision process (MDP) model they can deliver. 2 At the same time, state abstraction techniques allow for a reduction of the size of 3 an MDP while maintaining a bounded loss with respect to the original problem. 4 It may come as a surprise, therefore, that no such guarantees are available when 5 combining both techniques, i.e., where MBRL merely observes abstract states. Our 6 theoretical analysis shows that abstraction can introduce a dependence between 7 samples collected online (i.e., in the real world), which invalidates most results 8 for MBRLs in this setting. Collecting samples using a simulator can avoid this 9 problem. We conclude that we should be careful when applying MBRL methods 10 to abstracted real-world data. 11

Many methods for Model-based Reinforcement Learning (MBRL) provide guaran-1 tees for the accuracy of the Markov decision process (MDP) model they can deliver.2 At the same time, state abstraction techniques allow for a reduction of the size of3 an MDP while maintaining a bounded loss with respect to the original problem.4 It may come as a surprise, therefore, that no such guarantees are available when5 combining both techniques, i.e., where MBRL merely observes abstract states. Our6 theoretical analysis shows that abstraction can introduce a dependence between7 samples collected online (i.e., in the real world), which invalidates most results8 for MBRLs in this setting. Collecting samples using a simulator can avoid this9 problem. We conclude that we should be careful when applying MBRL methods10 to abstracted real-world data.11
1 Introduction12
When trying to find good solutions to MDPs using Reinforcement Learning (RL) a fundamental13 problem is the exploration-exploitation dilemma: when to take actions to obtain more information,14 and when to take actions that maximize reward based on the current knowledge. Tabular MBRL15 methods have found good ways to deal with this dilemma [7, 28, 14].16
However, MDPs can be very large, which can be problematic for these methods. One way to deal17 with this is to reduce the size of the MDP. State abstractions are one way to do this [17, 1]. We18 are interested in approximate state abstractions since they allow for greater reductions of the MDP,19 though there is a trade-off with solution quality [1]. Specifically, we assume we have an approximate20 model similarity abstraction function φ [1] that maps states to abstract states. The environment21 returns states s, but the agent receives φ(s), see Figure 1. This setting, which was considered before22 [22, 2], is what we call Abstracted RL, and is the topic of this paper.23
Abstracted RL corresponds to RL in a Partially24 Observable MDP (POMDP), as previously de-25 scribed [5]. It is well known that policies for26 POMDPs that only base their action on the last27 observation φ(s) could be arbitrarily bad [26].28 However, when we assume that φ is an approx-29 imate model similarity abstraction [1] this worst30 case may not apply: Based on the observed ab-31
stract states the agent learns an (empirical) abstract model. If we could show that this learned model32 is close to an ‘abstract MDP’ (details in Section 2.2), we could give finite-sample guarantees on the33 performance in the original MDP by combining results from MBRL and abstraction.34
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
However, in MBRL, to guarantee (with high probability) that the learned model is close to the actual35 environment model, it is typical (e.g., [28, 14]) to use concentration inequalities such as Theorem36 2 from Weissman et al. [30]. But this theorem relies on independent and identically distributed37 (i.i.d.) samples for each state-action pair. In this paper, we analyze online collection of such samples38 in Abstracted RL and show that they are not independent1, which means that most guarantees for39 existing MBRL methods do not hold in the online Abstracted RL setting.240
On the positive side, when we have access to a simulator, we show how this can be used to collect the41 data such that the typical MBRL guarantees hold and we can learn an accurate model. We discuss42 that emulating this in the real world is possible, but extremely sample inefficient, thus highlighting43 the difficulty of assuming that we would have access to an i.i.d. dataset, as in some earlier works.44
2 Preliminaries45
We assume the environment the agent is acting in can be represented by an infinite horizon MDP46 M := 〈S,A, T,R, γ〉. Where S is a finite set of states s ∈ S, A a finite set of actions a ∈ A, T47 a transition function T (s′|s, a) = Pr(s′|s, a), R a reward function R(s, a) which gives the reward48 received when the agent executes action a in state s, and γ the discount factor (0 ≤ γ < 1).49 In RL the goal of the agent is to find an optimal policy π∗ : S → A which maximizes the expectation50 of the discounted cumulative reward. V π(s) denotes the expected value of the discounted cumulative51 reward under policy π starting from state s. Similarly, Qπ(s, a) denotes the expected value of the52 discounted cumulative reward when first taking action a from state s and then following policy π.53
2.1 Model-Based RL54
MBRL methods learn a model from the experiences, these are obtained by the agent acting in the55 MDP. The learned model is usually the empirical model, directly based on the experience the agent56 obtains [7, 28, 14]. Per state-action pair the agent stores the next-states reached after taking action a57 from state s in sequence Ys,a: Ys,a : {s′(1), s′(2), · · · , s′(m)}. We use Y to refer to the collection of58 all Ys,a. From this the empirical, or learned, model TY is constructed, that just counts how often we59 have seen the transition to a next-state, and normalizes this:60
∀s′∈S TY (s′|s, a) , 1
m m∑ i=1 1{Y (i)s,a = s′}, (1)
where 1{·} denotes the indicator function of the specified event, i.e., it is 1 if Y (i)s,a = s′ and 061 otherwise.62
To give finite-sample guarantees on the accuracy of the estimate TY , 3 concentration bounds such as63 Theorem 2.1 from Weissman et al. [30] are often used, e.g. in Strehl and Littman [28], Jaksch et al.64 [14]. However, these typically make use of the fact that samples are i.i.d. In most MBRL settings this65 is not a problem under some assumptions, e.g. when the MDP is communicating [25]. In this case66 due to the Markov property the obtained samples are i.i.d.67
In general, of course the hope is that with enough samples the learned model TY becomes accurate.68 With accurate we mean that the distance between TY (·|s, a) and T (·|s, a) will be small, where the69 distance is measured using the L1 norm, defined as:70
||TY (·|s, a)− T (·|s, a)||1 , ∑ s′∈S |TY (s′|s, a)− T (s′|s, a)|. (2)
Part of theorem 2.1 from Weissman et al. [30], slightly reworded, then gives a guarantee of accuracy:71 1We also show that samples are not identically distributed, but demonstrate that that problem would be
resolvable. 2The reader might be puzzled by this statement, since certain guarantees on the combination of abstraction and RL are known. This can be explained by the generality of Abstracted RL: in this setting there is a nonstationarity caused by the clustering of states with different dynamics. There is a lot of related work in other abstraction settings (e.g., state aggregation) where this complication does not occur due to the particularities of their setting [24, 11, 19, 20, 23, 10]. In section 4 we give details to back up our claim for individual papers.
3This is a crucial element in being able to guarantee good performance, where performance can be measured in different ways, e.g. in PAC-MDP terms [28] or in terms of regret [14]. We focus on the model quality.
Lemma 1 (L1 inequality [30]). Let Y s,a = Y (1), Y (2), · · · , Y (m) be i.i.d. random variables72 distributed according to T (·|s, a). Then, for all > 0,73
Pr(||TY (·|s, a)− T (·|s, a)||1 ≥ ) ≤ (2|S| − 2)e− 1 2m 2 . (3)
In this way, MBRL can upper bound the probability that the learned model, based on m samples, for74 a state-action pair (s, a) TY (·|s, a) will be far away (≥ ) from the true model T (·|s, a).75 The situation is more subtle if the MDP is not communicating, i.e., if there exists s1, s2 ∈ S for which76 there is no deterministic policy that eventually leads from s1 to s2. This can create a dependence77 between the samples [28]. Intuitively this happens because, if we look at one particular state-action78 pair (s, a), there might be a transition to state s′ such that the probability to return to s is 0. Thus if79 we would have n outcomes of (s, a) we would immediately know that at least n− 1 outcomes were80 not state s′. Since as soon as we observe s′, we know the agent would not be able to return to state s.81 Strehl and Littman [28] show that in this setting it is still possible to use Lemma 3 as an upper bound.82
2.2 State abstraction for given models83
In the planning setting, where the model is known a priori, a state abstraction can be formulated as84 a grouping or mapping from ground states to abstract states [18]. This is done with an abstraction85 function φ, a surjective function that maps from ground states, s ∈ S, to abstract states s̄ ∈ S̄:86 φ(s) : S → S̄. Here S̄ is defined as S̄ = {φ(s)|s ∈ S}.We use the ¯ notation to refer to the abstract87 space. We slightly overload notation and let s̄ both denote an abstract state as well as the set of88 ground states that map to the abstract state s̄, i.e., s̄ = {g ∈ S | φ(g) = s̄}, if s̄ ∈ S̄. The use should89 be clear from the context. We define the probability to transition to an abstract state Pr(s̄′|s, a) as90 follows:91
Pr(s̄′|s, a) , ∑ s′∈s̄′ T (s′|s, a). (4)
This is a very general form of state abstraction, that clusters together states with different dynamics92 into abstract states. Note that we do assume that the given state abstraction deterministically maps93 states to an abstract state. This in contrast to some related work on problems with block structure94 [10], where a Markov state can lead to multiple observations (abstract states in our terminology) that95 need to be aggregated appropriately to result in a small MDP [4, 10].96
Approximate model similarity abstraction Many different abstraction criteria exist [17], here we97 focus on approximate model similarity abstraction [1]. In this abstraction two states can map to the98 same abstract state if their behavior is similar, i.e., when the reward function and the transitions to99 abstract states are close. Approximate model-similarity is defined as follows:100
Definition 1. An approximate model-similarity abstraction, φmodel,η , for fixed η, satisfies:101
φmodel,η(s1) = φmodel,η(s2) =⇒ ∀a |R(s1, a)−R(s2, a)| ≤ η, ∀s̄′∈S̄,a |Pr(s̄′|s1, a)− Pr(s̄′|s2, a)| ≤ η. (5)
From now on we will just refer to φmodel,η as φ.102
We note that the abstraction we consider, approximate model-similarity abstraction, is still quite103 generic. It can cluster together states that have different transition and reward functions. However, in104 the online Abstracted RL setting, the differences in dynamics can cause a dependence between the105 samples, as we will show in detail in section 3. E.g. looking at (s̄, a), the probability that we reach a106 state s′ depends both on the probability that we reach a particular state s ∈ s̄ and then state s′ from s.107 Returning to abstraction of a given model, it is possible to construct an abstract MDP M̄ω from108 the model of an MDP M and an abstraction function φ, where ω is an action-specific4 weighting109 function, defined as follows:110
4The action-specific weighting function is a more general weighting function than is typically used, e.g. by Li et al. [18], which is not action-specific, i.e., it only depends on the state s. More formally it is the case where ∀a,a′∈A, s∈S ω(s, a) = ω(s, a′).
Definition 2. We refer to the weight associated with a ground state, s ∈ S, and action, a ∈ A, by111 ω(s, a). We have: ∀s∈S, a∈A 0 ≤ ω(s, a) ≤ 1 and ∑ s′∈φ(s) ω(s ′, a) = 1.112
The weighting function can be used to create abstract transition and reward functions, which are a113 weighted average of the ground function. In this way, from M , φ and any ω we can construct an114 abstract MDP M̄ω:115
Definition 3. Given an MDP M , φ, and ω, M̄ω = 〈S̄, A, T̄ω, R̄ω, γ〉 is constructed as:116
S̄ = {φ(s) | s ∈ S}, A = A, γ = γ, (6) ∀s̄∈S̄, a∈A R̄ω(s̄, a) = ∑ s∈s̄ ω(s, a)R(s, a), (7)
∀s̄,s̄′∈S̄, a∈A T̄ω(s̄′|s̄, a) = ∑ s∈s̄ ∑ s′∈s̄′ ω(s, a)T (s′|s, a). (8)
An abstract MDP M̄ω is just an MDP . This means we can use any planning method we like to find117 an optimal policy π̄∗ for M̄ω .118
What we are interested in is the performance of a policy on the abstract space, when applied on the119 original problemM . Any policy on the abstract space π̄ can be used inM as follows π̄(s) := π̄(φ(s)),120 leading to V π̄ ∗ . It has been shown that we can upper bound the loss in performance due to using an121 optimal policy for M̄ω , π̄∗ in M instead of using the optimal solution for M [8, 1, 29]:122
Lemma 2 (Lemma 4 [29]). An approximate model similarity abstraction (Definition 1), has sub-123 optimality bounded in η: ∀s∈S V ∗(s)− V π̄ ∗ (s) ≤ 2η+2γ(|S̄|−1)η(1−γ)2 .124
3 Abstracted MBRL and the problem of online data collection125
As explained, we are interested in Abstracted RL, where we have an approximate model similarity126 abstraction function φ and an MDP M . The agent acts in M but only observes φ(s) using abstraction127 function φ, as in Figure 1. This setting can also be seen as a POMDP, where the states are hidden and128 there is a deterministic observation function, o = φ(s). However, in contrast to the usual POMDP129 settings, we look for a myopic (memoryless) policy. While we know that in general this can lead to130 arbitrarily bad results [26], in this case the value loss would be bounded in the planning setting by131 Lemma 2. However, now we assume we are in the Abstracted RL setting, and the result for planning132 may not hold for the learned model.133
We assume we know S,A,R, γ, and φ (and thus S̄), but do not know the transition function.5 Since134 we do not know the transition function we can neither simply do planning on M nor can we construct135 an abstract MDP, using Definition 3, and solve that. Instead, we let the agent interact with M but136 use φ to let the agent observe φ(s), instead of s, and build a learned (abstract) model from the137 observations it obtains. We show the general Abstracted MBRL procedure in Algorithm 1.138
The agent collects data for every abstract state-action pair (s̄, a), which is stored as sequences Ȳs̄,a:139
Ȳs̄,a : {s̄′(1), s̄′(2), · · · , s̄′(m)}. (9)
Similar to before in (1), we construct a learned model T̄Y , now looking at the abstract next-states that140 were reached:141
T̄Y (s̄ ′|s̄, a) , 1
m m∑ i=1 1{Ȳ (i)s̄,a = s̄′}. (10)
If this model would be equal, or close, to the transition function T̄ω of an abstract MDP M̄ω, for142 some valid ω, we could upper bound the loss in performance due to applying learned policy π̄∗ to M143 instead of the optimal policy π∗ [1, 29].144
5The assumption that the reward function is known simplifies our arguments but can be relaxed.
Algorithm 1 Procedure: Abstracted MBRL Input: M,φ, δ, , π Ȳ = COLLECTSAMPLES(M,φ, δ, , π) The sampling results in sequences Ȳs̄,a, one for every pair (s̄, a): Ȳs̄,a = φ(s
′(1)), · · · , φ(s′(m)) = s̄′(1), · · · , s̄′(m) for all (s̄, a, s̄′) ∈ S̄ ×A× S̄ do T̄Y (s̄ ′|s̄, a) = 1m ∑m i=1 1{Ȳ (i) s̄,a = s̄
′} end for M̄Y := 〈S̄, A, T̄Y , R̄, γ〉 π̄∗Y = Value Iteration(M̄Y ) Apply π̄∗Y to M
Algorithm 2 COLLECTSAMPLES Online Input: M,φ, δ, , π s = initial state // The number of samples m is based on the simulator analysis, Theorem 1. κ = δ/(|S̄||A|) m = d 2[ln(2 |S̄|−2)−ln(κ)] 2 e
for all s̄ ∈ S̄ do Ȳs̄,a = [ ] end for while min(s̄,a) |Ys̄,a| < m do s̄ = φ(s) a = π(s̄) s′ = Step(s, a) Ȳs̄,a.append(φ(s′)) s = s′ end while Return: Return all Ȳs̄,a
145
Our main question is: do the finite-sample model learning guarantees of MBRL algorithms still hold146 in the Abstracted RL setting?147
3.1 Online data collection148
In this section we follow the MBRL method from Algorithm 1, collecting samples online using149 Algorithm 2.6 Starting from an initial state the agent follows a policy π. Instead of observing the150 states s, the agent observes abstract states s̄ = φ(s), see Figure 1.151
We make two important assumptions in order to make analysis possible. We assume that the MDP152 is ergodic [25] 7 and that the policy assigns a positive probability to every action in every abstract153 state. Together this can guarantee that Algorithm 2 can obtain any finite number of samples for every154 abstract state-action pair within finite time.155
Our question is, can we still use Lemma 1 to guarantee that we learn an accurate model?156
Since we learn an abstract transition model T̄Y , we want to be able to guarantee that this learned157 model will be close to the transition model of some abstract MDP. To define this transition model,158 we first look at how the data is collected.159
In the online data collection, a sample in Ȳs̄,a is drawn when the agent takes action a when it is160 in a ground state s ∈ s̄. Specifically the i-th abstract Ȳ (i)s̄,a = s̄′ is drawn from (ground) state161 X
(i) s̄,a = s ∈ s̄:162
Ȳ (i) s̄,a ∼ Pr(·|X (i) s̄,a = s, a). (11)
Let Xs̄,a = (X (i) s̄,a) m i=1 denote the sequence of ground states s ∈ s̄ from which the agent took action163 a. Each ground state gets a weight according to how often it was sampled from, which we formalize164 with the weighting function ωX : ∀(s̄,a),s∈s̄ ωX(s, a) , 1m ∑m i=1 1{X (i) s̄,a = s}. We use ωX to define165 T̄ωX analogous to (8):166
∀(s̄,a),s̄′ T̄ωX (s̄′|s̄, a) = ∑ s∈s̄ ωX(s, a) ∑ s′∈s̄′ T (s′|s, a). (12)
6The order of m in Algorithm 2, the number of samples we want to collect, is based on the analysis of Model-based Interval Estimation (MBIE) [28].
7An ergodic, or recurrent, MDP is an MDP where, under every stationary policy, every state is recurrent (i.e., asymptotically every state will be visited infinitely often) [25].
We want to have a concentration inequality to provide bounds on the deviation of the learned model167 T̄Y from T̄ωX , we refer to this inequality as the abstract L1 inequality, similar in form to (3):168
P (|T̄Y (·|s̄, a)− T̄ωX (·|s̄, a)|1 ≥ ) ≤ δ, (13)
where T̄Y (·|s̄, a) is defined according to (10) and T̄ωX according to (12).169 If we could directly obtain i.i.d. samples from T̄ωX and base our learned model T̄Y on the obtained170 samples, then we would be able to show that the abstract L1 inequality holds by applying Lemma 1.171 Since in this case, we would have m i.i.d. samples per abstract state-action pair, distributed according172 to T̄ωX (·|s̄, a).173 However, the samples are not guaranteed to be i.i.d. when the agent follows Algorithm 2 to collect174 the samples. Since every sample Ȳ (i) was obtained after taking action a from state X(i)s̄,a = s ∈ s̄, as175 in (11). These can have different distributions if X(i)s̄,a 6= X (j) s̄,a .176
Non Identically Distributed While Lemma 1 assumes i.i.d. random variables, we show that it also177 holds when the random variables are independent but not (necessarily) identically distributed.178 Lemma 3. Let Xs̄,a = s1, · · · , sm be a sequence of states s ∈ s̄ and let179 Ȳ s̄,a = Ȳ
(1), Ȳ (2), · · · , Ȳ (m) be independent random variables distributed according to180 Pr(·|s1, a), · · · ,Pr(·|sm, a) (Eqn. 4). Then, for all > 0,181
Pr(||T̄Y (·|s̄, a)− T̄ωX (·|s̄, a)||1 ≥ ) ≤ (2|S̄| − 2)e− 1 2m 2 . (14)
The proof can be found in Appendix B. It mostly follows the proof by Weissman et al. [30], which uses182 Hoeffding’s inequality [12] and the union bound [6].8 Lemma 3 shows that the fact that Hoeffding’s183 inequality does not need identically distributed data can be carried over to the setting from Lemma 1.184
Independence We may be tempted to assume the samples are independent, i.e.,185
∀s̄1,··· ,s̄m∈(S̄)m Pr(Ȳ (1) s̄,a = s̄1, · · · , Ȳ (m) s̄,a = s̄m) = Pr(Ȳ (1) s̄,a = s̄1) · · ·P (Ȳ (m) s̄,a = s̄m) (15)
however, this may not be the case:186 Observation 1. When collecting samples online, i.e., based on Algorithm 2, the samples cannot be187 assumed to be independent.188
The following counterexample illustrates this.189
Counterexample To show that the samples may not be indepen-190 dent, we will give a counterexample. We use the example MDP and191 abstraction in Figure 2, where we have 4 (ground) states, 3 abstract192 states and only 1 action. We look at the transition probability from193 abstract state A, T̄Y (·|A).194 We will consider two samples and show that for at least one com-195 bination of s̄1 and s̄2 the samples are not independent. Consider196 s̄1 = s̄2 = B. That is, the first two times that we experience a197 transition from the abstract state A, we end up in B.198
Let state 1 be the starting state. Then we have Pr(Ȳ (1)A = B) =199 Pr(B|1) = 0.6 and200
Pr(Ȳ (2) A = B) = ∑ s̄∈S̄ Pr(Ȳ (2) A = B|Ȳ (1) A = s̄) Pr(Ȳ (1) A = s̄)
(16) = 0 + 0.6 · 0.6 + 0.4 · 0.4 = 0.52. (17)
So then we end up with: Pr(Ȳ (1)A = B) Pr(Ȳ (2) A = B) = 0.6 · 0.52 = 0.321. And for the joint201
probability: Pr(Ȳ (1)A = B, Ȳ (2) A = B) = Pr(Ȳ (1) A = B) Pr(Ȳ (2) A = B|Ȳ (1) A = B) = 0.6 · 0.6 =202 0.36.203 8Hoeffding’s inequality and the union bound can be found in Appendix A.
Thus we have that Pr(Ȳ (1)A = B, Ȳ (2) A = B) 6= Pr(Ȳ (1) A = B) Pr(Ȳ (2) A = B), the samples are not204 independent. Leading us to the second observation:205 Observation 2. As independence cannot be guaranteed, Lemmas 1 and 3 cannot be readily applied206 to show that the abstract L1 inequality holds.207
3.2 Simulator data collection208
Here we also want to give a guarantee in the form of the abstract L1 inequality from (13). While in209 the previous section we found this was not possible because the samples were dependent, here we210 assume that we have access to a simulator. To some extent this is not surprising, but to the best of our211 knowledge, this is the first work that explicitly shows how to combine MBRL and abstraction, using212 a simulator. We assume that this allows us to select (or move to) any state and draw a sample from its213 transition function. This we call the independent samples assumption:214 Assumption 1 (Independent samples). We assume we can obtain independent samples, e.g. for any215 state-action pair (s, a) we can draw samples directly from its transition function T (·|s, a).216
In case a simulator of the MDP is available this is a reasonable assumption. For every (s̄, a) the217 simulator sampling procedure (Algorithm 3 in Appendix B) selects a prototype xs̄,a ∈ s̄ to sample218 from. We define a weighting function ωx(s, a) that has weight 1 if s is the prototype xs̄,a and 0219 otherwise:220
∀(s̄,a),s∈s̄ ωx(s, a) , 1{s = xs̄,a}. (18)
Then we use this ωx to define the abstract transition function T̄ωX according to (8). T̄ωx(s̄ ′|s̄, a) =221 ∑ s′∈s̄′ T (s ′|s = xs̄,a, a). This way the samples that we collect for one pair (s̄, a) are i.i.d., they are222 independent because of our assumption of independent samples and identically distributed because223 we sample from the prototype. This means we can use Lemma 1. We show that with the simulator224 we can combine MBRL and abstraction, and still learn an accurate model, that is, we can guarantee225 that T̄Y will be close to T̄ωx , with high probability:226 Theorem 1. Under assumption 1, and following the procedure in Algorithm 1, with the data227 collection from Algorithm 3 (Appendix B), with inputs |S̄|, A, and δ. For T̄Y constructed by the228 algorithm we have that with probability 1− δ, the following holds:229
∀(s̄,a) ||T̄Y (·|s̄, a)− T̄ωx(·|s̄, a)||1 ≤ . (19)
By Assumption 1 we can obtain any number of independent samples for each abstract state action230 pair (s̄, a). Using Lemma 1 we can then derive the number of samples m that is required for each231 pair (s̄, a) such that, after applying a union bound, we obtain the bounds in (19). The full proof can232 be found in Appendix B.233
4 Related work234
There is a lot of work that considers the combination of abstraction with either planning or (online)235 RL. In a lot of these works the dependence of samples that arises in Abstracted RL is not an issue236 due to various assumptions, similarly to how in MBRL dependence of samples is often not an issue237 because of the Markov property and the assumption that the MDP is communicating [25]. Often238 this is either due the assumption that data has been obtained i.i.d., the specific type of abstraction, or239 because access to an MDP model is assumed.240
One paper that does give a result for the Abstracted RL setting is the work by Abel et al. [2].241 They show that in this setting R-MAX [7] no longer maintains its guarantees when paired with any242 type of state-abstraction function, though their example is specifically for approximate Q-function243 abstractions. They also show that the expected trajectory of a learning agent in a constructed244 abstract MDP (Definition 3) is not the same as in Abstracted RL. Their work makes clear there is a245 complication when combining MBRL and abstraction, here we further investigated the cause of this246 complication, the dependence between samples.247
For planning in constructed abstract MDPs, some main results for exact state-abstractions come248 from Li et al. [18] and for approximate state-abstractions from Abel et al. [1]. The results from Abel249 et al. [1] allow for quantifying an upper bound on performance for policies found in a constructed250
abstract MDP, as in section 2.2. Taïga et al. [29] build on this by giving a result for performing251 RL on top of the constructed abstract MDP. They provide upper bounds for this setting when using252 MBIE with exploratory bonus (MBIE-EB) [28]. In addition, they give an example to show that in253 this combination you cannot guarantee optimal performance in the original MDP. Still, they show254 that an upper bound on the loss in value can be given.255
Both Paduraru et al. [24] and Jiang et al. [15] deal with the issue of dependence by making the256 explicit assumption that samples are obtained i.i.d. Paduraru et al. [24] consider the setting where257 we are given a dataset for a continuous domain and then use discretization to aggregate states into258 abstract states. They then give PAC-style guarantees on the learned abstract model and the value that259 a policy based on this model can achieve in the real MDP. Instead of using the L1 deviation bound260 from Weissman et al. [30], Paduraru et al. [24] use a similar bound for i.i.d. samples by Devroye and261 Gyorfi [9], which requires a minimum amount of samples. Another difference is that their results262 calculate the probability that the model will be -accurate given a fixed dataset. They assume that the263 data has been gathered i.i.d., but our Lemma 3 shows that merely independent data would be enough.264 At the same time, our results show that when we collect data online in the Abstracted RL setting,265 their guarantees will not hold.266
Jiang et al. [15] operate in the abstraction selection setting, where the agent is provided with a set of267 abstraction functions (state representations). They do not assume that any of the abstraction functions268 results in a Markov model, but they do assume a given dataset, with data that was collected i.i.d. They269 give a bound directly on how accurate the Q-values based on the (implicitly) learned model will be,270 rather than on the accuracy of the model itself. As we showed, the assumption that the data is i.i.d.271 is not a trivial assumption, since it means the data cannot just have been collected online. They do272 mention that samples will not be strictly independent if a fixed exploration policy is used to collect273 data but do not mention what the implications are.274
There are quite a few other papers in the abstraction selection setting, several of these assume that275 the given set of state representations contains a Markov model [11, 19, 23]. Hallak et al. [11] give276 asymptotic guarantees for selecting the correct model and on building an exact MDP model. The277 assumption that there is an MDP model in the given set of representations is crucial in their analysis278 since for this ‘true model’ the samples are i.i.d. Similarly, both Maillard et al. [19] and Ortner279 et al. [23] also assume that the given set of state representations contains a Markov model. They280 create an algorithm for which they obtain regret bounds, their analysis also makes use of the Markov281 representation.282
Other work in the abstraction selection setting does not assume that the set of abstraction functions283 contains a Markov model [16, 22]. However, Ortner et al. [22] use Theorem 2.1 from Weissman284 et al. [30] that requires i.i.d. samples, which we have shown here cannot be guaranteed in this setting.285 Lattimore et al. [16] operate in a setting more general than MDPs, where the dynamics of the true286 environment depend arbitrarily on a history of actions, rewards, and observations. The agent gets287 as input a finite set of environments, one of which is the true environment. Since the input includes288 the full model of each environment, the agent does not have to learn a transition model. Instead, to289 obtain regret bounds, they directly compare the rewards the agent obtains to the expected rewards of290 the given environments and eliminate environments that are implausible given the observed rewards.291
Another way to deal with the issue of dependence is by looking at convergence in the limit [27, 13, 20].292 Singh et al. [27] give an asymptotic result for the convergence of Q-learning and TD(0) in MDPs293 with soft state aggregation. Soft state aggregation means that a state s belongs to a cluster x with294 some probability P (x|s), this means a state s can belong to several clusters. The state-abstraction295 functions we consider are a special case of this, where each state is part of exactly one abstract state296 (or cluster). Their result relies on having a stationary policy that assigns a non-zero probability to297 every action in every state and the assumption that the MDP is ergodic. Together these imply there is298 a limiting state distribution, and using this they show convergence asymptotically. Our main interest299 is in finite-samples guarantees with policies that change due to exploration, whereas this work gives300 convergence guarantees in the limit using a fixed policy.301
Hutter [13] gives a variety of results focusing on both approximate and exact abstractions in envi-302 ronments without MDP assumptions. Several of these are in the planning setting, similar to those of303 Abel et al. [1]. Most relevant for us is their Theorem 12, which for online RL shows convergence in304 the limit of the empirical transition function under weak conditions, e.g. if the abstract process itself305
is an MDP. Under this condition however the problem reduces to RL in an (abstract) MDP, rather306 than Abstracted RL.307
Majeed and Hutter [20] build on the work by Hutter [13] and focus on the combination of model-free308 RL and exact abstraction. They show that, under the condition of state uniformity, q-learning can be309 shown to converge in the limit to the optimal solution. State uniformity means that histories that are310 grouped together have the same optimal q-values. In contrast to our setting, they look at an exact311 abstraction, extending it to approximate aggregation was left as an open question.312
Other related work is in the area of MDPs with rich observations or block structure [4, 10]. However,313 in that setting each observation can be generated only from a single hidden state, which means that314 the issue of non-i.i.d. data due to abstraction does not arise. In contrast, in our setting multiple315 (hidden) states generate the same observation. Azizzadenesheli et al. [4] state their setting can be316 seen as an aggregation problem, where the observations can be aggregated to form a small (latent)317 MDP. But in our case, we do not try to learn the MDP (as it is not small). Du et al. [10] describe318 that their setting is similar to exact model similarity (or bisimulation), but we focus on approximate319 model similarity which is what introduces the problems as described here.320
5 Discussion321
When collecting samples online in Abstracted RL, there is a potential dependence between samples,322 meaning we cannot use the typically used concentration results that assume i.i.d. samples, e.g.323 Theorem 2.1 from Weissman et al. [30], the empirical Bernstein inequality [3, 21] or the Chernoff324 bound. In case the samples are only weakly dependent, it may be that concentration inequalities325 for (weakly) dependent variables are a viable alternative through which we can come to guarantees326 on the learned model. Alternatively, it may be possible to change the sampling process to ensure327 independent samples. One way to ensure independent samples is to, as in the simulator setting, select328 a prototype state and only use the samples collected from this state. Though in this case, we will be329 discarding information when we reach a state s ∈ s̄ that is not the prototype.330 Our assumption on the simulator that we can go/reset to any state to draw samples from it can be331 relaxed, though it may mean that the procedure takes considerably more time. Consider the case332 where we cannot just reset the simulator to the state s from which we want to sample, and instead, it333 would behave like the MDP. In this case, we would have to take the right actions to arrive at the state334 s from which we would like to sample. Since we assume we do not know T , this may take a long335 time. This also shows the difficulty of assuming that in the MBRL setting somehow have access to an336 i.i.d. dataset, as has been assumed in some earlier work [24, 15].337
6 Conclusion338
We analyzed Abstracted RL: the combination of MBRL and state abstraction when the model of339 the MDP is not available. We have shown that in Abstracted RL samples obtained online cannot340 be assumed to be independent. Since many current guarantees from MBRL methods rely on this341 assumption, their guarantees do not hold in this setting. And in fact, no current methods exist that342 give (correct) finite-sample quality guarantees for the models learned in this setting. This also means343 that current results that rely on an i.i.d. assumption cannot be readily transferred to the Abstracted344 RL setting.345
In addition, we show that with a simulator, since we can draw independent samples, it is still possible346 to give guarantees on the accuracy of the model. However, having access to a simulator may often347 not be possible. An important step is to see if the MBRL guarantees can be adapted to Abstracted RL348 for online sample collection.349
References350
[1] David Abel, David Hershkowitz, and Michael Littman. Near optimal behavior via approximate351 state abstraction. In International Conference on Machine Learning, pages 2915–2923, 2016.352
[2] David Abel, Dilip Arumugam, Lucas Lehnert, and Michael Littman. State abstractions for353 lifelong reinforcement learning. In International Conference on Machine Learning, pages354 10–19, 2018.355
[3] Jean-Yves Audibert, Rémi Munos, and Csaba Szepesvári. Tuning bandit algorithms in stochastic356 environments. In International conference on algorithmic learning theory, pages 150–165.357 Springer, 2007.358
[4] Kamyar Azizzadenesheli, Alessandro Lazaric, and Animashree Anandkumar. Reinforcement359 learning in rich-observation mdps using spectral methods. arXiv preprint arXiv:1611.03907,360 2016.361
[5] Aijun Bai, Siddharth Srivastava, and Stuart J Russell. Markovian state and action abstractions362 for mdps via hierarchical mcts. In IJCAI, pages 3029–3039, 2016.363
[6] George Boole. An investigation of the laws of thought: on which are founded the mathematical364 theories of logic and probabilities. Dover Publications, 1854.365
[7] Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for366 near-optimal reinforcement learning. Journal of Machine Learning Research, 3(Oct):213–231,367 2002.368
[8] Richard Dearden and Craig Boutilier. Abstraction and approximate decision-theoretic planning.369 Artificial Intelligence, 89(1-2):219–283, 1997.370
[9] L. Devroye and L. Gyorfi. Nonparametric Density Estimation: The L1 View. Wiley Interscience371 Series in Discrete Mathematics. Wiley, 1985.372
[10] Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John373 Langford. Provably efficient rl with rich observations via latent state decoding. In International374 Conference on Machine Learning, pages 1665–1674. PMLR, 2019.375
[11] Assaf Hallak, Dotan Di-Castro, and Shie Mannor. Model selection in markovian processes. In376 Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and377 data mining, pages 374–382, 2013.378
[12] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of379 the American Statistical Association, 58(301):13–30, 1963.380
[13] Marcus Hutter. Extreme state aggregation beyond markov decision processes. Theoretical381 Computer Science, 650:73–91, 2016.382
[14] Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement383 learning. Journal of Machine Learning Research, 11(Apr):1563–1600, 2010.384
[15] Nan Jiang, Alex Kulesza, and Satinder Singh. Abstraction selection in model-based reinforce-385 ment learning. In International Conference on Machine Learning, pages 179–188, 2015.386
[16] Tor Lattimore, Marcus Hutter, Peter Sunehag, et al. The sample-complexity of general rein-387 forcement learning. In Proceedings of the 30th International Conference on Machine Learning.388 Journal of Machine Learning Research, 2013.389
[17] Lihong Li. A unifying framework for computational reinforcement learning theory. PhD thesis,390 Rutgers University-Graduate School-New Brunswick, 2009.391
[18] Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a unified theory of state abstraction392 for mdps. In ISAIM, 2006.393
[19] Odalric-Ambrym Maillard, Phuong Nguyen, Ronald Ortner, and Daniil Ryabko. Optimal394 regret bounds for selecting the state representation in reinforcement learning. In International395 Conference on Machine Learning, pages 543–551. PMLR, 2013.396
[20] Sultan Javed Majeed and Marcus Hutter. On q-learning convergence for non-markov decision397 processes. In IJCAI, pages 2546–2552, 2018.398
[21] Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance399 penalization. arXiv preprint arXiv:0907.3740, 2009.400
[22] Ronald Ortner, Odalric-Ambrym Maillard, and Daniil Ryabko. Selecting near-optimal ap-401 proximate state representations in reinforcement learning. In International Conference on402 Algorithmic Learning Theory, pages 140–154. Springer, 2014.403
[23] Ronald Ortner, Matteo Pirotta, Alessandro Lazaric, Ronan Fruit, and Odalric-Ambrym Maillard.404 Regret bounds for learning state representations in reinforcement learning. In Advances in405 Neural Information Processing Systems, pages 12738–12748, 2019.406
[24] Cosmin Paduraru, Robert Kaplow, Doina Precup, and Joelle Pineau. Model-based reinforcement407 learning with state aggregation. In 8th European Workshop on Reinforcement Learning, 2008.408
[25] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming.409 John Wiley & Sons, 2014.410
[26] Satinder P Singh, Tommi Jaakkola, and Michael I Jordan. Learning without state-estimation in411 partially observable markovian decision processes. In Machine Learning Proceedings 1994,412 pages 284–292. Elsevier, 1994.413
[27] Satinder P Singh, Tommi Jaakkola, and Michael I Jordan. Reinforcement learning with soft414 state aggregation. In Advances in neural information processing systems, pages 361–368, 1995.415
[28] Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for416 markov decision processes. Journal of Computer and System Sciences, 74(8):1309–1331, 2008.417
[29] Adrien Ali Taïga, Aaron Courville, and Marc G Bellemare. Approximate exploration through418 state abstraction. arXiv preprint arXiv:1808.09819, 2018.419
[30] Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger.420 Inequalities for the l1 deviation of the empirical distribution. Hewlett-Packard Labs, Tech. Rep,421 2003.422
Checklist423
1. For all authors...424 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s425 contributions and scope? [Yes]426 (b) Did you describe the limitations of your work? [Yes] In Section 6 we describe one the427 limitations of our work.428 (c) Did you discuss any potential negative societal impacts of your work? [N/A]429 (d) Have you read the ethics review guidelines and ensured that your paper conforms to430 them? [Yes]431 2. If you are including theoretical results...432
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We state our433 general assumptions on the environment in Section 2 and more specific assumptions in434 Section 3, Section 3.1 and Section 3.2.435
(b) Did you include complete proofs of all theoretical results? [Yes] In the Appendix.436 3. If you ran experiments...437
(a) Did you include the code, data, and instructions needed to reproduce the main experi-438 mental results (either in the supplemental material or as a URL)? [N/A]439
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they440 were chosen)? [N/A]441
(c) Did you report error bars (e.g., with respect to the random seed after running experi-442 ments multiple times)? [N/A]443
(d) Did you include the total amount of compute and the type of resources used (e.g., type444 of GPUs, internal cluster, or cloud provider)? [N/A]445
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...446 (a) If your work uses existing assets, did you cite the creators? [N/A]447 (b) Did you mention the license of the assets? [N/A]448 (c) Did you include any new assets either in the supplemental material or as a URL? [N/A]449
450
(d) Did you discuss whether and how consent was obtained from people whose data you’re451 using/curating? [N/A]452
(e) Did you discuss whether the data you are using/curating contains personally identifiable453 information or offensive content? [N/A]454
5. If you used crowdsourcing or conducted research with human subjects...455 (a) Did you include the full text of instructions given to participants and screenshots, if456 applicable? [N/A]457 (b) Did you describe any potential participant risks, with links to Institutional Review458 Board (IRB) approvals, if applicable? [N/A]459 (c) Did you include the estimated hourly wage paid to participants and the total amount460 spent on participant compensation? [N/A]461

Review summaries:
This paper deals with model learning when given access to a specific type of abstraction, model-irrelevance (or bisimulation). It states the when the data is collected online, learning an accurate model in the abstract MDP is not possible since the samples are no longer independent (as required by MBRL methods that use concentration inequalities to bound the error of the estimated model). It then shows that given simulator access, this problem no longer exists as samples can be collected from anywhere in the state space. 

In this work the authors present a theoretical analysis of the problems which arise when applying model based RL methods to abstracted MDPs. An abstracted MDP is obtained by mapping each state of the original MDP to a new abstract state. Subsequently, a model of the transition dynamics of this abstracted MDP can be learned. This learned model can be used to solve the abstracted MDP and the resulting policy can be applied to the original MDP. The authors,  examine whether there are any theoretical guarantees regarding the quality of the learned model of the abstracted MDP and of the produced policy. They conclude that collecting data in an online fashion can break the independence assumption made by most MBRL methods in the abstracted case. 

In this paper, the transition model accuracy guarantee in model-based RL on abstract (tabular) MDP is discussed.   The problem that the sample independence assumption, which is important for guaranteeing meaningful model accuracy bound, is not always valid when the state is abstracted by approximate model similarity abstraction.    To address this problem (to realize i.i.d sampling),  the use of a simulator, which can re-start from arbitral state-action pair, is proposed.