Year: 2019

Abstract:
Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL. However, the theoretical understanding of such methods has been rather limited. This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algorithms with theoretical guarantees. We design a meta-algorithm with a theoretical guarantee of monotone improvement to a local maximum of the expected reward. The meta-algorithm iteratively builds a lower bound of the expected reward based on the estimated dynamical model and sample trajectories, and then maximizes the lower bound jointly over the policy and the model. The framework extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires no explicit uncertainty quantification. Instantiating our framework with simplification gives a variant of model-based RL algorithms Stochastic Lower Bounds Optimization (SLBO). Experiments demonstrate that SLBO achieves stateof-the-art performance when only one million or fewer samples are permitted on a range of continuous control benchmark tasks.1

heading: 1 INTRODUCTION
text: In recent years deep reinforcement learning has achieved strong empirical success, including superhuman performances on Atari games and Go (Mnih et al., 2015; Silver et al., 2017) and learning locomotion and manipulation skills in robotics (Levine et al., 2016; Schulman et al., 2015b; Lillicrap et al., 2015). Many of these results are achieved by model-free RL algorithms that often require a massive number of samples, and therefore their applications are mostly limited to simulated environments. Model-based deep reinforcement learning, in contrast, exploits the information from state observations explicitly — by planning with an estimated dynamical model — and is considered to be a promising approach to reduce the sample complexity. Indeed, empirical results (Deisenroth & Rasmussen, 2011b; Deisenroth et al., 2013; Levine et al., 2016; Nagabandi et al., 2017; Kurutach et al., 2018; Pong et al., 2018a) have shown strong improvements in sample efficiency.
Despite promising empirical findings, many of theoretical properties of model-based deep reinforcement learning are not well-understood. For example, how does the error of the estimated model affect the estimation of the value function and the planning? Can model-based RL algorithms be guaranteed to improve the policy monotonically and converge to a local maximum of the value function? How do we quantify the uncertainty in the dynamical models?
It’s challenging to address these questions theoretically in the context of deep RL with continuous state and action space and non-linear dynamical models. Due to the high-dimensionality, learning models from observations in one part of the state space and extrapolating to another part sometimes
0* indicates equal contribution 1The source code of this work is available at https://github.com/roosephu/slbo
involves a leap of faith. The uncertainty quantification of the non-linear parameterized dynamical models is difficult — even without the RL components, it is an active but widely-open research area. Prior work in model-based RL mostly quantifies uncertainty with either heuristics or simpler models (Moldovan et al., 2015; Xie et al., 2016; Deisenroth & Rasmussen, 2011a).
Previous theoretical work on model-based RL mostly focuses on either the finite-state MDPs (Jaksch et al., 2010; Bartlett & Tewari, 2009; Fruit et al., 2018; Lakshmanan et al., 2015; Hinderer, 2005; Pirotta et al., 2015; 2013), or the linear parametrization of the dynamics, policy, or value function (Abbasi-Yadkori & Szepesvári, 2011; Simchowitz et al., 2018; Dean et al., 2017; Sutton et al., 2012; Tamar et al., 2012), but not much on non-linear models. Even with an oracle prediction intervals2 or posterior estimation, to the best of our knowledge, there was no previous algorithm with convergence guarantees for model-based deep RL.
Towards addressing these challenges, the main contribution of this paper is to propose a novel algorithmic framework for model-based deep RL with theoretical guarantees. Our meta-algorithm (Algorithm 1) extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires no explicit uncertainty quantification of the dynamical models.
Let V π be the value function V π of a policy π on the true environment, and let V̂ π be the value function of the policy π on the estimated model M̂ . We design provable upper bounds, denoted by Dπ,M̂ , on how much the error can compound and divert the expected value V̂ π of the imaginary rollouts from their real value V π, in a neighborhood of some reference policy. Such upper bounds capture the intrinsic difference between the estimated and real dynamical model with respect to the particular reward function under consideration.
The discrepancy bounds Dπ,M̂ naturally leads to a lower bound for the true value function:
V π ≥ V̂ π −Dπ,M̂ . (1.1)
Our algorithm iteratively collects batches of samples from the interactions with environments, builds the lower bound above, and then maximizes it over both the dynamical model M̂ and the policy π. We can use any RL algorithms to optimize the lower bounds, because it will be designed to only depend on the sample trajectories from a fixed reference policy (as opposed to requiring new interactions with the policy iterate.)
We show that the performance of the policy is guaranteed to monotonically increase, assuming the optimization within each iteration succeeds (see Theorem 3.1.) To the best of our knowledge, this is the first theoretical guarantee of monotone improvement for model-based deep RL.
Readers may have realized that optimizing a robust lower bound is reminiscent of robust control and robust optimization. The distinction is that we optimistically and iteratively maximize the RHS of (1.1) jointly over the model and the policy. The iterative approach allows the algorithms to collect higher quality trajectory adaptively, and the optimism in model optimization encourages explorations of the parts of space that are not covered by the current discrepancy bounds.
To instantiate the meta-algorithm, we design a few valid discrepancy bounds in Section 4. In Section 4.1, we recover the norm-based model loss by imposing the additional assumption of a Lipschitz value function. The result suggests a norm is preferred compared to the square of the norm. Indeed in Section 6.2, we show that experimentally learning with `2 loss significantly outperforms the mean-squared error loss (`22).
In Section 4.2, we design a discrepancy bound that is invariant to the representation of the state space. Here we measure the loss of the model by the difference between the value of the predicted next state and the value of the true next state. Such a loss function is shown to be invariant to one-to-one transformation of the state space. Thus we argue that the loss is an intrinsic measure for the model error without any information beyond observing the rewards. We also refine our bounds in Section A by utilizing some mathematical tools of measuring the difference between policies in χ2-divergence (instead of KL divergence or TV distance).
2We note that the confidence interval of parameters are likely meaningless for over-parameterized neural networks models.
Our analysis also sheds light on the comparison between model-based RL and on-policy model-free RL algorithms such as policy gradient or TRPO (Schulman et al., 2015a). The RHS of equation (1.1) is likely to be a good approximator of V π in a larger neighborhood than the linear approximation of V π used in policy gradient is (see Remark 4.5.)
Finally, inspired by our framework and analysis, we design a variant of model-based RL algorithms Stochastic Lower Bounds Optimization (SLBO). Experiments demonstrate that SLBO achieves state-of-the-art performance when only 1M samples are permitted on a range of continuous control benchmark tasks.

heading: 2 NOTATIONS AND PRELIMINARIES
text: We denote the state space by S, the action space by A. A policy π(·|s) specifies the conditional distribution over the action space given a state s. A dynamical model M(·|s, a) specifies the conditional distribution of the next state given the current state s and action a. We will use M? globally to denote the unknown true dynamical model. Our target applications are problems with the continuous state and action space, although the results apply to discrete state or action space as well. When the model is deterministic, M(·|s, a) is a dirac measure. In this case, we use M(s, a) to denote the unique value of s′ and view M as a function from S ×A to S . LetM denote a (parameterized) family of models that we are interested in, and Π denote a (parameterized) family of policies.
Unless otherwise stated, for random variable X , we will use pX to denote its density function.
Let S0 be the random variable for the initial state. Let S π,M t to denote the random variable of the states at steps t when we execute the policy π on the dynamic model M stating with S0. Note that Sπ,M0 = S0 unless otherwise stated. We will omit the subscript when it’s clear from the context. We use At to denote the actions at step t similarly. We often use τ to denote the random variable for the trajectory (S0, A1, . . . , St, At, . . . ). Let R(s, a) be the reward function at each step. We assume R is known throughout the paper, although R can be also considered as part of the model if unknown. Let γ be the discount factor.
Let V π,M be the value function on the model M and policy π defined as:
V π,M (s) = E ∀t≥0,At∼π(·|St) St+1∼M(·|St,At)
[ ∞∑ t=0 γtR(St, At) | S0 = s ] (2.1)
We define V π,M = E [ V π,M (S0) ] as the expected reward-to-go at Step 0 (averaged over the random initial states). Our goal is to maximize the reward-to-go on the true dynamical model, that is, V π,M ?
, over the policy π. For simplicity, throughout the paper, we set κ = γ(1 − γ)−1 since it occurs frequently in our equations. Every policy π induces a distribution of states visited by policy π: Definition 2.1. For a policy π, define ρπ,M as the discounted distribution of the states visited by π on M . Let ρπ be a shorthand for ρπ,M ?
and we omit the superscript M? throughout the paper. Concretely,we have ρπ = (1− γ) ∑∞ t=0 γ t · pSπt

heading: 3 ALGORITHMIC FRAMEWORK
text: As mentioned in the introduction, towards optimizing V π,M ? ,3 our plan is to build a lower bound for V π,M ? of the following type and optimize it iteratively:
V π,M ? ≥ V π,M̂ −D(M̂, π) (3.1)
where D(M̂, π) ∈ R≥0 bounds from above the discrepancy between V π,M̂ and V π,M ?
. Building such an optimizable discrepancy bound globally that holds for all M̂ and π turns out to be rather difficult, if not impossible. Instead, we shoot for establishing such a bound over the neighborhood of a reference policy πref.
V π,M ?
≥ V π,M̂ −Dπref,δ(M̂, π), ∀π s.t. d(π, πref) ≤ δ (R1) 3Note that in the introduction we used V π for simplicity, and in the rest of the paper we will make the
dependency on M? explicit.
Here d(·, ·) is a function that measures the closeness of two policies, which will be chosen later in alignment with the choice of D. We will mostly omit the subscript δ in D for simplicity in the rest of the paper. We will require our discrepancy bound to vanish when M̂ is an accurate model:
M̂ = M? =⇒ Dπref(M̂, π) = 0, ∀π, πref (R2)
The third requirement for the discrepancy bound D is that it can be estimated and optimized in the sense that
Dπref(M̂, π) is of the form E τ∼πref,M? [f(M̂, π, τ)] (R3)
where f is a known differentiable function. We can estimate such discrepancy bounds for every π in the neighborhood of πref by sampling empirical trajectories τ (1), . . . , τ (n) from executing policy πref on the real environment M? and compute the average of f(M̂, π, τ (i))’s. We would have to insist that the expectation cannot be over the randomness of trajectories from π on M?, because then we would have to re-sample trajectories for every possible π encountered.
For example, assuming the dynamical models are all deterministic, one of the valid discrepancy bounds (under some strong assumptions) that will prove in Section 4 is a multiple of the error of the prediction of M̂ on the trajectories from πref:
Dπref(M̂, π) = L · E S0,...,St,∼πref,M?
[ ‖M̂(St)− St+1‖ ] (3.2)
Suppose we can establish such an discrepancy bound D (and the distance function d) with properties (R1), (R2), and (R3), — which will be the main focus of Section 4 —, then we can devise the following meta-algorithm (Algorithm 1). We iteratively optimize the lower bound over the policy πk+1 and the model Mk+1, subject to the constraint that the policy is not very far from the reference policy πk obtained in the previous iteration. For simplicity, we only state the population version with the exact computation of Dπref(M̂, π), though empirically it is estimated by sampling trajectories.
Algorithm 1 Meta-Algorithm for Model-based RL Inputs: Initial policy π0. Discrepancy bound D and distance function d that satisfy equation (R1) and (R2). For k = 0 to T :
πk+1,Mk+1 = argmax π∈Π, M∈M V π,M −Dπk,δ(M,π) (3.3)
s.t. d(π, πk) ≤ δ (3.4)
We first remark that the discrepancy bound Dπk(M,π) in the objective plays the role of learning the dynamical model by ensuring the model to fit to the sampled trajectories. For example, using the discrepancy bound in the form of equation (3.2), we roughly recover the standard objective for model learning, with the caveat that we only have the norm instead of the square of the norm in MSE. Such distinction turns out to be empirically important for better performance (see Section 6.2).
Second, our algorithm can be viewed as an extension of the optimism-in-face-of-uncertainty (OFU) principle to non-linear parameterized setting: jointly optimizing M and π encourages the algorithm to choose the most optimistic model among those that can be used to accurately estimate the value function. (See (Jaksch et al., 2010; Bartlett & Tewari, 2009; Fruit et al., 2018; Lakshmanan et al., 2015; Pirotta et al., 2015; 2013) and references therein for the OFU principle in finite-state MDPs.) The main novelty here is to optimize the lower bound directly, without explicitly building any confidence intervals, which turns out to be challenging in deep learning. In other words, the uncertainty is measured straightforwardly by how the error would affect the estimation of the value function.
Thirdly, the maximization of V π,M , whenM is fixed, can be solved by any model-free RL algorithms with M as the environment without querying any real samples. Optimizing V π,M jointly over π,M can be also viewed as another RL problem with an extended actions space using the known “extended MDP technique”. See (Jaksch et al., 2010, section 3.1) for details.
Our main theorem shows formally that the policy performance in the real environment is nondecreasing under the assumption that the real dynamics belongs to our parameterized familyM.4
Theorem 3.1. Suppose that M? ∈ M, that D and d satisfy equation (R1) and (R2), and the optimization problem in equation (3.3) is solvable at each iteration. Then, Algorithm 1 produces a sequence of policies π0, . . . , πT with monotonically increasing values:
V π0,M ? ≤ V π1,M ? ≤ · · · ≤ V πT ,M ?
(3.5)
Moreover, as k →∞, the value V πk,M? converges to some V π̄,M? , where π̄ is a local maximum of V π,M ? in domain Π.
The theorem above can also be extended to a finite sample complexity result with standard concentration inequalities. We show in Theorem G.2 that we can obtain an approximate local maximum in O(1/ε) iterations with sample complexity (in the number of trajectories) that is polynomial in dimension and accuracy ε and is logarithmic in certain smoothness parameters.
Proof of Theorem 3.1. Since D and d satisfy equation (R1), we have that
V πk+1,M ?
≥ V πk+1,Mk+1 −Dπk(Mk+1, πk+1) By the definition that πk+1 and Mk+1 are the optimizers of equation (3.3), we have that
V πk+1,Mk+1 −Dπk(Mk+1, πk+1) ≥ V πk,M ? −Dπk(M?, πk) = V πk,M ?
(by equation R2)
Combing the two equations above we complete the proof of equation (3.5).
For the second part of the theorem, by compactness, we have that a subsequence of πk converges to some π̄. By the monotonicity we have V πk,M
? ≤ V π̄,M? for every k ≥ 0. For the sake of contradiction, we assume π̄ is a not a local maximum, then in the neighborhood of π̄ there exists π′ such that V π ′,M? > V π̄,M ?
and d(π̄, π′) < δ/2. Let t be such that πt is in the δ/2-neighborhood of π̄. Then we see that (π′,M?) is a better solution than (πt+1,Mt+1) for the optimization problem (3.3) in iteration t because V π ′,M? > V π̄,M ? ≥ V πt+1,M? ≥ V πt+1,Mt+1 −Dπt(Mt+1, πt+1). (Here the last inequality uses equation (R1) with πt as πref.) The fact (π′,M?) is a strictly better solution than (πt+1,Mt+1) contradicts the fact that (πt+1,Mt+1) is defined to be the optimal solution of (3.3) . Therefore π̄ is a local maximum and we complete the proof.

heading: 4 DISCREPANCY BOUNDS DESIGN
text: In this section, we design discrepancy bounds that can provably satisfy the requirements (R1), (R2), and (R3). We design increasingly stronger discrepancy bounds from Section 4.1 to Section A.

heading: 4.1 NORM-BASED PREDICTION ERROR BOUNDS
text: In this subsection, we assume the dynamical model M? is deterministic and we also learn with a deterministic model M̂ . Under assumptions defined below, we derive a discrepancy bound D of the form ‖M̂(S,A)−M?(S,A)‖ averaged over the observed state-action pair (S,A) on the dynamical model M̂ . This suggests that the norm is a better metric than the mean-squared error for learning the model, which is empirically shown in Section 6.2. Through the derivation, we will also introduce a telescoping lemma, which serves as the main building block towards other finer discrepancy bounds.
We make the (strong) assumption that the value function V π,M̂ on the estimated dynamical model is L-Lipschitz w.r.t to some norm ‖ · ‖ in the sense that
∀s, s′ ∈ S, ∣∣V π,M̂ (s)− V π,M̂ (s′)∣∣ ≤ L · ‖s− s′‖ (4.1)
In other words, nearby starting points should give reward-to-go under the same policy π. We note that not every real environment M? has this property, let alone the estimated dynamical models.
4We note that such an assumption, though restricted, may not be very far from reality: optimistically speaking, we only need to approximate the dynamical model accurately on the trajectories of the optimal policy. This might be much easier than approximating the dynamical model globally.
However, once the real dynamical model induces a Lipschitz value function, we may penalize the Lipschitz-ness of the value function of the estimated model during the training.
We start off with a lemma showing that the expected prediction error is an upper bound of the discrepancy between the real and imaginary values.
Lemma 4.1. Suppose V π,M̂ is L-Lipschitz (in the sense of (4.1)). Recall κ = γ(1− γ)−1.∣∣V π,M̂ − V π,M? ∣∣ ≤ κL E S∼ρπ
A∼π(·|S)
[ ‖M̂(S,A)−M?(S,A)‖ ] (4.2)
However, in RHS in equation 4.2 cannot serve as a discrepancy bound because it does not satisfy the requirement (R3) — to optimize it over π we need to collect samples from ρπ for every iterate π — the state distribution of the policy π on the real model M?. The main proposition of this subsection stated next shows that for every π in the neighborhood of a reference policy πref, we can replace the distribution ρπ be a fixed distribution ρπref with incurring only a higher order approximation. We use the expected KL divergence between two π and πref to define the neighborhood:
dKL(π, πref) = E S∼ρπ
[ KL(π(·|S), πref(·|S))1/2 ] (4.3)
Proposition 4.2. In the same setting of Lemma 4.1, assume in addition that π is close to a reference policy πref in the sense that dKL(π, πref) ≤ δ, and that the states in S are uniformly bounded in the sense that ‖s‖ ≤ B, ∀s ∈ S. Then,∣∣V π,M̂ − V π,M? ∣∣ ≤ κL E
S∼ρπref A∼π(·|S)
[ ‖M̂(S,A)−M?(S,A)‖ ] + 2κ2δB (4.4)
In a benign scenario, the second term in the RHS of equation (4.4) should be dominated by the first term when the neighborhood size δ is sufficiently small. Moreover, the term B can also be replaced by maxS,A‖M̂(S,A)−M?(S,A)‖ (see the proof that is deferred to Section C.). The dependency on κ may not be tight for real-life instances, but we note that most analysis of similar nature loses the additional κ factor Schulman et al. (2015a); Achiam et al. (2017), and it’s inevitable in the worst-case.
A telescoping lemma. Towards proving Propositions 4.2 and deriving stronger discrepancy bound, we define the following quantity that captures the discrepancy between M̂ and M? on a single state-action pair (s, a).
Gπ,M̂ (s, a) = E ŝ′∼M̂(·|s,a) V π,M̂ (ŝ′)− E s′∼M?(·|s,a) V π,M̂ (s′) (4.5)
Note that if M, M̂ are deterministic, then Gπ,M̂ (s, a) = V π,M̂ (M̂(s, a))− V π,M̂ (M?(s, a)). We give a telescoping lemma that decompose the discrepancy between V π,M and V π,M ?
into the expected single-step discrepancy G. Lemma 4.3. [Telescoping Lemma] Recall that κ := γ(1− γ)−1. For any policy π and dynamical models M, M̂ , we have that
V π,M̂ − V π,M = κ E S∼ρπ,M A∼π(·|S)
[ Gπ,M̂ (S,A) ] (4.6)
The proof is reminiscent of the telescoping expansion in Kakade & Langford (2002) (c.f. Schulman et al. (2015a)) for characterizing the value difference of two policies, but we apply it to deal with the discrepancy between models. The detail is deferred to Section B. With the telescoping Lemma 4.3, Proposition 4.1 follows straightforwardly from Lipschitzness of the imaginary value function. Proposition 4.2 follows from that ρπ and ρπref are close. We defer the proof to Appendix C.

heading: 4.2 REPRESENTATION-INVARIANT DISCREPANCY BOUNDS
text: The main limitation of the norm-based discrepancy bounds in previous subsection is that it depends on the state representation. Let T be a one-to-one map from the state space S to some other space S ′,
and for simplicity of this discussion let’s assume a model M is deterministic. Then if we represent every state s by its transformed representation T s, then the transformed model MT defined as MT (s, a) , TM(T −1s, a) together with the transformed reward RT (s, a) , R(T −1s, a) and transformed policy πT (s) , π(T −1s) is equivalent to the original set of the model, reward, and policy in terms of the performance (Lemma C.1). Thus such transformation T is not identifiable from only observing the reward. However, the norm in the state space is a notion that depends on the hidden choice of the transformation T . 5
Another limitation is that the loss for the model learning should also depend on the state itself instead of only on the difference M̂(S,A)−M?(S,A). It is possible that when S is at a critical position, the prediction error needs to be highly accurate so that the model M̂ can be useful for planning. On the other hand, at other states, the dynamical model is allowed to make bigger mistakes because they are not essential to the reward.
We propose the following discrepancy bound towards addressing the limitations above. Recall the definition of Gπ,M̂ (s, a) = V π,M̂ (M̂(s, a))− V π,M̂ (M?(s, a)) which measures the difference between M̂(s, a)) and M?(s, a) according to their imaginary rewards. We construct a discrepancy bound using the absolute value of G. Let’s define ε1 and εmax as the average of |Gπ,M̂ | and its maximum: ε1 = ES∼ρπref
[∣∣∣Gπ,M̂ (S,A)∣∣∣] and εmax = maxS ∣∣∣Gπ,M̂ (S)∣∣∣ where Gπ,M̂ (S) = EA∼π [ Gπ,M̂ (S,A) ] . We will show that the following discrepancy bound DGπref(M̂, π) satisfies the property (R1), (R2).
DGπref(M̂, π) = κ · ε1 + κ 2δεmax (4.7)
Proposition 4.4. Let dKL and DG be defined as in equation (4.3) and (4.7). Then the choice d = dKL and D = DG satisfies the basic requirements (equation (R1) and (R2)). Moreover, G is invariant w.r.t any one-to-one transformation of the state space (in the sense of equation C.1 in the proof).
The proof follows from the telescoping lemma (Lemma 4.3) and is deferred to Section C. We remark that the first term κε1 can in principle be estimated and optimized approximately: the expectation be replaced by empirical samples from ρπref , and Gπ,M̂ is an analytical function of π and M̂ when they are both deterministic, and therefore can be optimized by back-propagation through time (BPTT). (When π and M̂ and are stochastic with a re-parameterizable noise such as Gaussian distribution Kingma & Welling (2013), we can also use back-propagation to estimate the gradient.) The second term in equation (4.7) is difficult to optimize because it involves the maximum. However, it can be in theory considered as a second-order term because δ can be chosen to be a fairly small number. (In the refined bound in Section A, the dependency on δ is even milder.) Remark 4.5. Proposition 4.4 intuitively suggests a technical reason of why model-based approach can be more sample-efficient than policy gradient based algorithms such as TRPO or PPO (Schulman et al., 2015a; 2017). The approximation error of V π,M̂ in model-based approach decreases as the model error ε1, εmax decrease or the neighborhood size δ decreases, whereas the approximation error in policy gradient only linearly depends on the the neighborhood size Schulman et al. (2015a). In other words, model-based algorithms can trade model accuracy for a larger neighborhood size, and therefore the convergence can be faster (in terms of outer iterations.) This is consistent with our empirical observation that the model can be accurate in a descent neighborhood of the current policy so that the constraint (3.4) can be empirically dropped. We also refine our bonds in Section A, where the discrepancy bounds is proved to decay faster in δ.

heading: 5 ADDITIONAL RELATED WORK
text: Model-based reinforcement learning is expected to require fewer samples than model-free algorithms (Deisenroth et al., 2013) and has been successfully applied to robotics in both simulation and in the real world (Deisenroth & Rasmussen, 2011b; Morimoto & Atkeson, 2003; Deisenroth et al., 2011) using dynamical models ranging from Gaussian process (Deisenroth & Rasmussen, 2011b; Ko
5That said, in many cases the reward function itself is known, and the states have physical meanings, and therefore we may be able to use the domain knowledge to figure out the best norm.
& Fox, 2009), time-varying linear models (Levine & Koltun, 2013; Lioutikov et al., 2014; Levine & Abbeel, 2014; Yip & Camarillo, 2014), mixture of Gaussians (Khansari-Zadeh & Billard, 2011), to neural networks (Hunt et al., 1992; Nagabandi et al., 2017; Kurutach et al., 2018; Tangkaratt et al., 2014; Sanchez-Gonzalez et al., 2018; Pascanu et al., 2017). In particular, the work of Kurutach et al. (2018) uses an ensemble of neural networks to learn the dynamical model, and significantly reduces the sample complexity compared to model-free approaches. The work of Chua et al. (2018) makes further improvement by using a probabilistic model ensemble. Clavera et al. (Clavera et al., 2018) extended this method with meta-policy optimization and improve the robustness to model error. In contrast, we focus on theoretical understanding of model-based RL and the design of new algorithms, and our experiments use a single neural network to estimate the dynamical model.
Our discrepancy bound in Section 4 is closely related to the work (Farahmand et al., 2017) on the value-aware model loss. Our approach differs from it in three details: a) we use the absolute value of the value difference instead of the squared difference; b) we use the imaginary value function from the estimated dynamical model to define the loss, which makes the loss purely a function of the estimated model and the policy; c) we show that the iterative algorithm, using the loss function as a building block, can converge to a local maximum, partly by cause of the particular choices made in a) and b). Asadi et al. (2018) also study the discrepancy bounds under Lipschitz condition of the MDP.
Prior work explores a variety of ways of combining model-free and model-based ideas to achieve the best of the two methods (Sutton, 1991; 1990; Racanière et al., 2017; Mordatch et al., 2016; Sun et al., 2018). For example, estimated models (Levine & Koltun, 2013; Gu et al., 2016; Kalweit & Boedecker, 2017) are used to enrich the replay buffer in the model-free off-policy RL. Pong et al. (2018b) proposes goal-conditioned value functions trained by model-free algorithms and uses it for model-based controls. Feinberg et al. (2018); Buckman et al. (2018) use dynamical models to improve the estimation of the value functions in the model-free algorithms.
On the control theory side, Dean et al. (2018; 2017) provide strong finite sample complexity bounds for solving linear quadratic regulator using model-based approach. Boczar et al. (2018) provide finitedata guarantees for the “coarse-ID control” pipeline, which is composed of a system identification step followed by a robust controller synthesis procedure. Our method is inspired by the general idea of maximizing a low bound of the reward in (Dean et al., 2017). By contrast, our work applies to non-linear dynamical systems. Our algorithms also estimate the models iteratively based on trajectory samples from the learned policies.
Strong model-based and model-free sample complexity bounds have been achieved in the tabular case (finite state space). We refer the readers to (Kakade et al., 2018; Dann et al., 2017; Szita & Szepesvári, 2010; Kearns & Singh, 2002; Jaksch et al., 2010; Agrawal & Jia, 2017) and the reference therein. Our work focus on continuous and high-dimensional state space (though the results also apply to tabular case).
Another line of work of model-based reinforcement learning is to learn a dynamic model in a hidden representation space, which is especially necessary for pixel state spaces (Kakade et al., 2018; Dann et al., 2017; Szita & Szepesvári, 2010; Kearns & Singh, 2002; Jaksch et al., 2010). Srinivas et al. (2018) shows the possibility to learn an abstract transition model to imitate expert policy. Oh et al. (2017) learns the hidden state of a dynamical model to predict the value of the future states and applies RL or planning on top of it. Serban et al. (2018); Ha & Schmidhuber (2018) learns a bottleneck representation of the states. Our framework can be potentially combined with this line of research.

heading: 6 PRACTICAL IMPLEMENTATION AND EXPERIMENTS

heading: 6.1 PRACTICAL IMPLEMENTATION
text: We design with simplification of our framework a variant of model-based RL algorithms, Stochastic Lower Bound Optimization (SLBO). First, we removed the constraints (3.4). Second, we stop the gradient w.r.t M (but not π) from the occurrence of M in V π,M in equation (3.3) (and thus our practical implementation is not optimism-driven.)
Extending the discrepancy bound in Section 4.1, we use a multi-step prediction loss for learning the models with `2 norm. For a state st and action sequence at:t+h, we define the h-step prediction ŝt+h
as ŝt = st, and for h ≥ 0, ŝt+h+1 = M̂φ(ŝt+h, at+h), The H-step loss is then defined as
L(H)φ ((st:t+h, at:t+h);φ) = 1
H H∑ i=1 ‖(ŝt+i − ŝt+i−1)− (st+i − st+i−1)‖2. (6.1)
A similar loss is also used in Nagabandi et al. (2017) for validation. We note that motivation by the theory in Section 4.1, we use `2-norm instead of the square of `2 norm. The loss function we attempt to optimize at iteration k is thus6
max φ,θ V πθ,sg(M̂φ) − λ E (st:t+h,at:t+h)∼πk,M?
[ L(H)φ ((st:t+h, at:t+h);φ) ] (6.2)
where λ is a tunable parameter and sg denotes the stop gradient operation.
We note that the term V πθ,sg(M̂φ) depends on both the parameter θ and the parameter φ but there is no gradient passed through φ, whereas L(H)φ only depends on the φ. We optimize equation (6.2) by alternatively maximizing V πθ,sg(M̂φ) and minimizing L(H)φ : for the former, we use TRPO with samples from the estimated dynamical model M̂φ (by treating M̂φ as a fixed simulator), and for the latter we use standard stochastic gradient methods. Algorithm 2 gives a pseudo-code for the algorithm. The nmodel and npolicy iterations are used to balance the number of steps of TRPO and Adam updates within the loop indexed by ninner.7
Algorithm 2 Stochastic Lower Bound Optimization (SLBO) 1: Initialize model network parameters φ and policy network parameters θ 2: Initialize dataset D ← ∅ 3: for nouter iterations do 4: D ← D ∪ { collect ncollect samples from real environment using πθ with noises } 5: for ninner iterations do . optimize (6.2) with stochastic alternating updates 6: for nmodel iterations do 7: optimize (6.1) over φ with sampled data from D by one step of Adam 8: for npolicy iterations do 9: D′ ← { collect ntrpo samples using M̂φ as dynamics } 10: optimize πθ by running TRPO on D′
Power of stochasticity and connection to standard MB RL: We identify the main advantage of our algorithms over standard model-based RL algorithms is that we alternate the updates of the model and the policy within an outer iteration. By contrast, most of the existing model-based RL methods only optimize the models once (for a lot of steps) after collecting a batch of samples (see Algorithm 3 for an example). The stochasticity introduced from the alternation with stochastic samples seems to dramatically reduce the overfitting (of the policy to the estimated dynamical model) in a way similar to that SGD regularizes ordinary supervised training. 8 Another way to view the algorithm is that the model obtained from line 7 of Algorithm 2 at different inner iteration serves as an ensemble of models. We do believe that a cleaner and easier instantiation of our framework (with optimism) exists, and the current version, though performing very well, is not necessarily the best implementation.
Entropy regularization: An additional component we apply to SLBO is the commonly-adopted entropy regularization in policy gradient method (Williams & Peng, 1991; Mnih et al., 2016), which was found to significantly boost the performance in our experiments (ablation study in Appendix F.5). Specifically, an additional entropy term is added to the objective function in TRPO. We hypothesize that entropy bonus helps exploration, diversifies the collected data, and thus prevents overfitting.
6This is technically not a well-defined mathematical objective. The sg operation means identity when the function is evaluated, whereas when computing the update, sg(Mφ) is considered fixed.
7In principle, to balance the number of steps, it suffices to take one of nmodel and npolicy to be 1. However, empirically we found the optimal balance is achieved with larger nmodel and npolicy, possibly due to complicated interactions between the two optimization problem.
8Similar stochasticity can potentially be obtained by an extreme hyperparameter choice of the standard MB RL algorithm: in each outer iteration of Algorithm 3, we only sample a very small number of trajectories and take a few model updates and policy updates. We argue our interpretation of stochastic optimization of the lower bound (6.2) is more natural in that it reveals the regularization from stochastic optimization.

heading: 6.2 EXPERIMENTAL RESULTS
text: We evaluate our algorithm SLBO (Algorithm 2) on five continuous control tasks from rllab (Duan et al., 2016), including Swimmer, Half Cheetah, Humanoid, Ant, Walker. All environments that we test have a maximum horizon of 500, which is longer than most of the existing model-based RL work (Nagabandi et al., 2017; Kurutach et al., 2018). (Environments with longer horizons are commonly harder to train.) More details can be found in Appendix F.1.
Baselines. We compare our algorithm with 3 other algorithms including: (1) Soft Actor-Critic (SAC) (Haarnoja et al., 2018), the state-of-the-art model-free off-policy algorithm in sample efficiency; (2) Trust-Region Policy Optimization (TRPO) (Schulman et al., 2015a), a policy-gradient based algorithm; and (3) Model-Based TRPO, a standard model-based algorithm described in Algorithm 3. Details of these algorithms can be found in Appendix F.4.9
The result is shown in Figure 1. In Fig 1, our algorithm shows superior convergence rate (in number of samples) than all the baseline algorithms while achieving better final performance with 1M samples. Specifically, we mark model-free TRPO performance after 8 million steps by the dotted line in Fig 1 and find out that our algorithm can achieve comparable or better final performance in one million steps. For ablation study, we also add the performance of SLBO-MSE, which corresponds to running SLBO with squared `2 model loss instead of `2. SLBO-MSE performs significantly worse than SLBO on four environments, which is consistent with our derived model loss in Section 4.1. We also study the performance of SLBO and baselines with 4 million training samples in F.5. Ablation study of multi-step model training can be found in Appendix F.5.10

heading: 7 CONCLUSIONS
text: We devise a novel algorithmic framework for designing and analyzing model-based RL algorithms with the guarantee to convergence monotonically to a local maximum of the reward. Experimental results show that our proposed algorithm (SLBO) achieves new state-of-the-art performance on several mujoco benchmark tasks when one million or fewer samples are permitted.
9We did not have the chance to implement the competitive random search algorithms in (Mania et al., 2018) yet, although our test performance with 500 episode length is higher than theirs with 1000 episode on Half Cheetach (3950 by ours vs 2345 by theirs) and Walker (3650 by ours vs 894 by theirs).
10Videos demonstrations are available at https://sites.google.com/view/algombrl/home. A link to the codebase is available at https://github.com/roosephu/slbo.
A compelling (but obvious) empirical open question then given rise to is whether model-based RL can achieve near-optimal reward on other more complicated tasks or real-world robotic tasks with fewer samples. We believe that understanding the trade-off between optimism and robustness is essential to design more sample-efficient algorithms. Currently, we observed empirically that the optimism-driven part of our proposed meta-algorithm (optimizing V π,M̂ over M̂ ) may lead to instability in the optimization, and therefore don’t in general help the performance. It’s left for future work to find practical implementation of the optimism-driven approach.
In our theory, we assume that the parameterized model class contains the true dynamical model. Removing this assumption is also another interesting open question. It would be also very interesting if the theoretical analysis can be applied other settings involving model-based approaches (e.g., model-based imitation learning).

heading: Acknowledgments:
text: We thank the anonymous reviewers for detailed, thoughtful, and helpful reviews. We’d like to thank Emma Brunskill, Chelsea Finn, Shane Gu, Ben Recht, and Haoran Tang for many helpful comments and discussions.

heading: A REFINED BOUNDS
text: The theoretical limitation of the discrepancy bound DG(M̂, π) is that the second term involving εmax is not rigorously optimizable by stochastic samples. In the worst case, there seem to exist situations where such infinity norm of Gπ,M̂ is inevitable. In this section we tighten the discrepancy bounds with a different closeness measure d, χ2-divergence, in the policy space, and the dependency on the εmax is smaller (though not entirely removed.) We note that χ2-divergence has the same second order approximation as KL-divergence around the local neighborhood the reference policy and thus locally affects the optimization much.
We start by defining a re-weighted version βπ of the distribution ρπ where examples in later step are slightly weighted up. We can effectively sample from βπ by importance sampling from ρπ
Definition A.1. For a policy π, define βπ as the re-weighted version of discounted distribution of the states visited by π on M?. Recall that pSπt is the distribution of the state at step t, we define βπ = (1− γ)2 ∑∞ t=1 tγ t−1pSπt .
Then we are ready to state our discrepancy bound. Let
dχ 2
(π, πref) = max{ E S∼ρπref
[ χ2(π(·|S), πref(·|S)) ] , E S∼βπref [ χ2(π(·|S), πref(·|S)) ] } (A.1)
Dχ 2 πref(M̂, π) = (1− γ) −1ε1 + (1− γ)−2δε2 + (1− γ)−5/2δ3/2εmax (A.2) where ε2 = ES∼βπref [ Gπ,M̂ (S,A)2 ] and ε1, εmax are defined in equation (4.2).
Proposition A.2. The discrepancy bound Dχ 2 and closeness measure dχ 2
satisfies requirements (R1) and (R2).
We defer the proof to Section C so that we can group relevant proofs with similar tools together. Some of these tools may be of independent interests and used for better analysis of model-free reinforcement learning algorithms such as TRPO Schulman et al. (2015a), PPO Schulman et al. (2017) and CPO Achiam et al. (2017).

heading: B PROOF OF LEMMA 4.3
text: Proof of Lemma 4.3. Let Wj be the cumulative reward when we use dynamical model M for j steps and then M̂ for the rest of the steps, that is,
Wj = E ∀t≥0,At∼π(·|St)
∀j>t≥0,St+1∼M(·|St,At) ∀t≥j,St+1∼M̂(·|St,At)
[ ∞∑ t=0 γtR(St, At) | S0 = s ]
By definition, we have that W∞ = V π,M (s) and W0 = V π,M̂ (s). Then, we decompose the target into a telescoping sum,
V π,M (s)− V π,M̂ (s) = ∞∑ j=0 (Wj+1 −Wj) (B.1)
Now we re-write each of the summands Wj+1 − Wj . Comparing the trajectory distribution in the definition of Wj+1 and Wj , we see that they only differ in the dynamical model applied in j-th step. Concretely, Wj and Wj+1 can be rewritten
as Wj = R + ESj ,Aj∼π,M [ EŜj+1∼M̂(·|Sj ,Aj) [ γj+1V π,M̂ (Ŝj+1) ]] and Wj+1 = R +
ESj ,Aj∼π,M? [ ESj+1∼M(·|Sj ,Aj) [ γj+1V π,M̂ (Sj+1) ]] where R denotes the reward from the first j steps from policy π and model M?. Canceling the shared term in the two equations above, we get
Wj+1 −Wj = γj+1 E Sj ,Aj∼π,M  E Ŝj+1∼M̂(·|Sj ,Aj) Sj+1∼M(·|Sj ,Aj) [ V π,M̂ (Sj+1)− V π,M̂ (Ŝj+1) ]
Combining the equation above with equation (B.1) concludes that
V π,M − V π,M̂ = γ 1− γ ES∼ρπ,A∼π(S)
[ E
S′∼M?(·|S,A) V π,M̂ (S′)− E Ŝ′∼M̂(·|S,A) V π,M̂ (Ŝ′)
]

heading: C MISSING PROOFS IN SECTION 4

heading: C.1 PROOF OF PROPOSITION 4.4
text: Towards proving the second part of Proposition 4.4 regarding the invariance, we state the following lemma:
Lemma C.1. Suppose for simplicity the model and the policy are both deterministic. For any oneto-one transformation from S to S ′, let MT (s, a) , TM(T −1s, a), RT (s, a) , R(T −1s, a), and πT (s) , π(T −1s) be a set of transformed model, reward and policy. Then we have that (M,π,R) is equivalent to (MT , πT , RT ) in the sense that
V π T ,MT (T s) = V π,M (s)
where the value function V π T ,MT is defined with respect to RT .
Proof. Let sT0 = T s, . . . , be the sequence of states visited by policy πT on model MT starting from s. We have that sT0 = T s = T s0. We prove by induction that sTt = T st. Assume this is true for some value t, then we prove that st+1T = T st+1 holds:
sTt+1 = M T (sTt , π T (sTt )) = M T (T st, πT (T st)) (by inductive hypothesis)
= TM(st, π(st)) (by defintion of MT , πT ) = T st+1
Thus we have RT (sTt , a T t ) = R(st, at). Therefore V πT ,MT (T s) = V π,M (s).
Proof of Proposition 4.4. We first show the invariant of G under deterministic models and policies. The same result applies to stochastic policies with slight modification. Let sT = T s. We consider the transformation applied to M and M? and the resulting G function
GT (sT , a) , |V π T ,MT (MT (sT , a))− V π T ,MT (M?,T (sT , a))|
Note that by Lemma C.1, we have that V π T ,MT (MT (sT , a)) = V π,M (T −1MT (sT , a)) = V π,M (M(s, a)). Similarly, V π T ,MT (M?,T (sT , a)) = V π,M (M?(s, a)). Therefore we obtain that
GT (sT , a) = G(s, a) (C.1)
By Lemma 4.3 and triangle inequality, we have that
1− γ γ ∣∣∣V π,M − V π,M̂ ∣∣∣ ≤ E S∼ρπ [∣∣∣Gπ,M̂ (S)∣∣∣] (triangle inequality) ≤ E S∼ρπref [∣∣∣Gπ,M̂ (S)∣∣∣]+ |ρπ − ρπref |1 ·max S
∣∣∣Gπ,M̂ (S)∣∣∣ (Holder inequality)
By Corollary E.7 we have that |ρπ − ρπref |1 ≤ γ1−γ ES∼ρπref [ KL(π(S), πref(S)) 1/2|S ]
= δγ1−γ . Combining this with the equation above, we complete the proof.

heading: C.2 PROOF OF PROOF OF PROPOSITION A.2
text: Proof of Proposition A.2. Let µ be the distribution of the initial state S0, and let P ′ and P be the state-to-state transition kernel under policy π and πref. Let Ḡ = (1 − γ) ∑∞ k=0 γ
kP k and Ḡ′ = (1 − γ) ∑∞ k=0 γ kP ′ k. Under these notations, we can re-write ρπref = Ḡµ and ρπ = Ḡ′µ. Moreover, we observe that βπref = ḠP Ḡµ.
Let δ1 = (1 − γ)−1χ2Ḡµ(P ′, P )1/2 and δ2 = (1 − γ)−1χ2ḠPḠµ(P ′, P )1/2 by the χ2 divergence between P ′ and P , measured with respect to distributions Ḡµ = ρπref and ḠP Ḡµ = βπref . By Lemma D.1, we have that the χ2-divergence between the states can be bounded by the χ2-divergence between the actions in the sense that:
χ2Ḡµ(P ′, P )1/2 = χ2ρπref (P ′, P )1/2 ≤ E S∼ρπref
[ χ2(π(·|S), πref(·|S)) ]1/2 χ2ḠPḠµ(P ′, P )1/2 = χ2βπref (P ′, P )1/2 ≤ E
S∼βπref
[ χ2(π(·|S), πref(·|S)) ]1/2 Therefore we obtain that δ1 ≤ (1− γ)−1δ, δ2 ≤ (1− γ)−1δ. Let f(s) = Gπ,M̂ (s). By Lemma D.4, we can control the difference between 〈ρπref , f〉 and 〈ρπ, f〉 by∣∣∣∣ E
S∼ρπref
[ Gπ,M̂ (S) ] − E S∼ρπ [ Gπ,M̂ (S) ]∣∣∣∣ = |〈ρπref , f〉 − 〈ρπ, f〉| ≤ δ1〈ḠP Ḡµ, f2〉1/2 + δ1δ1/22 ‖f‖∞ ≤ δ1ε2 + δ1δ1/22 εmax

heading: It follows that∣∣∣V π,M̂ − V π,M ∣∣∣ ≤ γ(1− γ)−1 ∣∣∣∣ E
text: S∼ρπ [ Gπ,M̂ (S) ]∣∣∣∣ (by Lemma 4.3) ≤ γ(1− γ)−1 (∣∣∣∣ E S∼ρπref [ Gπ,M̂ (S) ]∣∣∣∣+ ∣∣∣∣ E S∼ρπref [ Gπ,M̂ (S)− E S∼ρπ [ Gπ,M̂ (S)
]]∣∣∣∣) ≤ γ(1− γ)−1 ∣∣∣∣ E S∼ρπref [ Gπ,M̂ (S)
]∣∣∣∣+ γ(1− γ)−1δ1ε2 + γ(1− γ)−1δ1δ1/22 εmax ≤ (1− γ)−1ε1 + (1− γ)−2δε2 + (1− γ)−5/2δ3/2εmax

heading: C.3 PROOF OF PROPOSITION 4.1 AND 4.2
text: Proof of Proposition 4.1 and 4.2 . By definition of G and the Lipschitzness of V π,M̂ , we have that |Gπ,M̂ (s, a)| ≤ L|M̂(s, a)−M?(s, a)|. Then, by Lemma 4.3 and triangle inequality, we have that∣∣V π,M̂ − V π,M? ∣∣ = κ · ∣∣ E
S∼ρπ,M A∼π(·|S)
[ Gπ,M̂ (S,A) ] ∣∣ ≤ κ E S∼ρπ,M A∼π(·|S) [∣∣Gπ,M̂ (S,A)∣∣] ≤ κ E
S∼ρπ A∼π(·|S)
[ ‖M̂(S,A)−M?(S,A)‖ ] . (C.2)
Next we prove the main part of the proposition. Thus we proved Proposition 4.1. Note that for any distribution ρ and ρ′ and function f , we have ES∼ρ f(S) = ES∼ρ′ f(S)+〈ρ−ρ′, f〉 ≤ ES∼ρ′ f(S)+ ‖ρ− ρ′‖1‖f‖∞. Thus applying this inequality with f(S) = EA∼π(·|S) [ ‖M̂(S,A)−M?(S,A)‖ ] ,
we obtain that
E S∼ρπ
A∼π(·|S)
[ ‖M̂(S,A)−M?(S,A)‖ ] ≤ E
S∼ρπref A∼π(·|S)
[ ‖M̂(S,A)−M?(S,A)‖ ] + ‖ρπref − ρ‖1 max
S E
A∼π(·|S)
[ ‖M̂(S,A)−M?(S,A)‖ ]
≤ E S∼ρπref A∼π(·|S)
[ ‖M̂(S,A)−M?(S,A)‖ ] + 2δκB (C.3)
where the last inequality uses the inequalities (see Corollary E.7) that ‖ρπ − ρπref‖1 ≤ γ 1−γ ES∼ρπref [ KL(π(S), πref(S)) 1/2|S ]
= δκ and that ‖M̂(S,A) − M?(S,A)‖ ≤ 2B. Combining (C.3) and (C.2) we complete the proof of Proposition 4.2.
D χ2-DIVERGENCE BASED INEQUALITIES
Lemma D.1. Let S be a random variable over the domain S . Let π and π′ be two policies and and A ∼ π(· | S) and A′ ∼ π′(· | S). Let Y ∼ M(· | S,A) and Y ′ ∼ M(· | S,A′) be the random variables for the next states under two policies. Then,
E [ χ2(Y |S, Y ′|S) ] ≤ E [ χ2(A|S,A′|S) ] Proof. By definition, we have that Y |S = s,A = a has the same density as Y ′|S = s,A′ = a for any a and s. Therefore by Theorem E.4 (setting X,X ′, Y, Y ′ in Theorem E.4 by A|S = s, A′|S = s, Y |S = s, Y ′|S = s respectively), we have
χ2(Y |S = s, Y ′|S = s) ≤ χ2(A|S = s,A′|S = s)
Taking expectation over the randomness of S we complete the proof.

heading: D.1 PROPERTIES OF MARKOV PROCESSES
text: In this subsection, we consider bounded the difference of the distributions induced by two markov process starting from the same initial distributions µ. Let P, P ′ be two transition kernels. Let G = ∑∞ k=0 γ
kP k and Ḡ = (1− γ)G. Define G′ and Ḡ′ similarly. Therefore we have that Ḡµ is the discounted distribution of states visited by the markov process starting from distribution µ. In other words, if µ is the distribution of S0, and P is the transition kernel induced by some policy π, then Ḡµ = ρπ .
First of all, let ∆ = γ(P ′ − P ) and we note that with simple algebraic manipulation,
Ḡ′ − Ḡ = (1− γ)−1Ḡ′∆Ḡ (D.1)
Let f be some function. We will mostly interested in the difference between ES∼Ḡµ [f ] and ES∼Ḡ′µ [f ], which can be rewritten as 〈(Ḡ′ −G)µ, f〉. We will bound this quantity from above by some divergence measure between P ′ and P .
We start off with a simple lemma that controls the form 〈p− q, f〉 by the χ2 divergence between p and q. With this lemma we can reduce our problem of bounding 〈(Ḡ′ −G)µ, f〉 to characterizing the χ2 divergence between Ḡ′µ and Ḡµ.
Lemma D.2. Let p and q be probability distributions. Then we have
〈q − p, f〉2 ≤ χ2(q, p) · 〈p, f2〉
Proof. By Cauchy-Schwartz inequality, we have 〈q − p, f〉2 ≤ (∫ (q(x)− p(x))2
p(x) dx
)(∫ p(x)f(x)2 ) = χ2(q, p) · 〈p, f2〉
The following Lemma is a refinement of the lemma above. It deals with the distributions p and q with the special structure p = WP ′µ and q = WPµ.
Lemma D.3. Let W,P ′, P be transition kernels and µ be a distribution. Then, 〈W (P ′ − P )µ, f〉2 ≤ χ2µ(P ′, P )〈WPµ, f2〉
where χ2µ(P ′, P ) is a divergence between transitions defined in Definition E.3.
Proof. By Lemma D.2 with p = WPµ and q = WP ′µ, we conclude that
〈W (P ′ − P )µ, f〉2 ≤ χ2(q, p) · 〈p, f2〉 ≤ χ2(WP ′µ,WPµ)〈WPµ, f2〉 By Theorem E.4 and Theorem E.5 we have that χ2(WP ′µ,WPµ) ≤ χ2(P ′µ, Pµ) ≤ χ2µ(P ′, P ), plugging this into the equation above we complete the proof.
Now we are ready to state the main result of this subsection. Lemma D.4. Let Ḡ, Ḡ′, P ′, P, f as defined in the beginning of this section. Let δ1 = (1 − γ)−1χ2
Ḡµ (P ′, P )1/2 and δ2 = (1− γ)−1χ2ḠPḠµ(P ′, P )1/2. Then,∣∣〈Ḡ′µ, f〉 − 〈Ḡµ, f〉∣∣ ≤ δ1‖f‖∞ (D.2)∣∣〈Ḡ′µ, f〉 − 〈Ḡµ, f〉∣∣ ≤ δ1〈ḠP Ḡµ, f2〉1/2 + δ1δ1/22 ‖f‖∞ Proof. Recall by equation (D.1), we have
〈(Ḡ′ − Ḡ)µ, f〉 = (1− γ)−1〈Ḡ′∆Ḡµ, f〉 (D.3) By Lemma D.3,
〈Ḡ′∆Ḡµ, f〉 ≤ χ2Ḡµ(P ′, P )1/2〈Ḡ′PḠµ, f2〉1/2 (D.4)
By Holder inequality and the fact that ‖Ḡ‖1→1 = 1, ‖Ḡ′‖1→1 = 1 and ‖P‖1→1 = 1, we have
〈Ḡ′∆Ḡµ, f〉 ≤ χ2Ḡµ(P ′, P )1/2〈Ḡ′PḠµ, f2〉1/2
≤ χ2Ḡµ(P ′, P )1/2‖Ḡ′PḠµ‖1/21 ‖f2‖1/2∞
≤ χ2Ḡµ(P ′, P )1/2‖f‖∞ (by ‖Ḡ′PḠµ‖1 ≤ ‖Ḡ′‖1→1‖P‖1→1‖Ḡ‖1→1‖µ‖1 ≤ 1)
≤ (1− γ)δ1‖f‖∞ (D.5)
Combining equation (D.3) and (D.5) we complete the proof of equation (D.2).
Next we bound 〈Ḡ′PḠµ, f2〉1/2 in a more refined manner. By equation (D.1), we have 〈Ḡ′PḠµ, f2〉1/2 = ( 〈ḠP Ḡµ, f2〉+ 1
1− γ 〈Ḡ′∆ḠP Ḡµ, f2〉 )1/2 ≤ 〈ḠP Ḡµ, f2〉1/2 + (1− γ)−1/2〈Ḡ′∆ḠP Ḡµ, f2〉1/2 (D.6)
By Lemma D.3 again, we have that
〈Ḡ′∆ḠP Ḡ, f2〉2 ≤ χ2ḠPḠµ(P ′, P )〈Ḡ′PḠPḠµ, f4〉 (D.7)
By Holder inequality and the fact that ‖Ḡ‖1→1 = 1, ‖Ḡ′‖1→1 = 1 and ‖P‖1→1 = 1, we have 〈Ḡ′PḠPḠµ, f4〉 ≤ ‖Ḡ′PḠPḠµ‖1‖f4‖∞ ≤ ‖f‖4∞ (D.8)
Combining equation (D.6), (D.8) gives
(1− γ)−1/2〈Ḡ′∆ḠP Ḡ, f2〉1/2 ≤ ((1− γ)−1χ2ḠPḠµ(P ′, P )1/2)1/2‖f‖∞ = δ1/22 ‖f‖∞ (D.9)
Then, combining equation (D.3), (D.4), (D.9), we have
〈(Ḡ′ − Ḡ)µ, f〉 = (1− γ)−1χ2Ḡµ(P ′, P )1/2〈Ḡ′PḠµ, f2〉1/2 (by equation (D.3), (D.4))
= δ1〈Ḡ′PḠµ, f2〉1/2
≤ δ1〈ḠP Ḡµ, f2〉1/2 + δ1(1− γ)−1/2〈Ḡ′∆ḠP Ḡµ, f2〉1/2 (by equation (D.6))
≤ δ1〈ḠP Ḡµ, f2〉1/2 + δ1δ1/22 ‖f‖∞ (by equation (D.9))
The following Lemma is a stronger extension of Lemma D.4, which can be used to future improve Proposition A.2, and may be of other potential independent interests. We state it for completeness. Lemma D.5. Let Ḡ, Ḡ′, P ′, P, f as defined in the beginning of this section. Let dk = (ḠP )kḠµ and δk = (1− γ)−1χ2dk−1(P ′, P )1/2, then we have that for any K,
∣∣〈Ḡ′µ, f〉 − 〈Ḡµ, f〉∣∣ ≤ δ1〈d1, f2〉−1/2 + δ1δ1/22 〈d2, f4〉−1/4 + δ1 . . . δ 2−K+1 K 〈dk, f2 K 〉2 −K + δ1 . . . δ 2−K+1 K ‖f‖∞
Proof. We first use induction to prove that:
|〈(Ḡ′ − Ḡ)µ, f〉| ≤ K∑ k=1  ∏ 0≤s≤k−1 δ2 −s s+1  〈dk, f2k〉2−k
+  ∏ 0≤s≤K−1 δ2 −s s+1  〈Ḡ′∆(ḠP )KḠµ, f2K 〉2−K (D.10) By the first equation of Lemma D.4, we got the case for K = 1. Assuming we have proved the case for K, then applying
〈Ḡ′∆(ḠP )KḠµ, f2 K 〉 = 〈Ḡ∆(ḠP )KḠµ, f2 K 〉+ (1− γ)−1〈Ḡ∆(ḠP )K+1Ḡµ, f2 K
〉 (D.11)
≤ 〈Ḡ∆(ḠP )KḠµ, f2 K 〉
+ (1− γ)−1χ2dK (P ′, P )1/2〈Ḡ′∆(ḠP )K+1Ḡµ, f2
K+1
〉1/2
≤ 〈Ḡ∆(ḠP )KḠµ, f2 K 〉+ δK+1〈Ḡ′∆(ḠP )K+1Ḡµ, f2 K+1 〉1/2
By Cauchy-Schwartz inequality, we obtain that
〈Ḡ′∆(ḠP )KḠµ, f2 K 〉2 −K ≤ 〈Ḡ∆(ḠP )KḠµ, f2 K 〉2 −K +
δK+1〈Ḡ′∆(ḠP )K+1Ḡµ, f2 K+1 〉2 −K−1
Plugging the equation above into equation (D.10), we provide the induction hypothesis for the case with K + 1.
Now applying 〈Ḡ′∆(ḠP )KḠµ, f2K 〉2−K ≤ ‖f‖∞ with equation (D.10) we complete the proof.

heading: E TOOLBOX
text: Definition E.1 (χ2 distance, c.f. Nielsen & Nock (2014); Cover & Thomas (2012)). The Neyman χ2 distance between two distributions p and q is defined as
χ2(p, q) , ∫ (p(x)− q(x))2
q(x) dx =
∫ p(x)2
q(x) dx− 1
For notational simplicity, suppose two random variables X and Y has distributions pX and pY , we often write χ2(X,Y ) as a simplification for χ2(pX , pY ). Theorem E.2 (Sason & Verdú (2016)). The Kullback-Leibler (KL) divergence between two distributions p, q is bounded from above by the χ2 distance:
KL(p, q) ≤ χ2(p, q)
Proof. Since log is a concave function, by Jensen inequality we have
KL(p, q) = ∫ p(x) log p(x)
q(x) dx ≤ log
∫ p(x) · p(x)
q(x) dx
= log(χ2(p, q) + 1) ≤ χ2(p, q)
Definition E.3 (χ2 distance between transitions). Given two transition kernels P, P ′. For any distribution µ, we define χ2µ(P ′, P ) as:
χ2µ(P ′, P ) ,
∫ µ(x)χ2(P ′(·|X = x), P (·|X = x))dx
Theorem E.4. Suppose random variables (X,Y ) and (X ′, Y ′) satisfy that pY |X = pY ′|X′ . Then
χ2(Y, Y ′) ≤ χ2(X,X ′) Or equivalently, for any transition kernel P and distribution µ, µ′, we have
χ2(Pµ, Pµ′) ≤ χ2(µ, µ′)
Proof. Denote pY |X(y | x) = pY ′|X′(y | x) by p(y | x), and we rewrite pX as p and pX′ as p′. By Cauchy-Schwarz inequality, we have:
pY (y) 2 = (∫ p(y|x)p(x)dx )2 ≤ (∫ p(y|x)p′(x)dx )(∫ p(y|x)p(x) 2
p′(x) dx ) = pY ′(y) (∫ p(y|x)p(x) 2
p′(x) dx
) (E.1)
It follows that
χ2(Y, Y ′) =
∫ pY (y) 2
pY ′(y) dy − 1 ≤
∫ dy ∫ p(y|x)p(x) 2
p′(x) dx− 1 = χ2(X,X ′)
Theorem E.5. Let X,Y, Y ′ are three random variables. Then, χ2(Y, Y ′) ≤ E [ χ2(Y |X,Y ′|X) ] We note that the expectation on the right hand side is over the randomness of X .11 As a direct corollary, we have for transition kernel P ′ and P and distribution µ,
χ2(P ′µ, Pµ) ≤ χ2µ(P ′, P )
Proof. We denote pY ′|X(y|x) by p′(y | x) and pY |X(y|x) by p(y|x), and let p(x) be a simplification for pX(x). We have by Cauchy-Schwarz,
pY (y) 2 pY ′(y) =
(∫ p(y|x)p(x)dx )2∫ p′(y | x)p(x)dx ≤ ∫ p(y|x)2 p′(y|x) p(x)dx
It follows that
χ2(Y, Y ′) =
∫ pY (y) 2
pY ′(y) dy − 1 ≤
∫ p(y|x)2
p′(y|x) p(x)dxdy − 1 = E
[ χ2(Y |X,Y ′|X) | X ]
Claim E.6. Let µ be a distribution over the state space S. Let P and P ′ be two transition kernels. G = ∑∞ k=0(γP ) k = (Id − γP )−1 and G′ = ∑∞ k=0(γP
′)k = (Id − γP ′)−1. Let d = (1 − γ)Gµ and d′ = (1− γ)G′µ be the discounted distribution starting from µ induced by the transition kernels G and G′. Then,
|d− d′|1 ≤ 1
1− γ |∆d|1
Moreover, let γ(P ′ − P ) = ∆. Then, we have
G′ −G = ∞∑ k=1 (G∆)kG
11Observe χ2(Y |X,Y ′|X) deterministically depends on X .
Proof. With algebraic manipulation, we obtain,
G′ −G = (Id− γP ′)−1((Id− γP )− (Id− γP ′)(Id− γP )−1
= G′∆G (E.2)
It follows that
|d− d′|1 = (1− γ)|G′∆Gµ|1 ≤ |∆Gµ|1 (since (1− γ)|G′|1→1 ≤ 1)
= 1
1− γ |∆d|1
Replacing G′ in the RHS of the equation (E.2) by G′ = G+G′∆G, and doing this recursively gives
G′ −G = ∞∑ k=1 (G∆)kG
Corollary E.7. Let π and π′ be two policies and let ρπ be defined as in Definition 2.1. Then,
|ρπ − ρ′|1 ≤ γ 1− γ ES∼ρπ [ KL(π(S), π′(S))1/2 | S ]
Proof. Let P and P ′ be the state-state transition matrix under policy π and π′ and ∆ = γ(P ′ − P ) By Claim E.6, we have that
|ρπ − ρπ ′ |1 ≤
1
1− γ |∆ρπ|1 =
γ 1− γ ES∼ρπ [ |pM?(S,π(S))|S − pM?(S,π′(S))|S |1 ] ≤ γ
1− γ ES∼ρπ [ |pπ(S)|S − pπ′(S)|S |1 ] ≤ γ 1− γ ES∼ρπ [ KL(π(S), π′(S))1/2 | S ] (by Pinkser’s inequality)
F IMPLEMENTATION DETAILS

heading: F.1 ENVIRONMENT SETUP
text: We benchmark our algorithm on six tasks based on physics simulator Mujoco (Todorov et al., 2012). We use rllab’s implementation (Duan et al., 2016) 12 to interact with Mujoco. All the environments we use have a maximum horizon of 500 steps. We remove all contact information from observation. To compute reward from states, we put the velocity of center of mass into the states.

heading: F.2 NETWORK ARCHITECTURE AND MODEL LEARNING
text: We use the same reward function as in rllab, except that all the coefficients Ccontact in front of the contact force s are set to 0 in our case. We refer the readers to (Duan et al., 2016) Supp Material 1.2 for more details. All actions are projected to the action space by clipping. We normalize all observations by s′ = s−µσ where µ, σ ∈ R
dobservation are computed from all observations we collect from the real environment. Note that µ, σ may change as we collect new data. Our policy will always produce an action a in [−1, 1]daction and the action a′, which is fed into the environment, is scaled linearly by a′ = 1−a2 amin + 1+a 2 amax, where amin, amax are the min or max values allowed at each entry.
12commit b3a2899 in https://github.com/rll/rllab/

heading: F.3 SLBO DETAILS
text: The dynamical model is represented by a feed-forward neural network with two hidden layers, each of which contains 500 hidden units. The activation function at each layer is ReLU. We use Adam to optimize the loss function with learning rate 10−3 and L2 regularization 10−5. The network does not predict the next state directly; instead, it predicts the normalized difference of st+1 − st. The normalization scheme and statistics are the same as those of observations: We maintain µ, σ from collected data in the real environment and may change them as we collect more, and the normalized difference is st+1−st−σµ .
The policy network is a feed-forward network with two hidden layers, each of which contains 32 hidden units. The policy network uses tanh as activation function and outputs a Gaussian distribution N (µ(s), σ2) where σ a state-independent trainable vector. During our evaluation, we use H = 2 for multi-step model training and the batch size is given by 256 H = 128, i.e., we enforce the model to see 256 transitions at each batch.
We run our algorithm nouter = 100 iterations. We collect ntrain = 10000 steps of real samples from the environment at the start of each iteration using current policy with Ornstein-Uhlunbeck noise (with parameter θ = 0.15, σ = 0.3) for better exploration. At each iteration, we optimize dynamics model and policy alternatively for ninner = 20 times. At each iteration, we optimize dynamics model for nmodel = 100 times and optimize policy for npolicy = 40 times.

heading: F.4 BASELINES
text: TRPO. TRPO hyperparameters are listed at Table 1, which are the same as OpenAI Baselines’ implementation. These hyperparameters are fixed for all experiments where TRPO is used, including ours, MB-TRPO and MF-TRPO. We do not tune these hyperparameters. We also normalize observations as our algorithm and OpenAI Baselines do.
We use a neural network as the value function to reduce variance, which has 2 hidden layers of units 64 and uses tanh as activation functions. We use Generalized Advantage Estimator (GAE) Schulman et al. (2015b) to estimate advantages. Both TRPO used in our algorithm and that in model-free algorithm share the same set of hyperparameters.
SAC. For fair comparison, we do not use a large policy network (2 hidden layers, one of which has 256 hidden units) as the authors suggest, but use exactly the same policy network as ours. All other hyperparameters are kept the same as the authors’. Note that Q network and value network have 256 hidden units at each hidden layers, which is more than TRPO’s. We refer the readers to Haarnoja et al. (2018) Appendix D for more details.
MB-TRPO. Model-Based TRPO (MB-TRPO) is similar to our algorithm SLBO but does not optimize model and policy alternatively during one iteration. We do not tune the hyperparameter nmodel since any number beyond a certain threshold would bring similar results. For npolicy we try {100, 200, 400, 800} on Ant and find npolicy = 200 works best in Ant so we use it for all other environments. Note that when Algo 2 uses 800 Adam updates (per outer iteration), it has the same amount of updates (per outer iteration) as in Algo 3. As suggested by Section 6.1, we use 0.005 as the coefficient of entropy bonus for all experiments.
Algorithm 3 Model-Based Trust Region Policy Optimization (MB-TRPO) 1: initialize model network parameters φ and policy network parameters θ 2: initialize dataset D ← ∅ 3: for nouter iterations do 4: D ← D ∪ { collect ncollect samples from real environment using πθ with noises } 5: for nmodel iterations do 6: optimize (6.1) over φ with sampled data from D by one step of Adam 7: for npolicy iterations do 8: D′ ← { collect ntrpo samples using M̂φ as dynamics } 9: optimize πθ by running TRPO on D′
SLBO. We tune multi-step model training parameter H ∈ {1, 2, 4, 8}, entropy regularization coefficient λ ∈ {0, 0.001, 0.003, 0.005} and npolicy ∈ {10, 20, 40} on Ant and find H = 2, λ = 0.005, npolicy = 40 work best, then we fix them in all environments, though environment-specific hyperparameters may work better. The other hyperparameters, including ninner, nmodel and network architecture, are never tuned. We observe that at the first several iterations, the policy overfits to the learnt model so a reduction of npolicy at the beginning can further speed up convergence but we omit this for simplicity.
The most important hyperparameters we found are npolicy and the coefficient in front of the entropy regularizer λ. It seems that once nmodel is large enough we don’t see any significant changes. We did have a held-out set for model prediction (with the same distribution as the training set) and found out the model doesn’t overfit much. As mentioned in F.3, we also found out normalizing the state helped a lot since the raw entries in the state have different magnitudes; if we don’t normalize them, the loss will be dominated by the loss of some large entries.

heading: F.5 ABLATION STUDY
text: Multi-step model training. We compare multi-step model training with single-step model training and the results are shown on Figure 2. Note that H = 1 means we use single-step model training. We observe that small H (e.g., 2 or 4) can be beneficial, but larger H (e.g., 8) can hurt. We hypothesize that smaller H can help the model learn the uncertainty in the input and address the error-propagation issue to some extent. Pathak et al. (2018) uses an auto-regressive recurrent model to predict a multi-step loss on a trajectory, which is closely related to ours. However, theirs differs from ours in the sense that they do not use the predicted output xt+1 as the input for the prediction of xt+2, and so on and so forth.
Entropy regularization. Figure 3 shows that entropy reguarization can improve both sample efficiency and final performance. More entropy regularization leads to better sample efficiency and higher total rewards. We observe that in the late iterations of training, entropy regularization may hurt the performance thus we stop using entropy regularization in the second half of training.
SLBO with 4M training steps. Figure 4 shows that SLBO is superior to SAC and MF-TRPO in Swimmer, Half Cheetah, Walker and Humanoid when 4 million samples or fewer samples are allowed. For Ant environment , although SLBO with less than one million samples reaches the performance of MF-TRPO with 8 million samples, SAC’s performance surpasses SLBO after 2 million steps of training. Since model-free TRPO almost stops improving after 8M steps and our algorithms uses TRPO for optimizing the estimated environment, we don’t expect SLBO can significantly outperform the reward of TRPO at 8M steps. The result shows that SLBO is also satisfactory in terms of asymptotic convergence (compared to TRPO.) It also indicates a better planner or policy optimizer instead of TRPO might be necessary to further improve the performance.

heading: G SAMPLE COMPLEXITY BOUNDS
text: In this section, we extend Theorem 3.1 to a final sample complexity result. For simplicity, let Lπ,Mπref,δ = V π,M −Dπk,δ(M,π) be the lower bound of V π,M ?
. We omit the subscript δ when it’s clear from contexts. When D satisfies (R1), we have that,
V π,M ? ≥ Lπ,Mπref,δ ∀π s.t. d(π, πref) ≤ δ (G.1)
When D satisfies (R3), we use L̂π,Mπref,δ to denotes its empirical estimates. Namely, we replace the expectation in equation (R3) by empirical samples τ (1), . . . , τ (n). In other words, we optimize
πk+1,Mk+1 = argmax π∈Π, M∈M
L̂π,Mπref,δ = V π,M − 1
n n∑ i=1 f(M̂, π, τ (i)) (G.2)
instead of equation (3.3).
Let p be the total number of parameters in the policy and model parameterization. We assume that we have a discrepancy bound Dπref(π,M) satisfying (R3) with a function f that is bounded with [−Bf , Bf ] and that is Lf -Lipschitz in the parameters of π and M . That is, suppose π is parameterized by θ andM is parameterized by φ, then we require |f(Mφ, φθ, τ)−f(Mφ′ , φθ′ , τ)| ≤ Lf (‖φ − φ′‖22 + ‖θ − θ′‖2) for all τ , θ, θ′, φ, φ′. We note that Lf is likely to be exponential in dimension due to the recursive nature of the problem, but our bounds only depends on its logarithm. We also restrict our attention to parameters in an Euclidean ball {θ : ‖θ‖2 ≤ B} and {φ : ‖φ‖2 ≤ B}. Our bounds will be logarithmic in B.
We need the following definition of approximate local maximum since with sampling error we cannot hope to converge to the exact local maximum.
Definition G.1. We say π is a (δ, ε)-local maximum of V π,M ?
with respect to the constraint set Π and metric d, if for any π′ ∈ Π with d(π, π′) ≤ δ, we have V π,M? ≥ V π′,M? − ε.
We show a sample complexity bounds that scales linearly in p and logarithmically in Lf , B and Bf . Theorem G.2. Let ε > 0. In the setting of Theorem 3.1, under the additional assumptions above, suppose we use n = O(Bfp log(BLf/ε)/ε2) trajectories to estimate the discrepancy bound in Algorithm 1. Then, for any t, if πt is not a (δ, ε)-local maximum, then the total reward will increase in the next step: with high probability,
V πt+1,M ? ≥ V πt,M ? + ε/2 (G.3)

heading: As a direct consequence, suppose the maximum possible total reward is BR and the initial total
text: reward is 0, then for some T = O(BR/ε), we have that πT is a (δ, ε)-local maximum of the V π,M ? .
Proof. By Hoeffiding’s inequality, we have for fix π and M̂ , with probability 1 − nO(1) over the randomness of τ (1), . . . , τ (n),∣∣∣∣∣ 1n n∑ i=1 f(M̂, π, τ (i))− E τ∼πref,M? [f(M̂, π, τ)] ∣∣∣∣∣ ≤ 4 √ Bf log n n . (G.4)
In more succinct notations, we have |D̂πk,δ(M,π)−Dπk,δ(M,π)| ≤ 4 √ Bf logn n , and therefore
|L̂π,M − Lπ,M | ≤ 4 √ Bf log n
n . (G.5)
By a standard ε-cover + union bound argument, we can prove the uniform convergence: with high probability (at least 1 − nO(1)) over the choice of τ (1), . . . , τ (n), for all policy and model, for all policy π and dynamics M ,
|L̂π,M − Lπ,M | ≤ 4 √ Bfp log(nBLf )
n = ε/4. (G.6)
Suppose at iteration t, we are at policy πt which is not a (δ, ε)-local maximum of V π,M ?
. Then, there exists π′ such that d(π′, πt) ≤ δ and
V π ′,M? ≥ V πt,M ? + ε. (G.7)
Then, we have that
V πt+1,M ?
≥ Lπt+1,Mt+1πt (by equation (G.1))
≥ L̂πt+1,Mt+1πt − ε/4 (by uniform convergence, equation (G.6)) ≥ L̂π ′,M?
πt − ε/4 (by the definition of πt+1,Mt+1)
≥ Lπ ′,M?
πt − ε/2 (by uniform convergence, equation (G.6)) = V π ′,M? − ε/2 (by equation (R2))
= V πt,M ? + ε/2 (by equation (G.7))
Note that the total reward can only improve by ε/2 for at most O(BR/ε) steps. Therefore, in the first O(BR/ε) iterations, we must have hit a solution that is a (δ, ε)-local maximum. This completes the proof.

Review summaries:
This paper proposed a new class of meta-algorithm for reinforcement learning and proved the monotone improvement for a local maximum of the expected reward, which could be used in deep RL setting. The framework seems to be quite general but does not include any specific example, like what non-linear dynamical model in detail could be included and will this framework cover the classical MDP setting? In theory, the dynamical model needs to satisfy L-Lipschitz. So which dynamical model in reality could satisfy this assumption? It seems that the focus of this paper is theoretical side. But the only guarantee is the non-decreasing value function of the policy. In RL, people may be more care about the regret or sample complexity. Previous model-based work with simpler model already can have such strong guarantees, such as linear dynamic (Y. Abbasi-Yadkori and Cs. Szepesvari (2011)), MDP (Agrawal and Jia (2017)). What kind of new insights will this framework give when the model reduces to simpler one (linear model)?