Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees

1 INTRODUCTION . In recent years deep reinforcement learning has achieved strong empirical success , including superhuman performances on Atari games and Go ( Mnih et al. , 2015 ; Silver et al. , 2017 ) and learning locomotion and manipulation skills in robotics ( Levine et al. , 2016 ; Schulman et al. , 2015b ; Lillicrap et al. , 2015 ) . Many of these results are achieved by model-free RL algorithms that often require a massive number of samples , and therefore their applications are mostly limited to simulated environments . Model-based deep reinforcement learning , in contrast , exploits the information from state observations explicitly — by planning with an estimated dynamical model — and is considered to be a promising approach to reduce the sample complexity . Indeed , empirical results ( Deisenroth & Rasmussen , 2011b ; Deisenroth et al. , 2013 ; Levine et al. , 2016 ; Nagabandi et al. , 2017 ; Kurutach et al. , 2018 ; Pong et al. , 2018a ) have shown strong improvements in sample efficiency . Despite promising empirical findings , many of theoretical properties of model-based deep reinforcement learning are not well-understood . For example , how does the error of the estimated model affect the estimation of the value function and the planning ? Can model-based RL algorithms be guaranteed to improve the policy monotonically and converge to a local maximum of the value function ? How do we quantify the uncertainty in the dynamical models ? It ’ s challenging to address these questions theoretically in the context of deep RL with continuous state and action space and non-linear dynamical models . Due to the high-dimensionality , learning models from observations in one part of the state space and extrapolating to another part sometimes 0 * indicates equal contribution 1The source code of this work is available at https : //github.com/roosephu/slbo involves a leap of faith . The uncertainty quantification of the non-linear parameterized dynamical models is difficult — even without the RL components , it is an active but widely-open research area . Prior work in model-based RL mostly quantifies uncertainty with either heuristics or simpler models ( Moldovan et al. , 2015 ; Xie et al. , 2016 ; Deisenroth & Rasmussen , 2011a ) . Previous theoretical work on model-based RL mostly focuses on either the finite-state MDPs ( Jaksch et al. , 2010 ; Bartlett & Tewari , 2009 ; Fruit et al. , 2018 ; Lakshmanan et al. , 2015 ; Hinderer , 2005 ; Pirotta et al. , 2015 ; 2013 ) , or the linear parametrization of the dynamics , policy , or value function ( Abbasi-Yadkori & Szepesvári , 2011 ; Simchowitz et al. , 2018 ; Dean et al. , 2017 ; Sutton et al. , 2012 ; Tamar et al. , 2012 ) , but not much on non-linear models . Even with an oracle prediction intervals2 or posterior estimation , to the best of our knowledge , there was no previous algorithm with convergence guarantees for model-based deep RL . Towards addressing these challenges , the main contribution of this paper is to propose a novel algorithmic framework for model-based deep RL with theoretical guarantees . Our meta-algorithm ( Algorithm 1 ) extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires no explicit uncertainty quantification of the dynamical models . Let V π be the value function V π of a policy π on the true environment , and let V̂ π be the value function of the policy π on the estimated model M̂ . We design provable upper bounds , denoted by Dπ , M̂ , on how much the error can compound and divert the expected value V̂ π of the imaginary rollouts from their real value V π , in a neighborhood of some reference policy . Such upper bounds capture the intrinsic difference between the estimated and real dynamical model with respect to the particular reward function under consideration . The discrepancy bounds Dπ , M̂ naturally leads to a lower bound for the true value function : V π ≥ V̂ π −Dπ , M̂ . ( 1.1 ) Our algorithm iteratively collects batches of samples from the interactions with environments , builds the lower bound above , and then maximizes it over both the dynamical model M̂ and the policy π . We can use any RL algorithms to optimize the lower bounds , because it will be designed to only depend on the sample trajectories from a fixed reference policy ( as opposed to requiring new interactions with the policy iterate . ) We show that the performance of the policy is guaranteed to monotonically increase , assuming the optimization within each iteration succeeds ( see Theorem 3.1 . ) To the best of our knowledge , this is the first theoretical guarantee of monotone improvement for model-based deep RL . Readers may have realized that optimizing a robust lower bound is reminiscent of robust control and robust optimization . The distinction is that we optimistically and iteratively maximize the RHS of ( 1.1 ) jointly over the model and the policy . The iterative approach allows the algorithms to collect higher quality trajectory adaptively , and the optimism in model optimization encourages explorations of the parts of space that are not covered by the current discrepancy bounds . To instantiate the meta-algorithm , we design a few valid discrepancy bounds in Section 4 . In Section 4.1 , we recover the norm-based model loss by imposing the additional assumption of a Lipschitz value function . The result suggests a norm is preferred compared to the square of the norm . Indeed in Section 6.2 , we show that experimentally learning with ` 2 loss significantly outperforms the mean-squared error loss ( ` 22 ) . In Section 4.2 , we design a discrepancy bound that is invariant to the representation of the state space . Here we measure the loss of the model by the difference between the value of the predicted next state and the value of the true next state . Such a loss function is shown to be invariant to one-to-one transformation of the state space . Thus we argue that the loss is an intrinsic measure for the model error without any information beyond observing the rewards . We also refine our bounds in Section A by utilizing some mathematical tools of measuring the difference between policies in χ2-divergence ( instead of KL divergence or TV distance ) . 2We note that the confidence interval of parameters are likely meaningless for over-parameterized neural networks models . Our analysis also sheds light on the comparison between model-based RL and on-policy model-free RL algorithms such as policy gradient or TRPO ( Schulman et al. , 2015a ) . The RHS of equation ( 1.1 ) is likely to be a good approximator of V π in a larger neighborhood than the linear approximation of V π used in policy gradient is ( see Remark 4.5 . ) Finally , inspired by our framework and analysis , we design a variant of model-based RL algorithms Stochastic Lower Bounds Optimization ( SLBO ) . Experiments demonstrate that SLBO achieves state-of-the-art performance when only 1M samples are permitted on a range of continuous control benchmark tasks . 2 NOTATIONS AND PRELIMINARIES . We denote the state space by S , the action space by A . A policy π ( ·|s ) specifies the conditional distribution over the action space given a state s. A dynamical model M ( ·|s , a ) specifies the conditional distribution of the next state given the current state s and action a . We will use M ? globally to denote the unknown true dynamical model . Our target applications are problems with the continuous state and action space , although the results apply to discrete state or action space as well . When the model is deterministic , M ( ·|s , a ) is a dirac measure . In this case , we use M ( s , a ) to denote the unique value of s′ and view M as a function from S ×A to S . LetM denote a ( parameterized ) family of models that we are interested in , and Π denote a ( parameterized ) family of policies . Unless otherwise stated , for random variable X , we will use pX to denote its density function . Let S0 be the random variable for the initial state . Let S π , M t to denote the random variable of the states at steps t when we execute the policy π on the dynamic model M stating with S0 . Note that Sπ , M0 = S0 unless otherwise stated . We will omit the subscript when it ’ s clear from the context . We use At to denote the actions at step t similarly . We often use τ to denote the random variable for the trajectory ( S0 , A1 , . . . , St , At , . . . ) . Let R ( s , a ) be the reward function at each step . We assume R is known throughout the paper , although R can be also considered as part of the model if unknown . Let γ be the discount factor . Let V π , M be the value function on the model M and policy π defined as : V π , M ( s ) = E ∀t≥0 , At∼π ( ·|St ) St+1∼M ( ·|St , At ) [ ∞∑ t=0 γtR ( St , At ) | S0 = s ] ( 2.1 ) We define V π , M = E [ V π , M ( S0 ) ] as the expected reward-to-go at Step 0 ( averaged over the random initial states ) . Our goal is to maximize the reward-to-go on the true dynamical model , that is , V π , M ? , over the policy π . For simplicity , throughout the paper , we set κ = γ ( 1 − γ ) −1 since it occurs frequently in our equations . Every policy π induces a distribution of states visited by policy π : Definition 2.1 . For a policy π , define ρπ , M as the discounted distribution of the states visited by π on M . Let ρπ be a shorthand for ρπ , M ? and we omit the superscript M ? throughout the paper . Concretely , we have ρπ = ( 1− γ ) ∑∞ t=0 γ t · pSπt 3 ALGORITHMIC FRAMEWORK . As mentioned in the introduction , towards optimizing V π , M ? ,3 our plan is to build a lower bound for V π , M ? of the following type and optimize it iteratively : V π , M ? ≥ V π , M̂ −D ( M̂ , π ) ( 3.1 ) where D ( M̂ , π ) ∈ R≥0 bounds from above the discrepancy between V π , M̂ and V π , M ? . Building such an optimizable discrepancy bound globally that holds for all M̂ and π turns out to be rather difficult , if not impossible . Instead , we shoot for establishing such a bound over the neighborhood of a reference policy πref . V π , M ? ≥ V π , M̂ −Dπref , δ ( M̂ , π ) , ∀π s.t . d ( π , πref ) ≤ δ ( R1 ) 3Note that in the introduction we used V π for simplicity , and in the rest of the paper we will make the dependency on M ? explicit . Here d ( · , · ) is a function that measures the closeness of two policies , which will be chosen later in alignment with the choice of D. We will mostly omit the subscript δ in D for simplicity in the rest of the paper . We will require our discrepancy bound to vanish when M̂ is an accurate model : M̂ = M ? =⇒ Dπref ( M̂ , π ) = 0 , ∀π , πref ( R2 ) The third requirement for the discrepancy bound D is that it can be estimated and optimized in the sense that Dπref ( M̂ , π ) is of the form E τ∼πref , M ? [ f ( M̂ , π , τ ) ] ( R3 ) where f is a known differentiable function . We can estimate such discrepancy bounds for every π in the neighborhood of πref by sampling empirical trajectories τ ( 1 ) , . . . , τ ( n ) from executing policy πref on the real environment M ? and compute the average of f ( M̂ , π , τ ( i ) ) ’ s . We would have to insist that the expectation can not be over the randomness of trajectories from π on M ? , because then we would have to re-sample trajectories for every possible π encountered . For example , assuming the dynamical models are all deterministic , one of the valid discrepancy bounds ( under some strong assumptions ) that will prove in Section 4 is a multiple of the error of the prediction of M̂ on the trajectories from πref : Dπref ( M̂ , π ) = L · E S0 , ... , St , ∼πref , M ? [ ‖M̂ ( St ) − St+1‖ ] ( 3.2 ) Suppose we can establish such an discrepancy bound D ( and the distance function d ) with properties ( R1 ) , ( R2 ) , and ( R3 ) , — which will be the main focus of Section 4 — , then we can devise the following meta-algorithm ( Algorithm 1 ) . We iteratively optimize the lower bound over the policy πk+1 and the model Mk+1 , subject to the constraint that the policy is not very far from the reference policy πk obtained in the previous iteration . For simplicity , we only state the population version with the exact computation of Dπref ( M̂ , π ) , though empirically it is estimated by sampling trajectories . Algorithm 1 Meta-Algorithm for Model-based RL Inputs : Initial policy π0 . Discrepancy bound D and distance function d that satisfy equation ( R1 ) and ( R2 ) . For k = 0 to T : πk+1 , Mk+1 = argmax π∈Π , M∈M V π , M −Dπk , δ ( M , π ) ( 3.3 ) s.t . d ( π , πk ) ≤ δ ( 3.4 ) We first remark that the discrepancy bound Dπk ( M , π ) in the objective plays the role of learning the dynamical model by ensuring the model to fit to the sampled trajectories . For example , using the discrepancy bound in the form of equation ( 3.2 ) , we roughly recover the standard objective for model learning , with the caveat that we only have the norm instead of the square of the norm in MSE . Such distinction turns out to be empirically important for better performance ( see Section 6.2 ) . Second , our algorithm can be viewed as an extension of the optimism-in-face-of-uncertainty ( OFU ) principle to non-linear parameterized setting : jointly optimizing M and π encourages the algorithm to choose the most optimistic model among those that can be used to accurately estimate the value function . ( See ( Jaksch et al. , 2010 ; Bartlett & Tewari , 2009 ; Fruit et al. , 2018 ; Lakshmanan et al. , 2015 ; Pirotta et al. , 2015 ; 2013 ) and references therein for the OFU principle in finite-state MDPs . ) The main novelty here is to optimize the lower bound directly , without explicitly building any confidence intervals , which turns out to be challenging in deep learning . In other words , the uncertainty is measured straightforwardly by how the error would affect the estimation of the value function . Thirdly , the maximization of V π , M , whenM is fixed , can be solved by any model-free RL algorithms with M as the environment without querying any real samples . Optimizing V π , M jointly over π , M can be also viewed as another RL problem with an extended actions space using the known “ extended MDP technique ” . See ( Jaksch et al. , 2010 , section 3.1 ) for details . Our main theorem shows formally that the policy performance in the real environment is nondecreasing under the assumption that the real dynamics belongs to our parameterized familyM.4 Theorem 3.1 . Suppose that M ? ∈ M , that D and d satisfy equation ( R1 ) and ( R2 ) , and the optimization problem in equation ( 3.3 ) is solvable at each iteration . Then , Algorithm 1 produces a sequence of policies π0 , . . . , πT with monotonically increasing values : V π0 , M ? ≤ V π1 , M ? ≤ · · · ≤ V πT , M ? ( 3.5 ) Moreover , as k →∞ , the value V πk , M ? converges to some V π̄ , M ? , where π̄ is a local maximum of V π , M ? in domain Π . The theorem above can also be extended to a finite sample complexity result with standard concentration inequalities . We show in Theorem G.2 that we can obtain an approximate local maximum in O ( 1/ε ) iterations with sample complexity ( in the number of trajectories ) that is polynomial in dimension and accuracy ε and is logarithmic in certain smoothness parameters . Proof of Theorem 3.1 . Since D and d satisfy equation ( R1 ) , we have that V πk+1 , M ? ≥ V πk+1 , Mk+1 −Dπk ( Mk+1 , πk+1 ) By the definition that πk+1 and Mk+1 are the optimizers of equation ( 3.3 ) , we have that V πk+1 , Mk+1 −Dπk ( Mk+1 , πk+1 ) ≥ V πk , M ? −Dπk ( M ? , πk ) = V πk , M ? ( by equation R2 ) Combing the two equations above we complete the proof of equation ( 3.5 ) . For the second part of the theorem , by compactness , we have that a subsequence of πk converges to some π̄ . By the monotonicity we have V πk , M ? ≤ V π̄ , M ? for every k ≥ 0 . For the sake of contradiction , we assume π̄ is a not a local maximum , then in the neighborhood of π̄ there exists π′ such that V π ′ , M ? > V π̄ , M ? and d ( π̄ , π′ ) < δ/2 . Let t be such that πt is in the δ/2-neighborhood of π̄ . Then we see that ( π′ , M ? ) is a better solution than ( πt+1 , Mt+1 ) for the optimization problem ( 3.3 ) in iteration t because V π ′ , M ? > V π̄ , M ? ≥ V πt+1 , M ? ≥ V πt+1 , Mt+1 −Dπt ( Mt+1 , πt+1 ) . ( Here the last inequality uses equation ( R1 ) with πt as πref . ) The fact ( π′ , M ? ) is a strictly better solution than ( πt+1 , Mt+1 ) contradicts the fact that ( πt+1 , Mt+1 ) is defined to be the optimal solution of ( 3.3 ) . Therefore π̄ is a local maximum and we complete the proof .

Summary: This paper proposed a new class of meta-algorithm for reinforcement learning and proved the monotone improvement for a local maximum of the expected reward, which could be used in deep RL setting. The framework seems to be quite general but does not include any specific example, like what non-linear dynamical model in detail could be included and will this framework cover the classical MDP setting? In theory, the dynamical model needs to satisfy L-Lipschitz. So which dynamical model in reality could satisfy this assumption? It seems that the focus of this paper is theoretical side. But the only guarantee is the non-decreasing value function of the policy. In RL, people may be more care about the regret or sample complexity. Previous model-based work with simpler model already can have such strong guarantees, such as linear dynamic (Y. Abbasi-Yadkori and Cs. Szepesvari (2011)), MDP (Agrawal and Jia (2017)). What kind of new insights will this framework give when the model reduces to simpler one (linear model)?