Deep Nonlinear Stochastic Optimal Control for Systems with Multiplicative Uncertainties

1 INTRODUCTION . Stochastic Optimal Control ( SOC ) is the center of decision making under uncertainty with a long history and extensive prior work both in terms of theory as well as algorithms ( Stengel , 1994 ; Fleming & Soner , 2006 ) . One of the most celebrated formulations of stochastic optimal control is for linear dynamics and additive noise , known as the Linear Quadratic Gaussian ( LQG ) case ( Stengel , 1994 ) . For stochastic systems that are nonlinear in the state and affine in control , the stochastic optimal control formulation results in the Hamilton-Jacobi-Bellman ( HJB ) equation , which is a backward nonlinear Partial Differential Equation ( PDE ) . Solving the HJB PDE for high dimensional systems is generally a challenging task and suffers from the curse of dimensionality . Different algorithms have been derived to address stochastic optimal control problems and solve the HJB equation . These algorithms can be mostly classified into two groups : ( i ) algorithms that rely on linearization and ( ii ) algorithms that rely on sampling . Linearization-based algorithms rely on first order Taylor ’ s approximation of dynamics ( iterative LQG or iLQG ) or quadratic approximation of dynamics ( Stochastic Differential Dynamic Programming ) , and quadratic approximation of the cost function ( Todorov & Li , 2005b ; Theodorou et al. , 2010b ) . Application of the aforementioned algorithms is not straightforward and requires very small time discretization or special linearization schemes especially for the cases of control and/or state dependent noise . It is also worth mentioning that the convergence properties of these algorithms have not been investigated and remain open questions . Sampling-based methods include the Markov-Chain Monte Carlo ( MCMC ) approximation of the HJB equation ( Kushner & Dupuis , 1992 ; Huynh et al. , 2016 ) . MCMC-based algorithms rely on backward propagation of the value function on a pre-specified grid . Recently researchers have incorporated tensor-train decomposition techniques to scale these methods ( Gorodetsky et al. , 2015 ) . However , these techniques have been applied to special classes of systems and stochastic control problem formulations and have demonstrated limited applicability so far . Other methods include transforming the SOC problem into a deterministic one by working with the Fokker-Planck PDE representation of the system dynamics ( Annunziato & Borzi , 2010 ; Annunziato & Borzì , 2013 ) . This approach requires solving the problem on a grid as well and therefore doesn ’ t scale to high dimensional systems due to the aforementioned curse of dimensionality . Alternative sampling-based methodologies rely on the probabilistic representation of the backward PDEs and generalization of the so-called linear Feynman-Kac lemma ( Karatzas & Shreve , 1991 ) to its nonlinear version ( Pardoux & Rascanu , 2014 ) . Application of the linear Feynman-Kac lemma requires the exponential transformation of the value function and certain assumptions related to the control cost matrix and the variance of the noise . Controls are then computed using forward sampling of stochastic differential equations ( Kappen , 2005 ; Todorov , 2007 ; 2009 ; Theodorou et al. , 2010a ) . The nonlinear version of the Feynman-Kac lemma overcomes the aforementioned limitations . However it requires a more sophisticated numerical scheme than just forward sampling , which relies on the theory of Forward-Backward Stochastic Differential Equations ( FBSDEs ) and their connection to backward PDEs . The FBSDE formulation is very general and has been utilized in many problem formulations such as L2 and L1 stochastic control ( Exarchos & Theodorou , 2018 ; 2016 ; Exarchos et al. , 2018 ) , min-max and risk-sensitive control ( Exarchos et al. , 2019 ) and control of systems with control multiplicative noise ( Bakshi et al. , 2017 ) . The major limitation of the algorithms that rely on FBSDEs , is the compounding of errors from Least Squares approximation used at every timestep of the Backward Stochastic Differential Equation ( BSDE ) . Recent efforts in the area of Deep Learning for solving nonlinear PDEs have demonstrated encouraging results in terms of scalability and numerical efficiency . A Deep Learning-based algorithm was introduced by Han et al . ( 2018 ) to approximate the solution of non-linear parabolic PDEs through their connection to first order FBSDEs . However , the algorithm was tested on a simple high-dimensional linear system for which an analytical solution already exists . A more recent approach by Pereira et al . ( 2019 ) extended this work with a more efficient deep learning architecture and successfully applied the algorithm for control tasks on non-linear systems . However , similar to Han et al . ( 2016 ) , the approach is only applicable to stochastic systems wherein noise is either additive or state dependent . In this paper , we develop a novel Deep Neural Network ( DNN ) architecture for HJB PDEs that correspond to SOC problems in which noise is not only additive or state-dependent but control multiplicative as well . Such problem formulations are important in biomechanics and computational neuroscience , autonomous systems , and finance ( Todorov & Li , 2005a ; Mitrovic et al. , 2010 ; Primbs , 2007 ; McLane , 1971 ; Phillis , 1985 ) . Prior work on stochastic optimal control of such systems considers linear dynamics and quadratic cost functions . Attempts to generalize these linear methods to the case of stochastic nonlinear dynamics with control multiplicative noise are only primitive and require special treatment in terms of methods to forward propagate and linearize the underlying stochastic dynamics ( Torre & Theodorou. , 2015 ) . Aforementioned probabilistic methods relying on the linear Feynman-Kac lemmma can not be derived in case of control multiplicative noise . Below we summarize the contributions of our work : • We design a novel DNN architecture tailored to solve second-order FBSDEs ( 2FBSDEs ) . The neural network architecture consists of Fully Connected ( FC ) feed-forward and LongShort Term Memory ( LSTM ) recurrent layers . The resulting Deep 2FBSDE network can be used to solve fully nonlinear PDEs for nonlinear systems with control multiplicative noise . • We demonstrate the applicability and correctness of the proposed algorithm in four examples ranging from , traditionally used linear and nonlinear systems in control theory , to robotics and biomechanics . The proposed algorithm recovers analytical controls in the case of linear dynamics while it is also able to successfully control nonlinear dynamics with controlmultiplicative and additive sources of uncertainty . Our simulations show the robustness of the Deep 2FBSDE algorithm and prove the importance of considering the nature of the stochastic disturbances in the problem formulation as well as in neural network architecture . The rest of the paper is organized as follows : in Section 2 we first introduce notation , some preliminaries and discuss the problem formulation . Next , in Section 3 we provide the 2FBSDE formulation . The Deep 2FBSDE algorithm is introduced in Section 4 . Then we demonstrate and discuss results from our simulation experiments in Section 5 . Finally we conclude the paper in Section 6 with discussion and future directions . 2 STOCHASTIC OPTIMAL CONTROL . In this section we provide notations and stochastic optimal control concepts essential for the development of our proposed algorithm and then present the problem formulation . Note that the bold faced notation will be abused for both vectors and matrices , while non-bold faced will be used for scalars . 2.1 PRELIMINARIES . We first introduce stochastic dynamical systems which have a drift component ( i.e . the non-stochastic component of the dynamics ) that is a nonlinear function of the state but affine with respect to the controls . The stochastic component is comprised of nonlinear functions of the state and affine control multiplicative matrix coefficients . Let x ∈ Rnx be the vector of state variables , and u ∈ Rnu be the vector of control variables taking values in the set of all admissible controls U ( [ 0 , T ] ) , for a fixed finite time horizon T ∈ [ 0 , ∞ ) . Let ( [ v ( t ) T w ( t ) T ] T ) t∈ [ 0 , T ] be a Brownian motion in Rnw+1 , where v ( t ) ∈ R , w ( t ) ∈ Rnw and the components of w ( t ) are mutually independent one dimensional standard Brownian motions . We now assume that functions f : [ 0 , T ] × Rnx → Rnx , G : [ 0 , T ] × Rnx → Rnx×nu , Σ : [ 0 , T ] × Rnx → Rnx×nw and σ ∈ R+ satisfy certain Lipschitz and growth conditions ( see supplementary materials ( SM ) A ) . Given this assumption , it is known that for every initial condition ξ ∈ Rnx , there exists a unique solution ( x ( t ) ) t∈ [ 0 , T ] to the Forward Stochastic Differential Equation ( FSDE ) , dx ( t ) = ( f ( t , x ( t ) ) + G ( t , x ( t ) ) u ( t ) ) ︸ ︷︷ ︸ drift vector dt+ σG ( t , x ( t ) ) u ( t ) dv ( t ) ︸ ︷︷ ︸ control multiplicative noise + Σ ( t , x ( t ) ) dw ( t ) ︸ ︷︷ ︸ only state-dependent noise = ( f ( t , x ( t ) ) + G ( t , x ( t ) ) u ( t ) ) dt+ C ( t , x ( t ) , u ( t ) ) d ( t ) x ( 0 ) = ξ , ( 1 ) where C ( t , x ( t ) , u ( t ) ) = [ σG ( t , x ( t ) ) u ( t ) , Σ ( t , x ( t ) ) ] and ( t ) = [ v ( t ) w ( t ) ] . 2.2 PROBLEM STATEMENT AND HJB PDE . For the controlled stochastic dynamical system above , we formulate the SOC problem as minimizing the following expected cost quadratic in control J ( t , x ( t ) ; u ( t ) ) = EQ [ T∫ t ( q ( s , x ( s ) ) + 1 2 u ( s ) TRu ( s ) ) ds+ φ ( x ( T ) ) ] ( 2 ) where q : Rnx → R+ is the running state-dependent cost function , R ( control cost coefficients ) is a symmetric positive definite matrix of size nu × nu and φ : Rnx → R+ is the terminal state cost . q and φ are differentiable with continuous derivatives up to the second order . The expectation is taken with respect to the probability measure Q over the space of trajectories induced by the controlled stochastic dynamics in equation 1 . We can define the value function as { V ( t , x ( t ) ) = inf u ( t ) ∈U ( [ t , T ] ) J ( t , x ( t ) ; u ( t ) ) V ( T , x ( T ) ) = φ ( x ( T ) ) . ( 3 ) Under the condition that the value function is once differentialble in t and twice differentiable in x with continuous derivatives , we can follow the standard stochastic optimal control derivation to obtain the HJB PDE ( written without all the functional dependencies for simplicity ) { Vt + q + V T x f − 12V T x GR̂ −1GTVx + 1 2 tr ( VxxΣΣ T ) = 0 V ( T , x ( T ) ) = φ ( x ( T ) ) , ( 4 ) with the optimal control of the form , u∗ ( t , x ) = −R̂−1G ( t , x ) TVx ( t , x ) . ( 5 ) where , R̂ , ( R + σ2GTVxxG ) . The derivation for both equations 4 and 5 can be found in SM B .

Summary: The authors in the paper studied the problem of stochastic optimal control using deep learning. The main contributions of this work can be summarized by introducing an additional backward stochastic differential equation (BSDE) over Pereira et al. (2019), where instead of predicting the gradient of the value function (V_x), the authors predicted the values required to compute the Hamiltonian H(V_x) instead. The authors then compared the results against both closed form solutions of optimal control and a numerical approximate controller by Li and Todorov (2007). However, it is unclear whether this particular modification leads to any incremental benefits over Pereira et al. (2019), nor is it compared against the feedforward networks of Han et al. (2017). Furthermore, one of the main benefits of this modifications the authors claimed is scalability to higher dimensions, yet we do not have any experiments in more than 6 dimensions in the quadcopter case.