Deep Nonlinear Stochastic Optimal Control for Systems with Multiplicative Uncertainties
Authors: Marcus Pereira
Ziyi Wang
Tianrong Chen
Evangelos Theodorou
Venue:  (2020)
Keywords: Deep Learning
Stochastic Optimal Control
Robotics
Biomechanics
LSTM

Abstract: We present a deep recurrent neural network architecture to solve a class of stochastic optimal control problems described by fully nonlinear Hamilton Jacobi Bellman partial differential equations. Such PDEs arise when one considers stochastic dynamics characterized by uncertainties that are additive and control multiplicative. Stochastic models with the aforementioned characteristics have been used in computational neuroscience, biology, finance and aerospace systems and provide a more accurate representation of actuation than models with additive uncertainty. Previous literature has established the inadequacy of the linear HJB theory and instead rely on a non-linear Feynman-Kac lemma resulting in a second order forward-backward stochastic differential equations representation. However, the proposed solutions that use this representation suffer from compounding errors and computational complexity leading to lack of scalability. In this paper, we propose a deep learning based algorithm that leverages the second order Forward-Backward SDE representation and LSTM based recurrent neural networks to not only solve such Stochastic Optimal Control problems but also overcome the problems faced by previous approaches and scales well to high dimensional systems. The resulting control algorithm is tested on non-linear systems in robotics and biomechanics to demonstrate feasibility and out-performance against previous methods.   

Under review as a conference paper at ICLR 2020
DEEP NONLINEAR STOCHASTIC OPTIMAL CONTROL FOR SYSTEMS WITH MULTIPLICATIVE UNCERTAIN- TIES
Anonymous authors Paper under double-blind review
ABSTRACT
We present deep recurrent neural network architecture to solve a class of stochas- tic optimal control problems described by fully nonlinear Hamilton Jacobi Bellman partial differential equations. Such PDEs arise when one considers stochastic dy- namics characterized by uncertaintics that are additive, state dependent and control multiplicative. Stochastic models with the aforementioned characteristies have been used in computational neuroscience, biology, finance and aerospace systems accurate representation of actuation than models with only additive uncertainty. Previous literature has established the inadequacy of the linear HJB theory and instead relies on non-linear Feynman-Kac lemma resulting in a second order -arcardrdda Stochastic Differential Equations representation. However, the proposed solutions that use this representation suffer from compound- ing errors and computational complexity resulting in lack of scalability. In this paper, we propose a deep learning based algorithm that leverages the second order Forward-Backward SDE representation and LSTM based recurrent neural networks to not only solve such Stochastic Optimal Control problems but also overcome the problems faced by traditional approaches and shows promising scalability. The resulting control algorithm is tested on nonlinear systems from robotics and biomechanies in simulation to demonstrate feasibility and out out-performance against previous methods.
and provide
INTRODUCTION
Stochastic Optimal Control (SOC) is the center of decision making under uncertainty with a long history and extensive prior work both in terms of theory as well as algorithms (Stengel 1994] Fleming & Soner 2006). One of the most celebrated formulations of stochastic optimal control for linear dynamics and additive noise, known as the Linear Quadratic Gaussian (LQG) case (Stengel 1994). For stochastic systems that are nonlinear in the state and affine in control, the stochastic optimal control formulation results in the Hamilton-Jacobe aallllllma (HJB) equation, which is a backward nonlinear Partial Differential Equation (PDE). Solving the HJB PDE for high dimensional systems is generally a challenging task and suffers from the curse of dimensionality.
Different algorithms have been derived address stochastic optimal control problems and solve the HJB equation. These algorithms can be mostly classified into two groups: (i) algorithms that rely on linearization and (ii) algorithms that rely on sampling. Linearization-bate first order Taylor's 's's approximation of dynamics (iterative LQG or iLQG) or quadratic approximation of dynamics (Stochastic Differential Dynamic Programming). and quadratic approximation of the cost function (Todorov & Li 20056 Theodorou et al. 20100). Application of the aforementioned algorithms is not straightforward and requires very small time discretization or special linearization schemes especially for the cases of control and/or state dependent noise. It is also worth mentioning these algorithms have not been investigated and remain open
algorithms rely
that the convergence properties questions. Sampling-based methods include the Markov-Chal -Chiin Monte Carlo (MCMC) approximation of the HJB equation (Kushner& & Dupuis] 1992; Huynh et al. 2016). MCMC-based algorithms rely on backward propagation of the value function pre-specified grid. Recently researchers have incorporated tensor-train decomposition techniques to scale these methods (Gorodetsky et al., 2015).. However, these techniques have been applied special classes of systems and stochastic control

---

Under review as a conference paper at ICLR 2020
problem formulations and have demonstrated limited applicability far. Other methods include transforming the SOC problem into a deterministic one by working with the Pokker-Plancee PDE representation of the system dynamics (Annunziato & Borzi 2010: Annunziato & Borzi, 2013). This approach requires solving the problem on a grid as well and therefore doesn't t scale to high dimensional systems due to the aforementioned curse of dimensionality.
Alternative sampling-basse based methodologies rely on the probabilistic representation of the backward PDEs and generalization of the so-called linear Feynman-Kae lemma (Karatzas & Shreve [991) to its nonlinear version (Pardoux & Rascanu 2014). Application of the linear Feynman- Maaiiann-Kaaa lemma requires the exponential transformation of the value function and certain assumptions related to the control cost matrix and the variance of the noise. Controls are then computed using forward sampling of stochastic differential equations (Kappen 2005| Todorov 2007 2009 Theodorou et al.) 2010a). The nonlinear version of the Feynman-Kac lemma overcomes the aforementioned limitations. However it requires a more sophisticated numerical scheme than just forward sampling, which relies on the theory of Forward-Backee to backward PDEs. The FBSDE formulation is very general and has been utilized in many problem formulations such as L2 and L1 stochastic control (Exarchos & Theodorou 2018 2016 Exarchos jet al. 2018), min-max and risk-sensitive control (Exarchos et al. (019)) and control of systems with control multiplicative noise (Bakshi et al.) 2017). The major limitation of the algorithms that rely on FBSDEs, is the compounding of errors from Least Squares approximation used at every timestep of the Backward Stochastic Differential Equation (BSDE).
Stochastic Differential Equations (FBSDEs) and their connection
Recent efforts in the area of Deep Learning for solving nonlinear PDEs have demonstrated encourag- ing results in terms of scalability and numerical efficiency. A Deep Learning-based algorithm was introduced by Han et al. (2018) to approximate the solution of non-linear parabolic PDEs through their connection first order FBSDEs. However, the algorithm was tested on a simple high- -ighddimensional linear system for which an analytical solution already exists. A more recent approach by |Pereira et al. (2019) extended this work with a more efficient deep learning architecture and successtully applied the algorithm for control tasks on non-linear systems. However, similar to Han et al. (2016), the approach is only applicable to stochastic systems wherein noise is either additive or state dependent.
In this paper, we develop a novel Deep Neural Network (DNN) architecture for HJB PDEs that correspond to SOC problems in which noise is not only additive or state-dependent but control multiplicative as well. Such problem formulations are important in biomechanics and computational neuroscience, autonomous systems, and finance (Todorov& & Li, 2005a Mitrovic et al. 2010: Primbs 2007. McLanne| 1971, Phillis 1985). Prior work stochastic optimal control of such systems considers linear dynamics and quadratic cost functions. Attempts to generalize these linear methods to the case of stochastic nonlinear dynamics with control multiplicative noise are only primitive and require special treatment in terms of methods to forward propagate and linearize the underlying stochastic dynamics (Torre & Theodorou. (015). Aforementioned probabilistic methods relying on the linear Peynman-Kat lemmma cannot be derived in case of control multiplicative noise.
Below we summarize the contributions of our work:
We design novel DNN architecture tailored to solve second-order FBSDEs (2FBSDEs). The neural network architecture consists of Fully Connected (FC) feed-forward and Long- Short Term Memory (LSTM) recurrent layers. The resulting Deep 2FBSDE network can be used to solve fully nonlinear PDEs for nonlinear systems with control multiplicative noise. We demonstrate the applicability and correctness of the proposed algorithm in four examples ranging from, traditionally used linear and nonlinear systems in control theory, to robotics and biomechanics. The proposed algorithm recovers analytical controls in the case of linear dynamics while it is also able to successfully control nonlinear dynamics with control- multiplicative and additive sources of uncertainty. Our simulations show the robustness of the Deep 2FBSDE algorithm and prove the importance of considering the nature of the stochastic disturbances in the problem formulation as well as in neural network architecture.
The rest of the paper is organized as follows: in Section 2 we first introduce notation, some prelimi- naries and discuss the problem formulation. Next, in Section 3 we provide the 2FBSDE formulation. The Deep 2FBSDE algorithm is introduced in Section [ Then we demonstrate and discuss results from our simulation experiments in Section Finally we conclude the paper in Section 6 with discussion and future directions.
2

---

Under review as a conference paper at ICLR 2020
2 STOCHASTIC OPTIMAL CONTROL
In this section we provide notations and stochastic optimal control concepts essential for the develop- ment of our proposed algorithm and then present the problem formulation. Note that the bold faced notation will be abused for both vectors and matrices, while non-bold faced will be used for scalars.
2.1 PRELIMINARIES
We first introduce stochastic dynamical systems which have drift component (i.e. the non-stochastic component of the dynamics) that is nonlinear function of the state but affine with respect to the controls. The stochastic component is comprised of nonlinear functions of the state and affine control multiplicative matrix coefficients. Let a E IR" be the vector of state variables, and u E IR" be the vector of control variables taking values in the set of all admissible controls (4( T)), for fixed finite time horizon T E [0, oo). Let ([v(t)= (o())])))en rE be Brownian motion in Rn where v() E R. w(()= E Rhw and the components of w(() are mutually independent one dimensional standard Brownian motions. We now assume that functions f [0, x R + R"=, G [O, R= Ra X,,, Σ [0, T x R IR X and R satisfy certain Lipschitz and growth conditions (see supplementary materials (SM) A).
Given this assumption, it is known that for every initial condition & R there exists unique solution (zxt)) ke(o,T] to the Forward Stochastic Differential Equation (FSDE),
(t) (t, + G(t, ((())((())(())) dt + oG oo (()))(()))(()) E(t, (t)))wo)
control multiplicativv noise (ft) + Gtt (t) C(t, ((t)
drift vector
only state-dependent noise (1)
(0)) = E.
where C(t, ((t), u(t)) = (o)((t E(t, (t))) and e(t) = (v(t)
(tt)]
2.2 PROBLEM STATEMENT AND HJB PDE
For the controlled stochastic dynamical system above, formulate the SOC problem minimizing the following expected cost quadratic in control
!
Bo (g(s, 1 u(() u Ru(s)) d(x(T))
(tt,x(t): Eo
(2)
where q _ + + is the running state-dependent cost function, R (control cost coefficients) is a symmetric positive definite matrix of size nu X Tha and Φ RT + R+ is the terminal state cost. and @ are differentiable with continuous derivatives up to the second order. The expectation taken with respect the probability measure Q over the space of trajectories induced by the controlled stochastic dynamics in equation] [
We can define the value function as
V (v)))(())
inf u())) (t,) (TT,x((T) =))(T)))
Under the condition that the value function once differentialble in and twice differentiable in with continuous derivatives, we can follow the standard stochastic optimal control derivation to obtain the HJB PDE (written without all the functional dependencies for simplicity)
V1 -V -eR--GGTV VTVV, + tr(V ΣΣT) =
(4)
=
with the optimal control of the form,
u (t, x)
G(t,
(5)
where, R A (R + σ² GT G). The derivation for both equations 4 and 5 can be found in SM
B
3

---

Under review as a conference paper at ICLR 2020
3 A FBSDE SOLUTION TO THE HJB PDE
The theory of BSDEs establishes a connection between the solution of a parabolic PDE and a set of FBSDEs (Pardoux & Peng 1990 EI Karoui al. 1997). This connection has been used to solve the HJB PDE in the context of SOC problems. Exarchos & Theodorou (2018) solved the HJB PDE in the absence of control multiplicative noise de with utilized the second order FBSDEs or 2FBSDEs to solve the fully nonlinear HJB PDE in the presence of control multiplicative noise, but did not consider any control in the FSDE. Lack of control leads to insufficient exploration and for highly nonlinear systems renders it impossible to find an optimal solution to complete the task. In light of this, we introduce new set of 2FBSDEs
of first order FBSDEs. Bakshi et al. (2017)
dæ
fdt Gu dt Cde
(FSDE)
dV
H(Vddt + V TYGGudd dt + VT Cde
(BSDE 1)
(6)
dV
= -H(V,) dt + Gu dt + Van Cde
(BSDE 2)
(00)) = V(T) = (Tx(T)), (T) = (x(())))
where u" the optimal control given by equation 5 and the operator H is defined as
H((-) _ a.)- + a. (OCCC)
(7)
Note that the functional dependencies are dropped in this and the following section for simplicity.
We can obtain this particular set of 2FBSDEs by substituting for the optimal control (equation (5) in
the forward process (equation []. The backward processes are obtained by applying Ito's lemma (Itō)
1944), which is essentially the second order Taylor series expansion, to the value function V and
its gradient Vg- Additionally, we substitute the expression for V, from equation 4 into BSDE 1 and obtain the final form of the 2FBSDEs (detailed derivation is included in SM(C
as
dæ
= fdi + G(u di + ou do) + Edw
dv
= - (q - VV ( Vis G)R GGR
+ GT ou de) + V2 Edw
(8)
dV
(A Va- V2=ef Vax G(u"dald dt + ou' ou'dv) + Van Edw
a00) = E, V(T) = ((x(T),, (T) = dx (T)),
with A = 0V2)) + t0 (Va C.
forward sample the FSDE. The second-order BSDEs (2BSDEs), on the other hand, need to satisfy a terminal condition and therefore have to be propagated backwards in time. However, since the uncertainty that enters the dynamics evolves forward in time, only the conditional expectations of the 2BSDEs can be back-propage pack-propagated. For more details please refer to [Shreve (2004) Chapter 2). Bakshi al. (017)) back-propagate approximate conditional expectations, computed using regression, of the two processes. This method however, suffers from compounding errors introduced by least squares estimation at every time step. contrast a Deep Learning (DL. based approach, first introduced by Han et al.] (2018), mitigates this problem by using the terminal condition as the prediction target for a forward propagated BSDE. This enabled by randomly initializing the value function and its gradient at the start time and treating them as trainable parameters of self-supervised deep learning problem. In addition, the approximation errors at each time step are compensated for by backpropagation during training of the DNN. This allowed using FBSDEs to solve the HJB PDE for digeeniimnii linear systems. A more recent approach, the Deep FBSDE controller (Pereira et al. 2019), futher extends the work successfully nonlinear systems, with and without control constraints, in simulation that correspond to first order FBSDEs. It also introduced LSTM-based network architecture, in contrast using separate feed-forward neural networks at every timestep as in Han et al. (2016), that resulted in superior performance, memory and time complexity. Extending this work, we propose a new framework for solving SOC problems of systems with control multiplicative noise, for which the value function solutions correspond to 2FBSDEs.
Using equation 8
4 DEEP 2FBSDE CONTROLLER
In this section, we introduce a new deep network architecture called the Deep 2FBSDE Controller and present training algorithm solve SOC problems with control multiplicative noise.
4

---

Under review as a conference paper at ICLR 2020
V
V
N-1
N-1
3
V
9,N01
V. A.N
CVV Vir V...Ven Vani VieN Vanxi VIEN
Kg
Kyu1
V
Aa
Vaus
Agss
Vin
FC2
RC
PC2
PC
PC2
LSTM
LSTM
L.77M
LSTM
Xn
Xu
Aw
AWws
Figure 1: Deep 2FBSDE neural network architecture. The blocks FC are fully connected layers with linear activations while the L.TM block represents recurrent layers of stacked LSTM cells with standard nonlinear activations. These layers are parameterized by and shared temporally. Additionally, V (z tu) and (a, to) are parameterized by and respectively. The self supervisd targets V. V2.n are the terminal conditions from equation g and V N
Oex (xN).
Time discretization: In order to approximate numerical solutions of the Stochastic Differential Equations (SDEs) we choose the explicit Euler- Maruyama aime-disrrttization scheme (Kloeden & Platen 2013 Chapter 9) similar to (Pereira et al. 2019 Han al. (0188). Here overload t as both the continuous time variable and discrete time index and discretize the task horizon < < as == (0, 1, N). where T N.. This is also used discretize all variables step functions their discrete time index lies the interval /A, (t 1)At). We use subscript t to denote the discretized variables in our proposed algorithm Alg.
Algorithm 1: Finite Time Horizon Deep 2FBSDE Controller
Given: E, f, G, Σ, a: Initial state, drift, actuation dynamics, state dependent noise matrix and control multiplicative noise std. deviation; Φ. q. R: Cost function parameters; N: Task horizon, K: Number of iterations, M: Batch size; At: Time discretization; A: regularization parameter; parameterized by trainable weight s/; Va.0- Vn (æ:; )) Gradient of value function at 0 parameterized by trainable weights ;: 0: Weights and biases of all fully-connected and LSTM layers;
Parameters: Vo Van; *):: Value function
Initialize all trainable paramters and states: 00, 30,
for to K do for = 1 to M do for = 0 to N- 1 do
Compute G matrix: = G G
Network prediction: Van,t: A; = fro, (fISTM oh- fro, (x:
Compute optimal control: ut" - -R + oGG: Vi G.) G, Va
Sample Brownian noise: Aw;
IAt); Aut (10,AA
Compute control: K; K(t, xx)) Gi (u: At out Auf)
Forward propagate the SDEs: w141 Vit1 Vi #11+1 f2rnSDE Vi, Vi Vi
a.0 Vi K!) + time discretized equation
end for end for Compute mini-batch loss: L = fLoss (VN, Vg Va,N, V VO.Ns Vun,N. V Vo 0 @k-1) Gradient update: Ok, uk,, ck <- Adam, Ok-1 ,bk-1 (k-1) end for return 8 K 5k
Network architecture: Inspired by the LSTM-based recurrent neural network architecture intro- duced by Pereira al. (2019) for solving first order FBSDEs, propose the network in fig adapted to the uncertainties in 2FBSDEs given by equation 8 Instead of predicting the gradient of the value function V,, every time step. the output of the LSTM used predict the Hessian of the value function V and A = (Va) + (rr(mm CCT) using two separate FC layers with
5

---

Under review as a conference paper at ICLR 2020
linear activations. Notice that (Vz) is rank 3 tensor and using neural networks to predict this term explicitly would render this method unscalable. We, however, bypass this problem by instead predicting the trace of the tensor product which is a vector allowing linear growth in output size with state dimensionality.. Of these, Vaz is used to compute the control term K =
G( di + ou ou"de). This in turn is used to propagate the stochastic dynamics in equation [ Both Van and A used to propagate 2 which is then used to propagate V. This is repeated until the end of the time horizon as shown in fig I
Finally, the predicted values of V. V2 and Ver are compared with their targets computed using I at the end of the horizon, to compute loss function for backpropagation.
Algorithm: Algorithm details the training procedure of the Deep 2FBSDE Controller. Note that superscripts indicate batch index for variables and iteration number for trainable parameters. The value function Vo and its gradient Va,0 (at time index = 0), are randomly initialized and trainable. Functions JL.STM. frC, and fFC, denote the forward propagation equations of standard LSTM layers (Hochreiter Schmidhuber 1997) and FC layers with tanh and linear activations respectively, and BSFB represents a discretized version of equation 8 using the explicit Euler-Maruyain discretization scheme. The loss function (C) is computed using the given terminal conditions as targets, the propagated value function (V), its gradient (V and Hessian (Va=) at the final time index as well as an L2 regularization term as follows
time
(9) A detailed justification of each loss term is included in SM The network is trained using any variant of Stochastic Gradient Descent (SGD) such as Adam (Kingma & Ba 2014) until convergence. The algorithm returns a trained network capable of computing an optimal feedback control at every timestep starting from the given initial state.
ci ||V* Vx + c2 l|V V2.N
V Va 1 + c4 V + -111
SIMULATION RESULTS
In this section we demonstrate the capability of the Deep 2FBSDE Controller on 4 different systems in simulation. We first compare the solution the Deep 2FBSDE Controller to the analytical solution for scalar linear system to validate the correctness of our proposed algorithm. We then consider cart-pole swing-up task and a reaching task for 12-state quadcopter. Finally, we tested on a 2-link 6-muscle (10-state) human arm model for planar reaching task. The results were compared against the Deep FBSDE Controller in Pereira et al. (2019), where in the effect of control multiplicative noise was ignored by only considering additive noise models resulting in first order FBSDEs, and the iLQG controller in Li & Todorov (2007). Hereon we use 2FBSDE, FBSDE and iLQG to denote the Deep 2FBSDE, Deep FBSDE and iLQG controllers respectively. The system dynamics can be found in SM El and the simulation parameters can be found in SMF
Nisyticall
210SCC
All the comparison plots contain statistics gath- ered over 128 test trials. For 2FBSDE and FB- SDE, we used time discretization Al 0.004 seconds for the linear system problem, At = 0.02 seconds for the cart-pole and quadcopter simulations and At = 0.02 seconds for the hu- man arm simulation. For the iLQG simulations, At = 0.01 seconds for cartpole and quadcopter and AL = 0.001 seconds for the human arm were used avoid numerical instability. These values were hand-tuned until reasonable perfor- mance was observed from iLQG. In all plots, the solid lines represent mean trajectories, and the shaded regions represent the 68% confidence region (mean + standard deviation).
122
0.8
States
0.1
0.
-0.2
0.8
1.0
Time
Figure 2: The 2FBSDE solution is very similar to the analytical solution for a scalar linear system.
Linear System: We consider a scalar linear time-invarian system
dxx(t) = (t) + Bu(t)dt + o (u())do(() + Cdw((),
(10)
with quadratic running state cost and terminal state cost q ==0=(tt), = ftma(t)?
The dynamics and cost function parameters are set as A = 0.2, B 1.0, 0.1, o 0.1,0=0.5.Q 0.5.Q= 0, Qr = 80, R 2, 20 1.0. The task is to drive the state to O. Note that the problem is different

---

Under review as a conference paper at ICLR 2020
2FBSDE
F3SDE
iLOG
target
Cant vattvtyyyyy (ms3) Cort/ostion
Time
Timee (5)
Figure 3: Cartpole Swing-up Task: 2FBSDE most aware of the presence of control multiplicative noise and uses least control effort to perform the task. This is evident from the cart velocity plot, directly actuated state, which tries to stay as close to zero as possible.
2FBSDE
FBSDE
iLGG
largel
Postton Ins
Yhu Aogle mmsh)
Time (S)
Time (5)
1ime (5)
Figure 4: Quadcopter Reach and Hover task: 2FBSDE clearly out-performs both FBSDE and iLQG, as indicated by the smaller variance all states.
than the for LQG due the presence of control multiplicative noise (i.e. v(() 0). Let us assume that the value function (equation 4 has the form V (t,)
((()). where S(T) = 0, P(T) = Qr.c(t) = _ 2 {c²P(ssds The values of P and Q are obtained by solving corresponding Riccatti equations, using ODE solvers, that can derived in a similar manner to the LQG case in [Stengel (1994 Chapter 5) and are used to compute the optimal control (equation at every timestep. The solution obtained from 2FBSDE compared against the analytical solution in fig. 2 The resulting trajectories have matching mean and comparable variance, which verifies the effectiveness of the controller on linear systems.
We applied the controllers to cartpole dynamics for swing-up task with time horizon of 2.0 seconds. The networks were trained using a batch size 256 each for 2000 iterations. We imposed a strict restriction impose any cost or target for the cart position state. We would like to highlight in fig. 3 that both 2FBSDE and iLQG choose to pre-swing the pendulum and use the system's 's momentum achieve the swing-up task. They thus respect the presence of control multiplicative noise entering the system as compared FBSDE which tries drive the pendulum the desired vertical position soon as possible by applying as much control as required. Additionally, we tested for a lower control cost R 0.1 (see SM for plots) and observed qualitatively different behavior from the controllers.
Cartpole:
the controllers with control cost coefficient R 0.5 and did not
The controller was also tested on a 12-state quadcopter dynamics model for task of reaching and hovering at a target position with time horizon of 2.0 seconds. The network was trained with batch size of 256 for 5000 iterations. Only linear and angular states are included in fig. since they most directly reflect the task performance (velocity plots included SMGG) The figure demonstrates superior performance of the 2FBSDE controller over the FBSDE and iLQG controller reaching the target state faster and maintaining smaller state variance, Moreover, these results also convey the importance of taking into account multiplicative noise in the design of optimal controllers as the state dimensionality and system complexity increases. We also tested the 2FBSDE controller
Quadcopter:
7

---

Under review as a conference paper at ICLR 2020
2+85UE
iLQG
(Temmnaa
CTerminal
-2
-4
-6-
177
6.4
06
6.8
1.0
6.10
0.1
0:30
0.7%
8.50
0.55
0.45
Additvv Nolse standare deviatan
Muropiiaatva Nalse standard devitation
Figure 5: Comparison 2FBSDE and iLQG for Human Arm system: Plots show terminal cost versus additive (E) and control multiplicative noise standard deviations (o). The FBSDE results are omitted as failure (divergence) occurred all test trials. When additive noise varied (left), 2FBSDE outperforms iLQG at lower E and performance deteriorates much slower than iLQG as Σ increases. When multiplicative noise is varied (right), iLQG exhibits very erratic behaviour while the terminal cost gradually increases with a for 2FBSDE.
for the same task (reach and hover) for a longer horizon of 3.0 seconds (see SM for plots) to verify its stabilizing capability in presence of control multiplicative noise.
2-link 6-muscle Human Arm: This is a 10-state bio-mechanical system wherein control multiplica-
tive noise models have been found to closely mimic empirical observations (Todorov] 22055.. The controllers were tasked with reaching target position in 1.0 second. The networks were trained with a batch size of 256 for 2000 iterations. Additive noise in the joint-angle acceleration channels and multiplicative noise in muscle activations were considered. In fig. S we show the log of terminal state cost (log (xxx)) versus different values of additive noise standard deviations (while holding multiplicative noise standard deviation fixed at 0.1) the left and versus different values of mul- tiplicative noise standard deviations (keeping additive noise standard deviation fixed at 0) on the right. As seen in the fig. the performance of iLQG is very erratic compared to 2FBSDE as is varied. The difference in behaviors can be attributed to the fact that iLQG is only aware of the first two moments of the uncertainty entering the system while the 2FBSDE, being a sampling based controller is exposed to the true uncertainty entering the system.
Discussion: Although the performance of iLQG for the cartpole and human arm tasks seem com- petitive to 2FBSDE, we would like to highlight the fact that iLQG becomes brittle as of the control multiplicative noise increased (or time horizon is increased) and requires very fine time discretizations, proper regularization scheduling and fine state cost coefficient tuning in order to be able to converge. Moreover, the control cost (R) had to be tuned to a high value in order to prevent divergence at higher noise standard deviations. This is in contrast to the results in Li & Todorov (2007), on account of two main reasons: firstly they do not consider the presence additive noise in the angular acceleration channels in addition to multiplicative noise in the muscle activation channels and secondly the paper does not provide all details to fully reproduce the published results nor is code with muscle actuation made publicly available. Some of the crucial elements to fully implement the human arm model such as relationships between muscle lengths, muscle velocities, and the respective joint angles had to be obtained by studying the work done by Teka et (2017).
6 CONCLUSIONS
In this paper, we proposed the Deep 2FBSDE Controller to solve the Stochastic Optimal Control problems for systems with additive, state dependent and control multiplicative noise. The algorithm relies on the 2FBSDE formulation with control the forward process for sufficient exploration. The effectiveness of the algorithm is demonsirated by comparing against analytical solution for linear system, and against the first order FBSDE controller and the iLQG controller on systems of cartpole, quadcopter and 2-link human arm in simulation. Potential future directions of this work include application to financial models with intrinsic control multiplicative noise and theoretical analysis of error bounds value function approximation as well investigating newer architectures.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, lan Goodfellow, Andrew Harp. Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large- scale machine learning on heterogeneous systems, 2015, URL available from tensorflow.org.
Software
Mario Annunziato and Alfio Borzi. Optimal control of probability density functions of stochastic processes. Mathematical Modelling and Analysis, 15(4):393-407 2010.
Mario Annunziato and Alfio Borzi A fokker tokker-planck control framework for multidimensional stochastic processes, Journal of Computational and Applied Mathematics. 237(1):487-500 2013.
K. S. Bakshi, D. D. Fan, and E. A. Theodorou. Stochastic control of systems with control multiplicative noise using second order fbsdes. In 2017 American Control Conference (ACC), pp. 424 431,, May 2017. doi:
R. Bellman and R. Kalaba. Selected Pupers On mathematicul trends in Control Theory. Dover Publications. 1964.
Nicole El Karoui, Shige Peng, and Marie Claire Quenez. Backward stochastic differential equations in finance. Mathematical finance, 7(1):1711, 1997.
Ht M ElKholy. Dynamic modeling and control of a quadrotor using linear and nonlinear approaches. Master of Science in Robotics, The American University in Cairo, 2014.
I. Exarchos and F. A. Theodorou. Teanning optimal control via forward and backward stochastic differential eequations] In American Control Conference (ACC), 2016, pp. 1161. IEEE, 2016.
I. Exarchos and E. A. Theodorou. Stochastic optimal control via forward and backward stochastic differential equations and importance sampling Automatica, 87:159-165, 2018.
I. Exarchos, E. A. Theodorou, and P. Tsiotras. Stochastic L -opiiall control via forward and backward sampling. Systems & Control Letters, 118:101-108, 2018.
Toannis Exarchos, Evangelos Theodorou, and Panagiotis Tsiotras. Stochastic differential games: A sampling approach via fhsdes. Dynamic Games and Applications, 9())486-505, Jun 2019. ISSN 2153-0793. doi: 10.10077/13235-015 011-0268---
URI.hctpa: /doii org/ 000
018 0268-4
W. H. Fleming and H. M. Soner. Controlled Markov processes and viscosity solutions. Applications of mathematics. Springer, New York, 2nd edition, 2006.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Alex Gorodetsky, Sertac Karaman, and Youssef Marzouk. Efficient high-dimensional stochastic optimal motion control using tensor-train decomposition. In Proceedings of Robotics: Science and Systems, Rome, Italy, July 2015. doi: 10. 15607//5/ 2015...0.0....0...
Jiequn Han, Arnulf Jentzen, and Weinan E. Solving high-dinmensional partall differeenial equations using deep learning] Proceedings of the National Academy of Sciences, 115(34):305-83100 2018. ISSN 0027-8424. doi: 10. 10..1073/pas..71551 1718942115 15. URL httph: /wtp://wwwww./.o// www.wppaa...o..
pnas wrg. http:pppwa..rrcccccoe///////////
Jiequn Han et al. Deep Learning Approximation for Stochastic Control Problems arXiv:1611.07422.. 2016.
arXiv preprint
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 133: 1997. doi: 10.1 1662nec/.... 1997.9.9.. 9.....88......
w9w.9997........ 1735
V. A. Huynh, S. Karaman, and E. Frazzoli. An incremental sampling-bused algoriit for stochastic optimal control. J. Robotic Res., 3504)3305-333, 2016. doi: 10.11777022833669115166666 URL httpp://xx doi. www.orgg110. 100.11770027834915166633
Kiyosi ltô. 109. stochastic integral. Proceedings of the Imperial Academy, 20(8):519-524, 1944.

---

Under review as a conference paper at ICLR 2020
II. J. Kappen. Linear theory for control of nonlinear stochastic systems. Phys Rev Lett, 95:200201, 2005. Joumal Article United States.
Karatzas E. Shreve. Brownian Motion and Stochastic Calculus (Graduate Texts in Mathematics). Springer, 2nd edition, August. 1991. ISBN 0387976558.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Peter E Kloeden and Eckhard Platen. Numerical solution of stochastic differential equations, volume 23. Springer Science & Business Media, 2013.
H. J. Kushner and P. G. Dupuis. Numerical Methods for Stochustic Control Problems in Continuous Time. Veringe--Verlag, London, UK, UK, 1992. ISBN 0-387-97834-8.
Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems, ICINCO (1), pp. 222- 20992,992200 2004.
Weiwei Li and Emanuel Todorov. Iterative lincarization methods for approximately optimal control and estimation of non- conelincar stochastic system. International Journal of Control, 10(9))1439911 2007.
McLane. Optimal stochastic control of linear systems with state- and control-dependent disturbances. JEEE Transactions on Automatic Control, 16(6):793-798, December 1971. ISSN 0018-9286. doi: 10.1109/TAC. 1971. 1099888..28.
Djordje Mitrovic, Stefan Klanke, Rieko Osu, Mitsuo Kawato, and Sethu Vijayakumar. A computational model of limb impedance control based on principles of internal model uncertainty. PLOS ONE, 5(10):1-11, 10 2010. doi: 00.177/ijourmallpoe00003333333 0013661. URL http::/:://a...or111111.3333 D013601
pone
Etienne Pardoux and Shige Peng. Adapted solution a backward stochastic differential equation. Systems & Control Letters, 14(1):55-61, 1990.
Etienne Pardoux and Aurel Rascanu. Stochastic Differential Equations, Backward SDEs, Partial Differential Equations, volume 69. 07 2014. doi: 10.1007/978- 3-319-05714-9
Marcus A Pereira, Ziyi Wang, Toannis Exarchos, and Evangelos A Theodorou. Learning deep stochastic optimal sdes. In Robotics: science and systems, 2019.
control policies using
Y. Phillis. Controller design of systems with multiplicative noise. IEEE Transactions Automatic Control, (10):1017-1019, October 1985. ISSN 0018-9286. doi: 198. 103828.
Primbs. Portfolio optimization applications of stochastic receding horizon control. In 2007 American Control Conference, pp. 1811-1816, July 2007. doi: 10.1 10.1109/ACC.2007..44 4282251..
Steven E Shreve. Stochastic calculus for finance II: Continuous- time models, volume 11. Springer Science Business Media, 2004.
R. F. Stengel. Optimal control and estimation. Dover books on advanced mathematics. Dover Publications, New York, 1994.
Wondimu W Teka, Khaldoun C Hamade, William H Barnett, Tacgyo Kim, Sergey N Markin, Ilya A Rybak, and Yaroslav Molkov. From the motor cortex the movement and back again. PloS one, 12(6)=01 79288, 2017.
E.A. Theodorou, Buchli, and S. Schaal. generalized path integral approach to reinforcement learning. Journal of Muchine Learning Research, (11)3317--1181 2010a.
E.A. Theodorou, Tassa, and Todorov. Stochastic differential dynamic programming. American Control Conference, pp. 1125-1132. 2010b.
E. Todorov. Stochastic optimal control and estimation methods adapted to the noise characteristics of the sensorimotor system. Neural Computation, 17(5):1084, 2005.
E. Todorov. Lincarly-solvable markov decision problems. In B. Scholkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems 19, Vancouver, BC, 2007. Cambridge, MA: MIT Press.
E. Todorov. Efficient computation of optimal actions. Proceedings of the national academy of sciences, 106(28): 11478-11483, 2009.
10

---

Under review as a conference paper at ICLR 2020
E. Todorov and W. Li. generalized iterative lig method for locally-optimal feedback control of constrained nonlinear stochastic systems. pp. 300-306, 2005a.
E. Todorov and Weiwei Li. generalized iterative lgg method for locally-optimal feedback control of constrained nonlinear stochastic systems. Proceedings of the 2005, American Control Conference, 2005., pp. 300-306 vol. I, June 2005b. doi: 10.1 109/ACC.2005. 1...99999.9.99499
G. DeLa Torre and E.A. Theodorou. Stochastic variational integrators system propagation and linearization. 2015.

---

Under review as a conference paper at ICLR 2020
SUPPLEMENTARY MATERIALS
A ASSUMPTIONS
Assumption 1. There exists a constant such that
(ff + (t) n, u) (f + (t,) x/, u()l S k([== - x'1 + 1m u
(11)
(C, C(L u k([ a0| + lu w'l)
(12)
[(f + Gt) x, Ctt, k(1 [a),,
(13)
where C = Go 2). vt E [), Vz, x'' E R"= and Vu, u' E 1
B HJB PDE DERIVATION
Applying the dynamic programming principle (Bellman & Kalaba 1964) to the value function we have
V(I, ((1))(())=
inf (4a)
Eq
1 T ds + V(t dt, d()) (g(a(s)) + ju(s)
(14)
2
Then, we can approximate the running cost integral with step function and apply Ito's lemma obtain
Itō 1944 to
v(t, inf Ee
(g(x(1)) 2 =u(() Ru(t)) (u(t)"RR(t))deee V(t dt, axt + dr))
inf Eq
(a(x(()) + (1()) Ru(()) v((,)(1)) v(()
Held
V.= (t, + (ddd
(tt, x(t,)(())d + (000))) u()))
inf (o(=(1)) + _ ((() v(), ((())
weul
vo (t, x(1)) f(t, [())at + G(t, (()))())
tr ((())
C(t, (t), C(t,
u(t) (1, (()) dt
We can cancel ((t) on both sides and bring the terms not dependent on controls outside of the infimum to get
inf (Lu" Ru + ViTT Gu + tr(V CC*) wEM[o..]
Ve+ V
(15)
Note that the functional dependencies have been dropped for brevity. Now we can the optimal control by taking the gradient of the terms inside the infimum with respect to u and setting it to zero. Before that we first substitute for C [oGu Σ| that the term inside the trace operator becomes V Gun" G V ΣΣT By using the property of trace operators, we can write, tr(Vam CCT) = tr(a" Gun GT + Vaa ΣΣT) tr(o² u "T" Va Gu) + tr(Va= ") Note that second term not function of therefore zero when we take the gradient. Additionally, the first term a sealar and taking its trace returns the same scalar. Now, taking the gradient w.r.t u, we get,
Ru" (V."G)" + +op Vun = 0 + R + G G)u' (G"V
Ru" -
=u" -
where, R é (R+o2 G).
12

---

Under review as a conference paper at ICLR 2020
The optimal control above be plugged back into equation 1S to obtain
Vi + V f + + 2 Vo
GR "RR 'R"'' - V2 ''"v2
+ 2
3 v GR " GR
(((Vo222>") -0
V V + + Ve GR " (R G)R VG'V
-R Vo tr(V -
V.+ V + V/' GR RR G
V. G tr(V ΣΣ =0
1 V4 + + 2 V2' GR lG" Vo + tr((Va2222)) =
which gives the HJB PDE as mentioned in equation
C DERIVATION OF THE 2FBSDEs
Firstly apply Ito's differentiation rule V:
dV Vdtt + V22 dæ + tr (Van CC
1
(16)
2
Since V is the value function, we can substitute in the HJB PDE for V and forward dynamics for dæ to get:
(q V f VVTGR -F'GR''G
+ "G"
tr (Vaz V>>>>")) dt
2
Vi fdt + G(u + dt tou ou Edw) + 2 tr (Van CC dt
---1-V1' _ GR G tr (Ve dt
V (ftt G(u +Gdt+G(udd dt + ou dv) )u'dv) Edw}
+ 2 tr (Von ΣΣT o Vaa"a"" Gu")dt
q f 1 2
G
2 VTGGR Gi--GR--GG V.r GR-
Ve [su Gl ou ou'de) + Edw)
(1- (1-v2+ Vr TRR Gv 5 -
GR C
GR
GR GR.GGR--C GR
vo (sul + di ou du) + Edw)
V
1
V. - TR TR
dt
2
(su -Gu t ou ou) + Edw
- (g } ve GR TR + V dt + V.. GT"G(ud di + ou'dd) + V Edw.
(17)
For Vs. we can again apply Ito's differentiation rule:
1
dV=
Vaidt + VTT dx + 2
tr (Va
(18)
We can again substitute in the forward dynamics for dar and get:
dV=
O Vadt
(19)
+ aa V. fdi + G(u ou'' ou)dd) + Edw)
(8am V2 V=
(A Vea Vaw G(u di + ou du)
(20)
Van Edw.
(21)
13

---

Under review as a conference paper at ICLR 2020
We have A V.Vo Lr(OezVa
Note that the transpose Vaw dropped since it is symmetric.
D Loss FUNCTION
The loss function used in this work builds the loss functions used in(Han et. (201)) and Pereira (2019 Because Deep 2FRSDE Controller propagates BSDEs, in addition to using the propagated value function (Vi), the propagated gradient of the value function (Va,a) can also he used in the loss function to enforce that the network meets both the terminal constraints ie. 6(œx) ) and di (xxx respectively. Moreover, since the terminal cost function is known, its Hessian can be computed and used enforce that the output of the FC1 layer at the terminal index Vaa.N) be equal
target Hessian the terminal cost function Φa (æx Although this is enforced only at the terminal time index (Va..v), because the weights of a recurrent noural network are shared across time, in order to be able to predict Ve=.N accurately all of the prior predictions Vaa,, will have to be adjusted accordingly, thereby representing some form of propagation of the Hessian of the value function.
Additionally, applying the optimal control, which the network prediction, the forward process introduces an additional gradient path through system dynamics every time step. Although this makes training difficult (gradient vanishing problem) allows the weights to now influence what the next state (i.c. the next time index) will be. As result, the weights can control the state at the end time index and hence the target (V* (æx)) for the neural network prediction itself. This can be added to the loss function which accelerate the minimization of the terminal cost achieve the task objectives. The final form of our loss function is
L= c1 CIV* -N +2 VV22
Va.N co IIV + 1
where V = =((NN) V (x(x and Vis = (xN)
E NON-LINEAR SYSTEM DYNAMICS AND SYSTEM PARAMETERS
For all simulation experiments, we used the same quadratic state cost the form g(x) [ - Dmmges)) > TQ(x Eusger) for both the running and terminal state costs.
E. CARTPOLE
The cartpole dynamics is given by
O2x2
02x2
sin (16 +
dt
(udt + oude)
dw.
02x2
12x2
mh th Thp cin
me+mi
17.....0 182 ooe win fi- (m un
[[mg + sin? 8)
((motmmp sin 8)
The model and dynamics parameters are set as m, 0.01 kg. m. = kg. - 0.5 m, == 0.125. The initial pole and cart position are 0 rad and 0 m with no velocity, and the target state is a pole angle of π rad and zeros for all other states. The control costs R are 0.5 and 0.1 for experiments in fig and fig. [e]espoctivell. The state cost matrix is the same for both experiments
diag [0.0, 6.0, 0.3, 0.3].
E.2 QUADCOPTER
The quadcopter dynamics used can he found in FlKholy 2014). Its dynamics is given by
14

---

Under review as a conference paper at ICLR 2020
H
1000 1O00
10000
8
2
0
&
0
di +
(sin Φ y + cos øb cos sin (cos @ sin di sin cas sin の)
U2
df
Us
0
044
9
cos o cos
Auy 10
1am 0/
1ax
Ise <n
Luv
1uy
Ixz d6
o0
Tax
1za
Tun
U2
du)
06x6
06x6
dw.
U3
06x6
Isx6
where
1
1
Ug
0
-1
0
1
Ug
0
1
T3
114 d
d
d
T4
The states positions, velocities, angles and angular The model and dynamics parameters
and the controls the four rotor torques (r). 0.47 kg. Jaz Jy 4.86 10-3 kg m?? Ta= 8.8 x "kgm"kgm I 0.225 m, 0.05. The initial state conditions used are zeros, and the target state I meter each in the north-cast-down directions and all other states zero. The control cost matrix is R. 24xx4-
The state cost matrix is Q diag|20.0, 20.0, 50.0, 5.0, 5.0, 5.0, 1.25, 1.25, 5.0, 0.25, 0.25, 0.25).
E.3 2-LINK 6- MUSCLE HUMAN ARM
The forward rigid-body dynamics of the 2-link human arm stated in Li & Todorov (004 is given by:
i = _M C(e, 0) B8)
where 0 E R² represents the joint angle vector consisting of 0 (shoulder joint angle) and 0 (elbow joint angle). M(0) 2x22 the inertia matrix which is positive definite and symmetric. C(O, 0) IR the vector centripetal and Coriolis forces. 5 E R=*2 is the joint friction matrix. 7 E R is the joint torque applied on the system. The torque is generated by activation of 6 muscle groups. Following equations for components of each of the above matrices and vectors:
ã1 2āzcos02
ãs + 02cos02
=
+ azcos0z
āz
(20 + 62)
C
a2sin02
01
bi1
b12
B
b21
b22
à I + Ia M2l?
02 m2 T.182
ā3 I
Where b1 b22 0.05, 612 621 0.025, m. the mass (k.4kg.1kg) the length of link (30em. 33cm), is the distance between joint center and mass of link (1em, 16cm), I; represents for the moment of inertia (0. (2.022kgmm' 0.045kkmm*) The activation dynamies for each muscle is given by:
d odvju: as
Vie [1.6]
i tdeact
15

---

Under review as a conference paper at ICLR 2020
where ldeact 66mssc, Iº vector of muscle activations, I2 is vector of instantaneous neural inputs. result of these dynamics, new state variables (a) will be incorporated into the dynamical system. With the muscle activation dynamics proposed above, the joint torque vector can be computed as follows:
M(0)T(a.1(0),0(000))
wherein T(a,1,)) 1, is tensile force generated by each muscle, whose expression given by:
T(a, 1,) A(C. FP(I))
Ny0h
A(a,1) = =1 (-(0.56N,,0)
a
N₂(1) 2.11 4.16
12.93
1.87
((()) = exp
1.03
5.72
if 0
Fv(1,)=
-5.12+ 080)0'
0.62 0.62- (-3.12+4.111 2..04
otherwise
Fy(() 0.02 exp(13.8 18.7()
The equation for (()) is obtained from Teka et al. (207 Equation 3). The parameters used in the equations above are mostly obtained by work in
Teka et al. (2017
which we provide here in the following table:
Muscle
Maximal Force, Fmaz (N) Optimal Length, Lops (m)
Velocity, e (m/sec)
SF
420
0.185
RsF
SE
0.170
Ór R.R
EF
1010
0.180
-(82 -
BB
1880
0.055
-(82 -0- (0)- RE)
BF
460
0.130
-O,BEFS (Ø2 0 R
BE
630
0.150
R.RF (Ø2 0.)RBE
Where the last column consists of moment arms which are approximated according to (Li & Todorov] 2004 figure B). T is the length of each muscle, which calculated by approximating the range of each muscle group in Li Todorov| (2004) using the data joint angle ranges and linear relationships with corresponding joint angles as in Teka et al. (2017). Each of the fitted linear functions for muscle lengths and cosine functions for moment arms are available in the provided MATLAB code.
The above system is deterministic model of the 2-link 6-muscle human arm. However, in the main paper consider experiments with both additive and contral multiplicative stochasticities. A statc-space SDE model for
16

---

Under review as a conference paper at ICLR 2020
the human given follows,
81
8
02
02
01
01
02
02
02x2
04x2
Idenct
dt +
(udt + oudv)
Σxx2
dw.
022
ddecct
-02_
ddeact
T8x6
06x2
deace
414
ddcccl
-01
-a5
ddeact
deaas
where, 0 = (0i, 8b]" _M(0)) C(0, 18)) and (u1, u2, M3, u4, u5, u6]
In this system, the control cost coefficient matrix is R 21axo- The state cost matrix is diag[10.r (10.0, 10.0, 0.1, 0.000. 0.001, 0.001, 0.001, 0.001, 0.001].
F SIMULATION PARAMETERS
The precise training/network parameters shown the following table
system
batch size
iterations
A
dt (sec)
T (sec)
h
CartPole
256
2000
0.0005
0.02
[8,
QuadCopter
256
5000
1
1
0.0005
0.02
2
[8, 8]
HumanHand 256
2000
0
0.0005
0.02
2
[8, 8.
where iterations is the number of epochs. 61 to сa represent of the weight of the components the loss function in SMD, À the weights for L2 regularization of neural network. df is the time discretization. T is the time horizon for the task. is the output size of cach LSTM layer (cell state dimension is the same).
G ADDITIONAL PLOTS
G.1 CARTPOLE SWING-UP WITH LOWER CONTROL COST
The plots in fig. 6 show the comparison of the 2FBSDE, FBSDE and iLQG controllers when the restriction on the controls is lowered as compared to that in the main paper. Here, the control cost is set to I = 0.1 as compared R. 0.5 in the main paper. Clearly, the behavior very different. Now, all three controllers try reach the targets as soon as possible by applying the required control effort thus ignoring the presence control multiplicative noise in the system.
G.2 2-LINK 6-MUSCLE HUMAN ARM SINGLE CONTROL TRAJECTORY SAMPLE
Fig. 7 includes single coutrol trajectory well mean and variance for the human arm simulation demonstrate the feasibility of contral trajectories from 2FBSDE.
G.3 QUADCOPTER VELOCITY STATES
Fig. shows the velocity trajectories of the quadcopter experiment in the main paper.
17

---

Under review as a conference paper at ICLR 2020
2FBSDE
FBSDE
iLQG
target
Cart Position (m)
Pole: Angle (rad)
Pole Angulnn Rabel (radts)
Cant Cart Velocity (mbb)
175
175
700
175
770
Time
Time
Figure 6: Comparison of 2FBSDE, FBSDE and iL.QG on Cartpole with control cost of 0.1.
187
5- 230
125
050
182
3 180
3.0
-01
Lime
Figure 7: Single control trajectory well mean and variance of 2FBSDE on Human Arm. The single trajectory demonstrates the feasibility of the resulting controls.
2FBSDE
FBSDE
LQG
target
Velocctyiii5
Y Velocity(mvs))
Veloctyimma
Roll Rate(racos)
Pitch Rate((acls))
Yow Rate(radis) (radis)
Time (5)
Time (5)
Time
Figure 8: Comparison of 2FBSDE, FBSDE and iLQG on the quadcopter velocities and angular rates.
Again the variance of 2FBSDE trajectories are smaller than FBSDE and iLQG.
18

---

Under review as a conference paper at ICLR 2020
2FBSDE
iLQG
Shouklen Angle trad)
2.0
2.0
1.7
Elbow Angle [rac]
Figure 9: Phase plot at control multiplicative standard deviation = 0.1 and additive noise standard deviation 0.11. The yellow circle denotes the starting point, and yellow star denotes the target.
mean
target
Annteelleee (rad)
Mich Kngil (rwdd
Yea
405
Vatoocrr (mu4)
Veecciy (mbs)
Fal Pate treei))
Pitch pate (raats)
Yas Iatt (owits))
Ums (4)
Gmm (4)
Bmm 6c)
Figure 10: Quadcopter state trajectories of 2FBSDE for 3 seconds. The plots demonstrate the controller's capability to reach and hover.
G.4
QUADCOPTER LONGER TIME IIORIZON
Fig. 10 shows quadcopter state trajectories of 2FBSDE for longer time horizon (3 seconds).
G.5 PIIAS PLOT FOR HUMAN ARM SIMULATION
Fig. 9 is phase plot of the human arm experiment experiment at 0.1 and Σ 0.1.L It clearly shows that 2FBSDE can reach the target location and maintain small variance in the trajectory whereas iL.QG diverges from the target.
19

---

Under review as a conference paper at ICLR 2020
H NETWORK INITIALIZATION STRATEGIES
For linear dynamics and cartpole used the Xavier initialization strategy Glorot & Bengio 2010). This considered to standard method for recurrent networks that use tanh activations. This strategy was crucial to allow using large learning rates and allow convergence in ~ 2000 4000 iterations. Without this strategy. convergence extremely slow and prohibits the use for large learning rates.
On the other hand, for high dimensional systems like the quadcopter and human arm, this strategy failed to work. The reason partially explained in the loss function section of this supplementary text. Essentially, random initialization values cause snowballing effect on the propagation of Va,, causing the loss function to diverge and make training impossible as the gradient step in never reached. A simple solution to this problem to use zero initialization for all weights and biases. This prevents Va.1 : from diverging network predictions purely noise driven. This allows computing the loss function without diverging and taking gradient steps to start making meaningful predictions at every time step. Although this allows for training the network for high-dimensional systems, it slows down the process and convergence requires many more iterations. Therefore, further investigation into initialization strategies or other recurrent network architectures would allow improving convergence speed.
FC layers having (n2)) trainable parameters
zero and the state trajectory propagation
I MACHINE INFORMATION
We used Tensorflow Abadi et al. 2015 extensively defining computational graph in fig. D and model training. Because our current implem implementation involves many consecutive CPU-GPU transfers, we decided to implement the CPU version of Tensorflow as the data transfer overhead was very time consuming. The models were trained multiple desktops computers laptops to evaluate diffent models and hyperparameters. An Alienware laptop used with the following specs:
Processor: Intel Core 19-8950HK CPU 2.90GH2 12, Memory: 32GiB.
The desktop computers used have the following specs:
Processor: Intel Xeon(R) CPU E5- 1607v3 @ 3. 3.10GH2x4 Memory: 32GiB
20


Reviews:
[{"review_id": "rJeVbb-bKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary and Decision\n\nThe authors in the paper studied the problem of stochastic optimal control using deep learning. The main contributions of this work can be summarized by introducing an additional backward stochastic differential equation (BSDE) over Pereira et al. (2019), where instead of predicting the gradient of the value function (V_x), the authors predicted the values required to compute the Hamiltonian H(V_x) instead. The authors then compared the results against both closed form solutions of optimal control and a numerical approximate controller by Li and Todorov (2007). However, it is unclear whether this particular modification leads to any incremental benefits over Pereira et al. (2019), nor is it compared against the feedforward networks of Han et al. (2017). Furthermore, one of the main benefits of this modifications the authors claimed is scalability to higher dimensions, yet we do not have any experiments in more than 6 dimensions in the quadcopter case. \n\nWhile I believe the authors have a promising idea, the current paper do not provide enough justification to demonstrate an improvement. Therefore I recommend a weak reject. \n\n\nBackground\n\nIn the classical optimal control set up, it is often very difficult to recover a solution in closed form. Yet at the same time, current numerical methods are far from satisfactory. There are two main approaches to numerically solving optimal control: \n1. Numerically solve the Hamilton-Jacobi-Bellman (HJB) equation for the value function, which is difficult both due to non-linearity and curse of dimensionality. \n2. Use the forward-backward stochastic differential equation (FBSDE) representation of the solution for the HJB equation, but simulation of the backward equation is difficult due to requirement of meeting a terminal condition. \n\nHan et al. (2017) worked around the simulation difficulty by instead training a feedforward network to minimize the error of terminal condition. In the same paper, Han et al. showed this method can solve an HJB equation in 100 dimensions. Pereira et al. (2019) extended this idea to recurrent networks. Bakshi et al. (2017b) introduced the second BSDE to encourage more controlled exploration, which is used in the current paper under review. \n\nThe ultimate goal of this line of work is to solve highly complex non-linear stochastic control problems, where we have no hope of recovering an optimal in closed form. Therefore, any method with efficient approximate computation is highly desirable. In particular, methods involving an approximation with deep networks are quite promising and deserve further exploration. \n\n\nDetailed Comments \n\nThe authors in this paper essentially combined the idea of Bakshi et al. (2017b) and Pereira et al. (2019) in an attempt to improve both existing papers. Having a guided second order controlled BSDE will encourage further exploration, and intuitively this change should make training easier. Furthermore, by introducing the prediction of the quantity \\Omega_t, the authors avoid computing a rank 3 tensor. Hence, I believe this is a promising idea and deserves to be implemented and carefully studied. \n\nThe main concern of this work is that it's not clear whether this method is working. Essentially, I would like to see that despite introducing more complexity to the network compared to Pereira et al. (2019) and Han et al. (2017), the resulting network can still be trained to achieve improved results. It is very unsatisfying to only compare against a closed form solution and a simple approximate control. These experiments can serve to show that the trained network is not behaving erratically, but we cannot conclude any improvements. In fact, even when compared against the approximate control ILQG by Li and Todorov (2007), it's unclear if the current method is better when given the same computation budget. \n\nThe other main concern is scalability to higher dimensions. While computationally cheap, it is unclear whether or not predicting the quantity \\Omega_t does in fact work in high dimensions. This concern is mainly driven by the fact Han et al. (2017) has been able to solve control problems in 100 dimensions, where computing the rank 3 tensor is becoming costly. In this case, comparison against the closed form solution is sufficient to determine whether or not predicting \\Omega_t works in high dimensions. \n\n\nMinor Additional Comments\n\nThere are more minor points I would like to make, but these do not contribute to the review decision. \n1. On page 3, below equation (1), I believe C is a map [0,T] X R^{n_x} X R^{n_u} -> R^{ n_x * (n_w + 1) } \n2. On Page 3, below equation (2), it's strange seeing this notation C^{1,2} for a function of x only; in stochastic control and PDE literature, C^{1,2} typically denotes a function u(t,x) once differentiable in t and twice differentiable in x. \n3. On page 3, below equation (3), we do also require the matrix R to be symmetric? \n4. On page 3, for equation (4), it should be mentioned at some point in this paper that we can only recover an HJB equation without a sup term due to the loss \\ell(x,u) being quadratic in the control u. In general, the HJB equation and the FBSDEs can be much more difficult to work with. \n5. On page 4, it would be more satisfying to cite a more complete collection of works in stochastic control and FBSDEs. In particular, Bismut (1976) and Pardoux and Peng (1990) are seminal works that led up to El Karoui et al. (1997). It would be also nice to briefly mention a huge literature on the PDE methods to control. \n6. On page 5, the notation for \\psi and \\zeta were not introduced. I interpreted from context that these were parameters for predicting \\Gamma and \\Omega, but it would nice to have a definition. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Skg1ZY6utH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n######### Rebuttal Response:\nThanks for the clarifications and especially for updating the formatting. The current state does not convince me to rate the paper as weak accept but I increased my rating to weak reject. \n\n\"Pereira et. al. has shown that a recurrent network architecture using LSTM outperforms the fully connected networks at every time step proposed by Han et. al. in task completion, space and time complexity. Therefore, in this paper we choose to use an LSTM-based network architecture.\"\n\n-> Yes it might be true that a recurrent function approximator does in practice perform better than a feed-forward function approximator. However, in theory a feed-forward network should be sufficient as the value function does not depend on the previous states. Therefore, the question is, why does the LSTM perform better? Does the recurrent nature of the LSTM make the predictions smoother compared to a feed-forward network?  Can any other regularizing scheme be introduced s.t. the feed-forward networks performs equally well?\n\n######### Review:\n\nSummary: \nThe paper builds on the work of Pereira et. al. and uses forward backward stochastic differential equations to learn the Hessian of the Value function Vxx and \\partial _t V_x + 1/2 tr(\\partial_{xx} V_x CC^T). In contrast to the prior work, this paper introduces multiplicative noise for the control and uses second order optimization. The performance is evaluated on different control tasks, e.g., linear system, cartpole, quadcopter & human arm actuated by tendons. \n\nConclusion: \nAll in all, I like the proposed research of combining theoretical approaches and deep learning to perform trajectory optimization and I would like to see much more of this research like this within the ICLR community. Furthermore, I think that the paper has a contribution and that the paper was improved compared to the initial Neurips submission (i.e., adding ILQG as baseline). However, the writeup and formatting is still very much sub-standard and must be improved to make this paper worth publishing. The current write-up is not accessible for the ICLR community and the understandability must be significantly improved (Details are provided below). Therefore, I currently rate this paper as a clear rejection but I am happy to improve the score to 7-8 if the write up is improved during the rebuttal. \n\n\nTheoretical Structure: \nI like the introduction, which covers the topic but might be a bit too long. Maybe you want to shorten the introduction and add an additional related work section at the end. The stochastic control introduction is nice and has the correct level of abstraction for the reader. However, the paper introduces many complex concepts which are not essential for understanding the paper (e.g., filtered probability space etc.). One might want to trade off understandability vs. mathematical rigor especially, if the paper does not rely on these concepts. Furthermore, you might want to make eq 1 more explicit as the multiplicative action noise is not visible from eq 1. Section 3 'A FBSDE Solution to the HJB PDE' is the most problematic section of this paper, which is not understandable for the common ICLR reader. Eq. 6, which just appears without any derivation, is not understandable and the reader has no intuition how to derive this eq. Furthermore, Eq 6 (page. 4) uses notations which is only clearly introduced later within the paper or even the appendix (e.g., Y being the propagated value function, Z being the propagated gradient of the value function is only mentioned in the appendix, i.e., page 12. \\Gamma is only introduced in page 5. Yes, Eq. 7 defines these variables but the style of definition is not standard and one does not expect the variables to be defined in this style.). Could the authors please provide an intuitive derivation of these equation and use clearer notation (Why would one want to abstract V, V_x, V_x, \\mathcal|{H}(V_x) in the first place as these are intuitive for the ICLR community and sufficiently short?) Especially, as this section highlights the difference to the prior works of Pereira et. al., this section should be very clear. Section 4 is clear but should include the loss function as the loss is not trivial and essential for the optimization. Currently, the loss description is buried in the appendix. All in all, the theoretical explanation and the bloated notation should be simplified and every equation should be embedded into an intuitive derivation. Currently these explanations are not understandable without reading the appendix and prior work.  \n\n\nExperiments:\nThe experiments apply 2FBSDE to 4 different control tasks (Linear system, quadcopter, cartpole & human arm) and compare the performance to the prior work of FBSDE and iLQG. The number of baselines and systems is sufficient. However, the paper should provide more evaluations:\n\n(1) Plot the histogram of the obtained cost distributions. \n(2) Plot a single state- and action trajectory (and the action distribution). Using these plots, the level of noise and smoothness and hence the applicability to physical systems can be evaluated. \n(3) Plot the noise free trajectories and show that these mean trajectories reach the desired solution.\n(4) Specify the exact cost function for every experiment\n\nFurther Comments to the individual experiments: \n\nCartpole:\nThe Cartpole iLQG seems to perform much better (swing-up the pendulum faster, don't deviate so much from x=0, much more coherent velocity compared to 2FBSDE, FBSDE). Could the authors please discuss these aspects in more detail and present experiments with longer time-horizons to check whether the proposed method can stabilize the cart at [0, 0, 0, 0]. The current plots don't reach this target state. Your plots also hint that the cartpole does not need to pre-swing the pendulum, which is most likely due to the very low action cost. This selection of action cost significantly simplifies the problem. Could the authors please include a cartpole with higher action cost and show that 2FBSDE can learn to pre-swing the pendulum. \n\nThe quadcopter:\nCould the authors please specify the exact quadcopter dynamics. What kind of abstraction did you model? What are the control inputs? Furthermore, the citation for the dynamics is wrong and puts the supervisor of the master thesis as first author. Furthermore, can the authors please provide longer plots to highlight, which method can stabilize the system.\n\nHuman Arm:\nFor the human arm neither iLQG or 2FBSDE reach the desired target location. Can you explain why no trajectory optimization does reach the desired position. \n\nFormatting:\nPlease rework the formatting such that the inline math does not cause the formatting issues of different line spacings (e.g., sec. 2.1, sec. 5) and irregular whitespaces (e.g., last line of paragraph 2.1 Preliminaries). Please remove the color coding of text for the experiments and make sure that the legends are sufficiently large and include all lines. Currently the legends are missing the target state. You can also extend the figure captions to highlight the conclusion of the plots. Please rework the figures such that the figures do not cause so much whitespace (e.g., Figure 3, 4 & 5). When reconfiguring the plots, you gain space, which can be used for further explanation of the theory. Furthermore, you might want add dotted lines to the confidence intervals as the confidence intervals are important but the differences are not clearly visible from the plots. Also, the labelling in figure 3 seems wrong, the axis labeled cart velocity should be pendulum angle and the pendulum angle axis should be cart velocity. All in all, the formatting can be significantly improved, which is especially bad as this paper is most likely a resubmission from Neurips.  \n\n\nMinor Comments / Questions:\n- 'where l :Rnx×Rnu→R+ is the running cost and C1,23 \\phi :Rnx→R+ is the terminal state cost.' Is l and \\phi of class C^{1,2} or only one of them? The notation is confusing and should be simplified.\n- Can you comment on how important this multiplicative noise in physical system? ¬¬¬\n- Why are you using a LSTM instead of a simple feed-forward neural network as the ff-nn should be sufficient to model V(x) as the value function is not recurrent. Have you tried using a simple ff-nn?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeaEz_ptS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:** The paper contains a method tailored to a certain kind of SOC problems involving multiplicative noise. The central idea is to use a recurrent network to transform the observations into a representation that can be used with solvers specifically tailored towards that class of problems.\n\n**Decision:** I recommend to accept the paper for publication.\n\n**Arguments for decision:** The paper clearly adresses an important problem and poroposes a method capable of solving it. The method appears to be theoretically founded and the experimental validation seems solid. The relevance of the method is there as the problem class is prevalent in practical applications. The venue is a good fit as well, as the focus is the representation of a control problem in a way that allows more efficient solutions.\n\n**Feedback for improvement:**\n\n- The type setting could be improved at times. E.g. below equation (1).\n- I feel that the term \"exploration\" is overloaded. While it serves as an explicit mean to reduced the sample complexity of methods in RL, it appears to be about avoiding premature convergence in this work. I am too unfamiliar with the relevant SOC literature to judge how well the term fits, but coming from a ML background I stumbled over this expression.\n- Some of the experimental details, e.g. the exact choice of time discretisation, don't appear motivated well. \n- The paper needs to respect [1, 2] in the related work and show relations. From the perspective of learning state representations for optimal control, both works are relevant.\n- Is it necessary to start the discussion from the continuous case? While I appreciate the elegance of starting out with a continuous problem and then discretising at the last step, it felt like a barrier to understanding in my case, as my understanding of continuous optimal control is limited–and I feel the audience of ICLR might have the same problem.\n\n**References:**\n\n[1] Watter, Manuel, et al. \"Embed to control: A locally linear latent dynamics model for control from raw images.\" *Advances in neural information processing systems*. 2015.\n[2] Banijamali, Ershad, et al. \"Robust locally-linear controllable embedding.\" *arXiv preprint arXiv:1710.05373*(2017).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A nice paper, but quite some unclarities; it's unclear  in particular if the paper improves w.r.t. SOTA.  Esp. scaling is an issue here. Also, the understandability is below par and more work can make this into an acceptable submission.
Decision: Reject