Neural Solvers for Fast and Accurate Numerical Optimal Control

Synthesizing optimal controllers for dynamical systems often involves solving optimization problems with hard real–time constraints . These constraints restrict the class of numerical methods that can be applied : computationally expensive but accurate numerical routines often have to be replaced with fast and inaccurate methods , trading inference time for solution accuracy . This paper provides techniques to improve the quality of optimized control policies given a fixed computational budget . We achieve the above via a hypersolvers ( Poli et al. , 2020a ) approach , which hybridizes a differential equation solver and a neural network . The performance is evaluated in direct and receding–horizon optimal control tasks , where the proposed approach shows consistent Pareto improvements in solution accuracy and control performance . 1 INTRODUCTION . Optimal control of complex , high–dimensional systems requires the implementation of computationally expensive numerical methods for differential equations ( Pytlak , 2006 ; Rao , 2009 ) . Here , real–time and hardware constraints preclude the use of accurate and expensive methods , forcing instead the application of cheaper and less accurate algorithms . While the paradigm of optimal control has successfully been applied in various domains ( Vadali et al. , 1999 ; Lewis et al. , 2012 ; Zhang et al. , 2016 ) , improving accuracy while satisfying computational budget constraints is still a great challenge ( Ross & Fahroo , 2006 ; Baotić et al. , 2008 ) . To alleviate such computational limitations , we detail a procedure for offline optimization and subsequent online application of hypersolvers ( Poli et al. , 2020a ) to optimal control problems . These hybrid solvers aim to achieve the accuracy of higher–order methods by augmenting numerical results of a base solver with a learning component trained to approximate local truncation residuals . When the overhead of a single forward–pass of the learning component is kept sufficiently small , hypersolvers improve the computation–accuracy Pareto front of low–order explicit solvers ( Butcher , 1997 ) . However , direct application of hybrid solvers to controlled dynamical system involves learning truncation residuals on the higher–dimensional space of state and control inputs . To extend the range of applicability of hypersolvers to controlled dynamical systems , we propose two pretraining strategies designed to improve , in the set of admissible control inputs , on the average or worst–case hypersolver solution . With the proposed methodology , we empirically show that Pareto front improvements of hypersolvers hold even for optimal control tasks . In particular , we then carry out performance and generalization evaluations in direct and model predictive control tasks . Here , we confirm Pareto front improvements in terms of solution accuracy and subsequent control performance , leading to higher quality control policies and lower control losses . 2 NUMERICAL OPTIMAL CONTROL . We consider control of general nonlinear systems of the form ẋ ( t ) = f ( t , x ( t ) , uθ ( t ) ) x ( 0 ) = x0 ( 1 ) with state x ∈ X ⊂ Rnx , input uθ ∈ U ⊂ Rnu defined on a compact time domain T : = [ t0 , T ] where θ is a finite set of free parameters of the controller . Solutions of ( 1 ) are denoted with x ( t ) = Φ ( x ( s ) , s , t ) for all s , t ∈ T . Given some objective function J : X × U → R ; x0 , uθ 7→ J ( x0 , uθ ( t ) ) and a distribution ρ0 ( x0 ) of initial conditions with support in X , we consider the following nonlinear program , constrained to the system dynamics : min uθ ( t ) Ex0∼ρ0 ( x0 ) [ J ( x0 , uθ ( t ) ) ] subject to ẋ ( t ) = f ( t , x ( t ) , uθ ( t ) ) x ( 0 ) = x0 t ∈ T ( 2 ) where the controller parameters θ are optimized . We will henceforth omit the subscript θ and write u ( t ) = uθ ( t ) . Since analytic solutions of ( 2 ) exist only for limited classes of systems and objectives , numerical solvers are often applied to iteratively find a solution . For these reasons , problem 2 is often referred to as numerical optimal control . Direct optimal control If the problem ( 2 ) is solved offline by directly optimizing over complete trajectories , we call it direct optimal control . The infinite–dimensional optimal control problem is time–discretized and solved numerically : the obtained control policy is then applied to the real target system without further optimization . Model predictive control Also known in the literature as receding horizon control , Model Predictive Control ( MPC ) is a class of flexible control algorithms capable of taking into consideration constraints and nonlinearities ( Mayne & Michalska , 1988 ; Garcia et al. , 1989 ) . MPC considers finite time windows which are then shifted forward in a receding manner . The control problem is then solved for each window by iteratively forward–propagating trajectories with numerical solvers i.e . predicting the set of future trajectories with a candidate controller u ( t ) and then adjusting it iteratively to optimize the cost function J ( further details on the MPC formulation in Appendix B.2 ) . The optimization is reiterated online until the end of the control time horizon . 2.1 SOLVER RESIDUALS . Given nominal solutions Φ of ( 1 ) we can define the residual of a numerical ODE solver as the normalized error accumulated in a single step size of the method , i.e . Rk = R ( tk , x ( tk ) , u ( tk ) ) = 1 p+1 [ Φ ( x ( tk ) , tk , tk+1 ) − x ( tk ) − ψ ( tk , x ( tk ) , u ( tk ) ) ] ( 3 ) where is the step size and p is the order of the numerical solver corresponding to ψ . From the definition of residual in ( 3 ) , we can define the local truncation error ek : = ∥∥ p+1Rk∥∥2 which is the error accumulated in a single step ; while the global truncation error Ek = ‖x ( tk ) − xk‖2 represents the error accumulated in the first k steps of the numerical solution . Given a p–th order explicit solver , we have ek = O ( p+1 ) and Ek = O ( p ) ( Butcher , 1997 ) . 3 HYPERSOLVERS FOR OPTIMAL CONTROL . We extend the range of applicability of hypersolvers ( Poli et al. , 2020a ) to controlled dynamical systems . In this Section we discuss the proposed hypersolver architectures and pre–training strategies of the proposed hypersolver methodology for numerical optimal control of controlled dynamical systems . 3.1 HYPERSOLVERS . Given a p–order base solver update map ψ , the corresponding hypersolver is the discrete iteration xk+1 = xk + ψ ( tk , xk , uk ) ︸ ︷︷ ︸ base solver step + p+1 gω ( tk , xk , uk ) ︸ ︷︷ ︸ approximator ( 4 ) where gω ( tk , xk , uk ) is some o ( 1 ) parametric function with free parameters ω . The core idea is to select gω as some function with universal approximation properties and fit the higher-order terms of the base solver by explicitly minimizing the residuals over a set of state and control input samples . This procedure leads to a reduction of the overall local truncation error ek , i.e . we can improve the base solver accuracy with the only computational overhead of evaluating the function gω . It is also proven that , if gω is a δ–approximator of R , i.e . ∀k ∈ N≤K ‖R ( tk , x ( tk ) , u ( tk ) ) − gω ( tk , x ( tk ) , u ( tk ) ) ‖2 ≤ δ ( 5 ) then ek ≤ o ( δ p+1 ) , where δ > 0 depends on the hypersolver training results ( Poli et al. , 2020a , Theorem 1 ) . This result practically guarantees that if gω is a good approximator for R , i.e . δ 1 , then the overall local truncation error of the hypersolved ODE is significantly reduced with guaranteed upper bounds . 3.2 OPTIMAL CONTROL OPTIMIZATION WITH HYPERSOLVERS . Our approach relies on the pre–trained hypersolver model for obtaining solutions to the trajectories of the optimal control problem ( 2 ) . After the initial training stage , control policies are numerically optimized to minimize the cost function J ( see Appendix B.3 for further details ) . Figure 1 shows an overview of the proposed approach consisting in pre–training and system control . 4 HYPERSOLVER PRE–TRAINING AND ARCHITECTURES . We introduce in Section 4.1 loss functions which are used in the proposed pre–training methods of Section 4.2 and Section 4.3 . We also check the generalization properties of hypersolvers with different architectures in Section 4.4 . In Section 4.5 we introduce multi–stage hypersolvers in which an additional first–order learned term is employed for correcting errors in the vector field . 4.1 LOSS FUNCTIONS . Residual fitting Training the hypersolver on a single nominal trajectory { x ( tk ) } k results in a supervised learning problem where we minimize point–wise the Euclidean distance between the residual ( 3 ) and the output of gω , resulting in an optimization problem minimizing a loss function ` of the form ` ( t , x , u ) = 1 K K−1∑ k=0 ‖R ( tk , x ( tk ) , u ( tk ) ) − gω ( tk , x ( tk ) , u ( tk ) ) ‖2 ( 6 ) which is also called residual fitting since the target of gw is the residual R. Trajectory fitting The optimization can also be carried out via trajectory fitting as following ` ( t , x , u ) = 1 K K−1∑ k=0 ‖x ( tk+1 ) , xk+1‖2 ( 7 ) where x ( tk+1 ) corresponds to the exact one–step trajectory and xk+1 is its approximation , derived via ( 4 ) for standard hypersolvers or via ( 12 ) for their multi–stage counterparts . This method can also be used to contain the global truncation error in the T domain . We will refer to ` as a loss function of either residual or trajectory fitting types ; we note that these loss functions may also be combined depending on the application . The goal is to train the hypersolver network to explore the state–control spaces so that it can effectively minimize the truncation error . We propose two methods with different purposes : stochastic exploration aiming at minimizing the average truncation error and active error minimization whose goal is to reduce the maximum error i.e. , due to control inputs yielding high losses . 4.2 STOCHASTIC EXPLORATION . Stochastic exploration aims to minimize the average error of the visited state–controller space i.e. , to solve the following optimization problem min w E ( x , u ) ∼ξ ( x , u ) ` ( t , x , u ) ( 8 ) where ξ ( x , u ) is a distribution with support in X × U of the state and controller spaces and ` is the training loss function . In order to guarantee sufficient exploration of the state–controller space , we use Monte Carlo sampling ( Robert & Casella , 2013 ) from the given distribution . In particular , batches of initial conditions { xi0 } , { ui0 } are sampled from ξ and the loss function ` is calculated with the given system and step size . We then perform backpropagation for updating the parameters of the hypersolver using a stochastic gradient descent ( SGD ) algorithm e.g. , Adam ( Kingma & Ba , 2017 ) and repeat the procedure for every training epoch . Figure 2 shows pre–training results with stochastic exploration for different step sizes ( see Appendix C.2 ) . We notice how higher residual values correspond to higher absolute values of control inputs . Many systems in practice are subject to controls that are constrained in magnitude either due to physical limitations of the actuators or safety restraints of the workspace . This property allows us to design an exploration strategy that focuses on worst-case scenarios i.e . largest control inputs .

Summary: This paper extends the hypersolvers framework (Poli et al., 2020a) to the setting of optimal control. The hypersolvers framework combines a cheap numerical dynamical system solver with a neural network trained to approximate the truncation error of this solver, which together yield a cheap and relatively accurate solver. This paper shows how to utilize the hypersolvers framework within a controlled dynamical system by exploring different choices of loss functions, training methods, and network architectures for hypersolver pre-training. The authors further provide an augmentation to this framework, called multi-stage hypersolvers, to further account for misspecifications of the dynamics. Via experiments, the authors then demonstrate the performance of optimal control strategies (direct optimal control and MPC) utilizing hypersolvers in different settings (notably the pendulum, cart-pole, and quadcopter settings). They show e.g. that even in cases where their method receives misspecified dynamics, they are able to achieve control performance that is comparable to that obtained in cases where the underlying solver had knowledge of the correct dynamics.