Neural Solvers for Fast and Accurate Numerical Optimal Control

Synthesizing optimal controllers for dynamical systems often involves solving optimization problems with hard real–time constraints . These constraints restrict the class of numerical methods that can be applied : computationally expensive but accurate numerical routines often have to be replaced with fast and inaccurate methods , trading inference time for solution accuracy . This paper provides techniques to improve the quality of optimized control policies given a fixed computational budget . We achieve the above via a hypersolvers ( Poli et al. , 2020a ) approach , which hybridizes a differential equation solver and a neural network . The performance is evaluated in direct and receding–horizon optimal control tasks , where the proposed approach shows consistent Pareto improvements in solution accuracy and control performance . 1 INTRODUCTION . Optimal control of complex , high–dimensional systems requires the implementation of computationally expensive numerical methods for differential equations ( Pytlak , 2006 ; Rao , 2009 ) . Here , real–time and hardware constraints preclude the use of accurate and expensive methods , forcing instead the application of cheaper and less accurate algorithms . While the paradigm of optimal control has successfully been applied in various domains ( Vadali et al. , 1999 ; Lewis et al. , 2012 ; Zhang et al. , 2016 ) , improving accuracy while satisfying computational budget constraints is still a great challenge ( Ross & Fahroo , 2006 ; Baotić et al. , 2008 ) . To alleviate such computational limitations , we detail a procedure for offline optimization and subsequent online application of hypersolvers ( Poli et al. , 2020a ) to optimal control problems . These hybrid solvers aim to achieve the accuracy of higher–order methods by augmenting numerical results of a base solver with a learning component trained to approximate local truncation residuals . When the overhead of a single forward–pass of the learning component is kept sufficiently small , hypersolvers improve the computation–accuracy Pareto front of low–order explicit solvers ( Butcher , 1997 ) . However , direct application of hybrid solvers to controlled dynamical system involves learning truncation residuals on the higher–dimensional space of state and control inputs . To extend the range of applicability of hypersolvers to controlled dynamical systems , we propose two pretraining strategies designed to improve , in the set of admissible control inputs , on the average or worst–case hypersolver solution . With the proposed methodology , we empirically show that Pareto front improvements of hypersolvers hold even for optimal control tasks . In particular , we then carry out performance and generalization evaluations in direct and model predictive control tasks . Here , we confirm Pareto front improvements in terms of solution accuracy and subsequent control performance , leading to higher quality control policies and lower control losses . 2 NUMERICAL OPTIMAL CONTROL . We consider control of general nonlinear systems of the form ẋ ( t ) = f ( t , x ( t ) , uθ ( t ) ) x ( 0 ) = x0 ( 1 ) with state x ∈ X ⊂ Rnx , input uθ ∈ U ⊂ Rnu defined on a compact time domain T : = [ t0 , T ] where θ is a finite set of free parameters of the controller . Solutions of ( 1 ) are denoted with x ( t ) = Φ ( x ( s ) , s , t ) for all s , t ∈ T . Given some objective function J : X × U → R ; x0 , uθ 7→ J ( x0 , uθ ( t ) ) and a distribution ρ0 ( x0 ) of initial conditions with support in X , we consider the following nonlinear program , constrained to the system dynamics : min uθ ( t ) Ex0∼ρ0 ( x0 ) [ J ( x0 , uθ ( t ) ) ] subject to ẋ ( t ) = f ( t , x ( t ) , uθ ( t ) ) x ( 0 ) = x0 t ∈ T ( 2 ) where the controller parameters θ are optimized . We will henceforth omit the subscript θ and write u ( t ) = uθ ( t ) . Since analytic solutions of ( 2 ) exist only for limited classes of systems and objectives , numerical solvers are often applied to iteratively find a solution . For these reasons , problem 2 is often referred to as numerical optimal control . Direct optimal control If the problem ( 2 ) is solved offline by directly optimizing over complete trajectories , we call it direct optimal control . The infinite–dimensional optimal control problem is time–discretized and solved numerically : the obtained control policy is then applied to the real target system without further optimization . Model predictive control Also known in the literature as receding horizon control , Model Predictive Control ( MPC ) is a class of flexible control algorithms capable of taking into consideration constraints and nonlinearities ( Mayne & Michalska , 1988 ; Garcia et al. , 1989 ) . MPC considers finite time windows which are then shifted forward in a receding manner . The control problem is then solved for each window by iteratively forward–propagating trajectories with numerical solvers i.e . predicting the set of future trajectories with a candidate controller u ( t ) and then adjusting it iteratively to optimize the cost function J ( further details on the MPC formulation in Appendix B.2 ) . The optimization is reiterated online until the end of the control time horizon . 2.1 SOLVER RESIDUALS . Given nominal solutions Φ of ( 1 ) we can define the residual of a numerical ODE solver as the normalized error accumulated in a single step size of the method , i.e . Rk = R ( tk , x ( tk ) , u ( tk ) ) = 1 p+1 [ Φ ( x ( tk ) , tk , tk+1 ) − x ( tk ) − ψ ( tk , x ( tk ) , u ( tk ) ) ] ( 3 ) where is the step size and p is the order of the numerical solver corresponding to ψ . From the definition of residual in ( 3 ) , we can define the local truncation error ek : = ∥∥ p+1Rk∥∥2 which is the error accumulated in a single step ; while the global truncation error Ek = ‖x ( tk ) − xk‖2 represents the error accumulated in the first k steps of the numerical solution . Given a p–th order explicit solver , we have ek = O ( p+1 ) and Ek = O ( p ) ( Butcher , 1997 ) . 3 HYPERSOLVERS FOR OPTIMAL CONTROL . We extend the range of applicability of hypersolvers ( Poli et al. , 2020a ) to controlled dynamical systems . In this Section we discuss the proposed hypersolver architectures and pre–training strategies of the proposed hypersolver methodology for numerical optimal control of controlled dynamical systems . 3.1 HYPERSOLVERS . Given a p–order base solver update map ψ , the corresponding hypersolver is the discrete iteration xk+1 = xk + ψ ( tk , xk , uk ) ︸ ︷︷ ︸ base solver step + p+1 gω ( tk , xk , uk ) ︸ ︷︷ ︸ approximator ( 4 ) where gω ( tk , xk , uk ) is some o ( 1 ) parametric function with free parameters ω . The core idea is to select gω as some function with universal approximation properties and fit the higher-order terms of the base solver by explicitly minimizing the residuals over a set of state and control input samples . This procedure leads to a reduction of the overall local truncation error ek , i.e . we can improve the base solver accuracy with the only computational overhead of evaluating the function gω . It is also proven that , if gω is a δ–approximator of R , i.e . ∀k ∈ N≤K ‖R ( tk , x ( tk ) , u ( tk ) ) − gω ( tk , x ( tk ) , u ( tk ) ) ‖2 ≤ δ ( 5 ) then ek ≤ o ( δ p+1 ) , where δ > 0 depends on the hypersolver training results ( Poli et al. , 2020a , Theorem 1 ) . This result practically guarantees that if gω is a good approximator for R , i.e . δ 1 , then the overall local truncation error of the hypersolved ODE is significantly reduced with guaranteed upper bounds . 3.2 OPTIMAL CONTROL OPTIMIZATION WITH HYPERSOLVERS . Our approach relies on the pre–trained hypersolver model for obtaining solutions to the trajectories of the optimal control problem ( 2 ) . After the initial training stage , control policies are numerically optimized to minimize the cost function J ( see Appendix B.3 for further details ) . Figure 1 shows an overview of the proposed approach consisting in pre–training and system control . 4 HYPERSOLVER PRE–TRAINING AND ARCHITECTURES . We introduce in Section 4.1 loss functions which are used in the proposed pre–training methods of Section 4.2 and Section 4.3 . We also check the generalization properties of hypersolvers with different architectures in Section 4.4 . In Section 4.5 we introduce multi–stage hypersolvers in which an additional first–order learned term is employed for correcting errors in the vector field . 4.1 LOSS FUNCTIONS . Residual fitting Training the hypersolver on a single nominal trajectory { x ( tk ) } k results in a supervised learning problem where we minimize point–wise the Euclidean distance between the residual ( 3 ) and the output of gω , resulting in an optimization problem minimizing a loss function ` of the form ` ( t , x , u ) = 1 K K−1∑ k=0 ‖R ( tk , x ( tk ) , u ( tk ) ) − gω ( tk , x ( tk ) , u ( tk ) ) ‖2 ( 6 ) which is also called residual fitting since the target of gw is the residual R. Trajectory fitting The optimization can also be carried out via trajectory fitting as following ` ( t , x , u ) = 1 K K−1∑ k=0 ‖x ( tk+1 ) , xk+1‖2 ( 7 ) where x ( tk+1 ) corresponds to the exact one–step trajectory and xk+1 is its approximation , derived via ( 4 ) for standard hypersolvers or via ( 12 ) for their multi–stage counterparts . This method can also be used to contain the global truncation error in the T domain . We will refer to ` as a loss function of either residual or trajectory fitting types ; we note that these loss functions may also be combined depending on the application . The goal is to train the hypersolver network to explore the state–control spaces so that it can effectively minimize the truncation error . We propose two methods with different purposes : stochastic exploration aiming at minimizing the average truncation error and active error minimization whose goal is to reduce the maximum error i.e. , due to control inputs yielding high losses . 4.2 STOCHASTIC EXPLORATION . Stochastic exploration aims to minimize the average error of the visited state–controller space i.e. , to solve the following optimization problem min w E ( x , u ) ∼ξ ( x , u ) ` ( t , x , u ) ( 8 ) where ξ ( x , u ) is a distribution with support in X × U of the state and controller spaces and ` is the training loss function . In order to guarantee sufficient exploration of the state–controller space , we use Monte Carlo sampling ( Robert & Casella , 2013 ) from the given distribution . In particular , batches of initial conditions { xi0 } , { ui0 } are sampled from ξ and the loss function ` is calculated with the given system and step size . We then perform backpropagation for updating the parameters of the hypersolver using a stochastic gradient descent ( SGD ) algorithm e.g. , Adam ( Kingma & Ba , 2017 ) and repeat the procedure for every training epoch . Figure 2 shows pre–training results with stochastic exploration for different step sizes ( see Appendix C.2 ) . We notice how higher residual values correspond to higher absolute values of control inputs . Many systems in practice are subject to controls that are constrained in magnitude either due to physical limitations of the actuators or safety restraints of the workspace . This property allows us to design an exploration strategy that focuses on worst-case scenarios i.e . largest control inputs .

Summary: The paper descibes the application of hypersolves, which warm-start ODE solvers with neural nets that have distilled many precomputed solutions, to optimal control ODEs.  This warm-starting improves downstream accuracy when the computational budget is fixed.  This method is applied for trajectory optimziation for pendulum swing-up and MPC for cartpole and quadcopter systems.