Safe Pontryagin Differentiable Programming

1 Introduction . Safety is usually a priority in the deployment of a learning or control algorithm to real-world systems . For a physical system ( agent ) , safety is normally given in various constraints on system states and inputs , which must not be violated by the algorithm at any stage of the learning and control process , otherwise will cause irrevocable or unacceptable failure/damage . Those systems are referred to as safety-critical . The constraints in a safety-critical system can include the immediate ones , which are directly imposed on the system state and input at certain or all-time instances , and the long-term ones , which are defined on the trajectory of system states and inputs over a long period . Compared to the abundant results that focus on system optimality [ 1–3 ] , systematic and principled treatments for safety-critical learning and control problems seem largely insufficient , particularly in the following gaps ( detailed in Section 1.1 ) . First , existing safety strategies are either too conservative , which may restrict the task performance , or violation-tolerable , which only pursues the near-constraint guarantee and thus are not strictly constraint-respecting . Second , a systematic safety paradigm capable of handling different types of constraints , including system state and input ( or mixed ) , immediate , or/and long-term constraints , is still lacked . Third , some existing safety strategies suffer from huge computational- and data- complexity , difficult to be integrated into any differentiable programming frameworks to solve large-scale learning and continuous control tasks . To address the above research gaps , this paper aims to develop a safe differentiable programming framework with the following key capabilities . First , the framework provides a systematic treatment for different types of constraints in a safety-critical problem ; second , it attains provable safety- and accuracy- guarantees throughout the learning and control process ; third , it is flexible to perform safe learning of any unknown aspects of a constrained decision-making system , including policy , dynamics , state and input constraints , and control cost ; finally , it can be integrated to any differentiable programming framework to efficiently solve large-scale safe learning and control tasks . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1.1 Related Work . In machine learning and control fields , safety has been defined by different criteria , such as worstcase [ 4 , 5 ] , risk-sensitive [ 6 ] , ergodicity [ 7 ] , robust [ 8 , 9 ] , etc. , most of which are formulated by directly altering an objective function [ 10 ] . In this paper , we focus only on constrained learning and control problems , where constraints are explicitly formulated and must be satisfied . We categorize existing techniques into at-convergence safety methods , which only concern constraint satisfaction at convergence , or in-progress safety methods , which attempt to ensure constraint satisfaction during the entire optimization process . At-convergence safety methods . In reinforcement learning ( RL ) , a constrained agent is typically formulated as a Constrained Markov Decision Process ( CMDP ) [ 11 ] , seeking a policy that not only optimizes a reward but also satisfies an upper bound for a cumulative cost . A common strategy [ 12– 17 ] to solve CMDPs is to use the primal-dual method , by establishing the unconstrained Lagrangian and performing saddle-point optimization . In deep learning , the primal-dual method has been recently used [ 18 ] to train deep neural networks with constraints . In control , the primal-dual method has been used to solve constrained optimal control ( constrained trajectory ) problems [ 19–21 ] . While proved to satisfy constraints at convergence [ 22 , 23 ] , the primal-dual type methods can not guarantee constraint satisfaction during optimization , as shown in [ 13 , 24 ] , thus are not suitable for safety-critical tasks . In-progress safety methods . To enforce safety during training , [ 25 ] and [ 26 ] solve CMDPs by introducing additional constraints into the Trust Region Policy Optimization ( TRPO ) [ 27 ] . Since these methods only obtain the ‘ near constraint ’ guarantee , constraint violation is not fully eliminated . Another line of constrained RL [ 24 , 28–30 ] leverages the Lyapunov theory [ 31 ] to bound behavior of an agent . But how to choose a valid Lyapunov function for general tasks is still an open problem to date [ 32 ] , particularly for constrained RL , since it requires a Lyapunov function to be consistent with the constraints and to permit optimal policies [ 28 ] . Some other work also attempts to handle immediate constraints — the constraints imposed on agent state and input at any time . In [ 33 ] , a safe exploration scheme is proposed to produce a safe reachable region ; and it only considers finite state space . [ 34 ] develops a method that learns safety constraints and then optimizes a reward within the certified safe region ; the method defines constraints purely on agent state and thus may not be readily applicable to mixed state-input constraints . In control , in-progress safety can be achieved via two model-based frameworks : reachability theory [ 35 ] and control barrier functions [ 36 , 37 ] . Safe control based on reachability theory [ 35 , 38–40 ] explicitly considers adversarial factors and seeks a strategy that maintains the constraints despite the adversarial factors . This process typically requires solving the Hamilton-Jacobi-Isaacs equations [ 41 ] , which become computationally difficult for high-dimensional systems [ 35 ] . Control barrier functions [ 36 , 37 ] constrain a system only on safety boundaries , making it a less-conservative strategy for safety-critical tasks [ 42–44 ] . Most of the methods consider affine dynamics and directly use the given constraint function as a control barrier function . Such a choice could be problematic when a system is uncontrollable at the boundary of the sublevel set . Thus , how to find a valid control barrier function is still an ongoing research topic [ 45–47 ] . The above two control safety frameworks favorably focus on pure state constraints and can not be readily extended to other constraints , such as mixed state-input constraints or the cumulative constraints defined on the system trajectory . Interior-point methods and control . Interior-point methods ( IPMs ) [ 48–50 ] solve constrained optimization by sequentially finding solutions to unconstrained problems with the objective combining the original objective and a barrier that prevents from leaving the feasible regions . IPMs have been used for constrained linear quadratic regular ( LQR ) control in [ 51–57 ] . While IPMs for nonlinear constrained optimal control are studied in [ 58–62 ] , they mostly focus on developing algorithms to solve the unconstrained approximation ( from the perturbed KKT conditions ) and lack of performance analysis . Most recently , [ 63 ] uses the IPM to develop a zero-th order non-convex optimization method ; and [ 64 ] uses IPMs to solve reinforcement learning with only cumulative constraints . Despite the promise of the trend , the theoretical results and systematic algorithms regarding the differentiability of general constrained control systems based on IPMs have not been studied and established . Differentiable projection layer . In machine learning , a recent line of work considers embedding a differentiable projection layer [ 65–67 ] into a general training process to ensure safety . Particularly , [ 67 ] and [ 66 ] enforce safety by constructing a dedicated projection layer , which projects the unsafe actions outputted from a neural policy into a safe region ( satisfying safety constraints ) . This projection layer is a differentiable convex layer [ 68 , 69 ] , which can be trained end-to-end . In [ 65 ] , safety is defined as robustness in the case of the worst adversarial disturbance , and the set of robust policies is solved by classic robust control ( solving LMIs ) . An RL neural policy with a differentiable projection layer is learned such that the action from the neural policy lies in the robust policy set . Different from the above work , Safe PDP does not enforce safety by projection ; instead , Safe PDP uses barrier functions to guarantee safety constraint satisfaction . More importantly , we have shown , in both theory and experiments , that with barrier functions , differentiability can also be attained . Sensitivity analysis and differentiable MPCs . Other work related to Safe PDP includes the recent results for sensitivity analysis [ 70 ] , which focuses on differentiation of a solution to a general nonlinear program , and differentiable MPCs [ 68 ] , which is based on differentiable quadratic programming . In long-horizon control settings , directly applying [ 70 ] and [ 68 ] can be inefficient : the complexity of [ 68 , 70 ] for differentiating a solution to a general optimal control system is at least O ( T 2 ) ( T is the time horizon ) due to computing the inverse of Jacobian of the KKT conditions . Since an optimal control system has more sparse structures than general nonlinear or quadratic programs , by exploiting those structures and proposing the Auxiliary Control System , Safe PDP enjoys the complexity of O ( T ) for differentiating a solution to a general control system . Such advantages have been discussed and shown in the foundational PDP work [ 71 ] and will also be shown later ( in Section 8 ) in this paper . 1.2 Paper Contributions . We propose a safe differentiable programming methodology named as Safe Pontryagin Differentiable Programming ( Safe PDP ) . Safe PDP provides a systematic treatment of different types of system constraints , including state and inputs ( or mixed ) , immediate , and long-term constraints , with provable safety- and performance-guarantee . Safe PDP is also a unified differentiable programming framework , which can be used to efficiently solve a broad class of safety-critical learning and control tasks . In the spirit of interior-point methods , Safe PDP incorporates different types of system constraints into control cost and loss through barrier functions , approximating a constrained control system and task using their unconstrained counterparts . Contributions of Safe PDP are theoretical and algorithmic . Theoretically , we prove in Theorem 2 and Theorem 3 that ( I ) not only a solution but also the gradient of the solution can be safely approximated by solving a more efficient unconstrained counterpart ; ( II ) any intermediate results throughout the approximation and optimization are strictly safe , that is , never violating the original system/task constraints ; and ( III ) the approximations for both solution and its gradient can be controlled for arbitrary accuracy by barrier parameters . Arithmetically , ( IV ) we prove in Theorem 1 that if a constrained control system is differentiable , the gradient of its trajectory is a globally unique solution to an Auxiliary Control System [ 71 ] , which can be solved efficiently with the complexity of only O ( T ) , T is control horizon ; ( V ) in Section 7 , we experimentally demonstrate the capability of Safe PDP for efficiently solving various safety-critical learning and control problems , including safe neural policy optimization , safe motion planning , learning MPCs from demonstrations . 2 Safe PDP Problem Formulation . Consider a class of constrained optimal control systems ( models ) Σ ( θ ) , which are parameterized by a tunable parameter θ ∈ Rr in its control cost function , dynamics , initial condition , and constraints : Σ ( θ ) : control cost : J ( θ ) = ∑T−1 t=0 ct ( xt , ut , θ ) + cT ( xT , θ ) subject to dynamics : xt+1 = f ( xt , ut , θ ) with x0 = x0 ( θ ) , ∀t , terminal constraints : gT ( xT , θ ) ≤ 0 , hT ( xT , θ ) = 0 , path constraints : gt ( xt , ut , θ ) ≤ 0 , ht ( xt , ut , θ ) = 0 , ∀t . ( 1 ) Here , xt ∈ Rn is the system state ; ut ∈ Rm is the control input ; ct : Rn × Rm × Rr → R and cT : Rn × Rr → R are the stage and final costs , respectively ; f : Rn × Rm × Rr → Rn is the dynamics with initial state x0 = x0 ( θ ) ∈ Rn ; t = 0 , 1 , ... , T is the time step with T the time horizon ; gT : Rn × Rr → RqT and hT : Rn × Rr → RsT are the final inequality and equality constraints , respectively ; gt : Rn×Rm×Rr → Rqt and ht : Rn×Rm×Rr → Rst are the immediate inequality and equality constraints at time t , respectively . All inequalities ( here and below ) are entry-wise . We consider that all functions in Σ ( θ ) are three-times continuously differentiable ( i.e. , C3 ) with respect to its arguments . Although we here have parameterized all aspects of Σ ( θ ) , for a specific application ( see Section 7 ) , one only needs to parameterize the unknown aspects in Σ ( θ ) and keep others given . Any unknown aspects in Σ ( θ ) can be implemented by differentiable neural networks . For a given θ , Σ ( θ ) produces a trajectory ξθ = { xθ0 : T , uθ0 : T−1 } by solving the following Problem B ( θ ) : ξθ = { x θ 0 : T , u θ 0 : T−1 } ∈ arg min { x0 : T , u0 : T−1 } J ( θ ) subject to xt+1 = f ( xt , ut , θ ) with x0 = x0 ( θ ) , ∀t , gT ( xT , θ ) ≤ 0 and hT ( xT , θ ) = 0 , gt ( xt , ut , θ ) ≤ 0 and ht ( xt , ut , θ ) = 0 , ∀t . B ( θ ) Here , we use ∈ since ξθ to Problem B ( θ ) may not be unique in general , thus constituting a solution set { ξθ } . We will discuss the existence and uniqueness of { ξθ } in Section 4 . For a specific task , we aim to find a specific model Σ ( θ∗ ) , i.e , searching for a specific θ∗ , such that its trajectory ξθ∗ from B ( θ ∗ ) meets the following two given requirements . First , ξθ∗ minimizes a given task loss ` ( ξθ , θ ) ; and second , ξθ∗ satisfies the given task constraints Ri ( ξθ , θ ) ≤ 0 , i = 1 , 2 , ... , l. Note that , we need to distinguish between the two types of objectives : task loss ` ( ξθ , θ ) and control cost J ( θ ) , and also the two types of constraints : task constraints Ri ( ξθ , θ ) and model constraints gt ( θ ) . In fact , J ( θ ) and gt ( θ ) in Σ ( θ ) are unknown and parameterized by θ and can represent the unknown inherent aspects of a physical agent , while ` ( ξθ , θ ) and Ri ( ξθ , θ ) are given and known depending on the specific task ( they also explicitly depend on θ since θ needs to be regularized in some learning cases ) . Assume ` ( ξθ , θ ) and Ri ( ξθ , θ ) are both twice-continuously differentiable . The problem of searching for θ∗ can be formally written as : θ∗ = arg min θ ` ( ξθ , θ ) subject to Ri ( ξθ , θ ) ≤ 0 , i = 1 , 2 , ... , l , ξθ solves Problem B ( θ ) . P For a specific learning and control task , one only needs to specify the details of Σ ( θ ) and give a task loss ` ( ξθ , θ ) and constraints Ri ( ξθ , θ ) ≤ 0 . Section 7 will give representative examples .

Summary: In order to satisfy both immediate and long-term safety constraints, this paper proposes Safe Pontryagin Differentiable Programming (Safe PDP). The proposed method is based on interior-point methods, and it enables us to incorporate heterogeneous types of constraints. The advantages of Safe PDP lie both theoretically and empirically. In the experiments, the authors demonstrated that their proposed method is better than unconstrained methods (in one experiment, a baseline is ALTRO, though) in toy problems.