Safe Pontryagin Differentiable Programming
Authors: Wanxin Jin
Shaoshuai Mou
George J. Pappas
Venue: NeurIPS 2021 Poster (2021)
Keywords: safe reinforcement learning
safe learning and control
constrained learning and control
differentiable programming
bi-level optimization
safety-critical tasks

Abstract: We propose a Safe Pontryagin Differentiable Programming (Safe PDP) methodology, which establishes a theoretical and algorithmic  framework to solve a broad class of safety-critical learning and control tasks---problems that require the guarantee of safety constraint satisfaction at any stage of the learning and control progress.  In the spirit of interior-point methods,   Safe PDP handles different types of system  constraints on states and inputs by incorporating them into the cost or loss through barrier functions. We prove three fundamentals  of the proposed  Safe PDP:  first, both the  solution and its gradient in the backward pass can be approximated by solving their  more efficient unconstrained counterparts;  second,   the approximation for both the  solution and its gradient can be controlled for arbitrary accuracy by a  barrier parameter;   and third,   importantly, all intermediate results throughout the approximation and optimization  strictly respect the  constraints,  thus guaranteeing safety throughout the entire learning and control process. We demonstrate the capabilities of   Safe PDP in solving various safety-critical tasks,  including safe policy optimization, safe motion planning, and learning MPCs from demonstrations, on different challenging systems such as 6-DoF maneuvering quadrotor and 6-DoF rocket powered landing.

Safe Pontryagin Differentiable Programming
Wanxin Jin
Shaoshuai Mou Purdue University mous@purdue.co edu
George J. Pappas University of Pennsylvania pappasgaseas. upenn. edu
University of Pennsylvania wanxinjindgmail.cmm
Abstract
We propose a Safe Pontryagin Differentiable Programming (Safe PDP) methodol- ogy, which establishes a theoretical and algorithmic framework to solve broad class of safety-critical learning and control tasks-problems that require the guaran- tee of safety constraint satisfaction any stage of the learning and control progress. In the spirit of interior-point methods, Safe PDP handles different types of system constraints on states and inputs by incorporating them into the cost or loss through barrier functions. We prove three fundamentals of the proposed Safe PDP: first, both the solution and gradient in the backward pass can be approximated by solving their more efficient unconstrained counterparts; second, the approximation for both the solution and its gradient can controlled for arbitrary accuracy by a barrier parameter; and third, importantly, all intermediate results throughout the approximation and optimization strictly respect the constraints, thus guaranteeing safety throughout the entire learning and control process. We demonstrate the ca- pabilities of Safe PDP in solving various safety-critical tasks, including safe policy optimization, safe motion planning, and learning MPCs from demonstrations, on different challenging systems such as 6-DoF mancuvering quadrotor and 6-DoF rocket powered landing.
1 Introduction
Safety is usually a priority in the deployment of a learning or control algorithm to real-world systems. For physical system (agent), safety is normally given in various constraints on system states and inputs, which must not be violated by the algorithm at any stage of the learning and control process, otherwise will cause irrevocable or unacceptable failure/damage. Those systems are referred as safety-critical. The constraints in safety-critical system can include the immediate ones, which are directly imposed on the system state and input at certain or all-time instances, and the long-term ones, which are defined on the trajectory of system states and inputs over long period.
Compared to the abundant results that focus on system optimality [1-3], systematic and principled treatments for safety-crite critical learning and control problems seem largely insufficient, particularly in the following gaps (detailed in Section 1.1). First, existing safety strategies are either too conservative. which may restrict the task performance, or iollation-tolerabbe which only pursues the near-constraint guarantee and thus are not strictly constraint-respeccingg Second, a systematic safety paradigm capable of handling different types of constraints, including system state and input (or mixed), immediate, or/and long-term constraints, is still lacked. Third, some existing safety strategies suffer from huge computational- and data- complexity, difficult to be integrated into any differentiable programming frameworks to solve large-scale learning and continuous control tasks.
To address the above research gaps, this paper aims to develop safe differentiable programming framework with the following key capabilities. First, the framework provides systematic treatment for different types of constraints in caiticl problem; second, it attains provable safety- and accuracy- guarantees throughout the learning and control process; third, flexible to perform safe learning of any unknown aspects of constrained decision-making system, including policy, dynamics, state and input constraints, and control cost; finally, it can be integrated any differentiable programming framework to efficiently solve large-scale safe learning and control tasks.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

1.1 Related Work
In machine learning and control fields. safety has been defined by different criteria, such as worst- case [4, 5], risk-sensitive [6], ergodicity [7], robust [8, 9], etc., most of which are formulated by directly altering an objective function [10]. In this paper, we focus only on constrained learning and control problems, where constraints are explicitly formulated and must be satisfied. We categorize existing techniques into at-convergence safety methods, which only concern constraint satisfaction at convergence, or in-progress safety methods, which attempt to ensure constraint satisfaction during the entire optimization process.
At-convergence safety methods. In reinforcement learning (RL), a constrained agent is typically formulated as Constrained Markov Decision Process (CMDP) [11], seeking a policy that not only optimizes reward but also satisfies an upper bound for cumulative cost. A common strategy [12- 17] to solve CMDPs is to use the primal-dual method, by establishing the unconstrained Lagrangian and performing saddle-point optimization. In deep learning, the primal-dual method has been recently used [18] to train deep neural networks with constraints. In control, the primal-dual method has been used to solve constrained optimal control (constrained trajectory) problems [19-21]. While proved to satisfy constraints at convergence [22, 23), the primal-dual type methods cannot guarantee constraint satisfaction during optimization, as shown in [13, 24), thus are not suitable for safety-critical tasks.
In-progress safety methods. To enforce safety during training. [25] and [26] solve CMDPs by introducing additional constraints into the Trust Region Policy Optimization (TRPO) [27]. Since these methods only obtain the 'near constraint" guarantee, constraint violation is not fully eliminated. Another line of constrained RL [24, 28-30] leverages the Lyapunov theory [31] to bound behavior of an agent. But how to choose a valid Lyapunov function for general tasks is still an open problem to date [32], particularly for constrained RL, since requires a Lyapunov function to be consistent with the constraints and to permit optimal policies [28]. Some other work also attempts to handle immediate constraints the constraints imposed on agent state and input at any time. In [33], a safe exploration scheme is proposed to produce a safe reachable region; and it only considers finite state space. [34] develops a method that learns safety constraints and then optimizes a reward within the certified safe region; the method defines constraints purely on agent state and thus may not be readily applicable to mixed state-input constraints.
In control, in-progress safety can be achieved via two model-based frameworks: reachability theory [35] and control barrier functions [36, 37J. Safe control based on reachability theory [35, 38-40] explicitly considers adversarial factors and seeks a strategy that maintains the constraints despite the adversarial factors. This process typically requires solving the pacilloon-aaobi-ssaaae equations [41], which become computationally difficult for nigh-dimensional systems [35]. Control barrier functions [36, 37] constrain a system only on safety boundaries, making a less-conservative strategy for safety-critical tasks [42-44]. Most of the methods consider affine dynamics and directly use the given constraint function as a control barrier function. Such a choice could be problematic when a system is uncontrollable at the boundary of the sublevel set. Thus. how to find valid control barrier function is still an ongoing research topic [45-47]. The above two control safety frameworks favorably focus on pure state constraints and cannot be readily extended to other constraints, such as mixed state-input constraints or the cumulative constraints defined on the system trajectory.
Interior-point methods and control. Interior-point methods (IPMs) [48-50] 50] solve constrained optimization by sequentially finding solutions to unconstrained problems with the objective combining the original objective and a barrier that prevents from leaving the feasible regions. IPMs have been used for construined linear quadratic regular (LQR) control [51-57]. While IPMs for nonlinear constrained optimal control are studied in 288-22, they mostly focus on developing algorithms to solve the unconstrained approximation (from the perturbed KKT conditions) and lack of performance analysis. Most recently, [63] uses the IPM to develop a zero-th order non-convex optimization method; and [64] uses IPMs to solve reinforcement learning with only cumulative constraints. Despite the promise of the trend, the theoretical results and systematic algorithms regarding the differentiabilityy of general constrained control systems based on IPMs have not been studied and established.
Differentiable projection layer. In machine learning, a recent line of work considers embedding a differentiable projection layer [65-67] into a general training process to ensure safety. Particularly, [67] and [66] enforce safety by constructing a dedicated projection layer, which projects the unsafe actions outputted from a neural policy into a safe region (satisfying safety constraints). This projection layer is differentiable convex layer [68, 69], which can be trained end-to-end. In [65], safety is
2

---

defined as robustness in the case of the worst adversarial disturbance, and the set of robust policies is solved by classic robust control (solving LMIs). An RL neural policy with a differentiable projection layer is learned such that the action from the neural policy lies in the robust policy set. Different from the above work, Safe PDP does not enforee safety by projection; instead, Safe PDP uses barrier functions to guarantee safety constraint satisfaction. More importantly, we have shown, in both theory and experiments, that with barrier functions, differentiability can also be attained.
Sensitivity analysis and differentiable MPCs. Other work related to Safe PDP includes the recent results for sensitivity analysis [70], which focuses on differentiation of solution to general nonlinear program, and differentiable MPCs [68]. which is based on differentiable quadratic programming. In long-horizon control settings, directly applying [70] and [68] can be inefficient: the complexity of [68, 70] for differentiating a solution to general optimal control system is at least O(T²) (I' is the
time horizon) due to computing the inverse of Jacobian of the KKT conditions. Since an optimal control system has more sparse structures than general nonlinear or quadratic programs, by exploiting those structures and proposing the Auxiliary Control System, Safe PDP enjoys the complexity of OTT) for differentiating solution to a general control system. Such advantages have been discussed and shown in the foundational PDP work [71] and will also be shown later (in Section 8) in this paper.
1.2 Paper Contributions
We propose a safe differentiable programming methodology named as Safe Pontryagin Differentiable Programming (Safe PDP). Safe PDP provides systematic treatment of different types of system constraints, including state and inputs (or mixed), immediate, and long-term constraints, with provable safety- and performance guarnatee Safe PDP is also a unified differentiable programming framework, which can be used to efficiently solve a broad class of safety-critical learning and control tasks.
In the spirit of interior- point methods, Safe PDP incorporates different types of system constraints into control cost and loss through barrier functions, approximating a constrained control system and task using their unconstrained counterparts. Contributions of Safe PDP are theoretical and algorithmic. Theoretically, we prove in Theorem 2 and Theorem 3 that (I) not only a solution but also the gradient of the solution can be safely approximated by solving a more efficient unconstrained counterpart; (II) any intermediate results throughout the approximation and optimization are strictly safe, that is, never violating the original system/task constraints; and (III) the approximations for both solution and its gradient can be controlled for arbitrary accuracy by barrier parameters. Arithmetically, (IV) we prove in Theorem 1 that a constrained control system is differentiable, the gradient of its trajectory is a globally unique solution to an Auxiliary Control System [71], which can be solved efficiently with the complexity of only OTT), T is control horizon; (V) in Section 7, we experimentally demonstrate the capability of Safe PDP for efficiently solving various safety-critical learning and control problems, including safe neural policy optimization, safe motion planning, learning MPCs from demonstrations.
2 Safe PDP Problem Formulation
Consider a class of constrained optimal control systems (models) E(O), which are parameterized by tunable parameter 0 R" in its control cost function, dynamics, initial condition, and constraints:
control cost: J(O)- Σ 1==0
ce(wee ue 0) + eT (xT, 0)
subject to
Σ(0):
(1)
dynamics:
D141 f [x1,, Un 0) with 30 xo((), V7,
terminal constraints:
( 0) 1 O, hr(x,, 0) 1)=00
path constraints:
9, ( ue 0) < S. ha ut 0) 0, VL.
Here, Dt E R" is the system state; ur E R is the control input: Gp R" x R" x R' + R and or R" Rr + I are the stage and final costs, respectively; f F² x TM x RT + ET is the dynamics with initial state In = (f) E RT, t = 0. I,... T is the time step with T the time horizon; 9r R" R"" + Rd and hT R" x R" + R are the final inequality and equality constraints, respectively; 9. R" x R x R - R and he : R x RR x RR" - R are the immediate inequality and equality constraints at time 1, respectively. All inequalities (here and below) are entry-wise. We consider that all functions in E()) are three-times continuously differentiable (i.e., C2) with respect to its arguments. Although we here have parameterized all aspects of Σ(0), for specific application (see Section 7), one only needs to parameterize the unknown aspects in E()) and keep others given.
3

---

Any unknown aspects in E()) can be implemented by differentiable neural networks. For given 0,
E() produces a trajectory Eo- = (m u - by solving the following Problem B(0): E.- = f -1 arg min ( ](0)
subject to
#141 = fa,, un 0) with zo = (0) vt, gr (2T, (TT,
B(0)
- 0 and hr 0) 0
( ut, 0) S 0 and h1( ue, 0) O, Vt.
Here, we use E since Eo to Problem B(0) may not be unique in general, thus constituting solution set (Ea). We will discuss the existence and uniqueness of (Eo) in Section 4.
For a specific task, we aim to find a specific model E( i., searching for a specific 0", such that its trajectory Eg- from B(0") meets the following two given requirements. First, Eg- minimizes a given task loss (E.. 0); and second, E.- satisfies the given task constraints Ri(Eg. 0) < 0, i 1,2,..... Note that, we need distinguish between the two types of objectives: task loss E(E. 0) and control cost J(O), and also the two types of constraints: task constraints (E, 0) and model constraints ( In fact, (() and ( in Σ()) are unknown and parameterized by 0 and can represent the unknown inherent aspects of a physical agent, while E(E. 0) and (S.. 0) are given and known depending on the specific task (they also explicitly depend on A since A needs to be regularized in some learning cases). Assume E(Ee. O) and R2 (E. 0) are both twice-continnously differentiable. The problem of searching for 0* can be formally written as: e(Eo. 0)
0r arg min 0
subject to
R, SO =
P
εo solves Problem B(0).
For specific learning and control task, one only needs specify the details of Σ(0) and give task loss (Ee. 0) and constraints (E 0) S O. Section will give representative examples.
3 Challenges to Solve Problem P
Problem P belongs to bi-level optimization [72]-each time 0 is updated in the outer-level (including task loss l and task constraint R2) of Problem P. the corresponding trajectory & needs to be solved from the inner-level Problem B()). Similar to PDP [71]. one could approach Problem P using gradient-based methods by ignoring the process of solving inner-level Problem B(O) and just viewing Ee as explicit differentiable function of 0. Then, based interior-point methods [48], can introduce a logarithmic barrier function for each task constraint, In (-R,). R.). and a barrier parameter >> 0. This leads to solving the following unconstrained Problem SP(e) sequentially
0 (E) arg min E(E,, 0) - > In (R,, 0))
SP(e)
for fixed e. By controlling - 0, 0 (e( is expected to converge the solution 0" to Problem P. Although plausible, the above process has the following technical challenges to be addressed:
(1) As En is solution to the constrained optimal control Problem B(0), is E differentiable? Does the auxiliary control system [71] exist for solving ?Ea
00
00 (2) Since we want to obtain E and Øe. at as low cost as possible, instead of solving the constrained
Problem B(0), can
uncomstrained system to approximate both & and 0Eg 00 Importantly, can the accuracy of the approximations for Eo and 0g, be arbitrarily and safely controlled?
ae (3) Can we guarantee that the approximation for &o is safe in a sense that the approximation always respects the system original inequality constraints 9, S O and gr S 0?
(4) With the safe approximations for both &g and 2 can accuracy of the solution to the outer-level unconstrained optimization SP(s) be arbitrarily controlled towards 0"?
(5) With the safe approximations for both &e and 0Eg inequality constraints R, < 0 during the optimization for the outer-level SP(e)?
"a0 can we guarantee the safety of the outer-level
The following paper will address the above challenges. For reference, we give a quick overview: Challenge (1) will be addressed in Section 4 and the result is in Theorem 1; Challenges (2) and (3) will be addressed in Section 5 and the result in Theorem 2; Challenges (4) and (5) will be addressed in Section 6 and the result is in Theorem 3; and Section 7 gives some representative applications.

---

Differentiability for Σ()) and its Auxiliary Control System
4.1 Differentiability of &
For the constrained optimal control system Σ(0) in (1), we define the following Hamiltonian L, for t = 0, I,..., T1 and LT, respectively,
L; ci ( u. 0) X f (x,, un 0) + (ag, un 0) + w/ Ue 0),
(2a)
LT cr TT. 0) 1 0) w/rh( 0),
(2b)
where X E Rn is the costate, v, E _ and w, E R are multipliers for the inequality and equality constraints, respectively. The well-known second-order condition for &o to be local isolated (locally unique) minimizing trajectory to ( in Problem B(0) has been well-established in [73]. For completeness, we present it in Lemma A.2 in Appendix A. Lemma A.2 states that there exist costate sequence Xi.r- and multiplier sequences v6.g and w such that (Eo- >i.T. U. w satisfies the well-known Constrained Pontryagin Minimum Principle (C-PMP) given in (S.5) in Lemma A.2. Based on the above, can have the following result for the differentiability of Eg-
Lemma (Differentiability of Eo). Given fixed 0, assume the following conditions hold for E(0):
(i) the second-order condition (Lemma A.2) 2) is satisfied for Σ(0):
(ii) the gradients of all binding constraints at Eg are linearly independent (binding constraints include all equality constraints and all active inequality constraints);
(iil) strict complementarity holds at Eg. i.e., active inequality constraint has positive multiplier.
Then, for all O in neighborhood of 0, there exists a unique once-contimuously differentiable futererttiabeefuuttioion function (Eo, A... -... ( that satisfies the second-order condition (Lemma A.2) for the constrained opti- w at 0=0. Hence, &o a local isolated minimizing trajectory to (0)) Further, for all near 0, the strict complementarity is preserved, and the linear independence of the gradients of all binding constraints at Eg hold.
mal control system E()) with (Eo. >i.T. oo wh:r) (Eg. X u
The proof of Lemma can directly follow the well-known first-order sensitivity result Theorem 2.1 in [74]. Here, conditions (i)-(ii) are the sufficient conditions to guarantee the applicability of the well-known implicit function theorem [75] to the C-PMP. Condition (ii) is well-known and serves as sufficient condition for the constraint qualification to establish the C-PMP (see Corollary 3, pp. 22, [48]). Condition (iii) is necessary to ensure that the Jacobian matrix in the implicit function theorem is invertible, and it also leads to the persistence of strict complementarity, saying that the inactive inequalities remain inactive and active ones remain active and there is no 'switching' between them near O. Both our practice and previous works [68, 69, 71, 74, 76) show that the conditions (i)-(ii)) very mild and the differentiability of Eg can be attained almost everywhere in the space of A.
4.2 Auxiliary Control System to Solve 7E 89
If the conditions (i)-(ii) in Lemma for differentiability of &o hold, we next show that dEg 30 can also be efficiently solved by auxiliary control system, which is originally proposed in the foundational work [71]. First, we define the new state and input (matrix) variables X, E nxr and U2 e Rm respectively. Then, introduce the following auxiliary control system,
T-1
x, U
L
L
x. Uc
L
X₂ Ue
control cost:
J= Tr
L
L
Lud
1=
+ Tr
2 X L Xy (I
E(Ee):
(3)
subject to dynamics:
Xet1 F; F.U, + F.
with -u= X
terminal constraint:
G G = 0,
H + HG 0,
path constraint:
H H. H . 0.
Here, T.f and T denote the first- and second- order derivatives, respectively, of the Hamiltonian L, in (2) with respect to x; FÝ. HP G2 denote the first-order derivatives of f., he, g, with respect
5

---

to æ, respectively, where I. is the vector of stacking all active inequality constraints in gr and the similar convention applies to the other notations. All derivative matrices defining (Ee) are evaluated at (Ee. \i.r V ), where > vorr. and w are usually the byproducts of a constrained optimal control solver [77] or can be easily solved from the C-PMP given L.. as done in [71]. We note that (E is
Linear Quadratic Regulator (I.QR) system, as its control cost function is quadratic and dynamies and constraints are linear. For the above EEE)) we have the following important result without additional assumptions.
Theorem
30 AEe is globally unique minmizing trajectory to E(Eo)). Let the conditions (i)-(iii) in
Lemma I for differentiability of &o hold. Then, the auxiliary control system E(Sa) in (3) has a globally unique minimizing trujectory, denoted as { {X U0.T-13. which is exactly dea
ie.,
80
{ Ub U
0h. 00
with x
dx
and Up
Ou.
(4)
00
00
The proof of the above theorem is in Appendix B. Theorem 1 states that as long as the conditions (i)- (iii) in Lemma for differentiability of E are satisfied, without additional assumptions, the auxiliary control system Σ(Ee) always has globally unique minimizing trajectory, which exactly agg a0
Thus, obtaining 8Eg is equivalent to solving E(Ee). which be efficiently done thanks to the recent development of the constity-onsttaiined LQR algorithms OTT). The algorithm that implements Theorem is given in Algorithm in Appendix E.1.
all of which have a complexity of
Safe Unconstrained Approximations for Eo and
U 30
From Section 4, we know that one can solve the constrained system Σ(0) to obtain Eg and solve its auxiliary control system E(E,) to obtain 0E., Although theoretically appealing, there are several "0 difficulties in implementation. First, solving a constrained optimal control Problem B(0) is not as easy as solving an unconstrained optimal control, for which many trajectory optimization algorithms, e.g., iL.QR [81] and DDP [82], are available. Second, establishing E(Eo) requires the values of the multipliers vo.7 and w And third, to construct E(E,), one also needs identify all active inequality constraints g.. which can be numerically difficult due to numerical error (we will show this in later experiments). All those difficulties motivate to develop more efficient paradigm to obtain both E and aEg which is the goal of this section. 80
To proceed, we first convert the constrained system Σ(0) to an unconstrained system Σ(0, 7) by adding all constraints its control cost via barrier functions. Here, use quadratic barrier function for each equality constraint and logarithm barrier functions for each inequality constraint; and all barrier functions are associated with the same barrier parameter 7 > This leads to [0,, 7) be
control cost: J(0,7)
Σ
ca (ws. ue 0)-00)77 > In Lnn (-9,, uc 0))
(Ne, ree ue, 0)))) + cr( 0)-
27
7(0,)):
(5)
gr, (xFT, 0))
27 Σ
(br,;(a7 0))
subject to dynamics: 0t+1 = f(x,, Ue 0)
with 3o = xo (0), 1L.
The trajectory {( {
(0,) (0m) aou uar-i
produced by the above unconstrained system Σ(0,7) is
E(e,)) = { (0.0)) un. -1 }
E arg min (2o.T Mo:TT J(0,7)
SB(0, 7)
s.t. x:41 = f(0,, Ur 0) with xy = x.
that is, E(e..) is minimizing the new control cost J(0,7) subject to only dynamics. Then we have the following important result about the safe unconstrained approximation for Eg and Be using Σ(0,r).
80

---

Theorem 2. Let conditions (i)-(iii) in Lemma for differentiability of Eo hold. For any small >> 0,
(a) there exists a local isolated minimizing trajectory S(0.)) that solves Problem SB(O, (7), and
E(0,7) 7) is well-defined at E(e.)) i.e. (0(x) (0,)) u (0.5)
00 <0 and gr ( (0.7)
)0:
(b) E(0,)) is once-continuously differentiable with respect to (0, 7), and
(( Eo and (( og
7 0:
(6)
a0
00
(c) the trajectory derivative 0ce,1 is a globally unique minimizing trajectory to the auxiliary 80
control system Σ (((e))) corresponding to E(0,)) 7).
The proof of the above theorem is given in Appendix C. It is worth noting that the above assertions require no additional assumption except the same conditions (ii for differentiability of Eg. We make the following comments on the above results, using an illustrative cartpole example in Fig. 1.
First, assertion (b) states that by choosing small y > 0, one can simply use ((..)) and oéçe.)) of the unconstrained <3 optimal control system Σ(0.7) to approximate &a and oEa of the original constrained system E(0), respectively. Second, notably, assertion (b) also states that the above approximations can be controlled for arbitrary accuracy by simply letting + O, as illustrated in the upper panels in Fig. 1. Third, more importantly, assertion (a) states that the above approximations are always safe in sense that the approximation E(0,)) with any small yy > 0 is guaranteed satisfy all inequality constraints in the orig- inal E(0), as illustrated in the bottom panels in Fig. 1. Finally, similar to Theorem 1, assertion (c) states that the derivative @E(0.)) for Σ(e, 7) is globally unique mini- 50
x1o
"300
0.0
10
10
10-
10-2
10 10
Cart force Input
y=3.
Cart Cart position
10 15 0
20
25
0
10
15 Time
20
25
Time
Figure I: E(o.)) and
06 .) approximate
06
mizing trajectory to its corresponding auxiliary control system (E),)))) thus PDP [71] directly applies here.
Eo and dgg under different y 0. ae
In addition to the theoretical importance of Theorem 2, we also summarize its algorithmic advantage compared to directly handling the original constrained system Σ(0) and its auxiliary control system E(G,). First, solving the unconstrained Σ(0,7) 7) is easier than solving the constrained E() as more off-the-shelf algorithms are available for unconstrained trajectory optimization than for constrained one. Second, when solving 8E
using (E))),)) there is no need to identify the inactive and -a
active inequality constraints, as opposed to solving dEg using E(E,): thus it is easier implement and more numerically stable (we will show this later in experiments). Third, in contrast to Theorem I, the unconstrained Σ( 7) and E(E(e.:)) avoid dealing with the multipliers vo:r and Wo.r. Finally, by absorbing hard inequality constraints into the control cost through barrier functions, Σ 7) intro- duces the 'softness' of constraints and mitigates the discontinuous 'switching' between inactive/active inequalities over a large range of O. This leads to a more numerically stable algorithm, as we will show in later experiments. Implementation of Theorem 2 is given in Algorithm 2 in Appendix E.2.
778
6 Safe PDP to Solve Problem P
According to Theorem 2, we use the safe unconstrained approximation system Σ(0, 7) in (5) to replace the original inner-level constrained system Σ(0) in (1). Then, we give the following important result for solving Problem P. which addresses the Challenges (4) and (5) in Section 3.
Theorem 3. Consider all functions defining the constrained optimal control system Σ() are least three-times continuously differentiable, and let the conditions (i)-(ii)) in Lemma 1 for differentiabillty of Eo hold in a neighborhood of 0* Suppose that the second-order condition for a local isolated minimizor 0* to Problem P is satisfied, that the gradients V o R, ((a..'''))) 0²)) of all binding constraints (E 8) are linearly independent at 0¹, and that the strict complementary holds at 0" Then, for any small e > O and any small 7 >> O, the following outer-level unconstrained approximation
0' (c,7) arg min (E((,)); 0) - -)e E'E In (- Ra (-Ra (E((
SP(e,
7

---

with E(0.)) being the optimal trajectory to the inner-level safe unconstrained approximation system E(0,7) 7) in (5), has the following assertions:
(a) there exists a local isolated minimizor 0* (e,") to the above SP(e, and the corresponding
trajectory f (ec))))) from the inner-level approximation system Σ ((0 (c, (), 7) is safe with
respect the original outer-level constraints, i.e., R((E(o-(er)))) 0- (c, +)) < 0, i = ....?
(b) 0* (c, 7) is once-continuously differentiable with respect to both and % and
O' (c, 0"
( (0,0);
(c) for any 0 near 0 (c, 7), S(0,)) from the inner-level approximation system Σ(0,7) is safe with respect to the original outer-level constraints, i.e., R₂ (E(e.)))) 0) < 0, i i 1....
The proof of the above theorem is given in Appendix D. The above result says that instead of solving the original constrained Problem P with the inner-level constrained system E() in (I), one can solve an unconstrained approximation Problem SP(e, 7) with the inner-level safe unconstrained approxima- tion system (0, 7) in (5). Particularly, we make the following comments on the importance of the above theorem. First, claim (a) affirms that although the inner-level trajectory E(e.,) is an approxima- tion (recall Theorem 2), the outer-level unconstrained Problem 7) always has a locally unique solution 0* 7): furthermore, at 0" (e, 7), the corresponding inner-level trajectory E(or (e,7).)) is safe with respect the original outer-level constraints, i.c., R. (6(00000))))))) (c, < 0, 1,2, .... (. Second, claim (b) asserts that the accuracy of the solution 0" (c, 7) to the outer-level approximation Problem SP(c, 7) is controlled jointly by the inner-level barrier parameter ? and outer-level barrier parameter e: as both barrier parameters approach zero, 0" (s.7) is converging to the true solution 0' to the original Problem P. Third, claim (c) says that during the local search of the outer-level solution 0" 7), the corresponding inner-level trajectory E(e,)) is always safe with respect to the original outer-level constraints, i.e., R. (5(o,))) 0) < 0, 1.,,..., I. The above Theorem 3, together with Theorem 2 provide the safety- and accuracy- guarantees for the whole Safe PDP framework. Then entire Safe PDP algorithm is given in Algorithm 3 in Appendix E.3.
7 Applications to Different Safety-Critical Tasks
We apply Safe PDP to solve some representative anfety-critical learning/control tasks. For a specific task, one only needs to specify the parameterization detail of E(0), task loss E(E.. 0), and task constraints (Eo 0) in Problem P. The experiments are performed on the systems of different complexities in Table 1. All codes are available at https://gii thub. wwwm//awxinninnnaeooooaao
PDP.
Table 1: Experimental environments [71]
System Σ(0)
Dynamics f (00m)
Control cost J(0)))
Constraints g(Ocu))
Cartpole
cart & pole masses and length length and mass of links mass, wing length, inertia rocket mass, length, inertia
C: |
1 +
92 (
Two-link Robot arm
|0
(x - 22ooa)
||/a||///00 S U maxx,
6-DoF quadrotor
6-DoF rocket landing
|10%,, (1-=aa)))
cr=
Ocn= (xx=xx Unax)
Note that for each system, (0or)) includes the immediate constraints on system input and state ar at any time 11/12//c is the
instance; 9, is known;
norm; and time horizon T is around 50 for all systems.
Problem I: Safe Policy Optimization aims to find a policy that minimizes control cost J subject to constraints g while guaranteeing that any intermediate policy during optimization should never violate the constraints. To apply Safe PDP solve such a problem for the systems in Table 1, we set:
dynamics: 0141 f(0,, u.) with Co.
Σ(0)=
policy: u. = π (x,, 0).
where dynamics f is learned from demonstrations in Problem III, and π (0) is represented by a (deep) feedforward neural network (NN) with the NN parameter. In Problem P. the task loss E(Eo, 0) is set as J(Oanj). ). and task constraints (E 0) as g (( with both Oonj and a known. Then, safe policy optimization to solve Problem P using Safe PDP. The results for the robot arm and 6-DoF maneuvering quadrotor are in Fig. 2, and the other results and details are in Appendix F.I.
8

---

Safe ROP, e= 10
Unconstrained
Safe PDP, e- 10
Unconstrained
ter e0
4003
Ber
3553
10
Generatior Mt.
Mmaa
1502
10
Genereted [tu]]
Mmaar
15.5
Uminn
2503
2002
500
Iteration
1003
1520
13
503
1:03
1520
2300
Time
(a) Rohot-arm loss (b) Constraint violation during opt. (c) Quadrotor loss (d) Constraint violation during opt.
Figure 2: Safe neural policy optimization for robot-arm (a)-(b) and 6-DoF quadrotor (c)-(d).
Fig. 2a and 2c plot loss (control cost) versus gradient-descent iteration under different showing that the NN policy achieves good convergence when e S 10 (as asserted by Theorem 3). Fig. 2b and 2d show all indeterminate control trajectories generated from the NN policy during entire iterations; we also mark the constraints Uma and compare with the unconstrained policy optimization under the same settings. The results confirm that Safe PDP enables to achieve an optimal policy while guaranteeing that any intermediate policy throughout optimization is safe.
Problem II: Safe Motion Planning searches for a dynamics-feasible trajectory that optimizes a criterion and avoids unsafe regions (obstacles), meanwhile guaranteeing that any intermediate motion trajectory during search must avoid the unsafe regions. To apply Safe PDP to solve such problem, we specialize Σ(0) as (8) except that policy here u: = ut,,, 0). which is represented by Lagrangian polynomial [83] with & the parameters (pivots). In Problem P. task loss is set as J ((,)) and task constraints as ostr). with obbi and ear known in Table 1. The safe planning results using Safe PDP for cartpole and 6-DoF rocket landing are in Fig. 2, in comparison with ALTRO, state-of-the-an constrained trajectory optimization method [21]. Other results and more details are in Appendix F.2
Safe POP, 10-2
ALTRO
Safe PDP, 10-2
ALTRO
control
Dec 45 2300
Gimm tozs
snole Thrust [|ol]s
Fat es PiSCS
$400
c-10-)
Ezzo
10
Cart pos.
1000
2500 Iteration
33c0
15 Time
10
20
25
10
15
20
520
3002
1500
10
20 Time
30
Itertion
Time
(a) Cartpole loss
(b) Constraint violation during opt.
(c) Rocket loss
(d) Constraint violation during opt.
Figure Safe motion planning for cartpole (b)-(b) and 6-DoF rocket powered landing (c)-(d).
Fig. 3a and 3c plot the task loss versus gradient-descent iteration, showing that the trajectory achieves a good convergence with e S 10-2, Fig. 3b and 3d show all intermediate motion trajectories during entire optimization, with constraints marked. The results confirm that Safe PDP can find an optimal trajectory while always respecting constraints throughout planning process.
Problem III: Learning MPC from Demonstrations. Suppose for all systems in Table 1,, the control cost J( c0m)) dynamics f (0ay)) and constraints g, (f are all unknown and parameterized Table We aim to jointly learn f = {0eea. Odyn, 2sar) from demonstrations gdomio = demo demo of true expert system. In Problem P. set Σ(0) ) (1), consisting of J (( f f(0mm)) and ge (aar))
in
TAO:T
parameterized; set task loss ((E, 0)- = || demo -ell² which quantifies the reproducing loss between demm and Eg: and there is no task constraints. By solving Problem P. we can learn E(0) such that its reproduced &, is closest to given g demo The demonstrations g demoo here are generated with 0 known (two episode trajectories for each system with time horizon 7-50). The plots of the loss versus gradient-descent iteration are in Fig. 4, and more details and results are in Appendix F.3.
In Fig. 4a-4d, for each system, we use three strategies to obtain Ee and BEg for Σ(0): (A) use solver [77] to obtain Eo and use Theorem to obtain aEg. (B) use Theorem to approximate both E, and aEe by E(o.n) and #C(e.)), '9
ae res,eetieeelll 1=100 22 and (C) use a solver to obtain &o and Theorem 2 only for öSn Fig. 4a-4d show that for Strategies (B) and (C), the reproducing loss quickly "78
converges to zeros, indicating that the dynamics, constraints, and control cost are successfully learned to reproduce the demonstrations. Fig. 4a-4d also show numerical instability for strategy (A); this is due to the discontinuous 'switching" of active inequalities between iterations, and also the error in correctly identifying active incqualities (we identify them by checking It, -8 with d > 0 a small

---

threshold), as analyzed in Section 5. More analysis is given in Appendix F.3. Note that we are not aware of any existing methods that can handle jointly learning of cost, dynamics, and constraints here, and thus we have not given benchmark comparison. Fig. 4e gives timing results of Safe PDP.
10"
Time per iteration (c)
Heprobuuing loss
10
(en lar quand
Degrodacing 2o
Reproducing
2,00
Thorens 3 bndrward
Thurenn Lidinard
101
10a
200
9.30
10
100
0.25
40
150
lteration
teertion
Iteration
Iteration
Time horizon
(a) Cartpole (log-y)
(b) Robot-arm
(c) Quadrotor (log-y)
(d) Rocket
(e) Timing
Figure 4: Jointly learning dynamics, constraints, and control cost from demonstrations.
8 Discussion
Comparisons with other differentiable frameworks. Fig. compares Safe PDP, CasADi [70], and Differentiable MPC [68] for the computational efficiency of differentiating an optimal trajectory control system with different control horizons T. The results show a signifi- cantly computational advantage of Safe PDP over CasADi and Differentiable MPC. Specifically, Safe PDP has complexity of (TT). while CasADi and Diffcrentiable MPC have at least (T This is because both CasADi and differentiable MPC are based on the implicit function theorem [75] and need to compute the inverse of a Hessian matrix of the size proportional to x T. In contrast, Safe PDP solves the gradient of trajectory by constructing Auxiliary Control System, which can be solved using the Riccati equation.
Time for dilferentaatonn
CaSAEN
Safee POP
MPC
constrained optimal
1.00
305
horizon
Figure 5: Time for dif- ferentiating optimal tra- jectory with different T.
Limitation of Safe PDP. Safe PDP requires safe (feasible) initialization such that the log-barrier control cost or loss is well-defined. While restrictive, safe initialization common in safe learning [63, 84]. We have the following empiricism on how to provide safe initializations for different types of problems, as adopted in our experiments in Section 7. In safe policy optimization, one could first use supervised learning to learn a safe policy from some safe rajecttoiesséemoostttaiinn (not necessarily be optimal) and then use the learned safe policy to initialize Safe PDP. In safe motion planning, one could arbitrarily provide a safe trajectory (not necessarily optimal) to initialize Safe PDP. In learning MPCs, the goal includes learning of constraint itself, and there is no such requirement.
Strategies to accelerate forward pass of Safe PDP. There are many strategies to accelerate a long-horizon trajectory optimization (optimal control) in the forward pass of Safe PDP. (I) One effective way is to scale the (continuous) long-horizon problem into a smaller one (e.g., unit) by applying time-war warping function to the dynamics and cost function [85]. After solving the scaled short-horizon problem, re-scale the trajectory back. (II) There are also 'warm-up' tricks, e.g., one can initialize the trajectory at the next iteration using the result of the previous iteration. (III) One can also use hierarchical strategy to solve trajectory optimization from coarse to fine resolutions. We have tested and provided the comparison for the above three acceleration strategies in Appendix G.2.
Please refer to Appendix G for more discussion, which includes G.1: comparison between Safe-PDP and non-safe PDP; G.2: comparison of different strategies for accelerating hong-horizon trajectory optimization; G.3: trade-offs between accuracy and computational efficiency using barrier penalties; G.4: learning MPCs from non-optimal data; and G.5: detailed discussion on limitation of Safe PDP.
Conclusions
This paper proposes a Safe Pontryagin Differentiable Programming methodology, which establishes a provable and systematic safe differentiable framework to solve broad class of safety-crite cafety-critical control and learning tasks with different types of safety constraints. For a constrained system and task, Safe PDP approximates both the solution and its gradient in backward pass by solving their more efficient unconstrained counterparts. Safe PDP has established two results: one is the controlled accuracy guarantee for approximations of the solution and its gradient, and the other is the safety guarantee for constraint satisfaction throughout the control and learning process. We envision the potential of Safe PDP for addressing various safety-critice problems in machine learning, control, and robotics fields.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the NASA University Leadership Initiative (ULI) under grant number 80NSSC20M0161. The research of Prof. George J. Pappas is supported by the AFOSR Assured Autonomy in Congested Environments under grant number FA9550-19-1-00 19-1--265.. This work has been done primarily in the last semester of Wanxin Jin's in's Ph.D. study at Purdue University. Wanxin Jin thanks Prof. Zhaoran Wang for some discussion about this work.
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 51 (18854)):52295583 2015.
[2] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):4844485 2016.
[3] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[4] Arna Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.
[5] Matthias Heger. Consideration of risk reinforcement learning, International Conference on Machine Learning, pages 105-111, 1994.
[6] Ronald Howard and James Matheson. Risk-sensitive markov decision processes. Manage- ment science, 18(7):356-366 1972.
[7] Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in markov decision processes. International Conference on Machine Learning, 2012.
[8] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: arXiv:1706.0660 1706.0683, 2017.
[9] Kemin Zhou and John Comstock Doyle. Essentials of robust control, volume 104, Prentice hall Upper Saddle River, NJ, 1998.
[10] Javier Garcia and Fernando Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480) 2015.
[11] Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
[12] Eitan Altman. Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operations research, 48(3):387- 417, 1998.
[13] Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. Advances in Neural Information Processing Systems, 2019.
[14] Yinlam Chow, Mohammad Ghavamzadeh, Lueas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. International Conference on Machine Learning, 18(1):6070-6122 61200 2017.
[15] Shalabh Bhatnagar and K Lakshmanan. An online actor-critic algorithm with function ap- proximation for constrained markov decision processes. Journal of Optimization Theory and Applications, 153(3);688-77) 2012.

---

[16] Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes, Advances in Neural Information Processing Systems, 33, 2020.
[17] Miguel Calvo-Fullana, Santiago Paternain, Luiz FO Chamon, and Alejandro Ribeiro. State augmented constrained reinforcement learning: Overcoming the limitations of learning with rewards. arXiv preprint arXiv:2102.11194 2021.
[18] Yatin Nandwani, Abhishek Pathak, Parag Singla, al. A primal dual formulation for deep learning with constraints. Advances in Neural Information Processing Systems, 2019.
[19] Maitine Bergounioux, Kazufumi Ito, and Karl Kunisch. Primal-dual strategy for constrained optimal control problems. SIAM Journal on Control and Optimization, 37(4): 196-11161111 194, 1999.
[20] Matthew R Kirchner, Gary Hewer, Jérôme Darbon, and Stanley Osher. A primal-dual method for optimal control and trajectory generation in high-dimensional systems. In Conference Control Technology and Applications,, pages 1583-1590, 2018
[21] Taylor Howell, Brian E Jackson, and Zachary Manchester. Altro: A fast solver for constrained trajectory optimization. In IEEE/RSI International Conference Intelligent Robots and Systems, pages 7674-7679, 2019.
[22] Simon S Du and Wei Hu. Linear convergence of the primal-dual gradient method for convex- concave saddle point problems without strong convexity. In International Conference on Artificial Intelligence and Statistics, pages 196-205, PMLR, 2019.
[23] Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in nonconvex- nonconcave minimax optimization? In International Conference on Machine Learning, pages 4880-4889. PMLR, 2020.
[24] Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov- based safe policy optimization for continuous control. arXiv preprint arXiv: 10031, 2019.
[25] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pages 22-31. PMLR, 2017.
[26] Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection- constrained policy optimization. International Conference on Learning Representations, 2020.
[27] John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference Machine Learning, pages 1899. PMLR, 2015.
[28] Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach safe reinforcement learning. Advances in Neural Information Processing Systems, 2018
[29] Theodore Perkins and Andrew G Barto. Lyapunov design for safe reinforcement learning. Journal of Machine Learning Research, 3(Dec):803 832. 2002.
[30] Felix Berkenkamp, Matteo Turchetta, Angela P Schoellig, and Andreas Krause. Safe model- based reinforcement learning with stability guarantees. Advances in Neural Information Pro- cessing Systems, 2017.
[31] Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion. International journal of control, 55(3):531-53: 1992.
[32] Peter Giesl and Sigurdur Hafstein. Review on computational methods for Yyapunov functions. Discrete & Continuous Dynamical Systems-B, 20(8):2291, 2015.
[33] Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite markov decision processes with gaussian processes. Advances Neural Information Processing Systems, 2016.
12

---

[34] Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained markov decision processes. In International Conference on Machine Learning, pages 9797-9806. PMLR, 2020.
[35] Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J Tomlin. Hamilton-jacobi reachability: brief overview and recent advances. In IEEE Conference on Decision and Control, pages 2242-2253, 2017.
[36] Peter Wieland and Frank Allgöwer. Constructive safety using control barrier functions. IFAC Proceedings Volumes, 40(12):462 467, 2007.
[37] Aaron D Ames, Xiangru Xu, Jessy W Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control, 62(8):3861-3876, 2016.
[38] Jaime F Fisac, Anayo K Akametalu, Melanie N Zeilinger, Shahab Kaynama, Jeremy Gillula, and Claire J Tomlin. A general safety framework for learning- based control in uncertain robotic systems. IEEE Transactions on Automatic Control, 64(7):2737-273 2018.
[39] Sylvia L Herbert, Mo Chen, Soolean Han, Somil Bansal, Jaime F Fisac, and Claire J Tomlin. Fastrack: A modular framework for fast and guaranteed safe motion planning. In IEEE Conference on Decision and Control, pages 1517-1522 2017.
[40] Mo Chen, Jaime F Fisac, Shankar Sastry, and Claire J Tomlin. Safe sequential path planning of multi-vehicle systems via double-obstacle jamilton-jacobi-iie jacobi-isaacs variational inequality. In European Control Conference, pages 3304-3309, 2015.
[41] Lawrence Evans and Panagiotis E Souganidis. Differential games and representation formulas for solutions of bamilton-jacobi-isaaa equations. Indiana University muthematics journal, 33(5):773-797, 1984.
[42] Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In European Control Conference, pages 0220-4431, 2019.
[43] Jason Choi, Fernando Castaneda, Claire J Tomlin, and Koushil Sreenath. Reinforcement learning for safety-critical control under model uncertainty, using control Iyapumov functions and control barrier functions. arXiv preprint arXiv:2004.077: 075544 2020.
[44] Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In AAAI Conference on Artificial Intelligence, pages 3387-3395, 2019.
[45] Alexander Robey, Haimin Hu, Lars Lindemann, Hanwen Zhang, Dimos V Dimarogonas, Stephen Tu, and Nikolai Matni. Learning control barrier functions from expert demonstrations. In IEEE Conference Decision and Control, pages 3717-3724, 2020.
[46] Alexander Robey, Lars Lindemann, Stephen Tu, and Nikolai Matni. Learning robust hybrid control barrier functions for uncertain systems. arXiv preprint 1rXiv:2101.0649) 2021.
[47] Wanxin Jin, Zhaoran Wang, Zhuoran Yang, and Shaoshuai Mou. Neural certificates for safe control policies. arXiv preprint arXiv:2006.08465, 2020.
[48] Anthony V Fiacco and Garth P McCormick. Nonlinear programming: sequential unconstrained minimization techniques. SIAM, 1990.
[49] Yurii Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming. SIAM, 1994.
[50] Anders Forsgren, Philip E Gill, and Margaret H Wright. Interior methods for nonlinear optimization. SIAM review, 44(4):525-59) 2002.
[51] AEB Lim, JB Moore, and I. Faybusovich. Linearly constrained lq and lqg optimal control. IFAC Proceedings Volumes, 290)):110-11155, 1996.
13

---

[52] SJ Wright. Structured interior point methods for optimal control. In JEEE Conference on Decision and Control, pages 1711-1716, 1991.
[53] Stephen J Wright. Interior point methods for optimal control of discrete time systems. Journal of Optimization Theory and Applications, 77(1):161-18) 1993.
[54] Christopher V Rao, Stephen J Wright, and James B Rawlings. Application of interior-point methods to model predictive control. Journal of optimization theory and applications, 99(3):723-
757, 1998.
[55] Anders Hansson and S Boydt. Robust optimal control of linear discrete-time systems using primal-dual aherior-poimm methods. In American Control Conference, volume 1, pages 183-187, 1998.
[56] Anders Hansson. A primal-dual interior-poin method for robust optimal control of linear discrete-time systems. IEEE Transactions on Automatic Control, 45(9):1639-1655 2000.
[57] Christian Feller and Christian Ebenbauer. Relaxed logarithmic barrier function based model predictive control of linear systems. IEEE Transactions on Automatic Control, (2(3):1223-1252) 2016.
[58] Julien Larin,,-Varin, J Frederic Bonnans, Nicolas Bérend, Mounir Haddou, and Christophe Talbot. Interior-point approach to trajectory optimization. Journal of Guidance, Control, and Dynamics, 3025):1228-1233 2007.
[59] John Hauser and Alessandro Saccon. A barrier function method for the optimization trajectory functionals with constraints. In IEEE Conference Decision and Control, pages 864-869. IEEE, 2006.
[60] Paul Malisani, François Chaplais, and Nicolas Petit. An interior penalty method for optimal control problems with state and input constraints of nonlinear systems. Optimal Control Applications and Methods, 37(1): 33, 2016.
[61] Alexander Domahidi, Aldo U Zgraggen, Melanie N Zeilinger, Manfred Morari, and Colin N Jones, Efficient interior point methods for multistage problems arising in receding horizon control. In IEEE conference on decision and control, pages 668-674, 2012.
[62] Andrei Pavlov, Iman Shames, and Chris Manzie. Interior point differential dynamic program- ming. IEEE Transactions Control Systems Technology. 2021.
[63] Ilnura Usmanova, Andreas Krause, and Maryam Kamgarpour. Safe non-smooth black-box optimization with application to policy search. In Learning for Dynamics and Control, pages 980-989. PMLR, 2020.
[64] Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In AAAI Conference on Artificial Intelligence, pages 4947, 2020.
[65] Priya Donti, Melrose Roderick, Mahyar Fazlyab, and Zico Kolter. Enforcing robust control guarantees within neural network policies. In International Conference on Learning Representations, 2020.
[66] Bingqing Chen, Priya L. Donti, Kyri Baker, J. Zico Kolter, and Mario Bergés. Enforcing policy feasibility constraints through differentiable projection for energy optimization. In ACM International Conference on Future Energy Systems, page 199-210, New York, NY, USA, 2021. Association for Computing Machinery.
[67] Tu-Hoa Pham, Giovanni De Magistris, and Ryuki Tachibana. Optlayer- practical constrained optimization for deep reinforcement learning in the real world. In International Conference on Robotics and Automation, pages 6236-6243 IEFF, 2018.
[68] Brandon Amos, Ivan Dario Jimenez Rodriguez, Jacob Sacks, Byron Boots, and J Zico Kolter. Differentiable mpe for end-to-end planning and control. In Advances Neural Information Processing Systems, 2018
14

---

[69] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural
networks. International Conference Machine Learning, pages 136-145. PMLR, 2017.
[70] Joel AE Andersson and James B Rawlings. Sensitivity analysis for nonlinear programming in casadi. IFAC- IFAC-PupersOnLinee, 51(20):331-33, 2018.
[71] Wanxin Jin, Zhaoran Wang. Zhuoran Yang. and Shaoshuai Mou. Pontryagin differentiable pro- gramming: An end-to-end learning and control framework. In Advances Neural Information Processing Systems, 2020.
[72] Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. A review bilevel optimization: from classical to cvolutionary approaches and applications. IEEE Transactions on Evolutionary Computation, 22(2):276-25) 2017.
[73] J Pearson and R Sridhar. A discrete optimal control problem. IEEE Transactions automatic control, 11:111714,7, 1966.
[74] Anthony V Fiacco. Sensitivity analysis for nonlinear programming using penalty methods. Mathematical programming, 10(1):287-311, 1976.
[75] Walter Rudin et al. Principles of mathematical analysis, volume 3. McGraw-hill New York, 1976.
[76] Charles D Kolstad and Leon S Lasdon. Derivative evaluation and computational experience with large bilevel mathematical programs. Journal of optimization theory and applications. 65(3):485-499) 1990.
[77] Joel AE Andersson, Joris Gillis, Greg Horn, James B Rawlings, and Moritz Diehl. Casadi: software framework for nonlinear optimization and optimal control. Mathematical Programming Computation, 11 11(1):1-36, 2019.
[78] Athanasios Sideris and Luis A Rodriguez. A riccati approach to equality constrained linear quadratic optimal control. In American Control Conference, pages 5167-5172, 2010.
[79] Shuo Yang, Gerry Chen, Yetong Zhang, Frank Dellaert, and Howie Choset. Equality constrained linear optimal control with factor graphs. arXiv preprint arX1v:2011.01360, 2020.
[80] Forrest Laine and Claire Tomlin. Efficient computation of feedback control for equality- constrained lqr. In International Conference on Robotics and Automation, pages 0748-6754 IEEE, 2019.
[81] Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear bio- logical movement systems. In International Conference on Informatics in Control, Automation and Robotics, pages 222-229, 2004.
[82] David H Jacobson and David Q Mayne. Differential dynamic programming. Number 24. Elsevier Publishing Company, 1970.
[83] Milton Abramowitz and Irene A Stegun. Handbook of mathematical functions with formulas,,,, graphs, and mathematical tables, volume 55. US Government printing office, 1964.
[84] Felix Berkenkamp, Andreas Krause, and Angela P Schoellig. Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. Machine Learning, pages 1-35, 2021.
[85] Wanxin Jin, Todd D Murphey, Dana Kulić, Neta Ezer, and Shaoshuai Mou. Learning from sparse demonstrations. arXiv preprint arXiv:2008. UrXiv:2008.02159, 2020.
[86] Michael Athans. The matrix minimum principle. Information and control, 11(5-6):592-606 1967.
[87] Jean Dieudonné, Foundations of modern analysis. New York: Academic Press. Volume 1 of Treatise on Analysis, 2011.
[88] Rolf Johansson. System modeling and identification. Prentice-hall, 1993.
15

---

[89] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 2018.
[90] Wanxin Jin, Dana Kulić, Shaoshuai Mou, and Sandra Hirche. Inverse optimal control from incomplete trajectory observations. The International Journal of Robotics Research, 40(6- 7):848-865, 2021.
[91] Wanxin Jin and Shaoshuai Mou. Distributed inverse optimal control. Aulomatica, 129, 2021.
[92] Wanxin Jin, Dana Kulić, Jonathan Feng-Shun Lin, Shaoshuai Mou, and Sandra Hirche, Inverse optimal control for multiphase cost functions, IEEE Transactions on Robotics, 3566):1387-1398, 2019.
[93] Wanxin Jin, Todd D Murphey, and Shaoshuai Mou. Learning from incremental directional corrections, arXiv preprint arXiv:2011.15014 2020.
[94] Michael A Patterson and Anil V Rao. Gpops-ii: A matlab software for solving multiple-phass optimal control problems using hp-adaptive gaussian quadrature collocation methods and sparse nonlinear programming. ACM Transactions on Mathematical Software (TOMS), 11:13 2014.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Sections 8 and Appendix G.5 the paper.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Pleae find them in all theorems in the paper.
(b) Did you include complete proofs of all theoretical results? [Yes] Please find complete proofs for all theoretical results in the Appendix the supplementary file.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Please refer to the code at https: /tt::/giittt com/ wanxinjin/Safe-PPD
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see the Appendix F in the supplementary file.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see the Appendix in the supplementary file.
4. If you are using existing assets (e.g., code, data, models) or curating/releasnng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Please see the citation of the experimental environments in Table in the paper.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Some video demo links are included in Appendix F in the supplementary file.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
16

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "pPD2ln4FoX", "reviewer": "Reviewer_v3kP", "summary": "This paper provides an efficient method based on Pontryagin Differentiable Programming (PDP) for the control of safety-critical systems. The method entails formulating the control problem as a bi-level optimization problem, where\n* The lower-level problem solves for a trajectory that minimizes some control cost subject to dynamics, terminal constraints, and path constraints associated with the underlying system (all of which are potentially parameterized), and\n* The upper-level problem aims to pick the parameters of the lower-level problem to minimize some task-specific loss subject to task-specific constraints.\n\nThe authors provide an efficient method to solve problems of this form (e.g., both terminal and path constraints) by employing barrier methods to both rewrite the outer problem as an unconstrained optimization problem, and to efficiently solve for solutions of and implicit gradients through the inner problem. They then prove that under certain assumptions, their method satisfies all problem constraints, as needed for the safety-critical setting. They provide experimental validation for the problems of policy optimization, motion planning, and imitation learning in several dynamical systems settings.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a novel and interesting extension of PDP for safety-critical settings. The approach is well-motivated, the theory is well-grounded with reasonable assumptions, and the authors do a good job explaining the intuition and implications of the theoretical results even without including the full proofs in the main paper. The empirical evaluation does a good job showing the efficacy and generality of the proposed method. Overall, I think this is a very solid contribution and is well-suited for NeurIPS.\n\nOne suggestion for improvement is that non-safe PDP methods should be compared against in the experiments, in order to enable a better discussion of the salient tradeoffs (e.g., does constraint enforcement come at the cost of computational expense?).\n\nAnother suggestion is that the nascent literature on safe control via neural networks + differentiable optimization layers should also be discussed and ideally compared against in the experiments. (These methods do attempt to address similar settings as in the present work, and therefore can potentially provide some \"safe\" baselines.) The idea of this literature is that differentiable optimization layers can be constructed to capture the different types of constraints demanded by the setting (e.g., both immediate and long-term constraints, as addressed in the present work), and then used to constrain the output of a neural network. Some examples here include:\n* Pham, Tu-Hoa, Giovanni De Magistris, and Ryuki Tachibana. \"OptLayer - Practical constrained optimization for deep reinforcement learning in the real world.\" 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.\n* Donti, P. L., Roderick, M., Fazlyab, M., & Kolter, J. Z. (2021). Enforcing robust control guarantees within neural network policies. ICLR 2021.\n* Chen, B., Donti, P., Baker, K., Kolter, J. Z., & Berges, M. (2021). Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization. ACM e-Energy 2021.\n\nAs an additional question: In the experiments for Problem I, are the controllers evaluated on the true $f$, or the learned $f$? This should be made clearer, as evaluating on the learned $f$ could lead to potential over-stating of the results.\n\nMinor points, questions, and typo corrections (not affecting my score):\n* Line 113: \"arbitrarily\" --> \"arbitrary\"\n* Equation 1: Should the dynamics and path constraints have some notion of \"$\\forall t$\" somewhere? (Similar comment for later equations.)\n* Line 160: Since challenge 1 already seems to address how to obtain $\\partial \\xi_\\theta / \\partial \\theta$ efficiently (with $\\mathcal{O}(T)$ complexity), I was initially confused by the implicit claim in challenge 2 that  $\\partial \\xi_\\theta / \\partial \\theta$ is potentially expensive to obtain (and therefore needs to be cheaply obtained via an approximation). Perhaps the wording and presentation around the $\\mathcal{O}(T)$ bound could be modified in the first point in order to smooth the narrative.\n* Lines 160--169: These points should generally be checked for grammar and typos.\n* Line 239: \"motive\" --> \"motivate\"\n* Line 248, Theorem 2: \"differentiability\" is misspelled\n* Line 291, Theorem 3: \"minimizer\" is misspelled\n* Line 329: \"exiting\" --> \"existing\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While the authors do address the *assumptions* of their method, they do not really discuss its limitations -- notably, the potential computational cost with respect to other methods, in which settings the stated assumptions might break, and in which settings other methods  (even, e.g., traditional robust control) may be better-suited. It could be good to incorporate some additional discussion of these points in the related work and experiments (in the latter case, particularly by introducing some baselines).", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "jqyn-2jeJjU", "reviewer": "Reviewer_u2xh", "summary": "The authors propose a procedure for safe end-to-end learning of parameterized trajectory optimization problems. The entire learning problem is presented as a bi-level optimization. The outer optimization prescribes a loss function and safety constraints, which take as inputs the parameters and the output of an inner trajectory optimization problem using these same parameters. The overall map from parameters to the output of the outer problem is differentiated approximately with the help of: 1) approximating the inner and outer problems with unconstrained versions via log-barrier objective terms, and 2) using the discrete-time constrained PMP to differentiate the inner problem along an optimal trajectory. Safety is enforced by the log-barrier functions in the outer problem approximation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*** Originality ***\n\nThere is value in the clear manner in which the authors combine different methods with a machine learning audience in mind, and the authors' description of using log-barrier functions to enforce \"safety\" (even if this idea has been presented in the past). In particular, from a machine learning perspective, I enjoyed the authors' careful consideration of the quality of gradient approximations.\n\nHowever, the novelty of the paper is rather limited. Specifically, the results up to the end of Section 4, including Theorem 1, are directly from:\n \n-- Jin, Wang, Yang, and Mou, \"Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework\", NeurIPS 2020.\n\nThe use of log-barrier functions to travel within the interior of the constraint set is the primary characteristic of Interior Point Methods (IPMs). Viewing this as a notion of \"safe learning\" has been done in:\n\n-- Usmanova, Krause, and Kamgarpour, \"Safe non-smooth black-box optimization with application to policy search\", L4DC 2020.\n\n-- Liu, Ding, and Liu, \"IPO: Interior-Point Policy Optimization under Constraints\", AAAI 2020.\n\nNeither of these papers are cited by the authors.\n\nI appreciate the authors' clear presentation of differentiating solutions to trajectory optimization problems. However, the authors strongly overclaim originality in this regard; e.g., on line 96 they state: \"... all existing work has not studied/established the results regarding the differentiablity of general constrained optimal control systems ...\" This neglects: 1) all work in sensitivity analysis for nonlinear programming and optimal control (e.g., Andersson and Rawlings, \"Sensitivity Analysis for Nonlinear Programming in CasADi\", IFAC 2018), 2) study of differentiable solution maps in optimal control (e.g., Trelat, \"Some Properties of the Value function and its Level Sets for Affine Control Systems with Quadratic Cost\", JDCS 2000), and 3) recent work on differentiable MPC (Amos, Rodriguez, et al, \"Differentiable MPC for End-to-end Planning and Control\", NeurIPS 2018).\n\n*** Quality ***\n\nI like the bi-level optimization framework the authors have clearly laid out, and I appreciate the effort put into the experiments. However, neglected details in the authors' theoretical development and flaws in the experimental evaluation mar the quality of this paper.\n\n-- In Section 5, the unconstrained approximation of the trajectory optimization problem is not guaranteed to be safe w.r.t. the original nonlinear equality constraints (i.e., h_t and h_T). The quadratic barrier function used by the authors is insufficient in this regard. The authors do not discuss this point. On a minor related note, the authors should mention the interior of the safe set defined by the constraints must be non-empty to apply an IPM via log-barrier functions.\n\n-- In Section 6, Theorem 3 just summarizes characteristics of IPMs, and the authors do not discuss how the approximation of the inner trajectory optimization affects the quality of the outer solution (i.e., if the inner problem is solved with some `gamma > 0`, then `theta^*(epsilon)` does not approach `theta^*` as `epsilon -> 0`). This part is particularly concerning and represents a significant disconnect in the paper.\n\n-- Finding a feasible initial point is a key challenge when using IPMs. Given this, it is disappointing that the authors did not address the issue of finding a feasible initialization of the parameters `theta` in the main paper. In Section 7 (\"Problem II: Safe Motion Planning\"), this point is exacerbated by the authors' choice to experimentally compare their method with the ALTRO solver for trajectory optimization. ALTRO is not an IPM-based solver, and it can start with an infeasible trajectory guess; this kind of trade-off is common in nonlinear programming. The authors should have compared their method with an IPM-based trajectory optimization solver (e.g., DIRCOL, which uses IPOPT).\n\n-- In Section 5, the authors state: \"... solving a constrained optimal control ... is difficult since the popular trajectory optimization methods, such as iLQR and DDP, are designed only for unconstrained systems\" There is an entire field of literature and academic community dedicated to trajectory optimization under constraints. The authors even use popular tools from this field, such as CasADi and ALTRO, in their experiments. I urge the authors to clarify their meaning in Section 5 and how this motivates their work.\n\nIn general, the authors do not provide any informative evaluation of the limitations of their work. As outlined above, there are a number of clear opportunities for the authors to do this, many of which are rooted in limitations of IPMs.\n\n*** Clarity ***\n\nThe paper is well-organized and clearly written. The authors should fix any minor typos (e.g, line 104: \"board\", line 329: \"exiting\", line 291: \"minimizor\"), and improve the formatting and consistency of the references.\n\n*** Significance ***\n\nOverall, I think the idea of using log-barrier functions to enforce \"safety\" in an Interior Point Method (IPM)-style of optimization is a useful perspective for the machine learning community. While this idea exists in past work, I think the authors did well in communicating this idea through their current work. However, for this work to be complete, the authors need to spend more time evaluating and communicating the limitations of their method. Much of this could be connected to fundamental limitations of IPMs. Fair experimental comparisons to the state-of-the-art in trajectory optimization, optimal control, and sensitivity analysis would also help improve the quality of this paper. Finally, the authors should make an effort to discuss how the approximation of the inner trajectory optimization affects the quality of the outer solution -- an important aspect that the authors seem to have overlooked.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As discussed above, the authors need to add substantial discussion of the limitations of their work.  \n\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "hSsjwtw0oMr", "reviewer": "Reviewer_FvjM", "summary": "In order to satisfy both immediate and long-term safety constraints, this paper proposes Safe Pontryagin Differentiable Programming (Safe PDP). The proposed method is based on interior-point methods, and it enables us to incorporate heterogeneous types of constraints. The advantages of Safe PDP lie both theoretically and empirically. In the experiments, the authors demonstrated that their proposed method is better than unconstrained methods (in one experiment, a baseline is ALTRO, though) in toy problems.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper extends PDP (Jin et al., 2020) from unconstrained to constrained learning and control problems. The proposed Safe PDP algorithm is a natural extension of the original PDP method, and I find it nice in terms of good theoretical properties. In my opinion, however, this paper is hard to understand and fails to claim the authors' contribution in a proper way.\n\n\nFirst, the empirical experiments are not well conducted. Except for Problem 2 (where Safe PDP is compared with ALTRO), the authors simply compared their proposed method with unconstrained methods. What the authors should have claimed is that Safe PDP is better than other \"safe\" baselines (in terms of constraints' satisfaction, sample complexity, etc). If I understand correctly, the empirical advantage of Safe PDP is that 1) multiple immediate/long-term safety constraints can be handled, 2) safety constraints are strictly guaranteed, and 3) conservativeness can be tuned (i.e., performance should be better than other too conservative baselines). In short, \"Paper Contributions\" mentioned in Section 1.2 are not well discussed in the empirical experiments. I would like the authors to reconsider what kinds of experiments are needed to support their claims.\n\n\nSecond, the advantages of the proposed method have not fully discussed in a convincing way. In lines 86 - 88, there is the following statement \"The above two safe control frameworks favorably consider the pure state constraints and cannot be readily extended to other constraints, such as constraints defined on both state and input, long-term cumulative constraints defined on the system trajectory level, etc\". But, is this really true? As far as I know, there is a bunch of literature dealing with safety function (S \\times A \\rightarrow R) which is defined for state-action pairs. And, regarding lines 70 - 71, there are also several papers that deal with immediate constraint(s) and guarantee safety at every time step as being represented by\n\n- Turchetta, Matteo, Felix Berkenkamp, and Andreas Krause. \"Safe exploration in finite Markov decision processes with gaussian processes.\" Advances in Neural Information Processing Systems. 2016.\n- Wachi, Akifumi, and Yanan Sui. \"Safe reinforcement learning in constrained Markov decision processes.\" International Conference on Machine Learning. 2020.\n\nI feel this paper has many questionable statements in an attempt to make itself look good.\n\n\nThird, it is unclear how this paper is related to offline RL literature. In Problem III (lines 344 - 363), the dynamics are learned from true system expert demonstration. I personally think that demonstrations are not usually such ideal ones, and the authors should have made the problem settings consistent with typical offline RL studies. I think this paper should not totally ignore the offline RL pieces of literature if a problem like Problem III is solved in experiments. As a notable reference, please see\n\n- Le, Hoang, Cameron Voloshin, and Yisong Yue. \"Batch policy learning under constraints.\" *International Conference on Machine Learning*. 2019.\n\n\nMinor Comments\n\n- I think lines 1099 - 1104 (\"Why implementation of Theorem 2 is more numerically stable?\" in the Appendix) should be in the main paper (I understand that page limit is very severe, though).\n- In RL experiments, the constraints in terms of actions are often implemented by clipping. I wonder what happens if the action is clipped in Figure 2(b).\n\n\nQuestions\n\n- Safe PDP is totally dependent on the assumption of differentiability. How do you justify that the assumptions hold in a real environment? For example, how do you deal with constraints with steep changes?\n\n- What happens if the dynamics are learned from imperfect/noisy/non-expert demonstration?\n\n- Is it possible to conduct a theoretical analysis regarding sample complexity?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think limitations have been adequately addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "I don't have any ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "5VUGFbP9kiP", "reviewer": "Reviewer_Xjxt", "summary": "This paper proposes a Safe Pontryagin Differentiable Programming (Safe PDP) methodology. The contributions are at both theory and algorithmics for a new safe differentiable framework for various safety-critical learning and control tasks. The proposed approach is inspired by interior-point methods. Both backward and forward pass can be approximated and computed efficient by solving an unconstrained counterpart. The approximation can also be controlled for arbitrary accuracy using a barrier parameter. In addition, the authors can also show that all constraints are respected throughout the course of approximation and optimization, thus guaranteeing safety. Experiment results are shown in many learning and control tasks on different challenging control\nsystems such as 6-DoF maneuvering quadrotor and 6-DoF rocket powered landing.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI found this paper studies a very interesting and important research problem in control learning. The proposed approach will be very potential for many robot learning problems. The experiments on many robotic tasks are very convincing and non-trivial. \n\nCould the authors comment on how to improve the solving time of the forward pass? Each iteration, e.g. T=100, might take approximately 1s. If this can be improved, I believe Safe PDP could become more useful, e.g. integrated into an end-to-end task network. It might be interesting if the implementation can be leveraged to exploit GPUs, similar to the way OptNet and cvxpylayers do.\n\nIf possible for future work, it might be very interesting to see Safe PDP can be used as a layer in an end-to-end network. \n\nReference [1] might be related.\n\n\n[1] Priya L. Donti, Melrose Roderick, Mahyar Fazlyab, J. Zico Kolter:\nEnforcing robust control guarantees within neural network policies. ICLR 2021\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors response has address concerns from the reviewers, and all reviewers agree on an acceptance. I encourage the authors to revise the draft by including the discussion and the additional experiments from the rebuttal. 
Decision: Accept (Poster)