Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper provides an efficient method based on Pontryagin Differentiable Programming (PDP) for the control of safety-critical systems. The method entails formulating the control problem as a bi-level optimization problem, where
* The lower-level problem solves for a trajectory that minimizes some control cost subject to dynamics, terminal constraints, and path constraints associated with the underlying system (all of which are potentially parameterized), and
* The upper-level problem aims to pick the parameters of the lower-level problem to minimize some task-specific loss subject to task-specific constraints.

The authors provide an efficient method to solve problems of this form (e.g., both terminal and path constraints) by employing barrier methods to both rewrite the outer problem as an unconstrained optimization problem, and to efficiently solve for solutions of and implicit gradients through the inner problem. They then prove that under certain assumptions, their method satisfies all problem constraints, as needed for the safety-critical setting. They provide experimental validation for the problems of policy optimization, motion planning, and imitation learning in several dynamical systems settings.
main_review: This paper provides a novel and interesting extension of PDP for safety-critical settings. The approach is well-motivated, the theory is well-grounded with reasonable assumptions, and the authors do a good job explaining the intuition and implications of the theoretical results even without including the full proofs in the main paper. The empirical evaluation does a good job showing the efficacy and generality of the proposed method. Overall, I think this is a very solid contribution and is well-suited for NeurIPS.

One suggestion for improvement is that non-safe PDP methods should be compared against in the experiments, in order to enable a better discussion of the salient tradeoffs (e.g., does constraint enforcement come at the cost of computational expense?).

Another suggestion is that the nascent literature on safe control via neural networks + differentiable optimization layers should also be discussed and ideally compared against in the experiments. (These methods do attempt to address similar settings as in the present work, and therefore can potentially provide some "safe" baselines.) The idea of this literature is that differentiable optimization layers can be constructed to capture the different types of constraints demanded by the setting (e.g., both immediate and long-term constraints, as addressed in the present work), and then used to constrain the output of a neural network. Some examples here include:
* Pham, Tu-Hoa, Giovanni De Magistris, and Ryuki Tachibana. "OptLayer - Practical constrained optimization for deep reinforcement learning in the real world." 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.
* Donti, P. L., Roderick, M., Fazlyab, M., & Kolter, J. Z. (2021). Enforcing robust control guarantees within neural network policies. ICLR 2021.
* Chen, B., Donti, P., Baker, K., Kolter, J. Z., & Berges, M. (2021). Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization. ACM e-Energy 2021.

As an additional question: In the experiments for Problem I, are the controllers evaluated on the true $f$, or the learned $f$? This should be made clearer, as evaluating on the learned $f$ could lead to potential over-stating of the results.

Minor points, questions, and typo corrections (not affecting my score):
* Line 113: "arbitrarily" --> "arbitrary"
* Equation 1: Should the dynamics and path constraints have some notion of "$\forall t$" somewhere? (Similar comment for later equations.)
* Line 160: Since challenge 1 already seems to address how to obtain $\partial \xi_\theta / \partial \theta$ efficiently (with $\mathcal{O}(T)$ complexity), I was initially confused by the implicit claim in challenge 2 that  $\partial \xi_\theta / \partial \theta$ is potentially expensive to obtain (and therefore needs to be cheaply obtained via an approximation). Perhaps the wording and presentation around the $\mathcal{O}(T)$ bound could be modified in the first point in order to smooth the narrative.
* Lines 160--169: These points should generally be checked for grammar and typos.
* Line 239: "motive" --> "motivate"
* Line 248, Theorem 2: "differentiability" is misspelled
* Line 291, Theorem 3: "minimizer" is misspelled
* Line 329: "exiting" --> "existing"
limitations_and_societal_impact: While the authors do address the *assumptions* of their method, they do not really discuss its limitations -- notably, the potential computational cost with respect to other methods, in which settings the stated assumptions might break, and in which settings other methods  (even, e.g., traditional robust control) may be better-suited. It could be good to incorporate some additional discussion of these points in the related work and experiments (in the latter case, particularly by introducing some baselines).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a Safe Pontryagin Differentiable Programming (Safe PDP) methodology. The contributions are at both theory and algorithmics for a new safe differentiable framework for various safety-critical learning and control tasks. The proposed approach is inspired by interior-point methods. Both backward and forward pass can be approximated and computed efficient by solving an unconstrained counterpart. The approximation can also be controlled for arbitrary accuracy using a barrier parameter. In addition, the authors can also show that all constraints are respected throughout the course of approximation and optimization, thus guaranteeing safety. Experiment results are shown in many learning and control tasks on different challenging control
systems such as 6-DoF maneuvering quadrotor and 6-DoF rocket powered landing.

main_review: 
I found this paper studies a very interesting and important research problem in control learning. The proposed approach will be very potential for many robot learning problems. The experiments on many robotic tasks are very convincing and non-trivial. 

Could the authors comment on how to improve the solving time of the forward pass? Each iteration, e.g. T=100, might take approximately 1s. If this can be improved, I believe Safe PDP could become more useful, e.g. integrated into an end-to-end task network. It might be interesting if the implementation can be leveraged to exploit GPUs, similar to the way OptNet and cvxpylayers do.

If possible for future work, it might be very interesting to see Safe PDP can be used as a layer in an end-to-end network. 

Reference [1] might be related.


[1] Priya L. Donti, Melrose Roderick, Mahyar Fazlyab, J. Zico Kolter:
Enforcing robust control guarantees within neural network policies. ICLR 2021

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a procedure for safe end-to-end learning of parameterized trajectory optimization problems. The entire learning problem is presented as a bi-level optimization. The outer optimization prescribes a loss function and safety constraints, which take as inputs the parameters and the output of an inner trajectory optimization problem using these same parameters. The overall map from parameters to the output of the outer problem is differentiated approximately with the help of: 1) approximating the inner and outer problems with unconstrained versions via log-barrier objective terms, and 2) using the discrete-time constrained PMP to differentiate the inner problem along an optimal trajectory. Safety is enforced by the log-barrier functions in the outer problem approximation.
main_review: *** Originality ***

There is value in the clear manner in which the authors combine different methods with a machine learning audience in mind, and the authors' description of using log-barrier functions to enforce "safety" (even if this idea has been presented in the past). In particular, from a machine learning perspective, I enjoyed the authors' careful consideration of the quality of gradient approximations.

However, the novelty of the paper is rather limited. Specifically, the results up to the end of Section 4, including Theorem 1, are directly from:
 
-- Jin, Wang, Yang, and Mou, "Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework", NeurIPS 2020.

The use of log-barrier functions to travel within the interior of the constraint set is the primary characteristic of Interior Point Methods (IPMs). Viewing this as a notion of "safe learning" has been done in:

-- Usmanova, Krause, and Kamgarpour, "Safe non-smooth black-box optimization with application to policy search", L4DC 2020.

-- Liu, Ding, and Liu, "IPO: Interior-Point Policy Optimization under Constraints", AAAI 2020.

Neither of these papers are cited by the authors.

I appreciate the authors' clear presentation of differentiating solutions to trajectory optimization problems. However, the authors strongly overclaim originality in this regard; e.g., on line 96 they state: "... all existing work has not studied/established the results regarding the differentiablity of general constrained optimal control systems ..." This neglects: 1) all work in sensitivity analysis for nonlinear programming and optimal control (e.g., Andersson and Rawlings, "Sensitivity Analysis for Nonlinear Programming in CasADi", IFAC 2018), 2) study of differentiable solution maps in optimal control (e.g., Trelat, "Some Properties of the Value function and its Level Sets for Affine Control Systems with Quadratic Cost", JDCS 2000), and 3) recent work on differentiable MPC (Amos, Rodriguez, et al, "Differentiable MPC for End-to-end Planning and Control", NeurIPS 2018).

*** Quality ***

I like the bi-level optimization framework the authors have clearly laid out, and I appreciate the effort put into the experiments. However, neglected details in the authors' theoretical development and flaws in the experimental evaluation mar the quality of this paper.

-- In Section 5, the unconstrained approximation of the trajectory optimization problem is not guaranteed to be safe w.r.t. the original nonlinear equality constraints (i.e., h_t and h_T). The quadratic barrier function used by the authors is insufficient in this regard. The authors do not discuss this point. On a minor related note, the authors should mention the interior of the safe set defined by the constraints must be non-empty to apply an IPM via log-barrier functions.

-- In Section 6, Theorem 3 just summarizes characteristics of IPMs, and the authors do not discuss how the approximation of the inner trajectory optimization affects the quality of the outer solution (i.e., if the inner problem is solved with some `gamma > 0`, then `theta^*(epsilon)` does not approach `theta^*` as `epsilon -> 0`). This part is particularly concerning and represents a significant disconnect in the paper.

-- Finding a feasible initial point is a key challenge when using IPMs. Given this, it is disappointing that the authors did not address the issue of finding a feasible initialization of the parameters `theta` in the main paper. In Section 7 ("Problem II: Safe Motion Planning"), this point is exacerbated by the authors' choice to experimentally compare their method with the ALTRO solver for trajectory optimization. ALTRO is not an IPM-based solver, and it can start with an infeasible trajectory guess; this kind of trade-off is common in nonlinear programming. The authors should have compared their method with an IPM-based trajectory optimization solver (e.g., DIRCOL, which uses IPOPT).

-- In Section 5, the authors state: "... solving a constrained optimal control ... is difficult since the popular trajectory optimization methods, such as iLQR and DDP, are designed only for unconstrained systems" There is an entire field of literature and academic community dedicated to trajectory optimization under constraints. The authors even use popular tools from this field, such as CasADi and ALTRO, in their experiments. I urge the authors to clarify their meaning in Section 5 and how this motivates their work.

In general, the authors do not provide any informative evaluation of the limitations of their work. As outlined above, there are a number of clear opportunities for the authors to do this, many of which are rooted in limitations of IPMs.

*** Clarity ***

The paper is well-organized and clearly written. The authors should fix any minor typos (e.g, line 104: "board", line 329: "exiting", line 291: "minimizor"), and improve the formatting and consistency of the references.

*** Significance ***

Overall, I think the idea of using log-barrier functions to enforce "safety" in an Interior Point Method (IPM)-style of optimization is a useful perspective for the machine learning community. While this idea exists in past work, I think the authors did well in communicating this idea through their current work. However, for this work to be complete, the authors need to spend more time evaluating and communicating the limitations of their method. Much of this could be connected to fundamental limitations of IPMs. Fair experimental comparisons to the state-of-the-art in trajectory optimization, optimal control, and sensitivity analysis would also help improve the quality of this paper. Finally, the authors should make an effort to discuss how the approximation of the inner trajectory optimization affects the quality of the outer solution -- an important aspect that the authors seem to have overlooked.
limitations_and_societal_impact: As discussed above, the authors need to add substantial discussion of the limitations of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In order to satisfy both immediate and long-term safety constraints, this paper proposes Safe Pontryagin Differentiable Programming (Safe PDP). The proposed method is based on interior-point methods, and it enables us to incorporate heterogeneous types of constraints. The advantages of Safe PDP lie both theoretically and empirically. In the experiments, the authors demonstrated that their proposed method is better than unconstrained methods (in one experiment, a baseline is ALTRO, though) in toy problems.


main_review: This paper extends PDP (Jin et al., 2020) from unconstrained to constrained learning and control problems. The proposed Safe PDP algorithm is a natural extension of the original PDP method, and I find it nice in terms of good theoretical properties. In my opinion, however, this paper is hard to understand and fails to claim the authors' contribution in a proper way.


First, the empirical experiments are not well conducted. Except for Problem 2 (where Safe PDP is compared with ALTRO), the authors simply compared their proposed method with unconstrained methods. What the authors should have claimed is that Safe PDP is better than other "safe" baselines (in terms of constraints' satisfaction, sample complexity, etc). If I understand correctly, the empirical advantage of Safe PDP is that 1) multiple immediate/long-term safety constraints can be handled, 2) safety constraints are strictly guaranteed, and 3) conservativeness can be tuned (i.e., performance should be better than other too conservative baselines). In short, "Paper Contributions" mentioned in Section 1.2 are not well discussed in the empirical experiments. I would like the authors to reconsider what kinds of experiments are needed to support their claims.


Second, the advantages of the proposed method have not fully discussed in a convincing way. In lines 86 - 88, there is the following statement "The above two safe control frameworks favorably consider the pure state constraints and cannot be readily extended to other constraints, such as constraints defined on both state and input, long-term cumulative constraints defined on the system trajectory level, etc". But, is this really true? As far as I know, there is a bunch of literature dealing with safety function (S \times A \rightarrow R) which is defined for state-action pairs. And, regarding lines 70 - 71, there are also several papers that deal with immediate constraint(s) and guarantee safety at every time step as being represented by

- Turchetta, Matteo, Felix Berkenkamp, and Andreas Krause. "Safe exploration in finite Markov decision processes with gaussian processes." Advances in Neural Information Processing Systems. 2016.
- Wachi, Akifumi, and Yanan Sui. "Safe reinforcement learning in constrained Markov decision processes." International Conference on Machine Learning. 2020.

I feel this paper has many questionable statements in an attempt to make itself look good.


Third, it is unclear how this paper is related to offline RL literature. In Problem III (lines 344 - 363), the dynamics are learned from true system expert demonstration. I personally think that demonstrations are not usually such ideal ones, and the authors should have made the problem settings consistent with typical offline RL studies. I think this paper should not totally ignore the offline RL pieces of literature if a problem like Problem III is solved in experiments. As a notable reference, please see

- Le, Hoang, Cameron Voloshin, and Yisong Yue. "Batch policy learning under constraints." *International Conference on Machine Learning*. 2019.


Minor Comments

- I think lines 1099 - 1104 ("Why implementation of Theorem 2 is more numerically stable?" in the Appendix) should be in the main paper (I understand that page limit is very severe, though).
- In RL experiments, the constraints in terms of actions are often implemented by clipping. I wonder what happens if the action is clipped in Figure 2(b).


Questions

- Safe PDP is totally dependent on the assumption of differentiability. How do you justify that the assumptions hold in a real environment? For example, how do you deal with constraints with steep changes?

- What happens if the dynamics are learned from imperfect/noisy/non-expert demonstration?

- Is it possible to conduct a theoretical analysis regarding sample complexity?
limitations_and_societal_impact: I think limitations have been adequately addressed.
review_title: null

Meta-review: The authors response has address concerns from the reviewers, and all reviewers agree on an acceptance. I encourage the authors to revise the draft by including the discussion and the additional experiments from the rebuttal. 
Decision: Accept (Poster)