Infinite-Horizon Differentiable Model Predictive Control

Abstract: This paper proposes a differentiable linear quadratic Model Predictive Control (MPC) framework for safe imitation learning. The infinite-horizon cost is enforced using a terminal cost function obtained from the discrete-time algebraic Riccati equation (DARE), so that the learned controller can be proven to be stabilizing in closed-loop. A central contribution is the derivation of the analytical derivative of the solution of the DARE, thereby allowing the use of differentiation-based learning methods. A further contribution is the structure of the MPC optimization problem: an augmented Lagrangian method ensures that the MPC optimization is feasible throughout training whilst enforcing hard constraints on state and input, and a pre-stabilizing controller ensures that the MPC solution and derivatives are accurate at each iteration. The learning capabilities of the framework are demonstrated in a set of numerical studies. 

Reviews and author responses:

The paper shows how to use the Discrete-time Algebraic Riccati Equation (DARE) to provide infinite horizon stability & optimality to differentiable MPC learning.  The paper also shows how to use DARE to derive a pre-stabilizing (linear state-feedback) controller.  The paper provides a theoretical characterization of the problem setting, which shows that prior work on differentiable MPC learning may lead to unstable controllers without the proposed augmentations using DARE. 

I'm not sure I understand the implications of imitating "from an expert of the same class".  Can the authors elaborate?

Can the authors compare & contrast with this paper? 
https://arxiv.org/abs/1709.07174 
(I have my own views, but I'd like hear the authors' thoughts first)

My biggest complaint is with regards to the experiments.  Unless I'm mistaken, it seems there isn't a thorough empirical study of the theoretical claims, especially as it relates to previous work.  E.g., can one construct scenarios where the baseline approach (Amos et al., 2018) fails, and compare with the proposed approach?

The idea of pre-stabilization is interesting, and seems related to this paper: https://arxiv.org/abs/1905.05380



**** After Author Response ****
Thanks for the response, I am raising my score to weak accept.

Dear Reviewer 1. Thanks for your response. To clarify, in Lemma 1 we claim that the MPC problem is infinite horizon optimal for a large enough N given the P from the Riccati equation for a bounded set of initial conditions. In the proof, in appendix, we refer to the original result in the paper “constrained linear quadratic regulator”, where this claim is demonstrated under the assumption that the state and input constraints are convex compact sets with the origin in their interior. Hence for a sufficient N there is no sub-optimality. This is a known result in the control literature. We are sorry if this was not made clear and hope that the reviewer could still change their mind about our score in light of the updates in our revised paper. Thanks again for the constructive contribution. 

This exactly addresses my question -- I still maintain a weak accept as I think the contribution of this paper is interesting and relevant to the community, but the experimental results in the current form are of limited interest.

I have read through the other reviews and responses in this thread and maintain my original score of a weak accept as this is an interesting new method and the pre-stabilization is useful. I apologize for responding so late in the discussion period -- but the clarification that I *meant* to ask for in my original review is that you use the DARE of the unconstrained problem to get the terminal control cost Q_N=P (in the text following Eq. (8)) that is used in the constrained problem, not as part of the stabilization. Using this still seems sub-optimal as there could be some other terminal cost that is optimal for the constrained problem that is different from the unconstrained DARE solution. I could imagine a system with very extreme bounds where there is a final cost for the constrained system is extremely different than the final cost for the unconstrained system.  It would be useful to have a reference to the controls literature that discusses this disconnection.

A revised paper has now been submitted that extends the numerical experiments to a vehicle platooning problem that includes 18 state variables and 10 control inputs, and hope that this additional contribution addresses the reviewers' concerns on the simplicity of the numerical experiments and better demonstrates the properties of the algorithm on a more significant, real-world control problem. We have also revised the paper to make the contribution relative to (Amos et al. 2018) clearer in the introduction and section 4.1 paragraph 1, and have made other minor revisions to improve the readability of the paper, correct notational mistakes, and accommodate the additional experiments within the 8 page recommended limit. We would like to thank the reviewers again for their constructive feedback, and believe that the revised paper makes a much stronger contribution as a result. 




We would like to thank the reviewers for taking the time to provide comprehensive and constructive reviews.

An issue that has been highlighted by all three reviewers is the low complexity of the numerical experiments, so the response to all three reviewers starts with the same text that addresses this concern: 

We concur with Reviewer 1’s observation that the investigation of simple systems is necessary as a sanity check, and this was the rationale of the numerical experiments presented here, but we also agree that it is a fair observation to highlight the lack of complexity as a limitation of the paper. In direct response to Reviewer 4’s suggestion to investigate a nonlinear system that has been successively linearised, the DARE cannot be used for non-linear systems, unless in cases where the non-linearities are quite limited. This is also discussed in Appendix E of our paper. Extensions to some interesting non-linear cases are going to be the subject of a follow-up study. In direct response to Reviewer 3’s comment ‘can one construct scenarios where the baseline approach (Amos et al., 2018) fails?’ – one of the significant limitations of (Amos et al., 2018) is that state constraints are not included in their differentiable MPC formulation (see equation (10) in their paper). As a consequence, their approach very quickly fails in general for even the simple LTI case presented here when state constraints are included because the MPC optimization has become infeasible in the forwards pass, and so one of the major contributions of this paper is ensuring feasibility in the presence of state constraints. We therefore believe that the numerical demonstration with an LTI system is necessary to provide credibility for when the approach is extended to non-linear systems, but agree that a 2DOF system is insufficient. We propose to investigate a second LTI system with a larger amount of states and inputs, inspired by a real-world application. Would the reviewers be satisfied with this additional experiment? 

Comments specific to Reviewer 3:

•	By ‘the same class’ we mean that the expert and learner are both 2DOF LTI systems controlled with Quadratic box-constrained MPC controllers. This comment has been removed in the updated paper to improve clarity.

•	We thank the reviewer for the suggested references, which are both are of interest

•	In the first suggested paper (Y. Pan et al.) the ‘expert’ controller takes the form of a model predictive controller where the model is a gaussian process and the solution is provided using differential dynamic programming. The ‘learner’ however, is simply a deep neural network, for which it is generally impossible to determine whether the learned controller will satisfy hard constraints on the system state a-priori, or whether it will be stabilizing. The entire purpose of the work presented in this paper is to provide interpretable structure to a neural network when used for imitation learning, for which hard constraint satisfaction and stability can be guaranteed a-priori, provided that the prediction error of the MPC model is limited. Ultimately, our study, as well as the previous ones on differentiable MPC, aims towards future integration of more complex sensory (e.g. visual) information as done in Y. Pan et al. However, in this work we decided  to focus on improving one part of the stack, i.e. on establishing conditions for having a more stable and optimal method to imitate controllers from given state-action measurements. The integration of visual data and of convolutional networks is of great interest and will be addressed in future work.    

•	The second paper (R. Cheng et al.) deals with reinforcement learning, where a prior policy is introduced to reduce variance and, in the case of H-infinity robust control, to provide stability with respect to the “uncertainty” resulting in the use of an RL policy. The two policies are weighted and it is shown that this results in a regularization of the original loss. This is clearly inspired by robust control, and in the general case adding stability or robustness will lead to an inevitable level of conservatism or sub optimality with respect to the original reward or loss. In this paper the re-parameterisation does not lead to sub-optimality and does not alter the MPC problem.  


We would like to thank the reviewers for taking the time to provide comprehensive and constructive reviews.

An issue that has been highlighted by all three reviewers is the low complexity of the numerical experiments, so the response to all three reviewers starts with the same text that addresses this concern: 

We concur with Reviewer 1’s observation that the investigation of simple systems is necessary as a sanity check, and this was the rationale of the numerical experiments presented here, but we also agree that it is a fair observation to highlight the lack of complexity as a limitation of the paper. In direct response to Reviewer 4’s suggestion to investigate a nonlinear system that has been successively linearised, the DARE cannot be used for non-linear systems, unless in cases where the non-linearities are quite limited. This is also discussed in Appendix E of our paper. Extensions to some interesting non-linear cases are going to be the subject of a follow-up study. In direct response to Reviewer 3’s comment ‘can one construct scenarios where the baseline approach (Amos et al., 2018) fails?’ – one of the significant limitations of (Amos et al., 2018) is that state constraints are not included in their differentiable MPC formulation (see equation (10) in their paper). As a consequence, their approach very quickly fails in general for even the simple LTI case presented here when state constraints are included because the MPC optimization has become infeasible in the forwards pass, and so one of the major contributions of this paper is ensuring feasibility in the presence of state constraints. We therefore believe that the numerical demonstration with an LTI system is necessary to provide credibility for when the approach is extended to non-linear systems, but agree that a 2DOF system is insufficient. We propose to investigate a second LTI system with a larger amount of states and inputs, inspired by a real-world application. Would the reviewers be satisfied with this additional experiment? 

Comments specific to Reviewer 1:

“the DARE solution in (7,8) is derived to optimally control a LTI system *without* control/state bounds but is then used to control the LTI system *with* control/state bounds in (4)…”

The DARE solution simply stabilizes the system in the absence of constraints and thus improves the numerical conditioning of the problem of optimizing the perturbation (delta u). The optimization of this perturbation signal ensures that the predicted control sequence is optimal (in an open-loop sense) for the constrained problem. A feedback control law is obtained by repeating the optimization at each timestep using current information on the system state.


We would like to thank the reviewers for taking the time to provide comprehensive and constructive reviews.

An issue that has been highlighted by all three reviewers is the low complexity of the numerical experiments, so the response to all three reviewers starts with the same text that addresses this concern: 

We concur with Reviewer 1’s observation that the investigation of simple systems is necessary as a sanity check, and this was the rationale of the numerical experiments presented here, but we also agree that it is a fair observation to highlight the lack of complexity as a limitation of the paper. In direct response to Reviewer 4’s suggestion to investigate a nonlinear system that has been successively linearised, the DARE cannot be used for non-linear systems, unless in cases where the non-linearities are quite limited. This is also discussed in Appendix E of our paper. Extensions to some interesting non-linear cases are going to be the subject of a follow-up study. In direct response to Reviewer 3’s comment ‘can one construct scenarios where the baseline approach (Amos et al., 2018) fails?’ – one of the significant limitations of (Amos et al., 2018) is that state constraints are not included in their differentiable MPC formulation (see equation (10) in their paper). As a consequence, their approach very quickly fails in general for even the simple LTI case presented here when state constraints are included because the MPC optimization has become infeasible in the forwards pass, and so one of the major contributions of this paper is ensuring feasibility in the presence of state constraints. We therefore believe that the numerical demonstration with an LTI system is necessary to provide credibility for when the approach is extended to non-linear systems, but agree that a 2DOF system is insufficient. We propose to investigate a second LTI system with a larger amount of states and inputs, inspired by a real-world application. Would the reviewers be satisfied with this additional experiment? 

Comments specific to Reviewer 4:

We thank the reviewer for the thorough reading of the paper and will include the corrections in an updated paper.


This paper shows how to make infinite-horizon MPC differentiable
by differentiating through the terminal cost function and controller.
Recent work in non-convex finite-horizon continuous control [1,2,3] face
a huge issue in selecting the controller's horizon length and
better-understanding differentiable infinite horizon
control has potentially strong applications in these domains.
As a step in this non-convex direction, this paper provides a nice
investigation in the convex LTI case.
The imitation learning experiments on a small spring dynamical
system are a necessary sanity check for further work, but
many other more complex systems could be empirically studied
and would have made this paper stronger.

One point that would be useful to clarify: the DARE solution in (7,8) is
derived to optimally control a LTI system *without* control/state bounds but
is then used to control the LTI system *with* control/state bounds in (4).
Does this lead to suboptimal solutions to the true infinite-horizon problem?

[1] Chua, K., Calandra, R., McAllister, R., & Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. NeurIPS 2018.
[2] Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. Learning latent dynamics for planning from pixels. ICML 2019.
[3] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, Jimmy Ba. Benchmarking Model-Based Reinforcement Learning. arXiv 2019.

This paper continues the recent direction (e.g. Amos & Kolter 2017) of differentiating through optimal control solutions, allowing for the combination of optimal control methods and learning systems. The paper has some nice contributions and I find this research direction to be very exciting, which is why I think it merits acceptance, however I find the experiments (Section 4) could be greatly improved.

The main contribution of the paper are the analytical derivative of the solution to the DARE. The pre-stabilising controller reformulation is a neat trick. 

The main issue I have with this paper is that the experiments are performed only on a toy 2D problem. Even an LTI system can be interesting! Of course it is important to start with a toy problem, but once positive results have been shown, it would be much more convincing if the paper showed some more complicated system, possibly an iteratively linearised non-linear system. My feeling (and possibly many others') is that these type on differentiable controllers can be extremely powerful, however this power is sadly not demonstrated here.

errata:
before eq (3): dt is not a pertubation to the feedback control
eq (4) argmin over \delta u rather than \delta, presumably


Meta-review: This paper develops a linear quadratic model predictive control approach for safe imitation learning.  The main contribution is an analytic solution for the derivative of the discrete-time algebraic Riccati equation (DARE).  This allows an infinite horizon optimality objective to be used with differentiation-based learning methods.  An additional contribution is the problem reformulation with a pre-stabilizing controller and the support of state constraints throughout the learning process.  The method is tested on a damped-spring system and a vehicle platooning problem.

The reviewers and the author response covered several topics. The reviewers appreciated the research direction and theoretical contributions of this work.  The reviewers main concern was the experimental evaluation, which was originally limited to a damped spring system.  The authors added another experiment for a substantially more complex continuous control domain.  In response to the reviewers, the authors also described how this work relates to non-linear control problems.  The authors also clarified the ability of the proposed method to handle state-based constraints that are not handled by earlier methods.  The reviewers were largely satisfied with these changes.

This paper should be accepted as the reviewers are satisfied that the paper has useful contributions.

Decision: accept-poster