Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper continues the recent direction (e.g. Amos & Kolter 2017) of differentiating through optimal control solutions, allowing for the combination of optimal control methods and learning systems. The paper has some nice contributions and I find this research direction to be very exciting, which is why I think it merits acceptance, however I find the experiments (Section 4) could be greatly improved.

The main contribution of the paper are the analytical derivative of the solution to the DARE. The pre-stabilising controller reformulation is a neat trick. 

The main issue I have with this paper is that the experiments are performed only on a toy 2D problem. Even an LTI system can be interesting! Of course it is important to start with a toy problem, but once positive results have been shown, it would be much more convincing if the paper showed some more complicated system, possibly an iteratively linearised non-linear system. My feeling (and possibly many others') is that these type on differentiable controllers can be extremely powerful, however this power is sadly not demonstrated here.

errata:
before eq (3): dt is not a pertubation to the feedback control
eq (4) argmin over \delta u rather than \delta, presumably

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper shows how to use the Discrete-time Algebraic Riccati Equation (DARE) to provide infinite horizon stability & optimality to differentiable MPC learning.  The paper also shows how to use DARE to derive a pre-stabilizing (linear state-feedback) controller.  The paper provides a theoretical characterization of the problem setting, which shows that prior work on differentiable MPC learning may lead to unstable controllers without the proposed augmentations using DARE. 

I'm not sure I understand the implications of imitating "from an expert of the same class".  Can the authors elaborate?

Can the authors compare & contrast with this paper? 
https://arxiv.org/abs/1709.07174 
(I have my own views, but I'd like hear the authors' thoughts first)

My biggest complaint is with regards to the experiments.  Unless I'm mistaken, it seems there isn't a thorough empirical study of the theoretical claims, especially as it relates to previous work.  E.g., can one construct scenarios where the baseline approach (Amos et al., 2018) fails, and compare with the proposed approach?

The idea of pre-stabilization is interesting, and seems related to this paper: https://arxiv.org/abs/1905.05380



**** After Author Response ****
Thanks for the response, I am raising my score to weak accept.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper shows how to make infinite-horizon MPC differentiable
by differentiating through the terminal cost function and controller.
Recent work in non-convex finite-horizon continuous control [1,2,3] face
a huge issue in selecting the controller's horizon length and
better-understanding differentiable infinite horizon
control has potentially strong applications in these domains.
As a step in this non-convex direction, this paper provides a nice
investigation in the convex LTI case.
The imitation learning experiments on a small spring dynamical
system are a necessary sanity check for further work, but
many other more complex systems could be empirically studied
and would have made this paper stronger.

One point that would be useful to clarify: the DARE solution in (7,8) is
derived to optimally control a LTI system *without* control/state bounds but
is then used to control the LTI system *with* control/state bounds in (4).
Does this lead to suboptimal solutions to the true infinite-horizon problem?

[1] Chua, K., Calandra, R., McAllister, R., & Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. NeurIPS 2018.
[2] Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. Learning latent dynamics for planning from pixels. ICML 2019.
[3] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, Jimmy Ba. Benchmarking Model-Based Reinforcement Learning. arXiv 2019.
review_title: Official Blind Review #1

Meta-review: This paper develops a linear quadratic model predictive control approach for safe imitation learning.  The main contribution is an analytic solution for the derivative of the discrete-time algebraic Riccati equation (DARE).  This allows an infinite horizon optimality objective to be used with differentiation-based learning methods.  An additional contribution is the problem reformulation with a pre-stabilizing controller and the support of state constraints throughout the learning process.  The method is tested on a damped-spring system and a vehicle platooning problem.

The reviewers and the author response covered several topics. The reviewers appreciated the research direction and theoretical contributions of this work.  The reviewers main concern was the experimental evaluation, which was originally limited to a damped spring system.  The authors added another experiment for a substantially more complex continuous control domain.  In response to the reviewers, the authors also described how this work relates to non-linear control problems.  The authors also clarified the ability of the proposed method to handle state-based constraints that are not handled by earlier methods.  The reviewers were largely satisfied with these changes.

This paper should be accepted as the reviewers are satisfied that the paper has useful contributions.
Decision: Accept (Poster)