Infinite-Horizon Differentiable Model Predictive Control
Authors: Sebastian East
Marco Gallieri
Jonathan Masci
Jan Koutnik
Mark Cannon
Venue:  (2020)
Keywords: Model Predictive Control
Riccati Equation
Imitation Learning
Safe Learning

Abstract: This paper proposes a differentiable linear quadratic Model Predictive Control (MPC) framework for safe imitation learning. The infinite-horizon cost is enforced using a terminal cost function obtained from the discrete-time algebraic Riccati equation (DARE), so that the learned controller can be proven to be stabilizing in closed-loop. A central contribution is the derivation of the analytical derivative of the solution of the DARE, thereby allowing the use of differentiation-based learning methods. A further contribution is the structure of the MPC optimization problem: an augmented Lagrangian method ensures that the MPC optimization is feasible throughout training whilst enforcing hard constraints on state and input, and a pre-stabilizing controller ensures that the MPC solution and derivatives are accurate at each iteration. The learning capabilities of the framework are demonstrated in a set of numerical studies. 

Published as a conference paper at ICLR 2020
INFINITE-HORIZON DIFFERENTIABLE MODEL PREDICTIVE CONTROL
Sebastian East! Marco Gallieri', Jonathan Masci', Koutník' & Mark Cannon NNAISENSE, Lugano, Switzerland
*Department of Engineering Science, University of Oxford, Oxford, UK (sebastians east, mark. cannon) } @eng. . ox. ac uk (marco, jonathan, jan) ) @ @nnai sense. com
ABSTRACT
This paper proposes a differentiable linear quadratic Model Predictive Control (MPC) framework for safe imitation learning. The infinite-horizon cost is enforced using a terminal cost function obtained from the discrete-time algebraic Riccati equation (DARE), so that the learned controller can be proven to be stabilizing in closed-loop. A central contribution is the derivation of the analytical derivative of the solution of the DARE, thereby allowing the use of differentiation-barn based learning methods. A further contribution is the structure of the MPC optimization problem: an augmented I. Lagrangian method ensures that the MPC optimization is feasible throughout training whilst enforcing hard constraints on state and input, and pre- stabilizing controller ensures that the MPC solution and derivatives are accurate at each iteration. The learning capabilities of the framework are demonstrated in set of numerical studies.
INTRODUCTION
Imitation Learning (IL. Osa al. 2018) aims at reproducing existing control policy by means of function approximator and can be used, for instance, to hot-start reinforcement learning. Effective
learning and generalisation unseen data are paramount IL success, especially in safety critical
applications. Model Predictive Control (MPC, Maciejowski 2000 Camacho & Bordons, 2007
Rawlings & Mayne, 2009] Kouvaritakis & Cannon 2015 Gallieri 2016:] Borrelli al. 2017
Raković & Levine 2019) is the most successful advanced control methodology for systems with hard safety constraints. At each time step, a finite horizon forecast is made from a predictive model of the system and the optimal actions are computed, generally relying on convex constrained Quadratic Programming (QP, [Boyd & Vandenberghe) 20044 [Bemporad et al.) (0000. Stability of the MPC in closed loop with the physical system requires the solution of a simpler unconstrained infinite horizon control problem (Mayne et al.) 2000) which results a value function (terminal cost and constraint) and a candidate terminal controller to be accounted for in the MPC forecasting. For Linear Time Invariant (LTI) models and quadratic costs, this means solving (offline) a Riccati equation (Kalman 200I) a lincar matrix inequality (Boyd et al. 1994). Under these conditions, an MPC controller will effectively control a system, up to a certain accuracy, provided that uncertainties in the model dynamics are limited (Limon et al.) (209). Inaccuracies in the MPC predictions can reduce its effectiveness (and robustness) as the forecast diverges from the physical system trajectory over long horizons. This is particularly critical in applications with both short and long-term dynamics and is generally addressed, for instance in robust MPC (Richards) 2004 Raković el al. 2012), by using controller to stassiail the predictions.
This paper presents an infinite-horizon differentiable linear quadratic MPC that can be learned using gradient- methods. In particular, the learning method uses an MPC controller where the terminal cost and terminal policy are the solution of an unconstrained infinitehhorizo Linear Quadratic Regulator (LQR). A closed-form solution for the derivative of the Discrete-time -ime Algebraic Riccati Equation (DARE) associated with the LQR is presented so that the stationary solution of the forward pass is fully differentiable. This method allows analytical results from control theory
"oorrsponning author.

---

Published as a conference paper at ICLR 2020
to be used to determine the stabilizing properties of the learned controller when implemented in closed-loop. Once the unconstrained LQR is computed. the predictive model is pre-stabilised using a linear state-feedback controller to improve the conditioning of the QP and the numerical accuracy of the MPC solution and gradients. The proposed algorithm successfully learns an MPC with both local stability and intrinsic robustness guarantees under small model uncertainties.
Contributions This paper provides a framework for correctly learning an infinite-horizon. LTI quadratic MPC using recent developments differentiable QPs (Amos & Kolter 2017) and principles from optimal control (Blanchini & Miani 2007). A primary contribution is that the Discrete- time Algebraic Riccati Equation (DARE) is used to provide infinite-horizon optimality and stability, and an
analytical derivative of the solution of the DARE is derived so that iifferentiationnbaseed optimization can be used for training. This connects known results on MPC stability (Limon et al., [2003] (2009) and on infinite-horizon optimality (Scokaert & Rawlings 2018).
1998) imitation learning (Osa al.
further contribution is the MPC control formulation: pre-stabilizing linear state-feedback con- troller is implemented from the solution of the DARE, and then the total control input is obtained as a perturbation of the feedback control law from the solution of a convex QP. The pre-stabilizing controller ensures that the QP is well conditioned and promotes a highly accurate global solution, which in turn ensures that the gradients calculated in the backwards pass are accurate. Additionally, an augmented Lagrangian penalty method is used to enforce constraints on state and control input. This approach ensures that the hard constraints are strictly enforced if the penalty term sufficiently large, and also guarantees that the MPC problem is feasible throughout the training process. These contributions are in contrast to {Amos et al., 2018) which did not consider state constraints, and implemented differential dynamic programming (Tassa al., 2014) method to solve the MPC optimization for which convergence could not be guaranteed.
The framework is implemented platooning model, where it is demonstrated that the infinite horizon cost can be learned and the hard constraints be guaranteed using short finite prediction horizon.
set of second order mass-spring-damppe systems and vehicle
identity matrix. Omxn == m matrix of zeros. O,, == a vector of zeros. 1, == vector of n ones. All inequalities S and 2 are considered element-wise the context of vectors. p(A) largest absolute cigenvalue of given matrix A. vec Rm + R R is defined as vec ([c)--. .c,]) == (C1, ................................ Cn), i.c. the columns of a matrix stacked into a vector. For matrix A E IR the V m_n mm permutation matrix is implicitly defined by Vm.. vecAA vec. The Kronecker product, 8, is defined as in (Magnus & Neudecker 1999 pp. 440).
Notation I, :=
2 DIFFERENTIABLE MPC
Linear quadratic MPC
This paper considers linear time invariant systems of the form
Tt Ax, + Bui:
where E IR" is the system state, u, E Rmm is the control input, A E Ruxx is the state transition matrix, B E Raxm is the input matrix, I is the time, and dt E is the timestep (assumed constant). The control problem for this system is to determine the sequence of values of u, that achieve desired level of performance (e.g. stability, frequency response, etc...), and when the system is subject to hard constraints on control input, U4 E U. and state, _; E X, (or combination of both), a well studied framework for controller synthesis is MPC. The principle control input and state are optimized over a finite prediction horizon, then the first element of the obtained control sequence is implemented at the current time step and the process is repeated ad infinitum. For linear MPC it is common to use quadratic stage cost and box constraints on state and control ( x S Ig < x and u S uy < u where u S 0 S u), so that at each time index the vector of optimized control variables it is obtained from
MPC is that the system's
2

---

Published as a conference paper at ICLR 2020
N-1 Σ k==
N-1 Σ k=1
1 Qūx + =2 Z + ku Σ
Vo.N argmin
T
Rūg
5x
m rh + Kx
Σ
1
8k
à
k=0
k=1
s.t. Zu Zt:
1t = Ai + Büks
E {0, (0..... N N-1}.
r S 0R S u + rk
and
Fk 2 0, k E (0................... N N-1),
x 8 S i I x + 5k and
8k 2 0,
kE fL....... N].
where No:N is the predicted control trajectory, # is the predicted state trajectory, R E Rmxm > 0 represents the stage cost on control input, Q E _=x > 0 represents the stage cost on state, QN E RRax" 2 0 represents the terminal cost on state, N E N is the prediction horizon, T E R are slack variables for the control constraint, 8g E I are slack variables for the state constraint, and ku E R > 0 and h2 E R > 0 represent the cost of control and state constraint violations. The variables s and enforce the box constraints on state and control using the augmented Lagrangian method (Noedall & Wright 2006 11.22), and it can be shown that for sufficiently high kar and ki the constraints 1 S 2k S T and y I Wi S can be exactly guaranteed (Kerrigan Maciejowski 2000) (i.e.
0). The benefit of this approach is that ensures that the MPC optimization feasible at each iteration of the learning process, whilst still ensuring that the constraints are 'hard'. To close the MPC control loop, at each timestep, t, the first element of the optimized control sequence, ügj is implemented as Ug.
Pre-stabilised MPC If the control input is decomposed into u, Kz + d't, where K E m= is a stabilizing linear state-feedback matrix and ou, is a perturbation to the feedback control, system ① becomes
Ie+d = (A+ + BK)x + B8u,
(3)
and problem becomes
N-1
1 &y Q+x 2
8ūg argmin
(Kăx + )ūx)
TR(KE + 8ūx)
Σ
UNâN
2
k=
N-1
N
+ ku
Σ k=0
TTrnk + hig
n
(4)
R=1
s.t. 80 = It.
fx+1 = (A + BK) + BBüg.
k (O....,N- 1},
"1 г < Kit + 5ūg T + rh
and 2k 2 0, k (0. N ---1,, 1}, b NJ,
I -- 3k S ct < + 8k and
ss 20
so that ig = Kx, + Sūg is implemented as ue Using this decomposition, system controlled with the solution of 4 is precisely equal to system ! controlled with the solution of 2 but problem (4) is preferable from computational standpoint if A is open-loop unstable (i.e. p A) > 1) and N is "large', as this can lead to poor conditioning of the matrices defined Appendix A This is important
in the context of differentiable MPC, if A is being learned then there may be no bounds on its eigenvalues at any given iteration.
MPC derivative. Problems and (4 can be rearranged into the QP form (details in Appendix A
I
argmin 2
Hz + q s.t. S M S ub.
When a* is uniquely defined by (5, can also be considered as the solution of an implicit function defined by the Karush-Ko Kuhnh Kuhn-Tucker (KKT) conditions, and in Amos & Kolter (017) it was demon- strated that it is possible to differentiate through this function to obtain the derivatives of 2* with respect to the parameters H, q. M, and u. The MPC controller can then be used as layer in
'Note that 5 differs from the form presented in Amos & Kolter (0017). and is instead the form of problem solved by the OSQP solver used in this paper. Appendix demonstrates how to differentiate (5) using the solution returned by OSQP.
3

---

Published as a conference paper at ICLR 2020
neural network, and backpropagation can be used to determine the derivatives of an imitation cost function with respect to the MPC parameters Q, R, A, B, u, U. I. T. kg and lgg
Imitation Learning. A possible use case of the derivative of model predictive controller is imitation learning, where a subset of {cost function, system dynamics, constraints] are learned from observations of system being controlled by an 'expert.. Imitation learning can be performed by minimizing the loss
T
Σ T t=
1
Ber+NNd No.N (Zx 112 +1 21
(6)
where is the measured control input, N (2z) is the full MPC solution, and 3 > 0 a hyperpa- rameter. It is assumed that both the learning algorithm and MPC controller have completely precise measurements of both the state and control input. The first term of ( is the control imitation loss, and the second term penalises the one-step ahead prediction error w = Ax + Bu, It+dt In practice, the prediction error loss might not be needed for the MPC to be learned correctly, however its use can be instrumental for stability, as discussed in the next section.
3 TERMINAL COST FOR INFINITE HORIZON
Terminal cost. The anfinite-horizon discrete-ting Linear Quadratic Regulator (LQR, Kalman 2001
given with state feedback gain
== PB) PA,
where P is obtained solution of the DARE
P= A PA A PB(R+ B BT
(8)
The principle of the approach presented in this paper is the MPC controller 42 is implemented with QN P. Proposition I summarises the relevant properties of the proposed MPC, building on classic MPC results from Scokaert & Raw kawlings) (1998;; Limon et al. (2003 2009).
Proposition 1. Consider the MPC problem 4 with QN = P. where P and K solve 778 Define
V (1) the optimal objective in (4) with xy v. Denote the optimal stage cost with xt as f(x, 0n (x)) x. Fx + ()(x)) Ruy (r). Then, for the closed-loop system, it follows that:
1. For any N 2 1, there exists closed and bounded set, Ωg such that, if xo E SNN and w. = 0, Vt z 0, then the MPC solution is infinite-horize optimal for any 2 N.
2. There exist positive scalars d, a, such that, for any N 2 1, if = O, Vt 2 0 then the MPC constraints are feasible, vt 2 O, and the origin is asymptotically stable Vxp E N, with
TN = far E Rd ; V+(x) < C(x, 1z(x)) (N 1)d a}.
3. There exist a scalar, 2 O, such that, for any Z the MPC constraints are robustly feasible, Vt Z O, and the system is Input-to-State Stable (ISS) Yao E I' given an additive model error, w. such that: [| S μ, VI 2 O. In other words:
(a(++a))) I VV E((x,, (x+)) + a ((1
for some strictly increasing, bounded function, o(-), with (0)
4. The QP matrices, H, M and the vector in 5 have fiite norms for any N 2 1.
Proof. Proof of Proposition T is given in Appendix C
Implications. Proposition | has some important implications. First, point implies that there exists a state-dependant finite horizon length, N, which is sufficient to make the MPC problem infinite-horizon optimal. This N can be upper bounded for closed and bounded set of feasible states, g2 Scokkert & Rawlings (1998) proposed an iterative search that increases the horizon until optimality is verified; similar algorithm is discussed in Appendix B where learning is completed
4

---

Published as a conference paper at ICLR 2020
with large horizon and then iteratively reduced afterwards, although it is not implemented in this paper. Point 2,3 state that MPC that can provide stability and constraints satisfaction, hence safery, the model error is small. This also applies to small emrors in the QP solution. Finally, point states that the QP matrices have finite norm when the system dynamics are pre-stabilised using the LQR gair the MPC problem is well conditioned and can be solved reliably to high accuracy, even over long horizons. If the open-loop system is unstable then the terms of the matrices in Appendix [A] for the standard form are unbounded, so the QP solution may be poorly conditioned and the result inaccurate for long horizons. This can in turn invalidate the results of Amos & Kolter (2017) which assumes that the KKT conditions are exactly satisfied in order to compute its gradients.
DARE Derivative. In order to implement QN P in differentiable imitation learning framework such as that presented in Section 2 the solution of the DARE is differentiated as follows.
Proposition 2. Let be the stabilizing solution of 18). and assume that Z and (R + B PB)-1 exist, then the Jacobians of the implicit function defined by ( are given by
OvecP
Ovec
ŌvecP
OvecP ÖveeR
= 1/2
:vecB = 77 733
= 7, 74,
77 755
OvecA
ivecQ
where ZI..... Z3 are defined by
21 = I,- - (AT @ [L (PBM3B @ L,) (I, @ PBM²B
+ +PB & PB) @ (B" & BT]
Z2 (V n. (I, M))
Zg = (A ) [PB & (M2 @ MI + mm (Im & P
(I, + ) @ P)]
Z4 = In2
Zs = (A @ A JPPB PB) @ M2),
and M1, M2, My are defined by
PBMPB P. : = My M3 = R+ BT PB.
Proof. The proof of Proposition 2 is given in Appendix D
The sensitivity of the DARE solution has been investigated in the context of robustness to perturbations in the input matrices, e.g. Sun 1998); Konstantinoy al. (1993). and the analytical derivative of the continuous time algebraic Riccati equation was derived in Brewer (1977) by differentiating the exponential of the Hamiltonian matrix, but to the best of the authors knowledge this is the first presentation of an analytic derivative of the DARE using the differential calculus approach of Magnus & Neudecker (1999).
Algorithm 1: Infinite-horizon MPC Learning
Algorithm overview Algorithm D presents the overall procedure for learning a subset, S, of the MPC controller parameters, MM = {A, B, Q, R, x, E, u, T. kn, kz), with the key steps of the forwards and backwards pass of a gradient-basse based optimization method. In each forward pass the MPC terminal cost matrix, QN. and the pre-stabilizing controller, K, are set from the solution of the DARE, then the DARE and MPC QP solutions are differentiated in the backward pass obtain the gradients. Note that the horizon, N, is not differentiable, and that learning the entire set M simultaneously is challenging in general.
In: M \ S, N > 0, 2 0, N, > 0. Out: S
for == O...N, do
Forward Pass
(K, P) DARE
solution
Qr <
N6. + MPC QP 5
solution
L L Imitatiion loss
6
Backward Pass
Differentiate loss
6
Differentiate MPC QP solution, Wg.N-
using Appendix
B
Differentiate DARE, (P,K).
using Proposition 2
Update step + Gradient-based step
Note that any stabilising gain would he acceptable for the purpose of QP conditioning only.
5

---

Published as a conference paper at ICLR 2020
4 NUMERICAL EXPERIMENTS
In this section the performance of the algorithm was demonstrated through numerical experiments in two test cases: firstly set of second order mass-spring-dampee models provide performance baseline in an easily interpretable setting, and then on vehicle platooning problem to investigate a higher-dimensional real-world application.
4.1
Model & Expert Expert data was generated using a mass-spring-damppe model parameterized by mass, m > 0, damping coefficient, E R, stiffness, &t E R, and timestep dt R > 0, where
1
0
A =xxp(A,dt) A. &
B = (A I Be.
B. =
1
so that 2yz I22 is the position and velocity of the mass, and the uN E R a force applied to the mass.
Seven models were created with m 1, 1. and dt = 0.2, and c was varied as shown in Table D to affect the open-loop stability of the models (c > 0 = stable, = unstable). The expert data then generated by simulating each of the systems the initial condition zi (0,3) in closed-loop with an infinite-horizon MPC controller (i.e. the horizon was increased until the open-loop state predictions matched the closed-loop response), using Q = diag([1, 1)), R = 2, (u, T) (-20, 0.5). x -oc), and Φ) = (1. oo). The constraint set was chosen so that the constraints on both state and control input were strongly active at the solution whilst ensuring that the expert MPC optimization was feasbile. The values k, = b2 = 100 were found to be sufficient to enforce the hard constraints and were used for all experiments. It is important to note that the approach of (Amos et al. 2018) cannot be used reliably for even this simple example as does not consider state constraints, and when hard constraints are added to the method it fails in general because the optimization problem has become infeasible in the forwards pass at some time t.
Table 1: Damping coefficient used to generate the seven imitation systems.
System
1
2
3
4
5
6
7
1
0.5
0.1
-0.1
-0.3
-0.5
-0.6
Learning The learner and pert shared all system and con- troller information apart from the state transition matrix A, which was learned, and the MPC hori- zon length, which was imple- mented as each of N E (,,3, 3,6
N
N 4
N
Imitation Loss
10a
10a
10a
10a
10
100
10-2 2
10
2
10
10-4
10
10
1000
1000
in three separate experiments. A was initialized with the cor- rect state transition matrix plus
Model Loss
0.50
0.50
0.50
0.25
0.25
0.25
a uniformly distributed pseudo- random perturbation in the in- terval 0-0.5, 0.5) added to each element. The learner was sup- plied with the first 50 elements of the closed loop state trajectory and corresponding controls as a batch of inputs, and was trained to minimize the imitation loss (6) with 0. i.c. the state dy- namics were learned using pre- dicted control trajectories only,
0.00
0.00
0.00
1000
0
1000
1000
Iteration
Iteration
Iteration
Figure I: Mass--pring-damppae Imitation loss and model loss each iteration of the training process. Top row: imitation loss. Bottom row: model loss given by ||vec - vec.A, where Aj is the learned model iteration j. and A is the correct model. The model loss was not used as part of the training process, and shown only to indicate whether the model is converging correctly.
and the state transitions not made available to the learner (this is the same approach used in Amos et al. 2018). The experiments were implemented in Pytorch 1.2.0 using the built-in Adam optimizer (Kingma & Ba 2014) for

---

Published as a conference paper at ICLR 2020
1000 steps using default parameters. The MPC optimization problems were solved for the "ex- pert' and 'learner' using OSQP (Stellato et al. 2017) with settings (eps_abs= abs=1E-10, eps_ rel=1E-10, eps rim 10-, eps dual inf=1E-10).
Results Figure T shows the imitation and model loss at each of the 1000 optimization iterations for each of the tested horizon lengths. It can be seen that for all of the generated systems the imitation loss converges to low value, although this is a local minimum in general. In most cases, the learned model converges to a close approximation of the real model, although as the problem non-convex this cannot be guaranteed, and it is also shown that there are some cases in which the model does not converge correctly. This occurred exclusively for N 2, where neither system nor system converge to the correct dynamics. Additionally, it can be seen that both the imitation loss and model loss converge faster as the prediction horizon increased. This suggests that longer learning horizon improves the learning capabilities of the methods, but there is not sufficient data to demonstrate this conclusively.
To test generalization perfor- mance, each of the systems was re-initialized with initial condi- tion zo = (0.5, 2) and simu- lated in closed loop using the learned controller for each hori- zon length. The results are com- pared in Figure 2 against the same systems controlled with an infinite horizon MPC controller. The primary observation is that as the learned MPC horizon is increased to N = 6, the closed loop trajectories converge to pert trajectories, indicating that the infinite horizon cost has been learned (when using the infinite horizon cost with no model mismatch disturbance, the predicted MPC trajectory is exactly the same as the closed loop trajectory), and that the state constraints are guaranteed for N > 4. Furthermore, it can be seen that the learned controllers are stabilizing, even for the shortest horizon and the most unstable open-loop systems. This is also the case for systems 2 and 4 where the incorrect dynamics were learned, although in this case the state constraints are not guaranteed for 2.
= 2
N =
6
50
50
50
1
3
Figure 2: Mass-ppring-damper Closed-loop trajectories using the expert and learned controllers. Trajectories only shown for In (position), but (velocity) can be inferred. Expert controllers shown with solid lines, and learned controller shown with dotted lines. The hard state constraints are shown in the red regions.
4.2 VEHICLE PLATOONING
Model & Expert Vehicle platoon control is problem that has been studied using control theory (e.g. iZheng et al. (2019)), but here it is demonstrated that safe, stabilizing controller can be learned from examples of vehicles driv- ing formation. of platoon of n. vehicles for which the objec- tive is to stabilize the relative longitudinal positions of each vehicle to the steady-state conditions y: Vi-1 Yas and ;); -i-1 - 0 Vi, subject to the hard constraint that relative position of the vehicles is never lower than a safe threshold y yt-1 2 y Vi, and that the vehicles' ability to brake and accelerate is constrained by b < üi < a Vi where b << 0 < (note that only the relative positions and velocities of the vehicles is considered, as the global position and velocity of the platoon can be controlled separately by adding an equal perturbation each element of új) In appendix [F it is shown that this can be modelled as a discrete time LTI system. Expert data was generated from the model with n, 10 vehicles that x, E 18 and u, E 10 20 instances were generated using random feasible initial conditions with yes = 30 m and y = 10 m, and then simulated for 20 s in time intervals of dt 0.7 s with an infinite horizonn
21
1/2
In -1 yru
Figure 3: Platoon Model. ny vehicles in degree of freedom where y is longitudinal displacement.
3 an illustration
using Q I, and 2I

---

Published as a conference paper at ICLR 2020
Learning The learner and expert shared all system and controller information apart from the cost matrices Q and R. which were learned, and the MPC horizon length, which was implemented as each of N E {5, 10, 15, 20} in four separate experiments. The matrices Q and R were initialized as completely random diagonal matrices with each element uniformly distributed in the interval [0, 3). and the diagonal structure was maintained through training. 500 training iterations were used; otherwise the learning process (loss function, learning rate, etc...) was the same as in Section 4.1
Results Figure 4 shows the imitation and cost function losses at each of the 500 optimization iterations for each of the tested horizon lengths and initial con- ditions. As with the serrsspddipppppp experiments, is suggested that a longer prediction horizon improves training the imitation loss generally converges lower value for the examples with N E {15, 20). but only convergence to local minimum achieved in general. The cost eтror also does not converge in general (although better convergence is observed again for the longer horizon lengths), however for this learning prob- lem there is manifold of matrices Q and R with the same minimizing argument, divergence of the cost eтror does not necessarily indicate that the learned cost function 'incorrect'. Furthermore, in this case the model is known exactly, the closed-loop infinite-horizon properties can be obtained even without the correct cost function.
Cost Funetion Loss
0.02
Imitation Loss
10a
10
0.01
10-1
500
500
Iterations = N
Iterations
10 N = 15
N 20
Figure 4: Vehicle platooning. Imitation loss and cost function loss at each iteration of the training process. Left: imitation loss. Right: model loss given by ||vec vecQ,llž + l|vec vecR, 12. where Q and R are the correct cost matrices and 2, and R, are the cost matrices at iteration j.
Figure shows the model simulated from the same initial condition in closed loop using learned controller for each of the horizon lengths, together with the error between the MPC state predictions and ensuing closed-loop behaviour. All of the controllers are observed to successfully satisfy the hard constraints on vehicle separation, and all converge to the correct steady-state vehicle separation. The differences between the prediction capabilities of the controllers is highlighted by the state prediction errors, and it can be seen that for N = 20 the state predictions match the ensuing behaviour, indicating that the infinite horizon cost being used and that closed-loop loop stability is guaranteed, even without the use of terminal constraint set. It is also demonstrated for N < 20 that the largest errors occur
N = 5
N = 10
N = 15
N - 20
yf (m)
2000
2000
2200
20
20
20
20
State Prediction
Error (m)
0-
-1
-1
-1
0
(s)
A
(8)
t (8)
Figure Vehicle platooning. Closed loop simulation and prediction error for all horizon lengths. Top row: closed loop simulation where each shaded region is the safe separation dis- tance for each vehicle. Bottom row: prediction error given by [| - 2t, where is the state trajectory predicted by the MPC at time
8

---

Published as a conference paper at ICLR 2020
from predictions made at times when the state constraints are active, suggesting that these controllers deviate from their predictions to satisfy the constraints at later intervals.
4.3 LIMITATIONS
The above approach limited in scope to LTI systems, and more comprehensive solution would cover linear time varying systems (for which the MPC is still obtained from the solution of QP). In this case the infinite horizon cost cannot obtained from the solution of the DARE, and the extension of the methods presented in this paper to time varying non-linear models is triviaivial (see Appendix G for further discussion). Additionally, the derivative of the DARE in Proposition 2 involves multiple Kronecker products and matrix inversions (including n2 x n? matrix) that do not scale well to large state and control dimensions, although the dynamics of physical systems can usually be reasonably approximated with only a few tens of variables, so this may not become an issue in practice. The algorithm also requires a stabilizing solution of the DARE to exist; theories for the existence of stabilizing solutions are non-trivial (e.g. Ran & Vreugdenhil 1988). and is not immediately obvious how to enforce their existence throughout the training process (stabilizibility can be encouraged using the one-step ahead term
ACKNOWLEDGMENTS
The authors are grateful to Brandon Amos for providing support using his differentiable QP tool (https:/ / / g thub, comm / occus 1 ab/ optnet) in the preliminary work for this project (all of the methods presented in this paper were developed independently).
REFERENCES
Brandon Amos and J. Zico Kolter. OptNet: Differentiable Optimization as a Layer in Neural Networks. orXiv:1703.0000 [cs, math, stat], March 2017. htLl:://///axx www.aiii..rraaas......
1703.00443 XrXiv: 1703.00443.
Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and Zico Kolter. Differentiable MPC for End-to-end Planning and Control. In Advances in Neural Information Processing Systems 31, pp. 8289-8300. Curran Associates, Inc., 2018.
A. Bemporad, M. Morari, V. Dua, and F.N. Pistikopoulos. The explicit solution of model predictive control via multiparametric quadratic programming. In Proceedings of the 2000 American Control Conference, ACC (IEEE Cat. No. No.00CH363344 IEEE, 2000. doi: 20.1099/cc.2000.876663 URL
/ttt::::/ooi..r//100.
2000. 876624
Franco Blanchini and Stefano Miani. Set-Theoretic Methods in Control (Systems & Control: Founda-
fions & Applications). Birkhäuser, 2007. ISBN 081 0817632557.
R. V. Bobiti. Sampling driven stability domains computation and predictive control of constrained non- linear systems. PhD thesis, 2017.
wtt..n///e//lllle///eee.//// wttss//llee/7785440000 :ue www./////w.........
201 20171025 Bobiti. PdE.
Francesco Borrelli, Alberto Bemporad, and Manfred Morari. Predictive Control for Linear and Hybrid Systems. Cambridge University Press, 2017. ISBN 1107016886.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004. ISBN 0521833787.
Stephen Boyd, Laurent El Ghaoui, Eric Feron, and Vendataramanan Balakrishnan. Linear Matrix Inequalities System Control Theory (Studies in Applied Mathematics, Volume 15). Society for Industrial & Applied, 1994. ISBN 089871334X
J. Brewer. The derivative of the riccati matrix with respect to matrix. IEEE Transactions on Automatic Control, 2286)998 December 1977. doi: 10.1 19T.:...997.111065.55
E. F. Camacho and C. Bordons. Model Predictive control. Springer London, 2007. doi: 10.1007/ 978-0-85729-398-5.

---

Published as a conference paper at ICLR 2020
Marco Gallieri. Lasso-MPC Predictive Control with f1- Re-Reuularised Least Squares. Springer International Publishing, 2016. doi: 10.1007/978.3-31--27963--
10.1007/978- 3-1199996---31112263-------
R. A. Horn and R. Johnson. Matrix Analysis. Cambridge University Press, New York, NY, USA,
2nd edition, 2012. ISBN 0521 9780521548233
Vlad Ionescu and Martin Weiss. On computing the stabilizing solution of the discrete-time riccati equation. Linear Algebra and its Applications, 174:229 238, 1992. ISSN 0024-3795. doi: sci
URL http:/ /wtp://wwww.....mm www www.ncediiiicccoo.oooo rect com/
Rudolf Kalman. Contribution the theory of optimal control. Bol, Soc. Mat. Mexicana, 02 2001.
Eric C. Kerrigan and Jan M. Maciejowski. Soft constraints and exact penalty functions in model predictive control. In Proc. UKACC International Conference (Control, 2000.
H. K. Khalil. Nonlinear Systems. Pearson Education, 3rd edition, 2014.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2014.
MM Konstantinov. P Petkov, and ND Christov. Perturbation analysis of the discrete riccati equation. Kybernetika, 29(1):18-29) 1993.
B. Kouvaritakis and M. Cannon. Model Predictive Control: Classical, Robust and Stochastic. Advanced Textbooks in Control and Signal Processing, Springer, London, 2015.
D. Limon, Alamo, and E. F. Camacho. Stable constrained MPC without terminal constraint. Proceedings of the 2003 American Control Conference, 2003., (:4893-4898 vol.6, 2003.
D. Limon, T. Alamo, D. M. Raimondo, D. Muñoz de Peña. M. Bravo, Ferramosca, and Camacho. Input-to-Statt Stability: A Unifying Framework for Robust Model Predictive Control. In Nonlinear Model Predictive Control, pp. 1-26. Springer Berlin Heidelberg, 2009. doi: 1007
978-3-642-01094-
htLhttts:/ / doi org. 10. 1507 00009777 642
Jan Maciejowski. Predictive Control with Constraints. Prentice Hall, 2000. ISBN 0201398230.
Jan R. Magnus and Heinz Neudecker. Matrix Differential Calculus with Applications in Statistics and Econometrics. John Wiley, second edition, 1999. ISBN 047198632) 9780471986322 047 198633X 978047198633..
D. Q. Mayne, J. B. Rawlings, C. V. Rao, and P. O. M. Scokaert. Constrained model predictive control: Stability and optimality. 2000.
J. Nocedal and S. J. Wright. Numerical optimization. Springer verlag, 2006.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An Algorithmic Perspective on Imitation Learning. Foundations and Trends in Robotics, 7
(1-2):1-179, 2018. ISSN 1935-8253, 1935-8261. doi: 10.1561/2300000053.. URL http?
//arxiv. /1811 06711 arXiv: 1811.06711.
S. V. Raković, B. Kouvaritakis, R. Findeisen, and M. Cannon. Homothetic tube model predictive control. Automatica, 48: 48:1631-1638, 08 2012. doi: 10.
Savecsa V. Raković and William S. Levine (eds.). Handbook of Model Predictive Contro.. Springer International Publishing, 2019. doi: 10. 00.1007/97--3-119-778888 URL htthhttss:///ooorgg
10.1007/97 0070077778-70 3 77489-3
A.C.M. Ran and R. Vreugdenhil. Existence and comparison theorems for algebraic riccati equations for continuous- and discrete-time systems. Linear Algebra and its Applications, 99:63 83, 1988. ISSN 0024-3795. doi: https///bbaaogllooooooooo4ssssss
URL http:
7/www. www.sscienced.. rect www htt.o//......ooec......../oaaaaa
10

---

Published as a conference paper at ICLR 2020
J. B. Rawlings and D. Q. Mayne. Model Predictive Control Theory and Design. Nob Hill Pub, Llc, 2009. ISBN 0975937707.
A. G. Richards. Robust Constrained Model Predictive Control. PhD thesis, MIT, 2004.
P.O.M. Scokaert and J.B. Rawlings. Constrained linear quadratic regulation. IEEE Transactions on Automatic Control, 4368):1163-116,, 1998. doi: 10.1 10.110999.70499. URL https://doi. org/
10.1109/9. 704994
B. Stellato, G. Banjac, Goulart, A. Bemporad, and S. Boyd. OSQP: An operator splitting solver for quadratic programs. ArXiv e-prints, November 2017.
Sun. Sensitivity analysis of the discrete-time algebraic riccati equation. Linear Algebra and its Applicationns, 275-276:595 615, 1998. ISSN 0024-3795. doi: https://doi.ong orgg/
001101668002--775557711010111110 URL httpp://ww: sciencedirectt. com/ science/ / Aaticcee/LL1/002479999110000 Proceedings of the Sixth Conference of the Interna- tional Linear Algebra Society.
Y. Tassa, N. Mansard, and E. Todorov. Control-limited differential dynamic programming. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 1168-1175, May 2014. doi: 0..1009/CRAA2214.69977
Y. Zheng, Y. Bian, S. Li, and S. E. Li. Cooperative control of heterogeneous connected vehicles with directed acyclic interactions. IEEE Intelligent Transportation Systems Magazine, pp. 2019. doi: 10.1 10.1109/S 28899584.

---

Published as a conference paper at ICLR 2020
APPENDICES
A MPC QUADRATIC PROGRAM
Problem 2 is equivalent to
R + v QQ
VQRRi
zº argmin
NmxNM
2 +
ky Nm
2
O NNxXNu Nu
1g
Inm
INm
OC
00
Inm
Nn
0 Nm
I Nn
oc
25
X
V
INa
x
0
Inn
On
I Inn
XC
where
Q
2
R
Q
R
AN
QN
1
X
X
11
AN-BB
B
are of conformal dimensions. Using the above, problem (4) is then equivalent to
(Kύ + INM) TR(K + INm) + + Qú
argmin
2
Nuv x Nyn.
NnxNN
K R IN +Q)) Thx
Nm
ki Nu (K INm) (K INm)
K&
INM
x
O0
-Inm
K
0 Onn
Inm
s.t.
V
I Nn
V
INo
X
0 NN
IN=
where now
88
(A+ BK)
B
and V
(A + BK)N
[(A
BK)
B
B
and
K
Q
[ nx
I
V
[
Nn
K
Q
Nmxx
of conformal dimensions.
12

---

Published as a conference paper at ICLR 2020
B
OSQP DERIVATIVES
OSQP solves quadratic programs of the form 5 and returns values for 2, y, and 8 that satisfy
H + M yyy-
SEC, Nc(s),,
(Stellato al. 2017 $2), where C is the set {s ls:lss l.ssss u), and We is the normal cone of C. The values that are returned by the solver can used to determine whether the constraints are strongly active the solution, where y; 0 indicates that the constraints I; S M, and M,z S us are inactive, yi > indicates that M, I ue is strongly active, and < O indicates that I; S M,z is strongly active. The solution therefore be completely characterised by the KKT system
H Mu Mc
Mu
2
Yer Lyc.
(10)
where 21 2= (i: 03 and L {i: y4 < 0}. and the notation Ms indicates matrix consisting of the i E S columns of given matrix M, and ti indicates a vector consisting of the i S elements of given vector v. Equation (10) can then be differentiated using the techniques detailed in (Amos Kolter 2017 $3).
C PROOF OF PROPOSITION
Proof. (Proposition I The first point follows from (Scokaert & Rawlings, 1998). The next two points of Proposition D stem from the results in (Limon al., 2003 2009). In particular, the closed- loop is Lipschitz since the model is linear and the controller is the solution of a strictly convex QP. Moreover, the LQR provides a contractive terminal set. The final point follows from the fact that (A + BK) has eigenvalues in the unit circle, vN 2 1. Proof of point 4 is concluded by inspection of the QP matrices (Appendix [ and by application of Theorem 5.6. 12, page 298 of Horn & Johnson (2012) which states that, given bound, 7, on the spectral radius, then there exists matrix norm which is also less than p.
D PROOF OF PROPOSITION
Proof. (Proposition 2 If stabilizing solution (p(A BK) to (8 exists, is unique [Ionescu & Weiss 1992 Proposition 1). and the DARE can therefore be considered an implicit function of A, B. Q, and R. Using the assumption that (R+ BT PB) 2B exists, can concluded that Zi.... ., Zs and M1, M2, M3 exist (the Kronecker product and matrix addition, subtraction, and multiplication
always exist). Equation ( can be given by
P A MMA+ Q
(11)
which is differentiable, and M1, M2, My are also differentiable. Differentials are taken for II and each of M1, M3
dvee =(Vn, n,n II,, @ A T Mj dvee (AT s A J dvecQ
dvecM/ = [T,- - (PBM, & I.) - (I, @ PBM, B T dveeP
(PB & PB)dvecM2 - (L, + Vn.x) & P)yeedB
dvecM2 -M2 & Mvedve
dvecMs =dvecR + (BT x B dveeP (I,, V mm.m) (Im & B P)
then these can be combined using the differential chain rule (Magnus & Neudecker 18.2) to obtain
1999 Theorem
ZydvecP ZzdvecA ZzdvecB + Z,vvec + ZsdvecR.
The Jacobians, defined in Proposition 2 therefore exist if - exists.
13

---

Published as a conference paper at ICLR 2020
E VERIFICATION AND REDUCTION OF THE PREDICTION HORIZON
Algorithm 2: MPC horizon verification and reduc-
method is proposed for the reduction of the MPC prediction horizon after im- itation learning. The idea is to be able reproduce the infinite-horizon optimal MPC up to a tolerance e with high prob- ability. Do do so, we check that, for a candidate horizon N, the MPC action deltas, g satisfy | |I S 6, for all 2 N. This means that the optimal ac- tion is equal to the LQR up to a tolerance In order to provide high probability guarantee of this condition, we propose use of probabilistic verification ap- proach, similar to Bobiti (2017). This is described in Algorithm 2 In particu- lar, the condition is checked on high number, ng, of initial states. These states sampled uniformly from a set of in- terest X, which can be either the state constraints X or an estimate of the region of attraction, Nv. If verified, this set a region of attraction for the system with high probability. The relationship between the number of samples and the verification probability discussed in (Bobiti 2017 Chapter 5). The algorithm also checks whether the infinite horizon condition has been reached for the N used during training. Finally, a line search for suitable proposed using a scaling factor 7) E (0, 1). ). particular, the initial is downscaled until either an horizon is found or the set becomes empty. In latter case the search fails and the procedure returns to the training algorithm with an increased N. Noticeably, the proposed algorithm does not require to explicitly compute the terminal set in which the LQR is invariant and it could be used also for non-linear MPC an infinite-horizon (or a stabilising) terminal controller available.
tion
In:
N > 0, Xo C. (P, K) from 7 80 0,
ns 0. 17 E (0,1).
Out: N,
X < Xo
while X p do
+ N
while N > do
Xample n, uniform state samples, s.t.: X
Rü + Solution of MPC QP (15), Vx E Naampl
(z]ā())]| & e, Vk 2 N, Vo E Mesapii then
return TRUE
Ñ+Ñ-1
X c 7
Procedure failed
N <- N-N+ +1 Go to Algorithm
F PLATOON MODEL DERIVATION
The problem described in Section
can be decomposed into the regulation problem
1/2 V1
21
22
+ 1ysss
- 0,
y/n Mn-1
Zn Zn
Zn Zn-1
subject to the constraints
22 21
1 (as), and 1b S ÿ S la.
2n 2n-1
If each vehicle is modelled as mass then a continuous-tims LTT state space model can be formed as
2 2
1
22
21
Ú1
2m
2
n
2n
2n-1
+
(12)
22
82
&1
1
Ac
B.
which can then be given
[
I
0
0
x
+
B
n.
14

---

Published as a conference paper at ICLR 2020
If it is assumed that the control input is constant between sampling intervals t and + de, then this can be given discrete time as
[-1
I dtI O
2 (dtt
+
B
(13)
I
Bodt
where Ia E p2((a 1), and u R and subject the constraints
T1 Uss In
and b an al VI.
G NONLINEAR MODELS
As discussed in the main paper, our approach is currently limited to Linear Time Invariant (LTT) systems. In general, conditions for infinite-horizon optimality of systems that not LTT are trivial. Some of the results on MPC stability could however be maintained, for example in the case when the LQR value function, x Px, local control Lyapunov function (Khalil 2014 Mayne et al. 2000p. In this case, the stability and intrinsic robustness results are maintained (see Limon et al. 2003 2009). For these system, it would be possible use our method, for instance in combination with Amos al. (2018), to provide a stable Non-linear MPC. This is however big assumptions for systems that are very non-linear. Assessing this LQR controllability condition could be done, for instance, by training& local linear model around the target equilibrium (origin) and then checking whether the DARE is solvable. This should be performed before starting the imitation learning. We leave the study of more general systems to future work.
15


Reviews:
[{"review_id": "rkeMnzRk9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper shows how to use the Discrete-time Algebraic Riccati Equation (DARE) to provide infinite horizon stability & optimality to differentiable MPC learning.  The paper also shows how to use DARE to derive a pre-stabilizing (linear state-feedback) controller.  The paper provides a theoretical characterization of the problem setting, which shows that prior work on differentiable MPC learning may lead to unstable controllers without the proposed augmentations using DARE. \n\nI'm not sure I understand the implications of imitating \"from an expert of the same class\".  Can the authors elaborate?\n\nCan the authors compare & contrast with this paper? \nhttps://arxiv.org/abs/1709.07174 \n(I have my own views, but I'd like hear the authors' thoughts first)\n\nMy biggest complaint is with regards to the experiments.  Unless I'm mistaken, it seems there isn't a thorough empirical study of the theoretical claims, especially as it relates to previous work.  E.g., can one construct scenarios where the baseline approach (Amos et al., 2018) fails, and compare with the proposed approach?\n\nThe idea of pre-stabilization is interesting, and seems related to this paper: https://arxiv.org/abs/1905.05380\n\n\n\n**** After Author Response ****\nThanks for the response, I am raising my score to weak accept.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJeZNvcxcB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper continues the recent direction (e.g. Amos & Kolter 2017) of differentiating through optimal control solutions, allowing for the combination of optimal control methods and learning systems. The paper has some nice contributions and I find this research direction to be very exciting, which is why I think it merits acceptance, however I find the experiments (Section 4) could be greatly improved.\n\nThe main contribution of the paper are the analytical derivative of the solution to the DARE. The pre-stabilising controller reformulation is a neat trick. \n\nThe main issue I have with this paper is that the experiments are performed only on a toy 2D problem. Even an LTI system can be interesting! Of course it is important to start with a toy problem, but once positive results have been shown, it would be much more convincing if the paper showed some more complicated system, possibly an iteratively linearised non-linear system. My feeling (and possibly many others') is that these type on differentiable controllers can be extremely powerful, however this power is sadly not demonstrated here.\n\nerrata:\nbefore eq (3): dt is not a pertubation to the feedback control\neq (4) argmin over \\delta u rather than \\delta, presumably\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1x0gLTRKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows how to make infinite-horizon MPC differentiable\nby differentiating through the terminal cost function and controller.\nRecent work in non-convex finite-horizon continuous control [1,2,3] face\na huge issue in selecting the controller's horizon length and\nbetter-understanding differentiable infinite horizon\ncontrol has potentially strong applications in these domains.\nAs a step in this non-convex direction, this paper provides a nice\ninvestigation in the convex LTI case.\nThe imitation learning experiments on a small spring dynamical\nsystem are a necessary sanity check for further work, but\nmany other more complex systems could be empirically studied\nand would have made this paper stronger.\n\nOne point that would be useful to clarify: the DARE solution in (7,8) is\nderived to optimally control a LTI system *without* control/state bounds but\nis then used to control the LTI system *with* control/state bounds in (4).\nDoes this lead to suboptimal solutions to the true infinite-horizon problem?\n\n[1] Chua, K., Calandra, R., McAllister, R., & Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. NeurIPS 2018.\n[2] Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. Learning latent dynamics for planning from pixels. ICML 2019.\n[3] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, Jimmy Ba. Benchmarking Model-Based Reinforcement Learning. arXiv 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a linear quadratic model predictive control approach for safe imitation learning.  The main contribution is an analytic solution for the derivative of the discrete-time algebraic Riccati equation (DARE).  This allows an infinite horizon optimality objective to be used with differentiation-based learning methods.  An additional contribution is the problem reformulation with a pre-stabilizing controller and the support of state constraints throughout the learning process.  The method is tested on a damped-spring system and a vehicle platooning problem.

The reviewers and the author response covered several topics. The reviewers appreciated the research direction and theoretical contributions of this work.  The reviewers main concern was the experimental evaluation, which was originally limited to a damped spring system.  The authors added another experiment for a substantially more complex continuous control domain.  In response to the reviewers, the authors also described how this work relates to non-linear control problems.  The authors also clarified the ability of the proposed method to handle state-based constraints that are not handled by earlier methods.  The reviewers were largely satisfied with these changes.

This paper should be accepted as the reviewers are satisfied that the paper has useful contributions.
Decision: Accept (Poster)