DiLQR: Differentiable Iterative Linear Quadratic Regulator via Implicit Differentiation
Authors: Shuyuan Wang
Philip D Loewen
Michael Forbes
Bhushan Gopaluni
Wei Pan
Venue: ICML 2025 poster (2025)
Keywords: iLQR; Differentiable control; Learning based control
Primary area: reinforcement_learning->planning

Abstract: While differentiable control has emerged as a powerful paradigm combining model-free flexibility with model-based efficiency, the iterative Linear Quadratic Regulator (iLQR) remains underexplored as a differentiable component. The scalability of differentiating through extended iterations and horizons poses significant challenges, hindering iLQR from being an effective differentiable controller. This paper introduces DiLQR, a framework that facilitates differentiation through iLQR, allowing it to serve as a trainable and differentiable module, either as or within a neural network. A novel aspect of this framework is the analytical solution that it provides for the gradient of an iLQR controller through implicit differentiation, which ensures a constant backward cost regardless of iteration, while producing an accurate gradient. We evaluate our framework on imitation tasks on famous control benchmarks. Our analytical method demonstrates superior computational performance, achieving up to $\textbf{128x}$ speedup and a minimum of $\textbf{21x}$ speedup compared to automatic differentiation. Our method also demonstrates superior learning performance ($\mathbf{10^6x}$) compared to traditional neural network policies and better model loss with differentiable controllers that lack exact analytical gradients. Furthermore, we integrate our module into a larger network with visual inputs to demonstrate the capacity of our method for high-dimensional, fully end-to-end tasks. Codes can be found on the project homepage~\url{https://sites.google.com/view/dilqr/}.

DiLQR: Differentiable Iterative Linear Quadratic Regulator via Implicit Differentiation
Shuyuan Wang Philip D. Loewen'
Michael G. Forbes2
R. Bhushan Gopaluni! | Wei Pan
Abstract
1. Introduction
Differentiable control has emerged as a powerful approach in the fields of reinforcement learning (RL) and imitation learning, enabling significant improvements in sample ef- ficiency and performance. By integrating control policies into a differentiable framework, researchers can leverage gradient-based optimization techniques to directly optimize policy parameters. This integration allows for end-to-ene training, where both the control strategy and the underly- ing model can be learned simultaneously. enhancing the adaptability and precision of control systems.
While differentiable control has emerged as a powerful paradigm combining model-free flex- ibility with model-based efficiency, the iterative Linear Quadratic Regulator (iLQR) remains un- derexplored as differentiable component. The scalability of differentiating through extended it- erations and horizons poses significant challenges, hindering iLQR from being effective differen- tiable controller. This paper introduces DiLQR, a framework that facilitates differentiation through iLQR, allowing it to serve as a trainable and dif- ferentiable module, either as or within a neural network. A novel aspect of this framework is the analytical solution that it provides for the gradient of an iLQR controller through implicit differenti- ation, which ensures a constant backward cost re- gardless of iteration, while producing accurate gradient. We evaluate our framework on imitation tasks on famous control benchmarks. Our analyti- cal method demonstrates superior computational performance, achieving up 128x speedup and minimum of 21x speedup compared to auto- matic differentiation. Our method also demon- strates superior learning performance (10%x) com- pared to traditional neural network policies and better model loss with differentiable controllers that lack exact analytical gradients. Furthermore, we integrate our module into larger network with visual inputs to demonstrate the capacity of our method for tigh-dimensional, fully end-to- to- end tasks. Codes can be found the project homepage https: / sites.. google e. com/ view/ /ilgr/.
As numerical controller, the iterative Linear Quadratic Reg- ulator (iLQR) (Todorov et al., 2012) has been extensively adopted for trajectory optimization (Spielberg et al., 2021; Choi et al., 2023; Zhao et al., 2020; Mastalli et al., 2020) due its computational efficiency (Tassa et al., 2014; Dean et al., 2020; Collins al., 2021) and excellent control per- formance (Dantec et al., 2022; Xie et al., 2017; Chen et al., 2017). To make iLQR trainable as neural network module, naively differentiating through an iLQR controller may be reasonable choice, but the scalability of differentiating through hundreds of iterations steps poses significant chal- lenge, as the forward and backward passes during training are coupled. The forward pass involves iteratively solving an LQR optimization problem converge on the optimal trajectory. The backward pass computes gradients through hackpropagation, and becomes increasingly complex as it needs to traverse through all the layers of the forward pass, which requires significant computational resources (time and memory), especially for tasks requiring long iterations and long horizons. This coupling not only increases mem- ory usage, but also significantly slows down the training process, making difficult to scale to larger problems.
Efficient differentiable controllers are especially valuable in systems involving neural networks, such multi-modal frameworks (Mao et al., 2023; Xu et al., 2024b; Xiao et al., 2022) and deep reinforcement learning (Ye et al., 2021; van Hasselt et al., 2016), where an upstream neural network module is required. Developing differentiable controllers with efficient gradient propagation is crucial, they greatly enhance sample efficiency and reduce computational time for online tuning.
'The University of British Columbia, Vancouver, Canada 2 Honeyweel Process Solutions, Vancouver, Canada 'The Uni- versity of Manchester, Manchestor, England. Correspondence to: Wei Pan twei.pan@e @ manchester.cc.uk. Bhushan Gopaluni <bhushan sbhushan.gopaluniee @ ubc.ca>.
Proceedingy of the n2nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025, Copyright 2025 by the author(s).

---

DiLQQ: Differentiabl ILQR via Implicit Differentiation
Time
LQR
LQR
LQR
AenCM
L(T)
AutoDiff into 1LQR
LGR
Implicit Diff (analytical gradient)
Our Ixplcc10
r
L(T)
iteration!
iteration"
Forward Pass
Backward Gradient Pass
Figure 1: An overview of iLQR, and AutoDiff vs our proposed planner with implicit differentiation. As shown in the flowchart, autodiff must backpropagate through each layer of the LQR process, which leads to significantly increased memory usage store intermediate gradients and computational load. In contrast, our proposed planner, using implicit differentiation, only needs to handle the final layer. This results in constant computational costs and memory usage, making our method much more efficient.
Developing analytical solutions would greatly alleviate these challenges. DiffMPC (Amos et al., 2018) pioneered the use of analytical gradients in LQR control, leading to significant improvements in computational efficiency and generalization of the learned controller. Its success has inspired extensions in various planning and control applica- tions (East et al., 2020; Romero et al., 2024b; Karkus al., 2023; Cheng al., 2024; Soudbakhsh et al., 2023; Shrestha et al., 2023). Numerous studies have since shown that ana- lytical gradients significantly improve learning performance, reducing computational costs, and improving scalability in complex, long-horizon tasks (Jin al., 2020; Xu et al., 2024a; Jin et al., 2021; Böttcher et al., 2022; Zhao et al., 2022).
for optimal control problems with tunable additive cost functions and constrained dynamics described by first- order difference equations, focusing iLQR as the controller. Our analytical solution is exact, considering the entire iLQR graph. The method guarantees (11) computational complexity with respect to the number of iteration steps.
2. We propose forward method for differentiating linearized dynamics with respect to nonlinear dy- namics parameters, achieving speeds dozens of times faster than auto-differentiation tools such as torch . aut ad. jacobian. Furthermore, exploit the sparsity of the tensor expressions to com- pute some tensor derivatives that scale linearly with trajectory length.
In this paper, we introduce an innovative analytical frame- work that leverages implicit differentiation handle iLQR at its fixed point. This approach effectively separates the forward and backward computations, maintaining constant computational load during the backward pass, irrespective of the iteration numbers for iLQRR By doing so, our method significantly reduces computational time and the memory usage needed for training, thereby enhancing scalability and efficiency in handling non-convex control problems. See Figure/ for a conceptual comparison.
3.
We demonstrate the effectiveness of our framework in imitation and system identification tasks using the inverted pendulum and cartpole examples, showcas- ing superior sample efficiency and generalization com- pared to traditional neural network policies. Finally, integrate differentiable iLQR into a large net- work for end-to-end learning and control from pixels, demonstrating the extensibility and multimodal capa- bilities of our method.
This paper makes the following contributions.
Notation For function (().) where g depends on 6, we distinguish between partial and total derivatives. The partial derivative treats g as independent, while the total
1. We develop an efficient method for analytical differ- entiation. We derive analytical trajectory derivatives
2

---

DILQQ:: Differentiabl ILQR via Implicit Differentiation
derivative accounts for indirect dependencies via the chain rule: df ef de as + ef dg For readability, we use V&f as 3g shorthand for d/ ag. emphasizing the total variation. When f and are vectors or tensors, as represents its Jacobian. See eq. (2) for concrete examples.
through planning procedures by explicitly unrolling the op- timization algorithm itself, introducing drawbacks such as increased memory and computational costs and reduced computational stability (Zhao et al., 2022; Bai et al., 2019). DiftMPC (Amos et al., 201 is a representative work in the field of differentiable MPC. Significant progress has been made in the efficient differentiable LQR with box constraints by (Amos et al.. 2018). To differentiate iLQR, (Amos et al., 2018) proposes a methodology that differen- tiates through the last layer of iLQR to avoid unrolling the entire iLQR graph. However, (Amos et al., 2018; Dinev et al., 2022) treats the imput to the last layer of LQR as a constant, rather than a function of the learning parameters. Using implicit differentiation, we develop a framework that provides exact analytical solutions for iLQR gradients, im- proving the gradient computation presented in (Amos et al., 2018). Our approach not only addresses scalability issues, but also improves learning performance.
of
2. Related work on differentiable planning
Pure model-free techniques for policy search have demon- strated promising results in many domains by learning re- active policies that directly map observations to actions (Haarnoja et al., 2018; Sutton & Barto, 2018; Schulman et al., 2017; Fujimoto et al., 2018). However, due to the black box nature of these policies, model-free methods suf- fer from lack of interpretability, poor generalization, and high sample complexity (Ye et al., 2021; Yu, 2018; Bacon et al., 2017; Deisenroth & Rasmussen, 2011). I). Differentiable planning integrates classical planning algorithms with mod- em deep learning techniques, enabling end-to-end training of models and policies, thereby combining the complemen- tary advantages of model-free and model-based methods. Value Iteration Network (VIN) (Tamar et al., 2016) is a representative work that performs value iteration using con- volution on lattice grids and has been extended further (Niu et al., 2018; Lee et al., 2018; Chaplot et al., 2021; Schleich et al., 2019). These works have demonstrated significant performance improvements on various tasks.
3. Background
The Iterative Linear Quadratic Regulator (iLQR) addresses the following control problem:
T
min Σ
ge(We, uz)
(1)
1=1 s.t. Te4 = [e un), I1 = Finiti
" uuuus SuuSu u.
However, these works primarily focus on discrete action and state spaces. In the field of continuous control, most efforts have focused on differentiable LQR, including dif-
At each iteration step. it lincarizes the dynamics and makes quadratic approximation of the cost function to produce finite-time Linear Quadratic Regulator (LQR) problem.
ferentiating through finite horizon LQR (Amos et al., 2018;
Shrestha et al., 2023), infinite horizon (East et al., 2020;
Solving this auxiliary problem produces updates for the original trajectory. Here are some details.
Brewer, 1977). and constrained LQR (Bounou et al., 2023).
References (Jin et al., 2020; 2021; Böttcher et al., 2022) pro- pose frameworks that can differentiate through Pontryagin's 's Maximum Principle (PMP) conditions. However, the con- vergence speed of PMP-b methods slower than that of iLQR (Jin et al., 2020), due to the 1.5 order convergence rate of iLQR. More importantly, these methods and their enhanced approach (Xu et al., 2024a) assume broad range of forward pass solutions and do not align the gradient in the backward pass with the forward solution.
3.1. The Approximate Problem
Iteration i begins with the trajectory 7 = -Ti T/r), where -/ = {x1, wf}. For I = 1,,,..., T, we linearize the dynamics by defining
......TT
Of
D. [c]
Ox
Ou
For iLQR, which is powerful numerical control technique (Todorov et al.. 2012; Li & Todorov, 2004; Zhu et al., 2023), (Tamar et al., 2017) differentiates through an iterative LQR (iL.QR) solver to learn a cost-shaping term offline. Other numerical control techniques include
mr
[r
d, = [e ut)
Dr
u
methods based (Okada et al., 2017; Pereira et al., 2018), which provide methods to differentiate through path integral optimal con- trol, and (Srinivas et al., 2018), which shows how to embed differentiable planning (unrolled gradient descent over ac- tions) within dire-dire policy.
and form a quadratic approximation of the cost function using
Dg. Ox
0gt Ou
G [c Cr,u]
c
(3)
C. = Cr
Cr,oul
However, all of these methods require differentiation
Craz
Cr,ens)
3

---

DiLQQR: Differentiabl ILQR via Implicit Differentiation
where
The same is true for the coefficients in the quadratic approx- imation the loss function in the original problem. Careful accounting for the @-dependence at every level is required for accurate gradients.
pg. Ox.
8h gt Ou2
Ct
Cr.
Cr O
8gg gt. dudxx
Suppose the loss function L to be minimized by "learning" @ is expressed entirely in terms of the trajectory т. Then the influence of 0 on the observed L-values will be indirect, and we will need the chain rule to express the gradient of the composite function 0 - [()())):
These elements lead to an approximate problem whose un-
knowns are 87;- = Ti - 7/:
Vo(L ()( V-T(((0)) OT
(6)
min 511T
dn, C,ôr, + c. 8T.
00
2
(4)
1=0
In practical implementations, the partial derivatives required to form V.L are provided during the backward pass by automatic differentiation tools (Paszke al., 2019; Abadi et al., 2015). The main challenge, however, is to determine @c 00 i... the derivative of the optimal trajectory with respect to the learnable parameters. This is the focus of the next section.
s.t. 00 Djrr, 8x1 = 0; us 8.
3.2. The Trajectory Update
Problem (4) can be solved by the two-pass method detailed in (Tassa et al.. 2014). First, a backward pass is conducted, using the Riccati-Mayne method (Mayne et al., 2000) to obtain a quadratic value function and a projected-Newton method to optimize the actions under box constraints. Then forward pass uses the linear control gains K,, k₂ obtained in the backward pass to roll out a new trajectory. Let ôr* denote the minimizing trajectory in (4). We use the controls in 8 directly, but discard the states in favor of an update based on the original dynamics, setting
4.2. Fixed point differentiation
For a particular choice of 0, we can consider the sequence of trajectories produced by iLQR:
7 LQR +1 LQR 72 LQR
LQR T T
LQR,
LQR
(7)
Each iteration includes the three steps noted above: lin- earizing the system, conducting the backward pass, and performing the forward pass. Iterations proceed until the output 7* from an iLQR step is indistinguishable from the input, indicating that the process input trajectory. This trajectory T" is called a fixed point for the iLQR. We expect the value of 0 to influence the fixed point produced above.
++ " u + ōu:.
xtt1 = f(att) u4 ".
(5)
With these choices, defining rf+1 {artt ut oft") provides
a feasible trajectory for (1) that can serve as the starting point for another iteration.
no longer improve the
4. Differentiable iLQR
In general, an operator's fixed point can be calculated by various methods, typically iterative in nature. As pointed out in (Bai et al., 2019), naively differentiating through such a scheme would require intensive memory usage (Tamar et al., 2016; Lee al., 2018) and computational effort (Zhao et al., 2022). Instead, propose to use implicit differentiation di- rectly on the defining identity. This gives direct access to the derivatives required by decoupling the forward (fixed-point iteration as the solver) and backward passes (differentiating through the solver).
4.1. End-to-end learning framework
In the learning problem of interest here, the cost functions gr and system dynamics fe involve structured uncertainty pa- rameterized by a vector variable 0. For example, in a drone, 0 could represent physical parameters like mass or propeller length, while in a humanoid robot, it might refer to limb lengths or joint masses; additionally, 0 can include refer- ence trajectories for robot tracking, which help parametrize the cost function for control. Suppressing 0 in the notation is typical when 0 has fixed value, but now we face the challenge of choosing 0 to optimize some scalar criterion. This requires changing the notation to ft = fe(x,, u. 0) and 9t = ge (z. u, 6). As such, the derivatives shown in (2) and (3) must also be considered as functions of 0. So, along a given reference trajectory т, the dynamics in (1) will gener- ate three d-dependent matrices we must consider:
Let us write X .u.,...,))) for the components of a trajectory T = (I1, U2, ; ur), and abuse notation somewhat by identifying with (X, At fixed point (XUU) of the iLQR process for parameter 0, we have the following:
(II....,TT) and U
X- F(= U G(X U
(8)
A,( = 0f
Bi( = 0f and
0f
where and G summarize the operations that define a single
Ox
Ou
00

---

DiLQR: Differentiabl ILQR via Implicit Differentiation
iteration in the iLQR algorithm. (Thus eq. (8) formalizes the graphical summary in eq. (7).)
In each term on the right, the first matrix factor (e.g.. HH (aD) expresses the sensitivity of the optimal LQR tra- jectory with respect to the corresponding named ingredient of the formulation in eq. (4). Efficient methods for calculat- ing these terms are known: see (Amos et al., 2018; Amos & Kolter, 2017). The second factor in each term of (13) can be computed using automatic differentiation. The next subsections detail how to calculate these terms efficiently.
In eq. (8), the solutions X and U* depend on the parameter
0. By treating both X* and U explicitly as functions of 0. we can interpret eq. (8) as an identity valid for all 0. Differentiating through this identity yields new one:
VaX* OF
OF
0F
0x +
VeX*
VOU*
+
aU
00 ,
8G
aG
aG
4.4. Parallelization
VOU*= OX
V
+
au VaU* + a0
(Amos et al., 2018) propose a method that directly calcu- lates 8L aL aL and aL with a complexity of only O(T). B7) ad 80 We adopt these results in our framework. To facilitate par- allelization, we construct batches of binary loss functions.
Here, the matrix- -aluue partial derivatives of F and G above are evaluated at (X* (0), U'' (0) Likewise, V and Val* are the Jacobians (sensitivity matrices) that quantify the O-dependence of the optimal trajectory; both depend on 0. Rearranging eq. (9) produces a system of linear equations in which these two matrices provide the unknowns:
Be
Specifically, to compute 0Hi, we set the L., element in L to 1, while all other elements are set to 0, and then calculate aL Although this approach introduces more computations, the computations can be fully parallelized since each opera- tion is completely independent. As a result, the calculation ØF
8D
OF 0x
OF VOU*=
0F
(1
VoX
OU
00
,
(10)
can be parallelized efficiently. The same method also of aD aH aH and 2H applies to ad OC de
aG
0G OU VOU*
DG
Ox VOX* + +
00
The analytical solution for this system is given below.
4.5. Exploiting sparsity
Proposition 4.1. The Jacobians in eq. (10) are given by
Some care is required when coding the calculations for which eq. (13) provides the models. With X (II,..., .T) above, and the corresponding D (D.....,yy Dr)), the quantity AD ax suggests huge structure in- volving T22 submatrices of the general form 0D, However, the definitions in eq. (2) show that any such submatrix in which 1' * / will be zero. Thus the matrix aD is block diagonal. Our implementation never instantiates it Instead, we work directly with the coormation-bearing diagonal blocks aD.
VeX* = MFF + Fo(KK - Gx (e.MMB))-(GGAAA MF - -G))
VOU* = =K - (xK-GxMP)-(CGGG (Gx + Go),
(11)
where we denote M = =T--fx) and K- I - Gu, and
use the condensed notation
OF
OF
OF
ax shown above
Fx 0X
Fu = OU'
Fg = 00 ,
(12)
aG
Gu = 00 Go = 0G
8G
Gx = 0x 1
Da, 1 TT
00
See the Appendix A.I.
4.6. Forward algorithm
To be completely explicit, suppose parameter 0 is given. Then eq. (8) defines a fixed point 7* in terms of this particular 0, and this 7* provides the evaluation point (X*()), (0). o) for all the Jacobian matrices involving F and G in Equations (9) to (11).
It can be costly to evaluate matrices like aD a0 In Pytorch, for example, such tools such as torch. autograd. jacobian rely on backpropa- gation, which means that gradient information from one time step is not reused for the next time step. However, the derivation above makes it clear that knowing aD,-1 allows for direct calculation of aD.
80
4.3. Obtaining each term
The functions F and G whose Jacobians appear in eq. (12) are defined by rather complicated arg min operations. The Chain-Rule pattern below, which we can apply to either
We now propose an efficient forward approach that uses available information efficiently to accelerate later steps. We refer to aD, from (13) as Do here for clarity and to 80
H = F or H- = G, suggests that
distinguish it from other gradient notations, convention
we apply similarly to other gradients such as ddg
HH 0D
OH Ad Dd Ox
HH ac
0H de de Ox
Hx 3D 0x
+
+
8C 0x
Given a trajectory satisfying F++1 = =f(er; uz 0), the ma- trices Dr and de defined in eq. (2) are functions of T,, u,
OH 8D
0H 0d
OH ac
0H dc
Hu 0D OU
(13)
0d aU
OC OU
de at
ÖH 0D
HH ad
HH 8C
HH de
Hg =
+
aD 00
0d 00
0C 00
dc 00
5

---

DiLQQ:: Differentiabl ILQR via Implicit Differentiation
and 0. For time step i, we will have
We follow the examples and experimental setups from pre- vious works (Amos et al., 2018; Jin et al., 2020; Xu et al., 2024a; Watter et al., 2015) and conduct experiments on two well-known control benchmarks: CartPole and Inverted Pendulum. The experiments demonstrate our method's computational performance (at most 128x speedup) and superior learning performance (106 improvement). All ex- periments were carried out on a platform with an AMD 3700X 3.6GHz CPU, 16GB RAM, and an RTX3080 GPU with 10GB VRAM. The experiments are implemented with Pytorch (Paszke al., 2019).
OD:
aD:
0D, Oue Sug dzl
VoD,
+
Vo
(14)
00
dx
with
Oxa
dr du--1
Vgz 00
+
+
V (15)
dr--1 du-1 dr
where 8D. a2, aD and Bag REL, Bay are analyti- cally calculated in first so that on each time step we only need to instantly plug in the corresponding parameter values are the lin- ear control gain solved from FT-LQR. VOzT-1 is the stored information from time step t - 1 and reused here, and Va prepared for the next time step + 1. Finally
80 Oze Shss
30 B 8u--1
5.1. Computational Performance
to obtain the numerical gradients. Our and 0 Dz. d
The performance of our differentiable iL.QR solver shown in Figure 2. We compare it to the naive approach, where the gradients are computed by differentiating through the entire unrolled chain of iLQR. The results of the experiments clearly demonstrate the significant computational advantage of our method over Auto across all configurations.
aD,
I 8uc
Ved+ = V8
Dr
VaTte
00
(16)
V 2, de
0D [x)]]
Vn,de de
aDg
Backward pass efficiency: For example, for horizon of 10 and 300 iterations, AutoDiff takes 8.57 seconds com- pared to just 0.067 seconds with our method, resulting in 128x speedup. Even in the case with the smallest im- horiizn of 10 and 50 iterations, AutoDiff takes 1.41 seconds, while our method remains 0.067 seconds, still delivering a 21x speedup. These results highlight the clear scalability and efficiency our method, maintaining a near- constant computation time the number of iLQR iterations increases, while AutoDiff's 's time grows significantly with longer horizons and more iterations.
dr:
Oue
Ut
The calculation of VyCt and Vec, is similar.
Algorithm 1 Forward Algorithm
1: Input: aD, aD. aD.
au 0n, du, and dz. D: (i
2: Initialize variables V Vo 0
3: for time step / 1,2,..... T do
4: obtain Vœz, through (15)
5: obtain VeD, with Vaw, and (14), and obtain Vad
with Vex, and (16)
6: end for
5.2. Imitation Learning
7: return VoD, Vod
Imitation learning recovers the cost and dynamics of a con- troller through only actions. Similarly to (Amos et al., 2018), we compare our approach with Neural Network (NN): An LSTM-based approach that takes the state x as input and predicts the nominal action sequence, directly optimizing the imitation loss directly; SysId: Assumes that the cost of the controller is known and approximates the parameters of the dynamics by optimizing the next-state transitions; and DiffMPC (Amos et al., 2018). We evaluated two variations of our method: DiLQR.dx Assumes that the cost of the controller is known and approximates the parameters of the dynamics by directly optimizing the imi- tation loss; DiL.Q.costt: Assumes that the dynamics of the controller are known and approximates the cost by directly optimizing the imitation loss. Following DiffMPC (Amos et al., 2018), we adopt the same dataset size convention where "train=50" and "train=100" denote the number of expert trajectories available during training.
4.7. Methodological Comparison and Discussion
Differences between our method and DiffMPC (Amos et al., 2018) DiffMPC treats input X* and U* as constant and uses auto-diiferentiation to obtain aD and finally use the chain rule to obtain the derivative of the optimal trajec- tory. We improve DiffMPC by further considering the input X" and U as a function of 0, that is, (0) and (0), and leverage implicit differentiation on the fixed-point to solve the exact analytical gradient, improving the accuracy of the gradient. The box in 17 illustrates the differences between the two approaches
29
A'(F', 0f( dr
0 VoAL
0 0r
00
0 00
(17)
Imitation Loss: In Figure 3, we compare our method with NN and Sysid using imitation loss, under the train=100 set-
5. Experiments
6

---

DiLQR: Differentiabl ILQR via Implicit Differentiation
Hatson 1D
Hanson 20
Horzon 20
Autabiff Our method
AutoDIf Our method
AutoDif Our methed
Backogge Targ
93
100
190
203
206
200
53
100
153
264
200
300
50 190
133
204
250
300
LOR Ioeraions
ILOR Iterations
ILOR Iterations
Figure 2: Backward computation time comparison between AutoDiff and our proposed method across different iLQR iterations and LQR horizons. A computation time scales linearly with the number of iterations, while our method maintains constant computation time. The experiments are conducted under pendulum domain, with batch size 20.
Initation Comparison
Model Loss: In Figure 4, compare the model error learned from our approach to that of DiftMPC. Model loss is defined as the MSB(0 - 0), where 0 represents the param- eters of the cost function. In the dcost mode, our approach recovers more accurate model parameters than DiffMPC, reducing model loss by 32%, indicating improvement over our analytical results.
We further evaluate the dynamics learning mode. Model loss here is defined as the MSE(# - -). where A represents the parameters of the dynamics. Figure 5 shows DiffMPC's error plateaus early, while DiL.QR achieves 41% lower fi- nal error through steady optimization (with train=50), Al- though DiLQR converges slightly slower with train=100, it maintains strong physical consistency (2.76% negative parameters vs. DiffMPC's 16.85% Table 1), largely avoid- ing non-physical solutions that could destabilize control. These results demonstrate DiL.QR's s dual advantages in both accuracy and physical plausibility. See Appendix A.5 for clarification on the slight discrepancy with the rebuttal.
Figure 3: Learning results on the pendulum and cartpole. We select the best validation loss during training and report the test loss (Amos et al., 2018), averaged over five trials.
ting. Notably, our method performs the best in the dx mode across both tasks, achieving performance improvement of orders of magnitude- 10 and 10*-over the NN. In the dcost mode, our method is also dozens of times stronger than the NN but slightly weaker than Sysid. This is because Sysid directly leverages a system model with state estimates, while imitation learning relies solely on action data, which contains less information. The fact that our method achieves comparable results to Sysid in this mode demonstrates its effectiveness.
Agpregated Learning Curve on Dynamic (Cartpsle)
DILOR- train100
DifMMPC train100
DILOR train50
DifMPC train50
Aggregated Learning Curve Cost (Cartpole, Train100, Truncated
0.45
100
0.40
Figure 5: Model loss curves for dynamics learning under the cartpole domain, trained for 500 epochs.
0.50
0.25
Table I: Bad-Value Ratio (% of negative values in learned dynamics)
9.4
0.8
homaltood Epech (Tuncated at 80%
Train Size
DiLQR
DiffMPC
Figure 4: Model loss curves for cost function learning under the cartpole domain, trained for 500 epochs.
dx (train=100)
2.76% 7.23%
16.85% 17.82%
dx (train=50)
7

---

DiLQQ:: Differentiabl ILQR via Implicit Differentiation
Comparison to Other Differentiable Control Methods While SafePDP(Jin et al., 2021) and IDOC(Xu et al., 2024a) report an imitation loss of ~1e-2 under its official code- base, these results are obtained under experimental settings that differ from ours specificil, using ground-truth-nn parameter initialization and expert trajectories generated by an oracle solver (IPOPT). We progressively align these conditions with our own experimental setup. As shown in Figure 6, once the expert trajectories are replaced with those generated by iLQR, DiLQR exhibits a clear performance ad- vantage over both SafePDP and IDOC (As mngle-trajectory training is the default setting in both SafePDP and IDOC, all methods are trained using the same single expert trajectory generated by iLQR.). This highlights the methodological gap and underscores DiLQR's effectiveness for learning structured models in -R-eciifiii settings.
To quantify the contribution of each module, compare the computation time of different ablation configurations, including removing parallelization (Sec. 4.4), sparsity explo- ration (Sec. 4.5), and the forward algorithm (Sec. 4.6). We evaluate these configurations on Pendulum across varying horizons.
Computational Time Parallelization (Sec. 4.4) and the forward algorithm (Sec. 4.6) significantly reduce compu- tation time. As shown in Figure removing the forward algorithm leads to sharp increase in computation time across all horizons, while disabling the parallelization fur- ther exacerbates this effect, especially for longer horizons.
Discussion Among the tested modules, the forward algo- rithm (Sec. 4.6) has the greatest impact on computation time, especially for long-horizon settings. Parallelization also plays a critical role in reducing computational cost. These results validate our design choices and highlight the necessity of each component for scalable differentiable con- trol. Interestingly, at horizon 30, the computation without parallelization is slightly faster than with parallelization. This due to the lower clock frequency of our GPU com- pared to the CPU. In practice, adaptive switching scheme would be employed to select the optimal computation strat- egy based on the horizon and batch size.
Thining Loss Comcarionn OILOR SatePPP and DOC
LWferan Home
1757
400
teration
5.4. Visual control
Figure 6: Training loss comparison on the Cartpole task. DiLQR is compared against SafePDP and IDOC under two settings: (1) same initial dynamics parameters as DiLQR, and (2) same dynamics parameters plus same expert tra- jectories. Losses are plotted as reported by each method. DiLQR demonstrates significantly lower final loss.
We next explore a more complex, high-dimensional task: controlling an inverted pendulum system using images as input.
Encoder
==>
DILOR
Decoder
5.3. Ablation study
Figure 8: Diagram of the end-to-end control architecture. The encoder maps compressed set of four input frames to the physical state variables. The differentiable iLQR then steps the state forward using the encoder's parameters. The decoder takes the predicted state and generates a future frame to match true future observation.
Nido:
with by nl oxtion wit fensaed A Aorthim
Coroobation Tim2
In this task, the state of the pendulum is visualized by a rendered line starting from the center of the image, with the angle representing the position of the pendulum. The ob- jective is to swing up the underactuated pendulum from its downward resting position and balance it. The network ar- chitecture consists of mirrored encoder-decoder structure, each with five convolutional or transposed convolutional layers, respectively. To capture the velocity information, we stack four compressed images as input channels. An example of observations and reconstructions is provided in Figure 8.
4.2
30
6.
1toraon
Figure 7: Computation time comparison across different ablation settings as a function of horizon length. Error bars measure standard error across samples.
8

---

DiLQQ: Differentiabl ILQR via Implicit Differentiation
grated into policy network and trained using an actor-critic approach, enabling more efficient policy updates (Romero et al., 2024a; Romero et al.). With its ability to propagate gradients through the control process, our method could enhance RL's performance, potentially achieving state-of- the-art results in more advanced tasks.
Figure 9: Imagined trajectory in the pendulum domain. The first image (red) represents the real input, while the following images are "dreamed up" by our model based on the initial image.
Another direction is combining our method with perceptual control frameworks such as DiffTORI (Wan et al., 2024). While DifTTORT emphasizes generality and visual input handling, our approach offers precise and efficient gradients for structured systems. Integrating the two could enable scalable, end-to-end learning in high-dimensional, multi- modal tasks.
Our modular approach handles the coordination between the controller and decoder seamlessly. Figure 9 shows sample images drawn from the task depicting trajectory generated by our system. In this scenario, the system is given just one real image and, with the help of DiLQR, it can output sequence of predicted images, which closely approximate the actual trajectory of the pendulum. We further design autoencoder that connects the trained encoder and decoder to predict trajectory images in an autoregressive manner. As shown in Figure 10, the network incorporating differentiable iLQR achieves significantly higher prediction accuracy with lower variance.
6.3. Comparison with General Differentiation Frameworks.
Our method addresses key limitation in general-purpose frameworks such Theseus (Pineda et al., 2022) JAXopt, which can differentiate through fixed-point equations of the form x* = f(="). In contrast, the iLQR optimality condition is inherently recursive: both the dynamics and cost functions depend on the optimal trajectory x* itself, yielding relation of the form x = fx-
Images
Such self-referential structure breaks the assumptions of standard auto-implicit differentiation, which cannot han- dle recursive dependencies without problem reformulation. Our method bridges this gap by recasting the iLQR recur- sion as fixed-point equation suitable for exact differentia- tion non-going beyond an implementation tweak to address a structural limitation in existing frameworks.
Autoencoder
Time Step
6.4. Limitations
Figure 10: Comparison of image prediction across trajectory steps. The shaded area denotes the standard error over 25 trials.
Our method relies on the assumption that iLQR converges to a fixed point and requires access to both first-order and second-order derivatives of the dynamics. These constraints may limit its applicability to systems where such conditions hold.
6. Discussion
6.1. Scope of Experiments
7. Conclusions
In this paper, we focus on the theoretical aspects of differ- entiable control methods. While our experiments are based on simpler control tasks, the advantages of our approach promise to extend to more complex, real-world applications. Many prior works (Amos et al., 2018; Watter et al., 2015; Xu et al., 2024a; Jin al., 2020) also rely on such toy exam- ples to demonstrate foundational concepts. A demonstration of our method on a high- -igh-dimensionall rocket control task is available on the project page.
In this work, we introduced DiLQQ,, an efficient framework for differentiating through iLQR using implicit differen- tiation. By providing an analytical solution, our method eliminates the overhead of iterative unrolling and achieves O(1) computational complexity in the backward pass, sig- nificantly improving scalability. Experiments demonstrate that DiLQR outperforms existing methods in both train- ing loss and model loss, making it promising approach for real-time learning- based control applications. In future work, aim to explore relaxation techniques or alternative formulations to extend its applicability.
6.2. Future direction
One promising direction is embedding our differentiable controller into RI. frameworks. For instance, it could be inte-
9

---

DiLQQ:: Differentiabl ILQR via Implicit Differentiation
Impact statement
Brewer, J. The derivative of the riccati matrix with respect to matrix. IEEE Transactions on Automatic Control, 22 (6):980-958) 1977.
This paper presents work aimed at advancing the field of Machine Learning. The societal implications of our research discussed in the Discussion section, and do not find any requiring additional emphasis here.
Chaplot, D. S., Pathak, D., and Malik, J. Differentiable spatial planning using transformers. In International conference on machine learning, pp. 1484-1495. PMLR, 2021.
Acknowledgments
We gratefully acknowledge the financial support of the Nat- ural Sciences and Engineering Research Council of Canada (NSERC) and Honeywell Connected Plant. We also thank Dr. Brandon Amos, Dr. Nathan P. Lawrence, and the re- viewers for their insightful discussions.
Chen, J., Zhan, W., and Tomizuka, M. Constrained iterative lqr for on-road autonomous driving motion planning. In 2017 IEEE 20th International conference on intelligent transportation systems (IFSC), pp. 1-7. IEEE, 2017.
Cheng, S., Kim, M., Song, L., Yang. C., Jin, Y., Wang, S., and Hovakimyan, Difftune: Auto-tuning through auto- differentiation. IEEE Transactions on Robofics, 2024.
References
Abadi, M., Agarwal, A., Barham, P., Brevdo, E.,, Chen, Z.,
Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L.., Kudlur, M., Lev- enberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke. V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng. X. TensorFlow: Large- scale machine learning on heterogeneous systems, 2015. : / www. tersorrrrrrrro floowwllwww.......o www.woooo////oooooooooooooooo Software available from tensorflow.org.
Choi, S., Ji, G., Park, J., Kim, H., Mun, J., Lee, H., and Hwangbo, J. Learning quadrupedal locomotion deformable terrain. Science Robotics, 8(74)xeade2256, 2023.
Collins, J.,, Chand, S., Vanderkop, A., and Howard, D. re- view of physics simulators for robotic applications. IEEE Access, 9:51416-51431, 2021.
Dantec, E., Naveau, M.. Fernbach, P., Villa, N., Saurel, G., Stasse, O., Taix, M., and Mansard, N. Whole- body model predictive control for biped locomotion on a porque-controllee humanoid robot. In 2022 IEEE-RAS 2 st International Conference on Humanoid Robots (Hu- manoids), pp. 638-644. IEEE, 2022.
https:
Amos, B. and Kolter, J. 7. Optnet: Differentiable opti- mization as a layer in neural networks. In International Conference on Machine Learning, pp. 136-145, PMLR, 2017.
Dean, S., Mania, H., Matni, N., Recht, B., and Tu, S. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics, 20(4):633- 679, 2020.
Amos, B., Jimenez, I., Sacks, J., Boots, B., and Kolter, J. Z. Differentiable MPC for end-to-end planning and control. Advances in neural information processing systems, 31, 2018.
Deisenroth, M. and Rasmussen, C. E. PILCO: model- based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on chine learning (ICML-11), pp. 465-472, 2011.
Bacon, P.-L., Harb, J., and Precup, D. The option-critic architecture. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), 2017. doi: 10.1609/aaai. v31i1.10916.
Dinev, T., Mastalli, C., Ivan, V., Tonneau, S., and Vijayaku- mar, S. Differentiable optimal control via differential dy- namic programming. arXiv preprint arXiv:2209.01117 2022.
Bai, S., Kolter, J. Z., and Koltun, V. Deep equilibrium mod- els. Advances in neural information processing systems, 32, 2019.
Böttcher, L., Antulov-Fantulin, N., and Asikis, Al Pon- tryagin or how artificial neural networks learn to control dynamical systems. Nature communications, 13(1):333, 2022.
East, S., Gallieri, M., Masci, J., Koutnik, J., and Cannon, M. Infinite-horizon differentiable model predictive control. Proceedings of ICLR 2020, 2020.
Bounou, O., Ponce, J., and Carpentier, J. Leveraging proximal optimization for differentiating optimal con- trol solvers. In 2023 62nd IEEE Conference on Decision and Control (CDC), pp. 6313 6320. IEEE, 2023.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Interna- tional conference on machine learning. pp. 1596, 2018.
10

---

DiLQQ: Differentiabl ILQR via Implicit Differentiation
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off offlpo maximum entropy deep reinforce- ment learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Pereira, M., Fan, D. D., An, G. N., and Theodorou, E. MPC- inspired neural network policies for sequential decision making, arXiv preprint arXiv:
2018.
Pineda, L., Fan, T., Monge, M., Venkataraman, S., Sodhi, P., Chen, R. T., Ortiz, J., DeTone, D., Wang. A., Anderson, S., et al. Theseus: A library for differentiable nonlinear optimization. Advances in Neural Information Processing Systems, 35:3801-3818 2022.
Jin, W., Wang, Z., Yang, Z., and Mou, S. Pontryagin dif- ferentiable programming: An end-to-end learning and control framework. Advances in Neural Information Pro- cessing Systems, 33:7979-7992, 2020.
Romero, A., Aljalbout, E.,, Song, Y., and Scaramuzza, D. Actor-critic model predictive control: Differentiable opti- mization meets reinforcement learning.
Jin, W., Mou, S., and Pappas, G. J. Safe pontryagin differ- entiable programming. Advances in Neural Information Processing Systems, 34: 16050, 2021.
Romero, A., Song, Y., and Scaramuzza, D. Actor-critic model predictive control. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 14777-14784, 2024a. doi: 10. N0.1109/CRRA77177.2000 10610381.
Karkus, P., Ivanovic, B., Mannor, S., and Pavone, M. Diff- stack: A differentiable and modular control stack for autonomous vehicles. In Conference on robot learning, pp. 2170-2180. PMLR, 2023.
Lee, L., Parisotto, E., Chaplot, D. S., Xing, E., and Salakhut- dinov, R. Gated path planning networks. In Interna- tional Conference on Machine Learning. pp. 2947-2955. PMLR, 2018.
Romero, A., Song, Y., and Scaramuzza, Actor-critic model predictive control. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 14777-14784. IEEE, 2024b.
Li, W. and Todorov, E. Iterative linear quadratic regulator design for nonlinear biological movement systems. In First International Conference on Informatics in Con- trol, Automation and Robotics, volume 2. pp. 222-229. SciTePress, 2004.
Schleich, D., Klamt, T., and Behnke, Value iteration networks on multiple levels of abstraction, arXiv preprint arXiv:/90S /0068 2019.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms.
Mao, J., Qian, Y., Ye, J., Zhao, H., and Wang, Y. Gpt- driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415, 2023.
arXiv preprint arXiv: 1700107777.077.77 2017.
Shrestha, J., Idoko, S., Sharma, B., and Singh, A. K. End- to-end learning of behavioural inputs for autonomous driving in dense traffic. 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp- 10020-10027. IEEE, 2023.
Mastalli, C., Budhiraja, R., Merkt, W., Saurel, G., Ham- moud, B., Naveau, M., Carpentier, J., Righetti, L., Vi- jayakumar, S., and Mansard, N. Crocoddyl: An efficient and versatile framework for multi-contact optimal control. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 2536-2542, IEEE, 2020.
Soudbakhsh, D., Annaswamy, A. M., Wang, Y., Brunton, S. L.., Gaudio, J., Hussain, H., Vrabie, D., Drgona, J., and Filev, D. Data-driven control: Theory and applications. In 2023 American Control Conference (ACC), pp. 1922- 1939. IEEE, 2023.
Mayne, D. Q. Rawlings, J. B., Rao, C. V., and Scokaert, P. O. Constrained model predictive control: Stability and optimality. Automatica, 36(6):789-8)
2000.
Niu, S., Chen, S., Guo, H., Targonski, C., Smith, M., and Kovačević, J. Generalized value iteration networks: Life beyond lattices. I Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Spielberg, N. A., Brown, M., and Gerdes, J. C. Neural network model predictive motion control applied to auto- mated driving with unknown friction. IEEE Transactions Control Systems Technology, 1934)):11441944 2021.
Okada, M., Rigazio, I.,, and Aoshima, T. Path integral net- works: End-to-end differentiable optimal control. arXiv preprint arXiv:1706. 170.0777 2017.
Srinivas, A., Jabri, A., Abbeel, P., Levine, S., and Finn, C. Universal planning networks: Learning generalizable representations for visuomotor control. In International conference machine learning, pp. 4732-4741 PMLR, 2018.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.

---

DiLQQ: Differentiabl ILQR via Implicit Differentiation
Tamar, A., Wu, Y., Thomas, G., Levine, S., and Abbeel, Value iteration networks. Advances in neural information processing systems, 29, 2016.
Yu, Y. Towards sample efficient reinforcement learning. In IJCA,, pp. 5739-5743, 2018.
Zhao, I.,, Xu, H., and Wong, I.. Scaling up and stabilizing differentiable planning with implicit differentiation. arXiv preprint arXiv:2210. 22..:2320.. 2022.
Tamar, A., Thomas. G., Zhang, T., Levine, S., and Abbeel, P. Learning from the hindsight plan-episodie MPC improvement. In 2017 IEEE International Conference Robolics and Automation (ICRA), pp. 336-343, IEEE, 2017.
Zhao, W., Queralta, J. P., and Westerlund, Sim-to-real transfer in deep reinforcement learning for roboties: survey. In 2020 IEEE symposium series on compulational intelligence (SSCI), pp. 737-744. IEEE, 2020.
Tassa, Y., Mansard, N., and Todorov, E. Control-limited differential dynamic programming. In 2014 IEEE Inter- national Conference on Robotics and Automation (ICRA),
Zhu, J., Payne, J. J., and Johnson, A. M. Convergent ilqr for safe trajectory planning and control of legged robots. arXiv preprint arXiv:2304.03346 2023.
pp. 1168 1168-117. IEEE, 2014.
Todorov, E., Erez. T., and Tassa, Y. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ inter- national conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double Q-learning. In Proceedings of the 30th AAAI conference on artificial intelligence, vol- ume 30, pp. 2094-2100, 2016. doi: 10.1609/amai.v3001 10295.
Wan, W., Wang, Z., Wang, Y., Erickson, Z., and Held, D. Difftori: Differentiable trajectory optimization for deep reinforcement and imitation learning. arXiv preprint arXiv:2402.05422) 2024.
Watter, M.. Springenberg, J., Boedecker, J., and Riedmiller, M. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
Xiao, X., Zhang, T., Choromanski, K., Lee, E., Francis, A., Varley, J., Tu, S., Singh, S., Xu, P., Xia, F,, al. Learning model predictive controllers with real-time attention for real-world navigation. arXiv preprint arXiv:2209.1072 2022.
Xie, Z., Liu, C. K., and Hauser, K. Differential dynamic programming with nonlinear constraints. 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 695-702. IEEE, 2017.
Xu, M., Molloy, T. L., and Gould, S. Revisiting implicit differentiation for learning problems in optimal control. Advances in Neural Information Processing Systems, 36, 2024a.
Xu, Z.., Zhang, Y., Xie, E.,, Zhao, Z.., Guo, Y., Wong, K.- Y. K., Li, Z., and Zhao, H. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters, 2024b.
Ye, W., Liu, S., Kurutach, T., Abbeel, P., and Gao, Y. Mas- tering Atari games with limited data. Advances in Neural Information Processing Systems, 14:25476-25448 2021.
12

---

DiLQQ:: Differentiabl ILQR via Implicit Differentiation
A. Appendix
A.I. Proof of proposition 1
Proposition A.1. Define Fo
oF Fu
a8 Fx = OF Gg == 0G
G=:=
Gu == 80 Gx == 00
Define M == (T- [==))==+
80 and == I Gy.- The analytical form of the gradients
8x.
38
8x
ax d5
and dü are given as follows:
dX
= M(Fo + - -x
d0
(18)
dU
d0 = (K [K-GXMPA Go)
Proof. With the new notations, equations can be rewritten as:
(T Fx) dx"
Fu dU
Fg
dB
d8
(19)
dx*
du
G
+ (I
Gu) d0
Go
d8
Focusing on the first equation,
4X 27
can be represented with du.
20
dX
dU = (I - F + Fu
d0
d0
(20)
dU
= M + Fu do
Then, substituting 20 into the second equation of 19 obtain an equation with respect to only de
du
dU"
-M + FU d 1) (I Gv)
Go
(21)
d0
Solving equation 21 will give the solution
d8
dU
(K (GX
(22)
d0
Substituting 22 into 20, the solution to ax
can be obtained:
dØ
dX
M(Fo + Fo(K - GxMFE
(23)
d0
This completes the proof.
A.2. Experiments Details
We refer the methods in DiffMPC as DiffMPC.dx: Assumes the cost of the controller is known and approximates the parameters of the dynamics by directly optimizing the imitation loss; DiffMPC.cost: Assumes the dynamics of the controller are known and approximates the cost by directly optimizing the imitation loss. For all settings involving learning the dynamics (mpe.dx, DiftMPC.cost. DiLQR.dx, and MHL.R.coss.dxy. the pendulum domain, the parameters the masses of the arm, length of the arm, and gravity; and in the cartpole domain, the parameters are the cart's mass, pole's mass, gravity, and length. For cost learning in DiffMPC.cost and DiL.QR.cost we parameterized the controller's cost as the weighted distance to goal state C(T) = || T. As indicated in (Amos et al., 2018), simultaneously learning the weights w, and goal state was unstable. Thus, we alternated learning wy and Ty independently every 10 epochs.
parameterized version of the true dynamics used. In
Training and Evaluation We collected dataset of trajectories from expert controller (iLQR with true system parameters) and varied the number of trajectories our models were trained on. The NN setting was optimized with Adam with learning rate of 10 44 and all other settings were optimized with RMSprop with a learning rate of 10 - and a decay term 0.5.
13

---

DiLQQ: Differentiabl ILQR via Implicit Differentiation
A.3. Detailed Network Archttecture for Visual Control Task
Encoder The encoder is neural network designed encode input image sequences into low-dimensional state represen- tations. It is implemented as subclass of torch. nn, Module, and consists of five convolutional layers and a regression layer:
Convolutional layers: Each layer applies 2D convolutions, followed by batch normalization, ReLU activations, and max pooling. These operations progressively reduce the spatial dimensions of the input image.
Regression layer: After the final convolutional layer, the output is flattened and passed through three fully connected layers, mapping the extracted features to the desired output dimension, which represents the system state.
The forward pass takes an input tensor of shape (batch, 12, 224, 224] (representing four stacked RGB images) and processes it through the convolutional layers. The output is state vector of shape [batch, out_dim)
The decoder mirrors the structure of the encoder and is also subclass of torch. nn Modull.. It reconstructs images from the low-dimessional state vector. The decoder consists of five transposed convolutional layers followed by regression layer:
Decoder
Transposed convolutional layers: These layers progressively upsample the input, applying batch normalization and ReLU activations after each layer to restore the spatial dimensions.
Regression layer: This layer, consisting of three fully connected layers, transforms the low-dimensional input vector into form suitable for the initial transposed convolution.
The forward pass takes state vector of shape [batch, 3] input, upscales it through the transposed convolution layers, and outputs reconstructed image tensor of shape [batch, 3, 224, 224]. A Sigmoid activation is applied to ensure the pixel values remain within the range [O, 1].
A.4. Additional Experiments on Model Loss
Aggregated Learnnng Curve on Cost (Cartpole, Train100, Truncated
0.50
MSE (A] Oimordono)
0.45
34
0.40
0.35
0.30
66
0.25
0.4
Normalized Epoch (Tuncated 80%)
komes
HETVACE E2cch
Figure 11: Aggregated Cost Learning Curve (Cartpole, Train=100, 500 epoch).
Figure 12: Per-Dimension Cost Learning Trajectories.
Observation:
DiLQR converges significantly faster than DifIMPC (Figure 11). In terms of dim-wise (Figure 12), although DiLQR and DiffMPC exhibit similar trends in Dim I and Dim 2, in Dim 0 the difference is significant: DiLQR (blue) steadily converges toward the ground truth (black dashed), while DiffMPC (red) consistently diverges in the wrong direction, highlighting a failure to capture the correct gradient signal.
14

---

DiLQQ: Differentiabl ILQR via Implicit Differentiation
A.5. Note Diserepancy from Rebuttal.
In our rebuttal, we reported 0.0% bad-value ratio for DiLQR under train=100. That statistic was generated with the help of Al-assisted data summarization tool during fast-paced analysis, where truncated runs and inconsistent learning records were not fully filtered. In the final version, we reprocessed all experiments using stricter pipeline, including run alignment, truncation exclusion, and ground truth-based comparison. The resulting value (2.76%) remains significantly lower than DiffMPC (16.85%) and does not affect our original conclusions.
15


Reviews:
[{"review_id": "KofPyATgxg", "reviewer": "Reviewer_oCbp", "summary": "This paper presents a method for differentiating through iLQR. Naively autodifferentiating through a trajectory optimization problem backpropagates through the iterative optimization problem, incurring a growing computational burden. Like prior works such as DiffMPC, proposing smarter ways to compute the requisite objects such as providing analytic expressions can immensely reduce the computational and memory burden. This paper proposes an implicit differentiation approach to compute the exact iLQR graph without unrolling the entire optimization trajectory. The efficiency of this approach is demonstrated on various standard control tasks, demonstrating large improvements in efficiency and generalization compared to standard prior approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the positive evaluation and constructive suggestions. Below we address the specific questions raised:\n\n---\n\n**1. Computational Complexity Estimates (Q1):**  \nWe clarify the computational costs with corrected exponents and iteration factors:\n\n- **Forward pass (iLQR):**  \n  O(IT(n² + m²)) for T timesteps and I iterations (standard LQR complexity)\n\n- **Backward pass (diLQR.dx/cost):**  \n  O(T(n² + m²)) per parameter dimension (Theorem 3.1)  \n  *No dependence on I due to analytic formulation*\n\n- **Autodiff baseline:**  \n  O(IT(n² + m²)) where I = unrolled iterations\n\nThis explains Figure 2's constant backward cost vs. autodiff's linear scaling with I. We will add a FLOP count table in revision.\n\n---\n\n**2. Scaling Barriers (Q2):**  \nThe primary challenge lies in obtaining second-order matrices in Section 4.6:\n- Our codebase dedicates ~500 lines specifically to obtain these matrices element-by-element\n- Current engineering overhead includes:\n  • Hessian calculations for dynamics/cost  \n  • KKT matrix constructions  \n- While substantial, we believe future code optimizations can reduce this overhead (\nsuch as pre-calculating and storing analytical solutions for key matrix operations, and developing automated routines to substitute corresponding values)\n\nWe have also implemented our method on a challenging rocket control task with 13-dimensional state space, which currently represents the most advanced benchmark in analytical differentiable control literature ( from PDP and IDOC paper). Demo and code are available on our project page:  \n[Project Website](https://sites.google.com/view/dilqr/)\n\n---\n\n**3. Typos/Clarifications:**  \nWe will fix:\n- Appendix A title  \n- Line 228/229 notation (θX⋆ → ∂X⋆/∂θ)  \n- Line 264/265 clarification of Eq (14) dependencies  \n\nAll corrections will be highlighted in the camera-ready version.\n\n---\n\n**Acknowledgements**  \nWe thank the reviewer for catching these important technical nuances, which have improved our paper's precision.", "claims_and_evidence": "The main claims of the paper of this paper are that the derivatives at a fixed point of the iLQR process can be analytically computed through implicit differentiation as a function of various objects that can be themselves computed efficiently without unrolling all the iLQR iterations. This is supported by the explicit mathematical derivations and numerical experiments demonstrating the essentially constant backwards computation time as a function of iLQR iterations (Figure 2), where naive autodifferentiation incurs a visibly linear trend. The exactness of the computation is demonstrated by its improved performance on various benchmarks, including DiffMPC, which is derived on a similar principle except.", "methods_and_evaluation_criteria": "The point of this paper is to provide exact derivatives of the iLQR problem without a computation cost that grows with optimization iterations. The method is evaluated on well-known control benchmarks and the main criteria displayed are computational effort and incurred loss, matching the proposed benefits.", "theoretical_claims": "The main theoretical content of this paper is in the derivation of the analytical derivatives. These are largely applications of matrix calculus, and seem to be correct by my verification. It should be noted that a lot of subtlety can be hidden in these formulas, and that experimental evaluation is likely a better diagnosis of correctness, which this paper provides.", "experimental_designs_or_analyses": "Additional experiment information is contained in Appendix A.2 and A.3, and the relevant code is provided in an anonymized github. The benchmark methods are tuned appropriately.", "relation_to_broader_scientific_literature": "As aforementioned, I think analytic expressions that avoid unrolling the iLQR trajectory optimization are an important contribution to the controls community. This concept is not necessarily novel, but this paper introduces some interesting improvements and derivations that are likely interesting to the community.", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "As aforementioned, this paper presents some interesting improvements to iLQR that demonstrate increased performance relative to naive autodifferentiation and DiffMPC. The analytic formulas in this paper are likely interesting to the community, especially if they are drop-in improvements to existing iLQR applications.\n\nI have some questions to clarify how scalable these solutions in a below section.", "other_comments_or_suggestions": "Minor comments:\n\n- The title of Appendix A seems to be unchanged from the ICML default.\n\n- Line 228/229 first column: $\\theta_\\theta$ should be $\\nabla_\\theta$\n\n- Line 264/265 second column: should clarify exactly what it means to refer $\\frac{\\partial D_t}{\\partial \\theta}$ as $\\nabla_\\theta D_t$, since in eq (14) they both show up.", "questions_for_authors": "It might be helpful to give computation (e.g. flop) estimates for the proposed method (e.g. diLQR.dx and diLQR.cost). For example, iLQR requires second-order derivatives for the quadratic cost estimate. What is an estimate of the computational complexity of deriving the requisite derivatives given a generic $d_\\theta$-dimensional parameter space? How does this compare to standard autodiff? These estimates will be helpful to corroborate the numerical results in Figure 2.\n\nAdditionally, what do the authors anticipate is the main barrier to scaling this method to larger-scale experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8W1IaovUYz", "reviewer": "Reviewer_gH1Q", "summary": "This paper introduces a differentiable iLQR controller, DiLQR, to enable scaling iLQR to longer time horizons and iteration counts. DiLQR leverages implicit differentiation at the underlying fixed-point to recover analytic gradient updates, thereby reducing computation cost in the backward pass, bypassing the need for explicit unrolling of the optimization. The approach is tested on simple pendulum and cart-pole swing tasks and demonstrates improved performance over the considered baselines as measured by computation cost and prediction accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable feedback. Below we provide responses to each question.\n\n---\n\n**Q1: Application to Higher Dimensional RL Tasks**\n\nThe field of differentiable control remains relatively young. Current real-world applications primarily focus on autonomous vehicles and drones, as evidenced by recent works:\n\n[1] **Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning** (Science Robotics 2023)  \n[2] **Actor-Critic Model Predictive Control** (ICRA 2024)  \n[3] **Guiding RL with Incomplete Model Information** (IROS 2024)  \n\nIn simulation environments, our 6-DoF rocket control task (along with quadrotor control) represents the most complex benchmark among recent works in this field.\n\n**Key Challenges**:  \nThe primary difficulty lies not in computation, but in handling locomotion tasks where contact forces violate Lipschitz continuity and complicate gradient computation. Potential solutions include:  \n- Soft contact models  \n- Neural contact approximations  \n\n---\n\n**Q2: Extension Beyond Toy Examples**\n\nOur rocket control task with 13-dimensional state space currently represents the most challenging benchmark in analytical differentiable control, based on peer-reviewed literature:\n\n[1] Revisiting Implicit Differentiation for Learning Problems in Optimal Control (NeurIPS 2023)  \n[2] Pontryagin Differentiable Programming (NeurIPS 2020)  \n\nInitial results are available on our [project website](https://sites.google.com/view/dilqr/), with additional experiments underway. Rocket model and control file can also be found in the codebase.\n\n---\n\n**Q3: DiffMPC Comparison in Figure 3**\n\nWhile DiffMPC remains an excellent benchmark in the field, our method demonstrates approximately 5-7% improvement over it. This difference appears modest compared to the 1e4 magnitude improvement over SysID and NN baselines, making visual comparison challenging in the same plot. \n\nHowever, our approach provides superior parameter estimation accuracy - a crucial factor for industrial applications where physical interpretability matters, even when iLQR's robustness can compensate for parameter inaccuracies.\n\n---\n\n**Q4: Determining Rolling Direction**\n\nAs detailed in Page 8, Line 414:  \n\"We stack four compressed images as input channels to capture velocity information and determine the pendulum's swing direction.\"\n\nThis temporal stacking approach enables the system to infer motion dynamics from visual input.\n\n**Q5: Typographical Errors**\n\nWe sincerely appreciate the reviewer's careful reading and for bringing these typographical issues to our attention. We have carefully reviewed the manuscript and corrected all identified formatting and typographical errors in the final version. These corrections include:\n\n1. Fixed notation inconsistencies in equations (e.g., θX⋆ → ∂X⋆/∂θ)\n2. Corrected the Appendix A title\n3. Addressed all minor formatting issues throughout the text", "claims_and_evidence": "The claims are well-motivated and are supported by experimental result, while experiments currently focus on toy examples. It would be great to see experiments on extending evaluation to higher degrees of freedom tasks.", "methods_and_evaluation_criteria": "The evaluations highlight reduced compute time and increased accuracy over the considered baselines on the tasks considered.", "theoretical_claims": "The theoretical claims appear sound and are well-motivated.", "experimental_designs_or_analyses": "The experiments are well-motivated and borrow designs from prior work, while the paper would be significantly strengthened by showing successful application beyond low-dimensional toy domains. What challenges would the approach face when scaling up to more realistic tasks, particularly in light of mentioned extensions to reinforcement learning learning and real-world applications? Where will it become computationally infeasible?", "relation_to_broader_scientific_literature": "The comparison to the literature is mostly well-motivated. Discussion and experimental comparison w/ DiffMPC could be expanded, i.e. why not include them in Figure 3 and some of the compute time comparisons?", "essential_references_not_discussed": "The paper adequately cites many key related works.", "other_strengths_and_weaknesses": "- Figure 7 could be more informative by focusing on a single example, providing 4 context images, and comparing predictions to the ground-truth rollout (e.g. based on an optimal trajectory). In the single image case, how does the model know whether to swing left or right?\n- Several typos below should be corrected", "other_comments_or_suggestions": "- The method abbreviation diLQR should be more clearly introduced (first mention in experiments, “We evaluated two variations of our method: diLQR.dx: Assumes that the cost of the controller is known (…)”)\n- Notation should be unified DiLQR vs. diLQR vs. dilqr\n- Lines 228-229: typo in Jacobians\n- Line 306: “we compare our approach with Neural Network”\n- Line 605: Appendix A title “You can have an appendix here”", "questions_for_authors": "- The discussion mentions application to more advanced RL settings while being mindful regarding availability of first/second-order derivatives of the dynamics. Given that many recent RL results move towards dual-arm manipulation or 50+ degrees of freedom humanoid control, where do you see the limit of feasibility? Additional discussion would benefit the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "klcqmniFxd", "reviewer": "Reviewer_SxHA", "summary": "This paper proposes DiLQR, which derives the analytic gradient of a given scalar loss function with respect to the parameters in the iLQR system (e.g., parameters of the dynamics or cost functions) through the use of the implicit function theorem. Parallelization, and the sparsity of the problem are used to improve the computation speed for computing this analytic gradient. Experiments are performed on a state-based cartpole problem, showing the proposed method outperforms a baseline that just uses a ordinary nn as the policy. It is also shown to achieve a lower parameter estimation loss compared to a closely related prior work Amos et al. The method is also briefly demonstrated to work with images for a inverted pendulum task.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful comments and constructive feedback. Below we provide point-by-point responses to the raised questions.\n\n---\n\n**Q1: Comparison with DiffTORI**\n\nWe thank the reviewer for raising this point. **DiffTORI** ([Wan et al., NeurIPS 2024]) is a pioneering work that bridges differentiable control and RL, especially in high-dimensional settings. We fully agree its **generality** (e.g., model-agnostic design, multi-modal rewards) offers broader applicability for perception-driven tasks.\n\n**Our work** focuses on **structured control domains** where iLQR is the canonical solver. By deriving *exact gradients* for iLQR, we achieve:\n\n1. **Data efficiency**: Sub-2000 environment steps suffice to achieve the results in our paper, leveraging prior knowledge of dynamics.\n2. **O(1) backward complexity** (vs. autodiff's linear cost, **Fig. 2**), enabled by analytic gradient computation.\n\nThis aligns with the **PDP/DiffMPC lineage** ([Jin et al., NeurIPS 2020; Amos et al., NeurIPS 2018]), emphasizing precision for structured control. Crucially, our method and DiffTORI **address non-overlapping challenges**:\n\n- **DiffTORI**: General-purpose RL-control synergy with perception.\n- **Ours**: Analytic acceleration for iLQR-based pipelines with structured dynamics.\n\nWe will add more discussion and clarify this complementary in the final version.\n\n---\n\n**Q2: Differences with Theseus**\n\nOur method provides exact gradients for iLQR, which **Theseus cannot compute automatically**. While Theseus (similarly JAXopt) can differentiate through fixed-point equations ($x^*=f(x^*)$), iLQR's unique challenge is that its **dynamics (D) and cost (C) depend on x*** itself, creating a recursive relationship $x^*=f_{x^*}(x^*)$. \n\nThis cannot be resolved through auto-implicit-differentiation alone - it requires **reformulating LQR as a pure fixed-point problem**, which our paper implements theoretically (not just at the coding level). This is not a small step.\n\n---\n\n**Q3: Relation to DiffMPC**\n\nWe clarify that DiffMPC approximates gradients by treating D/C as independent of x* (i.e. x*≈f(x*)), while our method properly accounts for their **interdependence through implicit differentiation**. \n\nOur **key innovation** lies in embedding Amos's results into a correct fixed-point formulation that yields exact analytical gradients, representing a high-level advancement beyond prior work.\n\n---\n\n**Q4: Comparison with (Safe)PDP and IDOC**\n\nAs noted in our related work (L147), these prior methods compute gradients under the **assumption of optimal forward solutions**, which creates mathematical inconsistencies when the forward pass is suboptimal. In contrast, our approach maintains **alignment between forward and backward passes** throughout iLQR optimization.\n\nThis key difference is evidenced in our cartpole experiments:\n- **IDOC**: 4×10⁻² error  \n- **PDP**: 2×10⁻² error  \n- **Ours**: 1×10⁻⁴ error  \n\nAdditional results will be available on our [project page](https://sites.google.com/view/dilqr/).\n\nWe acknowledge that (Safe)PDP and IDOC offer **greater generality** in handling diverse solvers and constraints - an important contribution to the field.\n\n---\n\n**Q4: Losses for SysID**\n\nThe **SysID** approach follows a two-stage process:\n1. Estimates model parameters from data\n2. Uses the estimated model in MPC for trajectory prediction\n\nCompared to end-to-end learning, this represents a **modular strategy** that decouples system identification from control.\n\n---\n**Q5: NN Baseline Comparison**\n\nThis is exactly the key point we want to demonstrate! Our method essentially transforms the neural network's black-box approach into a white-box solution, achieving significant improvements in sample efficiency through structured prior knowledge.\n\n---\n**Q6: Visual Control Task**\n\nWe employ a basic **autoencoder (AE) framework**, which may exhibit sensitivity limitations. This work is primarily **conceptual**, with the visual control section demonstrating that:\n- iLQR can function as a differentiable controller  \n- It successfully integrates with vision modules\n\nWe believe that combining our method with DiffTORI could lead to significant improvements in handling multi-modal inputs.", "claims_and_evidence": "I think the claims made in the paper is in generally true, given the limited set of baselines it compared to. However, I feel the paper is lacking comparison to several more recent methods and frameworks that tackle almost the same problem:\n- Jin et al., Revisiting Implicit Differentiation for Learning Problems in Optimal Control, NeurIPS 2023 \n- Jin et al., Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework, NeurIPS 2020 \n\nThe above two papers solve almost the exact same class of problems, i.e., differentiable trajectory optimization (See Eq 2 in the first paper and Eq. 1 in the second paper), as defined in this paper, although they are not necessarily using iLQR as the solver. The lack of comparison to these more recent methods kind of weakens the claim on the effectiveness of the proposed method.\n- Pineda et al., Theseus: A Library for Differentiable Nonlinear Optimization\n\nThis paper provides a very general framework to differentiate through non-linear optimization problems (which includes trajectory optimization, though this paper handles the constraints in a soft way by adding the constraints as losses). Many computational and engineering optimization efforts are implemented in this paper to improve the computation speed, stability and quality of the obtained gradient (e.g., Theseus supports many different ways of differentiating through the non-linear optimization, such as implicit differentiation as used in DiLQR, and other highly optimized ways like truncated backward). I think a discussion to this paper, and how the proposed method differs from the methods already implemented in Theseus is needed.\n- Wan et al., DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning, NeurIPS 2024 \n\nThis paper shows that differentiable trajectory optimization can be scaled to high-dimensional image and point cloud observations on many common robotics benchmarks. Since it is claimed that DiLQR can be applied to image inputs as well, it would be good to discuss and maybe compare to this baseline to see how effective DiLQR is in such settings.", "methods_and_evaluation_criteria": "The method and evaluation criteria makes sense.", "theoretical_claims": "I went through the theoretical claims and did not find any obvious issues.", "experimental_designs_or_analyses": "Overall the experiment design seems to follow that in Amos et al. and is valid. \n\nI have the following questions regarding the experiments:\n- How is the imitation learning loss for sysid baseline computed? Since it does not learn the actions?\n- What would be the method’s performance if the dynamics and cost function does not assume pre-defined structures?  Since the NN baseline does not assume access to such prior information. \n- As mentioned above, I think it would make the paper much stronger and convincing if the DiLQR can be compared to more recent and advanced baselines, including:\n1. Jin et al., Revisiting Implicit Differentiation for Learning Problems in Optimal Control, NeurIPS 2023\n2. Jin et al., Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework, NeurIPS 2020 \n3. Wan et al, DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning, NeurIPS 2024 \n- Section 5.4 – what is the actual performance of the method on the inverted pendulum problem? Does it always perfectly solve the problem?", "relation_to_broader_scientific_literature": "This paper extends the work by Amos et al. to differentiate through iLQR, alone with some computational improvements. I feel this contribution alone, and the lack of comparison to more recent differentiable trajectory optimization methods, is not enough contribution to warrant acceptance at ICML.", "essential_references_not_discussed": "Already mentioned above in the lack of comparison and discussion to some very related works.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "- Figure 1 is not referenced in the main text.\n- Line 228:  “Likewise, θθX⋆ and θθU⋆ are the Jacobians” typos?", "questions_for_authors": "Already covered above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fUKl4URm4k", "reviewer": "Reviewer_f19d", "summary": "This paper introduces a framework that facilitates differentiation through iLQR, which provides the gradient of an iLQR controller through implicit differentiation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers' insightful comments and constructive feedback, which have helped improve our paper. Below we provide detailed responses to each point.\n\n---\n\n**Q1: No Theorem Proving Speedup**\n\nWe thank the reviewer for this important question. Our work is situated in the field of differentiable control, which aims to expose model structure to learning and improve sample and computational efficiency. In this area, it is commonly understood—though not typically accompanied by formal theorems—that incorporating control priors and structured solvers leads to faster learning. Foundational works such as:\n\n- *Revisiting Implicit Differentiation for Learning Problems in Optimal Control* (NeurIPS 2023)  \n- *Pontryagin Differentiable Programming* (NeurIPS 2020)  \n- *Infinite-Horizon Differentiable MPC* (ICLR 2020)  \n- *Differentiable MPC for End-to-End Planning and Control* (NeurIPS 2018)  \n\nshare a similar focus on demonstrating empirical improvements, without formal speedup theorems.\n\nSuch approaches have seen success in robotics, e.g.,  \n- *Reaching the Limit in Autonomous Racing: Optimal Control vs Reinforcement Learning* (Science Robotics 2023)  \n- *Actor-Critic Model Predictive Control* (ICRA 2024)  \n- *Guiding RL with Incomplete Model Information* (IROS 2024)  \ndemonstrating practical benefits in learning efficiency.\n\nIn addition, our method improves computational efficiency: by leveraging analytic gradients, we avoid differentiating through rollouts, reducing complexity to $ \\mathcal{O}(1)$ per iteration—a property determined by algorithm design.\n\n---\n\n**Q2: Experiments on More Complicated Tasks**\n\nAs noted in our discussion section, many prior works (**Amos et al., 2018; Watter et al., 2015; Xu et al., 2024a; Jin et al., 2020**) also use **simplified environments** to illustrate core ideas. Differentiable control methods are built upon **model-based control** and therefore require access to **structural priors**, which makes them less \"plug-and-play\" than general-purpose RL. However, the benefit is that such structure greatly reduces the **sample complexity**.\n\nTo go beyond toy tasks, we further evaluate our method on a challenging **rocket control task with a 13-dimensional state space**—the most complex benchmark among recent analytical differentiable control works. Videos are available on our project page:  \n[https://sites.google.com/view/dilqr/](https://sites.google.com/view/dilqr/) . Additional results are currently being prepared and will be shared shortly.\nThe rocket model and controller code are also open-sourced in our codebase.\n\n---\n\n**Q3: Sudden Drop and High Standard Deviation for Model Loss**\n\nThe sudden drop corresponds to learning progress in reducing model loss.  \nThe observed standard deviation (~0.17) is reasonable, especially considering the mean gap between methods is 0.47, indicating that the improvement is both significant and consistent.\n\n---\n\n**Q4: Typographical Errors**\n\nWe sincerely thank the reviewer for catching these formatting issues. We will carefully review the entire manuscript and correct all typographical/formatting errors in the final version.", "claims_and_evidence": "The authors theoretically prove the effectiveness of their method in Section 5 and provide experimental results in the same section to support their claims.", "methods_and_evaluation_criteria": "They perform experiments on CartPole and Inverted Pendulum and evaluate the results based on backward time, loss, and computation time, which make sense for the problem.", "theoretical_claims": "I've checked proposition 4.1.", "experimental_designs_or_analyses": "For the comparison of model loss between diffMPC and DiLQR, why is there a sudden drop with high standard deviation at the end for DiLQR?", "relation_to_broader_scientific_literature": "This paper provides an analytical solution for the gradient of an iLQR controller through implicit differentiation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n- The experimental results on speedup and performance improvement are compelling.\n- An ablation study is provided.\n\n**Weaknesses**\n- The logic of the proof needs to be polished. For example, there is no theorem proving that the proposed algorithm guarantees speedup.\n- Experiments on more complicated tasks, such as Atari games, could be considered in future experiments.", "other_comments_or_suggestions": "- The font size inside the figures should generally match the font size of the main text. Currently, some figures have text that is too small and blurry.\n- Figure 4(a) is missing.\n- In-text citations are all enclosed in parentheses. However, in some cases, this is not appropriate. For example, at the beginning of Section 4.4, the citation should be without parentheses.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission received mixed reviews.  Its contributions were viewed as interesting and well-executed, but on the other hand there were also some critiques, most notably a lack of comparison against and positioning within related methods.  The authors did a good job of addressing the latter concern during the discussion, therefore---assuming all of this content (as well as other information provided by the authors during the discussion) will appear in the camera-ready version---I recommend acceptance.
Decision: Accept (poster)