Data-Efficient Model Learning for Control with Jacobian-Regularized Dynamic-Mode Decomposition
Authors: Brian Edward Jackson
Jeong Hun Lee
Kevin Tracy
Zachary Manchester
Venue: CoRL 2022 Poster (2022)
Keywords: Koopman
learning
model-predictive control
optimal control

Abstract: We present a data-efficient algorithm for learning models for model-predictive control (MPC). Our approach, Jacobian-Regularized Dynamic-Mode Decomposition (JDMD), offers improved sample efficiency over traditional Koopman approaches based on Dynamic-Mode Decomposition (DMD) by leveraging Jacobian information from an approximate prior model of the system, and improved tracking performance over traditional model-based MPC. We demonstrate JDMD’s ability to quickly learn bilinear Koopman dynamics representations across several realistic examples in simulation, including a perching maneuver for a fixed-wing aircraft with an empirically derived high-fidelity physics model. In all cases, we show that the models learned by JDMD provide superior tracking and generalization performance within a model-predictive control framework, even in the presence of significant model mismatch, when compared to approximate prior models and models learned by standard Extended DMD (EDMD).

Data-Efficient Model Learning for Control with Jacobian-Regularized Dynamic-Mode Decomposition
Brian E. Jackson Roboties Institute Carnegie Mellon University brianjacks on cmu. eduu
Jeong Hun Lee Roboties Institute Carnegie Mellon University
jeonghunlee& cmu. edu
Kevin Tracy
Zachary Manchester
Robotics Institute
Robotics Institute
Carnegie Mellon University kt ktraccyccmu edu
Carnegie Mellon University za cm3 cmu, edu
Abstract: We present a data-efficient algorithm for learning models for model- predictive control (MPC). Our approach, Jacobian- Regularized Dynamic-Mode Decomposition (JDMD), offers improved sample efficiency over traditional Koop- man approaches based on Dynamic-Mode Mode Decomposition (DMD) by leverag- ing Jacobian information from an approximate prior model of the system, and improved tracking performance over traditional model-based MPC. We demon- strate JDMD's s ability to quickly learn bilinear Koopman dynamics representations across several realistic examples simulation, including a perching maneuver for fixed-wing aircraft with empirically derived high-fidelity physics model. In all cases, show that the models learned by JDMD provide superior tracking and generalization performance within model-predictive control framework, even in the presence of significant model mismatch, when compared to approximate prior models and models learned by standard Extended DMD (EDMD).
Introduction
In recent years, both model-based optimal-control [1, 2, 3,44 4] and data-driven reinforeement-leaming methods [5, 6, 7] have demonstrated impressive success on complex, nonlinear robotic systems. However, both approaches suffer from inherent drawbacks: Data-driven methods often require ex- tremely large amounts of data and fail generalize outside of the domain or task on which they were trained. On the other hand, model-based methods require accurate model of the system to achieve good performance. In many cases, high-fidelity models can be too difficult to construct from first principles or too computationally expensive to be of practical use. However, low-order approximate models that can be evaluated cheaply at the expense of controller performance are often available. With this in mind, we seek a middle ground between model- based and data-driven approaches in this work.
We propose a method for learning bilinear Koopman models of nonlinear dynamical systems for use in model-predictio control that leverages derivative information from approximate prior dynam- ies model of the system in the training process. Given the increased availability of differentiable simulators [8, 9], this approximate derivative information is readily available for many systems of interest. Our new algorithm builds on Extended Dynamic Mode Decomposition (EDMD), which learns Koopman models from trajectory data [10, 11, 12, 13, 14], by adding a derivative regular- ization term based on derivatives computed from prior model. We show that this new algorithm, Jacobian- Jacobian-regularized Dynamic Mode Decomposition (JDMD), learn models with dramatically fewer samples than EDMD, even when the prior model differs significantly from the true dynamics
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

of the system. We also demonstrate the effectiveness of these learned models in a model poodel-predictive control (MPC) framework. The result is a fast, robust, and sample-efficient pipeline for quickly training a model that can outperform MPC controllers using both approximate analytical models as well as models learned using traditional Koopman approaches.
Our work is most closely related to the recent works of Folkestad et. al. [13, 15, 16], which learn bilinear models and apply nonlinear model-predictive control directly on the learned bilinear dy- namics. Other recent works have combined linear Koopman models with model-predice control [12] and Lypuuovv control techniques with bilinear Koopman models [17]. Our contributions are:
A novel extension to Dynamic Mode Decomposition, called JDMD, that incorporates gra- dient information from an approximate analytic model
A recursive, batch QR algorithm for solving the least-squares problems that arise when learning bilinear dynamical systems using DMD-base algorithms, including JDMD and EDMD
The remainder of the paper is organized as follows: In Section 2 we provide some background on the application of Koopman operator theory to controlled dynamical systems and review some related works. Section then describes the proposed JDMD algorithm. In Section 4 we outline a memory-efficien technique for solving the large, sparse linear least-squares problems that arise when applying JDMD and other DMD-based algorithms. Section 5 then provides simulation results and analysis of the proposed algorithm applied control tasks on a cartpole, a quadrotor, and a small foam airplane with experimentally determined aerodynamics model, all subject significant model mismatch. We also discuss the suitability of the conventional open-loop prediction error as a metric for evaluating dynamics model used in closed-loop control frameworks. In Section 6 we discuss the limitations of our approach, followed by some concluding remarks in Section 7.
2 Background and Related Work
2.1 Koopman Operator Theory
The theoretical underpinnings of the Koopman operator and its application dynamical systems has been extensively studied [11, 18, 19, 20, 21, 22]. Rather than describe the theory in detail, highlight the key concepts employed by the current work and refer the reader to the existing literature on Koopman theory for further details.
We start by assuming controlled, nonlinear, discrete-time dynamical system,
x+ = =f(xru).
where I E X' C N. is the state vector, u E No is the control vector, and x is the state at the next time step. Assuming the dynamics are control affine, the nonlinear finite-dimensional system (1) can be represented exactly by an -imens--iimnnionall bilinear system through the Koopman canonical transform [22]. This bilinear Koopman model takes the form,
Ay + Bu +
Σ x 1C'y=((vu
where y (xx:) is nonlinear mapping from the fnite-dimensional state space X' to the infinite- dimensional Hilbert space of observables y. In practice, approximate (2) by restricting Y to be vector space, in which case é becomes dimee-dimeallonal nonlinear function of the state variables that can be either chosen heuristically based on domain expertise or by learning [23, 24, 25).
Intuitively, o "lifts" our state into a higher dimensional space y where the dynamics are approx- imately bilinear, effectively trading dimensionality for bilinearity. Similarly, we perform an "unlifting" operation by projecting lifted state y back into the original state space X. In this work,

---

since we embed the original state within the nonlinear mapping [11, 15, 26, 27, 28], ø is constructed in such a way that this unlifting is linear:
I = Gy.
(3)
We note that our proposed method does not require this assumption: any mapping could be used. The problem of finding an optimal mapping itself major area of research, and many recent studies have focused on jointly learning both the model and the mapping [23, 24, 25, 29, 30). While clearly advantageous, learning an optimal mapping is not the focus of this paper. Instead, we focus on incorporating prior information from an approximate model in way that is applicable to any lifting function, and we rely on simple mappings that are chosen heuristically in all of our examples.
2.2 Extended Dynamic Mode Decomposition
A lifted bilinear system of the form (2) be learned from P samples of the system dynamics (aJ -z 2z) using Extended Dynamic Mode Decomposition (EDMD) [15, 21]. We first define the following data matrices:
y1
1/2
y/
M1
ZI.P
u V/
U2y/2
upyp
Yitp = lyst
V/2
yp
(4)
Lu1 /1
U2 V/2
up
where uj is the i-th element of the control vector at time k. We then concatenate all of the model coefficient matrices from (2) as follows:
E = A B C
C"" E MNN.
(5)
The model learning problem can then be written as the following linear least-squares problem:
minimize EZ
Yitrll2
EDMD is closely related to classical feature-based machine learning approaches like the "kernel trick" used in support vector machines [31], but extends these ideas to bilinear models of controlled dynamical systems.
3
Jacobian-Regularizeed Dynamic Mode Decomposition
We now present JDMD straightforward adaptation of the original EDMD algorithm described in Section 2.2. Given P samples of the dynamics (x.t Tiz uz), and an approximate discrete-time dynamics model,
x+ = f(c, u),
evaluate the Jacobians of approximate model f each of the sample points: Āy
af. After choosing nonlinear mapping N= - NR,, our goal is to find a bilinear of B, du du
dynamics model (2) that matches the Jacobians of our approximate model, while also matching our dynamics samples. We accomplish this by penalizing differences between the Jacobians of our Icarned bilinear model with respect to the original states x and controls u, and the Jacobians we expect from our analytical model. These projected Jacobians are calculated by differentiating through the projected dynamics:
x = G (Δ(()) + Bu Σ w'Clslz)) = f(x, u).
(8)
Differentiating (8) with respect to and u gives us
0f A, Or (z,, Uz) G A uf ((zz)= = -GA((, -GEA,,
(9a)
of uz)) uy) = G B Cha
B, Du
(z,,
C=mzy)
(9b)
3

---

where ((x) = 0 dr is the Jacobian of the nonlinear map o. and
IN. u² IN, x²2 IN
IN=
(o(z) 0 0
A(z, u)
((x) ERN.NN, u(x,u)=
(0)(z) 0
R N.NN..
(10)
t IN,-
[00 6(x)
We then solve the following linear least-squares problem:
GEB, - BB(()-- (11)
minimize (1 (1-a)l - 1Tpll22 +a E
N.) rows and N₂- N₂ columns. Given that the number of rows this problem grows quadratically with the state dimension, solving it can be computationally challenging. The next section proposes an algorithm to address this challenge without resorting to distroyutedmmemrry system. This solution method also has the benefit of allowing incremental updates to the bilinear system, enabling online model learning.
Problem (11) has (N, + N3 N₂
4 Efficient Recursive Least Squares
In its canonical formulation, linear least squares problem can be represented the following unconstrained optimization problem:
min Fx d|2-
(12)
We assume F large, sparse matrix and that solving it directly using a QR Cholesky decom- position requires too much memory for single computer. While solving (12) using iterative method such as LSMR [32] or LSQR [33] is possible, we find that these methods do not work well in practice for solving (11) due to ill-conditioning. Standard recursive methods for solving these problems are able to process the rows of the matrices sequentially to build a QR decomposition of the full matrix, but also tend to suffer from ill-conditioning [34, 35, 36].
To overcome these issues, propose alternative recursive method based. We solve (12) by dividing rows of F into batches:
FT FFF=ATA+E
FF
(13)
The main idea is maintain and update an upper-triae trianguaa Cholesky factor U, of the first terms of the sum (13), Given U, we can calculate U441 using the QR decomposition, as shown [37]:
([ U
=411 = VUTU, F QRR
((
(14)
where QRR returns the upper triangular matrix R from the QR decomposition. For efficient implementation, this function should be "economy" or "Q-less" QR decomposition since the Q matrix is never needed.
We also handle regularization of the normal equations, equivalent to adding quadratic or Tikhonov regularization the original least squares problem, during the base case of our recursion,
U QRR ([A))
(15)
where is a scalar regularization weight. To ensure fair comparisons, the results presented in the next section for both EDMD and JDMD correspond to the best-perg À values found by sweeping over wide parameter range.

---

(a) Expert demonstration of high angle-of-attack perching maneuver that minimizes velocity the goal position with complex, post-stall aerodynamic forces.
(b) E-Flite AS3Xtra airplane model used hardware data collection.
(c) Wind-tunnel experimental setup for collecting aerody- namic data [38].
Figure Complex dynamics a perching fixed-wing airplane. High-angle-of-attack perching maneuvers (top) require the modeling of complex post-stall aerodynamic effects. The simulated aerodynamic forces were modeled using flight data collected from real-world hardware experiments (bottom) [38].
5
Experimental Results
This section presents the results of several simulation experiments evaluate the performance of JDMD. We specify two models for each simulated system: nominal model, which is simplified and contains both parametric and non-parametric model error, and true model, which is used exclusively for simulating the system and evaluating algorithm performance.
All models were trained by simulating "true" system with a nominal controller to collect data in of fixed-length trajectories were collected,
the region of the state space relevant the task. each sample rate of 20-25 Hz. The bilinear EDMD model introduced by Folkestad and Burdick [15]. When applying MPC to the learned Koopman models, the projected Jacobians (9) were used, since this projected system is much more likely to be controllable than the lifted one and reduces the computational complexity of the MPC controller. This results in a nonlinear model in the original state space, which is linearized about the reference trajectory to create a linear MPC controller. All continuous-time dynamics were discretized with an explicit fourth-order Runge-Kutta method. Code for all experiments available https: //githu..
trained using the same approach
5.1 Systems and Tasks
Cartpole:
We perform swing-up task on cartpole system. The true model includes Coulomb
friction between the cart and the floor, viscous damping at both joints, and a deadband in the control input that were not included in the nominal model. Additionally, the mass of the cart and pole model were altered by 20% and 25% with respect the nominal model, respec- tively. The following nonlinear mapping was used when learning the bilinear models: ((x) [1, Ir, sin(x), cos(z), sin(2r) sin(4x). T2()), Tz(z), [z()] e 333, where T2(x) is Cheby- shev polynomial of the first kind of order i. All reference trajectories for the swing up task were generated using ALTRO [37, 39].
Quadrotor: We track point-to-point linear reference trajectories from various initial conditions on full 3D quadrotor model. The true model includes aerodynamic drag terms not included in the nominal model, well parametric error of roughly 5% the system parameters (e.g. mass, rotor arm length, etc.). The model was trained using a nonlinear mapping of Φxx) =
5

---

[1, (x), sin(p), cos(p), R'v. u"RR" p p w x w| E R where is the quadro- tor's position, e and w are the translational and angular velocities respectively, and R is the rotation matrix representation of the quadrotor's attitude.
Airplane: We perform post-stall perching maneuver on high-fidelity model of fixed-wing
airplane. The perching trajectory is produced using trajectory optimization (see Figure la). Perching involves flight at high angles of attack, where the aerodynamic lift and drag forces are extremely complex and difficult to model from first principles. We leverage previous works that fit post-stall aerodynamics models using empirical data from wind-tunnel experiments [38, 40J. The true model includes these empirical nonlinear flight dynamics [38], while the nominal model uses simple flat- plate wing model with linear lift and quadratic drag coefficient approximations. The bilinear models nonlinear mapping e R68 which includes the aircraft attitude (expressed Modified Rodriguez Parameters [41]), powers of the angle of attack and side slip angle, the body frame velocity, various cross products with the angular velocity, and 3rd and 4th order Chebyshev polynomials of the states.
vector of
5.2 Sensitivity to Model Mismatch
While a significant amount of model mismatch is introduced in all examples, natural argument against model-based methods that they are only good as the model's ability to capture the salient dynamics of the system. Therefore, investigated the effect of in- creasing model mismatch by incrementally increasing the Coulomb friction coefficient between the cart and the floor for the cartpole stabilization task (recall the nominal model assumed zero friction). The results are shown in Table As expected, the number of train- ing trajectories required to find a good stabilizing controller increases with μr. We achieved the results above by setting a 0.01, corre- sponding to decreased confidence in our model, thereby placing greater weight on the experimental data. The standard EDMD ap- proach always required more samples, and was unable to find good enough model above friction values of 0.4. While this could likely be remedied by adjusting the nonlinear mapping é the proposed ap- proach works well with the given basis. Note that the nominal MPC controller failed stabilize the system above friction values of 0.1, so again, demonstrate that improve MPC perfor- mance substantially with just few training samples by combining analytical derivative information and data sampled from the true dynamics.
Nominal
EDMD
JDMD
1
0.0
3
2
0.1
/
19
2
0.2
X
6
2
0.3
X
15
2
0.4
X
X
3
0.5
X
X
7
0.6
X
X
12
Table 1: Training trajecto- ries required stabilize the cartpole with given friction coefficient.
Nominal
0.2
Tracking Error
100
Tracking Error
EDMD
JDMD
0.1
10
10
20
30
20
40
60
100
Number of Training Trajectories
Number of Training Trajectories
(a) Cartpole
(b) Airplanc
Figure 2: MPC tracking error vs training trajectories both the cartpole (left) and airplane (right). Tracking error is defined as the average 1.2 error over all the test trajectories between the reference and simulated trajectories. The median error shown as a thick line, while the shaded regions represent the 5% to 95% bounds on the 10 test trajectories.

---

(a) Point-to-point trajectories and initial conditions testing MPC on a 6-DOF quadrotor.
(b) Closed-loop trajectories of nominal MPC (black), EDMD (orange), and JDMD (cyan) for tracking a dy- namically infeasible, point-to-point trajectory (red).
Figure Visualizations of the tests on the full quadrotor model.
5.3 Sample Efficiency
We compare the sample efficiency of several algorithms on the cartpole swing-up task in Figure As shown, JDMD achieves the best performance overall, and does so with only two training trajectories. In comparison, traditional EDMD requires about 10 iterations to achieve consistent performance, Similar results were obtained for the airplane perching example (see Figure 2b), where EDMD requires over three times the number of samples (35 10) compared to JDMD and never achieves the same closed-loop performance.
5.4 Generalization
We demonstrate the generalizability of JDMD on quadrotor. The task is return the ori- gin, given an initial condition sampled from uniform distribution centered at the origin. Test initial conditions are sampled from distribu- tion larger than that of the training data. Given the goal of tracking a straight line back to the origin, we test 50 initial conditions, many of which are far from the goal, have large veloci- ties, nearly inverted (see Figure 3a). The results using MPC controller shown in Table 2, demonstrating the generalizability of JDMD, given that the algorithm was only trained on 30 initial conditions sampled relatively sparsely given the size of the sampling window, EDMD only successfully brings about 18% of the samples to the origin, while the majority the time resulting trajectories like those Figure 3b. JDMD improves tracking performance of nominal MPC, which is subject to a constant error due to model mismatch, as shown in Figure 3b.
Nominal
EDMD
JDMD
Success Rate
82
18%
80%
Median
0.30
0.63
0.11
5% Quantile
0.13
0.08
0.03
95% Quantile
0.38
2.62
0.23
Table 2: Performance summary of MPC tracking of 6-DOF quadrotor. Other than success rate, all values are the tracking eгтor of the successfully stabilized trajectories.
5.5 Model Prediction Error vs. Controller Performance
Much of the previous literature focuses on open-loop prediction eгror for evaluating learned- dynamics models [11, 12, 15, 19, 22, 42]. While intuitive, we argue that this is a poor metric when the end goal is closed-loop control performance. As shown in the histogram of open-loop prediction error in Figure 4a, the open-loop prediction error of JDMD (trained with 8 trajectories) significantly higher over 100 test trajectories, with 74% of tests resulting in a prediction error of approximately 1.5 compared to 25% for EDMD (trained with 24 trajectories). Despite worse open-loop prediction performance, the JDMD model outperforms the EDMD model in closed-loop
7

---

30
Nominal
Number of Trajectories
60
40
EDMD
20
JDMD
40
20
10
20
0
0
0.5
1.5
0
0.5
1.5
0.5
1.5
Prediction Error
Tracking Error
Jacobian Error
(a)
(b)
(c)
Figure 4: Histograms displaying prediction, tracking, and Jacobian enror over 100 test trajectories with randomly sampled initial conditions for the airplane perching problem. The EDMD model is trained with 24 trajectories and the JDMD model is trained with 8. While JDMD has poor open- loop prediction (a), it has better closed-loop tracking performance (h) and better matching of the dynamics Jacobians (c).
tracking (see Figure 4b). Given that MPC, like most closed-loop controllers, relies on the behav- ior of the model under small perturbations (i.e. derivative information). the difference in tracking performance may be explained by JDMD achieving a distribution with much lower Jacobian error than EDMD (see Figure 4c). This suggests that open-loop prediction error is not necessarily good metric for evaluating models that will be used control applications, and that models sufficient for closed-loop control may be learned with far less data.
Limitations
Many of the limitations of the proposed approach derive from the limitations of Koopman ap- proaches more broadly, such the sensitivity to the nonlinear mapping selected and the limitation to control-affine continuous dynamics. While the presented single rigid-body systems such as the quadrotor or airplane have similar dimensionality to many autonomous systems of interest, exten- sions to systems with many degrees of freedom may be difficult computationally, given that Jacobian matrices grow in size with the square the state dimension. As with most data-driven techniques, is difficult to claim that our method will increase performance in all cases. It is possible that hav- ing an extremely poor prior model may hurt rather than help the training process, especially if the derivative information from the approximate model has the incorrect sign.
7 Conclusions and Future Work
We have presented JDMD, simple but powerful extension to EDMD that incorporates derivative information from an approximate prior model. We have tested JDMD in combination with simple linear MPC control policy across range of systems and tasks, and have found that the resulting combination can dramatically increase sample efficiency over EDMD, often improving over a nom- jish a few sample trajectories. We also argued that the conventional open-loop
inal MPC policy with just dynamics prediction error a poor metric for evaluating models used in closed-loop control frame- works. Substantial areas for future work remain: most notably, demonstrating the proposed pipeline on hardware. Additional directions include applications on sytems with many degrees of freedom such those whose dynamics are governed by discretized PDEs, online learning or adaptive control applications, combining simulated and real data through the use of modern differentiable physics engines [9, 8), residual dynamics learning, as well as the development of specialized numerical methods for solving nonlinear optimal control problems using the learned bilinear dynamics.

---

References
[1] Farshidian, M. Neunert, A. W. Winkler, G. Rey, and Buchli. An efficient optimal planning and control framework for quadrupedal locomotion. In 2017 (IEEE) International Conference on Robotics and Automation ((CCAA), ), pages 93-100. doi: 10.
[2] S. Kuindersma, Permenter, and R. Tedrake. An efficiently solvable quadratic program for stabilizing dynamic locomotion, pages 2589-25811512 2594. ISSN 9781479936854. 2014.6907230.
[3] M. Bjelonic, R. Grandia, Harley, C. Galliard, S. Zimmermann, and M. Hutter. Whole-Body MPC and Online Gait Sequence Generation for Wheeled-Legge Legged Robots. pages 8388-8395. ISSN 9781665417143, doi: 111.11.111//RRSSSS11 11....222..666663
[4] Subosits and J. C. Gerdes. From the racetrack to the road: Real-time trajectory replanning for autonomous driving. 4(2):309-320, 16::10.1109/TIV22199920000
[5] N. Kamchanachari, M. I. Valls, S. David Hoeller, and M. Hutter. Practical Reinforce- ment Learning For MPC: Learning from sparse objectives in under an hour on a real robot. pages 1-14. doi: (-11999//EHH2---00000000 2th2-b-0004046900
URL https:// /tcpps://cc...o org/ 3929/
[6] D. Hoeller, F. Farshidian, M. Hutter, F. Farshidian, and D. Hoeller. Deep Value Model Predictive Control. 100:990-1004. 110.9999/TH12---000006 URLhttps://dobt org/10 00t.009992//ttt-----00666-555 000368961.
[7] Z. Li, X. Cheng, X. B. Peng, Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforcement
Learning for Robust Parameterized Locomotion Control of Bipedal Robots. 2021-May:281 11 2817. ISSN 9781728190778
10..110////A//0000000 2021 20119960669 .9560769.
[8] T. A. Howell, S. L. Cleac'h, "eac'h,11 J. Z. Kolter, M. Schwager, and Z. Manchester. Dojo: A differen- tiable simulator for robotics. arXiv preprint arXiv:2203.00806 2022.
[9] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEF, 2012.
[10] A. Meduri, P. Shah, J. Viereck, M. Khadiv, I. Havoutis, and L. Righetti. BiConMP: A Nonlinear Model Predictive Control Framework for Whole Body Motion Planning. 108.10,4855 2201.0760) URL haxts wwtp:/oorraabaooooo 222g//2b/220111.
2201 07601v1.
[11] D. Bruder, X. Fu, and R. Vasudevan. Advantages of Bilinear Koopman Realizations for the Modeling and Control of Systems with Unknown Dynamics. 6(3):436944376 1oi:10.11000 109/ LRA.2021.3068117
[12] M. Korda and I. Mezić. Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control. 93:149-160. doi: 101:10.1016/j.aa att.faaaaammaaa1..00000000 https: /oii wrg/10........ 1016 http://.....aaaaaaaaaaaam www.ttttttit.................... 2018. 03. 046.
URL
[13] C. Folkestad, D. Pastor, and J. W. Burdick. Episodic Koopman Learning of Nonlinear Robot Dynamies with Application to Fast Multirotor Landing. pages 9216-9222 ISSN 9781 9781728173955. doi: 1..11...1//CCAAAA. 1109/CCAA4445.2200.117777
[14] H. Suh and R. Tedrake. The Surprising Effectiveness of Linear Models for Visual Foresight in Object Pile Manipulation. 17:347-363. 200:100.48500amxx.222...0000 URL https:
//arxiv. wwt//bs/ 0900933.....

---

[15] C. Folkestad and J. W. Burdick. Koopman NMPC: Koopman
Koopman-based based Learning and Nonlinear Model Predictive Control of Control-affine Systems. In Proceedings IEEE International Conference on Robotics and Automation, volume 2021-May, pages 7350-7356. Institute of Electrical and Electronics Engineers Inc. ISBN 978-1-72819-077-1 doi: 109/CCA44806 2021.9562002.
[16] C. Folkestad, Wei, and W. Burdick. Quadrotor Trajectory Tracking with Learned Dynamics: Joint Koopman- based based Learning of System Models and Function Dictionaries. URL http://arxisa .rreiiiiorrr///oooo abs 2110 11111110044 10341.
[17] A. Narasingam, J. Sang, and I. Kwon. Data-driven feedback stabilization of nonlinear systems: Koopman-basel model predictive control. pages 1-12.
[18] S. L. Brunton, J. L. Proctor, and J. N. Kutz. Discovering governing equations from data by sparse identification nonlinear dynamical systems. Proceedings of the national academy of sciences, 11 113(15)3932-3333) 2016.
[19] S. L. Brunton,
Proctor, and J. N. Kutz. Sparse identification of nonlinear dynam- ics with control (sindye). IFAC-PapersOnLine, 49(18):700-715, 2016. ISSN 2405-8963. httptp/s//oboooo//////0011/ff/id
URL stttps:/wwer sciencedirect. com/ scisscienn eeceeenne//tttte / 6316882298.8 10th IFAC Symposium on Nonlinear Control Systems NOLCOS 2016.
[20] J. L. Proctor, S. L. Brunton, and J. Nathan Kutz. Generalizing koopman theory to allow for inputs and control. 17(1):909-930. doi: 1137/1611062266. URL htt.:t////// /ww. siam. org/ journals/////o
html.
[21] M. O. Williams, G. Kevrekidis, and C. W. Rowley. A Data-Driven Approximation of the Koopman Operator: Extending Dynamic Mode Decomposition. 25(6):1307-1346 doi: 000000077700322000---5----7--000 article/10 110.10..1177/0000300 015- 9258-5.
URL http::://///::k.ee springeeeeeeeeeeeeeeee.eee com/
[22] A. Surana. Koopman Operator Based Observer Synthesis for Control-Affine Nonlinear Sys- tems; Koopman Operator Based Observer Synthesis for Control-Affine Nonlinear Systems. ISBN 978-1-5090-1837-6 10010...09/CCDC22111.7777
[23] C. Folkestad, D. Pastor, I. Mezic, R. Mohr, M. Fonoberova, and J. Burdick. Extended Dynamic Mode Decomposition with Learned Koopman Eigenfunctions for Prediction and Control. In 2020 American Control Conference (ACC), pages 3906-39 3913. IEEE, 2020.
[24] C. Folkestad. S. X. Wei, and J. W. Burdick. Koopnet: Joint learning of koopman bilinear models and function dictionaries with application quadrotor trajectory tracking. In 2022 International Conference on Robotics and Automation (ICRA), pages 1344-1350. IEEE, 2022.
[25] Q. Li, F. Dietrich, E. M. Bollt, and I. G. Kevrekidis. Extended dynamic mode decomposi- tion with dictionary learning: A data-driven adaptive spectral decomposition of the koopman operator. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(10):003111 2017.
[26] G. Mamakoukas, M. Castano, X.
Tan, and Murphey. Local Koopman operators for data- driven control of robotic systems. In Robotics: science and systems, 2019.
[27] B. Huang, Ma, and Vaidya. Feedback stabilization using koopman operator. 20/8 IEEE Conference on Decision and Control (CDC), pages 6434-6439. IEEE, 2018.
[28] X. Ma, Huang, and U. Vaidya. Optimal quadratic regulation of nonlinear system using koopman operator. In 2019 American Control Conference (ACC), pages 491 1-191... IEEE, 2019.
10

---

[29] R. Wang, Y. Han, and U. Vaidya. Deep koopman data-driven optimal control framework for autonomous racing. Early Access, 5, 2021.
[30] E. Kaiser, N. Kutz, and S. L.. Brunton. Data-driven discovery of koopman eigenfunctions for control. Machine Learning: Science and Technology, 2(3):035023, 2021.
[31] N. Kutz, S. Brunton, B. W. Brunton, and J. Proctor. Dynamic mode decomposition: data-driven modeling of complex systems. SIAM, 2016.
[32] D. C.-L. Fong and M. Saunders. LSMR: An Iterative Algorithm for Sparse Least-Squares Problems. 53(5):2950-2977) ISSN 1064-8275. doi:10.1137/11 1171111111//0079677... URL https:
/ //epubs.. am. www.or///ooooo httgg////aab//////gggggggg abs/ wttpa/a.............///.11 /011/////0001111110777777
[33] C. C. Paige and M. A. Saunders. LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares. 8(1):43-71. ISSN 0098-3500, 1557-7295. doi: 140014/4//58883559999 URL https: //11. acm. /ttgg//oo/1././// 1145 1ee...111111154441110 3559999
[34] P. Strobach. Recursive Least-Squares Using the QR Decomposition. In P. Strobach, edi- tor, Linear Prediction Theory: A Mathematical Basis for Adaptive Systems, Springer Series in Information Sciences, pages 63-101. Springer. ISBN 978-3-642-75206-3 doi:10.1007/ 978-3-642-75206-3444 URL http:::://o..org/000000////////0 --0-----4---2552 4.
[35] A. Sayed and T. Kailath. Recursive Least-Squarer Adaptive Filters, volume 20094251 of Elec- trical Engineering Handbook,, pages 1-40. CRC Press. ISBN 978- 198N 4607-6. doi: 10. 10.11000...011999112244666677111
hwt.:h:::/wwwccrcttttttt htttt:://ww.crccetbbseee.. cttpr//cnnneaaeeeecooooooooooo..
abs/10. 11t.1121//9000 abbs/10..2200/9991220006667---
[36] A. Ghirnikar and S. Alexander. Stable recursive least squares filtering using an inverse QR decomposition. In International Conference on Acoustics, Speech, and Signal Processing, pages 1623-1626 vol.3. doi:10.1 1101100100//AAASP1999000.01 115736.
[37] T. A. Howell, B. E. Jackson, and Z. Manchester. ALTRO: A Fast Solver for Constrained Trajectory Optimization. pages 7674-7679. ISSN 9781728140044 doi: 109/1R0000/777 8019.8967788.
[38] Z. Manchester, Lipton, Wood, and S. Kuindersma. Variable forward-Sweep Wing Design for Enhanced Perching in Miero Aerial Vehicles. In AIAA Aerospace Sciences Meeting- URL https:tt////mm //tp///reexx rexlab........ stanford.d http:/dddddppppppppeeee httper/s//Mohhhigg______..
pdf.
[39] B. E. Jackson, Punnoose, D. Neamati, K. Tracy, R. Jitosho, and Z. Manchester. ALTRO- Control: ALTRO-C: Fast Solver for Conic
C: A Fast Solver for Conic Model-Predictive Control. ISSN 9781728190778. doi: 101.1109//CAAA455002000001111110 URL
com/.
[40] Moore, R. Cory, and R. Tedrake. Robust post-stall perching with simple fxcedwwin glider using LQR-Trea Bioinspiration & Biomimetics, 9(2):225013, May 2014. doi: 10. (0.1088/174--3182299 01177444881182///050011 URL. https: /ttp///oi.io/ .org 101.0.11881120 312////1/97833-/////// 2/025013. Publisher: IOP Publishing.
[41] H. Schaub and Junkins. Analytical Mechanics of Space Systems. AIAA Education Series. AIAA, Reston, VA, second edition, 2009. ISBN 1-60086-721-9.
[42] F. Djeumou, C. Neary, E. Goubault, S. Putot, and U. Topcu. Neural networks with physics- informed architectures and constraints for dynamical systems modeling. In Learning for Dy- namics and Control Conference, pages 263-277. PMLR, 2022.


Reviews:
[{"review_id": "vtMDwetlZ0Y", "reviewer": "Reviewer_5W5c", "summary": "The paper proposed a method for regularised fitting of Koopman models based on use of prior model knowledge introduced to the algorithm through Jacobians. In doing so, its main contribution is to extend the Extended Dynamic Mode Decomposition (EDMD) approach for model predictive control (MPC) applications. The performance of the approach is evaluated in a series of experiments for the control of three simulated benchmark problems, and shows improved performance as compared to traditional MPC or EDMD.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is very clearly written. The quality of the figures is excellent.\n- The mathematical development is appropriate.\n- The evaluations provide a thoughtful analysis of the proposed approach (beyond simply claiming that it works better than the state of the art) and go a good way toward characterising how it performs and why it behaves the way it does.\n- The apparent performance gains are impressive.\n- The video is excellent in explaining the work.\n\nWeaknesses:\n- Evaluations are only presented for a single trial, so it is impossible to judge whether the results are representative or statistically significant.\n- The assumption that the inverse mapping is linear (equation 3) seems very strong, and not applicable in practice.\n- No results are presented for a physical system, causing doubt as to the practical applicability.\n- The discussion of limitations is brief and more or less denies that there are any. This is implausible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper is of good quality, and addresses an important problem. It presents an elegant and seemingly effective approach to the problem in hand that I would expect to perform well if applied to a real, physical system. I did not detect any issues with the mathematical development. The evaluations are thoughtful and engaging. The results appear to be impressive, but lack statistical rigour making their reliability dubious.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Please discuss the plausibility of the assumption made in equation 3 and its implications for practical use.\n- Please repeat all evaluations for multiple trials with randomised data sets and provide statistics on the results. As a minumum I would expect the mean+/-standard devation performance to be presented. \n- Please fix the formatting problem on Lines 182-184."}}, {"review_id": "r7IV3xYUkU", "reviewer": "Reviewer_R4eq", "summary": "Combining prior knowledge with small data model learning is a significant problem for robust Model Predictive Control. The authors proposed a novel model learning method, JDMD that adds the Jacobian information of the dynamical system to Dynamic Mode Decomposition, DMD. Because DMD treats a nonlinear system as a linear system, the nonlinear control problem can be solved as the linear control. The proposed method devised the efficient computation algorithm for model learning. In addition, the authors demonstrated the validity of sample efficiency of the gradient information-based DMD by trajectory tracking problem of three numerical simulations, Cartpole, Airplane, and, Quadrotor.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. Jacobian-Regularized DMD (JDMD)\\\nJDMD may be so robust than the full data-driven learning that this method can reflect the dynamical property a designer knows. The efficient recursive algorithm based on the QR decomposition is also proposed.\n1. Experimental results show that JDMD improves the Quadratic Problem to track the reference trajectory. \\\nThis paper evaluated sample efficiency for model learning in Sec. 6.2, extrapolation data performance in Sec. 6.3, and bias error of prior knowledge in Sec. 6.5 comparing with the non-gradient information model learning. Every result indicates the proposed method improves the MPC score.\n\nWeaknesses\n\n1. There is no model error evaluation.\\\nThe model error between true dynamics and the estimated model is interesting to confirm the proposed method is superior. Furthermore, The Jacobians matrix error among true, prior, and estimated would assess the performance of the proposed algorithm. \n1. The limitation of the target dynamics system and ambiguous design of the prior knowledge\\\nThe Conventional bilinear Koopman Model[13] assumes that the dynamical system is the Control-affine, but this paper defines the more general form as shown in Equation (1). This assumption is critical to lift the dynamical system onto the bilinear model[Surana, 2016]. A detailed description is necessary to clarify the limitation of the JDMD. On the other hand, obtaining the dynamical system Jacobians sometimes is a challenge, even if the dynamics model has already been received. The clear guide of the prior design is significant for applying the wide system.\n\nSurana, Amit. \"Koopman operator based observer synthesis for control-affine nonlinear systems.\" 2016 IEEE 55th Conference on Decision and Control (CDC). IEEE, 2016.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I consider the limitations of the proposed method to apply broad dynamical systems. JDMD method needs control affine system and the Jacobian knowledge, and the discussion seems not enough about these demands. Furthermore, because this paper presently concentrates on evaluating MPC score, the model error of prediction state and Jacobians should study to indicate the JDMD supremacy.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Model Error Evaluation\\\nSee above\n2. detail prior Jacobian knowledge \\\nHow did the prior Jacobian knowledge defined in each experiment?\n3. I need more description to clarify the difference between Lifted and Projected MPC.\\\nThe MPC of the previous EDMD[13] is solved in the observables space $\\mathcal{Y}$. Is the state space where the QP problem solves a difference?\n4. Does the $\\phi(x)$ term appear in $\\hat{B}$ of equation (10)?\\\nI think differentiation of equation (8) with respect to $u_i$ gives us the term of $C_i\\phi(x)$.\n"}}, {"review_id": "p-wync6wmM5", "reviewer": "Reviewer_Uwdc", "summary": "Motivated by Koopman operator theory, this paper considers the setting where a given nonlinear dynamical system is approximated by a lifted bilinear system, where the parameters of the latter are fit on (state, control, next-state) triplets with respect to both next-state prediction loss as well as a dynamics Jacobian matching loss. The paper sketches efficient least squares implementations for this approach. Experimental results show that adding a Jacobian loss term improves MPC tracking error; that learning is more data efficient; leads to better generalization to initial conditions as well as greater robustness to model mismatch.\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper reminded the reviewer of a recent approach called TaSIL: Taylor Series Imitation Learning (https://arxiv.org/abs/2205.14812) which experimentally and theoretically shows the benefit of Jacobian matching in addition to vanilla action-prediction matching in Imitation Learning settings. One can view the proposed approach in this paper from a similar lens where the original nonlinear dynamics is an expert that the lifted bilinear dynamics attempts to imitate. It would be interesting to see if TaSIL analysis applies in this setting or not.\n\n- The paper is clearly written and the experiments are nicely organized and convincing.\n\nWeaknesses:\n\n- No real on-robot demonstrations.\n- Eqn 3 seems like an overly strong assumption\n- Learning is partial in the sense that it only covers the coefficient matrices of Eqn 2, not the embedding function phi or the unlifting matrix G. It is not clear how to set these effectively.\n- While effective in practice, the Projected bilinear MPC is a heuristic without a sound theoretical justification.\n- The empirical results are on smallish systems - it is not clear how far such an approach can be pushed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Nice, clear formulation that leads to an effective dynamics learning and MPC approach, with several empirical gains albeit demonstrated in smallish systems (and not on real hardware).", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- A plot of performance as a function of alpha would be insightful to add to understand the relative important of Jacobian matching and also sensitivity of performance.\n\n- How are fixed basis functions phi and unlifting matrix G to be chosen for a new problem?\n\n"}}, {"review_id": "TviTWulWnS5", "reviewer": "Reviewer_4PJQ", "summary": "The proposed approach improves the sample efficiency of Koopman-based models by introducting an auxillary loss minimizing the difference between the linearized Koopman dynamics and a linearized approximate model, coined Jacobian regularization. The method is evaluated using a downstream trajectory tracking task using MPC, on cartpole, quadcopter and plane dynamics in simulation.  ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think ideas of combining black-box models with physics-based inductive biases is an interesting research direction. Moreover, using about derivative or Jacobian information is a seldom-explored direction in machine learning. I thought this paper was also clearly written and had a very pleasing quality to the presentation. \n\nMy main issue with the paper is its motivation and execution. This is a 'solution-first' paper rather than a 'problem-first' one, as Koopman models are part of the motivation. I took issue with this beacuse Koopman dynamical systems are rather niche, and I have not seen convincing evidence that they are a particularly promising direction of research. Nonlinearities with infinite taylor series require infinitely-sized latent spaces, and therefore Koopman dynamical systems are best suited to systems with polynomial nonlinearities like Van der Pol oscillators, whereas robotics typically trignometric nonlinearities.  Moreover, compared to direct function approximation of the dynamical system, Koopman dynamical systems introduce three approximators (encoder, dynamics and decoder) which means there are additional sources of error and drift. I believe this paper is missing a baseline of a standard 2 layer MLP (perhaps with sine and cosine features) that approximates the dynamics directly. For the trajectory tracking MPC task, this MLP can also be linearized offline using forward-mode automatic differentiation. \n\nMy second issue is with the hand-crafted latent space. Since the dynamic state $x$ is part of the latent state $\\phi(x)$, this suggests that the Koopman model is already predicting the next state directly using feature regression, i.e. $x_{t+1} = \\mathbf{w}^T\\phi(x_t)$? The fact of the Koopman model having a linear decoder was very strange to me, and suggests this model is infact something simpler, since previous Koopman methods i have seen use some form of neural network autoencoder. I think this feature regression baseline should also be used a baseline, since it is also straightforward to learn and linearize.\n\nNext, while the Jacobian regularization is an interesting one, the paper does not really engage with the general question of 'how should I use an approximate oracle model?' and rather proposes one solution. I think the paper could be improved if it evaluates alternative approaches as well, such as learning a residual model, or augmenting the dataset with noise and encouraging predictive similarity when predicting outside of the data distribution. Without this investigation, the paper is improving on vanilla Koopman models by providing some form of oracle knowledge, which is not so insightful as the advantage is clear.  \n\nRegarding the evaluation, I found the trajectory tracking task on simple environments like cartpole a bit lacking for a venue like CoRL, especially as there are no real-world experiments. Moreover, there seems to be a complete absence of reporting confidence intervals and evaluating over random seeds.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The submission is highly focused around one particular solution, and does not engage outside Koopman models or other methods of using approximate oracle models. I have concerns over the motivation for Koopman, and I find the experiments a bit limited in complexity and statistical evaluation. Therefore, I am leaning towards rejection. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "I refer to my question above on the need for Koopman models given the feature regression it is essentially doing if the true state is in the latent state. ", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)