Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Combining prior knowledge with small data model learning is a significant problem for robust Model Predictive Control. The authors proposed a novel model learning method, JDMD that adds the Jacobian information of the dynamical system to Dynamic Mode Decomposition, DMD. Because DMD treats a nonlinear system as a linear system, the nonlinear control problem can be solved as the linear control. The proposed method devised the efficient computation algorithm for model learning. In addition, the authors demonstrated the validity of sample efficiency of the gradient information-based DMD by trajectory tracking problem of three numerical simulations, Cartpole, Airplane, and, Quadrotor.
strengths_and_weaknesses: Strength
1. Jacobian-Regularized DMD (JDMD)\
JDMD may be so robust than the full data-driven learning that this method can reflect the dynamical property a designer knows. The efficient recursive algorithm based on the QR decomposition is also proposed.
1. Experimental results show that JDMD improves the Quadratic Problem to track the reference trajectory. \
This paper evaluated sample efficiency for model learning in Sec. 6.2, extrapolation data performance in Sec. 6.3, and bias error of prior knowledge in Sec. 6.5 comparing with the non-gradient information model learning. Every result indicates the proposed method improves the MPC score.

Weaknesses

1. There is no model error evaluation.\
The model error between true dynamics and the estimated model is interesting to confirm the proposed method is superior. Furthermore, The Jacobians matrix error among true, prior, and estimated would assess the performance of the proposed algorithm. 
1. The limitation of the target dynamics system and ambiguous design of the prior knowledge\
The Conventional bilinear Koopman Model[13] assumes that the dynamical system is the Control-affine, but this paper defines the more general form as shown in Equation (1). This assumption is critical to lift the dynamical system onto the bilinear model[Surana, 2016]. A detailed description is necessary to clarify the limitation of the JDMD. On the other hand, obtaining the dynamical system Jacobians sometimes is a challenge, even if the dynamics model has already been received. The clear guide of the prior design is significant for applying the wide system.

Surana, Amit. "Koopman operator based observer synthesis for control-affine nonlinear systems." 2016 IEEE 55th Conference on Decision and Control (CDC). IEEE, 2016.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I consider the limitations of the proposed method to apply broad dynamical systems. JDMD method needs control affine system and the Jacobian knowledge, and the discussion seems not enough about these demands. Furthermore, because this paper presently concentrates on evaluating MPC score, the model error of prediction state and Jacobians should study to indicate the JDMD supremacy.
issues: 1. Model Error Evaluation\
See above
2. detail prior Jacobian knowledge \
How did the prior Jacobian knowledge defined in each experiment?
3. I need more description to clarify the difference between Lifted and Projected MPC.\
The MPC of the previous EDMD[13] is solved in the observables space $\mathcal{Y}$. Is the state space where the QP problem solves a difference?
4. Does the $\phi(x)$ term appear in $\hat{B}$ of equation (10)?\
I think differentiation of equation (8) with respect to $u_i$ gives us the term of $C_i\phi(x)$.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Motivated by Koopman operator theory, this paper considers the setting where a given nonlinear dynamical system is approximated by a lifted bilinear system, where the parameters of the latter are fit on (state, control, next-state) triplets with respect to both next-state prediction loss as well as a dynamics Jacobian matching loss. The paper sketches efficient least squares implementations for this approach. Experimental results show that adding a Jacobian loss term improves MPC tracking error; that learning is more data efficient; leads to better generalization to initial conditions as well as greater robustness to model mismatch.


strengths_and_weaknesses: Strengths:

- This paper reminded the reviewer of a recent approach called TaSIL: Taylor Series Imitation Learning (https://arxiv.org/abs/2205.14812) which experimentally and theoretically shows the benefit of Jacobian matching in addition to vanilla action-prediction matching in Imitation Learning settings. One can view the proposed approach in this paper from a similar lens where the original nonlinear dynamics is an expert that the lifted bilinear dynamics attempts to imitate. It would be interesting to see if TaSIL analysis applies in this setting or not.

- The paper is clearly written and the experiments are nicely organized and convincing.

Weaknesses:

- No real on-robot demonstrations.
- Eqn 3 seems like an overly strong assumption
- Learning is partial in the sense that it only covers the coefficient matrices of Eqn 2, not the embedding function phi or the unlifting matrix G. It is not clear how to set these effectively.
- While effective in practice, the Projected bilinear MPC is a heuristic without a sound theoretical justification.
- The empirical results are on smallish systems - it is not clear how far such an approach can be pushed.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Nice, clear formulation that leads to an effective dynamics learning and MPC approach, with several empirical gains albeit demonstrated in smallish systems (and not on real hardware).
issues: - A plot of performance as a function of alpha would be insightful to add to understand the relative important of Jacobian matching and also sensitivity of performance.

- How are fixed basis functions phi and unlifting matrix G to be chosen for a new problem?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The proposed approach improves the sample efficiency of Koopman-based models by introducting an auxillary loss minimizing the difference between the linearized Koopman dynamics and a linearized approximate model, coined Jacobian regularization. The method is evaluated using a downstream trajectory tracking task using MPC, on cartpole, quadcopter and plane dynamics in simulation.  
strengths_and_weaknesses: I think ideas of combining black-box models with physics-based inductive biases is an interesting research direction. Moreover, using about derivative or Jacobian information is a seldom-explored direction in machine learning. I thought this paper was also clearly written and had a very pleasing quality to the presentation. 

My main issue with the paper is its motivation and execution. This is a 'solution-first' paper rather than a 'problem-first' one, as Koopman models are part of the motivation. I took issue with this beacuse Koopman dynamical systems are rather niche, and I have not seen convincing evidence that they are a particularly promising direction of research. Nonlinearities with infinite taylor series require infinitely-sized latent spaces, and therefore Koopman dynamical systems are best suited to systems with polynomial nonlinearities like Van der Pol oscillators, whereas robotics typically trignometric nonlinearities.  Moreover, compared to direct function approximation of the dynamical system, Koopman dynamical systems introduce three approximators (encoder, dynamics and decoder) which means there are additional sources of error and drift. I believe this paper is missing a baseline of a standard 2 layer MLP (perhaps with sine and cosine features) that approximates the dynamics directly. For the trajectory tracking MPC task, this MLP can also be linearized offline using forward-mode automatic differentiation. 

My second issue is with the hand-crafted latent space. Since the dynamic state $x$ is part of the latent state $\phi(x)$, this suggests that the Koopman model is already predicting the next state directly using feature regression, i.e. $x_{t+1} = \mathbf{w}^T\phi(x_t)$? The fact of the Koopman model having a linear decoder was very strange to me, and suggests this model is infact something simpler, since previous Koopman methods i have seen use some form of neural network autoencoder. I think this feature regression baseline should also be used a baseline, since it is also straightforward to learn and linearize.

Next, while the Jacobian regularization is an interesting one, the paper does not really engage with the general question of 'how should I use an approximate oracle model?' and rather proposes one solution. I think the paper could be improved if it evaluates alternative approaches as well, such as learning a residual model, or augmenting the dataset with noise and encouraging predictive similarity when predicting outside of the data distribution. Without this investigation, the paper is improving on vanilla Koopman models by providing some form of oracle knowledge, which is not so insightful as the advantage is clear.  

Regarding the evaluation, I found the trajectory tracking task on simple environments like cartpole a bit lacking for a venue like CoRL, especially as there are no real-world experiments. Moreover, there seems to be a complete absence of reporting confidence intervals and evaluating over random seeds.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The submission is highly focused around one particular solution, and does not engage outside Koopman models or other methods of using approximate oracle models. I have concerns over the motivation for Koopman, and I find the experiments a bit limited in complexity and statistical evaluation. Therefore, I am leaning towards rejection. 
issues: I refer to my question above on the need for Koopman models given the feature regression it is essentially doing if the true state is in the latent state.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposed a method for regularised fitting of Koopman models based on use of prior model knowledge introduced to the algorithm through Jacobians. In doing so, its main contribution is to extend the Extended Dynamic Mode Decomposition (EDMD) approach for model predictive control (MPC) applications. The performance of the approach is evaluated in a series of experiments for the control of three simulated benchmark problems, and shows improved performance as compared to traditional MPC or EDMD.
strengths_and_weaknesses: Strengths:
- The paper is very clearly written. The quality of the figures is excellent.
- The mathematical development is appropriate.
- The evaluations provide a thoughtful analysis of the proposed approach (beyond simply claiming that it works better than the state of the art) and go a good way toward characterising how it performs and why it behaves the way it does.
- The apparent performance gains are impressive.
- The video is excellent in explaining the work.

Weaknesses:
- Evaluations are only presented for a single trial, so it is impossible to judge whether the results are representative or statistically significant.
- The assumption that the inverse mapping is linear (equation 3) seems very strong, and not applicable in practice.
- No results are presented for a physical system, causing doubt as to the practical applicability.
- The discussion of limitations is brief and more or less denies that there are any. This is implausible.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: Overall, the paper is of good quality, and addresses an important problem. It presents an elegant and seemingly effective approach to the problem in hand that I would expect to perform well if applied to a real, physical system. I did not detect any issues with the mathematical development. The evaluations are thoughtful and engaging. The results appear to be impressive, but lack statistical rigour making their reliability dubious.
issues: - Please discuss the plausibility of the assumption made in equation 3 and its implications for practical use.
- Please repeat all evaluations for multiple trials with randomised data sets and provide statistics on the results. As a minumum I would expect the mean+/-standard devation performance to be presented. 
- Please fix the formatting problem on Lines 182-184.
review_title: null

Meta-review: metareview: ### Initial Meta-review:
#### Strength:

This paper is clearly written and provides mathematically sound formulation. The ideas presented in this paper appear to be an interesting research direction. The results are nicely organized and convincing.

#### Weakness:

No hardware evaluation is provided. The evaluations are performed on very simple tasks with a single trial. Statistical evaluation is highly suggested with multiple trials. The assumption of equations 3 appears to be overly strong, whose practical plausibility should be discussed. In addition, effect of modelling error should be discussed.

### Final Meta-review:
The authors have addressed the questions and concerns raised by the reviewers and revised the paper in a detailed manner. One of the reviewers mentioned that the score will be raised from weak accept to strong accept. Future hardware evaluation would be highly suggested.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)