Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this paper, the authors propose KNODE-MPC-Online, an MPC that relies on Knowledge Based Neural Ordinary Differential Equations as dynamic models and updates the model online using neural networks. Experiments are conducted both in simulation and on a real robot.
strengths_and_weaknesses: Pros:
* The paper is well written and well structured
* Related work is sufficiently addressed
* Real world experiments are conducted and limitations are discussed
* The approach is interesting and pseudocode is provided in the form of algorithm

Cons:
* Some clarifications should be provided about the approach which is currently not reproducible
according to how it is described:
  - How do you compute the M matrix?
  - It is unclear how you use learned model in the MPC (Although there is a section about that in the paper). Specifically, given you learn several models, how do you 'choose' (or mix) them in order to obtain your prediction?
* It would be interesting to compare against other learning methods (even if not online) with MPC
to have a better sense of how it performs
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well written and well structured, but it lacks some clarity in the approach which make it currently difficult to re-implement.

Although the paper is somewhat incremental, some clarity would make it much better and potentially useful. Experiments are interesting,
but it would have been nice to have a larger set of baselines to compare against that make use of learning with MPC (even if not online).
issues: * Clarity
* Some more experiments missing (minor)
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The major contribution of this paper is to augment the existing work [20] to be implemented for online training. For the learning of dynamics to execute online, neural networks are stacked with weighted outputs to progressively capture the new features in the vehicle dynamics. Experiments are conducted to show the capability of the proposed method to adapt to changing dynamics of a quadrotor.
strengths_and_weaknesses: The major strength of the paper is to design an iterative update scheme to incorporate newly learned NN to describe the dynamical model from online data better. The more precise dynamics will yield better knowledge in the MPC controller and hence improve tracking performance subject to changes in the dynamics.

The major weakness of the paper is the lack of description/explanation of the selection mechanism of the stacked NN. Specifically, the authors mentioned the selection matrix M_\psi, which “couples the neural network with knowledge.” However, there’s no description of how the selection matrix is formed or designed. The ambiguity introduces further questions to the proposed framework. For example, what if a newly introduced NN contains the information captured by a previously stacked NN? This question tailors more towards the efficiency of stacking NN’s where it is possible that the NN’s contain repeated knowledge.

Another weakness is that the paper frequently refers to the adaptiveness of the proposed framework. Yet the literature review covers no results from the adaptive control community. However, there are recent adaptive control designs that work fairly well for quadrotor control. The authors should include it in the literature review and cite related works, e.g., 
[1] Kostadinov, D. and Scaramuzza, D., 2020, August. Online weight-adaptive nonlinear model predictive control. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1180-1185). IEEE.
[2] Hanover, D., Foehn, P., Sun, S., Kaufmann, E. and Scaramuzza, D., 2021. Performance, precision, and payloads: Adaptive nonlinear mpc for quadrotors. IEEE Robotics and Automation Letters, 7(2), pp.690-697.
[3] Joshi, G., Virdi, J. and Chowdhary, G., 2020. Asynchronous deep model reference adaptive control. arXiv preprint arXiv:2011.02920.
[4] Wu, Z., Cheng, S., Ackerman, K.A., Gahlawat, A., Lakshmanan, A., Zhao, P. and Hovakimyan, N., 2022, May. L 1 Adaptive Augmentation for Geometric Tracking Control of Quadrotors. In 2022 International Conference on Robotics and Automation (ICRA) (pp. 1329-1336). IEEE.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend this paper to be accepted, given that the authors can clearly explain how the selection matrix M_\psi is designed and include the references of adaptive control for quadrotors.  

issues: A couple of minor issues are listed below in addition to the major ones summarized in the “Strengths And Weaknesses” section:
1. In line 49, towards the end, there is a typo ‘hysical.’
2. In line 81, a period is missing between [11] and knowledge-based
3. In figure 2, given that the heatmap does not show a clear tendency in the 3x3 grids, it might be more straightforward to turn the MSE into a table with numbers.
4. In figure 3, the plot on the right can be more informative if the horizontal axis is time instead of y position (to show the altitude holding capability).
5. Please remove the last paragraph in Section 7 as it is not related to the paper's focus.
review_title: null

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper developed a framework for online updating of the dynamics modeled by KNODE for MPC.
This ensured adaptability to changes in the environment, and even when unknown weights were actually attached to the drone, that change was reflected appropriately in the dynamics to achieve optimal control.
strengths_and_weaknesses: Strengths:
- The online adjustment of dynamics enhanced adaptability to the environment.
- The proposed method introduced a trick to avoid forgetting the past, which is an important point in online learning.


Weaknesses:
- Online dynamics learning itself is not new.
- In avoiding past forgetting, the append-type method diverges in both memory and computational cost.
- The adaptability of the proposed method is not well demonstrated with only one example.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: In the context of model-based RL, it is not uncommon to update dynamics online, which is not consistent with the claims of this paper.
Also, if online learning is anticipated, a structure such as the proposed method, which ultimately requires infinite memory and computational cost, is not appropriate.
As mentioned in the Limitations section, I cannot find the value of this method unless the proposed method incorporates tricks to keep the computational cost constant.
issues: I would like to see a proper survey of previous studies, as online methods for updating dynamics are not uncommon.

The specific MPC solver was unknown unless I am missing something.
Some solvers are computationally very expensive, and optimization should not be possible on time for methods such as the proposed method, where the computational cost of the dynamics explodes.
With this in mind, please clarify the solver and discuss the realtimeness of the optimization.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an online learning framework in the context of model predictive control. The method includes a joint, parallel framework where the control scheme is updated based on new, incoming models; and the models are trained and updated based on the acquisition of new data in an online fashion. The models are based on the now-well-known neural ODE framework. The paper evaluates the method with a series of simulated quadrotor experiments w/r/t tracking error. 
strengths_and_weaknesses: Strengths
- The justification for KNODE presented in lines 82-92 is appreciated.
- The limitations section discusses several of my concerns, indicating that the the authors seem to anticipate several questions about the work that might arise

Weaknesses
- Maintaining a potentially large number of neural networks seems burdensome and possibly unnecessary
- There is a lack of a discussion or theory w/r/t convergence guarantees. The paper seems to imply that "more data is better" but this is not always the case, both in terms of the data that is generated itself and/or in terms of the algorithms that are using these data
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper presents an interesting idea and a relevant topic for robotics (and other applications). The paper would be significantly improved with a broader set of evaluation baselines and/or more exposition on theoretical guarantees in the absence of more rigorous experimental (simulation) evalutions. 
issues: Line 85: "First, it requires less data for training...". Less data than *what*? GPs? Standard neural networks? System identification?

I am a bit confused by equations (3) and (4). What is the time step in equation (3), i.e the difference between t_i and t_{i+1}? Is this a sort of design parameter, based on the dynamics of the system under consideration? Also, t_s is in T, i.e. any sampling time in a set. But how is T defined? Is it based on m, since we are averaging over a bunch of different start times? I conceptually understand what is going on here but the terms and nomenclature are slightly confusing. 

How sensitive is this approach, generally speaking, to the numerical integration of equation (4). Numerical solvers / integrators are not always stable; I am curious to see if there are practical issues here. 

Lines 113-118: I appreciate the discussion of t_{col} and the implications of longer or shorter collection intervals. However, it would be nice to see a discussion of how to characterize this tradeoff or identify the optimal tradeoff (perhaps this is coming later). 

In equation (5) and the surrounding discussion, it is proposed to keep all (I think???) of the neural network-based models. This seems burdensome in terms of both time and memory. Can the authors comment on this choice? Why is it necessary to keep all of the preceding models? The authors make both an explicit claim and seem to have the implicit assumption that "more data means better models". I can think of counterexamples. Perhaps the authors feel the same way. The paper also provides no stability or asymptotic guarantees - i.e. that over time the model does indeed get better. Furthermore, the system needs to handle "jumps" in the model, which might cause locally "strange" behavior. i.e. the system experiences something it has not seen before and therefore the NN updating might not be smooth. In short: this type of behavior could be unsafe, even if the overall trend of the iteration is one of improvement. 

Line 124: "to tackle this problem, we repeatedly add new neural networks in parallel" but it is unclear what is done with them. Does the selection matrix, M_phi, continue to adapt? Is this learned or learnable? Is this the mechanism that provides stability for learning? 

The end of section 3.2 discusses the coupling between generating new data, training, and what model(s) the controller might be using In lines 130-133, it states that new data must be produce by an updated controller, and then refers to Alg. 2 Line 4. It is unclear how *waiting* (what the while loop does on line 4) implements this requirement. 

It is still unclear why all the neural networks, f_theta, are needed when the recursion only depends on hat{f} and phi. Isn't it ok to discarbe and of the i^th or earlier neural networks? I am sorry I do not have a specific suggestion here, other than to say there is some confusion here, as well as some broader questions. 

I appreciate the experimental setup and evaluation in section 5. However, the evaluation seems to be too "inward facing". As the authors acknowledge in the related works subsection, there is a lot going on in this area right now. And yet the evaluation includes MPC, KNODE-MPC, and the paper's online variant. As I mentioned earlier, I appreciate the authors' justification for choosing KNODE; however, it is still unclear whether this method would outperform other model-predictive RL (or control) methods that do or do not use machine learning. This question is especially concerning given that KNODE-MPC-Online seems to barely outperform vanilla MPC. 

Lines 245-248: I agree that the continuous time nature of the dynamics could be a limitation. However, I was thinking more about mode switching and hybrid systems in this context, not necessarily the stochastic nature of the problem. Can't the uncertainty / stochastic systems still be modeled in a (discretized) variant of continuous dynamics? I feel like hybrid systems is the real problem here. 

Lines 249-252, the authors acknowledge one of my earlier concerns. One thing that is unclear is how substantial this growth in computational load is: is it exponential in the amount of time under experiment? And then, if we do network pruning or reduced order modeling, do we reduce the accuracy of the models in such a way to render the method ineffective? Furthermore, I had an earlier comment about the tradeoffs in increasing or decreasing collection time. Perhaps I missed this, but a more rigorous assessment of this tradeoff would be nice.
review_title: null

Meta-review: metareview: ### Strengths
- interesting combination of MPC and KNODE (knowledge-based neural differential equations)
- hardware experiments with the crazyflie (with an external computer)

### Weaknesses
- no detail about the solver and, more importantly, about the overall computing cost of the approach, especially for long experiment (more data)
- not many baselines (eg non-MPC baselines)
- some "ablations" might be useful to understand the contribution of each component
- no source code is provided
- the related work section can be improved
- no theoretical guarantees
- statistics: how many times was the *training* algorithm run with different seeds?

### Post-rebuttal update
I would like to thank the authors for their effort to improve their paper. 

I agree with one of the reviewers that the technique itself is not truly original (MPC + model learning is not novel). In addition, the statistics are too weak for empirical results: a single experiment with only 5 seeds is not enough to conclude that the training works reliably (the training algorithm might have been lucky... or the authors might have chosen their network well). However, I do think that combining learning and MPC is a very interesting and promising topic and this paper is an interesting contribution in that direction.
confidence: 2: The area chair is not sure
Decision: Accept (Poster)