Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper considers the problem of fly drones with wind. Neural networks are used to learn the effects of aerodynamic disturbances, which are the uncertainties between the true and estimated Control Contraction Metrics (CCMs). An RL agent is used to learn a policy, which computes a correction to the nominal contraction metric. The corrected contraction metric is then used in an MPC framework to construct the contraints of the MPC. It The approach is evaluated in simulation scenarios. 
strengths_and_weaknesses: Strengths:
1. Theoretical results.
2. Experiments in a simulation scenario.

Weakness:
1. No hardware experiments.
2. Concerns about the design of the reward function. (See Issues)
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Although no hardware experiments were conducted, the authors presented strong theoretical results and experimental results in simulation.
issues: 1. The purpose of \mu^M_1 and \mu^M_2 is to make \hat{C}_m as close as to the true value C_m. If I understand correctly, the third item in R_{contraction} is for this purpose. But I don't understand how it works. This item can only make sure that \hat{C}_m \prec - 2\lambda M, but it does not imply that \hat{C}_m is close to C_m.
2. In line 137, it seems there should be a Z_{\theta}.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose the QuaDUE-CCM framework for robust trajectory tracking for quadrotors with highly variable uncertainties due to aerodynamic forces. The authors provide convergence guarantees as well as simulation results that demonstrate the effectiveness of the framework.
strengths_and_weaknesses: The paper provides good experimental results that compare QuaDUE-CCM against state-of-the-art methods as well as convergence guarantees thoroughly supported by proofs. Unfortunately, I was unable to follow most of the proofs. Personally, I think the paper would have been easier to follow if it focused more on formulation of the framework and guarantees, and moved the proofs to the appendix. A section referring to how QuaDUE-CCM applies to other systems would help the paper as well.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The experimental results and guarantees seem thorough. However, I found the paper very confusing, mainly due to my lack of expertise, but also due to the dense and hard to follow writing.
issues: See above
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method for estimating uncertainties in quadrotor dynamics using distributional reinforcement learning and control contraction metrics (CCM). In particular, distributional RL is used to estimate the disturbances that the system is subjected to, and this is used in a CCM-based trajectory tracking control framework. The approach is demonstrated in simulation, and is shown to improve tracking error relative to baseline algorithms.
strengths_and_weaknesses: Strengths: The paper is generally well-written, mathematically solid, and provides an interesting synthesis between distributional RL and control contraction metrics which may be of interest for future research.

Comments, concerns, suggestions, and questions:
- There are some ambiguities in notation which sometime make it difficult to follow the math:
    - It seems $\lambda$ is overloaded in the proof of Theorem 5 (contraction rate vs. step size)?
    - It is unclear how $\Delta_1$, $\Delta_2$ factor into (1) or into the control affine form $\dot x = f(x) + B(x)u + w$.
    - $e_{fk}$ and $T_i$ are not defined
    - Lie derivative $\partial_{(\cdot)}$ is not defined
    - Several variables in Theorem 5 are not defined
- There is missing previous related work [1, 2] which also estimates worst-case disturbances in the context of learning-based motion planning to determine how they affect the tracking performance of a CCM-based controller.
- Related to the last note, it would be good to compare the proposed distributional RL approach to a “robust CCM” baseline [1, 2, 3], which attempts to do feedback control assuming worst-case disturbance, and the improvements in conservativeness afforded by using a distributional approach.
- How are $\omega_{c,i}$ chosen in the reward function? I couldn’t find information on that in the main body.
- In the QP in Figure 1, in general, it is difficult to guarantee a priori whether there exists a control input $u$ within the control limits $u_min$ and $u_max$ that satisfies the contraction condition, especially for a $\hat C$ which is being learned. In practice, do you run into feasibility problems in the QP? More comments clarifying this in the text would be helpful.
- In parts of the paper, to aid it would be good to have more intuition describing the math: it is unlikely that people are intimately familiar with both CCMs and distributional RL. For instance, before (2), you might describe what the “maximal form” is, and why it’s important that it has contractive properties, e.g., “the approximate Wasserstein distance between $Z_1$ and $Z_2$ shrinks upon applying the Bellman operator, which is important for convergence guarantees later on”.
- It would be good to have more details on the training of M, u, and the selection of the contraction rate \lambda.
- It would help to move some of the QuaDUE-CCM algorithm details from the appendix into the main body - right now the algorithm is described rather vaguely. More generally, in the main body of the paper, please reference the proofs and additional relevant material in appendix.

[1] Chou, Ozay, and Berenson. Model Error Propagation via Learned Contraction Metrics for Safe Feedback Motion Planning of Unknown Systems. CDC 2021.

[2] Chou, Ozay, and Berenson. Safe Output Feedback Motion Planning from Images via Learned Perception Modules and Contraction Theory. WAFR 2022.

[3] Singh, Landry, Majumdar, Slotine, and Pavone. Robust Feedback Motion Planning via Contraction Theory.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is generally well-written and is an interesting combination of CCMs and distributional RL, which hopefully may inform a future line of research -- thus, I recommend acceptance.
issues: See the comments above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes: (1) QuaDUE, a distributional RL based uncertainty estimator that evaluates aerodynamic effects that are difficult to model directly; (2) QuaDUE-CCM, a quadrotor trajectory tracking system that incorporates QuaDUE into a MPC controller based on control contraction metric. Thorough theoretical analysis (in both the paper and the supplementary document) shows that the QuaDUE is guaranteed to converge, and training acceleration could be achieved when the approximation error of the learned distribution is low. In the experiment, it could be seen that within the shown learning period QuaDUE outperforms DDPG in terms of the learned cumulative return. In addition, compared to DDPG-CCM, QuaDUE-CCM exhibits higher success rates in trajectory tracking tasks when external forces in the simulated environment are large.
strengths_and_weaknesses: Strengths: The theoretical analysis of convergence and training acceleration of QuaDUE demonstrates the method's capability with solid proof. The description of QuaDUE-CCM is comprehensive, and the supplementary document provides useful explanations about the quadrotor dynamic model, algorithm of kino-JSS and more details about the QuaDUE-CCM implementation.

Weaknesses: It's not clear that the learning curves shown in Figure 3 converge, hence I think it's not quite convincing to view the result as an evidence of the convergence guarantee described in theoretical analysis. Besides, QuaDUE shows similar learning performance as C51-DDPG, and QuaDUE-CCM has similar trajectory tracking performance as QuaDRED-MPC. Thus, experiment results seem to illustrate that the advantage of distributional RL methods in uncertainty estimation, and the proposed algorithms show a common performance among them.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper proposes a novel distributional RL based uncertainty estimator, and extends it to a quadrotor trajectory tracking system that is based on a control contraction metric based controller. Detailed theoretical analysis with respect to the convergence and training acceleration property of the proposed estimator is given. Simulation experiment results show that the proposed tracking system has higher success rates in trajectory tracking tasks with large external force than GP-MPC, a state-of-the-art trajectory tracking method, and DDPG-CCM, a system built with traditional RL.

Post-Rebuttal Comments: I appreciate the clarification provided by the authors, and I remain in favor of acceptance. Taking the authors' responses and the CoRL review criteria into account, I have decided to keep my rating of weak accept.
issues: How the nominal model is learned with the nominal function via contraction metric learning is not described in the paper, and I think it would be more clear to the audience if the methodology of contraction metric learning is included. Besides, in the experimental study of QuaDUE-CCM, some other method being compared include a nonlinear MPC controller, but no description about its formulation is given.
review_title: null

Meta-review: metareview: The reviewers evaluated this paper positively as it contains a solid theory (proof of convergence) for a new algorithm that combines distributional RL and control contraction metrics . The experiments are applied to a quadrotor trajectory following problem and showed good performance. One reviewer advocated an oral presentation, but I would rather recommend a poster presentation here due to missing hardware experiments and a rather narrow evaluation of the algorithm (only one domain).
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)