QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking
Authors: YANRAN WANG
James O'Keeffe
QIUCHEN QIAN
David Boyle
Venue: CoRL 2022 Poster (2022)
Keywords: Quadrotor trajectory tracking
Learning-based control

Abstract: Accuracy and stability are common requirements for Quadrotor trajectory tracking systems. Designing an accurate and stable tracking controller remains challenging, particularly in unknown and dynamic environments with complex aerodynamic disturbances. We propose a Quantile-approximation-based Distributional-reinforced Uncertainty Estimator (QuaDUE) to accurately identify the effects of aerodynamic disturbances, i.e., the uncertainties between the true and estimated Control Contraction Metrics (CCMs). Taking inspiration from contraction theory and integrating the QuaDUE for uncertainties, our novel CCM-based trajectory tracking framework tracks any feasible reference trajectory precisely whilst guaranteeing exponential convergence. More importantly, the convergence and training acceleration of the distributional RL are guaranteed and analyzed, respectively, from theoretical perspectives. We also demonstrate our system under unknown and diverse aerodynamic forces. Under large aerodynamic forces (>2~ m/s^2), compared with the classic data-driven approach, our QuaDUE-CCM achieves at least a 56.6% improvement in tracking error. Compared with QuaDRED-MPC, a distributional RL-based approach, QuaDUE-CCM achieves at least a 3 times improvement in contraction rate.

QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking
Yanran Wang
James O'Keeffe Imperial College London j okeeffe@imperial ac.uk
Imperial College London yanran. wang20@imperial. ac.
Qiuchen Qian
David Boyle
Imperial College London qiuchen. qian19@imperiai............. ac. uk
Imperial College London
david. hew.dddoooyeeimmrillee.. ac uk
Abstract: Accuracy and stability are common requirements for quadrotor tra- jectory tracking systems. Designing accurate and stable tracking controller remains challenging, particularly in unknown and dynamic environments with complex aerodynamic disturbances. We propose Quantil-approximatio--ae basedd Distributional--einfooedd Uncertainty Estimator (QuaDUE) to accurately identify the effects of aerodynamic disturbances, i.e., the uncertainties between the true and estimated Control Contraction Metrics (CCMs). Taking inspiration from con- traction theory and integrating the QuaDUE for uncertainties, our novel CCM- based trajectory tracking framework tracks any feasible reference trajectory pre- cisely whilst guaranteeing exponential convergence. More importantly, the con- vergence and training acceleration of the distributional RL are guaranteed and analyzed, respectively, from theoretical perspectives. We also demonstrate our system under unknown and diverse aerodynamic forces. Under large aerodynamic forces (>2 m/ compared with the classic data-driven approach, our QuaDUEE CCM achieves at least 56.6% improvement in tracking error. Compared with QuaDRED-MPC a distributional RL-based approach, QuaDUE-CCM achieves at least 3 times improvement in contraction rate.
Keywords: Quadrotor trajectory tracking, Learning-based control
1 Introduction
Designing precise and safe trajectory tracking controller for autonomous Unmanned Aerial Ve- hicles (UAVs), such as quadrotors, unknown and dynamic environments with unpredictable aerodynamic forces. To achieve reliable trajectory tracking for agile quadrotor flights, two key challenges are: 1) solving for highly variable uncertainties caused by these aerodynamic forces; and 2) ensuring the tracking controller performs precisely and reliably under these dynamic uncertainties.
critical yet extremely different problem, particularly in
Previous research shows that aerodynamic effects deriving from drag forces and moment variations caused by the rotors and the fuselage [1] the primary source of uncertainty, where these aerody- namic effects appear prominently at flight speeds greater than mss in wind tunnel experiments [2]. The causes of these effects are extremely complex: combinations of individual propellers, air- frame [3], rotor-rotor and airframe-roion turbulent interactions [4], and other turbulent propagation [5]. These aerodynamic effects are therefore chaotic and difficult to model directly.
To address these concerns, two areas of research have emerged. The first uses a distributional Re- inforcement Learning (RL) [6] method interact with the complex and changeable uncertainties. The second extends the contraction theory [7] to a control-affine system, called Control Contraction Metric (CCM). This contraction certificate guarantees that the system will exponentially converge
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Contraction- Metric-based Control
Distributional Reinforcement Learning
CCM-based MPC
Control Input ti
M(n, wa Qu + Run wubjeet
Ci 2AM -.0
(CCM)
Reol Gucarater
mn x M vil
Menin Mossn
M
State
Nominal Model
glain
Figure 1: QuaDUE-CCM: a distributional-RL. RRL-based estimator and a CCM-based controller
to track any feasible reference trajectory [8]. In applications such as quadrotor tracking, we are concerned with the whole tracking process rather than just stabilizing fixed point. One common problem with certificate control approaches, like CCM-based control, is that they heavily rely on the accurate knowledge of the system model. Thus, when the model is uncertain, robust adaptive versions must be considered.
We propose apaantll-aaproxximatiin based Distributional Uncertainty Estimatior for Control Contraction Metric (QuaDUE-CCM) as a precise, reliable and systematic quadrotor tracking fram- work for with high variance aerodynamic effects. Our contributions can be summarized:
1) QuaDUE, a distributional-RR-. sufficiently estimate variable uncertainties of contraction metrics. In the cases tested, we show that QuaDUE outperforms traditional RL, such as Deep Deterministic Policy Gradient (DDPG) [9], and prior Distributional RL approaches, such as C51 [6];
uncertainty estimator with quantile approximation that can
2) QuaDUE-CCM. quadrotor trajectory tracking framework that integrates distributional-Ru based uncertainty estimator into CCM- based nonlinear optimal control problem;
3) Theoretical understanding and mathematical proofs, i.c., the convergence guarantee and ac- celeration analysis are provided for the interpretability of the distributional RL.
Related Work
Quadrotor Uncertainty Modelling: Precise dynamics modeling in quadrotor autonomous naviga- tion is challenging, since a great variety of aerodynamic effects, such as unknown drag coefficients and wind gusts, can be generated by agile flight in high speeds and accelcrations. Data-driven ap- proaches, such as Gaussian Processes (GP) [I, 10] and neural networks [11] combined with Model Predictive Control (MPC), have been shown accurate modelling of aerodynamic effects. Due the nonparametric nature of GP, the GP-based approach hardly scale to the large datasets of complex environments [12]. While neural network-based approaches learn the nonlinear dynamical effects more accurately [13], achieving adaptability and robustness from these learning-based approaches are still ongoing challenges. One important reason is that these training datasets are collected from simulation and/or real-worke historical records, from which it is difficult to fully describe the com- plexity in their cnvironments [14].
Reinforcement Learning for uncertainty: Compared with the existing GP-based and neural- network-based approaches, RL, an adaptive and interactive learning approach, is introduced to model highly dynamic uncertainties in recent work [15]. Distributional RL constructs the entire distributions of the action-value function instead of the traditional expectation, where, some ex- tent, it addresses the key challenge of traditional RL, i.e., biasing the actions with high variance values in policy optimization [9]. Since some of these values will be overestimated by random chance [16], such actions should be avoided in risk-sensitive safety-critical applications such as autonomous quadrotor navigation. More importantly, recent work [17] shows that the distributional
RI. has better interpretability, where we can understand or even accelerate the entire RI training

---

process. This is an encouraging trend, especially for safety-critictt applications such as quadrotor autonomous navigation, towards closing the gup between theory and practice in distributional RL.
Control Certificate Techniques: Control certificate techniques guarantee the synthesis of con- trol policies and certificates [18], where the controller can optimize control policies whilst ensuring the satisfaction of the certificate properties. A Control Lyapunov Function (CLF)-based controller [19] ensures that the system state is Lyapunov stable, and a Control Barrier Function (CBF)-based controller [20] ensures that the system state is maintained in defined safety sets given by the bar- rier function. Contraction is a property of the closed-loop system [7]. [21] reformulates the CCM certificate as a Linear Matrix Inequality problem and solves using Bum-of-Squaree (SoS). However, the SoS- based approach cannot extend to general robot systems since an assumption is that the sys- tem dynamics need to be polynomial equations or can be approximated as polynomial equations. control framework, which extends neural net- works to certificate learning for contraction metrics. Moreover, based on the framework proposed by Tsukamoto et al., [8, 25, 26, 27, 28] are used to address higher dimensional control problems.
Then (22, 23, 24] propose contraction-metric-hee
3 Preliminaries and notations
Nominal Quadrotor Dynamic Model and its Control-affine Form:
the quadrotor is assumed as six Degrees of Freedom (DoF) rigid body of mass m, i.e., three linear motions and three angular motions [1]. Different from (29, 30], the aerodynamic effect (disturbance) e is integrated into the
quadrotor dynamic model as follows [10]:
PWB = VwB
VwB = gw + m
(QWB +c+
1 qwB - [(WW))wWB
Wp = J- wB Jwn)
where PWB, VWB and qw are the position, linear velocity and orientation expressed in the world frame, and wa is the angular velocity expressed in the body frame (more detailed definitions please see [10].. Then we reformulate Equation in its control-affine form: á = fxx efx) + g(z) u + w, and g : R" + Raxm are assumed by the standard as Lipschitz continuous [31, 8]. x P VwB, qw Wa]" E X, u = Ti E U and w E W are the state, input and additive uncertainty of the dynamic model, where T; is the thrust of the i-th (i E (0) motor, x S R", U S I. and W S Rn are compact sets as the state, input and uncertainty space, respectively. e is the collective thrust by VID-Fusion [32] the k-th timestamp. Given an input: 12 U and an initial state xo E X. our goad is to design a quadrotor tracking controller u such that the state trajectory x can track any reference state trajectory Øref (satisfied the quadrotor dynamic limits) under bounded uncertainty
Where f R" + R²
[0, 0, Σ T. eyk is the aerodynamic effect estimated
w. R220 W.
Control Contraction Certificates: most existing methods on quadrotor trajectory tracking have so far been demonstrated in stabilization, i.c., -ratttpint-trackknn controller [31, 33, 14]. These exxedpooint-rackiie controllers are relatively efficient in a low-speed and low-variance trajectory. However, to track high-speed and high-variance trajectory in agile flight, we need broader spec- ification [34] rather than Lyapunov guarantees that simply stabilize in a fixed point.
Contraction theory [7] gives an analysis on the convergence evolution between the pairs of close trajectories in nonlinear systems, i.e., incremental stability. When extended to a control-affine system, the change rate is defined as dá = A(x, u)ox + ((x)ou, where A(x, u) : dw
0f(a) +
dh; EE u dan BBx)) g(x), ) b, is the i-th vector of B, and u' is the i-th element of u. Just as CLF is an extension from the idea of Lyapunov function V(x) X + R, the contraction theory has been extended to a CCM M (xx) X + R" i.c. a distance measurement between neighboring trajec- tories (a contraction metric) that shrinks exponentially, for tracking controller design [35, 36, 37]. As the matrix-valued function (xx) denotes symmetric and continuously differentiable metric, 5x T M& defines a Riemannian manifold [38, 7]. If there exists M > O (uniformly positive defi- nite) and exponential convergence of dæ to O, the Riemannian infinitesimal length will converge to 0, then the system is contraction. We have the following theorem:
Theorem 1 (Contraction Metric [7, 34]): The Lie derivative 0,(-) of a matrix-valued func- 2M i'. In control-affine system, if tion M (Im) along vector Ra is O.M == Ei= Dzmy
3

---

Estiattr
Cumulative Return
Uncertainty QuoDuE
Equction
1100
Brd
Quadratic Programming
CCM-baased
-2000
CDPG
QuaDUE
Guearoter
100
200
300
400
500
630
700
800
900
1000
Episodes
Figure 2: The position of QuaDUE-CCM in the Quadrotor system:
Learning curves of RL algorithms. The simulated speed is set as 0.4.
Figure 3:
2AM < 0, then the inequality ||x(() &re S Re- |ax(0) o (0) with 4I 0, and À > 0 holds, where A A(ar, u) and B ((x) aM .t, sym(M) = M M and Øre a sequential reference trajectory. Therefore, we say the system is contraction. (Proof see Supple- mentary part A)
there exists: M + (Y + BK))
are defined above, and M +()))) E==1 8x,.
Overview of the Control Framework: the overall framework is shown in Fig. 2. The aim of this work is to design quadrotor controller, which combines wind estimation i.e., VIID-Fusion [32], for tracking the reference trajectory xiref(t) of the nominal model (Equation 1) under highly variable aerodynamic disturbances. We propose a CCM-based MPC, where the parameter 0 corresponds weights of neural network that estimates the uncertainty A and A the CCM dynamics. Combining with the neural network, a Distributional-RL Saaatttionna-R---aeeee Estimator, i.c., QuaDUE, is used to learn the uncertainties adaptively whilst satisfying the constraints of the contraction metric.
Distributional Reinforcement Learning for CCM
Quantil-approxximaton based Distribbutional--ennnfreedd Uncertainty Estimator: in policy evaluation setting, given 5-tuple Markov Decision Process [39]: MDP = (S, A, P.R.R. R, (where S, A, P. R and y E [0, 1] are the state spaces, action spaces, transition probability, immediate reward function and discount rate), distributional Bellman equation defines the state-action dis-
tribution Z and the Bellman operator T as [6, 40]: T a) 22 R(s, a) + 7Z(s', a'), where E 5 and a E A are the state and action vector. π is a deterministic policy under the policy evalua- tion. In policy control setting, based the quantile approximation [40], the distributional Bellman optimality operator T is defined as: TZ(s, a) _ R(s,a) + +Z(ss, argmaz EE (2 a'))) where P.R.
as) = = ~ Sq(a..)) E Zo is quantile distribution mapping one state-action pair (s, a) to uniform probability distribution supported on Za is the space of quantile distribution within N supporting quantiles. &z denotes Dirac with 2 R. Then the maximal form of the Wasserstein metric is proved to be a contraction in [6] as:
do (IIw, Z1 Iw, S doe (Zz, Zz)
where W,, p [1, oo] denotes the p- w-Wasserstein distance. Ilw, is quantile approximation under the minimal 1- I-Wasserstein distance W, The maximal form of the p- p-Wasserstein metric is defined
as dp == supW, W Zz). Equation 2 illustrates that, for a fixed policy æ. the Bellman operator T over value distribution Z1 and Z2 is contraction in the maximal form of the Wasserstein metric, which important support for the convergence guarantee below, i.c. Proposition 2 (Policy Evaluation).
Ritiibuutional-RL--aaaee Estimator for CCM Uncertainty: denoting M + sym(M + BK)) in Theorem 1 by C. (zt, Freff, Uref, Onomonal- metricc. where moominal-matris the parameter of the neural network in the contraction metric learning [8] for the nominal function Equation 1 with

---

e, = 0, then the truth Om under uncertainty becomes: Cm = -E== 0M 8x 2' Sy +
AM (x) A (x)u = C. + (x) + A
We employ an agent of QuaDUE, the detailed algorithm for which is given in Supplementary Al- gorithm 1, to estimate the uncertain terms in Om: 4 (x) and A (x). Therefore, an estimation Cm is constructed by: Cm Cm + M4 (z) + MM ( where Iu (2), M (x)) i.e., symmetric matrix is the output (action) of QuaDUE. 0, is again the neural network parameters. Then our goal of the Distributional-RL- --basdd Estimator i..., QuaDUE
M
clear: learn policy 11, 122 such that the estimation Cm as close as to the true value Cm- Notice here that we use the same neural network Opt for 1B,,,, (1)) and pd (x)). In Section 5 below, we will demonstrate our reasoning and give proof.
A framework of QuaDUE is illustrated in Fig. The QuaDUE learns policy which combines with the CCM uncertainties A (z), A (x) and other dynamic constraints. Then we design the reward function Re+1 (s, a, 0) Reontraction (s, a, 0) + Rerack (s, a), where @ O represents the neural network parameters. Thus, the reward function is defined in detail as:
Reontraction (s, a, 0)
We,1 [ml M (s) [e - ml))D(()
- Wo,3 (0m + 2AM IN
(3)
Rerack a) = (2)( (s) re ) H (x, Æreffe) - u, a) H,,,
where m. ii are hyper-parameters, H1 and H2 are positive definite matrices, and [A]ND is for penalizing positive definiteness where [A]ND 0 iff. A x O, and [A]ND 2 0 iff. A ł O.
These QuaDUE outputs, i.e., 1.. 1 and p.M. ( are then fed into the CCM constraints derived
from the nominal model. Specifically, the estimated Om is used as the best guess of Cm for MPC to satisfy the true constraints. The MPC formulation combining with CCM constraints are shown in Fig. 1, where the learned uncertainties are used to construct the contraction constraints of the MPC. Then the MPC is specified to quadratic optimization formulation and implemented using CasADi [41] and ACADOS [42].
Certified Learning on a Contraction Metric for the Nominal Model: as shown in Fig. certified learning controller proposed in [34] is constructed to learn a Contraction Metric for the nominal model (Equation
1). An expert control policy is employed by solving nonlinear MPC problem [10]. The contraction metric M and control policy is represented as two neural networks [8], where the output of the metric network and the metric achieved M sym(A). The parameters of the metric and policy networks are optimized simultaneously in the training process, where the loss function is: L Limetric + Lepoticy- = (1/Neraim) (Z, [ml (M]ND((())) +,[M + m/]ND((;;) + E,(mm + 2XM]nD(a; + (1/Ntrain) Σ: [4 Mexpert,ill
5 Properties of QuaDUE: Convergence and Acceleration analysis
In this section, the properties of the proposed DistrbutiinnllRL--aae Estimator i.e., QuaDUE are analyzed, including convergence guarantees and training acceleration analysis. In particular, we empirically verify a more stable and accelerated convergence process than traditional RL algorithms (shown in Section 6), and demonstrate theoretical understanding of the proposed QuaDUE.
Convergence Analysis of QuaDUE: Equation shows that the Bellman operator T is p-
contraction under the p- Wasserstein metric dp [6]. The Wasserstein distance W, between a dis- tribution Z and its Bellman update T can be minimized iteratively in Temporal Difference learn ing. Therefore, we present the convergence analysis of QuaDUE for Policy Evaluation and Policy Improvement, respectively.
Proposition 2 (Policy Evaluation [40, 14]): Given deterministic policy æ, quantile approximator Iw, and ZR a) = Iw, T Z a), the sequence Z (s, a) converges to unique fixed point
Zr under the maximal form of co- Wasserstein metric d See Supplementary part A for Proof.
Proposition& 3 (Policy Improvement [14]): Denoting an old policy by wold and new policy by Tnews there exists TE] [[[ a)) Knew (8, 2 B[Z(s, a)) Modd (8, a), Va and Va A. See Supple-
mentary part A for Proof.
5

---

Based on Propasition 2 and Proposition 3, we are ready to present Theorem 4 to demonstrate the convergence of QuaDUE.
Theorem (Convergence): Denoting the policy of the i-th policy improvement by x², there exists T + nT", + &c, and (s, (s) (8, Z B[Zx(s,a)) (s, Vs E S and Va E A. See
Supplementary part A for Proof.
Training Acceleration of QuaDUE: training acceleration is investigated for our QuaDUE in order to guarantee a more stable and predictable RL training process. For implementation execution, we use Kullback-Leibler (KL) divergence [43] instead of Wasserstein Metric. KL divergence is proven and implemented 'usable' in [6]. Thus we use KL divergence analyze the acceleration in RL training process, which is closer to the real implementation.
the histogram distributional loss, where p² (x) and 40 s.a are the true and approximated density function of Z(s, a). We denote (0) E (s..a)>p- [Jo(s, a)] is the expectation of Js. where Pr the generated distribution un- der the deterministic policy π. Since the unbiased gradient estimation of KL divergence, there exits (ea))>, Jo tween Pa 9: and 40" *.a A good degree of the approximation leads to small K, which then effects the training acceleration of the distribution RL [17], as shown in Theorem
Similar to [14],
denote Jo(s,a) = DK (90pp""""g)))))))))
(0 (0) ] kO², where we represent the approximation error be-
Next we define a stationary-poimt for the derivative of (o) [17]] if there exits (Or)]l < T (T E (0, 1)). the updated parameters Or after I steps is first-order 7-stationary point. Denote F(s) as the feature on each state and let the support of (s, a) have is partitions. Assuming [[((s]] < ( we immediately present Theorem 5 for the training acceleration.
Theorem (Training Acceleration): In the training process of the distribution RL (i.e., QuaDUE):
1) Let sampling steps T = 4kl2 E E, (60)/7², if there exists k S 0.2579/0?, the Ja optimized by stochastic gradient descend converges to a r-stationary point.
2) Let sampling steps T = (0)) / (,T?), if there exists x. > 0.257²/0², the Jo optimized by stochastic gradient descend will not converge r-stationary point. (Proof see Supplementary part
A)
Theorem 5 demonstrates two cases of training acceleration in our QuaDUE. In (1) of Theorem 5, the convergence of QuaDUE is accelerated where there is a low approximation error x between (x'( and 9.th In (2) of Theorem 5, a relatively larger 0.257 /o2 will give no guarantee for accelcration convergence. But if (1/T) EnD (()])) I 4kT² is satisfied, the distribu- tional RL still converges to the stationary optimization point. This understanding from theoretical analysis is consistent with our empirical experiment results in Section 6: when we add small external forces (disturbances) in the specific scenarios Fig. 4, the performance of the two distributional RL - i.e. 'QuaDUE-CCM' and 'QuaDRED-MPR verify this observation from simple Atari games.
is less than 'DDPG-CCM'. The cases in [40] also
6 Evaluation of Performance
To evaluate the performance of the proposed QuaDUE-CCM, we compare against the convergence performance of DDPG 19], C51-DDPG [6] in training process, the tracking performance of [14] under highly variable aerodynamic disturbances (uncertainties). The parameters of our QuaDUE-CCM are summarized in Table according the benchmark works (see [1, 14, 44]). The parameters We,1, We,2 and Wr,3 in Equa- tion 3 are set 1e le and 1e
GP-MPC [1, 10), DDPG-CCM and QuaDRED-
1e_$[]] [8]. The matrices Hi and Hg in Equation 3 are chosen as Hi
diag 2.5e - 2,255 2, 3, le ,2.5 -3 2.5e -3.5-, and 11, =
-3 2,255 -3 .2.5e -3, le Ie le 5) 1.25e 1.255 1.25e 1.25e 1.25e J), respectively [14].
we have two main training process: firstly, we learn a Contraction Metric for the nominal model (shown in Fig. 1), where both metric and policy networks are 2 hidden layers of 64 neurons and Nirain = Ied. Then we operate a training process by generating external forces in RotorS [45], where the programmable external forces are in the horizontal plane with range [-3,3] (m/s²). The training process has 1000 iterations where the quadrotor state is recorded at 16 H 2. The convergence curves of the training are illustrated in Fig. 3,
Comparative Performance of QuaDUE Training:

---

generated
Trajectory generated by COM without external distribances consiliration
conea
Trajectory penerated by QuaDue-CVMe with esternal disturbonces consideration
100
Dynamic external egg
Sampling Time 16Hz
Figure 4: The simulation scenario in RotorS: both reference trajectories with/without external forces are generated by Kino-JSS [10].
Figure 5: Specifie scenarios results: Wind esti- mation n, (expressed in body frame), position error (m) and derivatives of tracking error.
where we show the comparative training performance of DDPG, C51-DDPG and our QuaDUE. Our performance illustrates that the two distributional RL approaches, C51-DDPG and QuaDUE, outperform the traditional DDPG RL approach, whilst our proposed QuaDUE achieves the largest cumulative return. More importantly. our QuaDUE maintains the highest convergence speed, which again verifies the theoretical guarantees demonstrated Section 5.
Comparative Performance of QuaDUE-CCM in Variable Aerodynamic Disturbance: we eval- uate our proposed QuaDUE-CCM with variable aerodynamic forces generated in ZONE and ZONE 2 of the specific scenario, described in Fig. 4. We compare the tracking position er- rors and derivatives of tracking error with different and opposite heading aerodynamic forces with [0.0, 1.5, 0.0] m//² and [0.0, -1.0, 0.0] m/s² in ZONE 1 and ZONE 2, respectively. As shown Fig. 5, the three benchmarks are: 1) GP-MPC, a state-of-the-an trajectory tracking adaptive approach extending from RI RI-CBF-CLFFOO [31] to solve the CCM uncertainties along with a nonlinear MPC; 3) Qu-DRED-MPC, a novel learning- based MPC for uncertainties combining with distributional RL agent and nonlinear MPC [14].
method [1, 10]; 2) DDPG-CCM,
In Fig. 5, the two distributional-RA estimator-baus based frameworks (i.e., QuADRED-MPC and QuaDUE-CCM) react to the sudden acrodynamic effects more sufficiently than the traditional RL-based estimator (i.e., DDPG). Among the two distributional- based methods, the QuaDRED-MPC' tracking performance slightly better (accumulated error 6.73% less) than our proposed QuaDUE-CCM. This is mainly because the QuaDRED-MPC assesses the environment uncertainties and then directly feeds the uncertainty estimation into the nominal model while the QuaDUE-CCM is an indirect method required to satisfy the contraction metrics, which makes the tracking actions more conservative. However, based on has 20.7% bigger derivative of tracking error
Table 1: Parameters of QuaDUE-CCM
Parameters
Definition
Values 0.0015
la.
Learning rate of actor
lo.
Learning rate of critic
0.0015
Actor neural network: fully connected with two hidden layers (128 neurons per hidden layer)
Oa
Critic neural network: fully connected with two hidden layers (128 neurons per hidden layer)
D
Replay memory capacity
104
B
Batch size
256
Discount rate
0,9995
Training episodes
1000
T.
MPC Sampling period
50ms
N
Time steps
20
the contraction theory, our proposed QuaDUE- than QuaDRED-MPC and the contraction rate is Ce- with = 3.59 and X 1.046. This again means that, under variable uncertainties, our QuaDUE-CCM converges to a reference trajectory in a higher contraction rate (exponential contraction) both with theoretical (contraction theory) and empirical guarantees.

---

Table 2: Comparison of Trajectory Tracking under Programmatic External forces
Contra. Rate
Ex. Forces (m/s² )
Method
Succ. Rate
Time (s)
Err. (m)
(X/C)
GP-MPC
100%
23.54
3.44
0.097-0.63/66.99
Z1: [0, 0.5, 0.0]
DDPG + CCM
97.6%
24.11
3.82
0.285-1.01/3.54
Z2: [0, -0.5,
MPPC
95.3%
25.54
4.30
0.100-00.64/6.41
QuaDUE-CCM
95.1%
25.84
4.78
0.302-1.07/3.54
GP-MPC
69.8%
38.62
24.27
0.093-0.6116.55
Z1: [0, 2.5, 0.0]
DDPG + CCM
79.8%
33.10
18.62
0.240-0.92/3.83
Z2: [0, -2.5, 0.0]
QuaDRED-MPC
89.1%
29.57
14.93
0.089-0.58/6.49
QuaDUE-CCM
87.2%
30.19
15.49
0.278-1.01/3.63
GP-MPC
0%
Z1: [-3.0, 3.5, 0.0]
DDPG CCM
48.5%
43.64
27.24
0.257-0.81/3.85
Z2: [-2.0, -2.0, 0.0)
QuADRED-MPC
83.3%
36.40
17.66
0.079-0.53/6.69
QuaDUE-CCM
84.1%
36.45
17.13
0.265-0.9573.59
Then larger and more complex external forces, i.e., [-2.5, 2.5, 0.0] and [-3.0, 3.0, 0.0] (m/s²), generated in ZONE and ZONE 2, respectively, as shown in Fig. 4. In Table 2. we compare the success rate, operation time, accumulated tracking eπrо and contraction rate with the correspond- ing variable external forces in ZONE and ZONE 2. Our results show that RL--based methods are not always better, especially with relatively small external forces. For example, the success rate of RL-based methods (i.e., *DDPG + CCM', "QuaDRED-MMCC and 'QuaDUF-CCMM) lower than 'GP-MPC'. We also find that the two distributional-RL. -L-baase methods have close tracking errors, where in some cases "QuADREDMMPC' is sightly better than 'QuaDUECCMM" However, compared 'QuaDRED-MPC', our 'QuaDUE-CCM" achieves 302%, 312% and 335% improvements in con- traction rate.
7 Conclusion
this paper, we propose a precise and reliable trajectory tracking framework, QuaDUE-CCM, for quadrotors in dynamic and unknown environments with highly variable aerodynamic ef- fects. QuaDUE-CCM combines Distributional-RL--aal address large and variable uncertainties quadrotor tracking. A Quantil-apprxiimaaion-bbsss Distributional-rcinfforee Uncertainty Estimator, QuaDUE, is proposed to learn the uncertaintics of contraction metrics adaptively, where the convergence and acceleration in the RL training process are analyzed from theoretical perspectives. Based on the contraction theory. CCMs are used to guarantee exponential convergence any feasible reference trajectory, which significantly improve the reliability of our tracking systems. We empirically demonstrate that our proposed approach can track agile trajectories with at least 36.2% improvement under large aerodynamic effects, where experimental data can favourably verify our theoretical results.
uncertainty estimator and CCMs to
Limitation
As mentioned in Sections & 6, comparison with GP-MPC, our QuaDUE- CCM performs weakly under small disturbances. The potential reasons include QuaDUE-CCME indirectly variable uncer- tainties modelling and the relatively conservative outputs caused by its theoretical guarantees. Our ongoing work concerns incorporating direct uncertainty estimation, i.e., adding uncertainties into the controller directly, and certificate-based uncertainty estimation improve the performance un- der relatively small uncertainties. Another limitation is the computational complexity, where we only use 16 Hz for the CCM uncertainty estimation, whereas GP-MPC uses 50 Hz. To improve this, QuaDUE-CCM may be deployed dedicated hardware, for example, via FPGA implementation. Ongoing work concerns evaluating performance in real-world flight tests, during which our system will face greater variety of aerodynamic disturbances.

---

Acknowledgments
This work was partially supported by UKRI [grant number NE/TO1 1467/1].
References
[1] G. Torrente, E. Kaufmann, P. Föhn, and D. Scaramuzza. Data-driven mpe for quadrotors. IEEE Robotics and Automation Letters, 6(2):3769-3776, 2021.
[2] M. Faessler, A. Franchi, and D. Scaramuzza. Differential flatness of quadrotor dynamics sub- ject to rotor drag for accurate tracking of high-speed trajectories. IEEE Robotics and Automa- lion Letters, (00x6620-266 2017.
[3] G. Hoffmann, H. Huang. S. Waslander, and C. Tomlin. Quadrotor helicopter flight dynamics and control: Theory and experiment. In AIAA guidance, navigation and control conference and exhibit, page 6461, 2007.
[4] C. R. Russell, J. Jung, G. Willink, and B. Glasner. Wind tunnel and hover performance test re- sults for multicopter uas vehicles. In American Helicopter Society (AHS) International Annual Forum and Technology Display, number ARC-E-DAA-TN31096, 2016.
[5] D. Kaya and A. T. Kutay. Aerodynamic modeling and parameter estimation of a quadrotor helicopter. In AIAA Ammosheric Flight Mechanics Conference, page 2558, 2014.
[6] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective reinforcement learning. In International Conference on Machine Learning, pages 149-458. PMLR, 2017.
[7] W. Lohmiller and J.-J. E. Slotine. On contraction analysis for non-linear systems. Automatica, (4(6):683-666, 1998.
[8] C. Dawson, S. Gao, and C. Fan. Safe control with learned certificates: A survey of neural Iyapunov, barrier, and contraction methods. arXiv preprint arXiv:2202.1176; 2022.
[9] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 10XXv:1500..
2015.
[10] Y. Wang. J. O' Keeffe, Q. Qian, and D. Boyle. Kinojgm: A framework for efficient and accurate quadrotor trajectory generation and tracking in dynamic environments. In 2022 International Conference on Robotics and Automation (ICRA), pages 1006--10443 IEEE, 2022.
[1]] N. A. Spielberg, M. Brown, and J. C. Gerdes. Neural network model predictive motion control applied to automated driving with unknown friction. IEEE Transactions Control Systems Technology, 2021.
[12] Salzmann, E. Kaufmann, M. Pavone, Scaramuzza, and M. Ryll. Neural-mpe: Deep learning model predictive control for quadrotors and agile robotic platforms. arXiv preprint arXiv:2203.0m 07747,, 2022.
[13] A. Loquercio, E. Kaufmann, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza. Learning high-speed flight in the wild. Science Robotics, 6(59):ebbg5810, 2021.
[14] Y. Wang, J. O' Keeffe, Q. Qian, and D. Boyle. Interpretable stochastic model predictive con- trol using distributional reinforced estimation for quadrotor tracking systems. In 6/th IEEE International Conference on Decision and Control (CDC) [to appear]. IEEE, 2022.
[15] Q. Zhang, W. Pan, and V. Reppa. Medel-reference reinforcement learning for collision-fis tracking control of autonomous surface vehicles. IEEE Transactions Intelligent Transporta- tion Systems, 2021.
[16] Y. Ma, D. Jayaraman, and O. Bastani. Conservative offline distributional reinforcement learn- ing. Advances in Neural Information Processing Systems, 34, 2021.
[17] K. Sun, Y. Zhao, Y. Liu, E. Shi, Y. Wang, A. Sadeghi, X. Yan, B. Jiang, and L. Kong. Towards understanding distributional reinforcement learning: Regularization, optimization, accelera- tion and sinkhorn algorithm. arXiv preprint 2 2011.

---

[18] S. Jha, S. Raj, S. K. Jha, and N. Shankar. Duality-based nested controller synthesis from stl specifications for stochastic linear systems. In International Conference on Formal Modeling and Analysis of Timed Systems, pages 235-251. Springrr, 2018
[19] H. K. Khalil. Nonlinear control, volume 406. Pearson New York, 2015.
[20] A. D. Ames, X. Xu, J. W. Grizzle, and Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control, 62(8):3861- 3876, 2016.
[21] S. Singh, B. Landry, A. Majumdar, J.-J. Slotine, and M. Pavone. Robust feedback motion planning via contraction theory. The International Journal of Robotics Research, 2019.
[22] H. Tsukamoto and S.-J. Chung. Neural contraction metrics for robust estimation and control: convex optimization approach. IEEE Control Systems Letters, 5(1):211-216, 2020.
[23] H. Tsukamoto, S.-J Chung, and J.-J E. Slotine. Neural stochastic contraction metrics for learning-based control and estimation. IEEE Control Systems Letters, 5(5):1825-1880, 2020.
[24] G. Chou, N. Ozay, and D. Berenson. Model error propagation via learned contraction metrics for safe feedback motion planning of unknown systems. In 2021 60th IEEE Conference on Decision and Control (CDC), pages 3576-3583. IEEE, 2021.
[25] Z. Qin, K. Zhang, Y. Chen, J. Chen, and C. Fan. Learning safe multi-agent control with decentralized neural barrier certificates. arXiv preprint arXiv:2101.05433 2021.
[26] Z. Qin, D. Sun, and C. Fan. Sablas: Learning safe control for black-box dynamical systems. IEEE Robotics and Automation Letters, 7(2): 1928-19999, 2022.
[27] Y.-C. Chang and S. Gao. Stabilizing neural control using self-learned almost lyapunov critics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1803- 1809. IEEE, 2021.
[28] G. Chou, N. Ozay, and D. Berenson. Safe output feedback motion planning from images via learned perception modules and contraction theory. arXiv preprint arXiv:2206.06553 06553, 2022.
[29] M. Kamel, T. Stastny, K. Alexis, and R. Siegwart. Model predictive control for trajectory tracking of unmanned aerial vehicles using robot operating system. In Robot operating system (ROS), pages 3-39. Springer, 2017.
[30] D. Falanga, P. Fochn, Lu, and D. Scaramuzza. Pampe: Perception-awarr model predictive control for quadrotors. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1-8. IEEE, 2018
[31] J. Choi, F. Castañeda, C. J. Tomlin, and K. Sreenath. Reinforcement learning for safety- critical control under model uncertainty, using control Iyapunov functions and control barrier functions. In Robotics: Science and Systems (RSS), 2020.
[32] Z. Ding, Yang, Zhang, C. Xu, and Gao. Vid-fusion: Robust visual-inertial-dymame odometry for accurate external force estimation. arXiv preprint arXiv:2011.03993 2020.
[33] C. Dawson, B. Lowenkamp. Goff, and C. Fan. Learning safe, generalizable perception- based hybrid control with certificates. IEEE Robotics and Automation Letters, 2022.
[34] D. Sun, S. Jha, and C. Fan. Learning certified control using contraction metric. arXiv preprint arXiv:20/1. 12569, 2020.
[35] I. R. Manchester and J.-J. E. Slotine. Control contraction metrics: Convex and intrinsic criteria for nonlincar feedback design. IEEE Transactions on Automatic Control, 624663304,,,053, 2017.
[36] H. Tsukamoto and S.-J. Chung. Robust controller design for stochastic nonlinear systems via convex optimization. IEFE Transactions on Automatic Control, 66(10):4731-47 4746, 2020.
10

---

[37] P. Zhao, A. Lakshmanan, K. Ackerman, A. Gahlawat, M. Pavone, and N. Hovakimyan. Tube- certified trajectory tracking for nonlinear systems with robust control contraction metrics. IEEE Robotics and Automation Letters, 7(2):5528 25355 2022.
[38] S. Gallot, D. Hulin, and J. Lafontaine. Riemannian geometry, volume 2. Springer, 1990.
[39] M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[40] W. Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[41] J. A. Andersson, J. Gillis, G. Horn, J. B. Rawlings, and M. Diehl. Casadi: software frame- work for nonlinear optimization and optimal control. Mathematical Programming Computa- fion, 11(1):1-36, 2019.
[42] R. Verschueren, G. Frison, D. Kouzoupis, N. van Duijkeren, A. Zanelli, R. Quirynen, and M. Dichl. Towards a modular software package for embedded optimization. IFAC- PapersOnLine, 51(20):374-3888 2018.
[43] M. Joyce. Kullback-leibler divergence. In International encyclopedia of statistical science, pages 720-722. Springer, 2011.
[44] S. Zhang and H. Yao. Quota: The quantile option architecture for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5797-5804, 2019.
[45] F. Furrer, M. Burri, M. Achtelik, and R. Siegwart. Rotors-as modular gazebo mav simulator framework. Robot operating system (ROS), pages 595-625. Springer, 2016.


Reviews:
[{"review_id": "sr_4BEOYX6-", "reviewer": "Reviewer_6Poj", "summary": "This paper considers the problem of fly drones with wind. Neural networks are used to learn the effects of aerodynamic disturbances, which are the uncertainties between the true and estimated Control Contraction Metrics (CCMs). An RL agent is used to learn a policy, which computes a correction to the nominal contraction metric. The corrected contraction metric is then used in an MPC framework to construct the contraints of the MPC. It The approach is evaluated in simulation scenarios. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Theoretical results.\n2. Experiments in a simulation scenario.\n\nWeakness:\n1. No hardware experiments.\n2. Concerns about the design of the reward function. (See Issues)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although no hardware experiments were conducted, the authors presented strong theoretical results and experimental results in simulation.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1. The purpose of \\mu^M_1 and \\mu^M_2 is to make \\hat{C}_m as close as to the true value C_m. If I understand correctly, the third item in R_{contraction} is for this purpose. But I don't understand how it works. This item can only make sure that \\hat{C}_m \\prec - 2\\lambda M, but it does not imply that \\hat{C}_m is close to C_m.\n2. In line 137, it seems there should be a Z_{\\theta}."}}, {"review_id": "VxE2vDT4wBM", "reviewer": "Reviewer_VCFb", "summary": "This paper proposes: (1) QuaDUE, a distributional RL based uncertainty estimator that evaluates aerodynamic effects that are difficult to model directly; (2) QuaDUE-CCM, a quadrotor trajectory tracking system that incorporates QuaDUE into a MPC controller based on control contraction metric. Thorough theoretical analysis (in both the paper and the supplementary document) shows that the QuaDUE is guaranteed to converge, and training acceleration could be achieved when the approximation error of the learned distribution is low. In the experiment, it could be seen that within the shown learning period QuaDUE outperforms DDPG in terms of the learned cumulative return. In addition, compared to DDPG-CCM, QuaDUE-CCM exhibits higher success rates in trajectory tracking tasks when external forces in the simulated environment are large.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The theoretical analysis of convergence and training acceleration of QuaDUE demonstrates the method's capability with solid proof. The description of QuaDUE-CCM is comprehensive, and the supplementary document provides useful explanations about the quadrotor dynamic model, algorithm of kino-JSS and more details about the QuaDUE-CCM implementation.\n\nWeaknesses: It's not clear that the learning curves shown in Figure 3 converge, hence I think it's not quite convincing to view the result as an evidence of the convergence guarantee described in theoretical analysis. Besides, QuaDUE shows similar learning performance as C51-DDPG, and QuaDUE-CCM has similar trajectory tracking performance as QuaDRED-MPC. Thus, experiment results seem to illustrate that the advantage of distributional RL methods in uncertainty estimation, and the proposed algorithms show a common performance among them.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a novel distributional RL based uncertainty estimator, and extends it to a quadrotor trajectory tracking system that is based on a control contraction metric based controller. Detailed theoretical analysis with respect to the convergence and training acceleration property of the proposed estimator is given. Simulation experiment results show that the proposed tracking system has higher success rates in trajectory tracking tasks with large external force than GP-MPC, a state-of-the-art trajectory tracking method, and DDPG-CCM, a system built with traditional RL.\n\nPost-Rebuttal Comments: I appreciate the clarification provided by the authors, and I remain in favor of acceptance. Taking the authors' responses and the CoRL review criteria into account, I have decided to keep my rating of weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "How the nominal model is learned with the nominal function via contraction metric learning is not described in the paper, and I think it would be more clear to the audience if the methodology of contraction metric learning is included. Besides, in the experimental study of QuaDUE-CCM, some other method being compared include a nonlinear MPC controller, but no description about its formulation is given."}}, {"review_id": "N1y21vkC1q", "reviewer": "Reviewer_U1k2", "summary": "The authors propose the QuaDUE-CCM framework for robust trajectory tracking for quadrotors with highly variable uncertainties due to aerodynamic forces. The authors provide convergence guarantees as well as simulation results that demonstrate the effectiveness of the framework.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides good experimental results that compare QuaDUE-CCM against state-of-the-art methods as well as convergence guarantees thoroughly supported by proofs. Unfortunately, I was unable to follow most of the proofs. Personally, I think the paper would have been easier to follow if it focused more on formulation of the framework and guarantees, and moved the proofs to the appendix. A section referring to how QuaDUE-CCM applies to other systems would help the paper as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The experimental results and guarantees seem thorough. However, I found the paper very confusing, mainly due to my lack of expertise, but also due to the dense and hard to follow writing.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See above", "reviewer_expertise": "1: The reviewer's evaluation is an educated guess"}, "extra_text": {}}, {"review_id": "LHKYwTFQD_", "reviewer": "Reviewer_PQAG", "summary": "This paper proposes a method for estimating uncertainties in quadrotor dynamics using distributional reinforcement learning and control contraction metrics (CCM). In particular, distributional RL is used to estimate the disturbances that the system is subjected to, and this is used in a CCM-based trajectory tracking control framework. The approach is demonstrated in simulation, and is shown to improve tracking error relative to baseline algorithms.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is generally well-written, mathematically solid, and provides an interesting synthesis between distributional RL and control contraction metrics which may be of interest for future research.\n\nComments, concerns, suggestions, and questions:\n- There are some ambiguities in notation which sometime make it difficult to follow the math:\n    - It seems $\\lambda$ is overloaded in the proof of Theorem 5 (contraction rate vs. step size)?\n    - It is unclear how $\\Delta_1$, $\\Delta_2$ factor into (1) or into the control affine form $\\dot x = f(x) + B(x)u + w$.\n    - $e_{fk}$ and $T_i$ are not defined\n    - Lie derivative $\\partial_{(\\cdot)}$ is not defined\n    - Several variables in Theorem 5 are not defined\n- There is missing previous related work [1, 2] which also estimates worst-case disturbances in the context of learning-based motion planning to determine how they affect the tracking performance of a CCM-based controller.\n- Related to the last note, it would be good to compare the proposed distributional RL approach to a “robust CCM” baseline [1, 2, 3], which attempts to do feedback control assuming worst-case disturbance, and the improvements in conservativeness afforded by using a distributional approach.\n- How are $\\omega_{c,i}$ chosen in the reward function? I couldn’t find information on that in the main body.\n- In the QP in Figure 1, in general, it is difficult to guarantee a priori whether there exists a control input $u$ within the control limits $u_min$ and $u_max$ that satisfies the contraction condition, especially for a $\\hat C$ which is being learned. In practice, do you run into feasibility problems in the QP? More comments clarifying this in the text would be helpful.\n- In parts of the paper, to aid it would be good to have more intuition describing the math: it is unlikely that people are intimately familiar with both CCMs and distributional RL. For instance, before (2), you might describe what the “maximal form” is, and why it’s important that it has contractive properties, e.g., “the approximate Wasserstein distance between $Z_1$ and $Z_2$ shrinks upon applying the Bellman operator, which is important for convergence guarantees later on”.\n- It would be good to have more details on the training of M, u, and the selection of the contraction rate \\lambda.\n- It would help to move some of the QuaDUE-CCM algorithm details from the appendix into the main body - right now the algorithm is described rather vaguely. More generally, in the main body of the paper, please reference the proofs and additional relevant material in appendix.\n\n[1] Chou, Ozay, and Berenson. Model Error Propagation via Learned Contraction Metrics for Safe Feedback Motion Planning of Unknown Systems. CDC 2021.\n\n[2] Chou, Ozay, and Berenson. Safe Output Feedback Motion Planning from Images via Learned Perception Modules and Contraction Theory. WAFR 2022.\n\n[3] Singh, Landry, Majumdar, Slotine, and Pavone. Robust Feedback Motion Planning via Contraction Theory.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is generally well-written and is an interesting combination of CCMs and distributional RL, which hopefully may inform a future line of research -- thus, I recommend acceptance.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See the comments above.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)