Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes a learning-based controller for quadrotors that is robust to external disturbances such as unsteady wind fields. The learned controller is trained via model-free reinforcement learning in simulation, where it is exposed to random force disturbances while being tasked to track random trajectories. After training the controller for 2e7 environment steps, the controller is deployed zero-shot on a physical quadrotor. 

The presented method is compared against a differential flatness-based controller and a sampling-based model predictive controller on the task of trajectory tracking with and without external wind disturbances. The proposed learning-based controller outperforms both baselines.
review: Strengths
=========
- The paper is clearly written, reads well, and is easy to follow.
- The experiments are performed on a physical robot.
- The paper is very clear and transparent about the limitations of the presented work. 


Constructive Criticism
==================
- L16: UAV = *unmanned* aerial vehicle, not uninhabited aerial vehicle
- In the simulation, the proposed method is conditioned on ground-truth translational disturbance, while in the real world, the translational disturbance is estimated by an L1 adaptive controller. Is there a reason to not run the same L1 adaptive controller in simulation as well? Would doing so potentially reduce the sim2real gap?
- The proposed controller is able to reject external disturbances based on the information contained in the observed positional tracking error and the estimated force disturbance generated from the L1 adaptive controller. Previous work (ref [28] in the paper) proposed to use a history of state-action pair observations instead to achieve a similar effect. Would the proposed controller potentially benefit from such a history-based observation as well?
- Minor in Figure 1: always add a grid to every plot, it improves the reading experience.
- Minor: how does the performance of MPPI change with varying sample numbers?
- Minor thought to think about, does not require to be addressed: how much sense does it make to evaluate tracking performance on infeasible trajectories? Isn't such a metric ill-posed in the first place? Evaluating tracking performance on infeasible trajectories becomes extremely sensitive to controller tuning.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: My recommendation is based on the good presentation of the paper, the convincing set of baselines that the presented approach is evaluated against, and the reported performance of the method.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper is about precision trajectory tracking for quadrotors. It presents a learning-based approach that can track challenging trajectories in the presence of large disturbances. After RL training in simulation, the approach transfers to the real world. The experimental results are impressive. 
review: The work is really nice and clearly presented. No apparent weaknesses.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This a very nice paper and a pleasure to read. The algorithm is well explained, the results are convincing, and the demonstration on hardware is excellent.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes an algorithm for controlling a quadrotor to track arbitrary trajectories. Deep RL is used along with disturbance adaptation in real-time.  With respect to previous works, the algorithm can track infeasible trajectories (in the sense of non-differentiable ones) with better accuracy. Extensive simulation and experimental results are presented. 
review: Combining off-line learning and online adaptation (in a feedforward-feedback sense) seems a very promising technique for the control of dynamical systems.   
The concept is not entirely new, a recent work introduced a very similar concept with feedforward-feedback adaptive control: 
Verginis et al, 'Non-parametric neuro-adaptive control', ECC 2023, https://ieeexplore.ieee.org/document/10178288
and an association/comparison with it should be made. 

The paper is generally well-written, and the algorithm is verified extensively in simulation and experiments. 
I believe it mostly needs clarification in Section 3 regarding the proposed methodology, which is not completely clear.  
More specific comments follow below. 
 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I believe there is theoretical novelty in the paper and good experimental demonstrations. However, the paper needs to be revised, mostly on its presentation in the sense of clarification of the algorithm.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper proposes a control architecture for a quadrotor to track arbitrary (possibly non-feasible) trajectories subject to uncertainties using reinforcement learning. Two components are learned: one is a feedforward embedding h, which takes the inputs of the position error to future planned positions on the trajectory and projects it to a latent space; the other one is the feedback policy that takes in the state information, latent-space represented terms from the embedding h, and disturbance and outputs the control actions. Both the two components are trained via reinforcement learning in simulations. The online deployment uses the uncertainty estimation obtained via the L1 adaptive control. Experimental validations are provided to demonstrate the performance of the proposed architecture.
review: Below are my reviews of this paper

0. The paper is written well. Many questions while reading the paper are answered in the supplementary material. The experimental validations look good. 

1. This is an interesting paper that tries to build the latent space based on the position error to the desired future positions, which should carry useful information on whether the desired positions to follow next will be feasible. The idea of training with simulated (known) disturbance and deploying an online estimator of the disturbance for the actual system is not new: the Rapid Motor Adaptation (RMA) mechanism (reference [26]) has proposed a similar idea, and it worked well for quadrupedal and quadrotors. The authors should know that RMA has been applied to quadrotors [R1] and must compare the difference in methodology and experimental verifications.

2. The authors only compare the proposed method's performance with conventional non-learning controllers. However, given that neural networks (NNs) contain way more parameters than conventionally designed controllers, such a comparison can be an apple-to-orange comparison, which for sure the NN-parameterized controller will beat the conventional ones given the former's larger capabilities with more parameters. This is another reason why a comparison to the RMA [R1] should be conducted.

3. When referring to the L1 adaptive control literature, the authors may want to cite the guaranteed bounded disturbance error using the state predictor with the piecewise constant adaptation law, e.g., proposition 4 in [11] and lemma 3 in [R2].

4. In the experiments, it's fundamental to add unknown weights as a benchmark comparison (unknown mass induces unknown forces when the quadrotor tries to generate forces in the horizontal directions). Please consider adding this comparison.

5. In Figs. 5 and 6 in the appendix, the authors didn't show the tracking performance of the baseline L1 adaptive control method (only L1-MPC was shown), which was mentioned in Table 2. Please add the corresponding results.

Reference
[R1] Zhang, Dingqi, Antonio Loquercio, Xiangyu Wu, Ashish Kumar, Jitendra Malik, and Mark W. Mueller. "Learning a Single Near-hover Position Controller for Vastly Different Quadcopters." In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1263-1269. IEEE, 2023. 
[R2] Zhao, Pan, Yanbing Mao, Chuyuan Tao, Naira Hovakimyan, and Xiaofeng Wang. "Adaptive robust quadratic programs using control Lyapunov and barrier functions." In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 3353-3358. IEEE, 2020.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The recommendation is made based on the comments listed above. Please provide extra experimental results and supply more details to the questions above.

updated on Aug. 18: the reviewer's questions have been addressed.
review_title: null

Meta-review: ## Summary of Paper
The paper combines ideas from control and RL to come up with quadrotor controllers that are robust to external disturbances. The controllers are trained in simulation and extensively evaluated on a real quadrotor.

## Summary of Reviews
The reviewers found the paper to be well written and interesting. They had a few questions about the details of the method and requested stronger baselines.

## Influence of Rebuttal
With the extensive new results and the clarifications all concerns of the reviewers have been addressed.

## ToDos for Final Version
Please incorporate all the material from the rebuttal in the final version and also see comment by Reviewer fQDU on \sigma.
Decision: Accept (Oral)