Real-Time Generation of Time-Optimal Quadrotor Trajectories with Semi-Supervised Seq2Seq Learning
Authors: Gilhyun Ryou
Ezra Tal
Sertac Karaman
Venue: CoRL 2022 Poster (2022)
Keywords: Motion Planning
Model Learning

Abstract: Generating time-optimal quadrotor trajectories is challenging due to the complex dynamics of high-speed, agile flight. In this paper, we propose a data-driven method for real-time time-optimal trajectory generation that is suitable for complicated system models. We utilize a temporal deep neural network with sequence-to-sequence learning to find the optimal trajectories for sequences of a variable number of waypoints. The model is efficiently trained in a semi-supervised manner by combining supervised pretraining using a minimum-snap baseline method with Bayesian optimization and reinforcement learning. Compared to the baseline method, the trained model generates up to 20 % faster trajectories at an order of magnitude less computational cost. The optimized trajectories are evaluated in simulation and real-world flight experiments, where the improvement is further demonstrated.


Real-Time Generation of Time-Optimal Quadrotor Trajectories with Semi-Supervised Seq2Seq Learning
Gilhyun Ryou, Ezra Tal, and Sertac Karaman Laboratory for Information and Decision Systems (LIIS),
Massachusetts Institute of Technology,
Cambridge, Massachusetts 02139 [ghryou, eatal, pertac)@mit edu
Abstract: Generating time-optimal quadrotor trajectories is challenging due to the complex dynamics of high-speed, agile flight. In this paper, we propose data-driven method for real-time time-optimal trajectory generation that is suit- able for complicated system models. We utilize temporal deep neural network with sequence-to-sequence learning to find the optimal trajectories for sequences of variable number of waypoints. The model is efficiently trained in semi- supervised manner by combining supervised pretraining using minimum-snap baseline method with Bayesian optimization and reinforcement learning. Com- pared to the baseline method, the trained model generates up to 20 % faster trajec- tories at an order of magnitude less computational cost. The optimized trajectories are evaluated in simulation and real-world flight experiments, where the improve- ment further demonstrated.
Keywords: Motion planning, Model learning
1 Introduction
Generating a feasible minimum- -ime quadrotor trajectory, i.c., the fastest trajectory that can be flown accurately and reliably on the actual vehicle, is challenging due to the complex nonlinear quadro- tor dynamics. At high speeds, these dynamics include phenomena, such as unsteady aerodynam- ics, estimator error, and battery dynamics, that make it difficult to incorporate realistic feasibility bounds in trajectory generation. Popular methods address this difficulty by using simplified fea- sibility models [1, 2, 3] or by minimizing snap (i.e., the fourth temporal derivative of position) as a proxy for feasibility [4, 5]. Given a waypoint sequence and a time allocation over the segments between these waypoints, a corresponding polynomial minimum-snay trajectory can be obtained efficiently through convex optimization. However, finding the optimal, i.e., minimum-time and fea- sible, minimum-snap time allocation still amounts to a much harder nonlinear optimization problem.
Existing methods for finding (near-) joctinl time allocations suffer from downsides with regard to applicability, evaluation time, or their ability to incorporate realistic feasibility constraints. For example, while gradient descent (with without regularization) has been demonstrated with sim- plified dynamics [5. 6, 71, its large number of evaluations makes it computationally too expensive to consider a more realistic feasibility test. Additionally, the algorithm may not converge to a feasible solution altogether [8]. On the other hand, methods that are capable of incorporating complex real- istic feasibility constraints are typically not suitable for online motion planning. For instance, their learned feasibility representation may be specific to given sequence of waypoints [9].
In this paper, we address these shortcomings through a novel deep neural network model that gen- erates the optimal minimum-snan -nap time allocation for any given sequence of quadrotor waypoints. Specifically, we utilize temporal neural network model with ing to handle sequences with variable number of waypoints. Our work contains several contribu- tions. Firstly, we design the deep neural network trajectory generation model and train using a realistic six- faster trajectories than standard minimum-snen methods at fraction of the computational cost, mak- ing it suitable for high-rate online motion planning. Secondly, we propose novel training process
to-sequee (seq2seq) learn-
flight dynamics feasibility model. The resulting model generates
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Supervised Learning P1
Supervised Learning R2 Repeoser
Reinfrcement Learning MERR
2X0
220
Decoder
Waypssien Thes dlocctcttionn [4]
Sroottrmes seeghts 22
Thaining
(a) The proposed seq2seq model
(b) Average improvement over training phases.
Figure (a) The proposed model with fully-connected (FC) layers, bi-directional GRU encoder (ENC), and basic GRU decoder (DEC). Attention values are estimated from the context vector Cemes ie., all encoder hidden states, and the preceding hidden state. The model takes the variable-lengle waypoint sequence x as input and outputs the time allocation and the smoothness weights X as well the reconstructed waypoint sequence x5" (b) The three-phase training for forward-facing trajectory with differential flatness feasibility constraint.
that utilizes Bayesian optimization to enable supervised pretraining on a data subset. This pretrain- ing serves as guiding step that increases the efficiency of reinforcement learning by improving the sampling quality. Thirdly, we extend the minimum-snap optimization formulation by adding an individual snap penalty weight for each trajectory segment, enabling more aggressive minimum- snap trajectories. Fourthly, we demonstrate the improvement of the proposed model over standard minimum-snaa snap trajectory planning in both simulation and real-world flight experiments.
Preliminaries
2.1 Quadrotor Trajectory Planning
We study the problem of planning a time-optimal quadrotor trajectory that passes through m + given waypoints
-000 5²
pn Each waypoint ø' consists of prescribed position 87
and yaw angle pyy- A trajectory is continuous function that maps time position and yaw, i.e.,
p(() = [1) (t] The time-optimal planning problem can be formulated as follows:
minimize XEB
T
subject to p(0) =
= 8. (E)=1#;)=
i=1, m,
(1)
T T= Σ= Tie p PEPT.
where x = 21 Im is the time allocation over the segments between waypoints, and the function space Pr is the set of all feasible trajectories, ie., all trajectory functions that the quadrotor successfully track over [0, T). This set is defined by physical phenomena (e.g.. acrodynamics, thrust limitations, and vehicle mass and inertia), as well as hardware and software limitations of the estimation and control systems. Consequently, fast and agile motion planning is subject complex feasibility boundary that makes directly solving (1) challenging.
Widely used quadrotor motion planning algorithms employ minimum snap trajectory generation [4, 5], which does not directly consider the complex feasibility bounds. Instead, it minimizes the fourth- order derivative of position, i.e., snap, and the yaw acceleration to generate smooth trajectories that are less likely to violate feasibility constraints, as follows:
minimize σ (p. 1m) subject to p(0) =p' p (EE=1E3)) = i= .... m, PERm
(2)
where
T Hy
d2 Pib
op. T)
+
dt
(3)
d2t
with /4 and (e weighing parameters. By considering piecewise polynomial trajectories, the convex program (2) can be solved efficiently [5]. For convenience, denote the resulting minimum-snap

---

trajectory X(x. One way to obtain the time allocation x, which appears constraint in (2), is to first find the optimal allocation ratio
minimize x
(x((x,),)) subject to T
1 Li=
(4)
Since the optimal allocation ratio is independent of the duration T [5]. a uniform scaling factor can be used to find the the fastest trajectory that satisfies the feasibility constraint, i.e., X(kx, p) E Per [10]. This constraint can be evaluated in various ways, ranging from a simple check based on differential flatness of the idcalized quadrotor dynamics [4] to experimental evaluation in flight tests [9]. While the resulting time allocation kx on the feasibility boundary, is typically not time- optimal because there may exist different mion-minimmm -nap)) time allocation ratio that remains feasible for smaller k..
2.2 Bayesian Optimization
Bayesian optimization (BO) is class of machine learning algorithms efficiently solve optimiza- tion problems with black-box objective or constraint functions that are expensive to evaluate. The Gaussian process classifier (GPC) is widely used surrogate model to approximate constraint functions based on (noisy) evaluations [11]. Each point is selected as the maximum of the ac- quisition function ox (D). which balances uncertainty reduction in the surrogate model against the anticipated improvement in the objective function, given all data D obtained in previous evaluations. Multi-fidee Multi-fidelity Bayesian optimization (MFBO) extends the notion of BO to incorporate evaluations from various sources. The key idea is that combining cheap low-fidelity evaluation with expen- sive high-fidelity measurements improves overall efficiency, For instance, rough simulation or an expert's s opinion may serve as low-fidelity model, while high-accuracy simulation or real-world experiment serves as high-fidelity model.
Ryou et al. [9] utilize MFBO to find the minimum-time allocation x for quadrotor trajectory gen- eration. By efficiently combining evaluations from analytical models, numerical simulation, and real-world flight experiments, the algorithm finds faster trajectories that remain dynamically fea- sible on the actual vehicle, when compared to standard minimum-snap algorithms. However, the learned surrogate constraint model does not generalize beyond the predefined waypoint sequence p, making the algorithm unsuitable for online motion planning.
2.3 Deep Learning-based Approaches
Deep learning- based approaches have been applied in the field of quadrotor motion planning and control, e.g., to improve model predictive control parameters through deep reinforcement learn- ing [12, 13] and for neural network flight control [14, 15, 16, 17]. de Almeida et al. [8] apply deep learning to generate minimum-snap trajectories. They train a network model to generate the time allocation between the prescribed waypoints. By replacing the time-consuming nonlinear optimiza- tion (4) with neural network inference, an algorithm speed-up is achieved. However, since it uses a fully-connected network, the algorithm limited sequences with specific number of waypoints. Additionally, the model is trained only with supervised learning using labels from minimum- snap optimization, so it does not provide minimum-ting solutions.
Sequence-to-sequenee models take variable-length input sequence and transform it to an out- put sequence [18), which makes then particularly suitable for natural language processing appli- cations [19, 20, 21, 22, 23, 24, 25]. Seq2seq learning has also been applied towards various mo- tion planning problems, such predicting the future trajectory from past states and control in- puts [26, 27, 28, 29] and generating trajectories that are predictable for other agents, e.g., in multi- agent settings [30] and for socially-acceptable trajectory generation [31].
3 Algorithm
We propose seq2seq learning with a specialized training scheme to build model that generates the time-optimal trajectory for any given sequence of waypoints. By using minimum-snap trajectories, we obtain finite parameterization of the trajectory in terms of the time allocation over the segments between waypoints. Additionally, we introduce smoothness weights Xwi, leading to the following
3

---

snap objective (cf. (3))
Σ Twi (1mm
m
d
Pr
2
dl det
2 di
op, Xe: xx:
Mr
+ Mb
(5)
d'
Σ/= Ti m, Ei=1 Twi = m and X refers to the time allocated to the segment between the i-th and 1-th waypoints. The smoothness weights enable faster trajectories that remain feasible by allowing modification of the objective locally increase aggressiveness, according to
where 7; = EE= Jejs
minimize T, subject to T= =E X(X,, Xwx P) E Pr. X, R
(6)
Our goal is to build a model that outputs the optimal time allocation x; and smoothness weights X for any given waypoint sequence P with variable length. Solely using supervised learning to train the model is impractical, as obtaining the optimal time allocation and smoothness weights with regard to a realistic feasibility model (i.e., solving (6)) is computationally expensive, e.g., taking almost an hour for single set of waypoints using the BO approach from [9]. At the same time, reinforcement learning is inefficient and may fail, because randomly sampled time allocations and smoothness weights often fail to solve (5) altogether or require absurd paths with extreme excursions between waypoints. In order address these challenges, we propose a three three-phase training process consisting of pretraining, guiding, and generalizing. We first pretrain the model using supervised learning with minimum-snap labels obtained from (4), which not time-optimal but gives realistic trajectories. Next, we guide the model towards better solutions using optimal time allocation and smoothness weights obtained using BO [9] for subset of training dataset. The model exploits the progress from pretraining and guiding to efficiently sample in the final reinforcement learning phase, where the entire training dataset used to generalize the policy. In the remainder of this section we describe the dataset, the seq2seq model, and the three training phases. Additional details are given in the appendix.
3.1 Waypoint Sequences Dataset Generation
We create a dataset of sensible waypoint sequences by randomly sampling sequences in the unit cube 0.0.5, 0.5] and accepting samples based on two topological constraints: the summed Menger curvature [32] (i.e., the reciprocal of the radius of the circle that passes through a subsequence of three waypoints) and the summed Euclidean distance between subsequent waypoints, as follows:
Jeurvature
E (5, 20),
Idistance =
-E mh
(f E [0,30]. (7)
R(P,, .pr
2)+2
An initial minimum-smme snap trajectory obtained from (4) is used to reject waypoint sequences that result in trajectories that go beyond [-1, 1]s and to set the yaw component for each waypoint tangential to the local velocity. We unwrap the reference yaw, that the yaw difference between subsequent waypoints less than x rad. Finally, the actual waypoint positions are obtained by scaling with the desired space scale Lspace-
3.2 Sequence-to-Sequenec Model
The proposed seq2seq model, shown in Fig. 1a, consists of encoder, decoder, varia- tional autoencoder (VAE), and attention modules. Encoder input is normalized as z [Pt/ (Tsp(epp)))/)) cos( sin( SFos]. where fEos is indicator function for the final way- point in the sequence. The final hidden state of the bi-directional gated recurrent unit (GRU) en- coder serves as a feature vector that represents the waypoint sequence. Before it is passed to the decoder, the feature vector is densified by the VAE to prevent memorization behavior and force the model to summarize the waypoint sequence in a low-dimensional latent space [33]. Additionally, the hidden encoder states are used to generate content-based attention information that helps guide the decoder [19]. The feature vector and attention information are fed to the decoder network, which basic GRU [34].
The decoder output is passed through fully-connected neural network and then converted to the latent time allocation X;, the latent smoothness weights Ew. and the reconstructed waypoints x5"

---

through additional separate fully fulny-connected networks. Finally, a softmax (i.e., normalized exponen- tial) function is used to make sure that the output time allocation x, and the smoothness weights X are positíve. Motivated by examples where performance on primary task is improved by training on additional auxiliary tasks [35, 36, 37], we also train the model to minimize the reconstruction loss
LRcon = | xx""|| 1 of the waypoints. The similarity in feature space between the waypoints and the time allocation and smoothness weights enables the reconstruction loss minimization to guide the training on the primary tasks and to articulate the latent feature space. Indeed, we observed that adding this auxiliary task improves the stability of the training process.
3.3 Pretraining: Supervised Learning with Snap Minimization
In the first learning phase, the seq2seq model is pretrained using (4), While the resulting time allocations are not time-optimal, they provide a decent starting point for the learning process. Before generating the time allocations. we perform a grid search to find the penalty weights fr, and py that result in the fastest feasible trajectories on subset of the waypoint sequences dataset. In order to further reduce trajectory times, we considered adding weighted regularization term [5, 38]. However, through a grid search using a data subset, we found that there was no constant weight that improves the average performance, i.e., that did not also lead to many slower trajectories.
Based on the minimum-snap time allocations x; the seq2seq model is trained by minimizing
(8) where CELBO is the evidence lower bound (ELBO) loss from the VAE, and X₃ and X are the poftmax-normallne outputs of the seq2seq model. In effect, (4) uses uniform smoothness weights, but training the model with constant MM converges to zero neural network weights because the soft- max function normalizes the zero output
L LRecon + LELBO Xx 1ee-xxMs1 x1 + ||>u
MS
uniform X.. In order to avoid this behavior-whieh causes issues in the next learning phase because all gradients will be zero werrowwwaae set X to the inverse Ms ) Intuitively, these weights prioritize snap minimization on segments with small time allocation, which makes sense because these segments are flown faster on average and often require relatively increased smoothness to remain feasible.
of the time allocation, i.e., MS = m n2,z1 of
3.4 Guiding: Supervised Learning with Bayesian Optimization
In the second training phase, we improve the model by using time-optimal time allocations and smoothness weights obtained for subset of the dataset by solving (6) with BO. The BO algorithm initialized using the solution from (4) and efficiently samples the 2m-dimensional space of time allocations and smoothness weights to build surrogate model of the feasibility constraint Pr [9]. We further improve efficiency by combining analytical and numerical feasibility evaluations us- ing multi-fidelity BO, as described in Section 4.2. Based on the optimal time allocations x and smoothness weights xBO obtained using MFBO for a portion of waypoint sequenees dataset, the seq2seq model is trained by minimizing the following loss function:
L = LRssnn + LELBO + ||xx - x 2001 + ||xB
BO ||
which has similar form as (8).
3.5 Generalizing: Reinforcement Learning
In the third and final phase, we use reinforcement learning to generalize the seq2seq model that was overfitted to the BO data subset. A Markov decision process is formulated with the preceding waypoint 2 1 as the current state s; and the latent time allocation ετ,5 and latent smoothness weight uw, as the action ag At the end of each episode, consisting of variable-length sequence of mn+1 1 waypoints, the minimum feasible trajectory duration T x for the normalized time allocation x, and smoothness weights x computed. The episode reward is the improvement of the trajectory time based on and Xw,, compared to the baseline TMs:
r r(x,, x = 1 - Tx, TMs-
(10)
Since the reward is only obtained a the end of the episode, we estimate the discounted reward the i-th step, as = Ei 7 (rsr-r(s))))) aa) = (x,, xw), where 7 E [0, 1] is the discount factor.
5

---

We use the REINFORCE algorithm [39] to find the policy Th that maximizes the expected reward, i... T arg max, Eag... (n
(ā,, a. During exploration, the action
N( Frs) is sampled from the normal distribution around the model output aş. The objective function La- = > ri log To (a;j aa)) is estimated with batches of Nonch sequences, as follows:
Nexch
Lg
Nbach
1=1 (re rb) log T (ā,))a)))
(11)
k=1
Nuaa E ry subtracted to reduce reward variance. where the baseline reward 7ь Nbatch Σn
Moreover, all rewards are clipped to 1-0.5, 0.5] for further variance reduction. The objective func- tion (11) minimized together with the reconstruction loss and the VAE variational loss, i.e.,
L = LRcon + LELBO + Le.
(12)
Experimental Results
In order to validate the performance of our proposed algorithm, we trained the seq2seq model, eval- uated the trajectories generated by the model, and tested these trajectories flight experiments. Specifically, we trained two types of models: (i) using analytical feasibility evaluations based on the reference motor speeds obtained from the simplified quadrotor dynamics differential flatness transform, and (ii) using numerical evaluations of trajectory-tracking accuracy in six-degree-of- freedom (6DOF) flight dynamies simulation. All training data consists of sequences of five to four- teen waypoints that are scaled to be used in a 10 x 10 m 10 space, ie.., Lipace 10 Additional implementation details and experimental results can be found the appendix and at btttps://youte baaaatuu.eee. be/ be/1CD6UHBG7AY.
4.1 Differential Flatness Feasibility Constraint
We define the feasible set Pr all trajectories that can be flown without violating the admissible motor speeds [0, 2200] rad/s according to idealized quadrotor dynamics differential flatness trans- form from [4]. Based on this set, train models: one that considers only waypoint positions and not yaw (i.e., constant yaw at zero), and one for tangential (i.e., forward-facing) yaw refer- ence (see Section 3.1). The pretraining and generalization stages were each terminated after 150 training epochs based on convergence of the training loss and the intermediate guiding stage was terminated after 1000 epochs when the performance gap between the training and validation data reached a set threshold. The BO stage requires more training epochs because it uses much smaller training dataset than the other stages. Figure 1b shows the improvement over the three training stages for the model with forward-facing yaw reference. It can be seen that during the first stage the model approaches the performance of the minimum-snap labels. The BO data subset used in the second learning stage has superior labels, due to the more sophisticated optimization of (6). Overfitting to this dataset initially results in decreased performance on the validation dataset, but improves efficiency by guiding the RL algorithm. In the final stage, the model is generalized to con- sistent performance on the validation data. Moreover, Table I shows that all fully-trained models actually outperform the BO labels that were obtained using a limited number of iterations to curb computational cost.
(a) 95th percentile (14.94 % improvement)
(b) 75th percentile (8.88 improvement)
Figure 2: Trajectories obtained using minimum-snap optimization (top row) and the proposed seq2seq model (bottom row) with reference motor speeds.
With an average inference time of 0.01 s, our model is able to generate trajectories much faster than minimum-snap optimization algorithms, which require at least several seconds to solve the

---

Table 1: Trajectory time reduction relative minimum -nap trajectories for BO labels obtained using BayesOpt on (6), and for the seq2seq model after Phase 2 and Phase 3 of learning. BO set refers to the data subset with BO labels; Val. set refers to the validation dataset.
Feasibility constraint
Differential flatness
6DOF Simulation
Yaw reference
Constant
Forward-facing
Forward-facing
BO labels
5,375 %
4.746 %
5,552 %
Phase 2 (BO)
BO set
Val. set
3.884 %
-2.682 %
4.178 %
-2.368 %
5.048 %
-2.582 %
Phase 3 (RI.)
BO set
Val. set
6.971 %
6.638 %
6.191 %
6.031 %
7.169 %
7.291 %
Table 2: Trajectory time reduction by the seq2seq (s2s) model relative to minimum-snan snap (MS) trajectories for the validation data set with forward-facing yaw and using the differential flatness feasibility constraints. The bottom four rows contain data for the sample (i.c., the waypoint sequence and corresponding trajectory) the percentile rank.
Percentile
95th
75th
7th
1st -3.79 %
Time reduction
14.94 %
8.88 %
1.08 %
Waypoints
13
12
11
7
Duration [s]
MS
22.8
15.8
15.9
9.1
Top speed [mn/s]
MS s2s
6.7
8.3
8.8
8.6
6.9
7.7
9.1
8.4
Length [m]
MS
s2s
83.4
87.8
71.9
67.1
62.1
67.7
38.5
39.5
Smoothness cost
(s2s / MS)
1.6
0.8
1.2
1.2
nonconvex optimization (4). As shown in Table 2, the trained model also outperforms minimum- snap in terms of trajectory time on 93 % of the validation dataset. The improvement is greatest for trajectories with a large number of waypoints, where the seq2seq model has most options to modify the trajectory. The bottom row of Table shows that the seq2seq trajectories have higher smoothness cost than those from the minimum-sna method, i.c., they are less smooth and more aggressive. In practice, the proposed model generates lengthier paths that can be flown at higher speed or lowers the flight speed to allow tighter turns, as shown in respectively Fig. 2a and Fig. 2b.
4.2 6DOF Simulation Feasibility Constraint
We now consider the feasible set Py that contains all trajectories that can be flown with less than 20 cm position tracking error and deg yaw tracking error based on 6DOF flight dynamics and inertial measurement [40] and flight control [41] simulation. During the pretraining and RL stages, simulation is used for all feasibility evaluations, whereas it is combined with low-fidelity differen- tial flatness evaluations during MFBO label generation. Training in simulation enables our model to consider more realistic feasibility constraints that incorporate vehicle aerodynamics, sensor noise, control system limitations etc. It also permits increasingly aggressive trajectories, including motor saturation, provided that an acceptable tracking error can be maintained. Consequently, our seq2seq model further exploits the vehicle capabilities, leading to faster trajectories. While the first two learning stages are again terminated after respectively 150 and 1000 epochs, we only perform three RL epochs in the final stage due to the increased computational cost of improvement evaluation. De- spite this reduced RL training phase, our model for forward-facing yaw trajectories still outperforms minimum- snap trajectory generation on 87 % of the dataset with overall average improvement of 7.3 %, as shown in Table 1.
We randomly select ten waypoint sequences with corresponding minimum-snap and seq2seq trajec- torics. Over a series of flight tests, we uniformly scale the corresponding time allocations until the aforementioned tracking eπror constraints active. As shown in Table 3, the seq2seq time allo- cation ratios also result in quicker real-world trajectories. Figure 3a compares an example seq2seq trajectory with the corresponding minimum-snap trajectory. While the trajectory from our model has a lower top speed, has tighter, more aggressive turns that utilize more of the motor speed capacity, leading to a 22 % reduction in trajectory duration.
In addition, we choose two sequences that contain several waypoints linked to moving obstacles, as shown in Figure 3b. The proposed Seq2Seq model is evaluated by updating the trajectory in real-time as the locations of these waypoints change. When compared to a real-time baseline, which reuses the initial trajectory's minimum-snap time allocation, the proposed model produces more
7

---

compact trajectories with shorter flight times. Detailed experimental results are available in the accompanying video.
Table 3: Length and duration minimum-snap (MS) trajectories with simulation (sim) and real-
world (exp) feasibility constraints, as well as duration improvements by seq2seq (s2s) model.
Trajectory
3
4
6
7
9
10
Length [mn]
MS
sim
26.5
30.5
30.3
44.0
47.3
41.3
52.7
66.1
50.1
70.4
Duration [s]
MS sim
7.2
9.7
9.3
13.2
13.0
12.9
18.4
18.9
16.4
21.5
Imp. [%]
s2s sim
11.9
13.7
9.4
13.5
11.3
8.1
12.0
8.6
Duration [s]
MS
exp
5.8
8.4
10.6
12.7
11.7
13.6
15.8
16.2
20.2
Imp. [%)]
s2s
exp
4.3
1.3
10.5
10.4
4.2
14.0
9.2
5.2
8.9
22.1
(a) Traj. 10 (22.11 % woaldworld improvement..
(b) Online trajectory adaptation.
Figure 3: Trajectories obtained using minimum-snap optimization (left, top row) and the proposed seq2seq model (left, bottom row) with reference motor speeds, and real-time trajectory generation through moving obstacles (right).
Conclusion
5.1
Contributions
We presented a seq2seq model that generates time-optimal quadrotor trajectories from waypoint sequences. In order to train the model, we introduced novel three-stage learning process that combines the advantages of supervised learning, Bayesian optimization, and reinforcement learning- The resulting model outperforms minimum snap trajectory generation in both computation time and time-optimality of the generated trajectories in simulation and real-world flight experiments.
5.2 Limitations
The main limitation of the proposed method its lack theoretical performance guarantees. While the trained model outperforms minimum-snan trajectory generation for the vast majority of the val- idation dataset, it generates slower trajectories for small subset of waypoint sequences, e.g., the edge case where all but waypoints are located very closely together. We expect that utilizing ensemble modeling, i.e., training several models with varying parameters, or increasing the size of the data subset labeled with BO may relieve this issue.
Additionally, the learning process computationally expensive. Using multi-core implementation on powerful desktop computer, the complete process of data labeling and training described in Section 4 took on the order several weeks for the differential flatness feasibility constraints and on the order of several months for the 6DOF simulation. In particular, the costly computation of RL rewards with the 6DOF simulation may be sped up by using a multi-fidelity formulation that incorporates cheap low-fidelity evaluations from the differential flatness feasibility model, similar our approach for BO.
Finally, our model still needs some modifications to make it suitable for online motion planning. While it can generate trajectories in real time, currently limited to waypoint sequences that start and end in static hover. Our future work includes extending the seq2seq model to allow dynamic start and end points with nonzero velocity, yaw rate, acceleration etc., and deploying the model on an embedded system for online motion planning.

---

Acknowledgments
This work was partly supported by the Army Research Office through grant W911NF1910322.
References
[1] Borrelli, D. Subramanian, A. U. Raghunathan, and L. T. Biegler. Milp and nip techniques for centralized trajectory planning of multiple unmanned air vehicles. In 2006 American Control Conference, pages 6-pp. IEEE, 2006.
[2] W. Sun, G. Tang, and K. Hauser. Fast uav trajectory optimization using bilevel optimization with analytical gradients. In 2020 American Control Conference (ACC), pages 82-87. IEEE, 2020.
[3] J. Tordesillas, B. T. Lopez, and J. P. How. Faster: Fast and safe trajectory planner for flights in unknown environments. In 2019 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 1934-1940. IEEE, 2019.
[4] D. Mellinger and V. Kumar. Minimum snap trajectory generation and control for quadrotors. In 2011 IEEE International Conference on Roboties and Automation, pages 5520-2525.. IEEE,
2011.
[5] C. Richter, A. Bry, and N. Roy. Polynomial trajectory planning for aggressive quadrotor flight dense indoor environments. In Robotics Reseurch, pages 649-666. Springer, 2016.
[6] M. Cutler and How. Actuator constrained trajectory generation and control for pariable-pitch quadrotors. In AIAA Guidance, Navigation, and Control Conference, page 4777, 2012.
[7] B. Landry, R. Deits, P. R. Florence, and R. Tedrake. Aggressive quadrotor flight through clut- tered environments using mixed integer programming. In 2016 IEEE international conference on robotics and automation (ICRA), pages 1469-1475, IEEE, 2016.
[8] M. M. de Almeida, R. Moghe, and M. Akella. Real-time minimum snap trajectory genera- tion for quadcopters: Algorithm speed-up through machine learning. 2019 International Conference Robotics and Automation (ICRA), pages 683-689, IEEE, 2019.
[9] G. Ryou, E. Tal, and S. Karaman. Multi-fidelity black-box optimization for time-optimal quadrotor maneuvers. The International Journal of Robotics Research, 40(12-114;1152--13355 2021.
[10] A. Bry, C. Richter, A. Bachrach, and N. Roy. Aggressive flight of fixed-wing and quadrotor aircraft in dense indoor environments. The International Journal of Robotics Research, 34(7): 969-1002, 2015.
[11] H. Nickisch and C. E. Rasmussen. Approximations for binary Gaussian process classification. Journal of Machine Learning Research, (0ct):203552077, 2008.
[12] T. Zhang, G. Kahn, S. Levine, and P. Abbeel. Learning deep control policies for autonomous aerial vehicles with mpe-guided policy search. In 2016 IEEE international conference on robotics and automation (ICRA), pages 528-535. IEEE, 2016.
[13] Y. Song and D. Scaramuzza. Policy search for model predictive control with application to agile drone flight. IEEE Transactions on Robotics, 2022.
[14] Hwangbo, I. Sa, R. Siegwart, and M. Hutter. Control of a quadrotor with reinforcement learning. IEEE Robotics and Automation Letters, 2(4):2096-2103 2017.
[15) N. O. Lambert, D. S. Drew, J. Yaconelli, S. Levine, R. Calandra, and K. S. Pister. Low-level control of quadrotor with deep model-based reinforcement learning. IEEE Robotics and Automation Letters, 4(4):4224 4230, 2019.
[16] Y. Song, M. Steinweg, E. Kaufmann, and D. Scaramuzza. Autonomous drone racing with deep reinforcement learning. In 202/ JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1212-1212 IEEE, 2021.

---

[17] G. Shi, W. Hönig, Y. Yue, and S.-J. Chung. Neural-swarm: Decentralized close- proximity mul- tirotor control using learned interactions. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3241-3247. IEFE, 2020.
[18] I. Sutskever, O Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
[19] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 1eXiv:1409.0477 2014.
[20] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: 600.60090044,,.. 2016.
[21] A. Graves, A.T Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6645-6649. leee, 2013.
[22] A. M. Rush, S. Chopra, and J. Weston, A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:/509.000 2015.
[23] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio. End-to-end attention based large vocabulary speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4945-4949. IEEE, 2016.
[24] L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville. Describing videos by exploiting temporal structure. In Proceedings of the IEEE international conference on computer vision, pages 4507-4515, 2015.
[25] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko. Sequence to sequence-video to text. In Proceedings of the IEEE international conference on computer vision, pages 4534-4542, 2015.
[26] N. Lee, W. Choi, P. Vernaza, C. B. Choy, P. H. Torr, and M. Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 336 336-345, 2017.
[27] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-feasiin trajectory forecasting with heterogeneous data. In European Conference on Computer Vision, pages 683-700. Springer, 2020.
[28] B. Ivanovic, K. Leung, E. Schmerling, and M. Pavone. Multimodal deep generative models for trajectory prediction: conditional variational autoencoder approach. IEEE Robotics and Automation Letters, 6(2):295-302, 2020.
[29] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-971, 2016.
[30] Y. Yuan, X. Weng, Y. Ou, and K. M. Kitani. Agentformer: Agent-aware transformers for socio- temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 98 3-98233, 2021.
[31] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2255-2264, 2018.
[32] K. Menger. Untersuchungen über allgemeine metrik. vierte untersuchung, zur metrik der kur- ven. Mathematische Annalen, 103:466-500 5016 1930.
[33] A. M. Dai and Q V. Le. Semi-supervised sequence learning. Advances in neural information processing systems, 28, 2015.
[34] K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches.. arXiv preprint arXiv: 1259, 2014.
10

---

[35] V. Guizilini, R. Ambrus, W. Burgard, and A. Gaidon. Sparse auxiliary networks for unified monocular depth prediction and completion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11078-11088, 2021.
[36] A. Valada, N. Radwan, and W. Burgard. Deep auxiliary learning for visual localization and odometry. In 2018 IEEE international conference on robotics and automation (ICRA), pages 9939-6946 IEEE, 2018.
[37] A. Nediyanchath, P. Paramasivam, and P. Yenigalla. Multi-head attention for speech emotion recognition with auxiliary learning of gender recognition. In ICASSP 2020-2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7179-7183.
IEEE, 2020.
[38] M. Burri, H. Oleynikova, M. W. Achtelik, and R. Siegwart. Real-time visual- innttil mapping, le-loaalization and planning onboard mavs in unknown environments. In 2015 JEEE/RS/ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 1872-1878. IEEE, 2015.
[39] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, ((3):229-256, 1992.
[40] W. Guerra, E. Tal, V. Murali, G. Ryou, and S. Karaman. FlightGoggles: Photorealistic sen- sor simulation for perception-driven robotics using photogrammetry and virtual reality. arXiv preprint arXiv: 1905. 13377, 2019.
[41] E. Tal and Karaman. Accurate tracking of aggressive quadrotor trajectories using incre- mental nonlinear dynamic inversion and differential flatness. IEEE Transactions on Control Systems Technology, 2203)1203-1 2020.


Reviews:
[{"review_id": "A7_HSKNge-Q", "reviewer": "Reviewer_7heK", "summary": "The work presents an approach for real time inference of near-optimal time allocation for trajectory segments of minimum snap trajectories for quadrotors. To this end, the work utilizes a seq2seq model that is trained in three stages i) imitation learning from suboptimal trajectories ii) Bayesian Optimization using a smaller portion of optimal trajectories iii) RL refinement. To demonstrate the quality of the proposed method, the trajectories generated with this method are tested in simulation and on a real quadrotor platform.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Praise:\n* The work follows a generally interesting avenue of real time trajectory generation for very computation constrained platforms, such as drones.\n\n* The work demonstrates results using a real quadrotor platform.\n\n\nConcerns:\n   * To be honest I am not sure what specifically is called a “trajectory” in the context of this work, i.e. what parameters of the trajectory are minimized, what is trajectory representation? It does seem that the output of the seq2seq model is simply time allocations and smoothness weights, but how is trajectory itself generated, i.e. polynomial parameters or maybe quadrotor thrusts/velocity commands? \n\n   * What are the assumptions in this work? From what I understand the work assumes not only knowledge of the full dynamics model of the quadrotor, but also presence of a tracking controller that can go between provided trajectory key points, given the budget of time. The combination of both seems to constitute the feasibility set P_T.\n \n   * The method is rather complex using a 3 stage training procedure and 3 different approaches, which is definitely error-prone and, likely, introduces lots of tuning and hyperparameters. Modern RL methods under similar assumptions can solve for direct quadrotor commands (with implicit planning) and do it in hours (worst case days), not 4 weeks as stated in the Limitation section while showing very compelling results for various tasks in quadrotor flight (see [5] below just as an example).\n\n   * On the other hand,  work [1] presents seemingly strong baseline for simultaneous planning and tracking (there is generally a larger body of work from the same lab for time-optimal, high-speed robust quadrotor flight). It surely uses simplified assumptions on the planning part, yet demonstrates high performance even beating human pilots. The work under consideration does not seem to have an advantage here since it also lacks specific guarantees (it uses NN trained through BC and RL) and, I believe, relies on the accuracy of the model. Hence, it would be interesting to have some comparison or strong justification why such comparison is not appropriate.\n\n   * It is not completely clear why the authors need smoothness weights and time allocation at the same time. Both represent just normalized (over the entire trajectory) scaling of the cost per segment, i.e. two scaling parameters seem redundant.\n\n   * Literature overview is not complete, please, include (at least) these works with explanation of differences/advantages of your work:\n      * [1] Angel et al. Model Predictive Contouring Control for Time-Optimal Quadrotor Flight. \n      * [2] Panicka et al. Learning Minimum-Time Flight in Cluttered Environments.\n      * [3] Romero et al. Time-Optimal Online Replanning for Agile Quadrotor Flight.\n      * [4] Loquercio et al. Learning High-Speed Flight in the Wild.\n      * [5] Molchanov et al. Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "As stated in previous sections, I have problems with understanding the significance of the potential impact this work can make, especially considering missed references in the related work section.\nI am, clearly, not an expert in the area of planning for minimum snap quadrotor trajectories, hence, maybe, after some strong arguments I might change my opinion. But, for now, I am inclined toward rejection of the paper. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See Strengths & Weaknesses ", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "5_-11Spdsv6", "reviewer": "Reviewer_SHLP", "summary": "1. This paper proposes a data-driven method for time-optimal trajectory generation for quadrotors.\n2. Supervised pretraining and bayesian optizatimization & reinforcement learning are combined, which is interesting. The three stage training scheme is one of the core contribution.\n3. Superior perfromance is validated both by simulation and onboard tests. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. Three-stage training scheme is interesting.\n2. Modeling min-time trajectory generation problem as a seq2seq learning problem should be a good contribution to the community.\n\n### Weaknesses\n1. **The literature review and experimental comparisons are not sufficient**. The authors may consider comparing your methods with the following: 1) \"Optimal Time Allocation for Quadrotor Trajectory Generation\" by Fei Gao et al; 2) Time-Optimal Planning for Quadrotor Waypoint Flight by David Scaramuzza et al; They are both open-sourced and highly related to your topic. Minimum snap (MS) is now a relatively weak baseline and 20% faster than MS may not be a very supersing achievement.\n\n2. **Why using BO for the labeling of the sceond stage?**. BO may be too slow for this task and may limit the scale of the dataset for the second stage. The authors may consider using stronger methods as listed in 1. It can also be expected that the performance of the proposed pipeline is affected by the scale of the dataset for the second stage. A stronger method for the second stage may lead to a much stronger performance.\n\n3. **Still some gap to online motion planning** The authors do not incorporate the proposed method in a planning system in the wild. It will be much challenging for the proposed method to work generally in the wide. It is still unknown whether it can generalize that well. Although this point is also listed in the Limitations section, the reviewer thinks this is still a marjor weak point.\n\n4. **The scalability problem**. The authors mention that, even for this model in limited ODD, it still takes weeks to train. Many intermediate methods are time consuming. The problem comes that whether the pipeline is still feasible for a motion planning method in the wild (the scale of the dataset should be much much larger than the current one). The authors may provide further thoughts on this issue.\n  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The idea is interesting, but still some gap to a solid contribution.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The four weak points in the previous section may all be justified in the response.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "165j5kMyEyt", "reviewer": "Reviewer_a94g", "summary": "The trajectory optimization problem for a quadrotor to fly through multiple way-points in as short a time as possible has very complex dynamics. The problem is how to deal with the varying number of way-points and large computational cost, especially when the trajectory must be generated in real-time. Usually, simplified dynamics models or polynomial approximations of limited intervals are used, but these methods have limited performance. In this paper, the authors have succeeded in constructing a seq2seq model that can generate trajectories with a lower computational cost by approximating the exact solution with a DNN model and then generalizing the trained DNN model by reinforcement learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is consistent and clearly presented from the problem statement to the proposal of a solution and its evaluations through experiments on real systems.\n- The most significant contribution of the paper is the proposal of a framework for solving the problem stated in the abstract, but the reader does not have sufficient information necessary to implement the proposed framework. For example, the network structure of the seq2seq model and the method of preparing training data.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The consistency of the paper, the clarity of the presentation, and the implementation in real environments are highly evaluated, but on the other hand, the novelty of the learning algorithm is not high, and the evaluation on generalization performance is not sufficient. Therefore, it was evaluated one rank lower than the highest.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- In order to discuss the scalability of the proposed method, specifics on the network size of the seq2seq model and the details of the GPGPU environment used in the experiments are required.\n- In addition to real-time performance, it would be better to discuss battery consumption.\n"}}]

Author rebuttal:

Decision: Accept (poster)