Learning to Predict Vehicle Trajectories with Model-based Planning
Authors: Haoran Song
Di Luan
Wenchao Ding
Michael Y Wang
Qifeng Chen
Venue: CoRL2021 Poster (2021)
Keywords: Trajectory Prediction
Autonomous Driving

Abstract: Predicting the future trajectories of on-road vehicles is critical for autonomous driving. In this paper, we introduce a novel prediction framework called PRIME, which stands for Prediction with Model-based Planning. Unlike recent prediction works that utilize neural networks to model scene context and produce unconstrained trajectories, PRIME is designed to generate accurate and feasibility-guaranteed future trajectory predictions. PRIME guarantees the trajectory feasibility by exploiting a model-based generator to produce future trajectories under explicit constraints and enables accurate multimodal prediction by utilizing a learning-based evaluator to select future trajectories. We conduct experiments on the large-scale Argoverse Motion Forecasting Benchmark, where PRIME outperforms the state-of-the-art methods in prediction accuracy, feasibility, and robustness under imperfect tracking.

Learning to Predict Vehicle Trajectories with Model-based Planning
Haoran Song Di Luan Wenchao Ding Michael Yu Wang Qifeng Chen
The Hong Kong University of Science and Technology {hsongad, wdingae, dluan, mywang, eqf)eust.hk
Abstract: Predicting the future trajectories of on-road vehicles is critical for autonomous driving. In this paper, we introduce novel prediction framework called PRIME, which stands for Prediction with Model-based Planning. Unlike recent prediction works that utilize neural networks to model scene context and produce unconstrained trajectories, PRIME is designed to generate accurate and future trajectory predictions. PRIME guarantees the trajec- tory feasibility by exploiting model-based generator to produce future trajec- tories under explicit constraints and enables accurate multimodal prediction by utilizing learning- based evaluator select future trajectories. We conduct ex- periments on the large-scale Argoverse Motion Forecasting Benchmark, where PRIME outperforms the Mfft-oof-the-at methods in prediction accuracy, feasibil- ity, and robustness under imperfect tracking.
Keywords: trajectory prediction, autonomous driving
1 Introduction
In the architecture of autonomous driving, prediction serves as the bridging module that reasons future states based on the perceived information from upstream detection and tracking and provides the predicted future states facilitate the downstream planning. Therefore, making accurate and reasonable trajectory predictions for on-road vehicles is vital for planning safe, efficient, and com- fortable motion for self-driving vehicles (SDVs).
The widely known challenge of trajectory prediction lies in modeling multi-agent interaction and inferring multimodal future states under driving scenarios. Traditional methods [1, 2, 3, 4, 5] pro- duce motion forecasting by handcrafted rules models with embedded physical and environmental features, which are insufficient for modeling interactive agents in complex scenes. Learning-baas based approaches [6, 7, 8), with deep neural networks to fuse scene context information and generate future trajectories, significantly promote the prediction accuracy and dominate the recent motion forecasting benchmarks for autonomous driving [9, 10].
Despite achieving steady improvement in accuracy, much less attention has been paid the feasibil- ity and robustness of learning-basee trajectory prediction. Indeed, most traffic participants operate under their inherent kinematic constraints (e.g., non-holonomic motion constraints for vehicles) while in compliance with the road structure (e.g., lane connectivity, static obstacles) and semantic information (e.g., traffic lights. speed limits). All these kinematic and environmental constraints explicitly regularize the trajectory space. However, most existing approaches model traffic agents as points and generate future trajectories without imposing constraints. Such constraint-fres predictions may be incompliant with kinematic or environmental characteristics and thus give rise to massive uncertainty in the predicted future states. As result, the downstream planning module would in- evitably undergo some extra burdens, and even the "freezing robot problem" " [11]. Furthermore, the trajectory predictions typically generated by neural network regression have high dependences on long-term tracking. For some dense driving scenarios where the target would be momently occluded or suddenly appears within the sensing range, tracking results are discontinuous or not accumulated enough. The prediction accuracy would thereby degrade under such imperfect tracking cases.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Torget Slar
Map M
Reachable Poths Trajectories
Predicted Trajectories
yP
T
Modeling Modeling Agent-map Interactions
T fur E
Model-based Generator
Learning-based Evaluator
Figure I: Illustration of the PRIME framework. The model-based generator (left) samples feasible future trajectories T for the target agent by taking its teme- state sa.r and HD map M, while im- posing explicit constraints C to guarantee trajectory feasibility. The learning- Cearning-based evaluator (right) receives the feasible trajectory set T and all observed tracks S to model the implicit interactions in scene context and then selects final set of trajectories Tear T as the prediction result.
Toward overcoming these challenges, propose PRIME, novel architecture for vehicle trajec- tory prediction, as illustrated Fig. 1. The core idea is to exploit model-based motion planner as the prediction generator to produce Ceasibility-guarantee constraints, together with a deep neural network as the prediction evaluator to enable accurate mul- timodal prediction by learning complex implicit interactions. To the best of our knowledge, PRIME the first to incorporate an interpretable motion planner in prediction learning and also the only method that ensures kinematic and environmental feasibility in data-driven trajectory prediction. We conduct experiments on the large-scale Argoverse motion forecasting benchmark and achieves better prediction accuracy over the daat-offthe-ar Furthermore, PRIME shows significant superi- ority in trajectory feasibility guarantee and prediction robustness under imperfect tracking. These attributes would facilitate more flexible and safe motion planning for SDVs.
future trajectories under explicit physical
2 Related Work
Prediction and Planning are closely intertwined in autonomous driving [12, 13, 14, 15]. Planning is to generate constraint-complian trajectory candidates and, after considering safety, comfort, path progress, etc., select the best trajectory for execution by the SDV (ego agent). Prediction facilitates the trajectory selection in planning by inferring future trajectories of the surrounding vehicles (target agents). Their different focuses make the corresponding mainstream solutions diverge. Model-based approaches [16, 17, 18, 19] are preferred in planning due to their interpretability and reliability in computing safe trajectories under explicit constraints. Learning-base methods [6, 7, 20, 21), in contrast, prevail in prediction by utilizing its advantage in modeling implicit interactions.
Some learning-basse based prediction works incorporate the goal-directed idea from planning to infer the possible goals and then produce goal-conditioned trajectories [22, 23, 24, 25). Moreover, the frameworks are introduced make predictions conditioned on ego intentions [26] or motion plans [27, 28]. Although much emphasis on improving the point-level prediction accuracy, the data-driven frameworks cannot ensure the given constraints are indeed im- posed trajectory generation. Despite DKM [29] embeds the two-axle vehicle kinematics [30] in the output layer to ensure kinematic feasibility, yet still no guarantee on environmental compliance. Inspired by the popular sampling-bases paradigm in vehicle motion planning [13, 17]. we employ model-based planner for providing cessibllit-guaraatee based part is reduced to evaluate future trajectories. With making the most of model-based planning and learning-bur based prediction, PRIME handles complex agent-map interactions while fulfilling en- vironmental and kinematic constraints.
novel planning-predictto
trajectory sets, and thereby the learning-
Modeling agent-map interactions fundamental for capturing information from scene map and dynamic agents. The rasterized representation [20, 31, 32] is proposed for learning- bassi methods, which renders traffic entities into images by different colors intensities and then encodes rasters with Convolutional Neural Networks. As an alternative, the vectorized representation [33, 34, 25] vectorizes scene context as nodes to construct a graph, which exploits High Definition (HD) maps more explicitly and improves prediction accuracy. By contrast, we address the agent-map modeling

---

with a hierarchical structure that incorporates the lane-association ideas from [3] while extends to learn global scene context. To be specific, our prediction generator acts locally in a planning manner to generate path-conditioned trajectory sets, and the prediction evaluator learns global understanding of the scene context by aggregating all trajectory and map features.
Generating multimodal trajectories is essential for handling the intrinsic multimodal pre- diction distributions. Stochastic models are mostly built upon conditional variational autoen- coder [7, 35, 36, 37, 38] and generative adversarial network [39,40 40, 41, 42], while sampling with un- controllable latent variables at inference may impede their deployment on safety-critical driving sys- tems. Deterministic models are mainly based on mode trajectory regression [43, 44, 20, 34]. To alleviate mode collapse in prediction learning. recent works decompose the task into classifica- tion over anchor trajectories [45] or condioondt trajectories [25], followed by trajectory offset regression. However, no feasibility could be ensured for the regressed results. CoverNet [32] for- mulates multimodal prediction by directly classifying on pre-constructed trajectory set, but still, its predictions may violate the agent kinematics or scene constraints. For our framework, leveraging model-based planning as the prediction generator brings superiorities in 1) maintaining multimodal distributions by generating trajectories on diverse reachable paths, 2) ensuring trajectory feasibil- ity by imposing real-time constraints, 3) mitigating the high reliance on long-term tracking, and 4)
producing trajectories with continuous information rather than just discrete locations.
3 Overview
Problem formulation. Assume the self-driving vehicle is equipped with detection and tracking modules to provide observed states of on-road agents A and has access to HD map M. Let s denote the state of agent a; E A at frame I, including position, heading, velocity, turning rate and
actor type, and s; = { Tp+1 ,s TT++2 .....! denotes the state sequence in the observed period Tp. Given any agent as the prediction target, we denote it by atar and its surrounding agents by Anbrs {a1 02 ..., } for differentiation, with their state sequence correspondingly given as {otar) } U Antrs- Our objective is to predict multimodal future trajectories Taar = (T& = 1, 2, ..., K} together with corresponding trajectory probability (px J. where Ta denotes a predicted trajectory for target agent Atar with continuous state information up to the prediction horizon Tr, K is the number of predicted trajectories. Additionally, it is required to ensure each prediction Ta E Ttar is feasible with existing constraints C, which includes environmental constraints CM and the kinematic constraints Clar.
Star and Subrs {s1. S2. ..., Sm.. Accordingly, S = {Star} } U Sabrs and
Our framework. The two-stage architecture of PRIME consists of model-based generator G and learning- basen evaluator E. Concretely, the generator G (shar; M, C)) + (P,7) is tasked to pro- duce the trajectory space for the target, which is approximated by a finite set of feasible trajectories T This part starts with searching set of reachable paths P = [[P]li = 1,2, 2, ..... .) from HD map M, which provides reference path for trajectory generation. Then classical sampling maaeeng-bbsedd based planner is utilized to generate trajectory samples under constraints in C, and thus provide the feasible future trajectory set T U₁= (Tr, lk 1.2, ny} for the target. Ty,k denotes the k-th feasible trajec-
tory generated from path P,. and the total number of trajectories is = E R2j. The model-based part is specialized in generating trajectories with feasibility guaranteed but ignores multi-agent inter- actions. The evaluator E : (P) + (Tear; {p&}) takes charge of learning implicit interactions, which features with dual representation for spatial information and with the attention mechanism to process the varying sizes of / reachable paths, m surrounding agents, and n feasible trajectories. Notably, the evaluator E is reduced to score trajectories and select prediction results Trar C T, rather than regressing position or displacement as most learning-baus based frameworks do.
4 Model-based Generator
4.1 Path Search
Unlike motion planning, where the reference path for the controllable ego agent is given, the future paths of uncontrollable targets in prediction are unknown. Therefore, we conduct the path search in advance of trajectory generation such that any prediction target could be associated with a set of potential paths p+ Our path search algorithm G Gpath* : (M, slar)) + P+ is implemented by Depth-
3

---

First-Search on HD map M. with more details described in the supplementary material. Yielding potential path P, E P with the centerline coordinates of each lane segment sequence, we expect all the paths of p to provide sufficient coverage for the future trajectory space of Otar- As no dynamic constraint imposed in this phase, for target with current state s some paths in P+ may not be reachable at frame = TF. For instance, high-speed vehicle cannot change to the opposite lane with a U-turn in few seconds. Such unreachable paths could be recognized in the following trajectory generation phase as no trajectories samples towards them are feasible. Finally, a set of reachable paths P C pt would be reserved.
4.2 Trajectory Generation
Given the potential paths in P as dynamic references, we choose to generate future trajectories in planning manner. For SDV, motion planning typically aims at finding an optimal trajectory to connect the current state and goal state, essen- tially different from prediction that infers probable trajectories for vehicles with unknown intentions. Despite this, the model- based generator in planning, which computes large number of trajectory samples for the follow-up selection, could also be exploited in prediction.
path centérline
trajectory i(s(t), d(t)
Star
s(t)
d(L)
F(s)
We adopt the trajectory generation phase of Frenét plan- ner [17] in our trajectory generator Gtraj : (P+,stars C) r T. Given a reference path in p+, a dynamic curvilinear frame is given by the tangential vector Ł,- and normal vector n, at certain point r on the path centerline. The Cartesian coordinate I = (i, y) could be converted to the Frenét coordinate (s, d), with the relation
Figure 2: Trajectory generation in
a Frenét Frame
H(s(t), ((t))) = F(s(t)) + d(t)ū,
(1)
in which F' represents a vector pointing from the path root, and d denote the covered are length and the perpendicular offset, as illustrated in Fig. 2. The trajectory generation phase first projects the current state sa onto the Frenét frame and obtains the state tuple [sn. &o, So, do, do. do)- The longitudinal movement st) and lateral movement d(t) within the prediction horizon TF are then generated independently by connecting the fixed start state with different end states using paramet- ric curves to cover different driving maneuvers, Compared with planning, prediction receives less accurate state estimation for targets and does not need grained trajectories. In our trajectory gen- erator, therefore, some high-order state variables are simplified to zero. For longitudinal movement, we sample the target velocity (TT) + &; in the range of [max (0, áp 8 TF), min( $o + + while leaving sTy) unconstrained. The constants ,= 8 and å are given by considering the actor type of otar and speed limit in M, to control the longitudinal velocity à; in a reasonable range. Each longitudinal trajectory st( is calculated using quartic polynomial
s.t. [s(0), (0), 5(0), s(Tr), ((TT)] = [So, 8o 0, $; O].
For lateral movement, we sample the target offset d(Tr) <- d, in the range of [ddiane/ 2, diane/ 2], where diane denotes lane width. Each lateral trajectory (t) is calculated using a quintic polynomial
s.t. d()), a())) d(T)) dd(T)))d(Tr)).d)) ddTTr),
[do, dg. 0, 0,
With the resulted longitudinal and lateral trajectory set Tion and Tiat. full trajectory (s(t), d(t)) is formed by every combinations in Tion x Tiat- Next, the trajectories incompliant with given con- straints C would be filtered out. We first project the Frenét coordinates (s, d) back to global coordi- nates (I, y) to check if the trajectory collides with static obstacles given in Cy. For collision free trajectories, their high-order state variables are then converted by the Frendf-Cartsiia-ttaaffooaai
8, d, d, - [7, U.K.K,
(2)
to check if any velocity u, acceleration a or curvature к exceeds the kinematic constraints given in Ciar. Finally. each reference path P, E P would generate a set of n, Ceasiblitt-guuarnteed guaranteed future trajectories [T = 1, 2,.,,..,;; and all the trajectories together form an overall trajectory space T. Although the constraints are set conservatively with leaving some margin for the learning-based evaluator, our model-based generator effectively narrows down the trajectory space T by imposing constraints. This unique advantage would set our framework with higher accuracy and robustness.

---

Sequences ( mmm
Attention (Poth to Track)
Attention (Agent to Agent)
Track Encoding (m
PC loyers
72
Reachable = [9 [P
Scores
Path Encoding x L
Attention (Path to Future)
Attention (Future to Future)
P1 P2 .n Px
Feosilee Trojectories
11
X n
Future Encoding
x
Predicted Trajectories
Other's Track
Target's Treck
Reachable Poth
Feasible Future Trajectory
Predicted Trajectory
PRIME framework overview. The model-based generator searches reachable paths P through the map and produces feasible future trajectories T. The learning-base evaluator encodes the traffic entities in (P, T, S) and learns implicit interactions in the subsequent attention modules. Afterwards, each future trajectory Tj could query its track tensor X, (Stor) from P2T, interaction tensor Y, (Star) from A2A and future tensor (T,,,) from F2F, and it is scored by feeding the concatenation of these tensors to fully-connected layers. Finally, the evaluator ranks all feasible future trajectories by scoring and outputs final of K predicted trajectories.
Figure 3:
5
Learning-based Evaluator
5.1 State Representation
The prediction evaluator aggregates P, and future trajectory set T. To make it compatible with most existing trajectory prediction datasets, state sequence s; is reduced to history track in the learning part. Before feeding to the network, we discretize each history track 8; and future trajectory Ti. as a location sequence with time interval AT, and each reference path Py as a waypoint sequence with distance interval AD. Since the longitudinal movement and lateral offset indicate how an agent moves relative to reference path, they represent the local spatial relationship more straightforwardly. For this reason, we use the Frenét coordinates (s, d) in addition to the Cartesian coordinates (ar, y) to form a dual spatial representation. Here, the spatial information (a., y, 8, d) of future trajectories TT is given by the generator, while the (s, d) coordinates of history tracks in S are obtained by projecting (æ, y) coordinates on the corresponding reference path. Additionally, we adopt the approach of [34] to add binary mask b to history track's 's representation (I, 8,8,8888 d, b) to indicate if the location is padded.
context, including observed state sequences S, path set
5.2 Encoding Scene Context
Prior to capture interrelationships between traffic entities, we first encode each kind of entity in the scene. All encoders are structured with temporal convolutional layer followed by long short-term memory (LSTM) layer. The track encoder and the future encoder employ a unidirectional LSTM and make the last hidden state h(.)) as the motion encoding for history track and future trajectory, while the path encoder uses a bidirectional LSTM and provides sequence of hidden states H(-) as the path spatial encoding. Given the scene context description (S, P, T). each reachable path P, P is encoded as H(2)), where j = 1, ..., Considering that the Frenét representation dependent with the path frame, we encode all history tracks with respect to each reference path P,, which results groups of track encodings (h(star), ), ((s1).... ((ss.)), Each future trajectory TA E T relative to its reference path P,, so future trajectories are encoded correspondingly to form groups of future encodings (b(TT,))l: 1,2,..., n2).
5.3 Modeling Interactions
Next to capture the implicit interactions resulted from the static environment and multiple dynamic agents. To fuse the spatial-temporal information from varying numbers of entities in the scene con-
5

---

text, the attention mechanism [46] is adopted to construct four modules, namely, path to track (P2T), path to future (P2F), agent agent (A2A), and future to future (F2F). They are implemented in the same way of scaled dot-product attention and use linear layers for mapping key, query and value. The overall workflow is shown in Fig. 3. In the upper branch, P2T brings the spatial information of each path encoding P, into the corresponding track encodings (h((tar)), h(ss), h(ssm)). The track encodings are further processed by a self-attention structure in A2A, aiming to capture the in- teractions between agents in the past time domain. The lower branch lays emphasis on updating the features contained in future encodings. P2F brings the spatial information of path encoding H (P;)) into the corresponding future encodings {I((T,,)le
mz.. It is followed by F2F that fuses all future encodings - {A(T,,))kk = 1, 2, ny} from different paths Py(j = 1,,,....I) using self-attention. In particular, F2F obtains a global understanding of the reachable space given by P and, by this way, attempts further perceive the differences between different trajectories in T. For any future trajectories T E T. the corresponding track tensor X, (Ster),, interaction tensor ( and future tensor ((T,,)) could be obtained from P2T, A2A and F2F modules, which are then concatenated together to form a full description U, J = Concat (X y y ((Tz,)))
[k =
5.4 Trajectory Scoring, Learning, and Inference
With U,, as a full description, we score all the n trajectories TGK using a maximum entropy model:
exp(f(U,,x)) (Txxk) E/=1 En exp(f(U,,)))
in which f(-) is implemented using a 3-layer MLP at the end of the evaluation network E. The score label (Tx,,) is resulted from calculating the accumulated squared distance error Dist(-) between the future trajectory Tr.k and the ground truth trajectory Ter within the prediction horizon Ty:
exp(
Dist T j.b, TCT)/T)
(T,,)) E E exp( Dist Ter)/T)
(4)
where T is set as temperature factor. The overall network is trained by cross entropy between the evaluated scores and the labeled scores L = (ansEEnntroy((((T... (T,x))) For the inference stage that requires K predicted trajectories, we adopt the non-maximum suppression (NMS) algorithm to remove near-duplicate trajectories, as did in [25]. According to the predicted scores, this method greedily picks trajectories from T and excludes the lower scored trajectory between very close ones. Finally, K trajectories with descending order of scores form the prediction result Tear (Tijk 1.2.... .... K}. and the prediction probability pk is derived by p = (Tx)// (Tx).
Experiments
Dataset. Argoverse [10] is one of the largest publicly available motion forecasting datasets, which contains over 324K data sequence collected from complex urban driving scenarios. The training, validation, and test sets are taken from disjoint parts of the cities. Each sequence lasts for 5 seconds, containing the centroid locations of each tracked agent sampled at 10 Hz, which one vehicle with relatively complex motion is marked the prediction target. The objective is to predict its locations 3 seconds into the future, given an initial 2-second observation.
Metrics. We follow the Argoverse evaluation criteria under K = 1 and K = 6. Minimum Aver- age Displacement Error (minADE<) is the average L2 distance error of the best predicted trajec- tory. Minimum Final Displacement Error (minFDEx)) is the L2 distance error of the best predicted trajectory at the final timestamp. Miss Rate (MRx) is the ratio of scenarios that none of K pre- dicted trajectories has less than 2 meters L2 final displacement error. For multimodal prediction, the probability-bar based metrics p-minADE and p-minFDEx are calculated by adding -log(p) to minADE& and minFDE,, where p corresponds to the probability of the best predicted trajectory- In the Argoverse benchmark, best refers to the predicted trajectory with the minimum endpoint error.
Implementation Details. Our implementation is detailed in the supplementary material. Among the state-of-the-aar methods, only LaneGCN [34] is open-source. So we use its official implementation and Argoverse baselines [10] for additional tests about trajectory feasibility and imperfect tracking.

---

K=1
K=6
Method
Infeasibility (%)
minADE
(%)
minADE
p-minADE
p-minFDE
Argo-CV
3.53
7.89
83.48
3.39
7.57
5.18
9.36
81 1.68
0.00
Argo-LSTM--at
2.96
6.81
81.22
2.34
5.44
4.14
7.23
69. .66
43.53
Argo-NN+map
3.65
8.12
83.55
2.08
4.03
3.87
5.82
58. 21
86.39
LaneGCN [34]
1.71
3.78
59.05
0.87
1.36
2.66
3.16
16.34
16.52
Alibaba-ADaab
1.97
4.35
63.76
0.92
2.67
3.23
15.86
TNT [25]
1.78
3.91
59.72
0.94
1.54
2.73
3.33
13.28
Jean [21]
1.74
4.24
68.56
1.00
1.42
2.79
3.21
13.08
Poly
1.70
3.82
58.80
0.87
1.47
2.67
3.26
12.02
PRIME (Ours)
1.91
3.82
58.67
1.22
1.56
2.71
3.05
11.50
0.00
Table Comparison with the Argoverse baselines and the state-of-the-art methods on the Argoverse the key metric.
test set. All metrics are lower the better and Miss Rate (MR, K=6)
6.1 Comparison with State-of-the-art
We compare our proposed PRIME against the Argoverse baselines [10] (CV, LSTM+map, NN+map), the top-3 methods the Argoverse Motion Forecasting Competition 2020 (Jean [21], Poly, Alibaba-ADLab) and the recently published state-of-the-ans LaneGCN [34] and TNT [25]. The performance comparison under Argoverse test
shown in Table It could be noted that PRIME outperforms all other methods on Miss Rate (K = 6). which is the official ranking metric in Argoverse Competition 2020. It reflects that our method produces accurate multimodal predic- tions consistently in diverse scenarios. We also achieve the best on the probability-base metric p- minFDE,, which would be highly beneficial to weigh between multiple predictions making deci- sions and motion plans. From the methods with public details, including LaneGCN [34]. TNT [25], and Jean [21], we can find they all perform the learning-based paradigm that utilizes neural networks to model traffic entities and generates future trajectories, while PRIME is the only one that integrates a model- based motion generator into prediction learning. Notably, due to the lack more detailed on-road information in the Argoverse dataset, such as vehicle types, bounding box, static obstacles, etc., the quantitative result is achieved by imposing general constraints on the model- based gener- ator. This indicates there exists more space to improve when deploying our framework in real autonomous driving system. Furthermore, handling environmental and dynamic constraints in an interpretable model-based manner and generating trajectories with continuous state information is significant for real-world deployment, which could not be reflected from the evaluation metrics.
6.2 Ablation Studies
We ablate the F2F module and Frenét representation (denoted by SD) from the complete evaluation network to study their impacts. Table 2 reports the re- sults the Argoverse validation set. With P2T, P2A, and A2A attention mod- ules capturing the basic interactions be- tween map and agents, the base model performs at the same level with TNT (MR, = 9% reported in [25]), indicat- ing that these modules are effective in capturing agent-map interactions. As for the Frenét represen- tation providing the local spatial relationship and the F2F module fusing all feasible trajectories to get a global understanding of the reachable space, they both promote the performance. By compar- ison, the inclusion of Frenét representation is more effective. Additionally, the complete network makes the best performance with only 1.02M parameters, which indicates that separating the func- tion of trajectory generation would reduce the learning burden while achieving high performance.
Modules
p-minADEs p-minFDE, MRa(%) Params
Base
2.33
2.63
8.52
0.69 M
Basc+F2F
2.31
2.61
8.23
0.72 M
Base+SD
2.29
2.58
0.99
Basc+F2F4SD
2.29
2.57
7.51
1.02 M
Table 2: Ablation studies on the Argoverse validation set.
7

---

6.3 Trajectory Feasibility
As typical non-holonomic motion system, vehicles constrained by inherent kinematic charac- teristics. So we investigate the ratio of infeasible trajectories produced by prediction models. Since the high-order states (velocity, acceleration, turning rate) cannot be estimated accurately from discrete locations predicted by common learning-basse models, we evaluate the trajectory feasibil- ity only using curvature. By interpolating the predicted positions with pairwise cubic splines, we get the curvature at each point. A trajectory is labeled as infeasible if the curvature K > 1/3 (i.e.. the minimum turning radius is 3 meters) any of its points. The ratio of infeasible trajectories is shown in the last column of Table 1. Except for the physical baseline Argo-CV (Constant Veloc- ity), the others, as representatives of the unconstrained bearning-based models, have at least 16.5% infeasible predictions. Although we only use curvature for judgment and set fairly conservative threshold (the minimal turning radius for regular sedan is around 4.5 6.0 meters), the infeasible predictions still take up considerable proportion, which would cause redundant burdens for SDVs to make decisions and plans. By contrast, the model-based generator in our framework can handle any kinematic and environmental constraints, thereby ensuring trajectory feasibility.
6.4 Imperfect Tracking
While most motion forecasting datasets provide tracking results of certain dura- tion for prediction targets, a self-driving vehicle would inevitably encounter real- world situations where the target is lost in some timestamps or not tracked long enough yet. Then the prediction model is required to robustly handle imperfect tracks rather than being restricted to fixed- duration tracking inputs. To let the mod- els (ours, LaneGCN, and NN+map base- line) be aware of imperfect tracks, we re- trained them by randomly dropping out tracked locations. For processing such in- puts while keeping network structures, we pad the locations of dropped timestamps with the nearest tracked location and add a dimension of the binary mask to denote the padded location. The drop rate ran- domly sampled from 0 ~ 0.6 for each data sequence in training but fixed in testing. The drop rate is pointwise applied, i.e., 0.6 drop rate may drop more or less than 60% of locations on a track. The last timestamp is always kept to ensure the prediction target could be detected at inference. Fig. 4 shows how the miss rate varies with track drop rate, we observe that our model performs stably, with only 3.6% relative increase on MRs, while the relative increase is around 30% ~ 40% for the others. The result indicates that the baaning-based prediction models rely on long-term tracked results to regress trajectories, while our framework design relieves that to a certain extent, thereby improving the prediction robustness.
60%
Miss Rate (K (k=6)
50%
NN m
AMR, - -00.6% (37%))
6
40%
LaneGCN: AMR; 3..2% (30%*)
30%
Ours:
AMR 0.29% (3.6%11
20%
10%
0
10%
20%
30%
40%
50%
60%
Track Drop Rate
Figure 4: Comparison of prediction robustness under imperfect tracking.
Conclusion
We present the prediction framework PRIME that learns predict vehicle trajectories with model- based planning. PRIME guarantees the trajectory feasibility by exploiting model-based generator to produce future trajectories under explicit constraints. It makes accurate trajectory predictions by employing a learning- based evaluator to capture implicit interactions in scene context and select future trajectories by scoring. With the novel framework design, PRIME outperforms the state-of- the-art in prediction accuracy, feasibility, and robustness. Moreover, the advantages of reasonably regularizing trajectory space, predicting trajectories with continuous state, and the compatibility with rn-d information would set our framework highly useful in real system deployment.
8

---

Acknowledgments
This research work was supported in part by The Hong Kong University of Science and Technology under the project "LDSERFF-utonomy Through Learning".
References
[1] D. Helbing and P. Molnar. Social force model for pedestrian dynamics. Physical review E, 51 (5):4282, 1995.
[2] A. Houenou, P. Bonnifait, V. Cherfaoui, and W. Yao. Vehicle trajectory prediction based on motion model and maneuver recognition. In IROS, 2013.
[3] J. Ziegler et al. Making bertha drive-an autonomous journey on historic route. IEEE Intelligent transportation systems magazine, 6(2):8-20, 2014.
[4] S. Lefevre, D. Vasquez, and C. Laugier. A survey on motion prediction and risk assessment for intelligent vehicles. ROBOMECH journal, 1(1):1-14, 2014.
[5] W. Schwarting, A. Pierson, J. Alonso-Mora, S. Karaman, and D. Rus. Social behavior for autonomous vehicles. PNAS, 116650)229922-24978 2019.
[6] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social Istm: Human trajectory prediction in crowded spaces. CVPR, 2016.
[7] N. Lee, Choi, P. Vernaza, C. B. Choy, H. Torr, and M. Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In CVPR, 2017.
[8] M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. In CVPR, 2018.
[9] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: multimodal dataset for autonomous driving. CVPR, 2020.
[10] M.-F. Chang, J. Lambert, P. Sangkloy, Singh, Bak, Hartnett, D. Wang, Carr, S. Lucey, D. Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019.
[11] P. Trautman and A. Krause. Unfreezing the robot: Navigation in dense, interacting crowds. In IROS, 2010.
[12] D. Ferguson, T. M. Howard, and M. Likhachev. Motion planning in urban environments. Journal of Field Robotics, 25(11-12):939 9200, 2008.
[13] J. Wei, J. M. Dolan, and B. Litkouhi. A prediction-and cost bbaeid algorithm for robust autonomous freeway driving. In IEEE Intelligent Vehicles Symposium, 2010.
[14] C. Katrakazas, M. Quddus, W.-H. Chen, and L. Deka. Real-time motion planning methods for autonomous on-road driving: State-of-the-arr and future research directions. Transportation Research Part C: Emerging Technologies, 60:416-44] 2015.
[15] W. Schwarting, J. Alonso-Mora, and D. Rus. Planning and decision-making for autonomous vehicles. Annual Review of Control, Robotics, and Autonomous Systems, 2018.
[16] M. Pivtoraiko, R. A. Knepper, and A. Kelly. Differentially constrained mobile robot motion planning in state lattices. Journal of Field Robotics, 26(3):308-333 333,, 2009.
[17] M. Werling, J. Ziegler, S. Kammel, and S. Thrun. Optimal trajectory generation for dynamic street scenarios in a frenet frame. In ICRA, 2010.
[18] M. McNaughton, C. Urmson, J. M. Dolan, and J.-W. Lec. Motion planning for autonomous driving with a conformal spatiotemporal lattice. In ICRA, 2011.
[19] E. Galceran, A. G. Cunningham, R. M. Fustice, and E. Olson. Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction. In RSS, 2015.

---

[20] H. Cui, V. Radosavljevie, F.-C. Chou, T.-H. Lin, T. Nguyen, T.-K. Huang, J. Schneider, and N. Djurie. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In ICRA, 2019.
[21] Mercat, T. Gilles, N. EI Zoghby, G. Sandou, D. Beauvois, and G. P. Gil. Multi-head attention for multi-modal joint vehicle motion forecasting. In ICRA, 2020.
[22] B. D. Ziebart, N. Ratliff, G. Gallagher, Mertz, K. Peterson, J. A. Bagnell, M. Hebert, A. K. Dey, and S. Srinivasa. Planning- based prediction for pedestrians. In IROS, 2009.
[23] E. Rehder, F. Wirth, M. Lauer, and C. Stiller. Pedestrian prediction by planning using deep neural networks. In ICRA, 2018.
[24] K. Mangalam, Girase, S. Agarwal, K.-H. Lee, Adeli, Malik, and A. Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In ECCV, 2020.
[25] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp. B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, al. Tnt: Target-driven trajectory prediction. In CoRL, 2020.
[26] N. Rhinehart, R. McAllister, K. Kitani, and Levine. Precog: Prediction conditioned on goals in visual multi-agent settings. In ICCV, 2019.
[27] H. Song, W. Ding, Y. Chen, S. Shen, M. Y. Wang, and Q. Chen. Pip: Planning-informed trajectory prediction for autonomous driving. In ECCV, 2020.
[28] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-feasibbe trajectory forecasting with heterogencous data. In ECCV, 2020.
[29] H. Cui, T. Nguyen, F-C. Chou, T.-H. Lin, J. Schneider, D. Bradley, and N. Djuric. Deep kinematic models for kinematically feasible vehicle trajectory predictions. In ICRA, 2020.
[30] R. Rajamani. Vehicle dynamics and control. Springer Science & Business Media, 2011.
[31] N. Djurie, V. Radosavljevic, H. Cui, T. Nguyen, F--C. Chou, T.-H. Lin, N. Singh, and Schnei- der. Uncertainty-avae -waare short-term motion prediction of traffic actors for autonomous driving. In WACV, 2020.
[32] T. Phan-Minh, E. C. Grigore, F. A. Boulton, O. Beijhom, and E. M. Wolff. Covernet: Multi- modal behavior prediction using trajectory sets. In CVPR, 2020.
[33] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In CVPR, 2020.
[34] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forecasting. In ECCV, 2020.
[35] N. Rhinehart, K. M. Kitani, and P. Vernaza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In ECCV, 2018.
[36] J. Hong, B. Sapp, and J. Philbin. Rules of the road: Predicting driving behavior with convo- lutional model of semantic interactions. In CVPR, 2019.
[37] Y. C. Tang and R. Salakhutdinov. Multiple futures prediction. In NeurlPS, 2019.
[38] S. Casas, C. Gulino, S. Suo, K. Luo, R. Liao, and R. Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In ECCV, 2020.
[39] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In CVPR, 2018
[40] A. Sadeghian, V. Kosaraju, A. Sadeghian, N. Hirose, H. Rezatofighi, and S. Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In CVPR, 2019.
[41] Zhao, Y. Xu, M. Monfort, Choi, C. Baker, Y. Zhao, Y. Wang, and Y. N. Wu. Multi-agent tensor fusion for contextual trajectory prediction. In CVPR, 2019.
10

---

[42] X. Li, G. Rosman, Gilitschenski. C.-I. Vasile, J. A. DeCastro, S. Karaman, and D. Rus. Vehicle trajectory prediction using generative adversarial network with temporal logic syntax tree features. IEEE Robotics and Automation Letters, 2021.
[43] N. Deo and M. M. Trivedi. Convolutional social pooling for vehicle trajectory prediction. In CVPR Workshops, 2018.
[44] S. Casas, W. Luo, and R. Urtasun. Intentnet: Learning to predict intention from raw sensor data. In CoRL, 2018.
[45] Y. Chai, B. Sapp. M. Bansal, and D. Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In CoRL, 2019.
[46] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is you need. In NeurlPS, 2017.


Reviews:
[{"review_id": "xPxWTm9yPOz", "reviewer": "Reviewer_Q68c", "summary": "This paper aims to generate trajectories that abide by both kinematic and contextual constraints. The kinematic constraints include motion planning feasibility. The contextual constraints include road compliance, following traffic lights, following the speed limits. The key idea of this paper is to (1) generate a lot of plans and trajectories (2) train an evaluator to give the scores each trajectory.\n\nThe plans and trajectories are generated by doing depth first search in HD map for plans, and then use the dynamics to figure out the trajectories.\n\nThe evaluator is trained with the objective that regresses the predicted score to the labeled score using a cross entropy loss. The labeled score is computed by how far they are from the ground truth trajectory. The model takes both states, paths, and trajectories from the previous step, and pass them through a bunch of attention layers and output the score.\n\nThe authors show low p-ADE and miss rate meaning that it can cover the ground truth trajectory within 2 meters within one of the 6 predicted trajectories with 89% accuracy on Argoverse. The authors show that it is robust when tracking drops. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and clear.\nThe method is quite different from a more mainstream approach of using deep NN to generate trajectories but instead learning an evaluator. This is important and can inspire many more avenues in this direction.\nThe results demonstrate the improvement over the previous state-of-the-arts. The paper shows the robustness when tracking is dropped.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes an interesting idea of combining planning and learning in motion forecasting. The results surpassed other SOTA. While the paper does a great job considering robustness to tracking, I wonder if there is a reference of the actual problem in tracking. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "What are the failure modes of missing trajectories in the 11%? Can you provide common example scenarios?\n\nHow realistic is the tracking dropping? The experimental settings seem a bit constructed. Were there some references to that this is a common issue in practice? "}}, {"review_id": "lJi-lwYZkzp", "reviewer": "Reviewer_SJ8P", "summary": "The authors propose a prediction model that ranks different plans generated conditioned on reachable paths in the road network. A planner submodule is used to plan along the path, and an evaluator is used to rank the trajectories after encoding them. The authors detail the structure of the evaluator and choice of planner submodule, post-processing selection, as they obtain favorable results on Argoverse.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: Results  on Argoverse seem good, and some ablations show the exploration of design tradeoffs. Model-based trajectories can improve prediction at long horizons for majority of the driving scenarios, and can leverage the rich literature on planners, including different goals, rewards and constraints. I liked the exploration of tracking impact as it’s often neglected, and the use of local/global coordinates, which make sense and has been also shown in the recent CVPR WAD workshop discussions.\n\nWeaknesses: The paper’s main claim to fame (prediction via planning) and the main structure (i.e use of ranking) has been significantly explored before. While the specific approach is new, the readers would benefit from a more balanced view that targets the (much needed) set of open questions in the field that the paper touches on, such as exploration of the impact of the choices for planner, evaluator, lane handling, planner sample parameters (the path selected, any other parameters that prevent impoverished prediction set?).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the approach is not completely novel, and reuses elements from many existing works, it is overall a new combination, is presented clearly, and achieved good result on a valid benchmark. Would be good to see a better presentation with respect to other works that do planning for prediction, or generate-then-evaluate, and to explore the tradeoffs that drove the error down for this system.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "* How does the approach depend on the quality and diversity of the proposals made under the assumption of limited number of samples (MoN, N~5..7)? Do more suggested plans hurt performance, does the performance depend on good and “representative” action set? (the main problem is that it’s hard to generalize a models and plans to cover edge cases, which is why people look at model-free trajectory generation)\n* Why do we want planning-based predictions? Are we learning feasible or rewarding trajectories, and if so, why the assumption on a reward/planning structure? How does this compare to other approaches such as IRL / reward shaping (“Maximum Entropy Deep Inverse Reinforcement Learning“, “Social behavior for autonomous vehicles”), learned planning for  corrections of predictions (“Differentiable Logic Layer for Rule Guided Trajectory Prediction”), and injection into the discriminator (“Vehicle Trajectory Prediction Using Generative Adversarial Network With Temporal Logic Syntax Tree Features“) beyond citing, as the heavier machinery is in writing planners, seems like testing the other approaches for a comparison would’ve been easy and worthwhile. Would be also good to cite “What the Constant Velocity Model Can Teach Us About Pedestrian Motion Prediction“, as it makes a relevant claim -- learning-based generators are still quite sub-optimal given the motion statistics, to the point that simplified models (such as path following) can outperform them.\n* While it’s interesting to ablate the local/global coordinates representation and the F2F attention, the interesting ablations would be to compare the evaluator and planner contributions, since individually they have appeared in several works. Moreover, while adhering to lanes is important, this has been used in predictors before, so would be interesting to compare the specific representation for lane following vs. other approaches.\n* Where are the cases where planning is not adhered to and how bad are such mistakes? The main reason for model-free prediction is to handle rare events, but these are not always captured by path following.\n* In cases where multiple trajectories are generated for the same path, or nearby examples -- how do they differ? What do they capture? Would be good to hear/see examples, in addition to failure cases shown in the supp. mat.\n"}}, {"review_id": "dGPa8kFtyz", "reviewer": "Reviewer_fWJ2", "summary": "This paper proposes a technique to generate robust and feasible vehicle trajectory predictions. A model-based motion planner is designed to search reachable reference paths, around which the candidates of trajectory predictions are sampled, guaranteed to be compliant with kinematic constraints and road structures. The trajectories will be scored by a learning-based evaluator, which takes map structure and agent interaction into consideration. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The paper is well-written and easy to follow. \n2) The way to utilize map information is interpretable, and results are guaranteed to be feasible.\n3) Most model-based methods fail to generate diverse trajectories in interactive scenes, while learning-based methods suffer from trajectory quality. This paper combines the advantages of both methods and performs well in the argoverse competition.\n \nWeaknesses:\n1) The novelty of this paper is the combination of a model-based planner and a learning-based evaluator. However, the planner itself is widely used in industry, and the evaluator follows LaneGCN’s structure. I am not sure about the inference time for this proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would recommend this paper because it proposes a new method to make trajectory generators more practical. The experimental results are convincing.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I am curious about the forward inference time. ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "5ob_4q1FKAp", "reviewer": "Reviewer_PMNK", "summary": "This paper proposes a novel prediction framework called PRIME that learns to predict vehicle trajectories with a two-part approach with a model-based generator and a learning-based evaluator. This two-part system allows PRIME to generate accurate and feasibility-guaranteed predictions. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1) I would suggest that the authors consider adding a discussion of the limitations of the approach to the end of the paper.\n\nMinor comments\n\n1) Figure 4 is small and difficult to read\n2)Figure 3 caption is too long and should be reworked", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written and tackles a  relevant problem. The authors give a thorough description of their approach as well as experiments validating their claims. The author's two-part approach to trajectory prediction is novel and seems promising.\n\nHowever, a more robust experiment where more detailed on-road information would give a better picture of how much the PRIME improves feasibility when compared to the other baselines.\n\nAlso important is the need of a more\ndeveloped discussion around the limits of the approach.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "The paper is well written and tackles a well-studied but\nstill relevant problem. The authors provide a good\nliterature review and clearly explain their architecture,\npresenting some experiments to support their claim.  I believe it is suitable for CoRL."}}]

Author rebuttal:

Meta-review: Summary: This work proposes a prediction mechanism that combines model-based path and trajectory planners with a learned ranking system. The proposed method is supported by results on the Argoverse Benchmark. 

Clarity: All reviewers commented on the good composition of the paper. They seemed to agree that the paper was clearly and concisely written and overall effective in convincing the reader of the promise of the proposed approach. 

Originality: Reviewers noted that while the paper adopts existing techniques and a popular framing of the problem, it does combine specific techniques in a novel way. 

Quality: Reviewers were overall convinced that the results presented on the Argoverse Benchmark demonstrated a believable improvement over the state-of-the-art, however, also made suggestions on how to make the case even stronger. 

Significance: Reviewers had mixed feedback on the impact of the paper. It seems like reviewers were mainly looking to the experimental results to convince them and noted some gaps and lingering questions. 

Pros: 
The paper proposes a compelling way to combine model-based and learning-based tools to solve an important problem in robotics. The paper also addresses salient details such as the effect of tracking error and ablation experiments. 

Main opportunities for improvement: 
The authors are encouraged to focus efforts on addressing the questions raised and do the suggested additional analyses the reviewers requested in the experimental results section. The goal of these additions should be to resolve any lingering questions about the effectiveness of the proposed approach. 

Thank you for considering our feedback, and we look forward to seeing the updated paper.

============== Final Decision

All reviewers recommend acceptance. 

I am not 100% certain if this should be a poster v an oral presentation. The case for oral is that two reviewers gave a strong accept with strong scores in all sub-categories. However, I ultimately recommended poster because sub-scores in impact and originality are mixed (in particular, 2 in impact from reviewer PMNK  and fair in originality from reviewer SJ8P). 
Decision: Accept (Poster)