Multi-Predictor Fusion: Combining Learning-based and Rule-based Trajectory Predictors
Authors: Sushant Veer
Apoorva Sharma
Marco Pavone
Venue: CoRL 2023 Poster (2023)
Keywords: trajectory prediction
rule-based planning

Abstract: Trajectory prediction modules are key enablers for safe and efficient planning of autonomous vehicles (AVs), particularly in highly interactive traffic scenarios. Recently, learning-based trajectory predictors have experienced considerable success in providing state-of-the-art performance due to their ability to learn multimodal behaviors of other agents from data. In this paper, we present an algorithm called multi-predictor fusion (MPF) that augments the performance of learning-based predictors by imbuing them with motion planners that are tasked with satisfying logic-based rules. MPF probabilistically combines learning- and rule-based predictors by mixing trajectories from both standalone predictors in accordance with a belief distribution that reflects the online performance of each predictor. In our results, we show that MPF outperforms the two standalone predictors on various metrics and delivers the most consistent performance.

Multi-Predictor Fusion: Combining Learning-based and Rule-based Trajectory Predictors
Sushant Veer!, Apoorva Sharma', Marco Pavone
NNVIIIA Research, Stanford University (sveer, apoorvas, pegoorrass.mpvons. mpavone @nvidia com
Abstract: Trajectory prediction modules are key enablers for safe and efficient planning of autonomous vehicles (AVs), particularly in highly interactive traffic scenarios. Recently, learning-based trajectory predictors have experienced con- siderable success in providing state-of-the-art performance due to their ability to learn multimodal behaviors of other agents from data. In this paper, we present an algorithm called multi-predictor fusion (MPF) that augments the performance of learning-based predictors by imbuing them with motion planners that are tasked with satisfying bogic-based rules. MPF probabilistically combines learning- and rule-based predictors by mixing trajectories from both standalone predictors in ac- cordance with belief distribution that reflects the online performance of each predictor. In our results, we show that MPF outperforms the two standalone pre- dictors on various metrics and delivers the most consistent performance.
Keywords: trajectory prediction, rule-based planning
Introduction
The behavior of a traffic agent can depend on very large number of factors such as the desired nav- igation goal, personal driving style, local driving customs, etc., therefore, predicting future trajecto- ries of traffic agents is a fundamentally challenging task. Deep learning-based approaches [1, 2, 3] have emerged as front runners in this task due to their ability to extract patterns from complex high- dimensional data and generate multimodal predictions. However, improving the interpretability of these predictors and endowing them with ability scenarios remains an open challenge. Forcing the learned predictor to strictly conform with spe- cific set of logic rules can make the predictions brittle and sensitive to the completeness of the rules; for instance, when an agent violates the traffic rules, a strict traffic-law abiding predictor would forecast incorrect trajectories. In this paper, we present an approach to bake rules the predictor while retaining the flexibility afforded by a data-driven only approach and demonstrate ability to perform well on various real-world trajectory prediction datasets and metrics.
about the traffic law and nuanced traffic
We introduce multi-predictor fusion (MPF) that combines learning-basee trajectory predictors with planning-based predictors using Bayesian belief updates. We maintain a belief distribution over the standalone predictors. The probability assigned each predictor by the belief distribution is updated by monitoring their individual performance online; the belief probability mass is iteratively shifted towards the better-perfoe perter-performing predictor. Through this approach, MPF is able to relax the use of rules if the data-driven approach is performing better, and vice-versa.
MPF provides various benefits: First, we observe that learning-based predictors and rule-based pre- dictors perform well complementary data regimes. Learning-based predictors perform better in scenarios where multiple future outcomes are feasible due to their ability to reason about multi- modality. On the other hand, rule-based planners can incorporate traffic laws and perform better when the agent's intended goal is unambiguous or the scene is ditti---istribttion. The scatter plot in Fig. ((a) illustrates the complementary performance on different data: the average displacement error (ADE) for a large number of scenes lies far away from the identity line, suggesting that one of the predictors is performing much better them. Therefore, soft switching between the
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Agenti
RI Plenner ADE (m)
Trajectory
Predictor
Fuser Bellet Update
Trajectory Predictions
Scene Information
Bellet Sampler
Metrics
Rule-based Trajectory
Predictor
Trajectron++ ADE (m)
(a) Scatter Plot
(b) MPF Architecture
Figure 1: Motivation for predictor fusion and the architecture for multi-predictor fusion. The scatter plot depicts the Average Displacement Error (ADE) each agent-centric prediction in the nuPlan-mini dataset for learned (Trajectron++ [1]) and rule-hierarchy (RH) planner based predictor. (b) Architecture of MPF predictor.
predictors via the belief distribution can improve the overall prediction performance, and, in fact does, as we will discuss later in Section 4. Second, MPF is highly modular and can be used with any off-the-shelf predictors' without re-training them. This ability to polug-and-pley predictors facili- tates easier updates to the prediction module, e.g., adaptation to traffic laws in different geographical regions. Finally, MPF is backpropagatable which facilitates end-to-end data-driven training / fine tuning of the AV stack.
MPF includes three key components: learned trajectory predictor, rule-based trajectory predictor, and fuser (see Fig. 1(b)). We can use any learned trajectory predictor in MPF. For the rule-based predictor we design a motion planner that plans for the agent that we are predicting for according to a hierarchy of rules [4, 5, 6]. Rule hierarchies mirror the tendency of human drivers to relax lower priority rules (e.g.. speed limit) in favor of higher priority rules (e.g., collision avoidance), which which the key realizing realistic driving behaviors [7]. Furthermore, they can be encoded as a scalar reward that be used for fast online planning [6]. The fuser compares the performance of all predictors according to some task-specific metric (e.g.. displacement error, planning cost, etc.) and maintains belief distribution on the standalone predictors using 7-generalized Bayes updates [8].
Statement of Contributions. The contributions of this paper threefold: (i) We present new method for fusing learned and rule-based trajectory predictors such that they operate within the data regime that they are proficient at. (ii) We develop rule-based trajectory predictor using rule hierarchies [6] that achieves a level of performance comparable to Trajectron++ [1]. (iii) Finally, we demonstrate that MPF delivers more consistent performance across various prediction metrics and on multiple trajectory prediction datasets than the learning- and rule-based predictors alone.
2 Related Works
Rule-based Trajectory Prediction. Human behavior typically conforms to a set broad rules, customs, and norms. Thus, many works on trajectory prediction have relied on building rase-based models, with the underlying assumption that agents will in compliance with a specified rule set. Rules range from simple heuristics [9], enforcing physical constraints, rules encoding semantic traffic rules and norms, such as lane following [10]. We refer to [11] for a thorough survey on such methods. The prediction quality of such models is generally limited by the rigidity or incompleteness of rulesets. While recent work has demonstrated that appropriately designed rules can be competitive in structured environments [7], such models still struggle in corner cases.
Learning-based Trajectory Prediction. These challenges has led to a proliferation of learning- based approaches which aim to sidestep these issues by learning directly from datasets of observed agent behavior. In particular, many deep learning architectures have been developed for this task, varying in how they encode temporal sequences of data (e.g. using recurrent neural network (RNN) [12, 13] Transformer architectures [3]), how they reason about agent interactions (e.g. using pool- ing operations [12), graph neural networks (GNNs) [2. 1], or attention mechanisms [3]), and how
MPFF can be used for any number and type of predictors (learning- or rule- rule-based); however, in this paper, will limit our scope to fusing learnin- and rale-based predictor.

---

they represent their predictions (e.g. single deterministic prediction [12], probability distribution over future trajectories [1, 14], or a set of samples [15]). Much of the focus of this research area has been in ensuring that these models are able to condition on all available contextual cues, such as lane geometry and semantic map information. Advances in model architectures paired with the availability of high-quality large-scale datasets [16, 17, 18, 19] have led impressive performance, especially when evaluated held-out test splits.
Fusing Learned and Rule-based Trajectory Prediction. Pure learning baaning-based approaches can struggle in corner cases, producing predictions that fail to obey common-sense road rules, e.g.. drifting outside the road boundaries. To address such issues, recent work has considered fusing some rule-driven structure into learning-based trajectory prediction models. One style of approach is to first generate set of candidate plans that satisfy kinematic feasibility and semantic traffic rules via explicit rules-based motion planning, before using a learned model to subselect from these fixed "anchors" to produce predictions [20, 21J. Alternatively, [22] consider applying rules, encoded in the form of signal-temporal-logii specifications, as a post-processing step to "correct" the predic- tions made by a learning- -ased model. These approaches connect learned prediction and rule-based reasoning in series; in contrast, we propose architecture which puts the two in parallel, dynam- ically switching between the two a per-agent basis. As such, it is trivial to alter one component (e.g. adjusting the rules for the rule-based model) without retraining the bearning-based components. The closest work in the literature to our parallel architecture is [23]. However, [23] uses a different approach than us to fuse the learning- and planning-base predictors. Furthermore, unlike us, [23] assumes a priori availability the agent's 's ground-truth intent; we present a thorough comparison with the fusion approach in [23, Section II-F] in Appendix C provided in the supplementary.
Recursive Bayesian Multi-Model Filtering. Our approach builds on core ideas of recursive Bayesian filtering, cornerstone of probabilistic robotics. In particular, our approach is similar in spirit to early work in human trajectory forecasting which leveraged an interacting multiple mod- els (IMM) filter [24, 25] fuse predictions between basic single-rule models (constant velocity, constant acceleration, constant turning) [26]. Similar ideas have been applied for vehicle prediction, fusing simple kinodynamics based predictions with semantic map-based predictions (lane follow- ing, lane changing, etc.) [27]. In this work, we take a similar approach to fuse predictions from a learning- based model with a novel, more-expressive, rule-based prediction model.
3 Multi-Predictor Fusion
In this section, we will discuss the key building blocks of MPF, shown in Fig. 1(b). First, we define our notation: Let x E X' be the state of the traffic agent for which we are predicting and y E y be the joint state of traffic agents in the scene. Let 7 and yestr be the discrete-time state trajectories over some time duration from t to + 7 that lie in the space denoted by X" and y" respectively. The scene map is described by vector m that lies in the space of map features M.
3.1 Learning- Based Trajectory Predictor
Learning-based trajectory predictors, denoted by 4, typically take the form of generative networks that produce probability distribution of length T. conditioned on the historical state trajectories of all agents in the scene and the map. For the sake of exposition, we have included the map information in the learning- predictor, however, this is not strictly needed; learned predictors that do not use map information can also be used our approach. At each prediction run, we sample N trajectories from the learned predictor, ie., { IN ~ P(z++14471e |N: Hit, m, l), and pass them to the fuser, as shown in Fig. 1(b).
on future trajectories over a horizon
3.2 Rule-based Trajectory Predictor
To complement the learned predictor, we propose rule-based trajectory prediction model which predicts agent behavior based specified traffic code. Importantly, rather than forcing agents to satisfy strict set of rules, we specify this code in the form of a rule-hierarchy, allowing agents to relax satisfaction of less important rules (e.g., speed limit) if needed to ensure satisfaction of more important rules (e.g. collision avoidance). This flexibility allows for handling edge-case scenarios
3

---

in scalable manner due to its ability to naturally adapt to exceptional scenarios without additional supervision and has been shown to produce realistic driving behaviors [7].
Rules. We express the rules that we want vehicle to satisfy as boolean expressions Φ A" x y x M + (True, False} in the form of Signal Temporal Logic (STL) [28] using STLCG [29]. Each STL rule is equipped with a robustness metrie X" y x M - R which returns positive values if the rule is satisfied and negative values otherwise; larger positive robustness values indicate greater satisfaction of the rule while smaller negative values indicate greater violation.
Rule Hierarchy. A rule hierarchy := { is defined as a sequence of rules indexed in decreasing order of importance. The robustness vector p E RR of the rule hierarchy is defined as the vector p = (p1, (P1,-.. :.. Pa)) of the individual robustness of each rule Φ;. The rule hierarchy induces a ranking of trajectories in accordance with the importance of the rules they satisfy; trajectories that satisfy more important rules receive superior rank than those that satisfy less important rules. For instance, for a 2-rule hierarchy {Ø: {Ø:) j trajectories that: satisfy both rules have rank 1, satisfy Φ1 but violate Φ2 have rank 2, satisfy Φ2 but violate Φ1 have rank 3, and violate both rules have rank 4; see [6. Defintion 1] for a formal definition of the rank of a trajectory.
Planner. The planner first chooses the nearest lane to the agent (in terms of the Cartesian distance as well as the orientation) as the lane for the agent to follow; the desired lane is denoted by the black dashed line in Figs. 2(a) and 2(b). The rule hierarchy we use in this paper is presented in Fig. 2(c) which contains four rules in decreasing importance: collision avoidance, follow the center polyline of the lane to be followed, orient along the center polyline of the lane to be followed, and speed limit. The planner tasked with selecting trajectories with the highest possible rank. This can be achieved efficiently by leveraging the preservin reward function R p + R(p) R of rule hierarchy that embodies the property: trajectories with higher rank receive a higher reward than trajectories with a lower rank. We borrow the rank-preserving reward function from [6, Theorem l]:
1 R(p) Σ a an- + -Pkk
(I)
where a > 2. We generate a trajectory tree with K branches { K #1t14r } using splines [14] the same manner as described in [30] and compute the rank- lank-preserving rewards (R,,..., ] for each trajectory; trajectory tree generation and reward computation are paral- lelized on the GPU. The trajectory tree and the rewards of the branches can be visualized in Figs, 2(a) and 2(b); trajectory tree branches with warmer colors receive higher reward while branches with cooler colors receive a lower reward. All vehicles, other than the one for which we are predicting, are assumed to move with constant velocity over the prediction horizon for computing the reward. We adopt the constant-velocity prediction model for two main reasons: (i) Its computational effi- ciency allows batching predictions for all traffic agents, enabling fast online runtimes; see Section 4. (ii) The constant-velocity predictor has been shown to perform reasonably well [31].
Trajectory Prediction. Rather than choosing the trajectory with the highest reward, we transform the trajectory tree to a discrete Boltzmann distribution ((#441447|M-- m, by viewing the rewards as the negative of the Boltzmann energy as follows:
exp(R4/ (sp(Re/C)
P: = ΣK exp(Rx/()
(2)
where Ç > O is the temperature of the Boltzmann distribution, controlling the degree of opti- mality we expect from real-world agents with respect to the chosen rule hierarchy. At each pre- diction run, we sample N trajectories from the rule-hierarchy predictor, i.e., (#24144T,) IN ~ P(#41477M6- r) and pass them the fuser, as shown in Fig. 1(b).
3.3 Fuser
To fuse the learned trajectory predictor with the rule-based predictor, we propose recursive Bayesian filtering scheme, similar to the interacting multiple model (IMM) filter. At a high level, approach maintains belief over which of the learned and mule-based models best fits particular agent's 's behavior, and fuses their predictions according to this belief. During runtime, as we observe

---

Avoid
Pasyline Poaition
Orient Lane Polyline
Heading
40
20
Obsarve Speed
2.0E
400
(a) Straight Road
(b) Turn
Rules
Figure 2: Demonstration of the rule-hierarchy planner. In the first two figures, the pink vehicle is the agent we are planning for. (a) Planning on straight road. (b) Planning on a turn. (c) Rule hierarchy used in this paper.
agent behavior, we measure the performance of each individual model, and use this estimate to re- cursively update this belief. In the rest of this section, we will discuss the mechanics of the belief update and the building blocks of the fuser (shown in Fig. ((b) in more detail.
3.3.1 Performance Metric
[0, oo) can be any function that assesses how well the 1N observed state of the agent F++11 conforms to the past predictions 1./++,, and ( = This block, effectively, serves as a run-time monitor on the "quality" of both the individual predictors. Examples of performance metries include average displacement error, final displacement error, like- lihood of kernel density estimate, downstream planning cost, or any other task task-specific metrie.
The performance metric I X x X
3.3.2 Belief Update
We follow Bayesian approach to update the belief distribution. At the beginning of each episode choose prior belief bn. e.g., an uninformed prior 0.5), that reflects our initial knowledge about the correctness of the two models. Each belief update consists of two parts: (i) the observation step, and (ii) the mixing step. In the observation step, we collect the observed agent state 24+1, and use Bayes rule to obtain the conditional probabilities
HHHHH)) m)) P( Bt- mO, m,,)). 1)6
1/-1 P(1 Te+1: -
(3)
P(xe+1)
14 = P(r 1e+1, Ut- Ut- m) P(x++1
Dt H:t 1b P((++1)
(4)
Using the above, express 6/41 in terms of bi follows:
P(a Mt Hit m, 1)bb
1/41 = ab/+1
where a
(5)
P(r++1 W:tH:t. m. r)b;
the ratio of the likelihoods. In practice, we only obtain samples from each trajectory predictor, and cannot directly evaluate likelihood of the trajectories?. Instead, we can estimate the likelihoods in by using the performance metrics I = r((=1+1, ) 1x+1) and [" := I([ Te++)
as the likelihood of observing Tz+1, i.c., a = 11/1". Intuitively, a high performance metric corre- sponds the predictor doing well at predicting the agent trajectory, and hence a high performance metric should correspond the only condition need to ensure about I is that it should be integrable over its domain. This can always be achieved for positive performance metrics (e.g.. ADE) by defining I' the compo- sition of the performance metric with negative exponential. Rather than using the standard Bayes update directly, we use the ryggeerrllized Bayes update [8] here which introduces a scalar parameter E (0, 1) in (5):
high likelihood of observing the agent's trajectory. Mathematically,
by = a ab
(6)
We found that non parametric likelihood estimation, e.g. using kernel density estimation, led to numerical instability
5

---

The parameter is called the learning rate and controls how fast the belief is updated on encountering new "evidence", i.e., Tt+1- Choosing an < has been shown to produce better performance when the probabilistic mechanism that generates the evidence inconsistent with all models [8]; indeed, this is likely the case in real-world trajectory predictions, because no learned or rule-based model can describe all real-world agent behavior.
After factoring in the observation, we next account for the chance that the optimal model for a particular agent might switch, e.g. if an agent switches their behavior. To account for this, we update the belief by mixing with the prior, analogous to the mixing step in IMM. Thus, the full belief update which perform is given by:
b = (1-
1 I
+ 700-
where y E (0, 1) is small positive constant close to zero (e.g.. 0.1) representing the chance that the agent behavior switches at any given timestep. Note that this corresponds to geometric distribution over the duration that a given model describes agent behavior, the mean time between switching corresponds to 1/7
3.3.3 Belief Sampler
Finally, to make predictions, we sample predictions from both models according to the current be- lief. As shown in Fig. 1(b), the belief sampler receives a set of N trajectories from each model, i.e., (1441a47,)
(eG++11++/Wtt-H m,,) and { The belief sampler makes N draws from the belief distribution b to identify which predictor to sample from. Let the number of draws from the learned predictor be N' and the number of draws from the rule-based predictor be N", where N + N = N. Then, the belief sampler draws N trajectories from {=t+ 1a+,,,) and N" trajectories from { IN to construct the predic- tions of MPF. These trajectory can either be re-sampled uniformly N" trajectories from { J or 1=+++ J.W respectively, since the trajectories drawn from the predictors are independent identically distributed (i.i.d);; we use the latter approach.
P(3++1=-//Mtts m, r).
choose the first N' or
A complete algorithm for MPF is provided in Algorithm 1 in Appendix A.
4 Experimental Evaluation
In this section, we demonstrate MPF's ability to deliver consistent performance across predic- tion metrics and datasets. All experiments were conducted on desktop computer with AMD Threadripper Pro 3975WX CPU and an NVIDIA RTX 3090 GPU.
Datasets. We test MPF on three AV datasets: nuPlan- minan [32], nuScenes [18], and Lyft [17]. In particular, use the validation splits of each of these datasets. For muPlan-mint and nuScenes use the entire validation dataset while for Lyft we use 10% of the validation dataset. The data is loaded in an agent-centric manner using trajdata [19]. The number of prediction scenes (scene information for predicting the trajectory of an agent) and episodes (continuous sequence of agent trajectory rollout) for each dataset are provided in Table 2 in Appendix B.2. Predictors. In the results, we compare Trajectron++ [1], the rule hierarchy (RH) predictor that we developed in Section 3.2, and MPF that probabilistically fuses both the former models using the Bayesian belief approach discussed in Section 3.3.2.
Metrics. For each frame and agent in the dataset, we sample N trajectory predictions from each predictor denoted by (2 (where the superscript p denotes the predictor from which the trajectories are sampled). Let ( be the ground truth future trajectory of the agent for which we are predicting. We compute the following metrics: (i) Average Displacement Error (ADE) is average of the mean distance error across all N predicted trajectories and the ground truth 1 EM ΣT=1 ||= - IItT|l- (ii) Final Displacement Error (FDE) is average of the distance error between the last time step of all N predicted trajectories and the last time- step of the ground truth future trajectory: + D || Iz+T|l- (ii) minADE is the small- est mean distance error across all N predicted trajectories and the ground truth future trajectory: min N $ ET= || Iettll |l. (iv) minFDE is the smallest distance error between the last
1N
future trajectory: NT

---

NuPlan mini
Predictor
ADE
FDE
Mean (m)
minADE
minFDE
ADE
FDE
CVaRa (m)
minADE
minFDE
Mean Dial From Bosi (MDB) (98)
Tojje RIT (ound)
2.38 +
8.01
5:51 0.01
0.87 + 0.01
1,77
0.0
5-22 0.07
11:10 0.10
3.05 0.0
4.45++0..
30 64
0.02
2.12
0.00
XIF (oUI))
96 0.01
0.00
0.80
0.01
10.04 0.06
1.74 0.01
3.41
NuScenes
Predictor
ADE
FDF
Mean (m)
minADE
minFDE
ADE
FDE
CVaRau (m)
minADE
minRDE
Mean Diff From Best (MDB) (%)
Trije
2.40 D.D
5.48
0.02
0.8 = 0:00
106
0.01
4.56
0.04
10
0.07
2.16 = D1G
3.49 0.05
18 61
RH Pued (o00)) (oun))
1.82 0.01
424
0.03
0.95
2.05
0.00
5.32
0.05
11.76
0.11
4.07
0.06
0.11 0:111:0:11
0.01
0.01
0.53
0.01
0.04
10:34
0.08
204 0.03
3.73
Lyft
Predictor
ADE
FDE
Mean (m)
minADE
minFDE
ADE
FDE
CVaR, (m)
minADE
minFDE
Mean Diff From Best (MDB) (%6)
Trje-
2 78 + 0.00
6.4
0.01
105 - 0.00
1.30
0.00
6.23
0.01
20
0.03
3.12 - 0.1
5.07 + 0.02
& B- 79.09
Ri Pool (oue))
2.36 0.01
5.36
0.01
1.53
= 6.00
3.30
0.01
7.94
0.02
18.23 0.05
6.54 = 0.02
12.93 1 0.06
(oun)
0.00
0.00
1.25
0.00
2.87 0.01
0.02
Table 1: Results for 4 second prediction horizon with N = 20 rounded to the nearest second decimal.
time step of all N predicted trajectories and the last time-step of the ground truth future trajectory: min || Z++rl.- For all the above metrics, we report their mean and the 0.1-level conditional value-at-risk (CVaR) [33, 34] across all prediction scenes. CVaRo.1 is the mean of the top-10% tail of the histogram for each metric and provides insights about the "tail" performance.
The last metric we present the Mean Difference from Best (MDB) which combines all the above metrics to pro- vide a comparative performance measure. MDB com- putes the average percentage difference between a pre- dictor's performance on a metric from that of the best per- forming predictor on the same metric. Let there be / met- rics for each predictor p. denoted by 1ny, and let n be the metric value for the best performing predictor on that met- ric. Then, MDB is computed as: J ビド -n- 100. If predictor is the best at all metrics, then it will have an MDB of 0%. We report MDB across all eight met- rics discussed above, i.c., the mean and CVaRo.1 of ADE, minADE, FDE, and minFDE.
-20
-10
"
-00
x00
Figure 3: Incorrect lane choice.
Results and Discussion. We report the performance of our predictors on all datasets in Table along with 95% confidence intervals. MPF performs well on various metrics across all datasets. In particular, MPF demonstrates the lowest MDB across all datasets, highlighting its consistency across all metrics: Low MDB indicates that even if MPF is not the best predictor on a metric, is least close to the best performing predietor. For example, MPF does not have the best Mean ADE and FDE in all the three tables, but its performance is closer to RH predictor's 's than that of Trajectron++.
Remarkably, the handcrafted RH predictor with very simple rule hierarchy (Fig. 2(c)) has per- formance comparable to Trajectron++: in fact, MPF prefers RH predictor over Trajectron++, as indicated by the average belief for the RH predictor being greater than 0.5 across all datasets in Table 4 provided in Appendix B.4. RH predictor has the best mean ADE and mean FDE among all predictors, but has a higher mean minADE and mean minFDE. The violin plots in Fig. 4 plot the spread of ADE, FDE, minADE, and minFDE across all scenes in each dataset. In Fig 4 we observe that Trajectron++ yields metrics which follow a unimodal distribution smoothly tapering to the tail. In contrast, the RH predictor performs well on majority of scenes large mode near lower metric values), but also has a distinct mode with poor performance. This behavior can be attributed the RH predictor's is limited ability to reason about multimodality in the agent's 's goal. As an example, in Fig. 3, the RH planner assumes the agent aims to follow the lane corresponding to the black dashed curve which turns right. However, in this case there are other plausible options as well, including the ground truth in this scenario where the agent turns to the left, shown in the pink curve. By switching between the Trajectron++ and RH predictors, we are able to take advantage of the RH predictor's superior performance when it predicts the correct lane line follow and Trajectron++'s better per- formance when the RH predictor gets the desired lane line wrong. The improvement by MPF can be visually observed in Fig, 4 where the distribution for MPF mitigates the second mode of poor per-
7

---

Traj++
Troj+
Traj-+
RH Pred
RH Fred
RH Pred
RH Pred
ADE (m)
FDE (m)
minADE (m)
minFDÉ (i))
nuPlan mini
Traj-+
Tra++
Traf+
Tra--+
RH
MPT
NPT
MPT
NPT
ADE (m)
minADE
(b) nuScenes
Trajen
Trajet
Trifet
Trajet
RH Poad
RH Pred
RH Pred
RH Pred
MPF
MPF
MPF
ADE (m)
FDE (m)
minADE (m)
minFDE (I))
(c) Lyft
Figure 4: Violin plots for visualizing the spread of the metrics the datasets for all three predictors.
formance by the RH predictor and shifts the entire distribution towards lower metric values relative to Trajectron++. Furthermore, we observe that our 7-generalized Bayes filter approach performs better than the seisholled--ikklihoood approach in [23]; more details are provided in Appendix C.
Runtime Considerations. The average time for MPF to predict for a single agent at a given time- step 0.08665 seconds. Of this, the bulk of the time is spent querying the standalone predictors and very minuscule amount of time is spent on the fuser: on average, 0.06994 seconds are spent on querying trajectories from the rule hierarchy predictor, 0.01671 seconds from Trajectron+++, and 0.013 milliseconds the belief update and re-sampling from the fuser. The speed of MPF can be attributed to the high degree of parallelization on the GPU. There is scope for further computational speed improvement by parallelizing the querying of the standalone predictors.
Conclusion, Limitations, and Future Work
Conclusion. In this work, we demonstrated that cost function constructed by encoding traffic code in the form a rule-hierarchy can be leveraged to produce competitive future predictions for vehicles in traffic. Furthermore, we proposed MPF, a method to fuse the predictions from the rule-hierarchy planner with predictions from a learned trajectory forecaster to yield prediction framework which plays to the complementary strengths of each model. We evaluated the fusion approach real-world trajectory prediction datasets and demonstrated the yields improved predic- tions relative to learned baseline across suite of metrics.
Limitations. Our approach has two main limitations. The first limitation is the additional compu- tational expense of querying an ensemble of predictors compared to rumning single predictor. This challenge can be mitigated by parallelizing the predictor querying: however, it still involves some parallelization overhead and greater memory requirement on the GPU. The second limitation of MPF is the lack of multimodal behavior exhibited by the RH predictor, as discussed in Fig. 3. Fi- nally, we remark that MPF shares the limitation of all agent-centric predictors: the need to parallelize predictions queries across all agents to ensure feasible computational run times.
Future Work. This work provides various exciting opportunities for future work. On the practical front, we will explore methods, such as [21]. for introducing better multimodal behaviors for the planner in the rule hierarchy predictor, perform closed-loop evaluation of MPF, parallelize predic- tions for all agents in scene, and develop more detailed rule hierarchy. On the theoretical front, will explore the use of online learning techniques that provide convergence guarantees for the belief distribution.

---

Acknowledgments
We thank Boris Ivanovic for helpful discussions trajectory prediction and Yuxiao Chen for his help with spline trajectory tree generation and lane planning.
References
[1] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++:: Dynamically-feasin trajectory forecasting with heterogeneous data. In Proc. European Conf. on Computer Vision, pages 683-700. Springer, 2020.
[2] Y. Chen, Ivanovic, and M. Pavone. Scept: Scene-consistent, policy-based trajectory predic- tions for planning. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition, pages 17103-17112, 2022.
[3] Y. Yuan, X. Weng, Y. Ou, and K. M. Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting. Proc. IEEE Int. Conf. on Computer Vision, pages 9813 9823, 2021.
[4] A. Censi, K. Slutsky, T. Wongpiromsarn, D. Yershov, S. Pendleton, J. Fu, and E. Frazzoli. Liability, ethics, and culture-aware behavior specification using rulebooks. In Proc. IEEE Conf. on Robotics and Automation, pages 8536-8544 2019.
[5] W. Xiao, N. Mehdipour, Collin, A. Bin-Nun, E. Frazzoli, R. D. Tebbens, and C. Belta. Rule-based optimal control for autonomous driving. In Proc. ACM/IEEE Int. Conf. on Cyber- Physical Systems, pages 143-154, 2021.
[6] S. Veer, K. Leung, R. Cosner, Y. Chen, Karkus, and M. Pavone. Receding horizon planning with rule hierarchies for autonomous vehicles. arXiv preprint arXiv:2212.033223 2022.
[7] B. Hclou, A. Dusi, A. Collin, N. Mchdipour, Z. Chen, C. Lizarazo, C. Belta, T. Wongpirom- sarn, R. D. Tebbens, and O. Beijbom. The reasonable crowd: Towards evidence-basee and interpretable models of driving behavior. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots & Systems, pages 7708-6715 2021.
[8] Grünwald and T. van Ommen. Inconsistency of bayesian inference for misspecified linear models, and proposal for repairing it. Bayesian Analysis, 12(4):1099-1103, 2017.
[9] D. Helbing and P. Molnar. Social force model for pedestrian dynamics. Physical review E, (5):4282, 1995.
[10] A. Houenou, P. Bonnifait, V. Cherfaoui, and W. Yao. Vehicle trajectory prediction based on motion model and maneuver recognition. In Proc. JEEE/RSJ Int. Conf. on Intelligent Robots & Systems, pages 4363-4369, IEEE, 2013.
[11] Lefèvre, D. Vasquez, and Laugier. survey on motion prediction and risk assessment for intelligent vehicles. ROBOMECH journal, 1(1):1-14, 2014.
[12] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social LSTM: Human trajectory prediction in crowded spaces. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pages 961-971, 2016.
[13] A. Kamenev, L. Wang, O. B. Bohan, I. Kulkarni, B. Kartal, A. Molchanov, S. Birchfield, D. Nistér, and N. Smolyanskiy. Predictionnet: Real-time joint probabilistic traffic prediction for planning, control, and simulation. In Proc. IEEE Conf. on Robotics and Automation, pages 8936-8942, 2022.
[14] E. Schmerling, K. Leung. W. Vollprecht, and M. Pavone. Multimodal probabilistic model- based planning for human-robou interaction. In Proc. IEEE Conf. Robotics and Automation, pages 3399-3406, 2018.

---

[15] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi. Social GAN: Socially acceptable trajectories with generative adversarial networks. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pages 2255- 2264, 2018.
[16] Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Chai, B. Caine, et al. Scalability perception for autonomous driving: Waymo open dataset. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pages
2020.
[17] Houston, G. Zuidhof, L. Bergamini, Ye, Chen, A. Jain, Omari, V. Iglovikov, and Ondruska. One thousand and one hours: Self-driving motion prediction dataset. In Proc. Conf. Robot Learning, pages 409-418. PMLR, 2021.
[18] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, E. Liong, Q Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pages 1 11621-11631, 2020.
[19] B. Ivanovic. trajdata: A unified interface to many trajectory forecasting datasets, June 2022.
URL https: http////ttttu....// www/noomoooeeeeeecee com/nvr-avg/brajddet avg/trajdata.
[20] H. Song, D. Luan, W. Ding, M. Y. Wang, and Q. Chen. Learning to predict vehicle trajectories with model-based planning. In Proc. Conf. on Robot Learning, pages 1035-1045. PMIR, 2022.
[21] D. Li, Q. Zhang, Z. Xia, K. Zhang, M. Yi, W. Jin, and D. Zhao. Planning-inspired hierarchical trajectory prediction for autonomous driving, arXiv preprint arXiv:2304.1 1295, 2023.
[22] X. Li, G. Rosman, Gilitschenski, DeCastro, C.-I. Vasile, Karaman, and D. Rus. Dif- ferentiable logic layer for rule guided trajectory prediction. Proc. Conf. on Robot Learning, pages 2178-2194. PMLR, 2021.
[23] L. Sun, X. Jia, and A. Dragan. On complementing end-to-end human behavior predictors with planning. arXiv preprint 103 2021.
[24] H. A. P. Blom. An efficient filter for abruptly changing systems. In Proc. IEEE Conf. on Decision and Control, pages 656 656-658, 1984.
[25] H. A. P. Blom. An efficient decision-making free filter for processes with abrupt changes. IFAC Proceedings Volumes, (8(5):631-636 1985.
[26] N. Schneider and D. M. Gavrila. Pedestrian path prediction with recursive bayesian filters: A comparative study. In Proc. German Conf. on Pattern Recognition, pages 174-183. Springer, 2013.
[27] H. Gao, Y. Qin, C. Hu, Y. Liu, and K. Li. An interacting multiple model for trajectory pre- diction of intelligent vehicles in typical road traffic scenario. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[28] O. Maler and D. Nickovic. Monitoring temporal properties of continuous signals. In Y. Lakhnech and S. Yovine, editors, Formal Techniques, Modelling and Analysis of Timed and Tolt--oolerant Systems, pages 152-166. Springer Berlin Heidelberg, 2004.
[29] K. Leung, N. Aréchiga, and M. Pavone. Back Back-propagation through signal temporal logic specifications: Infusing logical structure into gradient-basse based methods. arXiv preprint arXiv:2008.00097 2020.
[30] Y. Chen, P. Karkus. B. Ivanovic, X. Weng, and M. Pavone. Tree-structured policy planning with learned behavior models. arXiv preprint 11199 2023.
10

---

[31] C. Schöller, V. Aravantinos, F. Lay. and A. Knoll. What the constant velocity model can teach
us about pedestrian motion prediction. IEEE Robotics and Automation Letters, 5(2): (22):666) 1696- 1703, 2020.
[32] H. Caesar, J. Kabzan, K. S. Tan, W. K. Fong, E. Wolff, A. Lang, L. Fletcher, O. Beijbom, and S. Omari. nuPlan: A closed-loop ml-based planning benchmark for autonomous vehicles. In Proc. IEEE Conf. on Computer Vision and Pultern Recognition ADP3 workshop. 2021.
[33] A. Majumdar and M. Pavone. How should robot assess risk? towards axiomatic theory of risk in robotics. In Proc. Int. Symp. on Robotics Research, pages 75-84. Springer, 2020.
[34] R. Rockafellar, S. Uryasev, al. Optimization of conditional value-at-risk. Journal of Risk, 2:21-42, 2000.

---

A Algorithm for MPF
The algorithm for MPF is provided in Alg. 1. We the agent-centric dataloader generated by trajdata [19] to access various prediction datasets ([32, 18, 17]) in a unified representation. In lines 15-16, trajectories from the standalone predictors are sampled, and in lines 8-13, the belief update discussed in Section 3.3.2 is performed.
Algorithm Multi- Predictor Fusion (MPF)
1: Input: learn predictorrrroo rule wullepredicctnn ctorr dataloader (agent-centric)
2: Output: MPF + O (list of MPF predictions)
3: Hyperparameters: numher of trajectory samples per scene N, prediction horizon T, Bayes learning rate convex combination factor with prior in (7), belief prior bo.
4: for episode in dataloader do
+ b <-
6: learn. mearnppeeictor..oo hittoorrrsttttt-ooo 0, celleprredictor.hsttrry--- 0.
7:
for scene in episode do
8:
if 0 then
3t observe.current current agett..gee..... (gent.state()
10:
fatelihhoddlearae + get.likelihood (learn peeeirnrredictor prestctor.history, x+)
11:
likelihood rule < get Likelihoodrrula (ru.s__rediictor predictor history, m.)
12:
+ (likelihood learneb htt. woodee
13:
new_belief 1+o
14:
b-- (1 - 7) new belief + 76p
15:
end if
(#i-14TT)) - wwt.:ppredicoor.smleeeeeeeeeeeee NN
17:
(z I/ + rule prwdictorr...... wwwple (scene,, N)
18:
learn searn_predicto historyy + (1(+1,)
19:
mule.predictor.histte + (x 11
20:
MPFd www.addeeeleeeee.eseeeee (fraatt ++=1
LN
21:
22:
end for
23:
end for
24:
return pred_
B Further Details on Experimental Evaluation
B.1 Hyperparameters
In our experimental evaluation set the hyperparameters in line 3 of Alg. as N 20. T 8, 0.1, = 0.1, and an uninformed prior by [0.5, 0.5]. The time dt between two timesteps is 0.5 seconds.
B.2 Dataset Details
The number of scenes and episodes per dataset are provided in Table below:
Dataset
Scenes
Episodes
nuPlan-man mini val
118030
6814
nuScenes val
66620
4140
Lyft val 10%
744448
49196
Table 2: Dataset Details
B.3 7) and study
We first set n = 0.1 and vary 7 (7). The mean difference from best (MDB) for MPF, compared against Trajectron++ and RH Predictor, for different choices of 7 are reported the left-hand side of Table 3. As can be noted from the last row, the performance of MPF is not very sensitive to the choice of Then we choose the smallest best performing (i.e., 0.1) and sweep across four values
12

---

of 1). The results for that are reported on the right-hand side of Table 3. With 1y we see a clear trend that smaller 1) perform better than larger when MDB is averaged across all datasets. A large updates the belief more aggressively based on the past evidence, potentially making more mistakes compared to a smaller y that results more cautious belief updates.
MDB (Sb)
(
0.1
MDB
(5)
Bhy 0.1
Dataset
@-y
0.05
@-
@
0.15
@*y
0.20
@t = 0.1
@q
@q
@q
1.0
NuPlan-mini
4.31
4.36
4.41
4.46
4.36
4.29
4.18
3.72
NuSecnes
6.19
5.17
5.16
5.16
5.17
5.23
5.12
6.45
Lyft
4.32
4.28
4.24
4.21
4.28
4.53
5.18
8.68
Avg. MDB Datasets
1.61
4.60
4.60
1.61
4.60
4.68
4.93
6.28
Table 3: MDB for different yj and ry in Bayes update.
B.4
Belief statistics
For the MPF results presented in Table 1, we additionally provide statistics on the belief distribution between the RH predictor and Trajectron++ predictor greater than 0.5, indicating that on-average, MPF relies more on RH predictor than Trajectron++ across all datasets. We note that the standard deviation for the belief of both predictors is always the same because they related by b' Trajectron++ and RH predictor, respectively.
Table 4. We observe that the mean belief for RH
b", where B' and b' the belief for
Dataset
Predictor
Mean
Std
Min
Max
RH Pred
0.53
0.06
0.26
0.95
NuPlan-mini
Traj++
0.47
0.06
0.05
0.74
RH Pred
0.54
0.03
0.07
0.92
NuScenes
Traj++
0.46
0.03
0.08
0.93
RH Pred
0.54
0.05
0.05
0.94
Lyft
Traj++
0.46
0.05
0.06
0.95
Table Belief statistics for MPF results in Table
C Comparison with [23]
In this section we compare the performance of our 77-generalized Bayes filter used MPF with the thresholded-ikkllioodd approach in [23, Section II-F]. We compute the likelihood of the observation given Trajectron++, i.e., PP 3t-H:; m. and switch the RH predictor when it dips below
threshold as follows:
RH Predictor, P( yt-H.t: m, < v, Trajectron++, otherwise
(8)
Since it is not clear what the best threshold should be, we sweep // across 0.1, 0.2, 0.3, 0.4, and 0.5 and report all of them Table To further mimic the approach in [23], we use the same historical buffer window (i.e., time delay switching) of 3-seconds; [23] uses 30-timesteps which corresponds to a 3-second window according to their dataset.
In Table 5, we notice that [23] serves as strong baseline which results mixed model that does better than Trajectron++ and RH predictor on many metrics (compare with Table 1). However, overall MPF outperforms the switching approach of [23], as summarized by the lowest MDB for all three datasets in Table The reason behind this is MPF's ability to mix trajectories at every time-step which allows it rapidly respond to performance degradation by predictor. MPF is able to achieve this because it maintains belief distribution over the standalone predictors, unlike [23].
In Table 5, MDB is computed across six predictors: five predictors based on [23] with varying and MPF.
13

---

NuPlan mini
Predictor
Mcan (m))
CVaiRaa (m)
Mcan From
(MDB)
01.01
0.78
0.00
4.51
-.044
9.88
3.44 D.06
16.42
2.27 1: 0.01
5.18
0.01
0.71 1: 0.00
0.97
1.42
0.00
9.77
0.06
2.17 0.7.2.000 0.01
3.12 3: 0.06
15.26
2 26
0.01
3.16
11160000 0.01
0.17 + 0.00
0.97
0.01
4.38
0.04
9.74
0.06
2.14
0.03
3.41 + 0.006
14.88
0.4
2.2 000622000 0.00
5.16
0.01
0.17
1: 0.00
0.97
- 0.01
1.36
0:04
9.74 0.06
2.13
2:222000 0.01
3.06
2: 11006
14 80
0.5
2.26 0:01
5.16
0.01
0.21
+: 0.00
0.98
0.01
4.37
0.04
9.78
0.007
2.15 0.04
3.35
4: 45.0000
15.
MPN louns)
1.95
4 13
0.02
0.54
4.34 0.02
10.04
0.06
0.03
40
NuScenes
Predictor
CVaR (m)
(MDB)
2:20 +: 0.01
5.08
0.02
0.81 J: 0.00
1.06 0.01
4.56
0.004
10.07
0.07
2.10 0.03
3.49 0.05
14.07
0.0
2.20
+ 0.01
1.48
0.02
0.81 + 0.00
1.06-+U00 0.01
4.56
0.04
10.05
= 0.01
2.17 117-+-0.00
3.50 + 1.005
14.
0.2
2.
0.01
3.08
0.02
0.52 0.000
1.07
2. 0.61
+.07
10.09
2: 0.07
2.18 2:2.00.0 0.03
3.53 4: 0.06
14:42
0.6
2.20
0.01
5.48
0.02
0.82 + 0.00
1.07
0.01
4.57
0.04
10.11
0.07
2.20 0.03
3.60
+: D.06
15:03
o-
2.41 0.01
5.50
0.02
0.80 0.01
1.12 0.01
4.64
10.26
0.08
2:31
3.94
+ 0.07
18.
(ouns)
2.00
0.02
0.54
0.01
4
10.34
0.088
2.04
0.03
3.73
0.06
Lyft
Predictor
2.78
0.00
6.03
0.01
1.05 1: 0.00
1.30 0.60
6:23
0.011
13.19 1 0.00
5.12 0.01
5.07 1: 0.02
6.00
2.78 + 0.00
6.03
0.01
1.05 + 0.00
1.31
0.00
6:23
+ 0.01
13:21 - 0.01
3.14 0.01
5.13 + 1.00
6:77
0.8
2. 1: 0.00
6.01
1: 0.01
1.00 1: 1..00
1.33
0.00
6.23
+: 0.01
13:25
2: 0.03
3.19 2: 0.01
5.31 4: 0.00
7.75
0- 0.8
78
0.00
6.06
0.01
1.07 + 0.00
1.40
0.01
6:33
0.02
13:37
0.03
3:18 0.01
5,83 4: 0.1G
11.47
0.5
0.00
1.12 0.00
1.57
0.59
0.02
14.31
7.22 1: 0.05
20.66
(oum)
Table 5: Results comparison between MPF and the Bayesian failure monitoring approach in [23, Section [[-F] for 4-second prediction horizon with N 20 rounded to the nearest second decimal.
14


Reviews:
[{"review_id": "rgGSHamBXD", "reviewer": "Reviewer_VeXK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "An algorithm called multi-predictor fusion (MPF) that augments the performance of learning-based predictors is presented. MPF probabilistically combines learning- and rule-based predictors, by essentially mixing trajectories from both standalone predictors in accordance with a belief distribution that reflects the online performance of each predictor. It is shown in the paper that MPF outperforms the two standalone predictors on various metrics and delivers the most consistent performance.", "summary_of_recommendation": "I have some background in the field and feel that the paper is of significant contribution to the filed and community.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "None", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I found that the paper is well developed, with clarity and sufficient contents, to describe the concept and to illustrate the performance of the proposed algorithm The topic is of great interest in the community, and the contributions of the paper are significant to the field.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "fFx8VLCZV4", "reviewer": "Reviewer_oqEg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposed a rule-based trajectory predictor using rule hierarchies that achieved comparable performance to Trajectron++. Then the authors proposed the multi-predictor fusion (MPF) algorithm, which fuses the learning- and rule-based predictors using recursive Bayesian multi-modal filtering. It was shown that MPF achieved more consistent performance across different prediction metrics than the learning- and rule-based predictors.", "summary_of_recommendation": "The most critical concern I have regarding this paper is the novelty and significance of the predictor fusion scheme. Thus, I lean towards rejecting this paper, given its current form. I am happy to raise my score if the authors can properly address my concerns during rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. The proposed recursive Bayesian inference scheme depends on both $\\gamma$ and $\\eta$. While the authors reported an ablation study on different values of $\\eta$ in Appendix B.3, there should be an ablation study on the effect of different $\\gamma$ values on the fusion performance. Besides, the authors claimed, \"The performance of MPF is not very sensitive to the choice of $\\eta$\". It is the case in NuPlan-mini and NuScenes, but the performance in Lyft has relatively large variation given different $eta$. Could the authors give some insights or hypotheses on it? \n\n2. In the rule-based predictor, it is assumed that \"All vehicles, other than the one for which we are predicting, are assumed to move with a constant velocity over the prediction horizon for computing the reward\". It is certainly a very big assumption. The authors should provide justification for this assumption. Also, does this assumption affect the prediction performance of RH prediction? Can any of those cases where RH has large prediction errors be explained by the violation of this assumption?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strength**: The proposed rule-based planner based on rule hierarchy is interesting. It is quite impressive that it achieved performance comparable to Trajectron++. \n\n**Weakness**:  A very relevant and crucially important prior work [1] was missing in the related work. In [1], various methods to fuse learning- and model-based prediction models were proposed and evaluated, including a recursive Bayesian inference scheme that is very similar to the one proposed in this work. Thus, I would not consider the idea of predictor fusion and the proposed fusion scheme, which are claimed as the main novelty of this work, as novel and significant. The authors should compare their work with [1] and justify the novelty of their work during rebuttal. \n\n[1] Sun, L., X. Jia, and A. Dragan. \"On Complementing End-To-End Human Behavior Predictors with Planning.\" Robotics science and systems (RSS). 2021.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "dil9ZJWcvM", "reviewer": "Reviewer_4qup", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": " The paper presents an algorithm called multi-predictor fusion (MPF) that enhances the performance of learning-based trajectory predictors for the safe and efficient planning of autonomous vehicles in highly interactive traffic scenarios. MPF combines learning- and rule-based predictors by mixing trajectories from both predictors in accordance with a belief distribution that reflects the online performance of each predictor. The proposed algorithm is shown to outperform the two standalone predictors on various metrics and delivers the most consistent performance in the results. The authors demonstrate that MPF augments the performance of learning-based predictors by imbuing them with motion planners that are tasked with satisfying logic-based rules.", "summary_of_recommendation": "Based on the strengths highlighted in the review, it appears that the paper proposes an interesting idea that may have an impact on real-world robotics applications. Therefore, I recommend accepting the paper for publication in CoRL. However, the weaknesses and concerns raised in the review should be addressed in the final version of the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* The literature review could benefit from more discussions on fusing learned and rule-based trajectory prediction to provide more context for the proposed method.\n* The paper should include more quantitative results on different datasets to demonstrate the effectiveness and generalization of the proposed method.\n* The computation expense of the proposed method may be a severe problem when applied to multi-agent motion prediction problems, and this limitation should be discussed in the paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n* The paper is well-written and easy to follow.\n* The proposed multi-predictor fusion module can be used with any off-the-shelf predictor, making it easily applicable to a variety of scenarios.\n* The idea of combining learning-based and rule-based motion predictors is beneficial to real-world robotics applications.\n\nWeaknesses:\n\n* Computation expense may be a key problem to solve for real-world robotics applications, as the proposed method may require significant computational resources.\n* The RH predictor does not exhibit multimodal behavior, which may limit its effectiveness in scenarios where multiple outcomes are possible.\n* The paper would be better to address how the rule-based motion predictor would perform in scenarios where an HD map is not available, such as in a large intersection. This limitation should be discussed in the paper.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a very interesting idea for a tile-based method for trajectory prediction that is comparable to trajectron++. All reviewers agreed on the strengths of the paper on the multi-predictor fusion module that can be used with any off-the-shelf predictor, making it a flexible solution with a possible impact on real-world robotics applications. The updated version of the paper has also integrated well most of the reviewers feedback. I would suggest that the authors rearrange some experiments in their camera ready version to better highlight their contributions and findings.
Decision: Accept (Poster)