Reasoning Multi-Agent Behavioral Topology for Interactive Autonomous Driving
Authors: Haochen Liu
Li Chen
Yu Qiao
Chen Lv
Hongyang Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Trajectory Prediction
Autonomous Driving
Primary area: other

Abstract: Autonomous driving system aims for safe and social-consistent driving through the behavioral integration among interactive agents. However, challenges remain due to multi-agent scene uncertainty and heterogeneous interaction. Current dense and sparse behavioral representations struggle with inefficiency and inconsistency in multi-agent modeling, leading to instability of collective behavioral patterns when integrating prediction and planning (IPP). To address this, we initiate a topological formation that serves as a compliant behavioral foreground to guide downstream trajectory generations. Specifically, we introduce Behavioral Topology (BeTop), a pivotal topological formulation that explicitly represents the consensual behavioral pattern among multi-agent future. BeTop is derived from braid theory to distill compliant interactive topology from multi-agent future trajectories. A synergistic learning framework (BeTopNet) supervised by BeTop facilitates the consistency of behavior prediction and planning within the predicted topology priors. Through imitative contingency learning, BeTop also effectively manages behavioral uncertainty for prediction and planning. Extensive verification on large-scale real-world datasets, including nuPlan and WOMD, demonstrates that BeTop achieves state-of-the-art performance in both prediction and planning tasks. Further validations on the proposed interactive scenario benchmark showcase planning compliance in interactive cases. Code and model is available at https://github.com/OpenDriveLab/BeTop.

Reasoning Multi-Agent Behavioral Topology for Interactive Autonomous Driving
Haochen Liul.2 Li 2.3
Yu Qiao²
Chen Ly
Hongyang Li
Nanyang Technological University
2 Shanghai Al Lab University of Hong Kong
Abstract
Autonomous driving system aims for safe and social-consistent driving through the behavioral integration among interactive agents. However, challenges re- main due to multi-agent scene uncertainty and heterogeneous interaction. Current dense and sparse behavioral representations struggle with inefficiency and incon- sistency in multi-agent modeling, leading to instability of collective behavioral patterns when integrating prediction and planning (IPP). address this, we ini- tiate topological formation that serves compliant behavioral foreground guide downstream trajectory generations. Specifically, we introduce Behavioral Topology (BeTop), pivotal topological formulation that explicitly represents the consensual behavioral pattern among multi-agent future. Be'Top is derived from braid theory distill compliant interactive topology from multi-agent future trajectories. synergistic learning framework (BeToopNet supervised by BeTTop facilitates the consistency of behavior prediction and planning within the predicted topology priors. Through imitative contingency learning, BeTTop also effectively manages behavioral uncertainty for prediction and planning. Extensive verification on large-scale real-world datasets, including nuPlan and WOMD, demonstrates that BeToop achieves state-of-Uue the-hhrrrt performance in both prediction and planning tasks. Further validations on the proposed interactive scenario benchmark show- case planning compliance in interactive cases. Code and model available https:// thub. com
Introduction
Autonomous driving system aspires to safe, humanoid, and socially compatible maneuvers [1]. This drives for formulation, prediction, and negotiation of collective future behaviors among interactive agents and autonomous vehicles (AVs) [2]. Remarkable accuracy is achieved by learning- based paradigms [3], including end-to-end modular design [4-7], social modeling [8, 9]. and trajectory- level integration [10-13]. However, substantial challenges arise in real-world cases due to scene uncertainty and volatile interactive patterns for multi-agent future behaviors.
To embrace compliant patterns for multi-agent future behaviors, current formulations fall into two mainstreams, dense representation and sparse representation (Fig. 1). Dense representation quan- tizes agent behaviors under ego-centric rasterization, forecasting bird's eye view (BEV) occupancy probabilities [14, 7, 15] or temporal flow [16-18]. It is easy to deduce interactions, perform scalable behaviors for agents [19], and align with BEV perceptions [20]. Still, dense representation hindered by frozen receptions. It causes safety- vulerrbllee intractability and occlusions potentially interacting with ego maneuvers [16, 21J. Contrary to pixel-wise behavioral probability, sparse representation forecasts agent-anchored set of trajectories [22-25] or intention distributions [10, 26, 27]. Its multi- modal formulation each agent marks the elasticity in diverse behavioral uncertainty and tractability under flexile spatial semanties. However, behavioral misalignment [28] and modality collapse [24]
Work done while Haochen's internship Shanghai AI Lab. Equal co-advising.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

BeTopNet
Top faride
Recvon
Driving acenario
Dorsa
Sprrse
Bcfop (Ouns]
Autonomous vahicle
Scena agents
Colltionn &
Overlop
Planning & Prediction
Braid crossings
Figure 1: Multi-agent Behavioral Formulation. (a) A typical driving scenario in Arizona, US [35]; (b) Dense representation conducts scalable occupancy prediction jointly, but restrained reception leads to unbounded collisions with planning; (c) Sparse supervision derives multi-agent trajectories with multi-modalities, while it struggles with conficts among integrated prediction and planning; (d) Tep reasons future topological behaviors for all scene-agents through braids theory, funneling interactive eventual agents (in highlighted colors) and guiding compliant joint prediction and planning.
impede compliant multi-agent modeling. requiring exponential computations with growth agent numbers [29]. Issues particularly result in unstable and slow behavioral learning when exposed to predictions and planning (IPP) [30]. Typical solutions by conditional prediction [31, 28, 32] or game- chme-theoretic reasoning [8, 33] often lead to nonstrategic maneuvers [34] due to non-compliant rollouts in adjusting interactive behaviors. This calls for a formuuullaionn for multi-au agent behaviors, which should stabilize collective behavioral patterns in compliant manner for IPP objectives.
The decision-making process for human drivers provides valuable insights. Humans primarily determine the future behavior of interacted agents for decision-making without relying on their specific states [2, 36]. Thus, effective strategy involves assessing agent- agent-wise behavioral impact on planning maneuvers, and reasoning about compliant interactions. Our fundamental insight is that compliant multi-agent behaviors exhibit topological formations, which can be identified by distilling consensual interactions from future behaviors. Prior works have approached this challenge through structural design [37, 38] or implicit relational learning [39-41] using GNNs [42] or Transformers [43]. Other studies quantify uncertainty by topological properties [44, 45]. Nevertheless, current literature is scarce formulating explicit future supervision of compliant multi aueti- behavioral patterns.
To this end, we launch the multi-agent behavior formulation termed as Behavioral Topology (BeToop). At its core, BeTopp explicitly forms the topological supervision of consensual multi-agent future interactions, and reasons to guide prediction and planning. BeToop stems from braid theory [46], which infers compliant interactions of multiple paths from the intertwining of their braids. This empowers Bo to intuitively distill forward intertwines (occupancy) as joint topology from braided multi-agent future trajectories (Fig. 1), marrying dense and sparse representation. With the aid of BeTTop, we introduce synergistic Transformer-base objectives. To implement, an iterative decoding strategy simultaneously reasons about Behavioral Topology and generates trajectory sets. Then the topology-guie -guided local attention, embedded in each decoder layer, selectively queries behavioral semantics from social-compliant agents within the predicted BeTTop priors. To further alleviate multi-agent uncertainty through topological guidance, contingency planning paradigm fitfully deployed. We lay out the imitative contingency learning process, which regulates the safety-ensured short-term plan. It maintains the long-range uncertainty by reasoned joint predictions from BeTop. Experimental results exhibit enhanced consistency and accuracy for prediction and planning in real-world scenarios. Testing in proposed interactive cases further highlights the planning ability of BeTopNet. To sum up, our contributions are three-fold:
learning stack, Be'TopNeet, for learning IPP
We bring in the concept of Behavioral Topology, a multi-agent behavioral formulation for topologi- cal reasoning that explicitly supervises consensual future interactions jointly for the IPP system.
A synergistic learning framework BeTopNet, offering joint planning and prediction guided by topology reasoning, is devised. Topology-guide local attention and imitative contingency planning could resolve scene compliance and multi-agent uncertainty.
Benchmarking on nuPlan [47] and WOMD [35], our approach demonstrates strong performance in both planning strategy and prediction accuracy. BeTopNet witnesses evident improvement

---

over previous counterparts, e.g., +7.9% in general planning score, +3.8% under interactive cases, +4.1% mAP for joint prediction, and +2.3% mAP for marginal prediction.
2 Related work
Multi-agent behavioral modeling. Carving the collective future behavior of diverse agents is imperative for socially-consistere driving maneuver. Farlier approaches centered around occupancy prediction [14, 5, 4, 48]. Forecasting the spatial presence under dense BEV representation [49, 20] offers flexibility of arbitrary agents [17, 50, 19] and alignment with perception [7, 51]. However, rigidity resolution induces scenario occlusion [16], rendering intractable occupancy [21]. In parallel, sparse representation consolidates multi-agent behavior into cohesive modalities across future trajectories [52, 22, 53, 54] or intentions [27, 10, 26, 55J. Joint future behaviors are derived through goal-based sampling recombination from marginal predictions [56-58]. However, the collection of joint modalities is susceptible to mode collapse and entails exponential complexity [29, 59]. Mcanwhile, topological representation has garnered traction motion primitives [60, 61] or tools [44, 45] for scenario quantification, yet topological properties delineating collective future behaviors remain largely unexplored. Our BeTTop targets the issue, marrying dense behavior probabilities by topology with the sparse motion from joint predictions to present structured future behaviors.
On the other hand, inconsistent communal agent behaviors have motivated leveraging future interac- tions. Implicit approaches obtain tacit interactions with attention [62, 32, 63] or GNN [24, 64, 37, 65] from final motion regressions. Nonetheless, the implicit supervisions are found inefficient in dynamic scenarios [22]. Contrarily, explicitly reasoning mutual behaviors by conditional factorization [31, 66),
relation reasoning [41, 40], or entropy-based methods [67, 68] offer consistent behavioral priors. However, hefty variance across agent dynamics and scenario geometries yield unstable inference. Distinguished from them, BeTop crafts compact topological supervision that stabilize future inter- actions among multi-agent behaviors. Derived from topological braids, BeTTop offers a topological equivalent behavioral representation to guide compliant forecasting and planning.
Integrated prediction and planning. IPP system aims to harmonize trajectory- baased learning of future interactive behaviors between the ego vehicle and social agents. Rule-based approaches [69-71] integrate handcrafted future interactions to evaluate candidate planning profiles, offering remarkable outcomes in rule-powered reactive simulation [47]. Still, the absence of real- world behaviors exhibits significant gaps in interactive scenarios. Learning-
methods yield imitative planning by integrating predictions within holistic modeling [72-74]. However, history-based coalitions pose challenges in supervising future homology among agents. Recently, hybrid pipelines [8, 75] have utilized post- peoceesroesingggg and optimization upon learning- -bsee models to realize behavior interactions among predictions and planning. This can entail significant computational overhead, and imitative planning tends to overestimate hereditary uncertainty in behavior predictions. Tree-based [10. 27] and contingency-enablee
[76, 11] works seek to balance planning preemption and aggression in the face of behavior uncertainty. Nonetheless, pipelines without holistic interactions fall into passive planning maneuvers and incur high exponential cost for predictions. In our work, BeTTop provides an explicit prior for future interactive behaviors which enhances compliant trajectory generations. Moreover, the synergistic prediction and contingency planning networks with BeToop effectively manage behavioral uncertainty.
3 Behavioral Topology
Presenting BeTop, commence the Behavioral Topology formulation and task statement of IPP for autonomous driving in Sec. 1. Then, we demonstrate the Be BeTTopNet network architecture (Fig. 3) for topological reasoning and IPP generation Sec. 3.2. Finally, in Sec. 3.3, we propose the imitative contingency learning process by topological guidance for the proposed network.
3.1 Formulation
Problem formulation. We consider the driving scenario with N, agents as A1: at presence O. along with the scenario map M. The states over historical horizon T,, are denoted as X, for AV and as X2:N, for scenario agents, respectively, where X,, (x TT.0 -T.00 Jn. [1, Na). The objective for integrated prediction and planning to jointly predict scene agents' trajectories Y 22N, as well as AV planning Y, over future horizon T as Y, IyTy Jm. Na).
3

---

Figure 2: BeT formulation. Joint future trajec- tories are transformed to braid sets, and then form joint topology through intertwine indicators.
Table 1: Analysis on different behavioral for- mulations. BeTop labels behave most similarly to human annotations [35], excelling over other formulations like k nearest GT or local attention.
Agent AV
Behavioral
WOMD
Formulations
Acc. f
AUC t
J
Expert [35]
1.000
1.000
GT top-k
0.833
0.702
Local attention [22]
0.951
0,522
JFP graph [67]
0.955
0,500
BeTTop (Ours)
0.967
0.731
Topological formulation. We leverage the braid theory [46], which probes explicit formulations for compliant multi-agent interactions from future data Y I:N.. Intuitively, denotes a transform process for iN with respective agent coordinates, and then gathers each future forward intertwine (occupancy) as joint interactions. Formally, consider the braid group B,, = {on) } by N₂ primitive braids nn each of which on = (ff'.... .R..) denotes a tuple of monotonically increased functions
f: IR3 x + R² x I mapping from Cartesian (z. y. 2) to lateral coordinate (17, t) for agent future Y. Specifically, the function J on is defined as f?" - (Y; - b,, R,,; S i, S Na. where b, and R., denote the left-hand transform matrix to local coordinate of agent A,. The joint interactive behaviors are identified as a set of braids having intertwines {a, C. B over others [45], as shown in Fig. 2. Opposite to implicit methods [22, 67, 41] banking on future distance heuristic, each intertwine the braid can signify an explicit behavioral response, distinguishing between assertive (ott elicit yielding from others) and passive (o yield to others) maneuvers. To avert difficulties in dynamic braid set inference, we redraft multi-agent braids from a topology reasoning perspective.
Named by BeToop, the goal is to reason a topological graph g = (V,E) for multi-agent future behaviors (Fig. 2). Expressly, node topology V = (Y,) J is denoted by multi-agent future trajectories. We can then reformulate the braid set {o as an edge topology eij - E E RRN S i,j < No for future interactive behaviors. Each topology element egj can be defined by two braid functions fī E a, assessing the future interwines along with Y,, Y, as: Eij = max, I () fy)))) Here I is an intertwine indicator by segment intersection [77] under lateral coordinates. With favorable properties proved in Appendix B, we can formulate the reasoning task as:
G = (max max E)
(1)
Agent future Y in node term V> is defined by Gaussian mixtures (GMM) and optimized in Sec. 3.3. The edge topology reasoning & can be specified as a probabilistic inference problem by:
max & = max ΣΣ< eij log Gij + - czg) log(1 ciy),
where 1 sij S No. Synergistic reasoning structures are then established optimizing G".
Comparative analysis. To highlight BeTop's position among various formulations, we first conduct a preliminary analysis to assess behavioral similarity by retrieving future interactive agent pairs using human annotations [35]. Human likeness is quantified by classification metrics, including accuracy and the area under the curve (AUC), with annotated interactive IDs. As depicted in Table 1, labeled BeTopp achieves the closest behavioral similarity compared with other well-accepted formulations in the community. Compared with retrieving k nearest strategy (k: = 6) by ground-truth future states, we observe advanced differentiation in non-interactive behaviors (+16.1 Acc., +4.13% AUC) by BeTopp We then look into the generic learning-based structure by attention [22] or dynamic graph [67] for interactive behaviors. Despite high accuracy, their inferior AUC scores imply difficulties in retrieving precise interactivity compared with BeTop (+19.9 AUC). We refer analytical content in Appendix B. This draft for a reasoning framework BeTop prompting joint behaviors.
3.2 BeTopNet
As presented in Fig. 3, we introduce the synergistic learning framework reasoning BeTop in response to the series of challenges. It encompasses a Transformer backended encoder-decoder network. With

---

Scene Encoder
Synergistic Heads
Imitative Contingency Learning
Planning
Transformer Encoder
Head
maxye
BeTop Guided Transtormer Decoder
argmin
Σ PG((
....
Agenl Encoder
Map Encoder
EAV
Topology Head
X
Recembon
Prediction
QA
SA
Sar
Sg
Figure 3: The BeTopNet Architecture. BeToop establishes an integrated network for topological behavior reasoning, comprising three fundamentals. Scene encoder generates scene-aware attributes for agent S and map MM. Initialized by S and Q4, synergistic decoder reasons edge topology é and trajectories Y! iteratively from ropology-guides local attention. Branched planning E Y with predictions and topology are optimized jointl by imitative contingency learning.
encoded scene semantics X; M, the proposed network features a synergistic decoder which reasons and guides Top Reason heads for topology & and IPP for V comprise the behavioral graph G.
Scene encoder. We leverage a scene-centric coordinate system following planning-orienter oriented princi- ple [7]. Scene attributes comprise historical agent states X R N T, D. and map polyline inputs M E RRN. Xmm,,.., where we portion Nm, map segments with length L,,, from full scene map. encoded separately SA E N.D and Sar E RN and concatenated as scene features S [S a; S (N.. xD A stack of Transformer encoders with local attention are directly employed capturing regional interactions from encoded scene semantics S SMM
Both attributes
Synergistic decoder. Retaining encoded scene features SA, SM, we zoom in the decoding strategy that asks for: 1) interactively reason simultaneous BeToop formulations; 2) selectively decoding of compliant interactive semantics leveraging reasoned topology priors. To this end, we introduce the iterative process of N Transformer decoder layers contributed all agents, pursuing the basis from [78]. To iron out the scene uncertainties, a multi-modal set of M decoding queries Q E IR X are initialized for multi-agent future trajectories. Mcanwhile, relative attributes Sy E Ne xN,x deployed through MLPs topology features Q) E IR Na N, X.N x for edge topology reasoning.
Next, we devise dual infostreams to the iterative decoding process for D of future trajectories and È of future topology. Given agent An, the decoding process in layer follows:
Q TopoDecoder (Q' Q'-I." .SA) ey TopoHead (Q
(3)
Q/ TransDecoder (Q" 1,n SM, S Vi-1 (hh Y IPPHead(Q""").
where both future trajectories Y, E and interactive topology é, E E in BeTоop are decoded in synergistic manners. Reasoned edge topology ěy, E MNN are garnered by topological decoder with query broadcasting Q 1,a, Reasoning nodes for Y,, a Transformer decoder with topology- local attention are drafted serving E as priors. We provide further details in Appendix C.I.
hopology-guidesd local attention. Querying whole-scene agent semantics results in misaligned interactive agents and sparse attention. This motivates design for local attention guided by the reasoned topology é E TR M.N as priors. Specifically, we retrieve the top- index c E R M priored from ét, for eventual interactive agents behaviors with A,, Interactive indices are directly leveraged in gathering S. selectively for local cross-attention. This process is formed as:
Cl..1 TopoAttn (Q" 1--- "A, e
MultiHeadAttm (q 2--2111 1-1." = SiE(
(5)
where e, argmaxx (ē)). Topology-guide agent features C/i" are then aggregated in each layer.
5

---

Reason heads, Given respective decoding features Q and QAA" for each layer, we affix reason heads accustomed to corresponding formulations for én and Y,,, Referred in Eq. (3), the topology head, planning head, and prediction head (IPP heads) are jointl devised by stacked MLPs in reasoning BeTop results. For agent A,, in each layer, reason heads decode GMM components of future states yn E RR-xXy x5 (referring to (Ma, My: log dn, log Ty,P) per step) with mixture score Pn E RM A&ns Pn} E Yn, as well as interactive edge topology éy, E MMMx for BeTop.
3.3 Imitative Contingency Learning
Pursuing the target in Eq. (1), BeTopNet learns end-to-end objectives imitating human-like multi-agent behaviors, integrating compliant behaviors by contingency planning under scenario uncertainties.
Imitation learning. Imitation objectives are firstly established in regulating multi-agent behavioral states {Y,} C V while maximizing their interactive distributions &. The imitative objective for Y is defined by the negative log- log-likelihood (NLL) from best-reasoned components m closest to ground-truths, as denoted: Ly = Σ LNLI - Y).. Followed Eq. (2), the behavioral distributions for edge topology are computed by binary cross-entropy (BCE) given gathered ex No H(em. en.j) over Na agents jointly.
RNN, formulated as LE Σ3
Integrated contingency planning. To integrate compliant behavior learning for G amidst multi- agent scenario uncertainties, contingency planning [79, 76] turned out an apt solution. Bridging immediate safe maneuvers TM to branched planning sets } with joint prediction, it adjourns uncertain decisions and ensures actual safety. While direct joint prediction may lose diversity [[1], reasoned topology E serves as suitable medium distilling future interactive agents for efficient joint combination. Given imitative AV planning outputs 7 C Y, with branching time 4, E (y, integrating contingency learning asks for a safe short-term plan тM E TM, TM E IR Mx to full marginal predictions YM = Y2.N.. as well as M branched planning sets T by joint predictions y This is defined by:
J:Mo Jm guided {T]
TM = argmin max OM Pxc + E P(YMICS (TH,, 131). +CY,
(6)
Y
where maxy Cm denotes worst-case cost fir TM Joint predictions Y, with scene probabilities p(ý,) are recombined by Kas interactive agent subsets, indexing Evv E KM from sorted AV topology: <Av argmaX (max G). It is described by joint costs C2 in guiding branched planning maneuvers. Specifically, both cost functions are defined by the repulsive potential field [8] discouraging planning proximity with respective prediction formulations.
Training loss. BeTopNet is trained end-to-end through imitative objectives and contingency planning costs by weighted integration for each layer, whenever applicable (for the datasets). Please refer to Appendix C.2 for additional details.
Experiment
With preliminary analysis in Sec. 3.1, this section further discovers the following questions: 1) Can BeTopp perform compliant planning via BeToopNet, especially in interactive scenarios? 2) Can BeTop achieve accurate marginal and joint predictions of heterogeneous agents under diverse real-world cases? 3) Can the formulated BeTop facilitate existing state-of-the-at prediction and planning methods? and 4) How do the functionalities in BeTopNet affect the performance?
Benchmark and metrics. BeTop is verified on diverse benchmarks. We leverage two large-scale real-world datasets, i.e., nuPlan [47] and Waymo Open Motion Dataset (WOMD) [35], which are presently the most diverse motion datasets in manifesting planning and prediction performance, For planning tasks in nuPlan, there are in total 1M training cases with 8s horizons. 8,300 separated testing set are chosen by Hear and Test]4-aandom benchmarks [73] for hard-core and general driving scenes. With further demands verifying maneuvers under interactive cases, we build the Test/4-Intee benchmark filtering 1.340 scenes by testing set. Scenarios ranging 15 seconds are tested under three tasks: 1) open-loop (OL), 2) close-loop non-reactive (CL-NR) simulations, and 3) reactive

---

Table 2: Performance comparison of open- and closed-loop planning nuPlan benchmarks. BeTTopNet positions top average planning score and non-reactive simulation amongst SOTA planning systems by all types (rule, learning, and hybrid), especially under difficult benchmarked scenarios.
Test14 Hard
Test14 Random
Type
Method
OLS
CLS-NR
CLS f
Avg.
OLS
CLS-NR
CLS
Avg.
Expert
Log Replay
1.000
0.860
0.688
0.849
1.000
0.940
0.759
0.900
IDM [70]
0.201
0,562
0.623
0.462
0.342
0.704
0.724
0.590
Rule
PDM-Closed [69]
0,264
0.651
0.752
0.556
0.463
0.901
0.916
0.760
GameFormer [8]
0.753
0.666
0.688
0.702
0.794
0.808
0.793
0.798
Hybrid
PDM-Hybrid [69]
0.738
0.660
0.758
0.719
0.822
0.902
0.916
0.880
UrvanDriver [74]
0.769
0.515
0.491
0.592
0.824
0.633
0.610
0.689
PDM-Open [69]
0.791
0.335
0.358
0.495
0.841
0.528
0.572
0.647
PlanCNN [72]
0.524
0.494
0.522
0.513
0.629
0.697
0.675
0.667
Learning
GC-PGP [83]
0.738
0.432
0.396
0.522
0.773
0,560
0,514
0.616
PlanTF [73]
0.833
0.726
0.617
0.725
0.871
0.865
0.806
0.847
Be'TopNet (Ours)
0.840
0.771
0.688
0.766
0.876
0.902
0.857
0.878
Table 3: nuPlan closed-loop planning results on the proposed interactive benchmark. Be achieves desirable PDMScore, with planning safety, road compliance, and driving progress.
Test14 Inter Progress
Type
Method
Col. Avoid f
Drivable 1
Direction
TTC
Comfort t
PDMScore
Expert
Log Replay
1.000
1.000
1,000
0.881
1.000
0.999
0.950
Rule
PDM-Closed [69]
0.886
1.000
1,000
0.818
0.853
0.999
0.833
Constant Acc.
0.449
0,509
0,651
0.048
0.419
1.000
0.108
UrbunDriver [74]
0.970
0,955
0.992
0.798
0.932
1.000
0.854
Leurning
PlanCNN [72]
0.902
0.895
0.973
0.678
0.859
0.999
0.720
PlanTE [73]
0.982
0,992
0.825
0.952
0.871
BeToopNet (Ours)
0.983
0.960
0.999
0.859
0.950
0.999
0.894
(CL-R) ones by nuPlan simulator. We report the official Planning Scores [80] computed by each task. The motion prediction tasks in WOMD share 487k training scenarios, with 44k validation and 44k testing set separately partitioned under two challenges: 1) The Marginal prediction challenge [81] forecasting multiple scene agents independently; 2) The Joint prediction challenge [82] predicting joint trajectory collections by two interactive agents. Primary metrics of mAP and Soft mAP are ranked for official leaderboards [81, 82]. We leave experimental details in Appendix D.
4.1 Main Result
Performance for interactive planning. Table demonstrates the planning results under difficult and regular test cases. Notably, BeTopNet marks top average planning scores, achieving +7.9% in hard cases and excels +6.2% (CLS-NR) in closed-loop simulations. Specifically, gains solid improvements against learning-based planners. This can be attributed to topological formulations learning stabilized joint behavioral patterns, boosting +6.2%, +4.3% non-reactive simulations by real- world logs and enhancing reactive simulation (+ (+11.5%, +6.3%). Contingency objectives enhance uncertainty compliance, leading to expanded results in hard scenarios. Meanwhile, BeToopNet also outperforms rule-based and hybrid planning agents asking for post-optimizations [8] hefty rules in coinciding with reactive simulation setups [69, 70]. We report + +15.8 and +18.4% results of non-reactive simulation in hard cases and close performance in general scenes. Moreover, interactive planning compliance is also verified in the proposed (est/44Inter benchmark centering on interactive scenarios. As in Table 3, BeTopNet fosters +3.8% planning score over previous methods, marking +5.5% driving progress and +2.9% driving compliance closest to human performance. Qualitative results of interactive scenarios in Fig. 5(a-d) further corroborate planning compliance by BeTop.
Performance for marginal and joint motion prediction. Marginal prediction results are in Ta- ble Without the aids of model ensembles or extra data [25, 59], Be BeToopNet outperforms existing approaches, manifesting +2.7% and +3.4% mAP metrics comparing concurrent methods [53, 86] for compliant predictions. Exhibited strong prediction displacement metric (-4.3% minFDE) over methods using extra pretraining [85], it should be noted that displacement metric is less illustrative as discounts uncertainty scoring. BeToopNet further outperforms +6.0% and +26. 1 1% Soft mAP over multi-agent predictors solely leveraging scenario attention [25] graph [24]. N. Table 5 exhibits the
7

---

Table 4: Performance of marginal prediction WOMD Motion Leaderboard. BeToopNet
surpasses existing motion predictors without model ensemble or using extra data. extra LIDAR
data and pretrained model. Primary metric.
Set split
Method
minADE
minPDE
Miss Rute +
mAP 1
Soft mAP
ReCoAt [84]
0.7703
1.6668
0.2437
0.2711
HDGT [24]
0,5933
1,2055
0.1854
0.3577
0.3709
MTR [22]
0,6050
1.2207
0.1351
0.4129
0,4216
MTR++ [25]
0,5906
1.1939
0.1298
0,4329
0,4414
Test
MGTR| [85]
0.5918
1.2135
0.1298
0,4505
0,4599
EDA [53]
0.5718
1.1702
0.1169
0.4487
0,4596
ControIMTR [86]
0.5897
1.1916
0.1282
0.4414
0.4572
BeToppNet (Ours)
0,5723
1.1668
0.1176
0.4566
0.4678
MTR [22]
0.6046
1.2251
0.1366
0.4129
Val
EDA [53]
0.5708
1.1730
0.1178
0.4353
BeTopNet (Ours)
0.5716
1.1640
0.1177
0.4416
Table 5: Performance of joint prediction on WOMD Interaction Leaderboard. BeTopNet
outperforms in both mAP metries. Primary metric.
split
Method
minADE 4 minFDE
Miss Rate
mAP
Soft mAP
Heat|Rm4 [37]
1.4197
3.2595
0.7224
0.0804
M21 [31]
1.3506
2.8325
0.5538
0.1239
GameFormer [8]
0.9721
2.2146
0.4933
0.1923
0.1982
Teat
AMP [32]
0,9073
2.0415
0.4212
0.2294
0.2365
MTR++ [25]
0.8795
1.9505
0,4143
0.2326
0.2368
NeToopNett (Ours)
0.9744
2.2744
0.4355
0.2412
0.2466
MTR [22]
0,9132
2.0536
0.4372
0.1992
Val
AMP [32]
0.8910
2.0133
0.4172
0.2344
NeToppNet (Ours)
0.9304
2.1340
0.4154
0.2366
joint prediction results. BeTopNee outperforms all methods both mAP metries (+4.1%% +3..7% 7% Soft mAP and mAP), presenting robust prediction compliance credit to Be formulations for stable future interaction patterns and aligned by local attention in BeToopNe. Particularly, BeTopp shows interactive compliance, improving +5.1% mAP over recent ruto-regressive approaches [32], boosting +25.4% mAP with same-theoretic methods [8] by large margin. Fig. 5 (e-h) demonstrates the qualitative prediction performance by BoToppet. At the time of submission, BeToopNet ranked st on both WOMD prediction leaderboards [82, 81].
4.2 Ablation Study
Instructed by the last two motivating questions, we investigate the effect of BeToop formulations and components inside BeTTopNet. For efficient study, we randomly partition 20% of WOMD train set for prediction, and directly report the planning results by Test/4-Randon benchmark, which are both representative for the original datasets verified by [22, 73].
Synergy with existing state-of-the-are methods. We first study the effect adjoining BеTop as synergistic objectives over existing SOTA methods in planning and prediction. Described in Table and Table 7, BeTTop augments +2.1% and 2.0% planning score with learning- baased and rule- based planners, respectively. Similar compliance effects are also witnessed in guiding strong motion predictors, bringing +1.1%. +2.4% improved mAP with 1.77% prediction errors of minADE.
Number of interactive agents for topology-guided local attention. determining the number K future interactive agents for BeTopNet in local attention, validate the prediction mAP under array of agent numbers. Shown in Fig. 4, we observe a converging effect, with maximum +3.7%% mAP by the growing number of interactive agents. A drop of 1..% mAP captured after the peak performance of = 32. It is due to falsely accepting non-interactive agent values by large K.
Different functionalities in BoToppet. We further investigate the effects of different functionalities for BeTopNet in Table Compared to the full model, ablations ID.1 and ID.2 underscore the imitative contingency learning process for costs (-2.9% CLS) and contingency branching (-1% CLSS-RR. Sole imitative BeToopet performs the best OL.S (ID.2), while the stabilizing effects found in Sec. 4.1 are verified (-2.8% CLS-NR) in comparing ID.3-ID.5 for joint interactive patterns.

---

Table 6: Results integrating Too by strong planning baselines in nuPlan benchmark.
Table Results of integrating BeTop by strong
prediction baselines in WOMD benchmark.
nuPlan
WOMD
Method
OLS
CLS-NR
CLS 1
Avg.
Method
minADE
minFDE
MR
mAP
PDM [69]
0.463
0.898
0.918
0.760
MTR [22]
0.6046
1.2251
0.1366
0.4164
PDM [69] +BeTop
0.488
0.916
0.902
0.770
MTR [23] +BeTop
0.5941
1,2049
0.1328
0.4249
PlanTF
0.871
0.864
0.847
EDA
0.5708
1.1730
0,4353
PlanIF [73] +BeTop
0,878
0,882
0.807
0.856
EDA [53] +BeTop
0.5742
1.1853
0.1181
0.4407
Figure 4: Results of different interactive agents number for local attention. We observe con- vergence effect for the selection of K.
Table 8: Results of BeTopNet planning perfor- mance with different components. Contingency is the key for closed-loop simulation.
Ablative
nuPlan CLS-NR t
Compoments
OLSt
CLS f
639
0.148
0
BeTopNet
0.876
0.902
0.857
0.855
0.146
branched plan
0.879
0.894
2
leurning
0.882
0.888
0.807
3
Be'Top only
0.877
0.876
0.804
6375
0.143
0.14
local attention
0.871
0.852
0.804
5
Encoders only
0.867
0.827
0.784
Number of Interactive agents
Hard
Test14 Ronddom
Test14 4-inter
b
1-Bs 1-8o
Procs
Pann
Marginal Predictions
Joint Predictions
p=1
1=0s p=0
Precs Prots
Figure 5: Qualitative results of planning and prediction in nuPlan and WOMD. BeTTopNet performs compliant reaction simulations in a) yielding for pedestrians; b) cruising in dense traffic. Interactive scenarios (c,d) further present the consistency of contingency learning. BeToopNet predicts both compliant marginal (e.f) and joint (g.h) multi-agent predictions under diverse scenarios. Future interactive behavior patterns can also be consistently reasoned (rendered in light red) with Be'Top.
5 Conclusion
In this paper, we present BeTop, a topological new-look for multi-agent behavioral formulation. Derived by braid theory, the reasoning tasks for BeTop are drafted supervising joint interactive patterns with integrated prediction and planning. A synergistic network, BeTopNet, is established with an imitative contingency learning process to boost compliant BeTop reasoning. Experiments on nuPlan and WOMD verify BeTooNeet's state-of-ther tftt--t performance in prediction and planning.
Limitation and Future work. Current BeTTop considers one-step future topology alone, and focuses on prediction and planning. Future work would be centered on developing a recursive version of BeTopp in multi-step, multi-agent reasoning and coordination. Another promising direction would be the connectivity of BeTop upon perceptions as tracking for the end-to-end paradigm, as well as an extension reasoning behaviors under 3D scenarios for multiple autonomous agents.

---

Acknowledgments
This work was supported in part by the Agency for Science, Technology and Research (A STARR) Singapore, under the MTC Individual Research Grant (M22K2c0079), the ANR-NRF Joint Grant NN.NNR22011NRF-AARO0 HM Science), the Ministry Education (MOE), Singapore, under the Tier 2 Grant 1-00E--2EE5022-000222 National Key R&D Program of China (20222D0160104) NSFC (62206172), and Shanghai Committee of Science and Technology (23YF1462000).
References
[1] Long Chen, Yuchen Li, Chao Huang, Bai Li, Yang Xing, Daxin Tian, Li Li, Zhongxu Hu, Xiaoxiang Na. Zixuan Li, Siyu Teng, Chen L.v, Jinjun Wang. Dongpu Cao, Nanning Zheng, and Fei-Yue Wang. Milestones in autonomous driving and intelligent vehicles: Survey of surveys. TIV, 2022.
[2] Wenshuo Wang. Lctian Wang, Chengyuan Zhang, Changliu Liu, and Lijun Sun. Social interactions for autonomous driving: A review and perspectives. Foundations and Trends in Robotics, 2022. 1, 2, 17
[3] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li, End-to-end autonomous driving: Challenges and frontiers. PAMI, 2024.
[4] Sergio Casas, Abbas Sadat, and Raquel Urtasun. MP3: A unified model to map, perceive, predict and plan. In CVPR, 2021. 1,3
[5] Shengehao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. ST-P3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In ECCV, 2022. 3
[6] Haochen Liu, Zhiyu Huang, Wenhui Huang, Haohan Yang, Xiaoyu Mo. and Chen Lv. Hybrid-prediition integrated planning for autonomous driving. arXiv preprint arXiv:2402.022 02426.. 2024.
[7] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang. Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning- oriented autonomous driving. In CVPR, 2023. 3.5
[8] Zhiyu Huang, Haochen Liu, and Chen Lv. GameFormer: Ghme-theoretic modeling and learning of transformer-be baased interactive prediction and planning for autonomous driving. In ICCV, 2023. 2, 8, 16, 19, 20, 21,
[9] Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani, AgentFormer: Agent-aware transformers for sncio-temporal multi-agent forecasting. In ICCV, 2021.
[10] Yuxiao Chen, Peter Karkus, Boris Ivanovic, Xinshuo Weng, and Marco Pavone, Tree-structured policy planning with learned behavior models. In ICRA, 2023. 1,
[11] Alexander Cui, Sergio Casas, Abhas Sadat, Renjie Liao, and Raquel Urtasun. LookOut: Diverse multi- future prediction and planning for self-driving. In ICCV, 2021. 3, 6
[12] Stefano Pini, Christian S Perone, Aayush Ahuja, Ana Sofia Rufino Ferreira, Moritz Niendorf, and Sergey Zagoruyko. Safe real-world autonomous driving by learning to predict and plan with a mixture of experts. In ICRA, 2023.
[13] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guidee control prediction for end-to-end autonomous driving: A simple yet strong baseline. In NewrlPS, 2022.
[14] Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan. Roberto Cipolla, and Alex Kendall. FIERY: Future instance prediction in bird's-eye view from surround monocular cameras. In ICCV, 2021. 3, 16
[15] Yihan Hu, Kun Li. Pingyuan Liang, Jingyu Qian, Zhening Yang. Haichao Zhang, Wenxin Shao, Zhuangzhuang Ding, Wei Xu, and Qiang Liu. Imitation with aatialltemmoral heatmap: 2nd place solution for nuplan challenge. arXiv preprint arXiv:2306.15: 15700, 2023.
[16] Reza Mahjourian, Jinkyu Kim, Yuning Chai, Mingxing Tan, Ben Sapp, and Dragomir Anguelov. Occupancy flow fields for motion forecasting in autonomous driving. RA-L, 2022. 1,3
[17] Haochen Liu, Zhiyu Huang, and Chen Lv, Multi modal hierarchical transformer for occupancy flow field prediction in autonomous driving. In ICRA, 2023.
10

---

[18] Ben Agro, Quinlan Sykora, Sergio Casas, and Raquel Urtasun. Implicit occupancy flow fields for perception and prediction in self-driving. In CVPR, 2023. 1, 16
[19] Jinkyu Kim, Reza Mahjourian, Scott Ettinger, Mayank Bansal, Brandyn White, Ben Sapp, and Dragomir Anguelov. StopNet: Scalable trajectory and occupancy prediction for urban autonomous driving. In ICRA,
2022. 1,3
[20] Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Hujjie Wang, Jia Zeng. Zhiqi Li. Jiazhi Yang. Hanming Deng, Hao Tian, Enze Xie, Jiangwei Xic, Li Chen, Tianyu Li, Yang Li, Yulu Gao, Xiaosong Jia, Si Liu, Jianping Shi, Dahua Lin, and Yu Qiao. Delving into the devils of bird's e-yd-----vvvww perception: A review, evaluation and recipe. PAMI, 2024. 1,3
[21] Hanchen Liu, Zhiyu Huang, and Chen Lv. Occupancy prediction-guidee neural planner for autonomous driving. In ITSC, 2023. 1,3, 16
[22] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schicle. Motion transformer with global intention localization and local movement refinement. NewrIPS, 2022. 1, 3, 4, 8, 9, 18, 19, 20, 22, 23
[23] Xiaosong Jia, Li Chen, Penghao Wu, Jia Zeng, Junchi Yan, Hongyang Li, and Yu Qiao. Towards capturing the temporal dynamics for trajectory prediction: a coarse-to-fine approach. CoRL, 2022.
[24] Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. HDGT: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. PAMI, 2023. 3, 7, 8, 16, 20
[25] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. MTR++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. PAMI, 2024. 7, 8, 16. 20
[26] Zhiyu Huang, Chen Tang, Chen Lv. Masayoshi Tomizuka, and Wei Zhan. Learning online belief prediction for efficient pomdp planning in autonomous driving. arXiv preprint arXiv:2401. 15315, 2024. 1,
[27] Zhiyu Huang, Peter Karkus, Boris Ivanovic, Yuxiao Chen, Marco Pavone, and Chen Lv. DTPP: Differen- tiable joint conditional prediction and cost evaluation for tree policy planning in autonomous driving. In ICRA, 2024. 1,3
[28] Zhiyu Huang, Haochen Liu, Jingda Wu, and Chen Lvv. Conditional predictive hehavior planning with inverse reinforcement learning for human-like autonomous driving. TITS, 2023. 1, 2, 16
[29] Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David Weiss, Benjamin Sapp, Zhifeng Chen, and Jonathon Shlens. Scene Transformer: A unified architecture for predicting future trajectories of multiple agents. In ICLR, 2022. 2,3
[30] Steffen Hagedom, Mareel Hallgarten, Martin Stoll, and Alexandru Condurache. Rethinking integration of prediction and planning in deep learning-base based automated driving systems: review. arXiv preprint arXiv:2308.0573 057331 2023. 2
[31] Qiao Sun, Xin Huang, Junru Gu, Brian C Williams, and Hang Zhao. M21: From factored marginal trajectory prediction to interactive prediction. In CVPR, 2022. 2, 3, 8, 16, 20
[32] Xiaosong Jia, Shaoshuai Shi, Zijun Chen, Li Jiang, Wenlong Liao, Tao He, and Junchi Yan. AMP: Autoregressive motion prediction revisited with next token prediction for autonomous driving. arXiv preprint arXi::2403. 13331, 2024. 2, 8, 16, 20, 22
[33] Jose Luis Vazquez Espinoza, Alexander Liniger, Wilko Schwarting, Daniela Rus, and Luc Van Gool. Deep interactive motion prediction and planning: Playing games with motion prediction models. In L4DC, 2022.
2, 16
[34] Wei Zhan, Changliu Liu, Ching-Yao Chan, and Masayoshi Tomizuka. A non-conservatively defensive strategy for urban autonomous driving. ITSC, 2016. 2
[35] Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Supp, Charles R. Qi, Yin Zhou, Zocy Yang, Aurélien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In ICCV, 2021. 2, 4, 6, 20, 23, 29
[36] Dan Xie, Tianmin Shu, Sinisa Todorovic, and Song Chun Zhu. Learning and inferring "dark matter" and predicting human intents and trajectories in videos. PAMI, 2017.
11

---

[37] Xiaoyu Mo, Zhiyu Huang, Yang Xing, and Chen Lv. Multi-agent trajectory prediction with heterogeneous edge-enhanced graph attention network. TI7S, 2022. 2,3,8, 20
[38] Zhiyu Huang, Xiaoyu Mo, and Chen L.v. Multi-modal motion prediction with transformer-based neural network for autonomous driving. In ICRA, 2022. 2
[39] Yuriy Biktairov, Maxim Stebelev, Irina Rudenko, Oleh Shliazhko, and Boris Yangel. PRANK: motion
prediction based on ranking. In NeurlPS, 2020. 2
[40] Dachee Park, Hobin Ryu, Yunsco Yang, Jegyeong Cho, Jiwon Kim, and Kuk-Jin Yoon. Leveraging future relationship reasoning for vehicle trajectory prediction. In ICLR, 2023. 3,
[41] Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. EvolveGraph: Multi-agent trajectory prediction with dynamic relational reasoning. In NeurIPS, 2020. 2, 3, 3, 16
[42] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICIR, 2017. 2
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017. 2
[44] Christoforos Mavrogiannis, Jonathan DeCastro, and Siddhartha S Srinivasa. Analyzing multiagent interac- tions in traffic scenes via topological braids. In JCRA, 2022. 2, 3, 16, 17
[45] Christoforos Mavrogiannis, Jonathan A DeCastro, and Siddhartha S Srinivasa. Abstracting road traffic via topological braids: Applications to traffic flow analysis and distributed control. I/RR, 2023. 2, 3, 16, 17
[46] Emil Artin. Theory of braids. Annals of Mathematics, 1947. 2, 4, 16
[47] Napat Karnchanachari, Dimitris Geromichalos, Kok Scang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong. Yiluan Guo, and Holger Caesar. Towards learning-based planning: The nuplan benchmark for real-world autonomous driving. In ICRA, 2024. 2, 3, 6, 20
[48] Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. ChauffeurNet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint 1rXiv:18/2.03079, 2018. 3
[49] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai BEVFormer: Learning bird's-cye s-cy--ewe representation from multi-camera images via spatiotemporal transformers. In ECCV, 2022.
[50] Alexey Kamenev, Lirui Wang. Ollin Boer Bohan, Ishwar Kulkarni, Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nistér, and Nikolai Smolyanskiy. PredictionNet: Real-time joint probabilistic traffic prediction for planning, control, and simulation. In ICRA, 2022. 3
[51] Zetong Yang. Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. In CVPR, 2024. 3
[52] Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. VectorNet: Encoding hd maps and agent dynamics from vectorized representation. In CVPR, 2020.
[53] Longzhong Lin, Xuewu Lin, Tianwei Lin, Lichao Huang, Rong Xiong, and Yue Wang. EDA: Evolving and distinct anchors multimodal motion prediction. AAAI, 2024. 3, 7, 8, 9, 19, 20, 22, 24
[54] Charlie Tang and Russ R Salakhudinov. Multiple futures prediction. In NeurlPS, 2019. 3
[55] Siyuan Qi and Song-Chun Zhu, Intent-aware multi-agent reinforcement learning. ICRA, 2018.
[56] Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. THOMAS: Trajectory heatmap output with learned multi-agent sampling. ICLR, 2022.
[57] Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde, GO- HOME: Graph-oricnted heatmap output for future motion estimation. In ICRA, 2022.
[58] Junru Gu, Chen Sun, and Hang Zhao. Dense End-to-end trajectory prediction from dense goal sets. In ICCV, 2021. 3
[59] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled Refaat, Nigamaa Nayakanti, Andre Comman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, and Benjamin Sapp. MultiPath+++ Efficient information fusion and trajectory aggregation for behavior prediction. In ICRA,
2022. 3.7
12

---

[60] Junha Roh, Christoforos Mavrogiannis, Rishabh Madan, Dieter Fox, and Siddhartha Srinivasa. Multimodal trajectory prediction via topological invariance for navigation uncontrolled intersections. In CoRL, 2020.
[61] Christoforos Mavrogiannis, Krishna Balasubramanian, Sriyash Poddar, Anush Gandra, and Siddhartha Srinivasa. Winding through: Crowd navigation via topological invariance. RA-L, 2022. 3, 16
[62] Nigamaa Nayakanti, Rami AI-Rfou, Aurick Zhou, Kratarth Goel, Khaled Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In ICRA, 2023. 3, 23
[63] Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu, HiVT: Hierarchical vector transformer for multi-agent motion prediction. In ICCV, 2022. 3
[64] Alexander Cui, Sergio Casas, Kelvin Wong, Simon Suo, and Raquel Urtasun. GoRela: Go relative for viewpoint-invariant motion forecasting. In ICRA, 2023. 3, 18
[65] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically- feasible trajectory forecasting with heterogeneous data. In ECCV, 2020.
[66] Luke Rowe, Martin Ethier, Eli-Henry Dykhne, and Krzysztof Czarnecki. FIMP: Factorized joint multi- agent motion prediction over learned directed acyelic interaction graphs. In CVPR, 2023, 3, 16
[67] Wenjie Luo, Cheol Park, Andre Cormman, Benjamin Sapp. and Dragomir Anguelov. JFP: Joint future prediction with interactive multi-agent modeling for autonomous driving. In CoRI,, 2022. 3, 3,4, 16
[68] Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forccasting. In ECCV, 2020. 3
[69] Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with misconceptions about leamiin-hhseed vehiele motion planning. In CoRI. 2023. 3, 7, 18, 19, 20, 21, 23
[70] Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observations and microscopic simulations. Physicul review E. 2000. 20, 21
[71] Peng Hang, Chen Lv, Yang Xing, Chao Huang, and Zhongxu Hu. Human-likk like decision making for autonomous driving: A noncooperative game theoretic approach. TITS, 2020. 3
[72] Katrin Renz, Kashyap Chitta, Otnicl- -oiidl Mereea, A. Sophia Koepke, Zeynop Akata, and Andreas Geiger. PlanTT: Explainable planning transformers via object-level representations, In CoRL, 2022. 3, 7, 20, 21
[73] Jie Cheng, Yinghing Chen, Xiaodong Mei, Bowen Yang. Bo L.,, and Ming Liu. Rethinking imitation-based planner for autonomous driving. In ICRA, 2024. 6, 7, 8, 9, 18, 20, 21, 22,24 24
[74] Oliver Scheel, Luca Bergamini. Maciej Wolczyk, Blakej Osiński, and Peter Ondruska. Urban Driver: Learning to drive from real-world demonstrations using policy gradients. CoRL, 2021. 3, 7, 20, 21
[75] Peter Karkus, Boris Ivanovic, Shie Mannor, and Marco Pavone. DiffStack: A differentiable and modular control stack for autonomous vehicles. In CoRL, 2022. 3
[76] Tong Li, Lu Zhang, Sikang Liu, and Shaojie Shen. MARC: Multipolicy and risk-aware contingency planning for autonomous driving. RA-L, 2023. 3,
[77] Franklin Antonio. Faster line segment intersection. In Graphics Gems III (IBM Version), pages 199-202. Elsevier, 1992. 4
[78] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for detr. In ICLR, 2022. 5
[79] Jason Hardy and Mark Campbell. Contingency planning over probabilistic obstacle predictions for autonomous road vehicles. TRO, 2013. 6
[80] Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong. Eric Wolff, Alex Lang, Luke Fletcher. Oscar Beijbom, and Sammy Omari. NuPlan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106. 11810, 2021. 7, 20, 23, 29
[81] Waymo. Waymo open dataset motion prediction challenge 2024. https://vayno.com com/ open/
wwlllenges//22//nottte www.ooonnneiiiiooneeeee 7, 8, 20, 22
13

---

[82] Waymo. Waymo open dataset interaction prediction challenge 2021. https://wayno: com/ open/ challenges/222//nnerrattt prwdictiin/.....e 7, 20, 22
[83] Marcel Hallgarten, Martin Stoll, and Andreas 7ell, From prediction to planning with goal conditioned lane
graph traversals. In ITSC, 2023. 7, 20, 21
[84] Zhiyu Huang, Xiaoyu Mo, and Chen Lv. ReCoAL: A deep learning-basel framework for multi-modal motion prediction in autonomous driving application. In ITSC, 2022. 8
[85] Yigian Gan, Hao Xiao, Yizhe Zhao, Ethan Zhang, 7he Huang, Xin Ye, and Lingting Ge. MGTR: Multi-granular transformer for motion prediction with lidar. In ICRA, 2024. 7,8
[86] Jiawei Sun, Chengran Yuan, Shuo Sun, Shanze Wang, Yuhang Han, Shuailei Ma, Zefan Huang, Anthony Wong, Keng Peng Tee, and Marcelo H Ang Jr. ControlMTR: Control-guided motion transformer with scene-compliant intention points for feasible motion prediction. arXiv preprint arXiv:2404.1024, 2024.
8, 8.20
[87] Mitchell Berger. Topological invariants in braid theory. Letters Mathematical Physics, 2001. 16
[88] Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. ego status all you need for open-loop end-to-end autonomous driving? In CVPR, 2024. 18, 20
[89] Charles Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning point for 3d classification and segmentation. In CVPR, 2017.
[90] Zetong Yang, Li Jiang, Yanan Sun, Bert Schicle, and Jiaya Jia, A unified query-based paradigm for point cloud understanding. CVPR, 2022. 19
[91] Zikang Zhou, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Query-centric trajectory prediction. In CVPR, 2023.
[92] Jie Cheng, Yingbing Chen, and Qifeng Chen. PLUTO: Pushing the limit of imitation learning-base -basd planning for autonomous driving. arXiv preprint arXiv:2404. 14327, 2024. 19, 20, 21
[93] NAVSIM Contributors.
NAVSIM: Data-driven non-reactive autonomous vehicle simulation. https:
//github. com/ http://tttoooovvvisiioaaaaaeeeee 2024. 20
[94] Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang. Zetong Yang, Hongyang
Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, Kashyap Chitta. NAVSIM: Data-driven non-reactive autonomous vehicle simulation and benchmarking. arXivv 2400.153499 2024. 20
[95] Lan Feng. Mohammadhossein Bahari, Kaouther Messaoud Ben Amor, Éloi Zablocki. Mathieu Cord, and Alexandre Alahi. UniTraj: A unified framework for scalable vehiclc trajectory prediction. In ECCV, 2024.
23
14

---

Appendix
A Discussions
16
Properties of BeTTop
16
C Implementation Details
18
C.I Model Structure
18
C.2 Imitative Contingeney Learning
19
Experimental Setup Details
20
Planning on nuPlan
20
D.2 Prediction on WOMD
20
D.3
Training Setup
21
Additional Quantitative Results
21
E.1 Planning
21
E.2 Prediction
22
Additional Ablation Studies
22
Additional Qualitative Results
23
License of Assets
23
15

---

A Discussions
Towards a better understanding of this work, we supplement intuitive questions that may raise. Note that the following list does not indicate the manuscript was submitted to previous venue or not.
Q1: How does BeTop bridge and discern with dense, sparse, and topological representations?
BeTop is derived from braid theory [46], reasoning the topology that explicitly labels consensual interactions as dense, occupancy-like intertwines from sparse, braided future trajectories of multi- agent behaviors. Unlike fixed occupancy [14, 21, 18], BeTop dynamically forecasts behavioral interactions by agents collectively, serving as a guiding medium for node reasoning in joint planning and prediction compared standard sparse predictions [25, 24, 66]. Differentiating from typical topological approaches, ВсTop explicitly formulates coordinated multi-agent behaviors and reasons topology in occupancy manners, rather than relying implicit relation graph learning [41, 67, 40] or complex braid inference [44, 45, 61].
Q2: Why is BeTop allied with contingency instead of conditional game-theoretic reasoning?
Contingency plays the most suitable role in BeTopNet, addressing multi-agent uncertainty by defer- ring uncertain planning with long-ten joint predictions from interactive agents under behavioral compliance. This aligns with BeTTop's reasoning targets, which aim to achieve a one-shot consensus among all behaviors by formulating future behavior as coordinated joint interactions. This synergy al- leviates the challenges led by game-theoretic reasoning [8, 33] or conditional integration [31, 28, 32], which are struggled by multi-step interactive rollouts and unstable joint behavioral patterns.
Q3: What would be the broader impact and future direction?
BeTTop steps the first trial towards an explicit topological formulation and reasoning paradigm for multi-agent interactive behaviors. This serves basis in exploring immense interactive behaviors in the real-world, and is reasoned by the autonomous agents jointly in a coordinated manner. For instance, BeTop with enlarged scalability and dimension may summarize the topology for all behaviors in 3D scenarios or even larger spaciality, and may reasoned through end-to-end BeToopeet as a foundation behavioral model. Moreover, we can consider BeTop's capability collective maneuvers, which can be further leveraged in coordinating naturalistic and efficient decision maainn for multiple autonomous agents.
B Properties of BeTTo
In this section, we supplement additional properties that characterize the formulation of Behavioral Topology (BeTTop). Analytically, Be'Top is highlighted with: 1) geometric invariant (Theorem B.1); 2) approximated topological invariant (Theorem B.2); and 3) asymmetric topology (Theorem B.3).
Theorem B.1 (Geometric Invariant). The topology results of & c g in BeToop remain unchanged given arbitrary geometrical transformations for the collective scene trajectories Y,..
Proof. Given arbitrary rotation H E R22x2 and shifting E RR2 Consider the mapping for elemen- tary topology Ciz E & from future trajectories (Y,, Y,), 1 jE Na), the local transformation f:g - h o f to i's coordinate is invariant, such that h(f(Y, Y,)) (HH(HY + HY, + b)). Hence, given the function sets fj E J. defined in Sec. 3.1; eij E is also invariant. O
Remark 1. The Theorem B.I proves the behavioral stability of BeTop given arbitrary multi-agent trajectories patterns for planning and predictions. Any rotations and movement of the original scene will not interfere with the formulated results of Be'Top.
Definition B.1 (Topological Invariant). Given future trajectory pairs (Y,, Y,), i + NA with certain current heading (B?. 89), the sum of future relative angles (winding number) Wij =
Dd Ti A0t, form first-order [87] topological invariant.
2
Proof. Consider the polar representation for the closed form vh((1) = (1)((()) Where the [0, Tm] + C((0), n). we define the winding function (x 1 Sc, dz/2,2
16

---

()(t), E [m] This Cauchy formula [2] can be further integrated as:
1 2
11,( 1 (0:(t) - ((())). [O))
(t)
log(
2T
We are interested the real (first-order) part of Ae(t) which is an invariant topologically. Hence,
the trajectory pairs (Y, 22), 0 < S < Tm can be described as: Y; =/s (0, Ty). The joint invariant across future Way Σi / (A.(t) - 2y)))) then becomes:
1 Ty
I, 2T Σ 09)
Wij
Σ (0 -3)
(8)
2m
As the current heading pair (0% 09) is certain, the invariant becomes Wij IF proofs the definition.
1 T
A0, which
Corollary B.1.1. Given any 10l [[ 2. 21. where 0 < < < No. (0,Ty), and the constant M), 7/j R, the transformed wiy E0' (1) sin A0i -7s sin 03) is also topological invariant.
Tú
Proof. The defined function sin(-) is monotone mapping under [- ]. Hence, this firstly enables Ed 1/ sin of uniquely defines Y, More than that, wi is the unique mapping value of myy with Y; under defined transformation, and thereby keep the invariant property.
Theorem B.2. The edge topology & G is an approximate of topological invariant, so that Ei E,0 i & N₂ is characterized by waz
Proof. Given future trajectories Y,, Y, consider the braid functions d; maps monotonically increased transformations ff fjj to i's local coordinate,as defined in Sec. 3.1. We assume continuous future horizon (o, Ty] where the headings for fJ(Y,) defined by relative angles (ty Thereby, the transformed lateral trajectory for agent j can be formed as: fo ny sin A0, (t)dt. Similarly, f( can be formed as So fy sin 0g (()dl, where //s original intersection function in Sec. 3.1 as:
my denotes small constant step Theee These form the
I (UIGD). + ((()) sin A01;
sin 0 ddt,
(9)
where I(.) = 0 denotes braid intertwine for interactive behaviors. As the term in Corollary B. of wi, (the sum of the right term) is an approximate (discretization) of the right term, this proves the edge topology Ciy E E maxr, I (+) as the approximation of topological invariant.
Remark 2.
The Theorem B.2 proves the generality of BeToop in terms of future interactive behavioral patterns. The approximated topological invariant property prompts a representative of various future states sharing similar behavioral or identical interactive patterns by BeTTop.
Theorem B.3 (Asymmetric Topology). Edge topology eji, 0 i,j S No.
C G not symmetric such that Fi, eij #
Proof. Given Eq. (9) defined in Theorem w.2.we can always construct a case 3t,,t, E (0,Ty), the intersection I (fl(x f)(r))) = 0, but I( (>')) ), f. f(())) * O, which prove the claim.
O
Remark 3. The Theorem B.3 proves a more naturalistic interactive behavior of BeTTop. It is likely in real-world scenario that the future behavior of agent A; is interacted by agent Aj. while A, does not.
Computational complexity. The complexity in computing full is O(22). In practice, we downscale the sourced as the agents of interests N, < Na. such that ONN,N.) It is much less than braid sequence inference [45] with maximum O(N,, - 1) computational costs. Further analytical proof of computational efficiency leveraging braids can be found in [44].
17

---

Cx Q
[M.K]
Add & Norm
Sa
MLP
[M.
6
2/ [M.N.]
FFN
MMWP
Concat.
QR
Topalogy
Topology-guided Local Attention
S
Head
Q
-1
MLP
Q
1R
K
Topology decoder
[N,.D]
OF [M.N,.D]
[M
Add & Norm
Concol
MLLP
MLP
P
Topology Decoder
Mulli Head Sell-Atlention
[M.My.D]
[M.
[MM,TT 6.
TC
Sg D]
-11
[M.D]
BeTopNet decoder layer
Planning Head
Figure 6: Structural details in BeToopNet. a) The learning structure of single synergistic decoder layer featuring TransDecoder and TopoAttn design; b) The structure inside topology decoder network of TopoDecoder; c) Branched planning head design corresponding contingency planning.
C Implementation Details
In this section, we instantiate further details for BeToop the configurations for Be BeTTopNet structure, and provide contingency learning paradigms for both prediction and planning challenges.
C.1 Model Structure
Subjecting to different testing requirements defined nuPlan and WOMD, we set up two model variants for BeTopNNe in formulating planning and prediction challenges. Apart from topology reasoning for interactive behaviors, for planning challenges, Be BeToppNet integrates both tasks of prediction and planning. In the prediction challenges under marginal and joint settings, BeToopNet is allocated only with the prediction parts. The structural details are illustrated below.
Scene inputs. Carving the driving scenarios involves historical agent states X and map polylines M as scene inputs. For planning settings, we collect scene agent states with past Ty seconds at 10Hz, leaving basic kinematic states as (I, y, Ex. by: 0), joining with agent shapes and types. We only keep the current state for ego vehicle (AV) in preventing closed-loop gap [73, 69, 88] with open-loop training as recently discussed in the community. Nm. 256 segments of the map with length Lm = 20 are gathered by scene-centric manners considering positions, traffic lights, and speed limits. The prediction task is built on WOMD considering T, seconds at 10Hz with scenarios of larger scalability. It is followed by full scene agents with Nm = 768 map segments of identical states for both X and M.
Scene encoder. Both scene attributes X, M are firstly encoded leveraging layered point encoder [89] to hidden dimension D shared throughout the BeTTopNet structures, with D = 128 for planning and D 256 for prediction tasks. A stack of Transformer encoders is then devised with 4 layers in planning and in prediction for A. SM- Due to scalable settings for prediction tasks, local attention with the nearest keys is built each layer. Following [22, 73], the dense prediction head adopted for all agents after the encoder, enhancing future semantics.
Synergistic decoder. Depicted in Fig. 6, the decoder structure is founded by an iterative stack of L Transformer decoders querying M modes of future trajectories Y with dual stream in reasoning topology E. Consisting of L = 6 and L decoders for prediction and planning respectively, it is initialized jointly by relative features Q and decoding queries U..". Relative attributes Sn computed efficiently following [64] for relative distance and headings. M = 6 learnable embedding are devised as 0.6 for planning, and we utilize the anchored ones [22] with M = 64 in prediction
tasks. As displayed in Fig. 6(a), dual queries Q 1,n
-²11." Q 1-11 are served from the last level. The
18

---

decoding process (following Eq. (3)) iteratively reasons é from TopoDecoder, serving as prior guiding agent semantics by TopoAttn inside TransDecoder, which concurrently aggregates scene semantics from agents SA and maps SM. Expressly, the structure of TopoDecoder (Fig. 6(b)) comprises simple MLPs and update Q2²" by concatenation sourcing query features Q/ with agent semantics SA and connect residuals Q/ from last layer. Decoding queries Q updated by a concatenation from aggregated agent feature Ch map features Ch M and agent semantics S directly from encoder. Agent feature C²2² is aggregated by TopoAttn, where the local attention is devised using deployments from [90] indexing K = 32 agents from reasoned topology é We omit the aggregation process for map features, which performs the vanilla Transformer decoder structure for planning and dynamic collection form by [22] under prediction tasks for hefty map features.
Reason heads. Following the contents Sec. 3.2, reason heads in decoding prediction and topology follow simple MLPs given respective decoding queries. For the planning head, it leverages cascaded design for branched contingency planning with multi-modalitten (Fig. 6(c)). Specifically, the short-planning TM is decoded by the AV future states {Ø] Im from first stage head with m E M modes, where to 3 denotes the branching time. They are then detached and leveraged as prior for the branched planning. Successive MLPs project and reshape the short-term contingency prior by IR xM xM for M, 6 branches planning Tj" under each of TM- Further concatenated by broadcasted decoding queries, the planning head generates M My trajectories Y, for AV.
C.2 Imitative Contingency Learning
Efficient joint prediction recombination. Retrieving the top-performed joint predictions from full marginal predictions YN sequentially time-consuming with exponentially complexity. Hence, we firstly downscale the potentially interested agents N, by sorting the AV-reasoned topology ét with the largest KM = 4 value as index given the planning task. For the joint prediction task, N₁ is annotated already from the original data. Then, we leverage the tensor broadcasting mechanism in efficiently retrieving M 6 largest joint distributions M(YM) and joint trajectories Y from N₁ interacted agents. Given a tensor P, E A initialized by ones, the joint score is computed by Ni times of iterative broadcasting p.. on the n-th dimension for P, as P, = max II." p(7) @ p... This process only costs 1.6ms in computing N, = 4 joint predictions for contingency planning.
Imitative contingency objectives. Followed by learning objectives derived in Sec. 3.3, the imitation objectives for each layer can be represented Lu. Ly + AiCg. A 50 weighting BCE loss for edge topology reasoning, the NLL loss for Ly is formulated as:
LNLL = log Tr + log By + 1og(1-p")
((#)
d=
2
du
2
2p dodg a2oy
logp(m"), (10)
2
2[1-p"]
where da, d, denotes the difference with ground-truths. In determining the component m², we leverage a winner-take-all (WTA) strategy [91] in planning by measuring the average displacements (ADE) with groung- truths.. For prediction tasks, m" is selected from the closest anchor as in [53]. For the learnable cost functions max M' (-), (-) in contingency planning, we leverage the repulsive potential field [8] delincating planning with prediction by Ø = ming (1+d For max Cm (-). is gathered across T considering the worst case under full marginal prediction YM comprising N, = 32 scene agents. For the branched cost C-(+), ø for each branch is computed considering joint prediction from N₂ = 4 agents. Following the objective defined in Eq. (6), the learnable contingency cost is defined as: La. Cm + ΣM P( Hence, the general objectives for planning become:
C == Lv + XILE + X2LcL.
(1))
where A is the contingency costs weight. Prediction tasks are updated only by LI.-
Inference. Different from the training process, for the planning task we directly select the full planning trajectory of , 8 seconds by highest scoring T" argmax, Y,, subjecting to the original task settings nuPlan. The scoring results are a combination from original confidence pr and the short-term cost C' [69]: C = Pi Am CMr where Xm 0.5 facilitates short-term planning compliance [92]. For the prediction task, a post-processing module following [53] leveraged in selecting M- 6 marginal or joint trajectories of T 8 seconds among agent types in WOMD.
19

---

D Experimental Setup Details
In this section, we provide extra details demonstrated in Sec. 4 for the experiment setups, including detailed settings for the proposed benchmark, testing metrics, state-of-the-an baselines, and training.
D.I Planning on nuPlan
Testing metrics. For open-loop planning tests, the open-loop score (OIS) serves as the general statistics weighting displacement metrics and miss rates. For closed-loop simulations, both metrics (CLS, CLS-NR) are weighted by a series of statistics measuring 1) driving safety. 2) planning progress, 3) driving comforts, and rule obeying. The PDMScore [93, 94] compared in Table is basically replica of the closed-loop score for efficient computations. It is denoted as:
W1 TTC + w2DC + W3EP
PDMscore = CA - DAC DDC
DDC
(12)
Σ W/
where the sub-metrics abbreviations referred in Table 3. All general metries range from 0 to 1.
Test14-Inter: We launch the Test14-Inter benchmark in verifying the planning systems under typical corner cases containing rich social interactions, or dynamic profiles by complex map forms. This is highly motivated by the issues raised in [69, 88], that massive scenarios may also be completed by a simple motion model. Specifically, we adopt mining heuristic defining corner cases by which human experts excel but the motion model (constant acceleration vehicle, CAV) fails. For efficient mining, we directly assess planning results by PDMScore and define the criteria as:
(PDMscore CAV < 7) A E 2 (1 - (1-7)),
(13)
where 7 = 0.1 denotes scoring threshold for cases that cannot be easily solved by regular motion profiles of the planning maneuvers. As future work, we aim to explore more interactive scenarios
aggravating by BeT as an enhancement.
Val14. In pursuing comprehensive comparisons with current methods, we also manifest Be in the Val14 set proposed in [69]. It is a subset of 1040 scenes from the validation set. However, since a portion of validation scenes are shared with the training set in nuPlan [80], we argue this less representative of testing fairness for learning- based methods. Hence, we only place it as supplementary.
Baselines. For all baselines presented the planning task, we directly report their previous benchmark results. Additional results in the proposed benchmark (Table 3) and ablation studies (Table 6) are re-implemented by the official releases [69, 72, 47, 69]. Expressly, we study the sfte-of-theean planning systems categorized by: 1) Rule-based: performing maneuvers by designate rules with the reactive agents [70] or mimicking the planning score [69]; 2) Hybrid: incorporating rules [69] or post-optimizations [8] with a learning- based model; and 3) Learning-bssed end-to-end planning with GNN [83, 74] or Transformer [72, 73] enabled models, as well as concurrent methods [92] augmented by representation learning. For ablation studies in Table 6, BeTopp is trained directly by the proposed topology decoder with the original PlanT [73] pipeline. For the PDM [69] as
rule-based planning system, integrate BeTTop by replacing the original rule-based motion model with predictions generated from Be BeTTopeet.
D.2 Prediction on WOMD
Testing metrics. For the prediction task, the mean AP (mAP) and Soft-mAP scores are assigned as the primary metries in computing multi-modal predictions modeled by marginal or joint distributions [35, 81, 82]. Displacement metrics of minADE and minFDE provide the multi-modal prediction errors closest to ground truths without considering the prediction scores.
Baselines. We also directly provide the prediction results displayed on the official leaderboards in Tables 4 and 5. Ablation studies in Table are reproduced by the official codes [22, 53). The prediction performance of BeTooNet is compared against SOTA baselines by: 1) GNN-enabled interactive graph [37, 24]: 2) conditional or thme-the hehavioral interactions [31, 8]; DETR- based Transformer attentions [22, 25, 53, 86]; and 4) auto-regressive modeling [32].
20

---

Table 9: Detailed nuPlan closed-loop simulation results in Val14 benchmark. BeTTopNet highlights
leading results among SOTA methods in safety and compliance, outperforms learning-base agents.
Val14
Type
Method
CA t
TTC 1
DDC1 1
DC
EP 1
Speed f
CLS-NR
CLS-R
Expert
Log Replay
0.987
0.944
0.981
0.993
0.989
0.965
0.937
0.812
IDM [70]
0,909
0.834
0.941
0.944
0.862
0,973
0.793
0.793
Rule
PDM-Closed [69]
0.981
0.933
0.998
0.955
0.921
0.998
0.932
0.930
Hybrid
GameFormer [8]
0,943
0.867
0.948
0.933
0.890
0.987
0.829
0.838
UrbanDriver [74]
0.856
0.803
0,908
1.000
0,808
0,915
0.677
0.648
PDM-Open [69]
0.745
0.691
0,879
0.995
0.698
0,977
0,502
0.548
PlanCNN [72]
0.869
0.814
0.850
0.814
0.806
0.980
0,669
0.646
Learning
GC-PGP [83]
0.858
0,801
0.897
0.900
0.603
0,993
0.611
0,549
PlanTT [73]
0,941
0.907
0.968
0.937
0.898
0.977
0.853
0.771
PLUTO [92]
0.961
0.933
0.985
0.964
0.895
0.981
0.890
0.800
Be'TopNet (Ours)
0.966
0.916
0.995
0.932
0,866
0.971
0.883
0.837
Table 10: Detailed nuPlan closed-loop planning results (PDMScore) on Test14-Random bench- mark.
Test Random
Type
Method
Col. Avoid
Drivable
Direction 1
Progress f
TTC
Comfort
PDMScore
Expert
Log Replay
0.996
0.962
0.996
0.664
0.985
1.000
0.832
Rule
PDM-Closed [69]
0,934
0.984
0.996
0.867
0,911
0.996
0.888
Constant Ace.
01846
0.907
0.915
0.436
0.804
1.000
0.592
UrbanDriver [74]
0.965
0.961
0.986
0.611
0.957
1.000
0.788
Learinng
PlanCNN [72]
0.935
0,938
0.971
0.888
0.989
0.736
PlanTF [73]
0,966
0.948
0.625
0.626
0,918
0.992
0.768
BeToppNet (Ours)
0.989
0.977
0.989
0.673
0,969
1,000
0.833
Table 11: Detailed nuPlan closed-loop plamning results (PDMScore) on Test14-Hard benchmark.
Test14 Hard Progress
Type
Method
Col. Avoid
Drivable
Direction
TTC
Comfont
PDMScore
Expert
Log Replay
0.985
0.945
0,970
0.658
0.955
1.000
0.786
Rule
PDM-Closed [69]
0.933
0.952
0.976
0.779
0.852
0.981
0.811
Constant Ace.
0.845
0.871
0.861
0.415
0.800
1.000
0.552
UrbunDriver [74]
0.946
0.944
0,992
0.581
0.903
1.000
0.731
Learning
PlanCNN [72]
0.909
0.908
0.937
0.555
0.860
0.992
0.675
PlanTF [73)
0.984
0.961
0.996
0.649
0.961
0.996
0.813
BeToopNet (Ours)
0.968
0.945
0.972
0,747
0.908
0.996
0,813
D.3 Training Setup
BeToopNet for both prediction and planning tasks are trained in end-to-end manners by A optimizer with 4 NVIDIA A100 GPUs. The learning rate is configured l scheduled with the multi-step reduction strategy. The planning model is trained by 25 epochs with batch size of 128, while the prediction task is trained with 30 epochs with a batch of 256.
E
Additional Quantitative Results
E.1 Planning
Additional planning results in Val14. We evaluate the closed-loop simulation performance under Val14 in Table 9, BeTopNet hovers strong planning results and is comparable (+4.6% CLS) to con- current learning- baaning-based methods [92] leveraging extra contrasting learning for training augmentations. BeTopNet is also featured by leading driving safety (+2.7% CA, +1.0% TTC) and compliance (+2.8% DDC) compared with other strong models [73, 8]. However, due to the data leakage of Vall4 with training by part of shared scenarios, only provide the results reference.
Additional planning effects Test14. To delve into the planning results of BeTopNet, we present statistics measuring by another detailed metric, PDMScore, for both of the Test/4 benchmarks in Table Exhibited Tables 10 and 11, BeTooeeet delivers strong maneuver safety and compliance, marking solid PDMScore from both benchmarks. Compared with learning-baat based methods, BeTopNet
21

---

Table 12: Marginal predictions WOMD Motion Leaderboard [81].
Primary metric.
Category
Method
minADE L
minFDE
Miss Rate 4
1AP f
Soft mAP t
MTR [22]
0.7642
1,5257
0.1514
0.4494
0.4590
Vehicle
EDA [53]
0.6808
1.3921
0.1164
0,4833
0.4972
BeToopNet (Ours)
0.6814
1,3888
0.1172
0.4860
0.4995
MTR [22]
0.3486
0.7270
0.0753
0.4331
0,4409
Pedestrian
EDA [53]
0.3426
0.7080
0.0670
0,4680
0.4778
BeTopNet (Ours)
0.3451
0.7142
0.0668
0.4777
0.4875
MTR [22]
0.7022
1.4093
0.1786
0.3561
0,3650
Cyclist
EDA [53]
0.6920
1.4106
0.1673
0.3947
0,4037
BeToppet (Ours)
0.6905
1.3975
0.1688
0,4060
0.4163
Table 13: Joint predictions on WOMD Interaction Leaderboard [82].
Primary metric.
Category
Method
minADE
minFDE
Miss Rate
mAP
Soft mAP
GameFormer [8]
1.0499
2.4044
0.4321
0.2469
0,2564
Vehicle
AMP [32]
0.9862
2.2286
0.3726
0.3104
0.3196
BeToopNet (Ours)
1,0216
2.3970
0.3738
0.3374
0.3308
GameFormer [8]
0.7978
1.8195
0.4713
0.1962
0.2014
Pedestrian
AMP [32]
0.6823
1.5244
0.3716
0.2359
0.2423
NetoopNet (Ours)
0). 0.7862
1.8412
0.4074
0.2212
0.2267
GameFormer [8]
1. L.0686
2.4199
0.5765
0,1367
0.1338
Cyclist
AMP [32]
1.0533
2.3715
0,5194
0,1420
0.1477
BeTopNet (Ours)
1.1155
2,5850
0,5253
0.1717
0.1756
excels in closed- driving progress (+15.1%, +7.5% EP), safety (+5.6% TTC, +2.9% CA), and the general score (+8.5% PDMScore). For rule-based systems, the leading performance is empirically by virtue of constant driving progress. This may refer to unresolved copy-cat problem [73] for imitative planners. It requires further integration and fallback with ule-bbsed methods for on-board AD system design in practice.
E.2 Prediction
Per-category marginal prediction. In Table 12, We mainfest the prediction performance of BeTTop- Net under each prediction category. Compared against the concurrent SOTA motion predictors [53]. BeToppet demonstrates superior mAP-based metrics among all types for compliant predictions. Specifically, overall improvements in Cyclist denote refined interactive patterns captured by Be'Top, as the cyclist predictions are the most uncertain task with less reliance on map information.
Per-category joint prediction. We further instantiate the per-category joint prediction of Be BoTopeet with SOTA methods in Table 13. Compared with concurrent methods [32] featuring auto-regressive decoding, BeTTopNet achieves robust displacement metries, while outperforming in prediction com- pliance of mAP metrics (+8.7%, + +0.9% mAP) due to advanced joint modality scoring stabilized by edge topology in BoToNeet. Moreover, the coordinated joint behaviors reasoned by BeToopett largely mitigate the unstable patterns against game-theoretic method [8] (-15.6% 11.7%% -9.7% Miss Rate) under similar model architecture.
F Additional Ablation Studies
Scaling effects of model and decoding agents. The scalability challenges begin with the scaling of our BeTopNet models to accommodate varying scene agents and map. Experimentally, we configure BeTopNet with different model scales evaluate whether our approach maintains its effectiveness.
In Table 14, BeTopNet is evaluated by three model scales varying in decoding modalities and dimensions. The results demonstrate that BeToppee consistently improves prediction accuracy, with an increase from 0.391 to 0.442 (+13.4% mAP) and a decrease in the Miss Rate (-11.9%). This showcases its enhanced robustness in handling multi-agent settings by enlarging model scales.
In Table 15, BeTopNet reports comparable computational costs compared [22], while with better prediction accuracy shown in Table 12. The similar latency is due to the topo-guided attention,
22

---

Table 14: Effects of varied model scale. BeTopNet shows scalability with the number of decoding modal- ities and feature dimensions.
Table 15: Effects of varied decoding agents.
Computational costs of [22] are reported in the
parenthesis after ours.
Scale
mAP
Miss Rate
Latency (ms)
# Params, (M)
Decoding Agents
Latency (ms)
GPU Memory (G)
Small
0.391
0.131
45
28.91
8
(84)
6,3 (5.2)
Medium
0.437
0.119
28.91
16
120 (123)
10.8 (7.1)
Base
0.442
0.117
70
45.38
32
166 (193)
19.2 (15.6)
Table 16: Effects of varied temporal granularity in BeTop. Future interactions are split into various intervals for multi-step BeToop labels. A fine-grained topology reasoning for the whole prediction horizon results in a slightly improved performance and increased computational costs simultaneously.
Inference Latency (ms)
Training
Params.
Interval
minADE
minFDE
Miss Rate
mAP
Latency (ms)
(M)
((Base)
0.637
1.328
0.144
0.392
70.0
101.6
45,380
2
0.633
1.325
0.145
0.394
75.5
110.6
45,382
0.634
1,326
0.142
0.391
80.0
133.4
45,386
0.641
1.347
0.147
0.389
90.0
255.0
45,393
Table 17: Effects of varied model foundation by Wayformer [62]. Synergistic decoder design by
BeTopNet demonstrate solid multi-agent interaction understanding compared with vanilla design.
Method
minADE minFDE
Miss Rate |
mAP*
Wayformer
0.661
1.417
0.199
0.281
Wayformer+BcTop
0.637
1.364
0.178
0.290
Beypoee
0.604
1.261
0.166
0.344
which reduces the KV features in agent aggregation during decoding. While Be'Top introduces extra computations for reasoning, it requires more GPU memory for cached topology tensors.
Temporal granularity in BeTop. Table 16 explores effect varied temporal granularity in BeTop. with minimal adjustments to BeTopNet. In study, future interactions are split into multi- step BeTop labels. Topology reasoning task is then deployed through expanded MLP Topo Head for output steps. Compared to the baseline 1-step reasoning. multi-step BeTop reasoning slightly improves performance (e.g.. 2-steps, +0.2 mAP), with a corresponding increase computational costs for additional steps. This highlights the potential of multi-step reasoning to enhance Be BeToopNet in interactive scenarios, while refining temporal granularity for more accurate and efficient interactions remains an open question. We believe how effectively leverage multi-step BeTopNet represents an interesting area for future exploration.
Synergy with additional model foundation. To understand the generalization under different model foundations, we conduct additional ablations integrating BeTop with reproduced Wayformer [62] in [95]. As reported in Table 17, incorporating BeToop as supervision improves vanilla Wayformer with - 6.2% Miss Rate and +3.2% mAP. Furthermore, integrating BeTopNet significantly boosts performance, achieving + 18.6% mAP and - 77.2% Miss Rate. This enhancement is due to syn- ergistic decoder design, which uses iterative Bop reasoning and Topo-guided attention to refine trajectories by selectively aggregating interactive features.
G Additional Qualitative Results
Additional planning results. We provide the qualitative closed-loop simulations for all of the benchmarks in 1est/4, as shown in Figs. 7 and 8.
Additional prediction results. We provide the qualitative prediction results for BeTTop with reasoned edge topology under both marginal (Fig. 10) and joint (Fig. 9) challenge settings.
H License of Assets
Data for nuPlan [80] and WOMD [35] are complied with CC-BY-NC 4.0 licence and Apache License 2.0; The code for re-implementations are under Apache License 2.0 for PDM [69] and MTR [22], and
23

---

Test I 4-Hard
Testl 4-Random
Qualitative results of BeTopNNe in nuPlan planning under Test14 simulations. Each row of the figures render closed-loop simulations at Is, 8s, and 15s temporal frames. As illustrated, BeTTopNet performs consistent planning under challenging driving scenarios of diverse categories.
Figure 7:
MIT License for EDA [53] and Plan [73], respectively. The source code and our trained models will be publicly available under the Apache License 2.0.
24

---

p=0
p=1
1=0s
1=86
1=0s
1-85
Joint Prababilty
Prediction
Planning
Figure 8: Qualitative results of BeTopNet in nuPlan planning under Test14-Inter. BeTopNet performs compliant planning under: (a) yielding to front agents; (b) cruising various road structure; (c-d) interactive behaviors among two or more agents with dense traffic.
Vehicle vehicle (V2V)
Vehicle to cyclist [V2C]
Vehicle pedeestrann [V2P)
p=0
p=1
t=0s
f=8s
e=1
Joint Probabillyy
Prediction
Edge Topology
Figure 9: Qualitative results of BeTopNet in WOMD joint prediction. Joint predictions among heterogeneous agents are categorized by each column (V2V, V2C, and V2P) with corresponding TopK reasoned topology. As depicted, BeTopNet edge topology reasoning compared with the human annotations of interactive agents (rendered in red). Moreover, BeTopNet may source potential interactions rendered in grey.
accurately capture the future interactive behaviors via
25

---

p=0
p=1
f=8s
e=0
e=]
Marginal Probabiity
Prediction
Edge Topology
Figure 10: Qualitative results of BeTooNet in WOMD marginal prediction. BeTopNet performs compliant and accurate marginal predictions multiple agents, reasoning diverse edge topology which stabilizes the behavioral patterns for future interactions.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Clear introduction with contributions and scopes in Sec. 1. We provide extra insightful Q&A in Appendix A to further position our scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Sec. 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

We provide comprehensive formulations in Sec. 3.1. We also provide full analytical proof Appendix B, as well as empirical verification in Fig. apart from experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Main experimental results of Sec. 4.1 are listed directly from the official leaderboards. We provide the methodology for formulation, model structure and optimiza- tion process in Sec. 3. The corresponding details for implementations and experiments are expressly illustrated in Appendices C and D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
28

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: All data, baselines and results are already publicly available [80, 35]. Code will be available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https:/nnip) cc/
public/gute httpii/gguidee//oeeuubmmim smoolicyyyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/nips. cec/public/guu http://gguiieeeodeuuubbiiiii wwwiibmisssonPeo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See details in Sec. 4. We also provide some settings in Appendix D and will release the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: All the experiments are tested by weighted mean metries for official compar- isons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide compute resource details and related information in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httppp/////////.....e/// cc i e/Ethi Seluuuddeeines"
Answer: [Yes]
Justification:
The research conforms the Code of Ethics all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed broader impacts in Q3 in Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the papers for the used datasets and models in the paper, and list corresponding licenses in Appendix H.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithheeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release assets including code and models. Details are in Ap- pendix H.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "IoNIGVjMOi", "reviewer": "Reviewer_PWLC", "summary": "This paper introduces a new approach, called Behavioral Topology (BeTop), to address the challenges in modeling multi-agent behaviors in autonomous driving. By utilizing braid theory, BeTop explicitly represents the consensual behavioral patterns among multiple agents, facilitating better prediction and planning. The framework, BeTopNet, incorporates this topological reasoning into a synergistic learning model that guides both behavioral prediction and planning. Good experiments on large-scale datasets, such as nuPlan and WOMD, demonstrate the superior performance of BeTopNet in both prediction and planning tasks, showcasing significant improvements over existing methods.", "questions": "Lack of Discussion on Recursive Settings: While the paper introduces a topological approach for multi-agent behavior modeling, it lacks an in-depth discussion on how this method scales and handles various steps of multi-agent settings. More insights into the limitations and potential scalability issues would strengthen the paper. While the braid theory is used to model interactions, a clearer and more comprehensive explanation of how this integrates with different numbers and types of agents would be helpful.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Good Presentation: The paper is well-organized and clearly presents the motivation, methodology, and results. The introduction of BeTop is logically structured, and the figures help in understanding the complex concepts.\n\nReasonable Formulation: The use of braid theory to represent multi-agent interactions is innovative and provides a solid theoretical foundation. This formulation helps in capturing the interactive behaviors more effectively compared to traditional dense or sparse representations.\n\nExtensive Experiments: The authors have conducted comprehensive experiments on large-scale real-world datasets. These experiments cover both prediction and planning tasks, providing a thorough evaluation of the proposed method.\n\nPerformance Improvement: The experimental results demonstrate that BeTopNet achieves improved performance in prediction and planning tasks, especially in planning scores and prediction accuracy, with detailed metrics provided to back these claims.", "weaknesses": "Lack of Discussion on Multi-Agent Settings: While the paper introduces a topological approach for multi-agent behavior modeling, it lacks an in-depth discussion on how this method scales and handles various multi-agent settings. More insights into the limitations and potential scalability issues would strengthen the paper.\n\nFormulation for Multi-Agent Settings: The paper could benefit from a more detailed formulation of the multi-agent setting. While the braid theory is used to model interactions, a clearer and more comprehensive explanation of how this integrates with different numbers and types of agents would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation and the helpful review of our work. We address your concerns below.\n\n>**W1/Q1:** Lack of Discussion on Multi-Agent Settings: While the paper introduces a topological approach for multi-agent behavior modeling, it lacks an in-depth discussion on how this method scales and handles various steps of multi-agent settings. More insights into the limitations and potential scalability issues would strengthen the paper. \n\n\n1. **How does this method scale:** \n\n- **Scaling model size:** In the rebuttal, we *additionally* ablate BeTopNet with different model scales. We adjust the number of decoding modalities and feature dimensions to get three sizes of models. The results are as follows and we have added the experiment in the revision.\n\n| Method| Description|mAP|Miss Rate|Latency (ms)|# Params. (M)|\n|-|-|-|-|-|-|\n| BeTopNet-Small|Mode=6, DecDim=256|0.391|0.131| 45|28.91|\n| BeTopNet-Medium|Mode=64, DecDim=256|0.437|0.119|65|28.91|\n| BeTopNet-Base|Mode=64, DecDim=512|**0.442**|**0.117** |70|45.38|\n\n- **Scaling number of decoded agents:** Besides, we provide the computational complexity when the number of decoded agents increase during prediction, which is also a common challenge for the multi-agent setting. A preliminary computation study is conducted for BeTopNet against MTR:\n\n|Method|# Decode Agents|Latency (ms)|GPU Memory (G)|\n|-|-|-|-|\n||\n| MTR [22]|8|84|5.2|\n| MTR [22]|16|123|7.1|\n| MTR [22]|32|193|15.6|\n||\n|BeTopNet (Ours)|8|89|6.5|\n|BeTopNet (Ours)|16|120|10.8|\n|BeTopNet (Ours)|32|166|19.2|\n\nWe can observe that BeTopNet reports comparable latency and memory costs compared to MTR, with better prediction accuracy shown in the paper. The similar latency is due to the topo-guided attention, which reduces the KV features in agent aggregation during decoding, thereby decreasing the main computation cost for multi-head attention tensors. While BeTop introduces extra computations for reasoning, it requires more GPU memory for cached topology tensors. In practice, these computational challenges might be addressed through knowledge distillation or half-precision computations to reduce GPU requirements. We will enrich the above discussion in our revised version.\n\n2. **How does this method handle various steps of multi-agent setting:** As mentioned in Lines 300-301, another challenge and direction for scalability is the development of Multi-step BeTop. In this paper, we clarify that multi-step BeTop refers to using multiple intervals to summarize entire future interactions with different temporal granularities. \n\nTo explore this, we add a *new ablation experiment* to evaluate the effect of multi-step BeTop with minimal adjustments to the current BeTopNet framework. \n\n**We refer to the Multi-step results and discussions in the *Global Rebuttal* above.**\n\n\n\n>**W2:** Formulation for Multi-Agent Settings: The paper could benefit from a more detailed formulation of the multi-agent setting. While the braid theory is used to model interactions, a clearer and more comprehensive explanation of how this integrates with different numbers and types of agents would be helpful.\n\nThanks for your question. \n\n1. To have a more intuitive understanding of how BeTop deals with varied agent numbers and categories:\n\n  - For **varied agent numbers**:  Each BeTop would outline whole scene agents. We leverage batched padding by the maximum scene agent number per batch, and generate the padding mask during batched BeTop formulation. Hence, BeTop can be formulated uniformly by batched padding mask.\n  - For **varied agent types**: Variations in agent types would not affect the formulation process. BeTop is only defined through states $(x, y, \\theta)$ from multi-agent future trajectories. In fact, the agent types are considered in multi-agent states $\\mathbf{X}$ and queries $\\mathbf{Q}_A$ as model inputs.\n\n2. The detailed formulation of BeTop (Lines 127-146) is summarized for the multi-agent future interactions.\n\n**The inputs :**\n\n  - (1) Multi-agent future trajectories $\\mathbf{Y}$ of states $(x, y, \\theta)$; Tensor shape: $[N_a, T_f, 3]$. (We omit subscript $a$ and $f$ below for simplicity.)\n  - (2) Agent padding mask $M$; Tensor shape: $[N]$\n\n**The formulation process:**\n\nLoop for index $i$, $j$ from 1 to $N$; \n\n  - (1) Calculate $e_{ij}$ by firstly index the future state tensor of sourced agent $\\mathbf{Y}[i]$ and targeted agent $\\mathbf{Y}[j]$; Both tensors have tensor shape $[T, 3]$;\n  - (2) Conduct the LocalTransformation() function by agent $i$ for sourced agent $\\mathbf{Y}[i]$ and targeted agent $\\mathbf{Y}[j]$. This is linked to the Braid function mapping by $(f^i_i, f^i_j)$ in Line 132-134, and refers to the process in Line 143-144; Both tensors output shape $[T]$ for lateral states;\n  - (3) Passed through the `SegmentIntersect()` function [76] for locally transformed $\\mathbf{Y}[i]$ and $\\mathbf{Y}[j]$ referred to the function `I(.,.)` in Line 144 with shape $[T]$; Conduct the max-pooling across $T$ for the finalized $e_{ij}$ summarizing the future interactions as one-step (one-interval)\n  - (4) Let $\\mathcal{E}\\_{ij} = e\\_{ij}$\n\nExpand the agent padding mask $MM^T$ ( tensor shape: $[N, N]$), and mask BeTop $\\mathcal{E}$\n\n**Output:** masked BeTop $\\mathcal{E}$ for varied agents; Tensor shape: $[N, N]$\n\nIn practice, the formulation process is conducted in batches and calculated using parallelized tensor multiplications instead of the loop iterations for more efficient computation.\n\nWe hope the detailed formulations will aid in a better understanding of the multi-agent settings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GshMBLCkdH", "reviewer": "Reviewer_M7XC", "summary": "The paper addresses the challenges of autonomous driving by integrating behavior among interactive agents, specifically focusing on issues caused by multi-agent scene uncertainty and heterogeneous interactions.  To tackle this, the paper introduces a topological formation called Behavioral Topology (BeTop), derived from braid theory, to represent consensual behavioral patterns among multiple agents. This formulation guides downstream trajectory generations and enhances the consistency of behavior prediction and planning.", "questions": "See weaknesses", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.\tThe experimental results seem to support the authors' claims.\n2.\tIt is developed based on exiting braids topology. Novel method for an existing problem setup. \n3.\tThe use of braid theory to distill compliant interactive topology from multi-agent future trajectories seems a good and intuitive idea to me.", "weaknesses": "The paper is generally not well-written, with extensive use of ChatGPT leading to paragraphs that are hard to follow.\n\n- As far as I understand, the paper uses braid topology for just one step in planning and prediction. In long-horizon planning and prediction, this may not be sufficient as the motion of vehicles is  no longer independent. Could you provide some basis to the idea why one step topology will be enough? \n\n-  There has been prior work using braid topology for planning (https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9812118). Benchmarking your method against this prior work would provide a clearer picture of this method. \n\n- The method appears very similar to existing methods like Wayformer, with the main difference being the use of braid topology. I would like to see a detailed comparison showing how much the encoding of braid topology improves performance compared to Wayformer, especially given that only one-step braid topology is used instead of long-horizon topology.\n\nOverall, the paper has potential, and I would be happy to discuss more with you. I would be willing to increase my score if my questions and concerns are addressed satisfactorily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions and we really appreciate your comments. We have carefully revised the draft for better readibility. We address each of the questions and confusion on the weaknesses as follows.\n\n>**W1:** As far as I understand, the paper uses braid topology for just one step. Could you provide some basis to the idea why one step topology will be enough?\n\nThank you for the question. We would like to clarify that \"one step\" in the BeTop label refers to an interval summarizing interactions across the entire future horizon (T=8s, 10Hz), rather than a single timestep, illustrated in Lines 143-145. As detailed in Line 144, $e_{ij}=1$ signifies any future interaction between agents $i$ and $j$, while $e_{ij}=0$ indicates no interaction across all timesteps. Variations in BeTop labeling capture interactions over the full horizon with different temporal granularity.\n\nAs mentioned in Lines 300-301, another challenge and direction for scalability is the development of Multi-step BeTop. In this paper, we clarify that multi-step BeTop refers to using multiple intervals to summarize entire future interactions with different temporal granularities. \n\n**We refer to the multi-step results and discussions in the Global Rebuttal above.**\n\n>**W2:** There has been prior work using braid topology for planning (https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9812118). Benchmark would provide a clearer picture of this method.\n\nThank you for your question. The work in reference [43] has been cited in the paper, but we understand that it may cause some confusion. The differences of our work from [43] are as follows:\n\n- The paper [43] *does not involve planning tasks*, but rather focuses on traffic scenario analysis using topological braids to calculate an Interaction Score, as in Lines 88-90. In contrast, BeTop formulates behavioral topology and integrates it into joint prediction and planning using BeTopNet.\n- In terms of formulation,  [43] converts braid interactions into braid words, which is suitable for calculating the TC index but challenging for reasoning due to exponential complexity. While BeTop uses topological representations to simplify the formulation process, allows for reasoning and planning tasks with quadratic complexity. \n- While [43] conducts case studies on real-world datasets using log-replayed trajectories, BeTopNet includes reasoning in both prediction and planning benchmarks. This distinction highlights how BeTop goes beyond simple quantification to actively engage in prediction and planning processes. \n\nWe will enrich these discussions to provide further clarity.\n\n>**W3:** I would like to see a detailed comparison showing how much the encoding of braid topology improves performance compared to Wayformer, especially given that only one-step braid topology is used instead of long-horizon topology.\n\nThank you for your question. We understand that the similarities in the encoder-decoder structure might cause some confusion, so we we'd like to firstly highlight a few key differences between BeTopNet and Wayformer [61]. In the meantime, we conduct *additional ablation studies* to demonstrate BeTop's effectiveness in the Wayformer framework.\n\n**Key Differences:**\n  - **Model Foundations**: As mentioned in Section 3.2 in the main paper and Section C.1 in the Appendix, BeTopNet follows the encoder and anchor-based decoder design of Motion Transformer (MTR) [22]. While Wayformer focuses on encoder novelty, BeTopNet centers on decoder design, employing a synergistic Transformer decoder that jointly decodes trajectories and reasons BeTop interactions. This is contributed by topo-guided attention, which leverages reasoned topology for compliant predictions.\n  - **Encoding**: BeTopNet's encoder is based on MTR and uses local self-attention to encode all scene agents. Wayformer handles surrounded agents to avoid out-of-memory issues.\n  - **Decoding Strategies**: BeTopNet uses iterative refining strategies to selectively aggregate encoded features and decode trajectories for each decoder layer. In contrast, Wayformer uses stacked vanilla Transformer decoders for one-shot trajectory output.\n  - **Attention Design**: Wayformer’s latent query attention focuses on encoding query reduction, similar to MTR's local attention. BeTopNet uses guided attention for cross-attention in decoder, with reduction based on sorted BeTop results supervised by BeTop labels.\n\nGiven differences, our studies primarily compare BeTopNet against MTR . In the rebuttal, we conduct additional ablations with BeTop in Wayformer to understand the generalization under different model foundations.\n\n**Ablative Experiments:**\n  - **Wayformer+BeTop**: The impact of using BeTop as additional supervision for interactions within the vanilla Wayformer model.\n  - **Wayformer+BeTopNet**: This combines the Wayformer encoder with the BeTopNet decoder design, showcasing our key contribution of topo-guided attention and iterative reasoning for complex interactions.\n\nThe ablations use protocols outlined in Lines 274-277 with end-to-end decoding approach (Modality=6). We use the reproduction from (https://github.com/vita-epfl/UniTraj), since Wayformer is not open-sourced. The results are as follows:\n\n|Method|mAP|minADE|minFDE|Miss Rate|\n|-|-|-|-|-|\n|Wayformer|0.281|0.661|1.417|0.199|\n|Wayformer+BeTop|0.290|0.637|1.364|0.178|\n|**Wayformer+BeTopNet**|**0.344**|**0.604**|**1.261**|**0.166**|\n\n Compared to the vanilla Wayformer, incorporating BeTop as supervision improves performance with a -6.2% Miss Rate and +3.2% mAP.  Furthermore, integrating BeTopNet significantly boosts performance, achieving a +18.6% mAP and -7.2% Miss Rate. This enhancement is due to our ***synergistic decoder***, which uses iterative BeTop reasoning and Topo-guided attention to refine trajectories by selectively aggregating interactive features. These results demonstrate BeTopNet's superior ability to enhance prediction and planning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3JoNwjkUhg", "reviewer": "Reviewer_PsGi", "summary": "The paper introduces a novel approach to enhance the safety and social consistency of autonomous driving systems through improved multi-agent behavioral integration. To address inefficiencies and inconsistencies in current behavioral representations, the authors propose Behavioral Topology (BeTop), a topological framework derived from braid theory that captures consensual behavioral patterns among multiple agents. This framework guides downstream trajectory generations and ensures stable collective behavior when integrating prediction and planning. The paper also presents BeTopNet, a synergistic learning framework supervised by BeTop that manages behavioral uncertainty and enhances prediction and planning consistency. Extensive experiments on large-scale real-world datasets, including nuPlan and WOMD, demonstrate that BeTop achieves state-of-the-art performance in prediction and planning tasks, showcasing its effectiveness in interactive scenarios", "questions": "1. It is unclear how is BeTop used during inference when future trajectories for surrounding agents are unavailable.\n\n2. It is unclear how are $Q_R$ and $Q_A$ initialized and defined.\n\n3. How do you ensure the robustness of BeTopNet in highly dynamic and unpredictable driving environments, such as those with sudden changes or unexpected behaviors from other agents?", "limitations": "Please refer to the weakness section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper introduces a novel approach to enhance the safety and social consistency of autonomous driving systems through improved multi-agent behavioral integration. To address inefficiencies and inconsistencies in current behavioral representations, the authors propose Behavioral Topology (BeTop), a topological framework derived from braid theory that captures consensual behavioral patterns among multiple agents. This framework guides downstream trajectory generations and ensures stable collective behavior when integrating prediction and planning. The paper also presents BeTopNet, a synergistic learning framework supervised by BeTop that manages behavioral uncertainty and enhances prediction and planning consistency. Extensive experiments on large-scale real-world datasets, including nuPlan and WOMD, demonstrate that BeTop achieves state-of-the-art performance in prediction and planning tasks, showcasing its effectiveness in interactive scenarios", "weaknesses": "In general, the paper is well written and there are no major weakness, however, there are some aspects that can be further discussed in the paper: 1.While the paper demonstrates effectiveness on specific datasets, it remains uncertain how well the method generalizes to diverse driving environments and conditions not covered in the training data. 2. The computational overhead associated with the topological framework and synergistic learning might be higher compared to simpler models, possibly affecting real-time performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our work. We address your questions below.\n\n> **W1:** While the paper demonstrates effectiveness on specific datasets, it remains uncertain how well the method generalizes to diverse driving environments and conditions not covered in the training data. \n\nThank you for your question. As mentioned in Lines 233-234 and Lines 239-240, we evaluate the performance of our method on both prediction and planning tasks using independent test sets, specifically Test14 and WOMD Test, as detailed in Tables 2-5 of the main text. These test sets are all private testing scenarios against the coverage in training data, ensuring that our method is evaluated objectively.\n\nFurthermore, as noted in Lines 708-712, the validation sets in nuPlan may share certain scenarios with the training set. Therefore we argue that the widely-used Val14 benchmark may not be fully representative, and have thus placed the Val14 results in the appendix (Table 9). In the main text, we focus on reporting test results for a more objective assessment of our method's generalizability.\n\n> **W2:** The computational overhead associated with the topological framework and synergistic learning might be higher compared to simpler models, possibly affecting real-time performance.\n\nAgreed. This may potentially lie in computations with model scale and decoded agents, which are the common challenges for the learning-based predictors with multi-agent settings. A preliminary computation study is conducted for BeTopNet against the MTR baseline using a single A100 GPU:\n\n| Method| Latency (ms) | GPU Memory (G) |\n|-|-|-|\n| MTR [22]| 84|5.2|\n| BeTopNet (Ours) | 89|  6.5 |\n\nWe can observe that BeTopNet reports comparable latency and memory costs compared to MTR, with better prediction accuracy shown in the paper. The similar latency is due to the topo-guided attention, which reduces the KV features in agent aggregation during decoding, thereby decreasing the main computation cost for multi-head attention tensors. While BeTop introduces extra computations for reasoning, it requires more GPU memory for cached topology tensors. In practice, these computational challenges might be addressed through knowledge distillation or half-precision computations to reduce GPU requirements. We will enrich the above discussion in our revised version.\n\n\n> **Q1:** It is unclear how is BeTop used during inference when future trajectories for surrounding agents are unavailable.\n\nBeTop serves as a label to supervise the reasoning process in BeTopNet. During both training and inference, the reasoned BeTop is used within BeTopNet to guide predictions. This can be referred to Figure 3, Reason heads in Lines 196-197, and Training loss in Lines 205-206. We will improve the content accordingly for better understanding.\n\n> **Q2:** It is unclear how are $𝑄_𝑅$ and $𝑄_𝐴$ initialized and defined.\n\n$\\mathbf{Q}_A$ is initialized by learnable embeddings. For the prediction task, the embedding will also be added with anchored features. These anchored features are predefined by K-means end-point anchors, referred to the clustering process in MTR [22].\n\n$\\mathbf{Q}_R$ is initialized by MLP encoding of relative features of $\\mathbf{S}_R$ [63].\n\nMore details can be referred to Lines 637-639 in the Appendix and references [22, 63]. We will enrich extra clarifications in the revised main context accordingly.\n\n> **Q3:** How do you ensure the robustness of BeTopNet in highly dynamic and unpredictable driving environments, such as those with sudden changes or unexpected behaviors from other agents?\n\nThank you for the insightful question. Our method has tried to tackle and validate these challenges as follows:\n\n- **Methodology**: BeTopNet ensures interactive robustness through its synergistic decoder design and topo-guided attention, which iteratively refines predictions by focusing on potential future interactions (by reasoned BeTop), and reacts to possible changes in the next layer of decoding.  For prediction/planning output, we use a Gaussian Mixture Model (GMM) to account for dynamic uncertainty. The contingency learning paradigm formulates predictions and planning guidance that balance short-term safety and long-term compliance. Expressely, this paradigm enhances short-term (0-3s) safe planning by considering worst-case predictions and conducting long-term branching (3-8s) for various prediction uncertainty.\n- **Experiments**: BeTopNet's robustness can be demonstrated in Test14-Hard and Test14-Inter benchmarks. Test14-Hard highlights scenarios where rule-based planning agents perform poorly, indicating difficult environments, while Test14-Inter focuses on highly dynamic situations where physics-based planners struggle. Our strong results in these benchmarks verify BeTopNet's capability to tackle such challenging scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kdi22axE29", "reviewer": "Reviewer_aQzs", "summary": "The paper presents a approach to address the challenges of autonomous driving in multi-agent scenarios and heterogeneous interaction. It introduces the concept of Behavioral Topology (BeTop) and its corresponding network BeTopNet. BeTop is based on braid theory and aims to provide a topological representation of multi-agent interactions to enhance the prediction and planning of autonomous vehicles (AVs). The proposed method focuses on creating a compliant behavioral pattern among agents, which guides the trajectory generation for AVs. Extensive experiments on large-scale datasets such as nuPlan and WOMD demonstrate the good performance of BeTop in both prediction and planning tasks.", "questions": "* The proposed method infer one type of agent behavior topology from one mode of future trajectories (e.g., 8s), while the topology of multi-agent interaction for real-world autonomous driving is usually multi-modal and dynamic. A discussion of modeling multi-step and dynamic topologies of vehicles for a long horizon would be beneficial.", "limitations": "* The current implementation of BeTop considers only one-step future topology. Extending this to multi-step reasoning could provide more robust predictions and planning.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and polished.\n* The idea of using braid theory to explicit formulate the interactions among agents is interesting and lays a solid mathematical foundation for BeTop.\n* The proposed method effectively integrates prediction and planning in a unified framework.\n* The proposed method and baselines are extensively evaluated on two large-scale real-world datasets to demonstrate the performance on both motion prediction and planning.\n* Ablation studies show that each component of the proposed method contribute to its performance on the planning task.", "weaknesses": "* The proposed method infer one type of agent behavior topology from one mode of future trajectories (e.g., 8s), while the topology of multi-agent interaction for real-world autonomous driving is usually multi-modal and dynamic. A discussion of modeling multi-step and dynamic topologies of vehicles for a long horizon would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We address your questions below.\n\n>**W1/Q1:** The proposed method infer one type of agent behavior topology from one mode of future trajectories (e.g., 8s), while the topology of multi-agent interaction for real-world autonomous driving is usually multi-modal and dynamic. A discussion of modeling multi-step and dynamic topologies of vehicles for a long horizon would be beneficial.\n\n\nThanks for your insightful suggestions. As mentioned in Lines 300-301, another challenge and direction is the development of Multi-step BeTop. In this paper, we clarify that multi-step BeTop refers to using multiple intervals to summarize entire future interactions with different temporal granularities. \n\nTo explore this, we add a *new ablation experiment* to evaluate the effect of multi-step BeTop with minimal adjustments to the current BeTopNet framework. In our study, future interactions are split into 1 (base), 2, 4, and 8 steps (intervals) for multi-step BeTop labels. The multi-step topology reasoning task is then deployed through the current BeTopNet decoder, with an expanded MLP Topo Head for output steps. A max-pooling over BeTop steps is performed to comply with the indexing for topo-guided attention. The ablation results are as follows: \n\n|BeTop reasoning steps/intervals|mAP|minADE|minFDE|Miss Rate|Inference Latency (ms)|Training Latency (ms)| # Params. (M)|\n|-----------------------------------|-----------|-----------|-----------|-----------|------------------------|-----------------------|------------|\n|1/Base|0.392|0.637|1.328|0.144|70.0| 101.6|45.380|\n|2|**0.394**|**0.633**|**1.325**|0.145|75.5|110.6|45.382|\n|4|0.391|0.634|1.326|**0.142**|80.0|133.4|45.386|\n|8|0.389|0.641|1.347|0.147|90.0|255.0|45.393|\n\nCompared to the baseline 1-step reasoning, multi-step BeTop reasoning slightly improves BeTopNet's performance (e.g., 2-steps, +0.2 mAP), with a corresponding increase in computational costs for additional steps.\n\nThis result highlights the potential of multi-step reasoning to enhance BeTopNet in interactive scenarios. One-step BeTop performs relatively well because the current topo-guided attention is optimized for single-interval reasoning. However, the 8-step configuration shows a slight drop, which might result from the minimal adjustments for BeTopNet. Direct reasoning and Max-pool over multi-step BeTop at the topo-attention may not predict and capture multi-interval interactions effectively, leading to potential noise or information loss.\n\nThe current approach focuses primarily on formulating and integrating BeTop into the integrated prediction and planning (IPP) tasks, but refining temporal granularity for more accurate and efficient interactions remains an open question. We believe how to effectively leverage multi-step BeTop represents an interesting area for future exploration. We will enrich the experiment and discussion above in the revision accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
*Dear Area Chairs and Reviewers,*

*We thank all the Reviewers for their careful reviews and valuable comments on our work. We have taken each comment into consideration, added more ablative experiments in the rebuttal, and clarified some technical details. Please see each response below. We are grateful for the opportunity to improve our work with your guidance.*

*Best Regards,*  
*The Authors*


**Here we refer to the general question proposed by #Rew.PWLC and #Rew.M7XC:**

**Ablations and discussions for multi-step:** As mentioned in Lines 300-301, another challenge and direction for scalability is the development of Multi-step BeTop. In this paper, we clarify that multi-step BeTop refers to using multiple intervals to summarize entire future interactions with different temporal granularities. 

To explore this, we add a *new ablation experiment* to evaluate the effect of multi-step BeTop with minimal adjustments to the current BeTopNet framework. In our study, future interactions are split into 1 (base), 2, 4, and 8 steps (intervals) for multi-step BeTop labels. The multi-step topology reasoning task is then deployed through the current BeTopNet decoder, with an expanded MLP Topo Head for output steps. A max-pooling over BeTop steps is performed to comply with the indexing for topo-guided attention following the current design in Line 144. The ablation results are as follows: 

|BeTop reasoning steps/intervals|mAP|minADE|minFDE|Miss Rate|Inference Latency (ms)|Training Latency (ms)| # Params. (M)|
|-|-|-|-|-|-|-|-|
|1/Base|0.392|0.637|1.328|0.144|70.0| 101.6|45.380|
|2|**0.394**|**0.633**|**1.325**|0.145|75.5|110.6|45.382|
|4|0.391|0.634|1.326|**0.142**|80.0|133.4|45.386|
|8|0.389|0.641|1.347|0.147|90.0|255.0|45.393|

Compared to the baseline 1-step reasoning, multi-step BeTop reasoning slightly improves BeTopNet's performance (e.g., 2-steps, +0.2 mAP), with a corresponding increase in computational costs for additional steps.

This result highlights the potential of multi-step reasoning to enhance BeTopNet in interactive scenarios. One-step BeTop performs relatively well because the current topo-guided attention is optimized for single-interval reasoning. However, the 8-step configuration shows a slight drop, which might result from the minimal adjustments for BeTopNet. Direct reasoning and Max-pool over multi-step BeTop at the topo-attention may not predict and capture multi-interval interactions effectively, leading to potential noise or information loss.

The current approach focuses primarily on formulating and integrating BeTop into the integrated prediction and planning (IPP) tasks, but refining temporal granularity for more accurate and efficient interactions remains an open question. We believe how to effectively leverage multi-step BeTop represents an interesting area for future exploration. We will enrich the experiment and discussion above in the revision. 

*Please refer to the rebuttal modules below for our point-to-point responses to each reviewer.*

Meta-review: This paper introduces Behavioral Topology (BeTop), a novel approach to modeling multi-agent behaviors in autonomous driving using braid theory. The authors present BeTopNet, a framework that incorporates topological reasoning into a synergistic learning model for both behavioral prediction and planning.

The reviewers generally agree on the innovative nature of the approach. Reviewer PWLC notes that "the use of braid theory to represent multi-agent interactions is innovative and provides a solid theoretical foundation," while Reviewer aQzs states that it "lays a solid mathematical foundation for BeTop." The comprehensive evaluation is another strong point, with Reviewer PWLC highlighting the "extensive experiments on large-scale real-world datasets" and Reviewer PsGi confirming that these experiments "demonstrate that BeTop achieves state-of-the-art performance in prediction and planning tasks."

However, the paper has some limitations. Reviewer PWLC points out that it "lacks an in-depth discussion on how this method scales and handles various multi-agent settings." A major concern raised by multiple reviewers (M7XC, PsGi, and aQzs) is the use of only one-step topology. Reviewer M7XC questions, "Could you provide some basis to the idea why one step topology will be enough?" Reviewer M7XC also suggests benchmarking against prior work using braid topology for planning to provide a clearer picture of the method's contribution.

The presentation of the paper received mixed reviews. While most reviewers found it well-organized, Reviewer M7XC notes that "The paper is generally not well-written, with extensive use of ChatGPT leading to paragraphs that are hard to follow."

Several important questions were raised by the reviewers. Reviewer PsGi asks how BeTop is used during inference when future trajectories for surrounding agents are unavailable, and how the robustness of BeTopNet is ensured in highly dynamic and unpredictable driving environments. Reviewer PWLC suggests providing a clearer explanation of how the method integrates with different numbers and types of agents.

Recommendation:
Despite the concerns raised, the paper's innovative approach and comprehensive experiments are seen as significant contributions to the field. Three out of four reviewers recommend acceptance (two "Accept" and one "Weak Accept"), while one reviewer gives a "Borderline Accept."

Based on these reviews, the recommendation is to accept the paper, contingent on the authors addressing the following key points in the camera-ready version:

Provide a more detailed discussion on multi-step topologies and their potential benefits.
1. Clarify how the method scales to various multi-agent settings.
2. Include a comparison with existing methods, particularly those using braid topology for planning.
3. Address the questions raised about inference and robustness in dynamic environments.
4. Improve the clarity and readability of the paper, addressing the writing concerns raised by Reviewer M7XC.
Decision: Accept (poster)