LaKD: Length-agnostic Knowledge Distillation for Trajectory Prediction with Any Length Observations
Authors: Yuhang Li
Changsheng Li
Ruilin Lv
Rongqing Li
Ye Yuan
Guoren Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Trajectory Prediction
Primary area: robotics

Abstract: Trajectory prediction is a crucial technology to help systems avoid traffic accidents, ensuring safe autonomous driving. Previous methods typically use a fixed-length and sufficiently long trajectory of an agent as observations to predict its future trajectory. However, in real-world scenarios, we often lack the time to gather enough trajectory points before making predictions, e.g., when a car suddenly appears due to an obstruction, the system must make immediate predictions to prevent a collision. This poses a new challenge for trajectory prediction systems, requiring them to be capable of making accurate predictions based on observed trajectories of arbitrary lengths, leading to the failure of existing methods. In this paper, we propose a Length-agnostic Knowledge Distillation framework, named LaKD,  which can make accurate trajectory predictions, regardless of the length of observed data. Specifically, considering the fact that long trajectories, containing richer temporal information but potentially additional interference, may perform better or worse than short trajectories, we devise a dynamic length-agnostic knowledge distillation mechanism for exchanging information among trajectories of arbitrary lengths, dynamically determining the transfer direction based on prediction performance. In contrast to traditional knowledge distillation, LaKD employs a unique model that simultaneously serves as both the teacher and the student, potentially causing knowledge collision during the distillation process. Therefore, we design a dynamic soft-masking mechanism, where we first calculate the importance of neuron units and then apply soft-masking to them, so as to safeguard critical units from disruption during the knowledge distillation process. In essence, LaKD is a general and principled framework that can be naturally compatible with existing trajectory prediction models of different architectures. Extensive experiments on three benchmark datasets, Argoverse 1, nuScenes and Argoverse 2, demonstrate the effectiveness of our approach.

LaKD: Length-agnostic Knowledge Distillation for Trajectory Prediction with Any Length Observations
Yuhang
Changsheng Li Beijing Institute of Technology lesbbit. edu. on
Beijing Institute of Technology $36983629Qqq- com
Ruilin L Beijing Institute of Technology 32202314548bit. edu. on
Rongqing Li Beijing Institute of Technology (ttpinggiiinnggmmammmmm com
Ye Yuan
Guoren Wang
Beijing Institute of Technology yuan-yeßbie edu. cn
Beijing Institute of Technology danggrbit@111 26 com
Abstract
Trajectory prediction is a crucial technology to help systems avoid traffic accidents, ensuring safe autonomous driving. Previous methods typically use fixed-length and sufficiently long trajectory of an agent as observations to predict its future trajectory. However, in real-world scenarios, we often lack the time to gather enough trajectory points before making predictions, e.g., when a car suddenly appears due to an obstruction, the system must make immediate predictions prevent collision. This poses a new challenge for trajectory prediction systems, requiring them to capable of making accurate predictions based on observed trajectories of arbitrary lengths, leading to the failure of existing methods. In this paper, we propose a Length-agnoste Knowledge Distillation framework, named LaKD, which can make accurate trajectory predictions, regardless of the length observed data. Specifically, considering the fact that long trajectories, containing richer temporal information but potentially additional interference, may perform better or worse than short trajectories, we devise dynamic length- agnostic knowledge distillation mechanism for exchanging information among trajectories of arbitrary lengths, dynamically determining the transfer direction based on prediction performance. In contrast traditional knowledge distillation, LaKD employs a unique model that simultaneously serves as both the teacher and the student, potentially causing knowledge collision during the distillation process. Therefore, we design a dynamic soft-masking mechanism, where we first calculate the importance of neuron units and then apply noft-masking to them, so to safeguard critical units from disruption during the knowledge distillation process. In essence, LaKD is general and principled framework that can be naturally compatible with existing trajectory prediction models of different architectures. Extensive experiments on three benchmark datasets, Argoverse 1, nuScenes and Argoverse 2, demonstrate the effectiveness of our approach.
'Changsheng Li (les bit.edu.cn) is the corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

HTVT
NIVT QCNett
OCNer
3.25
minADE
3.75
1.25
1.00
0.75
Model performance
Model performance
Longer trajectory on Argoverse 2.
Shorter trajectory performs better.
(a)
(b)
on Argoverse 1.
performs better.
Figure I: Figure ((a) and Figure ((b) show the prediction results of HiVT 60 and QCNet [59] on the Argoverse 1 4 and Argoverse 2 [5T] datasets by using observed trajectories of different lengths, respectively. Figure [[c]] and Figure [(d) display scenarios where longer trajectories perform better and shorter trajectories perform better, respectively. The red line represents the ground-trui -rtuuh future trajectory. The solid green and blue lines depict the observed trajectories, while the dashed green and blue lines illustrate the predicted trajectories.
and
Introduction
Predicting the future trajectories of dynamic agents in traffic scenarios is critical task in autonomous driving, enabling autonomous vehicles to make safe decisions [55] [8] Recently, numerous learning- based methods [60 43
39
48
50
57
59 have been proposed and have demonstrated their effec-
tiveness in trajectory prediction tasks. These methods typically rely on fixed-length and sufficiently long historical trajectories as observations for accurately predicting future trajectories. However, In real-world scenarios, there is often insufficient time to gather an adequate number of observed trajectory points. For example, when car suddenly appears around corner, the trajectory prediction model needs to immediately make predictions by utilizing a small number of observed trajectory points to avoid collisions. This poses new and challenging problem for trajectory prediction, requiring models to make accurate predictions based on observed trajectories of arbitrary lengths. However, the number of observed trajectory points decreases, the performance of existing methods declines significantly, as shown in Figures ((a) and [ Therefore, it is essential to investigate models capable of handling observed trajectories of arbitrary lengths to accurately predict future trajectories.
In this paper, we propose new knowledge distillation framework, Length-agnoste Knowledge Distillation, called LaKD, for trajectory prediction with observations of arbitrary lengths. Firstly, note that longer trajectories often contain more temporal information, which can potentially lead to higher prediction accuracy compared to shorter trajectories. As shown in Figure [[c] the blue vehicle's straight trajectory history can boost confidence in predicting continued straight paths. However, as the number of observed trajectory points increases, additional interference might be introduced. As depicted in Figure [(d] despite the longer trajectory of the blue vehicle, it encompasses significant interference, leading to less accurate predictions compared to shorter trajectories. Inspired by this, we devise dynamic length-agnostic knowledge distillation strategy to adaptively transfer knowledge among trajectories of different lengths. As we know, Knowledge Distillation (KD) techniques 22 have been widely applied various domains, including computer vision I natural language processing [2]. etc. The basic idea of traditional KD algorithms is to optimize smaller student model by distilling knowledge from larger teacher model. In contrast these KD methods, our strategy emphasizes dynamic knowledge transfer among trajectories of varying lengths, rather than the conventional KD of transferring knowledge from the teacher model to the student model. Our method shares a unique encoder for all trajectories of varying lengths to learn the latent representations trajectories of varying lengths. aims to distill the knowledge 'good' trajectory features to 'bad' trajectories, with the assessment of 'good' or "bad' trajectories based on their prediction performance. This strategy facilitates adaptive knowledge exchange between long and short trajectories. It aids long trajectories in filtering out interfering information and assists

---

short trajectories in capturing richer temporal information, ultimately obtaining the optimal feature representation for predicting the agent's trajectory.
is worth noting that utilizing a single encoder as both the teacher and student models may affect the prediction performance of 'good' trajectory when distilling from 'good' trajectory to 'bad' trajectory, leading to knowledge collision during the distillation process. A straightforward solution to train a separate encoder for each trajectory length, but this approach significantly increases computational complexity. To address this issue, we devise a dynamic soft-masking strategy. Since different neuron units in a neural network model usually play different roles for different input data 371, the core idea of our strategy is to perform soft- masking on the neuron units during gradient updates. Specifically, when training on 'good' observation trajectory, the importance of the neuron units in the network is calculated based on the gradients. Subsequently, during agnthssgo knowledge distillation, gradients of crucial neuron units are multiplied by a lower update weight to mitigate significant updates. Conversely, less important units' gradients are multiplied by higher update weight to prioritize their updates. Through this approach, knowledge conflicts can be effectively resolved during the knowledge distillation process.
Our contributions can be summarized as follows: (I) distillation framework for trajectory prediction with observations of any length. LaKD is plug-and- play and compatible with existing models, enabling them to gracefully handle observed trajectories of arbitrary lengths. (2) We design new knowledge distillation strategy that dynamically transfers knowledge among trajectories of varying lengths. This approach helps long trajectories filter out interfering information and enables short trajectories to capture richer temporal details. Additionally, we devise a dynamic soft- soft-masking strategy to protect erucial neuron units from disruption and prevent knowledge collision during transfer. (3) We perform extensive experiments on three widely-used benchmark datasets, and demonstrate that LaKD significantly outperforms the baselines. Moreover, we show the compatibility of LaKD by integrating it with different trajectory prediction models.
We propose LaKD, a length-agnostic knowledge
2 Related Works
Traditional Trajectory Prediction. Traditional trajectory prediction methods aim to predict future trajectories of agents given sufficiently long observed trajectories. To date, many methods have been proposed, including coordinate system based methods [50,
17 interactive behavior modeling The representative works among
based methods (28 27 23), multimodal approaches 44 46
coordinate system based methods are pairwise-relative 60, 8
57]. which can simultaneous predict trajectories for multiple agents while reducing memory consumption and inference latency, Meanwhile, interaction behaviors play an important role in trajectory prediction. To model interactive
[59
behaviors within scenes, methods such as Graph Neural Networks (28] 27 22 3
33
42 and
attention mechanisms [39
36 32
52 23 are introduced. Given the substantial uncertainty
surrounding road agents, researchers
exploring diverse methods by integrating multimodal
information into predicted trajectories, such as GAN-based [29
44 46 58 16,
26
g [3), VAE-basce
47 49 21 flow-based (30 (56). and diffusion models [10
15 24
35
generate multimodal
trajectories. However, these methods generally perform well with fixed-length and sufficiently long
historical trajectories but experience significant performance drop when the length of observable historical trajectories varies.
Instantaneous Trajectory Prediction. Recently, significant advances have been made in instanta- neous pedestrian trajectory prediction tasks, using very short (i.c., two frames) historical trajectories. For example, MOE [45) introduces a unified feature extractor and a pre-training mechanism to capture effective information from momentary observations. DTO 38 employs a knowledge distilla- tion technique to transfer knowledge from long trajectories to short ones. BCDiff 24 develops a bidirectional diffusion model that simultaneously generates both unobserved historical and future trajectories. However, when confronted with input data containing varying numbers of frames, they necessitate training a model for each case, resulting in limited generalizability and high computa- tional complexity. In contrast to these works, we focus on studying how to perform length-agnostic knowledge distillation to adaptively transfer knowledge among long and short trajectories, so as to accurately predict future trajectories with observations of arbitrary lengths.
3

---

Random Masking
Length-agnosti Knowledge Distillation
Dynamic Soft-Masking Mechanism
Impor fanee Score Computation
Porameter Update
Maskk
90 ---
Decuder
Randon
-1
Observed trajecterics
Ground-trth future trajectories
Predicted future trajectories
Determine distilitioon direction
..* Only training
Training and testing
Knowledge distillation
Latent features
Element-wise multiplication
Ayr
Fat Total loss
Gradient
Impartance score
Gradient musking
During training, randomly mask historical irajectories M times to generate observed trajectories of varying lengths. Subsequently, we design a length-agnoste knowledge distillation module to dynamically transfer knowledge across trajectories of different lengths. Finally, we devise a dynamic soft-masking mechanism during gradient updates to effectively prevent knowledge conflicts. During inference, random masking, knowledge distillation, and dynamic soft-masking are not implemented.
Figure 2: Illustration of our LaKD framework.
Trajectory Prediction with Complex Observations. Currently, there are limited works focusing on complex observed trajectories for trajectory prediction. The recently proposed GC- VRNN framework [53] facilitates the concurrent execution of incomplete trajectory completion and prediction tasks in unified framework. However, this model does not take into account important traffic information, e.g., lane, making it unsuitable for vehicle trajectory prediction tasks. The FLN framework [54), which most closely related our work, propagates long historical trajectory information into medium and short trajectories to optimize the fitting of invariant features across multiple subnetworks. However, this strategy requires maintaining three models simultancously during training, sharply increasing computational complexity. In addition, it is plug-and-play but can only be integrated with Transformer based models. Moreover, this method assumes that longer observed trajectories always contain more useful information for trajectory prediction, and transfers knowledge from longer trajectories to shorter ones. Different from FLN, we observe that longer observed trajectories do not necessarily contain more valuable information than shorter ones for trajectory prediction, and thus explore length-agnostic knowledge distillation to dynamically transfer knowledge among long and short trajectories, enabling method gracefully handle observed trajectories of arbitrary lengths.
3 Method
3.1
Problem Formulation
We denote the observed state sequence of the target agent as Xobs {x1, .2., ØT), where T represents the observed time steps of the target agent, and it can be of arbitrary length greater than i I, E R² is the location of the agent at time step i. Additionally, we define the ground-trume -tuth future trajectories X (IT+1, 2", DT+F). and the predicted future possible K trajectories as X [ { 24 ftw.::+Fleeeeeeee [) where F denotes the length of the future trajectory. Our objective is to develop flexible trajectory prediction method capable of handling the case of observed traeectories of arbitrary lengths. Given that longer trajectories contain richer temporal information yet may also entail additional interference, their performance relative to short trajectories can vary. Thus, we attempt to explore a length-agnostic knowledge distillation framework for dynamically transferring knowledge among long and short trajectories, enabling long trajectories to filter out interference and allowing short trajectories to capture richer temporal details. By doing so, we aim to enhance the performance of trajectory prediction with observations of any lengths.
Previoos works 45 24 have shown that when the agent has only one frame of historical trajectory data. it cannot be predicted due to the lack of basic information such velocity and direction.

---

3.2 Overall Framework
The overall framework of the proposed LaKD shown in Figure 2 parts: a -angt-aggosttic knowledge distillation mechanism and dynamic soft- soft-masking strategy. First, to enhance the model's ability handle observed trajectories X of arbitrary lengths, we propose a length-agnostic knowledge distillation mechanism. This mechanism first evaluates the performance of the predicted trajectories X, and then determines the direction of knowledge distillation accordingly. Finally, it promotes adaptive knowledge exchange among trajectories of varying lengths, helping long trajectories filter out interfering information and short trajectories capture richer temporal information. However, since this strategy uses a single encoder as both the teacher and student models, risks causing knowledge conflicts during distillation. Therefore, we propose dynamic soft-masking strategy to address this issue. Specifically, When training on a 'good' observation trajectory, the importance of neuron units in the network can be determined by their gradients. During length- agnostic knowledge distillation, crucial neuron gradients are multiplied by lower update weight to mitigate significant updates, while less important gradients are multiplied by a higher update weight to prioritize their updates. This strategy can effectively resolve knowledge conflicts during distillation, such that our LaKD can effectively perform trajectory prediction based on observations of arbitrary lengths. In essence, LaKD is a plug-and- and-play approach that can be easily integrated with existing trajectory prediction models, enabling accurate predictions based on observed trajectories of varying lengths.
Our framework consists of two
3.3 Length-agnosti Knowledge Distillation
In this section, we introduce our proposed angth-agnosste knowledge distillation mechanism, which can facilitate information exchange among trajectories of different lengths, thereby enhancing the model's ability to handle observed trajectories of arbitrary lengths.
First, we obtain o of M different lengths by performing M random masks on the same observed trajectory, where the m-th trajectory is denoted as X As shown in Figure 2 these trajectories are fed into the backbone & generate the latent features V,, n and predicted trajectories X,:
Vm m
4 e (X obs, m
ΦE),
x = #p (Vmi mi Op),
where eg and d denote the encoder and decoder of o with parameterized by ág and dD. respec- tively. The backbone & can be any trajectory prediction model, e.g., HiVT [60] and QCNet [59] used in this paper, making method plug-and-play
As aforementioned, longer trajectories contain richer temporal information but may also involve additional interference for trajectory prediction, thus we design a length-agnostic knowledge dis- tillation strategy, where knowledge transfer can occur from longer trajectories to shorter ones, as well as from shorter to longer trajectories. To dynamically determine the direction of knowledge transfer, we employ the prediction performance based on different observed trajectories to find a "good' trajectory, and attempts to distill the knowledge embedded in its latent features Vm m to those of 'bad' trajectories. To measure the prediction performance, we calculate the minimum distance D. between the predicted trajectories X,, and the ground-truth trajectories X using the lz norm:
T-F
Dm min iE (1,2,......)
| x,5l² 1| J=T+1
Σ
During training, if the prediction performance of the current observation trajectory is worse than that of a previous 'good' observation trajectory, we begin to distill knowledge from the 'good' trajectory to the current trajectory. In this paper, use the latent features knowledge for transfer, and use the KL divergence 20 to minimize the following distillation loss to achieve the goal:
Cx KL(V ml V good),
where V good represents the latent features of the 'good' trajectory. By Eq. 3 the features Vm.. of "bad' trajectories are expected to optimized towards those of 'good' trajectories, i.e., V good- This facilitates effective knowledge transfer from 'good' to 'bad' trajectories. It is worth noting that different from traditional knowledge distillation optimizing smaller student model by distilling knowledge from larger teacher model, utilize unique encoder to encode all trajectories of
5

---

arbitrary lengths, and distill knowledge from 'good' trajectory to 'bad' one. This may lead to knowledge collision during the distillation process, degrading the feature representation capability of the "good' trajectory. To this end, we devise dynamic soft-masking strategy to address the issue of knowledge collision.
3.4 Dynamic Soft-Masking
As we know, different neuron units in neural network model typically play distinct roles for various input data [37]. Thus, we attempt to perform soft-masking on the neuron units during gradient updates. Specifically, when training on a 'good' observation trajectory, we determine the importance of the neuron units based on their gradients. During length-agnostic knowledge distillation, the gradients of crucial neuron units multiplied by lower update weight to prevent significant updates, while the gradients of less important units are multiplied by a higher update weight to prioritize their updates. By this strategy, knowledge conflicts can be effectively resolved during the distillation process.
During the training process, the gradient neuron unit is large, it indicates that changing it will have significant impact the result [37 [9]. Building this, we aim to identify which units are essential for the model to produce accurate predictions. To do so, first calculate the importance scores of the different units in the network follows:
Importance Score of Neuron Unit.
ØL X
I
(4)
0g.
where B denotes the batch size. X₂ and X represent the predicted trajectories and the ground- truth trajectories, respectively. 9,, is introduced as virtual parameter for calculating the importance I of units, where we fix 9u to in the training. I, is the importance score of the u-th neuron unit.
Since the gradients of the neuron units are usually very small, they cannot he directly applied to the calculation of soft masking weights. Therefore, it is necessary to confine the values of importance scores within the range [0,1]. To achieve this, we first normalize the importance scores of all units within each layer, ensuring a mean of and standard deviation of Then, we apply the Tanh activation function to these normalized scores as:
í, (tanh I. (- 1)/2,
where y represents the average importance of all units in the (-th layer, while denotes their variance.
Accumulation Importance Scores. Due to the fact that different units in the model play varying roles for trajectories of varying lengths, it is necessary preserve the model's ability much as possible during training. Therefore, during the m-th training iteration, we need to comprehensively consider the importance of units from the previous m-1 training iterations, and employ the element-
wise maximum (EMax) operation for calculating the cumulative importance f( p(Sm-1) of the model up to the (m-1)-tt iteration:
(Smm-1) = EMax( (f(m-1) _ p(Sm-2) 1)
where we set (0) uniformly to O.
Dynamic Soft-Masking of Units. During early training stages, when the model's predictive capability initially constrained, the informativeness of unit importance scores is limited. As training progresses, the reliability of these scores gradually improves. Therefore, we introduce dynamic decay coefficient& & to control the strength of the soft-masking. The specific formulas for the decay coefficient and the dynamic soft-masking mechanism based on the importance of units are follows:
V (1 i ((5m-1)
{) V.,
min( Creg * 77,1) Lreg < otherwise,
(8)
0

---

where y is hyperparameter. Creg represents the regression loss between the predicted trajectories and the ground-truth trajectories, as used in HiVT [60] and QCNet [59]. V. and V. represent the gradients of the units before and after soft-masking, respectively. The dynamic soft-masking mechanism effectively addresses the issue of knowledge conflict between trajectories of different lengths, promotes cross-length information exchange and thereby enhances the model's ability to predict trajectories based on observations of arbitrary lengths.
3.5 Optimization and Inference
Optimization. Following HiVT [60] and QCNet the regression loss Creg, which regresses the trajectory closest the ground truth. In addition, We also use the cross-entropy loss as the classification loss Lets to optimize the trajectory prediction model. Finally, the total loss function can be expressed as follows:
55 we also adopt the negative log-likelihood as
L Greg + aL BLu,,
where and I are the hyperparameters used to balance the contributions of different loss functions. We provide the pseudo-code of the training procedure in Appendix A.
Inference. After training, the model can be utilized for trajectory prediction based on observations of arbitrary lengths. For a new observed trajectory of any length, we directly input it into the encoder and decoder for future trajectory prediction, bypassing knowledge distillation and soft masking.
4 Experiments
4.1 Experimental Settings
Dataset. We evaluate the performance of our method on three widely used datasets: Argoverse 1 4.. nuScenes 3 and Argoverse 2 [3]. The Argoverse 1 dataset comprises 323,557 real driving scenes from Miami and Pittsburgh. The observation duration is seconds with a sampling frequency of 10Hz. Traditional trajectory prediction approaches typically assume that the first 2 seconds represent the historical observed trajectories, while the last 3 seconds are considered as the future ground- truth trajectories. The nuScenes dataset comprises 32,186 training scenarios, 8,560 validation scenarios, and 9,041 test scenarios. Fach scenario spans 8 seconds, sampled at Hz. Traditional trajectory prediction approaches typically assume that the first 2 seconds (5 locations) are used as the observed trajectory, while the last seconds are designated the future ground-truue -rruh trajectory. The Argoverse 2 dataset includes 250,000 scenes spanning across six cities. The observation duration is 1 seconds with a sampling frequency of 10Hz. Traditional trajectory prediction approaches typically assume that the first 5 seconds are used as historical observed trajectories, while the last 6 seconds serve as future ground- -truh trajectories. By masking trajectories on these datasets, we aim to evaluate the effectiveness of our trajectory prediction method with observations of arbitrary lengths.
Evaluation Metrics. To comprehensively evaluate the model, we employ a set evaluation metrics based on the minimum Average Displacement Error (minADE) minimum Final Displacement Error (minF) DE), and Miss Rate (MR) as:
1
MinADE
H minADE
T-i ")
(10)
2=
1
minFDE
(minFDD
(11)
H im
1 Σ (MR T=
MR
(12)
where H denotes the maximum number of observation points, and represents the number of trajectories to be predicted. T = represents the number of observation points. We evaluate the performance for each observation length and then average the results across all lengths to obtain the final outcome.
7

---

Table 1: Comparisons of different methods Argoverse 1 and Argoverse 2, evaluated using minADE, minFDE and MR metries. The best results are highlighted in bold.
Kml
K-6
Dataset
Methods
minADE
minFDE
MR.
minADE
minFDD
MR
HivT-Origg
1,4733
3,1834
0,5267
0,7255
1.0740
0.1124
HiVT-RM
3,0599
0.7070
0.1053
Argoverse
HIVT-DTO
1,3999
3.0262
0.5056
0,7032
1,0350
0.1039
HITT-FLN
1.4011
3,0288
0,5051
0,7026
1,0325
0.1033
HiVT-LakD
2,8799
0.6807
0.0928
QCNct-Orig
1,1656
2.4021
0,3860
0,5791
0.7399
0.0734
QCNet-RM
1,0995
2,2550
0.3630
0,5684
0.7115
0,0703
Argoverse
QCNei-DTO
2,2303
0.3563
0,5418
0.0671
QCNet-FLN
1.6311
2.2083
0.3579
0,5411
0.6680
0.0671
OCNet-LaKD
0.9982
2.0718
0.3439
0,5240
0,6581
0.0640
HiVT-Orig
3,5973
8.3062
0,8518
1.5289
2.8261
U.4377
HiVT-RM
3.6580
8,4889
0.8647
1,5245
2. 8068
0.4716
nuScenes
HIVT-DTO
3,5860
8.2556
0.8514
1,5105
2.7379
0,4350
HiVT-FLN
8.1928
1,5094
HiVT-LakD
3.4296
7,8882
0.8369
1.4793
2.6934
0.4329
QCNet-Orig
4,3134
9,7857
0.8588
1,4719
2.5831
0,4600
QCNet-RM
4.1723
9.4672
0.8622
1.5255
0.4611
nuSeenes
QCNei-DIT
4.1447
9,4552
0.8580
1.4653
2.5798
0.4317
QCNet-FLN
4,1169
9,3639
0.8562
1.4676
2.5448
0,4344
QCNet-LaKD
4.0663
9.2524
0.8523
1.4594
0.4023
HiVT-Orig
2.5502
6,5586
0.7455
1.0561
2. 2.9093
0.3275
HiVT-RM
2.2848
6.0548
0.7249
0.9457
1 1.9283
0.2994
Argoverse
HiVT-DTO
2.2769
6.0548
0.7275
0.9324
0.2903
HiVT-FLNN 2.2786
6.0464
0.7240
0.9287
1. 1,8388
0.2891
HiVT-LakD
2.2066
5.8769
0.7161
0.9183
1 188666
0,2791
QCNet-Orig
2.1006
5.2219
0.6299
0.8339
1,3849
0.1884
QCNei-RM
1.7452
4,4404
0.5957
0.7508
1.3184
0.1671
Argoverse 2
QCNet-DTO
1,7713
4,4900
0.5979
0.7454
1,2924
0,1671
QCNet-FLN
1,6940
4.2373
0.5808
0.7370
1,2595
0.159
QCNet-LaKD
1.6574
4.1505
0.5753
0.7258
1.2420
0.1555
Backbone and Baselines. To demonstrate the compatible ability of our LaKD, we combine with two representative trajectory prediction methods: HiVT (60) and QCNet [59]] To verify the effectiveness of our method, we compare LaKD with Flexil ength Network (FLN)
54,, the work most related to ours. FL.N integrates trajectory data with diverse observation lengths and attempts to learn temporally invariant representations for future trajectory predictions. We also compare LaKD with (DTO) 388 DTO is initially developed for instantaneous trajectory prediction. To ensure fairness, we modify its framework to distill from complete trajectories into arbitrary length trajectories. Moreover, we take Orig and RM our baselines. Orig denotes using the original fixed-length observed trajectories as inputs for training the backbones, while RM involves randomly masking the original observed trajectories to generate trajectories of varying lengths inputs for training the backbones.
Implementation Details. During training, we set M in our LaKD to 3, and both a and 3 to 1. For Argoverse 1, nuScenes, and Argoverse 2, we set 7 to -1, -0.65, and -1.35, respectively. The dimensionality of the encoded latent feature V is set to 128. We utilize the AdamW optimizer [34]. setting the learning rate and weight decay parameters to 5e-4 and 1c-4, respectively. The batch size is set to 32. The experiments are implemented using PyTorc [40] on the NVIDIA GeForce RTX 4090.
4.2 Results and Analysis
Performance on Trajectory Prediction with Observations of Arbitrary Lengths. We evaluate the overall performance of our method, as listed in Table Based on Table 1, our method outper-

---

forms all others, particularly surpassing FLN, across all the three datasets. This demonstrates the effectiveness of our method in predicting future trajectories from observations of varying lengths. Moreover, our LaKD outperforms Orig, indicating the necessity of developing trajectory prediction method specifically designed to handle observations varying lengths. Finally, our LaKD achieves the best performance with various backbones, demonstrating the compatibility of our method. More detailed results are presented in Appendix A.2
Allation Study. We conduct ablation studies to validate the effectiveness of our proposed compo- nents using HiVT as the backbone on the Argoverse dataset. Since the Random Masking strategy was first presented in this paper, also conduct ablation study verify its effectiveness. The results are shown in Table 2 The experiments demonstrate that as we progressively remove the dynamic soft-masking mechanism (DSM), the Length-agnostic Knowledge Distillation (LaKD), and the Random Masking (PM), the performance of our method gradually declines, demonstrating the effectiveness of our proposed components. By combining these components, our method achieves the best performance.
Table 2: Ablation study of our method on the Argoverse dataset.
K=
Ka6
LaKD
DSM
minADE
minFDE
MR
minADE
minFDE
MH
1,4733
3.1834
0,5267
0,7255
1.0740
0.1124
1
3.0599
0.5104
0.7070
1.0447
0.1053
1.3619
2,9511
0.5051
0.6851
0.9965
0.0948
1.3317
Analysis of Different Mask Numbers M. We investigate the impact of different mask numbers M in our LaKD on the trajectory prediction performance. We use HiVT [60] as the backbone, and list the results in Table 3 Since our method involves randomly masking historical trajectories M times in each training iteration and continues for sufficient number of epochs, observation trajectories of all different lengths are seen during training, regardless of the value of M. Consequently, the model's performance does not significantly degrade as M changes, indicating that the model is not sensitive M. This makes M easy
real-world scenarios. For our experiments, we M 3.
Table 3: Analysis of our method with different M on the Argoverse I dataset.
K=6
M
minADE
minFDE
MR
minADE
minFDE
1.3457
2.9116
0.4943
1.3317
2,8799
0.4901
0.6807
0,9864
0.0928
1.3414
2,9017
0.4938
0.6814
0.9867
0.0934
1.3563
2,9359
0,5010
0.6851
0.9973
0.0961
6
L3386
2,9198
0.4977
0.6878
1.0025
0.0943
Qualitative Analysis. To intuitively demonstrate the effectiveness of LaKD, perform
qualitative experiment on the Argoverse 2 dataset, as shown in Figure 3 The first row features a scenario T-junction where the agent is about turn, with an observed trajectory spanning points. The second row illustrates scenario at a fork in the road, where the agent is preparing to change lanes, with an observed trajectory of 10 points. is observable that across different scenarios, our method exhibits higher accuracy compared to other models.
5 Conclusion
In this paper, propose with observations of any length. This framework enables long trajectories to filter out interfering information and short trajectories to capture richer temporal details. To address knowledge conflicts during distillation, we devise dynamic soft-masking mechanism to protect crucial neuron units from disruption, thereby enhancing prediction performance. Extensive experiments on the Argov- erse I, nuScenes, and Argoverse 2 datasets demonstrate the effectiveness of our approach and its compatibility with various trajectory prediction models.
knowledge distillation framework for trajectory prediction

---

(a) QCNet-Orig
(b) QCNet-FLN
(c) QCNet-LaKD
Figure 3:
Qualitative results the Argoverse 2 dataset using (a) QCNet-Orig, (b) QCNet-FLN, and (c) QCNet-LaKD. The observed trajectories, ground-truth trajectories and predicted trajectories shown in green, red and blue, respectively. Our predicted future trajectories are closer to the boound-truth, compared to other methods.
Acknowledgments and Disclosure of Funding
This work was supported by the NSFC under Grants 62122013, U2001211.
References
[1] Inhwan Bae, Jin-Hwi Park, and Hae- Jeon. Learning pedestrian group representations for multi-modal trajectory prediction. In European Conference on Computer Vision, pages 270-289, Springer, 2022.
[2] Cristian BUCILA. Rich CARUANA, and Alexandru NICULESCU-MIZIL Model compression. In KDD-2006 (proceedings of the Twelfth ACM SIGKDD international conference on knowledge discovery and data mining, August 20-23, 2006. Philadelphia. PA, USA), pages 535-544 335-541, 2006.
[3] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11633 2020.
[4] Hing-Fang Chang, John Lambert, Patsorn Sangkloy, Jagject Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon I Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 8748-8757, 2019. 2019.
[5] Guangyi Chen, Junlong Li, Jiwen Lu, and Jie Zhou. Human trajectory prediction via counterfac- tual analysis. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9824-9833, 2021.
[6] Hao Chen, Jiaze Wang, Kun Shao, Furui Liu, Jianye Hao, Chenyong Guan, Guangyong Chen, and Pheng-Ann Heng. Traj-mae: Masked autoencoders for trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8351-8362. 2023.
[7] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Bevdistill: Cross-modal bev distillation for multi-view 3d object detection. arXiv preprint arXiv:221..0933 2022.
10

---

[8] Alexander Cui, Sergio Casas, Kelvin Wong, Simon Suo, and Raquel Urtasun. Gorela: Go relative for siewpoint-invarian motion forecasting. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7801-7807. IEEE, 2023.
[9] Patrick Dendorfer, Sven Elffein, and Laura Leal-Taixé. Mg-gan: A multi-generrtor model preventing out-of-distribution samples in pedestrian trajectory prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13158-13167, 2021.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 14:8780-8794 2021.
[11] Kaituo Feng, Changsheng Li, Dongchun Ren, Ye Yuan, and Guoren Wang. On the road to portability: Compressing end-to-end motion planner for autonomous driving. arXiv preprint arXiv:2403.01233, 2024.
[12] Kaituo Feng. Changsheng Li, Ye Yuan, and Guoren Wang. Freekd: Free-direction knowledge distillation for graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 357-366, 2022.
[13] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2255-2264, 2018.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint urXiv:1503.0000 150.15253333/1 2015.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
[16] Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. Stgat: Modeling spatial-temporal interactions for human trajectory prediction. In Proceedings of the IEEE/CVF international conference computer vision, pages
2019.
[17] Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heteroge- neous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE transactions on pattern analysis and machine intelligence, 2023.
[18] Christos Katrakazas, Mohammed Quddus, Wen-Hua Chen, and Lipika Deka. Real-time mo- tion planning methods for autonomous on-road driving: State-of of-thc-art and future research directions. Transportation Research Part C: Emerging Technologies, 60:416-444 442, 2015.
[19] Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual learning of language models. In International Conference on Learning Representations (ICLR), 2023.
[20] Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
[21] Mihee Lee, Samuel S Sohn, Seonghycon Moon, Sejong Yoon, Mubbasir Kapadia, and Vladimir Pavlovic. Muse-vae: Multi-scale vae for environment-aware long term trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2221-2230 2022.
[22] Jiachen Li, Fan Yang. Masayoshi Tomizuka, and Chiho Choi. Exolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. Advances in neural information processing systems, 13:19783-19794, 2020.
[23] Rongging Li, Changsheng Li, Yuhang Li, Hanjie Li, Yi Chen, Ye Yuan, and Guoren Wang. ltpnet: Towards instantancous trajectory prediction for autonomous driving. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages (643-1654, 2024.
[24] Rongging Li, Changsheng Li, Dongchun Ren, Guangyi Chen, Ye Yuan, and Guoren Wang. Bediff: Bidirectional consistent diffusion for instantaneous trajectory prediction, Advances in Neural Information Processing Systems, 36, 2024.

---

[25] Yuhang Li, Changsheng Li, Baoyu Fan, Rongqing Li, Ziyue Zhang, Dongchun Ren, Ye Yuan, and Guoren Wang. Fdnet: Feature decoupling framework for trajectory prediction. In 2024 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS). IEFF, 2024.
[26] Yuke Li.
Which way are you going? imitative decision learning for path forecasting in dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 294-303, 2019.
[27] Junwei Liang, Lu Jiang, and Alexander Hauptmann. Simaug: Learning robust representations from simulation for trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pages 275-292. Springer, 2020.
[28] Junwei Liang, Lu Jiang, Kevin Murphy, Ting Yu, and Alexander Hauptmann. The garden of forking paths: Towards multi-future trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10508-10518, 2020.
[29] Rongqin Liang, Yuanman Li, Xia Li, Yi Tang, Jiantao Zhou, and Wenbin Zou. Temporal pyramid network for pedestrian trajectory prediction with multi-supervision. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 2029-2037, 2021.
[30] Rongqin Liang, Yuanman Li, Jiantao Zhou, and Xia Li. Stglow: a flow-based generative framework with gral-graphormer for pedestrian trajectory prediction. IEEE transactions on neural networks and learning systems, 2023.
[31] Chang Liu, Chongyang Tao, Jiazhan Feng, and Dongyan Zhao. Multi-granulan structural knowledge distillation for language model compression. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1001-1011, 2022.
[32] Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction with stacked transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7577-7586, 2021.
[33] Yuejiang Liu, Qi Yan, and Alexandre Alahi. Social nce: Contrastive learning of socially-aware motion representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15118-11 155119,, 2021.
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 10000 2017.
[35] Weibo Mao, Chenxin Xu, Qi Zhu, Siheng Chen, and Yanfeng Wang. Leapfrog diffusion model for stochastic trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5517-5526, 2023.
[36] Jean Mercat, Thomas Gilles, Nicole El Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Pita Gil. Multi-head attention for multi-modal joint vehicle motion forecasting, In 2020 JEEE International Conference on Robotics and Automation (ICRA), pages (638-9644. IEEE, 2020.
[37] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
[38] Alessio Monti, Angelo Porrello, Simone Calderara, Pasquale Coscia, Lamberto Ballan, and Rita Cucchiara. How many observations are enough? knowledge distillation for trajectory forecasting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 6553-6562. 2022.
[39] Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Ben- jamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In 2023 IEFE International Conference on Robotics and Automation (ICRA), pages 2980-2987. IEEE, 2023.
12

---

[40] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga. et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[41] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasbbee trajectory forecasting with heterogeneous data. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 683-700. Springer, 2020.
[42] Liushuai Shi, Le Wang, Chengjiang Long, Sanping Zhou, Mo Zhou, Zhenxing Niu, and Gang Hua. Sgen: Sparse graph convolution network for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8994-9003, 2021.
[43] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global inten- tion localization and local movement refinement. Advances in Neural Information Processing Systems, 35:6531 6543, 2022.
[44] Hao Sun. Zhiqun Zhao, and Zhihai He. Reciprocal learning networks for human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7416-7425, 2020.
[45] Jianhua Sun, Yuxuan Li, Liang Chai, Hao-Shu Fang, Yong-Lu Li, and Cewu Lu. Human trajectory prediction with momentary observation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6467-6476, 2022.
[46] Jianhua Sun, Yuxuan Li, Liang Chai, and Cewu Lu. Stimulus verification is universal and effective sampler in multi-modal human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22014-22023, 2023.
[47] Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu. Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13250-13259, 2021.
[48] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cormman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In 2022 International Conference on Robotics and Automation (ICRA), pages 7814-7821. IEEE, 2022.
[49] Chuhua Wang, Yuchen Wang, Mingze Xu, and David J Crandall. Stepwise goal-driven networks for trajectory prediction. IEÉE Robotics and Automation Letters, 7(2):2716-2723, 2022.
[50] Xishun Wang, Tong Su, Fang Da, and Xiaodong Yang. Prophnet: Efficient agent-centric motion forecasting with anchor informedd proposals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11995-22009 2023.
[51] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagject Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In Thirty-fifth fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.
[52] Conghao Wong, Beihao Xia, Ziming Hong, Qinmu Peng, Wei Yuan, Qiong Cao, Yibo Yang, and Xinge You. View vertically: A hierarchical network for trajectory prediction via fourier spectrums. In European Conference on Computer Vision, pages 682-700. Springer, 2022.
[53] Yi Xu, Armin Bazarjani, Hyung-gun Chi, Chiho Choi, and Yun Fu. Uncovering the missing pattern: Unified framework towards trajectory imputation and prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9632-9643, 2023.
[54] Yi Xu and Yun Fu. Adapting to length shift: Flexilength network for trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
13

---

[55] Ekim Yurtsever, Jacob Lambert, Alexander Carballo. and Kazuya Takeda. A survey of au- tonomous driving: Common practices and emerging technologies. IEEE access, 8:58443-58469, 2020.
[56] Bo Zhang, Tao Wang, Changdong Zhou, Nicola Conci, and Hongbo Liu. Human trajectory fore- casting using flow-based generative model. Engineering Applications of Artificial Intelligence, 115:105236, 2022.
[57] Zhejun Zhang, Alexander Liniger, Christos Sakaridis, Fisher Yu, and Luc V Gool. Real-time motion prediction via heterogeneous polyline transformer with relative pose encoding. Advances in Neural Information Processing Systems, 36, 2024.
[58] Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 12126-1215 2019.
[59] Zikang Zhou, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Query-centric trajectory prediction. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 17863-17873, 2023.
[60] Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8823 8833, 2022.
14

---

A Appendix
A.1 Training Procedure of LaKD
We present the training procedure for LaKD in Algorithm
Algorithm 1: Training Procedure of LaKD
while Model converges do
Sample trajectory (X .x*) from dataset; for m = 1 to M do
Random mask to get X
Obtain predicted trajectories X,, by Equation (1);
Compare the prediction performance of the current observation trajectory with the
previous 'good' observation trajectory by Equation (2), and then determine the direction of knowledge distillation;
Carry out knowledge distillation according to Equation (3);
Calculate the total loss function C by Equation (9);
Calculate importance scores of units 1 by Equations (4) and (5);
Calculate cumulative importance 1 1(5mm1) by Equation (6);
Constrain the gradient of units according to Equations (7) and (8);
Update parameters using the Adam W optimizer.
end end
A.2 Additional Experimental Results
In this section, we demonstrate our model's ability process observed trajectories of arbitrary lengths using three metries: minADE, minFDE, and MR. From the figures below, our method significantly outperforms other baselines in handling trajectory points of any length.
2.4
HIVT-rig
HIVT-Orig
HIVT-Orig
HiVT-RM
HiVT-RM
0.65
HİVT-RM
HIVT-DTO
HIVT-DTO
DIO
HIVT-FLNN
HIVT-FLN
HIVT-FLN
minADE,
2.0
HIVT-LaKD
minFDE)
HIVT-LaKD
HIVT-LaKD
MR-7
1.3
0.55
144
200
HIVT-Orig
15
HIVT-Orig HIVT-RM
HiVT-Orig
HIVT-RM
0.20
HIVT-RM
0.55
HIVT-DTO
HIVT DTO
DTO
HIVT-FLN
0.18
HIVT-FLN
0.00
HIVT-LaKD
minFDEs
HiVT-LaKD
HiVT-LaRD
0.02
1.3
0.80
1.2
11-
0.12
0.70
0.00
10
18
20
2
8
12
15
18
14
36
18
20
Figure 4: Comparison of Results Using HiVT the backbone on the Argoverse I dataset.
15

---

QCNet-Orig QCNet-RM
QCNet-Ort QCNet-RM
QCNet-
QCNet-RM
QCNet DTO
3.25
QCNet DTO
0.46
OCNet DTO
14-
OCNet-FLN QCNet-LakD
QCNet-FLN
0.42
OCNet-FLN
minADE
minFDE,
QCNet-Lat LD
OCNet-LaKD
2.75
0.40
0.38
12-
0.36
2.25
0.34
0.32
0.75
OCNet-Orig QCNet-RM
QCNet-Orig QCNet-RM
QCNet-Orig
QCNet-RM
0.70
OCNet-DTO
0.30
QCNet-DTO
0.10
OCNet-DTO
OCNet-FLN QCNet-LaKD
QCNet-FLN QCNet-LakD
OCNet-FLN
minFDEs
0.09
QCNDt-LaKD
0.90
MR.
0.50
0.75
0.55
0.85
0.06
10
12
18
20
10
16
18
14
16
Figure 5: Comparison of Results Using QCNet as the backbone on the Argoverse 1 dataset.
3.70
HIVT-Orig
8.5
HiVT-Orig
0.865
HIVT-Orig
HIVT-RM
8.5
HIVT-RM
HIVT-RM
HIVT-DTO
84-
HIVT-DTO
HIVT-DTO
HIVT-FLN
HIVT-FLN
HIVT-FLN
0.855
HiVT-LaKD
HIVT-LaKD
HIVT-LaKD
3.55
MRT
0.050
3.50
0.85%
1.45
7.9
3.10
7.8
0.835
HIVT-Orig
2.850
HIVT-Orig HIVT-RM
HIVT-Ori
HiVT-RM
HIVT-RM
HiVT-DTO
HiVT-DTO
HiVT-DTO
HIVT-FLN
2,500
HIVT-FLN
HIVT-FLN
1.52
HiVT-LaKD
HiVT-LaKD
HiVT-LaKD
MRG
1.50
2.250
0.45
2,700
Figure 6: Comparison of Results Using HiVT as the backbone on the nuScenes dataset.
16

---

4.40
QCNet-Orig QCNet-RM
QCNet-Orig
0.866
QCNet-Orig
9.9
QCNet-RM
0.884
OCNetR
QCNet DTO
9.5
QCNet DTO
OCNet-DTO
4.30
OCNet-FLN
QCNet-FLN
OCNet FLN
QCNet LakD
MinFDE
9.7
QCNet- LaKD
0.060
QCNet-LaKD
4.20
0.855
4.35
0.051
4.05
0.852
1.90
QCNet-Orig
2.70
QCNet-Orig
QCNet-Orig
QCNet-RM
QCNet-RM
QCNet-RM
1.38
OCNet-DTO
QCNet-DTO
0.46
OCNet-DTO
2.50
1.56
OCNet-FLN
QCNet-FLN
OCNet-FLN
1.54
QCNet-LaKD
minFDEs
QCNet-LaKD
LCNet-LaKD
2.50
1.52
0.41
0.41
0.48
Figure 7: Comparison of Results Using QCNet as the backbone the nuScenes dataset.
HiVT-Orig
9.5-
HIVT-Orig
HIVT-Orig
HIVT-RM
9.9
HIVT-RM
0.84
HIVT-MM
HIVT-DTO
HIVT-DTO
HIVT-DTO
HIVT-FLN
8.5
HIVT-FLN
0.82
HIVT-FLN
HIVT-LaKD
minFDE,
8.0
HIVT-LaKD
HIVT-LaKD
3.0
0.76
0.74
2.5
2.0
HIVT-Orig
HIVT-Orig HIVT-RM
HIVT-Oni
HiVT-RM
HIVT-RM
HiVT-DTO
HiVT-DTO
HiVT-DTO
HIVT-FLN HiVT-LaKD
HIVT-FLN
HIVT-FLN
minADEs
HIVT-LaKD
HIVT-LaKD
minEDE
MR.
12
1.0
18
76
Figure 8: Comparison of Results Using HiVT the backbone on the Argoverse 2 dataset.
17

---

QCNet-Orig
QCNct-Orig
0.6
OCNet-Orig
QCNet-RM
QCNet-RM
OCNet-RM
QCNet-DTO
QCNet-DTD
OCNet-DTO
QCNet-FLN
QCNet-FLN
QCNet-FLN
QCNet LakD
MinFDE;
QCNet LaKD
OCNet LaKD
MR-
0.70
QCNet-Orig QCNet-RM
QCNet-Orig QCNet-RM
QCNet-Orig Orig
2.25
2.75
0.50
QCNet-RM
QCNet DTO
2.50
QCNet
DTO
0.45
QCNel: DTO
2.00
OCNet-FLN
QCNet-FLN
OCNet-FLN
0.40
1.75
QCNet-Lake
MinFDEs
2.2
QCNet-Late
LaKD
1.50
1.75
0.25
130
1.50
0.20
0.15
18
10
18
22
10
18
Figure 9: Comparison of Results Using QCNet the backbone on the Argoverse 2 dataset.
A.3 Limitations
In this work, we aim distill knowledge from 'good' trajectory 'bad' trajectory for improving the prediction performance from observations of any lengths. However, how to determine 'good' or 'bad' trajectory an open problem. Currently, we adopt heuristic strategy by utilizing the distance between the predicted trajectory and the ground-trutt trajectory. More complex strategies, such as reinforcement learning, are worth further exploration and investigation.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions of the paper, including the development and evaluation of our proposed model.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We conduct detailed analysis of the limitations of the framework proposed in this paper in the Conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The paper provides comprchensive information necessary to reproduce the main experimental results, ensuring transparency and replicability of the findings.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release the code as open-source as soon as the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
public/
ss ssiossssioypoooyyooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
/ httpscccccccccccccccccccccc i ies CCodeeoodSSaaee bm ss i onnool icy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the training and test details in the Section and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the high cost of training, we did not perform multiple runs compute eтror bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information on the computer resources in the Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms fully with the NeurIPS Code of Ethics. We have reviewed the guidelines thoroughly and ensured that all aspects of our work adhere to the ethical standards set forth by NeurlPS.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses the potential positive societal impacts of the work per- formed, highlighting the benefits and advancements it can bring to the field and society at large.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper poses no such risks, because trajectory prediction models do not have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly credited and cited the creators or original owners of assets, including code and papers, used our work. We have explicitly mentioned the licenses and terms of use for these assets and have respected them accordingly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papelpappssss wwwthhocee com/ / /ataaaaasssssee has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "tMoL1Rolq1", "reviewer": "Reviewer_Rudm", "summary": "The paper presents a length-agnostic knowledge distillation framework, which is motivated from knowledge transfer among trajectories of different lengths. The authors address knowledge conflicts during distillation from a dynamic soft-masking mechanism. The evaluation is conducted using Argoverse 1, nuScenes, and Argoverse 2 datasets.", "questions": "- The qualitative results do not clearly demonstrate the scenarios where the authors have been motivated. As shown in Figure 3, there is no difference between (b) and (c), which makes me doubt about the contribution of this work. Not just closer to the ground-truth, it is strongly suggested where the proposed method works while others fail. \n- Different backbone models other than HiVT and QCNet would be needed to claim its generality.", "limitations": "The paper does not provide any failure cases or insights into the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem is clearly stated and the motivation is logical. \n- Dynamic knowledge transfer between trajectories of varying lengths sounds interesting.\n- It seems that the soft-masking strategy reasonably addresses the issue of knowledge collision as proposed.", "weaknesses": "- Lacks a proper qualitative results. \n- Limited choice of backbone models.\nSee below for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer of the constructive feedback. In light of these insightful comments, we would like to address them by providing the following clarifications.\n\n> **Question 1**: The qualitative results do not clearly demonstrate the scenarios where the authors have been motivated. As shown in Figure 3, there is no difference between (b) and (c), which makes me doubt about the contribution of this work. Not just closer to the ground-truth, it is strongly suggested where the proposed method works while others fail.\n\nThanks for raising this concern which helps us to clarify the contribution of our work. Due to the limited time during the submission phase, we simply visualized the results to demonstrate that our method can more accurately predict future trajectories. Although it seems that there is no significant difference among the three predictions in the above scenario of Fig. 3, their results are 3.5891/6.0794/1, 1.4298/3.0295/1 and 0.6915/1.0974/0 in terms of minADE/minFDE/MR, respectively, indicating that our LaKD provides more accurate predictions. Following your suggestion, we have added more results to demonstrate that our method works while others fail, as shown in the Figure 1 of the attached pdf in the \"global\" response. Thanks again for the insightful comments, and we will add these results in the final version. \n\n\n\n\n\n> **Question 2**: Different backbone models other than HiVT and QCNet would be needed to claim its generality.\n\nThanks for the comment. We chose to use HiVT and QCNet as our backbones in our paper because they are the most advanced models in the field of trajectory prediction and have recently ranked highly on the Argoverse dataset leaderboard. Based on your helpful suggestion, we conducted experiments using two other typical trajectory prediction models, TNT [1] and Vectornet [2], as our backbones to show the generality of our method. The results are as follows:\n\n|||||||||\n|-|-|-|-|-|-|-|-|\n|Dataset|Methods||K=1|||K=6||\n|||$\\mathrm{min\\overline{ADE}}$|$\\mathrm{min\\overline{FDE}}$|$\\mathrm{\\overline{MR}}$|$\\mathrm{min\\overline{ADE}}$|$\\mathrm{min\\overline{FDE}}$|$\\mathrm{\\overline{MR}}$|\n|Argoverse 1|VectorNet-Orig|3.5335|7.8259|0.8267|2.1173|3.6945|0.6751|\n||VectorNet-RM|1.7815|3.8589|0.6412|1.1016|1.9880|0.3531|\n||VectorNet-DTO|1.7523|3.8065|0.6409|1.0126|1.8096|0.3189|\n||VectorNet-FLN|1.7334|3.7027|0.6244|1.0088|1.7994|0.3132|\n||VectorNet-LaKD|1.6003|3.4628|0.5917|0.9933|1.7546|0.3016|\n|Argoverse 1|TNT-Orig|3.7318|7.7174|0.8600|1.8255|2.9818|0.4081|\n||TNT-RM|2.8629|6.3561|0.7925|1.1946|2.1685|0.3116|\n||TNT-DTO|2.7280|6.1935|0.7858|1.1242|2.0630|0.3084|\n||TNT-FLN|2.4241|5.5038|0.7593|1.0692|1.9485|0.2714|\n||TNT-LaKD|2.2174|5.0279|0.6974|1.0172|1.8571|0.2456|\n\n\n\n\nThe experimental results show that our method still achieves the best performance when using TNT and Vectornet as our backbones. We will add these results in the final version.\n\n\n\n\n\n> **Limitations**: The paper does not provide any failure cases or insights into the limitations.\n\nThanks for your comment. Due to the limitations of the paper's length, we have analyzed our algorithm's limitations in the appendix as : \n\nIn this work, we aim to distill knowledge from 'good' trajectory to 'bad' trajectory for improving the prediction performance from observations of any lengths. However, how to determine a 'good' or 'bad' trajectory is an open problem. Currently, we adopt a heuristic strategy by utilizing the distance between the predicted trajectory and the ground-truth trajectory. More complex strategies, such as reinforcement learning, are worth further exploration and investigation. \n\n\n\n[1] Zhao H, et al. \"Tnt: Target-driven trajectory prediction\" PMLR 2021.\n\n[2] Gao J, et al. \"Vectornet: Encoding hd maps and agent dynamics from vectorized representation\" CVPR 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "63Z5S6potG", "reviewer": "Reviewer_4u3J", "summary": "To tackle the length-agnostic trajectory prediction problem, the authors are motivated to utilize knowledge learned from both longer and shorter trajectories. They propose a plug-and-play self-distillation framework for trajectory prediction, which can be integrated with many different off-the-shelf trajectory prediction models. Specifically, for a given long trajectory, the authors first randomly mask the trajectory into several different lengths, maintaining the same prediction horizon but with different history lengths. They then determine the direction of knowledge distillation based on the performance of the model with longer or shorter history. The better-performing trajectory length is used to distill knowledge into the intermediate features of represented from other lengths. Additionally, the authors introduce a soft masking method at neuron level, updating the distillation of important neurons slowly and vice versa. They test their framework on Argoverse1, nuScenes, and Argoverse2 datasets, using off-the-shelf models HiVT and QCNet, and demonstrate performance gains.", "questions": "Q1. Could the authors explain further what do they mean by “Knowledge Collision”? \n\nQ2. May I ask how could a trivial degenerate solution, or a mode collapse, be avoided when conducting bi-directional self-distillation? I am aware that the authors try to classify the neurons in Section 3.4, but it seems to me that by the metrics as in Equation 4, the important neurons will get even more important length-agnostically, and therefore get distilled more both for long and for short trajectories. \n\nQ3. A question about the overall framework is that when we distill the representation of a long history into a short history, the short history lacks information about the longer past. For example, when trajectory A and trajectory B have the same last several time steps but differ in that one turns left earlier while the other turns right, the information intrinsic to these trajectories is different. Let’s refer to the last time stamps of both A and B as a shorter C, as A and B share the same last steps. The longer horizon trajectory encoder of A can capture this information, but when it is distilled to the encoder with a shorter history C, this distinguished information of A is not seen. But the proposed framework enforces distilling this uncertain knowledge of A to the short history C, even if in a future potential test situation, the longer trajectory turns in the opposite direction in the past, like B, than seen in the training set. May I ask how is the proposed framework valid, considering a similar situation?\n\nQ4. For Ablation Table 3, may I ask why the performance degrades when M is larger than 3? Intuitively, the performance should improve with a larger mask number?", "limitations": "Briefly discussed in Conclusion section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper is generally well-written and easy to follow.\n\nS2. The proposed plug-and-play method can work well with many different off-the-shelf model architectures, making it flexible. And the proposed method shows performance gains with different several recent trajectory prediction models.", "weaknesses": "W1. One constraint for most knowledge distillation scenarios is that for the input to the model being distilled and the model distilling, at least the information should be equivalent or nearly equivalent between the inputs to both models. I'm a bit confused about why the overall framework of bidirectional self-distillation works from an information theory / flow perspective. For my detailed confusion, please refer to my question section.\n\nW2. For the qualitative analysis in Fig. 3, I could hardly understand why this shows that the proposed LRKD method is better than the other methods. In the above scenario, there doesn't seem to be a significant difference between the three predictions. In the below scenario, the proposed method is closer to the ground truth, but I do not fully understand why the ground truth behaves in that manner and how this is inferred by the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. The following are our responses to the points you have raised.\n\n> **Comment 1**: I'm a bit confused about why the overall framework of bidirectional self-distillation works from an information theory / flow perspective.\n\nSorry for any confusion caused by our unclear presentation. Traditional knowledge distillation involves transferring knowledge from a deeper teacher model to a shallower student model given the same input, which is a form of model capability transfer. However, the bidirectional self-distillation in our LaKD first randomly masks the same trajectory to obtain trajectories of different lengths. Then, knowledge is transferred from the features of good trajectories to those of bad trajectories extracted by the same model, which is a form of data-level information transfer. We expect that this approach will enable trajectories of varying lengths to exhibit strong features for future trajectory prediction.\n\n\n\n> **Question 1**: Could the authors explain further what do they mean by “Knowledge Collision”?\n\nAs mentioned above, since our framework uses a single shared encoder to extract feature representations for trajectories of different lengths and then performs data-level information transfer through this encoder, the feature representation of good trajectories can be disrupted when knowledge is transferred to bad trajectories through updates to the shared encoder. We refer to this problem as \"Knowledge Collision\". We will add more details in the final version.\n\n\n\n> **Comment 2**: For the qualitative analysis in Fig. 3, I could hardly understand why this shows that the proposed LRKD method is better than the other methods.\n\nThanks for your comment.  Although it seems that there is no significant difference among the three predictions in the above scenario of Fig. 3, their results are 3.5891/6.0794/1, 1.4298/3.0295/1 and 0.6915/1.0974/0 in terms of minADE/minFDE/MR, respectively, indicating that our LaKD provides more accurate predictions. To more intuitively demonstrate the effectiveness of our LaKD method, we have included additional cases in the Figure 1 of the attached pdf to show that our method succeeds where others fail, as shown in the \"global\" response.\n\nRegarding the behavior of the ground truth in the below scenario of Fig. 3, we infer that the vehicle stops at the road ramp. We think this can be inferred by the features of the vehicle, e.g., speed, acceleration, etc.\n\n\n\n\n\n\n\n> **Question 2**: May I ask how could a trivial degenerate solution, or a mode collapse, be avoided when conducting bi-directional self-distillation?\n\nWe apologize for the confusion caused by our unclear presentation. In our method, we design a dynamic soft-masking strategy to effectively conduct bi-directional self-distillation. During the training process, we set different soft-masking weights based on the importance of the units. This protects the more important units from significant damage that could affect the model's performance on trajectories of different lengths. However, these important units are still slowly updating, and their importance does not necessarily increase, as shown in Equation 7. Additionally, our dynamic soft-masking mechanism recalculates the importance of units whenever a new batch of data is encountered, avoiding excessive protection of any single unit. Furthermore, when the model's performance is poor, highly important units also need to be trained. Therefore, we lower their soft-masking weights at the early stages of training to prevent mode collapse, as shown in Equation 8. We will add more details in the final version.\n\n\n\n> **Question 3**: A question about the overall framework is that when we distill the representation of a long history into a short history, the short history lacks information about the longer past.\n\nThank you for your insightful question. During the training process, we can perform our LaKD to distill knowledge from the longer trajectory A to the shorter trajectory C (Note that A and C belong to the same trajectory), allowing C to capture the intrinsic information of A and achieve similar prediction performance. By this means, we expect the shared encoder to extract feature representations for trajectories of different lengths as precisely as possible. During testing, we disable bi-directional self-distillation and only use the backbone module. When a new trajectory B (i.e., B and A belong to different trajectories) comes, we directly extract its features using the shared encoder for future trajectory prediction, which can capture the distinguishing information of B.\n\n\n\n\n\n\n\n> **Question 4**: For Ablation Table 3, may I ask why the performance degrades when M is larger than 3? Intuitively, the performance should improve with a larger mask number?\n\nThanks for your comment. Since our method involves randomly masking historical trajectories M times in each training iteration and continues for a sufficient number of epochs, observation trajectories of all different lengths are seen during training, regardless of the value of M. Consequently, the model's performance does not significantly fluctuant as M changes, indicating that the model is not sensitive to M. This makes M easy to set in real-world scenarios.\n\nRegarding the performance degradation when M is larger than 3, this occurs because, despite our use of a dynamic soft-masking mechanism to prevent knowledge collision, the feature representation of good trajectories may be compromised when knowledge is transferred to bad trajectories through updates to the shared encoder. This compromise becomes more severe as M increases, resulting in reduced performance. We will add these in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PSHDKwljKI", "reviewer": "Reviewer_G7Yk", "summary": "This paper presents the LaKD method to improve trajectory predictions for variable input observation lengths.\n\nLaKD incorporates two key ideas. The first idea is dynamic length-agnostic knowledge distillation. During training time, for each training sample, they augment it by randomly masking the input observation with different lengths. Then, they calculate the prediction errors when using each observation length. They select the prediction with the lowest error as the teacher prediction, and they have the other predictions distill information from the teacher prediction through a KL divergence loss on their feature embeddings. This way, the network is trained to predict good trajectories when given any observation length.\n\nIn order to prevent the knowledge distillation process from affecting the performance of the teacher prediction, they propose a second idea to apply gradient weigths based on the importance of the neurons on the prediction prediction. For neurons that have high importance to the teacher prediction, they multiply their corresponding gradients with a smaller weight.\n\nThis method is a plug-and-play that can be applied to almost any trajectory prediction model. The authors applied the method to the popular HiVT model and QCNet model. They performed evaluations on Argoverse 1, Argoverse 2, and nuScenes datasets. They compared against a few standard baselines such as Original and Random Masking, as well as a similar work FlexiLength Network (FLN). The result shows their LaKD method achieves better performance than the baselines. The authors also performed ablation studies to demonstrate the contributions from components.", "questions": "N/A", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The evaluation result is very thorough. The authors applied their method to two popular trajectory prediction models and performed evaluation on three public datasets.\n\n* The proposed method achieves better performance than the baselines, including a very recent work from CVPR 2024.\n\n* The paper is well-written and easy to follow.", "weaknesses": "* The performance improvement compared to the naive Random Masking baseline is not very significant. I doubt whether it's worth the complexity to use this method in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts on evaluating our work. Here are my responses to your comments:\n\n> **Comment 1**: The performance improvement compared to the naive Random Masking baseline is not very significant. I doubt whether it's worth the complexity to use this method in practice.\n\nThanks for your comment. Firstly, during testing, our model disables bi-directional self-distillation, and remains fully consistent with the backbone structure without requiring any additional inference time or resources. Additionally, our method enables existing models to effectively handle observed trajectories of varying lengths. Secondly, as shown in Table 1, LaKD outperforms Random Masking with relative improvements ranging from 3% to 11%. Additionally, Figures 4, 5, 6, and 7 in the appendix demonstrate that LaKD yields better results than Random Masking across various lengths of observed trajectories. For example, on the Argoverse1 dataset using HiVT as the backbone, LaKD achieves relative improvements of 21.2%/19.2%/10.0% at K=1 and 15.3%/18.8%/28.9% at K=6 over Random Masking in terms of minADE/minFDE/MR with 2-frame observed trajectories.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful and constructive feedback. We really appreciate the reviewers thought our work to be "well-motivated" (Rudm, 4u3J, G7Yk), "easy to follow" (Rudm, 4u3J, G7Yk), "well-written" (Rudm, 4u3J, G7Yk), "novel" (Rudm, 4u3J, G7Yk), "a promising/good topic" (Rudm, 4u3J, G7Yk), "effective in experiments" (Rudm, 4u3J, G7Yk), "sufficient experiments" (4u3J, G7Yk), "tackles an important problem" (Rudm, 4u3J, G7Yk), "insightful and sound" (Rudm, 4u3J, G7Yk), "with solid theoretical part" (Rudm, G7Yk).

We have made point-to-point response to the comments of each reviewer. Additionally, we provide further qualitative results in the attached PDF to address the reviewers' concerns. Once again, we thank all reviewers for their insightful comments which are very helpful for improving the quality of our paper.

Meta-review: All reviewers have positively rated the paper. The AC agrees with the reviewers. Here are some additional comments to potentially improve the final version:
1-  Please discuss if LaKD may face scalability issues with large datasets due to the intensive computation required for importance scores and gradient adjustments. 
2- Dynamic soft-masking may not fully resolve knowledge conflicts, as its effectiveness depends on accurately determining neuron importance, which can be challenging with diverse data patterns.

Thanks.
Decision: Accept (poster)