LaKD: Length-agnostic Knowledge Distillation for Trajectory Prediction with Any Length Observations

Authors: yuhang  li
changsheng  li
ruilin  lv
rongqing  li
ye  yuan
guoren  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 10:23:16

Abstract: Trajectory prediction is a crucial technology to help systems avoid traffic accidents, ensuring safe autonomous driving. Previous methods typically use a fixed-length and sufficiently long trajectory of an agent as observations to predict its future trajectory. However, in real-world scenarios, we often lack the time to gather enough trajectory points before making predictions, e.g., when a car suddenly appears due to an obstruction, the system must make immediate predictions to prevent a collision. This poses a new challenge for trajectory prediction systems, requiring them to be capable of making accurate predictions based on observed trajectories of arbitrary lengths, leading to the failure of existing methods. In this paper, we propose a Length-agnostic Knowledge Distillation framework, named LaKD,  which can make accurate trajectory predictions, regardless of the length of observed data. Specifically, considering the fact that long trajectories, containing richer temporal information but potentially additional interference, may perform better or worse than short trajectories, we devise a dynamic length-agnostic knowledge distillation mechanism for exchanging information among trajectories of arbitrary lengths, dynamically determining the transfer direction based on prediction performance. In contrast to traditional knowledge distillation, LaKD employs a unique model that simultaneously serves as both the teacher and the student, potentially causing knowledge collision during the distillation process. Therefore, we design a dynamic soft-masking mechanism, where we first calculate the importance of neuron units and then apply soft-masking to them, so as to safeguard critical units from disruption during the knowledge distillation process. In essence, LaKD is a general and principled framework that can be naturally compatible with existing trajectory prediction models of different architectures. Extensive experiments on three benchmark datasets, Argoverse 1, nuScenes and Argoverse 2, demonstrate the effectiveness of our approach.

Main paper: 
 introduction: 
 Predicting the future trajectories of dynamic agents in traffic scenarios is a critical task in autonomous driving, enabling autonomous vehicles to make safe decisions [55, 18] . Recently, numerous learningbased methods [60, 43, 39, 48, 50, 57, 59] have been proposed and have demonstrated their effectiveness in trajectory prediction tasks. These methods typically rely on fixed-length and sufficiently long historical trajectories as observations for accurately predicting future trajectories. However, In real-world scenarios, there is often insufficient time to gather an adequate number of observed trajectory points. For example, when a car suddenly appears around a corner, the trajectory prediction model needs to immediately make predictions by utilizing a small number of observed trajectory points to avoid collisions. This poses a new and challenging problem for trajectory prediction, requiring models to make accurate predictions based on observed trajectories of arbitrary lengths. However, as the number of observed trajectory points decreases, the performance of existing methods declines significantly, as shown in Figures 1(a ) and 1(b). Therefore, it is essential to investigate models capable of handling observed trajectories of arbitrary lengths to accurately predict future trajectories.In this paper, we propose a new knowledge distillation framework, Length-agnostic Knowledge Distillation, called LaKD, for trajectory prediction with observations of arbitrary lengths. Firstly, we note that longer trajectories often contain more temporal information, which can potentially lead to higher prediction accuracy compared to shorter trajectories. As shown in Figure 1(c) , the blue vehicle's straight trajectory history can boost confidence in predicting continued straight paths. However, as the number of observed trajectory points increases, additional interference might be introduced. As depicted in Figure 1(d) , despite the longer trajectory of the blue vehicle, it encompasses significant interference, leading to less accurate predictions compared to shorter trajectories. Inspired by this, we devise a dynamic length-agnostic knowledge distillation strategy to adaptively transfer knowledge among trajectories of different lengths. As we know, Knowledge Distillation (KD) techniques [2, 14] have been widely applied in various domains, including computer vision [7, 11] , natural language processing [31, 12] , etc. The basic idea of traditional KD algorithms is to optimize a smaller student model by distilling knowledge from a larger teacher model. In contrast to these KD methods, our strategy emphasizes dynamic knowledge transfer among trajectories of varying lengths, rather than the conventional KD of transferring knowledge from the teacher model to the student model. Our method shares a unique encoder for all trajectories of varying lengths to learn the latent representations of trajectories of varying lengths. It aims to distill the knowledge of 'good' trajectory features to 'bad' trajectories, with the assessment of 'good' or 'bad' trajectories based on their prediction performance. This strategy facilitates adaptive knowledge exchange between long and short trajectories. It aids long trajectories in filtering out interfering information and assists short trajectories in capturing richer temporal information, ultimately obtaining the optimal feature representation for predicting the agent's trajectory.It is worth noting that utilizing a single encoder as both the teacher and student models may affect the prediction performance of a 'good' trajectory when distilling from a 'good' trajectory to a 'bad' trajectory, leading to knowledge collision during the distillation process. A straightforward solution is to train a separate encoder for each trajectory length, but this approach significantly increases computational complexity. To address this issue, we devise a dynamic soft-masking strategy. Since different neuron units in a neural network model usually play different roles for different input data [37] , the core idea of our strategy is to perform soft-masking on the neuron units during gradient updates. Specifically, when training on a 'good' observation trajectory, the importance of the neuron units in the network is calculated based on the gradients. Subsequently, during length-agnostic knowledge distillation, gradients of crucial neuron units are multiplied by a lower update weight to mitigate significant updates. Conversely, less important units' gradients are multiplied by a higher update weight to prioritize their updates. Through this approach, knowledge conflicts can be effectively resolved during the knowledge distillation process.Our contributions can be summarized as follows: (1) We propose LaKD, a length-agnostic knowledge distillation framework for trajectory prediction with observations of any length. LaKD is plug-andplay and compatible with existing models, enabling them to gracefully handle observed trajectories of arbitrary lengths. (2) We design a new knowledge distillation strategy that dynamically transfers knowledge among trajectories of varying lengths. This approach helps long trajectories filter out interfering information and enables short trajectories to capture richer temporal details. Additionally, we devise a dynamic soft-masking strategy to protect crucial neuron units from disruption and prevent knowledge collision during transfer. (3) We perform extensive experiments on three widely-used benchmark datasets, and demonstrate that LaKD significantly outperforms the baselines. Moreover, we show the compatibility of LaKD by integrating it with different trajectory prediction models.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The abstract and introduction clearly state the main contributions of the paper, including the development and evaluation of our proposed model.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Traditional Trajectory Prediction. Traditional trajectory prediction methods aim to predict future trajectories of agents given sufficiently long observed trajectories. To date, many methods have been proposed, including coordinate system based methods [50, 17] , interactive behavior modeling based methods [28, 27, 25] , multimodal approaches [44, 46] . The representative works among coordinate system based methods are pairwise-relative [60, 8, 17, 59, 57] , which can simultaneous predict trajectories for multiple agents while reducing memory consumption and inference latency. Meanwhile, interaction behaviors play an important role in trajectory prediction. To model interactive behaviors within scenes, methods such as Graph Neural Networks [28, 27, 22, 5, 33, 41, 42] and attention mechanisms [39, 1, 6, 36, 32, 52, 23] are introduced. Given the substantial uncertainty surrounding road agents, researchers are exploring diverse methods by integrating multimodal information into predicted trajectories, such as GAN-based [29, 44, 46, 58, 16, 26, 9, 13] , VAE-based [47, 49, 21] , flow-based [30, 56] , and diffusion models [10, 15, 24, 35] to generate multimodal trajectories. However, these methods generally perform well with fixed-length and sufficiently long historical trajectories but experience a significant performance drop when the length of observable historical trajectories varies.Instantaneous Trajectory Prediction. Recently, significant advances have been made in instantaneous pedestrian trajectory prediction tasks, using very short (i.e., two frames) historical trajectories. For example, MOE [45] introduces a unified feature extractor and a pre-training mechanism to capture effective information from momentary observations. DTO [38] employs a knowledge distillation technique to transfer knowledge from long trajectories to short ones. BCDiff [24] develops a bidirectional diffusion model that simultaneously generates both unobserved historical and future trajectories. However, when confronted with input data containing varying numbers of frames, they necessitate training a model for each case, resulting in limited generalizability and high computational complexity. In contrast to these works, we focus on studying how to perform length-agnostic knowledge distillation to adaptively transfer knowledge among long and short trajectories, so as to accurately predict future trajectories with observations of arbitrary lengths. Trajectory Prediction with Complex Observations. Currently, there are limited works focusing on complex observed trajectories for trajectory prediction. The recently proposed GC-VRNN framework [53] facilitates the concurrent execution of incomplete trajectory completion and prediction tasks in a unified framework. However, this model does not take into account important traffic information, e.g., lane, making it unsuitable for vehicle trajectory prediction tasks. The FLN framework [54] , which is most closely related to our work, propagates long historical trajectory information into medium and short trajectories to optimize the fitting of invariant features across multiple subnetworks. However, this strategy requires maintaining three models simultaneously during training, sharply increasing computational complexity. In addition, it is plug-and-play but can only be integrated with Transformer based models. Moreover, this method assumes that longer observed trajectories always contain more useful information for trajectory prediction, and transfers knowledge from longer trajectories to shorter ones. Different from FLN, we observe that longer observed trajectories do not necessarily contain more valuable information than shorter ones for trajectory prediction, and thus explore a length-agnostic knowledge distillation to dynamically transfer knowledge among long and short trajectories, enabling our method to gracefully handle observed trajectories of arbitrary lengths.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We conduct a detailed analysis of the limitations of the framework proposed in this paper in the Conclusion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation: 
 We denote the observed state sequence of the target agent as X obs = {x 1 , x 2 , ..., x T }, where T represents the observed time steps of the target agent, and it can be of arbitrary length greater than 12 .x i ∈ R 2 is the location of the agent at time step i. Additionally, we define the ground-truth future trajectories as X gt = {x T +1 , x T +2 , ..., x T +F }, and the predicted future possible K trajectories asX = {(x k T +1 , xk T +2 , ..., xk T +F )} k∈[1,K], where F denotes the length of the future trajectory. Our objective is to develop a flexible trajectory prediction method capable of handling the case of observed trajectories of arbitrary lengths. Given that longer trajectories contain richer temporal information yet may also entail additional interference, their performance relative to short trajectories can vary. Thus, we attempt to explore a length-agnostic knowledge distillation framework for dynamically transferring knowledge among long and short trajectories, enabling long trajectories to filter out interference and allowing short trajectories to capture richer temporal details. By doing so, we aim to enhance the performance of trajectory prediction with observations of any lengths.The overall framework of the proposed LaKD is shown in Figure 2 . Our framework consists of two parts: a length-agnostic knowledge distillation mechanism and a dynamic soft-masking strategy. First, to enhance the model's ability to handle observed trajectories X obs of arbitrary lengths, we propose a length-agnostic knowledge distillation mechanism. This mechanism first evaluates the performance of the predicted trajectories X, and then determines the direction of knowledge distillation accordingly. Finally, it promotes adaptive knowledge exchange among trajectories of varying lengths, helping long trajectories filter out interfering information and short trajectories capture richer temporal information. However, since this strategy uses a single encoder as both the teacher and student models, it risks causing knowledge conflicts during distillation. Therefore, we propose a dynamic soft-masking strategy to address this issue. Specifically, When training on a 'good' observation trajectory, the importance of neuron units in the network can be determined by their gradients. During lengthagnostic knowledge distillation, crucial neuron gradients are multiplied by a lower update weight to mitigate significant updates, while less important gradients are multiplied by a higher update weight to prioritize their updates. This strategy can effectively resolve knowledge conflicts during distillation, such that our LaKD can effectively perform trajectory prediction based on observations of arbitrary lengths. In essence, LaKD is a plug-and-play approach that can be easily integrated with existing trajectory prediction models, enabling accurate predictions based on observed trajectories of varying lengths.In this section, we introduce our proposed length-agnostic knowledge distillation mechanism, which can facilitate information exchange among trajectories of different lengths, thereby enhancing the model's ability to handle observed trajectories of arbitrary lengths.First, we obtain X obs of M different lengths by performing M random masks on the same observed trajectory, where the m-th trajectory is denoted as X obs m . As shown in Figure 2 , these trajectories are fed into the backbone Φ to generate the latent features V m and predicted trajectories X m :EQUATIONwhere Φ E and Φ D denote the encoder and decoder of Φ, with parameterized by ϕ E and ϕ D , respectively. The backbone Φ can be any trajectory prediction model, e.g., HiVT [60] and QCNet [59] used in this paper, making our method plug-and-play.As aforementioned, longer trajectories contain richer temporal information but may also involve additional interference for trajectory prediction, thus we design a length-agnostic knowledge distillation strategy, where knowledge transfer can occur from longer trajectories to shorter ones, as well as from shorter to longer trajectories. To dynamically determine the direction of knowledge transfer, we employ the prediction performance based on different observed trajectories to find a 'good' trajectory, and attempts to distill the knowledge embedded in its latent features V m to those of 'bad' trajectories. To measure the prediction performance, we calculate the minimum distance D m between the predicted trajectories X m and the ground-truth trajectories X gt using the l 2 norm:EQUATIONDuring training, if the prediction performance of the current observation trajectory is worse than that of a previous 'good' observation trajectory, we begin to distill knowledge from the 'good' trajectory to the current trajectory. In this paper, we use the latent features as knowledge for transfer, and use the KL divergence [20] to minimize the following distillation loss to achieve the goal:EQUATIONwhere V good represents the latent features of the 'good' trajectory. By Eq. ( 3), the features V m of 'bad' trajectories are expected to be optimized towards those of 'good' trajectories, i.e., V good . This facilitates effective knowledge transfer from 'good' to 'bad' trajectories. It is worth noting that different from traditional knowledge distillation optimizing a smaller student model by distilling knowledge from a larger teacher model, we utilize a unique encoder to encode all trajectories of arbitrary lengths, and distill knowledge from 'good' trajectory to 'bad' one. This may lead to knowledge collision during the distillation process, degrading the feature representation capability of the 'good' trajectory. To this end, we devise a dynamic soft-masking strategy to address the issue of knowledge collision.As we know, different neuron units in a neural network model typically play distinct roles for various input data [37] . Thus, we attempt to perform soft-masking on the neuron units during gradient updates. Specifically, when training on a 'good' observation trajectory, we determine the importance of the neuron units based on their gradients. During length-agnostic knowledge distillation, the gradients of crucial neuron units are multiplied by a lower update weight to prevent significant updates, while the gradients of less important units are multiplied by a higher update weight to prioritize their updates. By this strategy, knowledge conflicts can be effectively resolved during the distillation process.Importance Score of Neuron Unit. During the training process, if the gradient of a neuron unit is large, it indicates that changing it will have a significant impact on the result [37, 19] . Building on this, we aim to identify which units are essential for the model to produce accurate predictions. To do so, we first calculate the importance scores of the different units in the network as follows:EQUATIONwhere B denotes the batch size. X b and X gt b represent the predicted trajectories and the ground-truth trajectories, respectively. g u is introduced as a virtual parameter for calculating the importance I u of units, where we fix g u to 1 in the training. I u is the importance score of the u-th neuron unit.Since the gradients of the neuron units are usually very small, they cannot be directly applied to the calculation of soft masking weights. Therefore, it is necessary to confine the values of importance scores within the range [0,1]. To achieve this, we first normalize the importance scores of all units within each layer, ensuring a mean of 0 and a standard deviation of 1. Then, we apply the Tanh activation function to these normalized scores as:EQUATIONwhere µ represents the average importance of all units in the l-th layer, while σ denotes their variance.Accumulation of Importance Scores. Due to the fact that different units in the model play varying roles for trajectories of varying lengths, it is necessary to preserve the model's ability as much as possible during training. Therefore, during the m-th training iteration, we need to comprehensively consider the importance of units from the previous m-1 training iterations, and employ the elementwise maximum (EMax) operation for calculating the cumulative importance Î(≤m-1) u of the model up to the (m-1)-th iteration:EQUATIONwhere we set Î(0) u uniformly to 0.Dynamic Soft-Masking of Units. During early training stages, when the model's predictive capability is initially constrained, the informativeness of unit importance scores is limited. As training progresses, the reliability of these scores gradually improves. Therefore, we introduce a dynamic decay coefficient ξ to control the strength of the soft-masking. The specific formulas for the decay coefficient and the dynamic soft-masking mechanism based on the importance of units are as follows:EQUATIONEQUATION)where γ is a hyperparameter. L reg represents the regression loss between the predicted trajectories and the ground-truth trajectories, as used in HiVT [60] and QCNet [59] . ∇ u and ∇u represent the gradients of the units before and after soft-masking, respectively. The dynamic soft-masking mechanism effectively addresses the issue of knowledge conflict between trajectories of different lengths, promotes cross-length information exchange and thereby enhances the model's ability to predict trajectories based on observations of arbitrary lengths.Optimization. Following HiVT [60] and QCNet [59] , we also adopt the negative log-likelihood as the regression loss L reg , which regresses the trajectory closest to the ground truth. In addition, We also use the cross-entropy loss as the classification loss L cls to optimize the trajectory prediction model. Finally, the total loss function can be expressed as follows:EQUATIONwhere α and β are the hyperparameters used to balance the contributions of different loss functions.We provide the pseudo-code of the training procedure in Appendix A.1.After training, the model can be utilized for trajectory prediction based on observations of arbitrary lengths. For a new observed trajectory of any length, we directly input it into the encoder and decoder for future trajectory prediction, bypassing knowledge distillation and soft masking.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We will release the code as open-source as soon as the paper is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental settings: 
 Dataset. We evaluate the performance of our method on three widely used datasets: Argoverse 1 [4] , nuScenes [3] and Argoverse 2 [51] . The Argoverse 1 dataset comprises 323,557 real driving scenes from Miami and Pittsburgh. The observation duration is 5 seconds with a sampling frequency of 10Hz. Traditional trajectory prediction approaches typically assume that the first 2 seconds represent the historical observed trajectories, while the last 3 seconds are considered as the future ground-truth trajectories. The nuScenes dataset comprises 32,186 training scenarios, 8,560 validation scenarios, and 9,041 test scenarios. Each scenario spans 8 seconds, sampled at 2 Hz. Traditional trajectory prediction approaches typically assume that the first 2 seconds (5 locations) are used as the observed trajectory, while the last 6 seconds are designated as the future ground-truth trajectory. The Argoverse 2 dataset includes 250,000 scenes spanning across six cities. The observation duration is 11 seconds with a sampling frequency of 10Hz. Traditional trajectory prediction approaches typically assume that the first 5 seconds are used as historical observed trajectories, while the last 6 seconds serve as future ground-truth trajectories. By masking trajectories on these datasets, we aim to evaluate the effectiveness of our trajectory prediction method with observations of arbitrary lengths.Evaluation Metrics. To comprehensively evaluate the model, we employ a set of evaluation metrics based on the minimum Average Displacement Error (minADE), minimum Final Displacement Error (minFDE), and Miss Rate (MR) as:EQUATIONEQUATIONEQUATIONwhere H denotes the maximum number of observation points, and K represents the number of trajectories to be predicted. T = i represents the number of observation points. We evaluate the performance for each observation length and then average the results across all lengths to obtain the final outcome. Backbone and Baselines. To demonstrate the compatible ability of our LaKD, we combine it with two representative trajectory prediction methods: HiVT [60] and QCNet [59] . To verify the effectiveness of our method, we compare LaKD with FlexiLength Network (FLN) [54] , the work most related to ours. FLN integrates trajectory data with diverse observation lengths and attempts to learn temporally invariant representations for future trajectory predictions. We also compare LaKD with (DTO) [38] . DTO is initially developed for instantaneous trajectory prediction. To ensure fairness, we modify its framework to distill from complete trajectories into arbitrary length trajectories. Moreover, we take Orig and RM as our baselines. Orig denotes using the original fixed-length observed trajectories as inputs for training the backbones, while RM involves randomly masking the original observed trajectories to generate trajectories of varying lengths as inputs for training the backbones.Implementation Details. During training, we set M in our LaKD to 3, and both α and β to 1.For Argoverse 1, nuScenes, and Argoverse 2, we set γ to -1, -0.65, and -1.35, respectively. The dimensionality of the encoded latent feature V m is set to 128. We utilize the AdamW optimizer [34] , setting the learning rate and weight decay parameters to 5e-4 and 1e-4, respectively. The batch size is set to 32. The experiments are implemented using PyTorch [40] on the NVIDIA GeForce RTX 4090.Performance on Trajectory Prediction with Observations of Arbitrary Lengths. We evaluate the overall performance of our method, as listed in Table 1 . Based on Table 1 , our method outper-forms all others, particularly surpassing FLN, across all the three datasets. This demonstrates the effectiveness of our method in predicting future trajectories from observations of varying lengths. Moreover, our LaKD outperforms Orig, indicating the necessity of developing a trajectory prediction method specifically designed to handle observations of varying lengths. Finally, our LaKD achieves the best performance with various backbones, demonstrating the compatibility of our method. More detailed results are presented in Appendix A.2.Ablation Study. We conduct ablation studies to validate the effectiveness of our proposed components using HiVT as the backbone on the Argoverse 1 dataset. Since the Random Masking strategy was first presented in this paper, we also conduct ablation study to verify its effectiveness. The results are shown in Table 2 . The experiments demonstrate that as we progressively remove the dynamic soft-masking mechanism (DSM), the Length-agnostic Knowledge Distillation (LaKD), and the Random Masking (PM), the performance of our method gradually declines, demonstrating the effectiveness of our proposed components. By combining these components, our method achieves the best performance. Analysis of Different Mask Numbers M . We investigate the impact of different mask numbers M in our LaKD on the trajectory prediction performance. We use HiVT [60] as the backbone, and list the results in Table 3 . Since our method involves randomly masking historical trajectories M times in each training iteration and continues for a sufficient number of epochs, observation trajectories of all different lengths are seen during training, regardless of the value of M . Consequently, the model's performance does not significantly degrade as M changes, indicating that the model is not sensitive to M . This makes M easy to set in real-world scenarios. For our experiments, we set M = 3. Qualitative Analysis. To intuitively demonstrate the effectiveness of our LaKD, we perform a qualitative experiment on the Argoverse 2 dataset, as shown in Figure 3 . The first row features a scenario at a T-junction where the agent is about to turn, with an observed trajectory spanning 5 points. The second row illustrates a scenario at a fork in the road, where the agent is preparing to change lanes, with an observed trajectory of 10 points. It is observable that across different scenarios, our method exhibits higher accuracy compared to other models. 
conclusion: 
 In this paper, we propose a length-agnostic knowledge distillation framework for trajectory prediction with observations of any length. This framework enables long trajectories to filter out interfering information and short trajectories to capture richer temporal details. To address knowledge conflicts during distillation, we devise a dynamic soft-masking mechanism to protect crucial neuron units from disruption, thereby enhancing prediction performance. Extensive experiments on the Argoverse 1, nuScenes, and Argoverse 2 datasets demonstrate the effectiveness of our approach and its compatibility with various trajectory prediction models.  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We specify all the training and test details in the Section 4 and the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Due to the high cost of training, we did not perform multiple runs to compute error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide the information on the computer resources in the Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discusses the potential positive societal impacts of the work performed, highlighting the benefits and advancements it can bring to the field and society at large. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks, because trajectory prediction models do not have high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have properly credited and cited the creators or original owners of assets, including code and papers, used in our work. We have explicitly mentioned the licenses and terms of use for these assets and have respected them accordingly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 and QCNet [59] on the Argoverse 1 [4] and Argoverse 2 [51] datasets by using observed trajectories of different lengths, respectively. Figure 1(c) and Figure 1 (d) display scenarios where longer trajectories perform better and shorter trajectories perform better, respectively. The red line represents the ground-truth future trajectory. The solid green and blue lines depict the observed trajectories, while the dashed green and blue lines illustrate the predicted trajectories. 
 
 Appendix: In this work, we aim to distill knowledge from 'good' trajectory to 'bad' trajectory for improving the prediction performance from observations of any lengths. However, how to determine a 'good' or 'bad' trajectory is an open problem. Currently, we adopt a heuristic strategy by utilizing the distance between the predicted trajectory and the ground-truth trajectory. More complex strategies, such as reinforcement learning, are worth further exploration and investigation.NeurIPS Paper ChecklistJustification: The research conducted in the paper conforms fully with the NeurIPS Code of Ethics. We have reviewed the guidelines thoroughly and ensured that all aspects of our work adhere to the ethical standards set forth by NeurIPS.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Previous works[45,24] have shown that when the agent has only one frame of historical trajectory data, it cannot be predicted due to the lack of basic information such as velocity and direction. 


Decision: All reviewers have positively rated the paper. The AC agrees with the reviewers. Here are some additional comments to potentially improve the final version:
1-  Please discuss if LaKD may face scalability issues with large datasets due to the intensive computation required for importance scores and gradient adjustments. 
2- Dynamic soft-masking may not fully resolve knowledge conflicts, as its effectiveness depends on accurately determining neuron importance, which can be challenging with diverse data patterns.

Thanks.

Reviews: review: paper_summary: paper_summary: The paper presents a length-agnostic knowledge distillation framework, which is motivated from knowledge transfer among trajectories of different lengths. The authors address knowledge conflicts during distillation from a dynamic soft-masking mechanism. The evaluation is conducted using Argoverse 1, nuScenes, and Argoverse 2 datasets.
strength_weakness: strength_weakness: - Lacks a proper qualitative results. 
- Limited choice of backbone models.
See below for details.
questions: questions: - The qualitative results do not clearly demonstrate the scenarios where the authors have been motivated. As shown in Figure 3, there is no difference between (b) and (c), which makes me doubt about the contribution of this work. Not just closer to the ground-truth, it is strongly suggested where the proposed method works while others fail. 
- Different backbone models other than HiVT and QCNet would be needed to claim its generality.
limitations: limitations: The paper does not provide any failure cases or insights into the limitations.
paper_summary: paper_summary: To tackle the length-agnostic trajectory prediction problem, the authors are motivated to utilize knowledge learned from both longer and shorter trajectories. They propose a plug-and-play self-distillation framework for trajectory prediction, which can be integrated with many different off-the-shelf trajectory prediction models. Specifically, for a given long trajectory, the authors first randomly mask the trajectory into several different lengths, maintaining the same prediction horizon but with different history lengths. They then determine the direction of knowledge distillation based on the performance of the model with longer or shorter history. The better-performing trajectory length is used to distill knowledge into the intermediate features of represented from other lengths. Additionally, the authors introduce a soft masking method at neuron level, updating the distillation of important neurons slowly and vice versa. They test their framework on Argoverse1, nuScenes, and Argoverse2 datasets, using off-the-shelf models HiVT and QCNet, and demonstrate performance gains.
strength_weakness: strength_weakness: W1. One constraint for most knowledge distillation scenarios is that for the input to the model being distilled and the model distilling, at least the information should be equivalent or nearly equivalent between the inputs to both models. I'm a bit confused about why the overall framework of bidirectional self-distillation works from an information theory / flow perspective. For my detailed confusion, please refer to my question section.

W2. For the qualitative analysis in Fig. 3, I could hardly understand why this shows that the proposed LRKD method is better than the other methods. In the above scenario, there doesn't seem to be a significant difference between the three predictions. In the below scenario, the proposed method is closer to the ground truth, but I do not fully understand why the ground truth behaves in that manner and how this is inferred by the model.
questions: questions: Q1. Could the authors explain further what do they mean by “Knowledge Collision”? 

Q2. May I ask how could a trivial degenerate solution, or a mode collapse, be avoided when conducting bi-directional self-distillation? I am aware that the authors try to classify the neurons in Section 3.4, but it seems to me that by the metrics as in Equation 4, the important neurons will get even more important length-agnostically, and therefore get distilled more both for long and for short trajectories. 

Q3. A question about the overall framework is that when we distill the representation of a long history into a short history, the short history lacks information about the longer past. For example, when trajectory A and trajectory B have the same last several time steps but differ in that one turns left earlier while the other turns right, the information intrinsic to these trajectories is different. Let’s refer to the last time stamps of both A and B as a shorter C, as A and B share the same last steps. The longer horizon trajectory encoder of A can capture this information, but when it is distilled to the encoder with a shorter history C, this distinguished information of A is not seen. But the proposed framework enforces distilling this uncertain knowledge of A to the short history C, even if in a future potential test situation, the longer trajectory turns in the opposite direction in the past, like B, than seen in the training set. May I ask how is the proposed framework valid, considering a similar situation?

Q4. For Ablation Table 3, may I ask why the performance degrades when M is larger than 3? Intuitively, the performance should improve with a larger mask number?
limitations: limitations: Briefly discussed in Conclusion section.
paper_summary: paper_summary: This paper presents the LaKD method to improve trajectory predictions for variable input observation lengths.

LaKD incorporates two key ideas. The first idea is dynamic length-agnostic knowledge distillation. During training time, for each training sample, they augment it by randomly masking the input observation with different lengths. Then, they calculate the prediction errors when using each observation length. They select the prediction with the lowest error as the teacher prediction, and they have the other predictions distill information from the teacher prediction through a KL divergence loss on their feature embeddings. This way, the network is trained to predict good trajectories when given any observation length.

In order to prevent the knowledge distillation process from affecting the performance of the teacher prediction, they propose a second idea to apply gradient weigths based on the importance of the neurons on the prediction prediction. For neurons that have high importance to the teacher prediction, they multiply their corresponding gradients with a smaller weight.

This method is a plug-and-play that can be applied to almost any trajectory prediction model. The authors applied the method to the popular HiVT model and QCNet model. They performed evaluations on Argoverse 1, Argoverse 2, and nuScenes datasets. They compared against a few standard baselines such as Original and Random Masking, as well as a similar work FlexiLength Network (FLN). The result shows their LaKD method achieves better performance than the baselines. The authors also performed ablation studies to demonstrate the contributions from components.
strength_weakness: strength_weakness: * The performance improvement compared to the naive Random Masking baseline is not very significant. I doubt whether it's worth the complexity to use this method in practice.
questions: questions: N/A
limitations: limitations: Yes

Comments: title: Official Comment by Authors
Official Comment by Authors
Thank you for your response
A more detailed explanation on question 3.
Raised my rating from 5 to 6
comment: Thank you for your decision to maintain the current rating. I appreciate your time and feedback.
Thank you very much for the score improvement and your constructive feedback. We will further polish the paper in the final revision. Thank you!
Thanks for providing the response. They are clear now. Please add new figure and results to the final version.
Thank you for your response. I will keep my rating.
Thank you very much for recognizing our work. I will provide a more detailed explanation of Question 3, as follows:
During the training process, we can perform our LaKD to distill knowledge from the longer trajectory A to the shorter trajectory C (Note that A and C belong to the same trajectory), allowing C to capture the intrinsic information of A and achieve similar prediction performance. By this means, we expect the shared encoder to extract feature representations for trajectories of different lengths as precisely as possible. Our Length-agnostic Knowledge Distillation framework will only enhance the model's ability to extract temporal features of trajectories, improving the model's capacity to capture every temporal change in the observed trajectory, without reducing the model's generalization ability.

During testing, we disable bi-directional self-distillation and only use the backbone module. When a new unseen trajectory B (i.e., B and A belong to different trajectories) comes, the model does not predict based on the previously seen trajectories A and C, but instead directly extracts its features using the shared encoder for future trajectory prediction. The model can capture the distinguishing information of B, recognize B's turning behavior, and make predictions based solely on B's observed trajectory, without being influenced by A and C.
Thank you so much for answering my questions! Most of my questions are well answered. It would be great if you could provide a more detailed explanation on question 3.