TSC-Net: Prediction of Pedestrian Trajectories by Trajectory-Scene-Cell Classification

Authors: BO HU, Tat-Jen Cham
Primary area: learning on time series and dynamical systems
Keywords: Trajectory-Scene-Cell, Trajectory Prediction, Attention

Abstract:
To predict future trajectories of pedestrians, scene is as important as the history trajectory since i) scene reflects the position of possible goals of the pedestrian ii) trajectories are affected by the semantic information of the scene. It requires the model to capture scene information and learn the relation between scenes and trajectories. However, existing methods either apply Convolutional Neural Networks (CNNs) to summarize the scene to a feature vector, which raises the feature misalignment issue, or convert trajectory to heatmaps to align with the scene map, which ignores the interactions among different pedestrians. In this work, we introduce the trajectory-scene-cell feature to represent both trajectories and scenes in one feature space. By decoupling the trajectory in temporal domain and the scene in spatial domain, trajectory feature and scene feature are re-organized in different types of cell feature, which well aligns trajectory and scene, and allows the framework to model both human-human and human-scene interactions. Moreover, the Trajectory-Scene-Cell Network (TSC-Net) with new trajectory prediction manner is proposed, where both goal and intermediate positions of the trajectory are predict by cell classification and offset regression. Comparative experiments show that TSC-Net achieves the SOTA performance on several datasets with most of the metrics. Especially for the goal estimation, TSC-Net is demonstrated better on predicting goals for trajectories with irregular speed.

Decision: accept (poster)
Reviewer scores: 8, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730039065117

thread_type:
official_review

<thread_object>
creator: reviewer_fgc6

<content>
summary:
The authors emphasize the importance of utilizing scene information for trajectory prediction and propose a method to do so. 
According to the authors, existing approaches have limitations: some methods use CNNs to convert scenes into feature vectors, leading to misalignment issues, while others employ heatmaps to represent trajectories as image features, neglecting pedestrian interactions. 
This paper introduces "trajectory-scene-cell feature" to represent both the trajectory and the scene with the same representation. 
By integrating these two types of information into a single representation, the method enables human-human and human-scene interactions. 
To decode future trajectory from this representation, a Trajectory-Scene-Cell Network is proposed. 
It achieves state-of-the-art prediction performance on some datasets and some metrics.

strengths:
* Previous pedestrian trajectory prediction papers tend not to emphasize the importance of effectively utilizing scene information, but this paper points out this gap.
* The paper is well-structured and easy to follow.
* The proposed method is intuitively understandable, technically well-implemented, and achieves comparable prediction performance.

weaknesses:
* The referenced trajectory prediction papers focus heavily on older works. It would be beneficial to include more recent papers.
* If the authors claim that the ETH-UCY dataset only contains binary-level semantics, why not experiment with vehicle trajectory datasets instead? Vehicle datasets offer richer semantic scene information, such as lanes, road edges, and traffic lights, so it seems more convincing to conduct experiments on vehicle datasets for claiming effective use of scene information. Moreover, numerous methods already proposed how to effectively utilize these scene information:
    - HiVT (CVPR, 2022): It proposes an integrated hierarchical transformer structure using both scene and trajectory vectors.
    - Leveraging Future Relationship Reasoning for Vehicle Trajectory Prediction (ICLR, 2023): It models agent interactions using vectorized scene information.
    - Apart from these vectorized methods, many models also use image features. Demonstrating state-of-the-art performance in vehicle scenarios by leveraging scene information would be more convincing.
* The CVAE-based trajectory decoder for goal prediction does not seem new. The sequential prediction of goals and trajectories has been widely explored. For instance, Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals (CoRL, 2022) presents a more sophisticated approach with predictions conditioned on traversals.
* The ablation study sounds insufficient. It is hard to verify the effectiveness of the proposed TSC. There is only one ablation experiments, but it is hard to see where the performance gain came from (from the proposed methods or finetuning hyperprameters). For example, it is necessary to observe how prediction performance changes when certain features ($z_{i,t}^{(sc)}$ or $a_{i,t,j,k}^{(step)}$) are excluded.
* There are other methods with better performance on the ETH-UCY dataset, making it difficult to claim state-of-the-art performance based on average metrics.
    - View Vertically (ECCV, 2022): 0.18/0.28

questions:
* The ETH-UCY dataset also includes RGB images of the scene. Instead of using binary label semantics, why not use the RGB images? It’s unclear why RGB information was used for the SDD dataset but not for ETH-UCY.
* In equations 7 and 9, is the network 𝑓 the same as the 𝑓 in equation 3? If they are different, the notation should be distinguished. If they are the same, how can they be identical when the inputs in equations 3 and 7 are different?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730296336699

thread_type:
official_review

<thread_object>
creator: reviewer_rvzk

<content>
summary:
In this study, the trajectory-scene-cell feature is introduced to integrate both trajectories and scenes into a unified feature space. By decoupling the trajectory in the temporal domain and the scene in the spatial domain, the trajectory and scene features are reorganized into various cell types that effectively align both aspects. This approach facilitates the modeling of interactions between humans as well as interactions between humans and their environments. Additionally, The TSC-Net is introduced, utilizing an innovative trajectory prediction approach that predicts both the goal and intermediate trajectory positions through cell classification and offset regression. Experiments conducted on two datasets demonstrate that TSC-Net improves the accuracy of pedestrian trajectory prediction.

strengths:
1.This paper proposes a novel joint representation called the trajectory-scene-cell feature and introduces an innovative trajectory prediction pipeline.

2.The structure of the paper is comprehensive, and the writing is clear. Compared to the baselines, it demonstrates commendable results on several public datasets.

weaknesses:
1.The layout design of Figure 2 is unappealing and does not effectively highlight the core innovations of this research. It is recommended to make the various steps of the framework more explicit in the figure, such as the model's inputs and outputs.

2.The paper is quite challenging to read due to the confusing formula symbols, such as the Formula 7 and Formula 8, what does $q$ represent for and what is the meaning of $i$, $j$ and $k$?

questions:
1. In Formula 11, what is the prior distribution of $\varepsilon$, and does $p(\varepsilon)$ represent the standardized normal distribution?

2. How is $Z_T$ used for guidance ,which is mentioned in section 3.4.2?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730503052386

thread_type:
official_review

<thread_object>
creator: reviewer_rsfq

<content>
summary:
This paper addresses the scene usage limitations of prior approaches in pedestrian trajectory prediction by introducing the trajectory-scene-cell (TSC) feature and TSC-Net. The authors convert raw trajectory and scene inputs into three types of TSC features: trajectory cells, goal scene cells, and step scene cells, which are utilized for trajectory prediction. The paper reframes the trajectory prediction task as a classification problem. Furthermore, the authors claim that their method achieves state-of-the-art performance and is demonstrated better
on predicting goals for trajectories with irregular speed.

strengths:
1.	The trajectory-scene-cell feature in pedestrian trajectory prediction is somewhat innovate.
2.	The paper is well-written and easy to follow.

weaknesses:
1.	My main concern is whether the TSC feature genuinely works better than the traditional CNN and heatmap approaches. Because in the performance comparison, TSC-Net significantly underperforms compared to Y-Net, NSP-SFM, V2-Net, and E-V2-Net. And there is no ablation study provided to demonstrate the performance of replacing the TSC feature with traditional CNN and heatmap approaches.
2.	The authors claim that Y-net and NSP-SFM performs better because their results come from clustering 1000 samples. However, I only find they also use best-of-20 (K=20) protocol in their paper. Moreover, if they indeed use extra samples, the result of the TSC-Net under the same setting could be given for comparison.
3.	The authors claim in abstract that TSC-Net is demonstrated better on predicting goals for trajectories with irregular speed. However, Figure 3 only provides a few examples for demonstration, which is insufficient to illustrate the overall situation. A more detailed statistical table should be included.
4.	The ETH-UCY and SDD datasets are relatively small and consist of simple trajectories. The performance of TSC-Net on more complex datasets, such as NBA[1], would provide a clearer understanding of its capabilities. 
[1] Conghao Wong, Beihao Xia, Ziqian Zou, Yulong Wang, and Xinge You. Socialcircle: Learning the angle-based social interaction representation for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19005–19015, 2024.

questions:
1.	Explain the underperformance of all four approaches in ETH-UCY and demonstrate TSC feature genuinely works better than the traditional CNN and heatmap approaches.
2.	A more detailed statistical table should be included for irregular speed as author claims.
3.	Performance on more complex datasets, such as NBA[1]. 
Other issues can refer to the weaknesses.

[1] Conghao Wong, Beihao Xia, Ziqian Zou, Yulong Wang, and Xinge You. Socialcircle: Learning the angle-based social interaction representation for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19005–19015, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730531573018

thread_type:
official_review

<thread_object>
creator: reviewer_igai

<content>
summary:
The paper proposes a novel framework, TSC-Net, with an innovative Trajectory-Scene-Cell (TSC) feature representation designed for pedestrian trajectory prediction.  In particular, it effectively captures both individual pedestrian behavior and the contextual scene information. By decoupling trajectory data in the temporal domain and scene information in the spatial domain, the model effectively captures both human-human interactions and human-scene dynamics, leading to improved prediction accuracy compared to existing methods. The experiments conducted on various datasets seem to validate the efficacy of the proposed method.

strengths:
+ The paper is well written and easy to follow, with clear explanations of methodologies, results, and implications.

+ It is a well-designed idea to amalgamate trajectory and scene information into a unified TSC feature space. Such a joint feature allows for better modeling of both human-human and human-scene interactions.

+ The authors reformulate the trajectory prediction task as a classification problem by estimating the likelihood of each cell containing the goal or the next steps, which distinguish the proposed methods from the existing regression-based trajectory prediction.

+ Extensive experimentation across multiple datasets, i.e., SDD and ETH-UCY, demonstrating the robustness and generalizability of TSC-Net. Such a clear improvement over state-of-the-art methods strongly demonstrate the model's efficacy.

weaknesses:
- The proposed model has shown tendencies to bias its trajectory predictions towards the centers of scene cells. This could lead to inaccuracies, particularly when pedestrians navigate along edges or less central paths.

- The discussion of limitations is relatively brief. If the authors can provide some vivid examples, then it should be easier for readers to better understand.

- It is still not clear if the proposed method is able to sufficiently address scenarios with higher pedestrian densities or complex environments especially in urban settings with numerous dynamic agents.

questions:
See the "Weakness" section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732370386409

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal by Authors

comment:
We appreciate the valuable comments and suggestions provided by all the reviewers. We are grateful for the time and effort all the reviewers dedicated to our manuscript. Please refer to our individual responses for detailed information on each review. We look forward to further comments from reviewers. The manuscript and pdf file in supplemental material are also revised accordingly and re-submitted.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a novel framework, TSC-Net, with an innovative Trajectory-Scene-Cell (TSC) feature representation designed for pedestrian trajectory prediction. The paper is well written and easy to follow, with clear explanations of methodologies, results, and implications. Extensive experimentation across multiple datasets, i.e., SDD and ETH-UCY, demonstrating the robustness and generalizability of TSC-Net. Such a clear improvement over state-of-the-art methods strongly demonstrate the model's efficacy. This paper lacks a discussion of the limitations of the proposed methods, especially for complex dynamic scenarios. The reviewers also pointed out the lack of experiments to verify the advantages of the method over traditional methods, the unclear description of the motivation and the lack of comparison with the latest methods. The author's detailed response can address most of the reviewer's concerns. So the final vote is acceptance.