TSC-Net: Prediction of Pedestrian Trajectories by Trajectory-Scene-Cell Classification
Authors: BO HU
Tat-Jen Cham
Venue: ICLR 2025 Poster (2025)
Keywords: Trajectory-Scene-Cell
Trajectory Prediction
Attention
Primary area: learning on time series and dynamical systems

Abstract: To predict future trajectories of pedestrians, scene is as important as the history trajectory since i) scene reflects the position of possible goals of the pedestrian ii) trajectories are affected by the semantic information of the scene. It requires the model to capture scene information and learn the relation between scenes and trajectories. However, existing methods either apply Convolutional Neural Networks (CNNs) to summarize the scene to a feature vector, which raises the feature misalignment issue, or convert trajectory to heatmaps to align with the scene map, which ignores the interactions among different pedestrians. In this work, we introduce the trajectory-scene-cell feature to represent both trajectories and scenes in one feature space. By decoupling the trajectory in temporal domain and the scene in spatial domain, trajectory feature and scene feature are re-organized in different types of cell feature, which well aligns trajectory and scene, and allows the framework to model both human-human and human-scene interactions. Moreover, the Trajectory-Scene-Cell Network (TSC-Net) with new trajectory prediction manner is proposed, where both goal and intermediate positions of the trajectory are predict by cell classification and offset regression. Comparative experiments show that TSC-Net achieves the SOTA performance on several datasets with most of the metrics. Especially for the goal estimation, TSC-Net is demonstrated better on predicting goals for trajectories with irregular speed.

Published as a conference paper at ICLR 2025
-SC-ÑET: PREDICTION OF PEDESTRIAN TRAJECTO- RIES BY TRAJECTORY-SCENE-CEL CLLENEECCLLCLSSSSICCAAAIIOO CLASSIFICATION
Bo Hu, Tat-Jen Cham
College of Computing and Data Science
Nanyang Technological University 50 Nanyang Ave, Block N4, Singapore hubo00056e ntu. edu. sg, jstjchnenttu. edu. sg
ABSTRACT
To predict future trajectories of pedestrians, scene is as important the history trajectory since i) scene reflects the position of possible goals of the pedestrian ii) trajectories are affected by the semantic information of the scene. It requires the model to capture scene information and learn the relation between scenes and trajectories. However, existing methods either apply Convolutional Neural Net- works (CNNs) to summarize the scene to a feature vector, which raises the feature misalignment issue, convert trajectory to heatmaps align with the scene map, which ignores the interactions among different pedestrians. In this work, we intro- duce the trajectory-scene-ceel feature represent both trajectories and scenes in one feature space. By decoupling the trajectory in temporal domain and the scene in spatial domain, trajectory feature and scene feature are re-organized in different types of cell feature. which well aligns trajectory and scene, and allows the frame- work to model both human-human and human-scene interactions. Moreover, the Snjjector--Scne--ee
Network (TSC-Net) with new trajectory prediction manner is proposed, where both goal and intermediate positions of the trajectory are pre- dict by cell classification and offset regression. Comparative experiments show that TSC-Net achieves the SOTA performance on several datasets with most the metries. Especially for the goal estimation, TSC-Net is demonstrated better on predicting goals for trajectories with irregular speed.
INTRODUCTION
The problem addressed in this work that of predicting pedestrian trajectories in the future given historical and scene information Liang et al. (2020a): Deo & Trivedi (2020); Chen et al. (2021a), which has strong technical relevance in many applications, such as seli-driving vehicles and human- robot interaction Gulzar et al.] (2021); Bennewitz et al. (2022). While there have been promising advances research in the past few years, this task remains challenging. The behavior
per- is determined by both self-intention and social interaction Hu & Cham (202)). In the context of pedestrians, we consider self-intention to determine the planned destination and intended route, governed by the individual's goal and prior knowledge of the environment, and is longer term attribute. Social interaction relates to shorter term reactive behavior, such as avoiding collision with other independent pedestrians, or more complex behavioral dynamics such as group movement. Since it is not possible to discer the exact intention of pedestrian based purely on past visual ob- servations, the prediction of motion becomes less certain the further into the future the prediction is attempted. Therefore, the appropriate approach in this situation is to model the distribution of goals (i.e. intended destinations), together with the distribution of paths to get there. If the environment is known, say in the form of scene map, this will influence the distribution of goals and paths, e.g. store entrance will be more probable as a goal than the exterior wall next to it, and paths are more likely to follow walkways. We consider these effects to be human-scens interactions, counterpart to suman-human interactions.
Most of the earlier deep learning- based methods for human trajectory prediction had been predom- inantly focused on human -urman interactions Gupta et al. (2018); Alahi et al. (2016); Zhhng et al.
2019): Yu al. (000)). Later works further considered how semantic information from the scene

---

Published as a conference paper at ICLR 2025
also significantly affect human decisions, and incorporated scene information into this task. The most intuitive way is to concatenate scene feature vector obtained from a Convolutional Neural Network (CNN). Both local scene maps Salzmann et al. (2020): Liang et al. (2019) and global maps Sadeghian et al. 2019); Fang et al. (2020) have been investigated as the prediction input in different methods. In these methods, local scene maps are found not to be suitable for longer range inference.
For example, the model may not have sufficient input to correctly predict that a pedestrian is heading for a bus stop, the bus stop outside the range of the input scene maps, and there no additional information available. In contrast, the global scene feature contains all necessary information; how- ever, trajectory features are local representations focusing on specific positions. The concatenation of trajectory feature and global scene feature will result in difficulty learning the relations between the trajectory and the scene, as the scale of these two feature types are misaligned. In Mangalam et al. (202T), an alternate strategy is provided to fuse scene maps and trajectories, where the tra- jectory is converted to pedestrian position-basee -based heatmaps at different time steps, and the heatmaps are concatenated to the scene maps. In this way, the coordinate information is well aligned with the scene semantic information in the spatial domain. However, each pedestrian is represented by a 3-dimensional tensor in Mangalam al. (202T), which makes the computational cost of computing human-human interactions very high.
In this work, propose novel trajectory and scene feature fusion technique. As trajectory in- formation is more dominant in the temporal domain while scene semantic information is more so in the spatial domain, to align these two feature types, trajectory and scene features are decoupled in temporal and spatial domains respectively and re-organized into spatial-temporal cells, which we call Trajectory-Scent Sccentror---eeeeellll Cell (TSC) features. Every cell feature consists of coordinate feature and the corresponding local scene feature. The raw trajectory and scene inputs are converted to three types of TSC features: trajectory cells, goal scene cells, and step scene cells. Based on the TSC features, we introduce a new two-stage trajectory prediction framework, the Trajectory-Scene- Network (TSC-Net). Different from directly regressing the coordinates of goal and future trajec- tory, TSC-Net completes the trajectory prediction by step-by-ster cell classification. The goal prediction network assigns a confidence score to each goal scene cell, of whether the goal is in the cell not. To obtain a more accurate position, positional offset from the cell center is also esti- mated for each cell. Similarly, the confidence and offsets are computed for every step scene cell in the trajectory completion network. The TSC features unify the trajectory feature and scene feature to the same feature space, which keep good alignment between trajectory and scene as trajectory cells and scene cells are represented in the same format the same scale. Moreover, TSC features allow unification of human- interaction and human- scene interaction modeling, where all the interactions are modeled by cell-level attentions in our framework.
The main contributions of this paper are as follows:
Based on well-reasoned analysis of requirements for integrating trajectory and scene features, we propose a novel joint representation, the imjjectory-scene-cell feature.
We introduce an innovative trajectory prediction pipeline, in which trajectory prediction is achieved via cell classification and sampling, enabled by our trajectory-scen--cell feature.
We demonstrate our approach outperforms most of existing methods in two datasets.
RELATED WORK
2.1 SEQURNCE MODELING
Sequence modeling is a fundamental component of the trajectory prediction task. To encode tra- jectory histories, Recurrent Neural Networks (RNN) based models /Zhan et al.) (2019): Kosaraju et al. (2019): Bartoli et al. (2018) and transformers with attention Bae et al. (2024); Hu & Cham (2022); ; Pang et al. (2021 1) are most commonly used. Some other methods directly concatenate his- tory coordinates and employ CNN or Multi-Layer Perceptron (MLP) as the encoder Mangalam et al. (202T: 2020), which is efficient but hard to generalize to different trajectory lengths. Beside the temporal encoder, the decoder and prediction parts also have different designs. Step-by-step pre- diction can be achieved by an augo-regressive structure, usually realized by RNN]Alahi et al. (2016); (201))
Zhang et al. (2019): Kosaraju al. (2019) and attention Yu et al. (2020): Hu & Cham (022)) In
2

---

Published as a conference paper at ICLR 2025
Gu et al. 2022), all future steps predicted simultaneously, while diffusion model Ho et al. (2020) can be employed to refine the future trajectory iteratively. Recently, Hierarchical prediction achieves state-of-theane art performance
many datasets. In these methods, prediction of the future trajectory is in two stages Mangalam et al. (2000) Yue et al. (2022), where the final step (goal) is first predicted and used as the guidance when predicting the intermediate steps. In Mangalam al. (022]), the future trajectory is further divided into goals, waypoints, and paths, to allow for longer term prediction. Liang et al. (2019) provided a gridding mechanism for rough goal prediction as an auxiliary loss over conventional RNN framework. Different from Liang et al. (2019), our TSC-Net predicts precise goal and trajectory purely on cell classification mechanism.
2.2 HUMAN- INTERACTION
Future trajectory is determined by self-intention and social interaction, with the latter requiring human-human relation modeling. The distance between two pedestrians is used as relation in Gupta et al. (2018), pooling all relations from neighbor set to feature vector. A social physics function can also be employed Yue et al. (2022), where the state of every pair of pedestrians is sent to a module to compute repulsion. A higher-order graph is introduced in Kim al. (2024) to capture collision-awan relations of pedestrians. The success of the transformer Vaswani et al. (2017) led to attention becoming the most popular way to model relations among pedestrians. In Zhang et al. (2019), sncention-baeed human-human interaction modeling is integrated into RNN-based trajectory prediction. In contrast, some other works employed attention model nomannhumas interaction (spatial relation) and history encoding (temporal relation) simultaneously. In Hu & Cham (2022). a spatial transformer and a temporal transformer are applied sequentially to trajectory history, while other structures such parallel spatial and temporal transformers are explored in Yu et al. (2020).
2.3 HUMAN-SCENE INTERACTION
None of the models can accurately predict the intention of a specific person. However, models can be applied to learn the conditional distribution of goals for multiple people; here, scene semantic information is critical, especially for longer-term prediction. A backbone network for global scene feature embedding is employed in Sadeghian al. (2019), while Dendorfer et al. (222T) took the cropped global scene as input which covers all steps of trajectory. Global scene features and trajec- tory features cannot be aligned well by direct concatenation, as will be discussed in Section 3.2 In Liang et al. (2019). local scene features surrounding every frame of the trajectory are extracted. Lo- cal scene Teature is useful for short-term obstacle avoidance, yet cannot provide clues for long-term goal prediction. Heatmap-based methods have been proposed Mangalam et al. (202T); Cao et al. (2020), where trajectory coordinates are converted to
heatmaps before concatenated to global scene features. Similarly. [ee et al.] (202)) applies heatmaps with coarse-to-fine struc- ture. [Liang et al. (2020b) applied discrete heatmaps that recorded the trajectory with a grid. Different from commonly used Euclidean space, Wong et al. (2024) models the interactions in angle-based space. Although heatmap- based methods can learn good correspondence between the trajectory and scene, they cannot model human- -uman- interactions because of the high dimensionality.
2.4 MULTI-MODALITY PREDICTION
As the indeterminacy of person's 's intention is undeniable, many stochastic prediction approaches have been proposed to capture the multi-modality of future trajectories especially the goals. Huang et al. (2019): Yu et al. (2020) directly added noise to the hidden states of the framework to obtain uncertainty, where the best output among multiple samples is selected compute loss. Instead of direct noise injection, several generative models are applied on this task. Sun et al. (2020);
Zhao al. (2019) employ generative adversarial networks (GANs) Goodfellow et al. (2014) to model the indeterminacy with a noise variable. Some other methods Chen et al. 2021b) Ivanovic & Pavone (2019); Tang & Salakhutdinov (2019); Yue et al. (2022): Bhattacharyya et al. (2019)
combined Variational Auto-Encoder (VAE) or conditional VAE (CVAE) Sohn al. (2015) in their trajectory prediction pipeline. Gu et al. (2022) applied a diffusion model et al. (2020) to generate future positions from noise variables, which achieved good performance on large dataset but has high computational cost. Lee et al. (2017) sampled future trajectories from latent space, which is optimized by a inverse reinforcement learning module.
3

---

Published as a conference paper at ICLR 2025
3 METHOD
3.1 PROBLEM DEFINITION
In this task, there are types of information: trajectories represented by coordinates, and the scene represented by RGB images or semantic segmentation maps. The vidco clip has T frames and N pedestrians (N varies for different video clip samples). The trajectory of the 1th pedestrian is represented by T 2D coordinates and denoted by Pt, (Zi,,, Vs,e), where i = 1..... N and t t= I.. The time invariant scene in the video clip is denoted by I. where I E RC Note that the trajectory and scene need to described in the same coordinate system, e.g. ; the scenes are RGB images represented in pixel space, and the trajectory coordinates are also in pixels. Given the scene Z and observed trajectories of every pedestrian p p(án) this task is to predict the future trajectories of all pedestrians p plout) = (P..)=1NN,,.711117
T.
where S
3.2 MULTI-MODEL FEATURE ALIGNMENT
Here, we explore and analyze the aligmment of coordinate- based trajectory features and image- scene features. To combine two different types of features, intuitive way is concatenating both types of feature embedding [Sadeghian et al. (2019); Liang et al. (2019). Usually, separate feature extraction networks employed for the feature types,
(c0)
(p. (in) (0
7;
where f(-) and g(-) are feature extraction networks for trajectory coordinates and the scene image respectively. The dimensions in z((²) provide information the pedestrians' locations. Different dimensions in z!-e) encode semantic information in different areas of the scene, such as the presence of obstacles or sidewalks. To build the relation between pedestrians and scene, the network needs
to find specific dimensions of (ec) based on the content of x; (co)_ However, for different steps or different pedestrians, the relations between dimensions inside z, are not fixed. This requires the network to have dynamic weights for different dimensions inside a feature vector, which is difficult for almost all operations, such convolution, MLP, and attention. Thus, directly concatenating the two types of feature cannot align trajectory and scene information well.
Instead of concatenation, was proposed in Mangalam al. (2021) to convert coordinates of trajectory to heatmaps, where
[sc) = [h (P.1), ... (p..),
2
(2)
In Eq. Eq.equation 2 h(.) transforms coordinates to heatmaps with size of H W, where a Gaus- sian peak is located at the input coordinates. Therefore, after concatenating the scene maps, the size is z, E HXC At each spatial location, for the feature Z,j. where j 1, 1....; H x W, the content to conveyed is more straightforward than feature embedding concatenation. More (co) should capture the presence or absence of a pedestrian at position j. while z/ encodes semantic information at the same location. With this structure, the relation between the two feature types is static in different dimensions and can be easily manipulated by simple opera- tions such as convolution, dynamic dimension selection is unnecessary. Therefore, the key
specifically, Zi.j
good feature alignment is that z (co) and z; (ee) should have the same scale and fixed relation. While heatmaps achieve good feature alignment, a pedestrian in Mangalam et al. (2021) is represented by tensor, which makes social interactions among different pedestrians much harder to compute. Hence, we propose the TSC feature representation, which is able to model social interactions, as well as keep good feature alignment.
3.3 TRAJECTORY-SCENT-
FEATURES
To achieve good feature alignment, the receptive fields of the two feature types should be small enough and matched each other. However, trajectory mainly spans the temporal domain while the scene map spans the spatial domain, To align these two feature types, we propose the trajectory- scene-cell (TSC) feature representation, where both the trajectory and scene mapped unified
4

---

Published as a conference paper at ICLR 2025
Trajectry
traj
f LguNS
7
Scene gleeat)
Figure 1: Illustration of the cell feature embedding. @ denotes the concatenation. The raw trajectory and scene inputs are converted to three types of TSC feature in the feature embedding stage.
cell feature space. More specifically, the trajectory is decoupled in temporal domain and converted to a sequence of trajectory cell feature z by combining corresponding scene information to each trajectory step. The scene is decoupled to spatial grid and converted to goal scene cell feature (groat) and step scene cell feature (step) by integrating center coordinate information each position in the grid, which is illustrated in Figure
3.3. TRAJECTORY CELL FEATURE.
To compute z, the trajectory is decoupled into separate temporal cells 2 for which each cell only represents the coordinates of one time step, i.e.,
4 = (pi.r.Pr.t. Pree),
(co)
(3)
where the coordinate feature embedding network f(-) is an MLP. It takes in three components of: 2D reference coordinates Pi,r. 31D shifted coordinates Pi,t: and 2D velocity Åi... The reference position Pi.r is the most recent observed frame at time while the shifted coordinates Pt,: is the relative position compared to the reference position, together with the time difference in a third dimension:
Pi,t [p..t Pi.r, r)-
(4)
The velocity P., is computed as the difference in coordinates between two neighboring frames, i.e.
Di. Pa,t Pi,-1 t 1 (p,e+1 Pi.s t=
(5)
For each (co) fit scene feature i,, u X u. It computed as
(sc) is assigned, which comes from an area centered Pi, with size
(uc) Ziet
(troos) (In,.- How..- #. -
(6)
where (.) CNN-based scene feature extraction network. Therefore, the pedestrian cell
feature 2 computed by concatenation of two feature types ,,,
Zi, Fiit
(co) (ec)
E RRD, where D
is the embedded feature dimension. Then the feature of observation becomes (ii)) DxNN7
3.3.2 GOAL SCENE CELL FEATURE.
To map the scene into the same feature space as a trajectory cell feature z, the scene is decoupled into types of scene cell features, a (goal) and a (step), To obtain (goal)) square area centered on the last observation position Pt.r with the size of LR x LR is cropped (and padded if necessary), followed by CNN (.)o maps the area to I. x I cells, where I, is the number of goal cells in one axis of the grid and the size of one cell. For each cell, the center coordinate feature assigned. Hence, the goal scene cell feature is computed by concatenating the coordinate features and scene feature:
(goal) = =[1 =[I(Or. (PP.ns Ris.s T (...k)) (gl.)))))gg (I(20al) F.i.k
5

---

Published as a conference paper at ICLR 2025
Step dool
alanel)
Qucry
Goal Encoder
CVAE
a
Decoder
Valor
Bayytues
Enoode
[scene map!
TSC Feature
History Encoder
TSC Feature Embedding
Pr
Ptr
Embeddlin
7
G+
Key Valac
d;
pdap)
Duary
Step
Step
pesp)
Updatte
Stop Step Traj-Net
Figure 2: Framework of TSC-Net. 1) Compute TSC features (illustrated in Figure T and encode features in historical frames, where the output is used in the next two steps. 2) Predict goals form OVAl,CVAE, the goal prediction network with CVAE structure, where the output used in trajectory completion as guidance. 3) Predict next step position with Traj-Net, the trajectory completion network. Step and 3 are applied eyclically until all steps are predicted. @D: concatenation.
where
=
[(
4 2
L
Bigi.
R.
j
R
2
(8)
I(goal)
Mr.
Me. ...
LB S0,,
LR
In this way, each pedestrian has unique but global scene representation a (goal) E DXNN1?
3.3.3 STEP SCENE CELL FEATURE.
Similarly, step scene cell features a (step) are formed by:
(step) J.J.
(Piir: Qif.jj Pi,t Pi,r, T, (...j..))) (step) (step)
jik
where
Gitt,ik ( (
1 1-1-1 r, (
1-1
(10)
(step) =I Wi.t 4 wi. 4f = 4
g (step) (.) maps the ir x area to x cells, where is the number of step cells in one axis of the grid and is the size of one cell. Step scene cells a (step) E Dx x xr are built for all time steps,
where the cropped scene has smaller size than goal scene and centered on the position of each time step. To simplify the implementation, we set = Ir.
TSC features can be treated positional encoding Xu et al. (202T); Dosovitskiy et al. (2020). The TSC feature embedding module provides positional encoding for every local scene and "scene encoding" for every position of the trajectory. Both trajectories and scenes converted into this format. Therefore, the information inside trajectory cell features and scene cell features are matched with each other and the relation among feature dimensions become static.
local scene feature concatenated with a learnable spatial-temporal
3.4 PREDICTING TRAJECTORY BY CELI CLASSIFICATION
TSC features enable both human-human and human-scene interactions to be modeled by cell-level attention operations. With help of TSC features, the trajectory prediction task becomes that of learning mapping from (in), a (goal), a (step)) to p poot() Our framework follows previous works Mangalam et al. (202T) in splitting the prediction into two stages of goal estimation and trajectory completion (Figure 2). Instead of auto-regressie regressive approaches, in method, the goal and trajectory predictions are completed by estimating the likelihood of each cell containing the goal or the next step. In other words, the trajectory prediction lask is reformulated classification problem.

---

Published as a conference paper at ICLR 2025
3.4.1 GOAL PREDICTION NETWORK.
In the goal prediction network, the history trajectory feature 21.r is first sent to the history encoder, which consists of K self-attention layers over spatial and temporal domains. Both human-human interactions and temporal relations within each trajectory are captured by the history encoder. The goal encoder has K self-attention layers for a a and also K cross-attention layers for attention between (goat) and the output of the history encoder 7i:7, where { the query since the target is to predict the presence of the goal in each goal scene cell. Note that self-attentions and cross-attentions in the goal encoder are only computed with the same pedestrian, as human-hume interaction information is already captured in the history encoder. A CVAE structure Kingma & Welling (203)) is applied for multi-modality goal prediction, where the ground truth embedding, CVAE encoder, and CVAE decoder are all cell-level MLPs. The condition of CVAR is the output of goal encoder ā After combining the condition and latent variable c, CVAE estimates the (goal) goal output o (gvaa)) E g3xNXxL For each cell, Oij,k E g includes the I-dimension confidence of whether the goal is located in the cell and the 2D offset between the goal and cell center.
3.4.2 TRAJECTORY COMPLETION NETWORK.
After sampling the goals pr, TSC features are computed, which represent the guidance when predicting the trajectory. The trajectory completion network shares the same encoder-decoder struc- ture except CVAE. When predicting the trajectory in frame t ++ 1, the step encoder first computes the self-attention for a,
(step) and then computes the cross- attenttntnnoon over a, ((step) and 2 with the query
for cross-attention being at (step) The output of the step encoder at ((step) is sent to the step decoder, a cell-level MLP, to predict the step output of frame t + 1: 0,41 includes one confidence score and offsets. The position of next step Pt+1 1 is determined by picking the cell with highest confidence and adding the corresponding offsets to the cell center. Given Pt+1, I/slep) is cropped, ai+1 and history encoder.
(step) 3 Here ((step) also
(step) computed, and 21,++1, E14+1 updated by the feature embedding module
The loss function of TSC-Net includes the CVAE loss from goal network and L2 reconstruction loss from trajectory completion network:
DKL (g(e)o (goal), a (00ol)) (c)) + A/ (goat)
o (goal) lle a (stpp)
(step) ||2,
(11)
where À and a are weights for different parts, e is the latent variables of CVAE, and DKL (xI(-Il.) is the Dallback-Leible divergence of two distributions.
3.5 HYBRID GOAL SAMPLING
In our model, CVAE is employed to generate multiple trajectories. However, different from previous methods, the goal CVAE network in model naturally have more sampling solutions the output of CVAE are cells with confidence scores. In the inference stage. for each pedestrian, CVAE first generates m LxL grids. Secondly, for each grid, n cells with highest confidence score are picked. Hence, mn predicted goals are obtained from this two-step sampling.
In practical, there are multiple (m, n) combinations, where following widely used setting mn 20. We find that larger m and smaller is likely to result in goals with small variance, which will achieve better performance for short or straight trajectories. Conversely, smaller n and larger m tends to larger variance, which is helpful to longer complex trajectories. Hence hybrid sampling strategy proposed to find the most appropriate (m, n.) combination for each pedestrian.
During training, hyper classifier is trained separately, where input is the Pi,1:r for pedestrian, and category label is the (m,n) combination index that achieves the best performance. the inference stage, pedestrian specific (m, n) combination is obtained from the classifier before run the TSC- Net. The hyper classifier can be treated as a model selection module that determines the hyper- parameters m and for each pedestrian. The hyper classifier is built by MLP and easy to train since the relation between trajectory shape and best (m, n) combination is stable and general over training and testing sets.
7

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
4.1 DATASETS AND METRICS
Stanford Drone Dataset (SDD) Robicquet et al. (016) is a large-scale benchmark which contains more than 11,000 pedestrians with 20 different scenes. The dataset is pre-processed for the trajectory prediction following NSP-SFM Yue et al. (2022). Intersection Drone Dataset (InD) Bock et al. 2020) contains about 10,000 pedestrians in 4 different road intersection scenes. ETH-UCY dataset Lerner et al. (2007); Pellegrini et al. (2009) includes 5 subsets, where the evaluation follows the leave-one-out validation strategy over 5 subsets. The scene semantic maps in this dataset have category labels: traversable area and non-traversable area.
Prediction Settings in the experiments include short-term prediction and long-term prediction. Most of previous works focus on the short-term setting with 7=20 and 7=8, where the source videos are down-sampled to 2.5 fps. The short-term setting is applied in the experiments and ETH-UCY datasets. Following Mangalam et al. (2021), we apply the long-term setting 7-33 and 5-5 with I fps frame rate, which is applied in the experiment on SDD and inD.
SDD
Metrics used in the experiments include Average Displacement Error (ADE) and Final Displace- ment Error (FDE) are measured Allai et al. (2014). where ADE is the average distance between prediction and ground truth over all frames and FDE is the distance between prediction and ground truth of the last frame. With multi- prediction, minimum ADE and FDE over 20 sampled trajectories for each person are presented.
4.2 ABLATION STUDY
We investigated several alternate selections for different parts of our model in SDD, where results shown in Table First, multiple sizes of the goal scene cells were evaluated, which is denoted as different Lx L. 1-15 outperforms both smaller and larger L values. Smaller L means the network has to find correct offsets in a larger area, while larger L means the network has to classify the correct cell among more cells. L=15 balances the number of cells and the area of each cell.
Different two-step sampling hyper-parameters were also evaluated. There are different (m, n) combinations that get 20 samples. The selection of m and n determined the range and precision of the goal sampling. Usually short and straight observed trajectories achieved better prediction performance with larger m, where more goals can be sampled in the same cell. In contrast, long and irregular observed trajectories achieved better prediction performance with larger 71, which force the sampled goals to be distributed in a larger area. Since hybrid sampling tries to find the most suitable (m, n) combination for each pedestrian, it achieves the least ADE and FDE compared to all fixed (m, n) combinations. (10, 2) has the second best ADE and FDE, which is a balance between the precision and range over all trajectories.
Table 1: Ablation study results in SDD dataset.
Input
Scene
-Pi.
Piir
Pr.1
Full
Full
Full
LXLL
15
5x5
25
15
(m.n)
Hyhrid
(20,1)
Hyhrid
(20,1)
(10,2)
(5.4)
(4.5)
(2,10)
(1.20)
Hybrid
Query
Scene
Traj
Scene
ADE
8.33
8.31
7.10
15.28
8.77
7.49
9.70
8.35
7.18
7.49
7.66
8.43
9.63
19.58
6.44
FDE
13.93
13.99
11.34
14.66
14.70
11.92
17.01
14:53
11.78
12.64
13.10
14.85
17.45
27.20
9.97
Moreover, to demonstrate the effectiveness of prediction by cell classification (denoted as "Scene"), we designed an alternate goal encoder and step encoder with a common auto-regressive structure. For the alternate encoders, the last frame trajectory cell was sent as query and the decoders were trained to directly regress the goal (or next frame) coordinates, which is denoted as "Traj".. The results show cell classification significantly outperforms the auto-regressive version. We believe this is because cell classification is easier to learn with the format of TSC features. In cell classification, one scene cell is responsible for small area. However, when directly predicting the next step position, the network has regress too many different possible areas based on the last pedestrian
8

---

Published as a conference paper at ICLR 2025
cell, which is more difficult. At last, different input components are compared by excluding one input feature type at one time (e.g.. '-Scnn' indicates the full input types without scene). It can be observed that the velocity Pi., plays the most crucial role during prediction.
4.3 COMPARISON WITH STATE-OF-THE-ART METHODS
Here, our method is quantitatively compared with a wide range of existing methods, with results shown in Table 2 For SDD, our TSC-Net outperforms SOTA methods by 0.64 on FDE and 0.08 on ADE. Note that heatmap based methods NSP-SFM and Y-Net Man- galam al. (2021) requires 10,000 samples in testing for every pedestrian before obtain the final 20 samples by clustering, where the samples can naturally cover larger possible positions. In con- trast, our hybrid sampling strategy only needs to sample more goals in the training stage learn the hyper-classifier Our performance also significantly surpasses scene feature embedding based methods, such as SoPhie (Sadeghian al. (2019) and MG-GAN Dendorfer Dendoroen al. (202I).
Yue et al. (2022)
Results from ETH-UCY dataset are also presented. The scene semantic maps in ETH-UCY dataset are only binary labels: traversable and untraversable and usually the whole scene is traversable. This meant that there is little relationship between the scene and trajectory features to be captured by TSC features, thus, in this scenario our TSC features degenerate to normal trajectory feature embeddings and the performance is limited by inadequate scene maps. From the results it can be seen that TSC-Net achieves the third lowest FDE on the average of all subsets except for the two methods using 10,000 samples. The results demonstrate that the cell classification manner has comparable performance even with normal feature embedding only.
Table 2: Comparison between method and existing methods on ETH-UCY dataset and SDD dataset. "ADE/FDE" are reported. * means the results come from clustering 10,000 samples, where all other methods require 20 samples only. Bold Best, Underline:Second best.
Methods
HOTEL
UNIV
ZARA1
ZARA2
Average
Y-Net Mangulam al. al.
2021
0,28/0.33
0,100..14
0,24X0.41
0,17X1,27
0,13/0.22
0.18/1,27
7,85 /1 111.85
NSP.SFM Yue
0.25/0.24
0.09/0.13
0.21/0.38
0.16/0.27
0.12/0.20
0.1 0.1770
6,52 /10.61
Social-GANI Gupta et al
2018
0.81/1.52
0.72/1.61 11
0.60/1 1.2
0.34/0.69
0.42/0.84 84
0.58/1.18
27.23/4 1.44
SoPhie Sadeghian al.
2019
0,70/1.43
0.76/1.67
0.54/1 .24
0.30/0.63
0.38/0.78
0,54/1.15
16. 16.27/29.38
Trajectro+++ Salzmann
2020
0.39/0.83
0.12/0.21
0.20/0.44
0.15/1.33
0.11/0.25
0.19A1.41
8,98
MG-GAN Dendorfer et al 2021
0.47/0.91
0.1 1/.4..2
0.54/1 1.54/1.07
0. 0.36/0.73
0.29/0.00 60
0.36/0.71
13 13.60/25.80 2.080
PCCSNET Sun
202
0,2840.54
0.11/1.19
0,29X0.60
0.21/1.44
0.15/0.34
0,21/1.42
8.62 /16.16
2022
0.39/0.66
0.13/0
0.22/0.45
7/0.30
0/13/0227
1.2
7.61 /14.30
Flow CowCChin[MMade Maeda & Ukita
2023
0.55/0.99
0,200033
0.29/0.54
0,22/U.40
0,2000.4
0,29XU.52
9.93/17.17
SICNe Dong ai. 2023
0,27A1.45
0.11
0,26/1.46
0.19/1.33
0,13/0.26
0.19X1.33
8. 8.44/13.65
TUTR
Shi et
(0123)
0.40/0.61
0.11/0.
0.23/0.42
1//0.4
0.13/0.25
1.2
7.76/12.69
V2-Nei Wong et all.
2002
0.23/0.37
0.11/U.16
0.21/0.35
0.19X133
0.14/0.24
0.18/1.28
7.12/11.39
LMTraj-SUP Bae 20244)
0.41/0.51
0.1
0.22/0.34
0.20/0.32
0.17/0.27
0.22/0.32
7. 7.80/10.10
E WWon et al.
2024
0.25/0.38
0.12/0.14
0.20/0.34
0.18/0.29
0.13/0.22
0.17/0.27
6.54/10.36
TSC-Net (Ours)
0.32/0.39
0.12/0.19
0.25/0.46
0,17/0,30
0.15/0.26
0,20X1.32
6.44/9.97
4.4 LONG- PREDICTION
In this section we evaluate our TSC-Net in long-term prediction setting. Our method achieves best FDE on Both SDD and InD and comparable ADE. With longer prediction range, the size of cell becomes larger, where less scene details are summarized to the representation. Thus, ADE of method is slightly larger than heatmap based method Y-Net. SocialCirele WWogg et al. (2024) achieves good performance on ETH-UCY dataset, however, it becomes difficult to precisely collect- ing information with aagle-based partition.
1,000 sampled goals of Y-Net and our method on SDD are visualized in Figure 3 In our method, goals obtained by (m,n) n) (50, 20). In Y-Net, 999 goals sampled from the estimated heatmap besides the soft-argmax sample. It can be observed that our method provides a goal dis- tribution covering more possibilities that accounts for sudden changes of velocity, e.g. in Figure

---

Published as a conference paper at ICLR 2025
Figure 3: Visualization of trajectory prediction results comparison between Y-Net and our method on SDD dataset with long-term setting. The first row show results from Y-Net, while the second row show results from our method. Blue and magenta curves: history and future of ground truth trajectory. Green dots: predicted goals.
sudden stops (first column) and sudden moves (second column), while Y-Net tends to predict the goals by extrapolating the observed trajectory along the map with constant speed.
Table Prediction results on SDD dataset and InD dataset with long-term prediction setting, where "ADE/FDE" are reported. +Results are reproduced and reported by Y-Net Mangalam et al. (2021) twe re-train the model with long-term setting based on released code. The results come from clustering 10,000 samples, where all other methods require 20 samples only.
Methods
Social-GANf Gupta al. 2018
PECNet
R-PECNett
Y-Ner"
E-V--Net-SC's Wong et 2024
TSC-Net (Ours)
Mangalam et 2020
Mangalam et 2021
SDD
155.32/307.88 88
72.22/1
261.27/750.45
47.94/66.7 71
52, 32.1699..8
51. 31.39/3..77
InD
38.57/84.61
20.25/32.95
341.8/1702.64
14.99/21.13
16.23/23.56
17.15/19.53
4.5 LIMITATIONS
Although TSC-Net has various advantages over both heatmap based and scene embedding based methods, and achieves promising performance, the predicted offsets are usually biased to the center of the cell, which makes the sampled goals hard to cover the edge area of cells. It may caused by non-ignorable amount of pedestrians without any movements in the dataset, as they share the same zero-offset. One possible solution is estimating whether pedestrian will remain static move, and sampling the goal separately in two situations. We leave to future work as integrating such a module is non-trivial.
CONCLUSIONS
In this work, novel TSC feature representation was proposed to represent the trajectory and scene in a unified feature space. TSC feature avoids the dynamic weight learning in scene embedding based methods and the high dimensionality issue in heatmap based methods. Both numan-humai and scene interactions can be modeled by cell-level attentions over TSC feature. With TSC features, an innovative framework TSC-Net was introduced, where future positions are predicted by cell classification and offset regression. TSC-Net was evaluated in several experiments and out- performed all compare methods on most of the metrics. Moreover, qualitative results demonstrated TSC-Net generates more reasonable distributions according to the training samples. Source code of the TSC-Net is released at @ggihubcco//uboo//SSSCC
ACKNOWLEDGMENTS
This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, well cash and in-kind contribution from Singapore Telecommunications Limited (Singtel), through Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alexandre Alahi, Vignesh Ramanathan, and Li Fei-Fei. Socially-aware large-scale crowd forecast- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2203-2210, 2014.
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 961-971, 2016.
Inhwan Bae, Junoh Lee, and Hae-Gon Jeon. Can language beat numerical regression? language- based multimodal trajectory prediction. In Proceedings of the IEEE/CVE Conference on Com- puter Vision and Pattern Recognition, pp. 753-766, 2024.
Federico Bartoli, Giuseppe Lisanti, Lamberto Ballan, and Alberto Del Bimbo. Context-aware tra- jectory prediction. In 2018 24th international conference on pattern recognition (ICPR), pp- 1941-1946. IEEE, 2018.
Maren Bennewitz, Wolfram Burgard, and Sebastian Thrun. Learning motion patterns of persons for mobile service robots. In Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), volume 4, pp. 3601-3606. IEEE, 2002.
Apratim Bhattacharyya, Michael Hanselmann. Mario Fritz, Bernt Schiele, and Christoph-Nikolt Strachle. Conditional flow variational autoencoders for structured sequence prediction. In 4th Workshop on Bayesian Deep Learning. bayesiandeepleamiing org, 2019.
Julian Bock, Robert Krajewski, Tobias Moers, Steffen Runde, Lennart Vater, and Lutz Eckstein.
The ind dataset: A drone dataset of naturalistic road trajectories german intersections. In 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 1929-19344 2020. doi: 10. 109/V444002 2020.9304839.
The Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long-term human motion prediction with scene context. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pp. 387-404. Springer, 2020.
Guangyi Chen, Junlong Li, Jiwen Lu, and Jie Zhou. Human trajectory prediction via counterfactual analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9824 9833, 2021a.
Guangyi Chen, Junlong Li, Nuoxing Zhou, Liangliang Ren, and Jiwen Lu. Personalized trajec- tory prediction via distribution discrimination. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15580-15589, 2021b.
Patrick Dendorfer, Sven Elflein, and Laura Leal-Taixé. Mg-gan: A multi-generator model preventing out-of-distribution samples in pedestrian trajectory prediction. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 13158-13167, 2021.
Nachiket Deo and Mohan M Trivedi. Trajectory forecasts in unknown environments conditioned grid-based plans. arXiv preprint arXiv:2001.00735, 2020.
Yonghao Dong, Le Wang, Sanping Zhou, and Gang Hua. Sparse instance conditioned multimodal trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9763-9772. 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2020.
Liangji Fang, Qinhong Jiang, Jianping Shi, and Bolei Zhou. Tpnet: Trajectory proposal network for motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6797-6806, 2020.
11

---

Published as a conference paper at ICLR 2025
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farte Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 17113-17122, 2022.
Mahir Gulzar, Yar Muhammad, and Naveed Muhammad. A survey on motion prediction of pedes- trians and vehicles for autonomous driving, IEEE Access, 9:137957-137969, 2021.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: So- cially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2255-2264, 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Bo Hu and Tat-Jen Cham. Entry-flipped transformer for inference and prediction of participant behavior. In European Conference Computer Vision, pp. 439-456. Springer, 2022.
Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. Stgat: Modeling spatial- temporal interactions for human trajectory prediction. In Proceedings of the IEEE/CVF interna- lional conference computer vision, pp. 6272-6281, 2019.
Boris Ivanovic and Marco Pavone. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2375-2384 2019.
Sungjune Kim, Hyung-gun Chi, Hyerin Lim, Karthik Ramani, Jinkyu Kim, and Sangpil Kim. Higher-order relational reasoning for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15251-15260, 2024.
Diederik P Kingma and Max Welling. 1rXiv:1312.61 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Vineet Kosaraju, Amir Sadeghian, Roberto Martin-Martin, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-ge gan and graph attention networks. Advances Neural Information Processing Systems, 32, 2019.
Mihee Lee, Samuel S Sohn, Seonghyeon Moon, Sejong Yoon, Mubbasir Kapadia, and Vladimir Pavlovic. Muse-vae: multi-scale vae for environment-awarc long term trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
2221-2236 2022.
Namhoon Lee. Wongun Choi, Paul Vernaza, Christopher Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 336-345, 2017.
Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by example. Computer Graphics Forum, 26(3):655-666 2007. doi: https://doi.or//10.111 11111/.111111111676655999 0167-8865.2007.01000
Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander G Hauptmann, and Li Fei-Fei. Peeking into the future: Predicting future person activities and locations in videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5725-5734, 2019.
Junwei Liang, Lu Jiang, and Alexander Hauptmann. Simaug: Learning robust representations from simulation for trajectory prediction. In Computer Vision-ECCY 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pp. 275 275-292. Springer, 2020a.
12

---

Published as a conference paper at ICLR 2025
Junwei Liang, Lu Jiang. Kevin Murphy, Ting Yu, and Alexander Hauptmann. The garden of forking paths: Towards multi-future trajectory prediction. In Proceedings of the IEEE/CVE Conference Computer Viston and Pattern Recognition, pp.
2020b.
Takahiro Maeda and Norimichi Ukita. Fast inference and update of probabilistic density estimation on trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9795-9805, 2023.
Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned tra- jectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 759-776. Springer, 2020.
Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals, waypoints & paths long term human trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12232-15242, 2021.
Bo Pang, Tianyang Zhao, Xu Xie, and Ying Nian Wu. Trajectory prediction with latent belief energy- model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pat- tern Recognition, pp. 11814-11824, 2021.
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll never walk alone: Modeling social behavior for multi-target tracking. In 2009 IEEE 12th International Conference Computer Vision, pp. 261-268. IEEE, 2009.
Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social eti- Human trajectory understanding crowded scenes. Computer VCCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pp. 549-565. Springer, 2016.
quette:
Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical con- straints. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1349-1358, 2019.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: fynamically-feasible trajectory forecasting with heterogeneous data. In Computer ESCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16. pp. 683-700. Springer, 2020.
Liushuai Shi, Le Wang, Sanping Zhou, and Gang Hua. Trajectory unified transformer for pedestrian trajectory prediction. In Proceedings of the LEEE/CVF International Conference on Computer Vision, pp. 9675-9684, 2023.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Hao Sun, Zhiqun Zhao, and Zhihai He. Reciprocal learning networks for human trajectory predic- tion. In Proceedings of the IEEEICVF Conference on Computer Vision and Pattern Recognition, pp. 7416-7425, 2020.
Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu. Three steps multimodal trajectory predic- tion: Modality clustering, classification and synthesis. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 15250-13259) 2021.
Charlie Tang and Russ R Salakhutdinov. Multiple futures prediction. Advances in neural information processing systems, 32, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2025
Conghao Wong, Beihao Xia, Ziming Hong, Qinmu Peng. Wei Yuan, Qiong Cao, Yibo Yang, and Xinge You. View vertically: A hierarchical network for trajectory prediction via fourier spec- trums. In European Conference Computer Vision, pp. 682-700. Springer, 2022.
Conghao Wong, Beihao Xia, Ziqian Zou, Yulong Wang, and Xinge You. Socialcircle: Learning the angle-based social interaction representation for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19005-19015, 2024.
Rui Xu, Xintao Wang, Kai Chen, Bolei Zhou, and Chen Change Loy. Positional encoding as spatial inductive bias in gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and 2021.
Pattern Recognition, pp.
Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pp. 507-523. Springer, 2020.
Jiangbei Yue, Dinesh Manocha, and He Wang. Human trajectory prediction via neural social physics. European Conference on Computer Vision, pp. 376-394. Springer, 2022.
Pu Zhang, Wanli Ouyang, Pengfei Zhang, Jianru Xue, and Nanning Zheng. Sr-Istm: State refinement for Istm towards pedestrian trajectory prediction. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12085-12094 2019.
Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12126- 12134, 2019.
14


Reviews:
[{"review_id": "Xl6rKltIdO", "reviewer": "Reviewer_iGai", "summary": "The paper proposes a novel framework, TSC-Net, with an innovative Trajectory-Scene-Cell (TSC) feature representation designed for pedestrian trajectory prediction.  In particular, it effectively captures both individual pedestrian behavior and the contextual scene information. By decoupling trajectory data in the temporal domain and scene information in the spatial domain, the model effectively captures both human-human interactions and human-scene dynamics, leading to improved prediction accuracy compared to existing methods. The experiments conducted on various datasets seem to validate the efficacy of the proposed method.", "questions": "See the \"Weakness\" section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well written and easy to follow, with clear explanations of methodologies, results, and implications.\n\n+ It is a well-designed idea to amalgamate trajectory and scene information into a unified TSC feature space. Such a joint feature allows for better modeling of both human-human and human-scene interactions.\n\n+ The authors reformulate the trajectory prediction task as a classification problem by estimating the likelihood of each cell containing the goal or the next steps, which distinguish the proposed methods from the existing regression-based trajectory prediction.\n\n+ Extensive experimentation across multiple datasets, i.e., SDD and ETH-UCY, demonstrating the robustness and generalizability of TSC-Net. Such a clear improvement over state-of-the-art methods strongly demonstrate the model's efficacy.", "weaknesses": "- The proposed model has shown tendencies to bias its trajectory predictions towards the centers of scene cells. This could lead to inaccuracies, particularly when pedestrians navigate along edges or less central paths.\n\n- The discussion of limitations is relatively brief. If the authors can provide some vivid examples, then it should be easier for readers to better understand.\n\n- It is still not clear if the proposed method is able to sufficiently address scenarios with higher pedestrian densities or complex environments especially in urban settings with numerous dynamic agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cgNPtiHHtP", "reviewer": "Reviewer_RsFQ", "summary": "This paper addresses the scene usage limitations of prior approaches in pedestrian trajectory prediction by introducing the trajectory-scene-cell (TSC) feature and TSC-Net. The authors convert raw trajectory and scene inputs into three types of TSC features: trajectory cells, goal scene cells, and step scene cells, which are utilized for trajectory prediction. The paper reframes the trajectory prediction task as a classification problem. Furthermore, the authors claim that their method achieves state-of-the-art performance and is demonstrated better\non predicting goals for trajectories with irregular speed.", "questions": "1.\tExplain the underperformance of all four approaches in ETH-UCY and demonstrate TSC feature genuinely works better than the traditional CNN and heatmap approaches.\n2.\tA more detailed statistical table should be included for irregular speed as author claims.\n3.\tPerformance on more complex datasets, such as NBA[1]. \nOther issues can refer to the weaknesses.\n\n[1] Conghao Wong, Beihao Xia, Ziqian Zou, Yulong Wang, and Xinge You. Socialcircle: Learning the angle-based social interaction representation for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19005–19015, 2024.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe trajectory-scene-cell feature in pedestrian trajectory prediction is somewhat innovate.\n2.\tThe paper is well-written and easy to follow.", "weaknesses": "1.\tMy main concern is whether the TSC feature genuinely works better than the traditional CNN and heatmap approaches. Because in the performance comparison, TSC-Net significantly underperforms compared to Y-Net, NSP-SFM, V2-Net, and E-V2-Net. And there is no ablation study provided to demonstrate the performance of replacing the TSC feature with traditional CNN and heatmap approaches.\n2.\tThe authors claim that Y-net and NSP-SFM performs better because their results come from clustering 1000 samples. However, I only find they also use best-of-20 (K=20) protocol in their paper. Moreover, if they indeed use extra samples, the result of the TSC-Net under the same setting could be given for comparison.\n3.\tThe authors claim in abstract that TSC-Net is demonstrated better on predicting goals for trajectories with irregular speed. However, Figure 3 only provides a few examples for demonstration, which is insufficient to illustrate the overall situation. A more detailed statistical table should be included.\n4.\tThe ETH-UCY and SDD datasets are relatively small and consist of simple trajectories. The performance of TSC-Net on more complex datasets, such as NBA[1], would provide a clearer understanding of its capabilities. \n[1] Conghao Wong, Beihao Xia, Ziqian Zou, Yulong Wang, and Xinge You. Socialcircle: Learning the angle-based social interaction representation for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19005–19015, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "icw3HziT4t", "reviewer": "Reviewer_RVzK", "summary": "In this study, the trajectory-scene-cell feature is introduced to integrate both trajectories and scenes into a unified feature space. By decoupling the trajectory in the temporal domain and the scene in the spatial domain, the trajectory and scene features are reorganized into various cell types that effectively align both aspects. This approach facilitates the modeling of interactions between humans as well as interactions between humans and their environments. Additionally, The TSC-Net is introduced, utilizing an innovative trajectory prediction approach that predicts both the goal and intermediate trajectory positions through cell classification and offset regression. Experiments conducted on two datasets demonstrate that TSC-Net improves the accuracy of pedestrian trajectory prediction.", "questions": "1. In Formula 11, what is the prior distribution of $\\varepsilon$, and does $p(\\varepsilon)$ represent the standardized normal distribution?\n\n2. How is $Z_T$ used for guidance ,which is mentioned in section 3.4.2?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.This paper proposes a novel joint representation called the trajectory-scene-cell feature and introduces an innovative trajectory prediction pipeline.\n\n2.The structure of the paper is comprehensive, and the writing is clear. Compared to the baselines, it demonstrates commendable results on several public datasets.", "weaknesses": "1.The layout design of Figure 2 is unappealing and does not effectively highlight the core innovations of this research. It is recommended to make the various steps of the framework more explicit in the figure, such as the model's inputs and outputs.\n\n2.The paper is quite challenging to read due to the confusing formula symbols, such as the Formula 7 and Formula 8, what does $q$ represent for and what is the meaning of $i$, $j$ and $k$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VpNmDe4la0", "reviewer": "Reviewer_fgC6", "summary": "The authors emphasize the importance of utilizing scene information for trajectory prediction and propose a method to do so. \nAccording to the authors, existing approaches have limitations: some methods use CNNs to convert scenes into feature vectors, leading to misalignment issues, while others employ heatmaps to represent trajectories as image features, neglecting pedestrian interactions. \nThis paper introduces \"trajectory-scene-cell feature\" to represent both the trajectory and the scene with the same representation. \nBy integrating these two types of information into a single representation, the method enables human-human and human-scene interactions. \nTo decode future trajectory from this representation, a Trajectory-Scene-Cell Network is proposed. \nIt achieves state-of-the-art prediction performance on some datasets and some metrics.", "questions": "* The ETH-UCY dataset also includes RGB images of the scene. Instead of using binary label semantics, why not use the RGB images? It’s unclear why RGB information was used for the SDD dataset but not for ETH-UCY.\n* In equations 7 and 9, is the network 𝑓 the same as the 𝑓 in equation 3? If they are different, the notation should be distinguished. If they are the same, how can they be identical when the inputs in equations 3 and 7 are different?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Previous pedestrian trajectory prediction papers tend not to emphasize the importance of effectively utilizing scene information, but this paper points out this gap.\n* The paper is well-structured and easy to follow.\n* The proposed method is intuitively understandable, technically well-implemented, and achieves comparable prediction performance.", "weaknesses": "* The referenced trajectory prediction papers focus heavily on older works. It would be beneficial to include more recent papers.\n* If the authors claim that the ETH-UCY dataset only contains binary-level semantics, why not experiment with vehicle trajectory datasets instead? Vehicle datasets offer richer semantic scene information, such as lanes, road edges, and traffic lights, so it seems more convincing to conduct experiments on vehicle datasets for claiming effective use of scene information. Moreover, numerous methods already proposed how to effectively utilize these scene information:\n    - HiVT (CVPR, 2022): It proposes an integrated hierarchical transformer structure using both scene and trajectory vectors.\n    - Leveraging Future Relationship Reasoning for Vehicle Trajectory Prediction (ICLR, 2023): It models agent interactions using vectorized scene information.\n    - Apart from these vectorized methods, many models also use image features. Demonstrating state-of-the-art performance in vehicle scenarios by leveraging scene information would be more convincing.\n* The CVAE-based trajectory decoder for goal prediction does not seem new. The sequential prediction of goals and trajectories has been widely explored. For instance, Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals (CoRL, 2022) presents a more sophisticated approach with predictions conditioned on traversals.\n* The ablation study sounds insufficient. It is hard to verify the effectiveness of the proposed TSC. There is only one ablation experiments, but it is hard to see where the performance gain came from (from the proposed methods or finetuning hyperprameters). For example, it is necessary to observe how prediction performance changes when certain features ($z_{i,t}^{(sc)}$ or $a_{i,t,j,k}^{(step)}$) are excluded.\n* There are other methods with better performance on the ETH-UCY dataset, making it difficult to claim state-of-the-art performance based on average metrics.\n    - View Vertically (ECCV, 2022): 0.18/0.28", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)