Forecasting Human Trajectory from Scene History
Authors: Mancheng Meng
Ziyan Wu
Terrence Chen
Xiran Cai
Xiang Sean Zhou
Fan Yang
Dinggang Shen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Human trajectory prediction
Scene history
Group trajectory
Cross-modal interaction

Abstract: Predicting the future trajectory of a person remains a challenging problem, due to randomness and subjectivity. However, the moving patterns of human in constrained scenario typically conform to a limited number of regularities to a certain extent, because of the scenario restrictions (\eg, floor plan, roads and obstacles) and person-person or person-object interactivity. Thus, an individual person in this scenario should follow one of the regularities as well. In other words, a person's subsequent trajectory has likely been traveled by others. Based on this hypothesis, we propose to forecast a person's future trajectory by learning from the implicit scene regularities. We call the regularities, inherently derived from the past dynamics of the people and the environment in the scene,  \emph{scene history}. We categorize scene history information into two types: historical group trajectories and individual-surroundings interaction. To exploit these information for trajectory prediction, we propose a novel framework Scene History Excavating Network (SHENet), where the scene history is leveraged in a simple yet effective approach. In particular, we design two components, the group trajectory bank module to extract representative group trajectories as the candidate for future path, and the cross-modal interaction module to model the interaction between individual past trajectory and its surroundings for trajectory refinement, respectively.  In addition, to mitigate the uncertainty in the evaluation, caused by the aforementioned randomness and subjectivity, we propose to include smoothness into evaluation metrics. We conduct extensive evaluations to validate the efficacy of proposed framework on ETH, UCY, as well as a new, challenging benchmark dataset PAV, demonstrating superior performance compared to state-of-the-art methods.

Forecasting Human Trajectory from Scene History
Mancheng Meng
Ziyan Wu²
Terrence Chen² Xiran Cai
Xiang Sean Zhou? Fan Yang²- Dinggang Shen 1.2
ShanghaiTech University (mengmch, caixr, dgehen)@Shanghaait @shanghaitech edu, en {ziyan. wu, terrence. chen, sean zhou, fan, yango3)@uii-ai. com
United Imaging Intelligence
Abstract
Predicting the future trajectory of person remains challenging problem, due randomness and subjectivity of human movement. However, the moving patterns of human in constrained scenario typically conform limited number of regularities to a certain extent, because of the scenario restrictions (e.g., floor plan, roads, and obstacles) and messon-peroon or person-object interactivity. Thus, an individual person in this scenario should follow one of the regularities as well. In other words, a person's 's subsequent trajectory has likely been traveled by others. Based on this hypothesis, we propose to forecast a person's 's future trajectory by learning from the implicit scene regularities. We call the regularities, inherently derived from the past dynamics of the people and the environment in the scene, scene history. We categorize scene history information into two types: historical group trajectory and types of information for trajectory prediction, we propose novel framework Scene History Excavating Network (SHENet), where the scene history is leveraged in particular, we design two components: the
interaction. To exploit these two
a simple yet effective approach. group trajectory bank module to extract representative group trajectories the candidate for future path, and the cross-modal interaction module to model the interaction between individual past trajectory and its surroundings for trajectory refinement. In addition, to mitigate the uncertainty in ground -trtth trajectory, caused by the aforementioned randomness and subjectivity of human movement, we propose to include smoothness into the training process and evaluation metrics. We conduct extensive evaluations validate the efficacy of our proposed framework on ETH, UCY, as well as a new, challenging benchmark dataset PAV, demonstrating superior performance compared to state-of-the-an methods. Code is available https://githhne wtt.a///.ooo/aaaaaaaaaaaaa SHENet
Introduction
Human trajectory prediction (HTP) aims to predict a target person's 's future path from video clip 19 20). This critical for intelligent transportation, of pedestrian in advance so that it can avoid potential collision. Surveillance systems with HTP capability can assist security officers to predict the possible escape path of suspects. Although much work has been done in recent years I enough to be applied real-world scenarios, primarily due to two challenges of the task: randomness and subjectivity of human movement. However, in the constrained real world scenarios, the challenges are not absolutely intractable. As shown in Figure T given previously-cape caattrdd video in this scene, the target person's 's future trajectory (red box) becomes human typically complies with several underlying regularities in this scenario, which the target person
enables vehicle to perceive the status
14 18, (35,, very few are reliable and generalizable
predictable the moving pattern of
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Siillanty
Interaction
Historical group trajectoriss
Observation
Surroundings
Figure 1: Illustration of utilizing scene history: historical group trajectories and individual-
surroundings interaction, for human trajectory prediction.
would follow. Therefore, to predict the trajectory, we first need understand these regularities. We argue that these regularities are implicitly encoded in historical human trajectories (Figure[]let individual past trajectory, surroundings, and interaction between them (Figure T right), which refer to as scene history.
We separate historical information into two categories: historical group trajectories (HGT) and setividual-surrounddie
interaction (ISI). HGT refers to the group representatives of all historical trajectories in scene. The reason of resorting to HGT is that, given new target person in the scenario, his/her path is more likely to share more similarity with one of the group trajectories than any individual instance of the historical trajectories, due to the aforementioned randomness, subjectivity, and regularities. Nevertheless, the group trajectories are less correlated to the individual's past status and the corresponding surroundings, influencing the person's 's future trajectory as well. ISI is required to more comprehensively leverage history information by extracting contextual information. Few existing methods have taken into account similarity between individual past trajectory and historical interaction, where tremendous efforts are spent on modeling the individual trajectory, the semantic information of environment information, and the relationship between them. Although MANTRA (20) models the similarity using the encoders trained in a reconstruction manner and MemoNet (32] simplifies the
trajectories. Most attempts [16] 19 27 335
explore only individual-surroundee
similarity by storing the intention of historical trajectories, they both perform similarity computation on instance level, instead of group level, making it sensitive the capability of the trained encoder. Based on the above analysis, we propose simple yet effective framework, Scene History Excavating Network (SHENet), leverage HGT and IST jointly for HTP. In particular, the framework consists of two major components: (i) a group trajectory bank (GTB) module, and (ii) a cross-modal interaction (CMI) module. GTB constructs the representative group trajectories from all historical individual trajectories and provides candidate path for future trajectory prediction. CMI encodes the observed individual trajectory and the surroundings separately, and models their interaction using a cross-modal transformer for refinement of the searched candidate trajectory.
In addition, to alleviate the uncertainty from two mentioned characteristies (i.e., randomness and subjectivity), we introduce curve smoothing (CS) into the training process and current evaluation metrics, average and final displacement errors (i.e.. ADE and FDE), resulting two new metrics CS-ADE and CS-FDE. Moreover, to facilitate the development of research in HTP, we collect a new challenging dataset with diverse movement patterns, named PAV. The dataset obtained by choosing the videos with fixed camera view and complex human movement from MOT15 dataset (11.
The contributions of the work can be summarized follows: 1) We introduce group history to search individual trajectory for HTP. 2) We propose a simple and effective framework, SHENet, to jointly utilize two types of scene history (i.e., historical group trajectories and dadividual-uurronddiin interaction) for HTP. 3) We construct a new challenging dataset, PAV; also, one novel loss function and two new metrics, considering randomness and subjectivity human moving patterns, are proposed to achieve better benchmark HTP performance. 4) We conduct comprehensive experiments on ETH, UCY, and PAV to demonstrate the superior performance of SHENet and the efficacy of each component.
2 Related Work
Unimodal Methods. Unimodal methods rely on learning the regularity of individual motion from past trajectories in order to predict future trajectories. For example, Social LSTM [ focuses

---

sujeciories
Trajectories bank Srank
trajectory [X,)
Group trajectory bank module
Search
gribdding
Concart
Linear
attention
Trajectory encoder
Observed trajectory Xy
Cross-modal
Transformer
Offsets
reshape
Predicted trajectory i
Cross-modal
interaction
aczne Sp
Scone encoder
Figure 2: The architecture of SHENet consists of two components: group trajectory bank module (GTB) and cross-modal interaction module (CMI), GTB clusters all historical trajectories into bank of representative group trajectories and provides candidates for final trajectory prediction. In the training phase, GTB can include the trajectory extend the expressivity, based on the eтror of predicted trajectory. CMI takes the past trajectory of target person and the observed scene input to a trajectory encoder and scene encoder for feature extraction, respectively, followed by a cross- cross-modal transformer to effectively model the interaction between the past trajectory and its surroundings and refine the provided candidate trajectory.
target person into group trajectory bank to
modeling the interaction between individual trajectories through social pooling module. STGAT S uses an attention module to learn spatial interactions and assign reasonable importance to neighbors. PIE (25 uses a temporal attention module calculate the importance observed trajectories at each time step. Giuliari et al. 6 use the Transformer networks 3 30 to predict the trajectories of individual people, which weight all observed points in the trajectories according to attention mechanism. Xu et al. [34] propose newly designed graph neural network to extract spatial-temporte feature representations of the observed trajectories, along with knowledge learning module to refine individual-level transferable representations. These methods consider the scene history in two aspects: individual's 's past trajectory and the interaction between the individual's 's past trajectory and its neighboring trajectories.
Multimodal Methods. Additionally, multimodal methods examine the influence of environment information on HTP. SS-LSTM (35 proposes a scene interaction module to capture global information of the scene, Trajectron++ 27 uses a graph structure model the trajectories and interacts with environmental information and other individuals. Zhou et al. (39 propose a context encoder to extract local features for each agent, and global interaction module aggregate the local context of individual agents. Liang et al. [16] integrate the interactive modules into a person interaction and person-scene) and add a person behavior module (person appearance) to encode rich visual information. Different from the RNNs 16 33 35) to encode past trajectories into hidden state vector, which prone loss of information, MANTRA (0) utilizes an external memory to model long-term dependencies. It stores the historical single-agent trajectories in a memory, and encodes the environment information to refine a searched trajectory from this memory.
module pprson-pprsoo,
Difference from Previous Works. Both unimodal and multimodal methods use single or partial aspects of the scene history, disregarding the historical group trajectory. In our work, we integrate the information from the scene history in a more comprehensive way and propose dedicated modules to handle different types of information, respectively. The main differences between our method
and the previous works, especially memory-basee methods [15] 20 21
32 and clustering-
methods 28 29 36), are as follows: i) MANTRA 20 and MemoNet 32 consider historical
individual trajectories, while our proposed SHENet focuses on historical group trajectories, which is more generalizable in different scenarios: In [15,, historical trajectories are not stored for search.

---

Algorithm Group trajectory bank constructing
Input: Complete trajectory set 2, input trajectory Zp [Xps 3 Yp] of person p. and our model
predicted trajectory Y, of this person
Output: Trajectory bank Zbank
added trajectory counter +- O, added trajectory threshold 3 2: 1* Step 1 Trajectory bank initialization */
I: Initialization: Zbank
Calculate the distance between each pair of trajectories in Z
4: Find medoids for each cluster (via clustering algorithms, K-medoids)
5: Assign each trajectory to the nearest medoid and obtain the initial trajectory bank Zhank by taking the average of trajectories belonging to the same medoid
6: 1* Step Trajectory search
7: Initialization i + 1, m t- Size (bonk))))))))))))))))))))))
8: repeat
9: Choose the past trajectory X; of group from Zbank
10:
Calculate similarity 8; between X; and Xp based on Equation
11: i ++1
12: until m
13: Find the most representative trajectory Yy according to the maximum value of
14: 1 Step 3 Trajectory update */
15: Initialization d < 0, distance threshold 0
16: Calculate the distance between the predicted trajectory Y, with the ground-truth trajectory Yp then
17: if
18: n f nl ++
Update the Zrank by adding the trajectory Y,
20: end if
if 2 3 then
22:
n n==0 0
Cluster newly added trajectories in Zbank, and merge the clustered trajectories into Zoank
24:
end if
return Boank
ii) 28 36 group person-neighbor, for trajectory prediction; 29 clusters the trajectories into fixed number categories for trajectory classification; Our SHENet generates representative trajectories as reference for individual person trajectory prediction.
3 Method
3.1
Overview
The architecture of the proposed Scene History Excavating Network (SHENet) is depicted in Figure 2 which consists of two major components: group trajectory bank module (GTB) and cross-modal interaction module (CMI). Formally, given all the trajectories Z in observed vidcos of this scene, a scene image Sp and a past trajectory of a target person in the last Tpos time steps, Xp = (x (=p)t=-Tpm." +y.. E Raa x2, where oy E R² represents the position of tthh person at time step I, Tequireeq to predict the future positions the person Y, = (0/ E TT x2 in the next Tyut frames, such that P, is as close to the ground-truth trajectory Y, = {ut) Tyue RT as possible. The proposed GTB first condenses Z representative group trajectories Zbank- Then, the observed trajectory X, is used as key to search a closet representative group trajectory in Zoank candidate future trajectory, Yp.. Mcanwhile, the past trajectory Xp and scene image Sp separately forwarded to the trajectory encoder and the scene encoder to yield the trajectory feature and the scene feature, respectively. The encoded features are fed into the cross-modas transformer to learn the offsets Op between Yp and the ground-truth trajectory Yp. By adding O, to Yp, we obtain our final prediction Y,. In the training phase, if the distance of the Y, to Yp is higher than threshold, the trajectory of the person (i.e., X, and Y,) will be added into the trajectory bank Zhank. After training. the bank is fixed for inference.

---

leyer
x
Modality Al
Qutput
Xg
Searched trajectory "
Ntra IRepaeed
Styg
Girundd truth
SA
FFN
LE
M
Op..
Offsets
Meo
CA
SA
FFN
GPL
%
xd
Cross (CA) layer
Modaliy
T
xx
Vodalit 8
Ng R gland
BTAM
An illustration of cross-modal transformer. The trajectory features and scene features are fed into the cross-modal transformer learn the offsets between the searched trajectory and the ground-truti trajectory.
Figure 3:
3.2 Group Trajectory Bank Module
The group trajectory bank module (GTB) is used to construct the representative group trajectories in the scene. The core functions of GTB are bank initialization, trajectory search, and trajectory update.
Trajectory Bank Initialization. Due redundancy of the large amount of recorded trajectories, instead of naively using them, we generate a set of sparse and representative ones as the initial members of the trajectory bank. Specifically,
denote the trajectories in the training data as fi=1 and split each Z; into a pair of observed trajectory X, and future trajectory Y,, so
Z = { that separated into an observed set X' { and a corresponding future set = (Y) ==1 Afterwards, calculate the Euclidean distance between each pair of trajectories in {X', V). and obtain trajectory clusters through the K-medoids clustering algorithm [23]. The initial member the bank Zhank is the average of the trajectories belonging the same cluster (see Algorithm I Step 1). Each trajectory in Zbank represents moving pattern of group of people.
Trajectory Search and Update. In the group trajectory bank Zbank, each trajectory can be treated as past-future pair. Numerically, Zbenk {2.} (=1 |2hant| {[X;; ! where [:] represents the combination of a past trajectory and a future trajectory, and Zbank| is the number of past-future pairs in Zbank- Given trajectory Zp [ Yp] of person p. we use the observed Xp as key to
compute its similarity score to the past trajectories (X,) 12aanel in Zbank and find representative trajectory Ypr: 1 S p' S Zbank| according the maximum similarity score (see Algorithm 1 Step 2). The similarity function can be formulated as:
Xp X, X
9;
1si5 Zhank]
By adding the offsets O, (see Equation 2) to the representative trajectory Ygr, obtain predicted trajectory Y, of the observed person (see Figure Although the initial trajectory bank works well for most cases, to improve the generalizability of the bank Zbank (see Algorithm [ Step 3), we decide whether to update Zhank according to the distance threshold 0.
3.3 Cross-modal Interaction Module
This module focuses on the interaction between individual past trajectory and the environment information. It consists of two unimodal encoders to learn human motion and scene information, respectively, and cross-modal transformer model their interaction.
Trajectory Encoder. The trajectory encoder adopts multi-head attention structure from a trans- former network [30] with Ntra self-attention (SA) layers. SA layers capture human motion across different time steps with the size of Tpas Ntra d, and project the motion features from dimension T Tpa Ntro to Tpae x d, where is the embed dimension of the trajectory
5

---

coder. Thus, we obtain the human motion representation X using the trajectory encoder Etra:
Xtra Etra (Xp) E Puupuu
Scene Encoder. Due to the compelling performance of the pretrained Swin Transformer 17 in feature representation, we adopt it scene encoder. It extracts size of Nota X h X w, where Nets (Nots = 150 in the pretrained scene encoder) is the number of semantic classes, such as person and road, and h and are the spatial resolutions. In order make the following module to conveniently fuse motion representation and environment information, we reshape semantic features from size (Nets x x w) (Neis x hw), and project them from dimension (Nas x hw) to (Nete x d) through a Layer Perception layer. As result, we obtain the scene representation Sae using the scene encoder Exce: Xace Ecee (Sp) E R R.Natax
semantic features with the
Cross-modal Transformer: Unimodal encoders extract features from its own modality, disregarding the interaction between human motion and environment information. The cross-modal transformer with Nero layers aims to refine the candidate trajectory Y (see Section (.2) by learning this interaction. We adopt two-stream structure: one for capturing the important human motion constrained by environment information, and the other for picking out the environment information related to human motion. The cross-attention (CA) layers and self-attention (SA) layers the main components of the cross-modal transformer (see Figure B To capture the important human motion affected by environment and acquire the environment information related the motion, CA layers treat one modality as query. and the other as key and value to interact with the two modalities. SA layers serve facilitate better internal connections, which calculate the similarity between an element (query) and other elements (key) in ccene-constrainel motion or motion-relevant environment information. Thus, we obtain the inter-modal representation (Xiray X via the cross- modal transformer Eero: Xtra Exro X x, ) offsets Op between the searched trajectory Yp' and the ground-truth trajectory Yp. we take the last element (LE) hera E Rd of Xire and the output hace E IR of Xsce after the global pooling layer (GPL). Offsets Op R. can be formulated follows:
Eero Xtro). To predict the
Op MI.
where [:
denotes the vector concatenation, and MLP is the Multi-Layer Perception layer.
3.4 Training
We train the overall framework of SHENet end-to-end to minimize an objective function. During the training, since the scene encoder has been pretrained on ADE20K (38), we freeze its segmentation part and update parameters of MLP head (see Section 3.3). Following the existing works [9] [9 20 35),
we calculate the mean squared error (MSE) between the predicted trajectory and the ground-truth trajectory on ETH/UCY datasets: Lira
Tyut l1 1/.12
Tyue
In more challenging PAV dataset, we use the curve smoothing (CS) regression loss, which serves to reduce the impact of individual bias. calculates the MSE after trajectory smoothing. CS loss be formulated as follows:
T Tjut
Les
Y, = CS(Y))
Tyae
(3)
where CS represents the function of curve smoothing
2
4 Experiments
4.1 Experimental Setup
PAV and Stanford Drone Dataset (SDD) 26 datasets. ETH/UCY datasets are the benchmark commonly used for human trajectory prediction (HTP). The benchmark contains videos from scenes, including ETH, HOTEL, UNIV, ZARAI,, and ZARA2 captured at 2.5HZ. Unimodal methods [0 31 35] only focus the trajectory data,
Datasets. We evaluate our method on ETH (24, UCY K.

---

however, multimodal methods [16 20 need to consider scene information. Since video from UNIV not available, [66 20 use fewer frames than the unimodal methods. We follow [16] to process the data and predict future fuu 12 frames given the observed Plaa 8 frames.
Compared the ETH/UCY datasets, PAV is more challenging with diverse movement patterns, including PETS09-S2LI (PETS) ΑΙЛ. ADL-Rundle-6 (ADL), and Venice-2 (VENICE), which are captured from static camera and provide sufficient trajectories for the HTP task. We divide the videos into training (80%) and testing (20%) sets, and PETS/ADL/VENICE contain 2,370/2,935/4, 2,00 training sequences and 664/306/650 testing sequences, respectively. We use Rpas 10 observed frames to predict future fut = 50 frames, such that we can compare the long-term prediction results from different methods.
Different from ETH/UCY and PAV datasets, SDD is a large-scale datatset captured on the university campus in bird's eye view. It consists of multiple interacting agents (e-g., pedestrians, bicyclists, and cars) and diverse scenes (e.g-, sidewalks and crossroads). Following [19], we use past frames to predict future 12 frames. More results conducted on SDD the the supplementary material.
Evaluation Metrics. For ETH and UCY datasets, we adopt the standard metrics [1] 16 20 335 for HTP: Average Displace-
CS-ADE
CS-FDE
Predited tojecsy
Predided injectry
Roug1
Rogg tajectry
Smocheed inuedoy
Srooffes headoy
ment Error (ADE) and Final Displacement Error (FDE). ADE is the average L2 error between predicted trajectories and ground- truth trajectories over all time steps, and FDE is the C2 error between the predicted trajectories and ground-truth trajectories at the final time step. The trajectories in PAV are with some jittering phenomena (e.g., abrupt and sharp turns). Thus, reason- able prediction may produce approximately the same error an unrealistic prediction by using traditional metrics: ADE and FDE (see Figure aa). In order to focus on the pattern and shape of the trajectory itself and also reduce the impact randomness and
8
Figure The illustration of our proposed metries, CS-A AS-ADE and CS-FDE.
Rough trajectory
trajectory
Figure 5: The visualization of some samples after curve smoothing.
subjectivity, we propose the CS-Metric: CS-ADE and CS-FDE (illustrated in Figure 4 CS-ADE is calculated follows:
M
Tut 1= JJ 1/ll2 N x T
CS ADE pp=l
Yp - -S(Y)) = -V Tyue
(4)
where CS is the curve smoothing function defined the same Les in Section 3.4 CS-ADE, CS-FDE calculates the final displacement error after trajectory smoothing:
Similar to
rut
CS FDEE
Y, CS(Y)
(T))))
(
Tyue
(5)
Figure 5 shows some samples from the training data, converting rough ground-truth trajectories to smooth ones.
Implementation Details. In SHENet, the initial size of group trajectory bank is set to Zhank = 32. Both the trajectory encoder and the scene encoder have 4 self-attention (SA) layers. The cross-modal transformer is with 6 SA layers and cooss-attention (CA) layers. We set all the embed dimensions to 512. For the trajectory encoder, learns the human motion information with size of Tas x 512 pas in ETH/UCY, Tpus = 10 in PAV). For the scene encoder, it outputs the semantic features with size 150 x 56 x 56. We reshape the features from size 150 x 56 x to 150 3136, and project them from dimension 150 x 3136 to 150 x 512. We train the model for 100 epochs on 4 NVIDIA Quadro RTX 6000 GPUs and use the Adam optimizer with a fixed learning rate le 5. More details are in the supplementary material.
7

---

4.2 Ablation Studies
In [ we evaluate each component of SHENet, including group trajectory bank (GTB) module, and cross-modal interaction (CMI) module that contains trajectory encoder (TE), a scene encoder (SE) and cross-modal transformer (CMT).
Table I: Ablation experiments on PAV dataset. + represents that lower is better. Bold/underline denotes the lowest/second low error.
Method
Evaluation metrics: /S-ADE + (in pixels)
CS-FDE
GTB
TE
SE
CMT
PETS
ADL
VENICE
AVG
X
X
X
37.51 / /94.99
17.23 / 43.13
9.82 21.68
21.52 53.07
X
V
X
x
38.51 / 104.42
19.13 51.10
11.46/ 24.14
23.03 59.89
X
x
39.53 / 105.41
/6.14/
9.77 21.65
21.81 57.18
X
36.62 / 99.55
00.12/45.22
9.15 20.62
20.63 55.13
34.49 78.40 14.42 38.67
7.76 18.31
18.89/ 45.13
Effect of GTB. We first investigate the performance of GTB. GTB improves FDE by 21.2% PETS compared to the CMI (i.e., TE, SE, and CMT), which a significant improvement, indicating the importance of GTB. However, GTB (1-st of Table [ alone is not enough and performs little worse than CMI. Thus, we explore the effects of various parts in CMI module.
Effect of TE and SE. To evaluate the performance of TE and SE, we concatenate the trajectory features extracted from TE and the scene features from SE together (3-rd row in Table(]). and improve performance ADL and VENICE with smaller movements, compared the case of using TE alone. This indicates that the incorporation of environment information into trajectory prediction can improve the accuracy of the results.
Effect of CMT. Compared to the third row of Table [ CMT (4-th row in Table I can result in promising improvement of the model performance. Notably, it performs better than the TE and SE concatenated on PETS, improving by 7.4% over ADE. Compared GTB alone, the full CMI improves by 12.2% over ADE on average.
4.3 Comparison with State-of-the-art Methods
Table 2: Comparison of ffute-of-the-aee oftthe-art (SOTA) methods ETH/UCY datasets. * represents that use a smaller set than unimodal methods. The best-of-20 is adopted for evaluation.
Evaluation metries: ADE 1 FDE (in meters)
Method
ETH
HOTEL
UNIV
ZARAI
ZARA2
AVG
SS-LSTM
35
1.01 / 1.94
0.60/ 1.34
0.71 1.52
0.41 /0.89
0.31 / 0.68
0.61 / 1.27
Social-STGCC
22
0.75 1.38
0.61
1.40 1.40
0.58 1.03
0.72
0.43/0.71
0.56 / 1.05
MANTRA 20
1.76
0,28 028/0.. 0.68
0.51 1.26
0.25 /.67
0.20
0.39 / 0.98
AgentFormer 337
0.52
0.84
0.15 / 0.22
0.34 0.72
0.18 / 0..3
0.16 0.30
0.27 / 0.48
YNet 19
0.47 0.72
0.12 0.18
0.27 / 0.47
0.20 0.34
0.15 0.24
0.24 / 0.39
SHENet (Ours)
0.41 0.61
0.13 0.20
0.25 / 0.43
0.21 0.32
0.15 0.26
0.23 / 0.36
On ETH/UCY datasets, we compare our model with the state-of-the-at methods: SS-LSTM [35] 2 MANTRA 20), AgentFormer [37]. YNet [19]. The results summarized in Table Our model reduces the average FDE from 0.39 to 0.36, and achieves 7.7% improvement compared with the state-of-the-an method, YNet. Particularly,
model outperforms previous methods significantly on ETH. when there are large movements of trajectories, and improves the ADE and FDE by 12.8% and 15.3%, respectively.
To evaluate the performance of our model in long-term predictions, we conduct experiments on PAV, with ppas = 10 observed frames and fu 50 future frames of each trajectory. Table 3 shows the

---

performance compared with previous HTP methods: SS-LSTM (35, Social-STGCN 1221. Next [6].
MANTRA (20). YNet [9]. Compared the stateoof-the-ar results of YNet, the proposed SHENet
achieves respective 3.3% and 10.5% improvement over CS-ADE and CS-FDE on average. Since
YNet predicts the heatmaps
of trajectory, performs
Table 3: Comparison with SOTA methods on PAV dataset.
little better when there are
Evaluation metries: CS-ADE // CS-FDE (in pixels)
Method
small movements of trajec- tories, e.g., VENICE. Nev- ertheless, our method is
PETS
ADL
VENICE
AVG
SS-LSTM 35
39.42 / 107.24
16.52/50.44 50.40
10.37 / 23363
22.10/ 60.42
Social-STGCN 22
43,40/ / 117.85
24.34/57.22
14.42/ 38.66
27.39/ /71.24
still competitive in VENICE, and is much better than oth- ers approaches on PETS with large movements and
Next [GG
37.54 98.56
16.82 46.39
8.37 19.32
20,91
54.76
MANTRA 20
106.89
17.26/
12.50/29.00
62.20
YNet 19
36.46 / 93.53
15.07 /11664
7.10/ 16.11
19.54 50.43
SHENet (Ours)
34,49 78.40
14.42/38.67
7.76 18.31
18.89/ 45.13
intersections. In particular, our method improves the CS-FDE by 16.2% on PETS, compared to YNet. We also achieve great improvements using the traditional ADE/FDE metries. The complete results are shown in the supplementary material.
4.4 Analysis
Distance Threshold used to determine the update of trajectory bank. The typical value of
according to the trajectory length. When the ground-truth trajectory is longer in terms of pixel, the absolute value of prediction er- ror is usually larger. However, their relative errors are comparable. Thus, the f is set to be 75% of the training error when the error converges. In our experiments, we in PETS, and / = 6 in ADL. The "75% of the
Table 4: Comparison between different parameters 0 on PAV dataset. Results are the average of the three scenarios.
0
25%
50%
75%
100%
ADE
22.82
20.48 51.67
18.89
20.78
FDE 58.28
45.13
54.52
training error" is obtained from the experimental result, as shown in Table
Cluster Number in K-medoids. We study the effect of setting different number of initial clusters K, shown in Table We note that the initial number of clusters is not sensi- tive prediction results, especially when the initial number clusters is 24-36. Therefore, can set K 32 in our experiments.
Table 5: Comparison between the initial number of
clusters K on PAV dataset.
K
24
28
32
36
ADE
19.28
19.23
18.89
19.17
FDE
45,48
45.59
45.13
45.62
Analysis for Bank Complexity. The time complexity of the searching and updating are O(N)
and (1)) Their space complexity is ONN). The group trajectory number N 1000. The time complexity of the clustering process is 0 + M K (,, and the space complexity is O + M3). is the number of trajectories for clustering. K' is the number of clusters, and M is the number of iterations for the clustering methods.
Predicionn Grutnd
Predicionn
Predition
Ground muth
Oound tufb
trutt
SS-LSTM
MANTRA
YNet
Ours
Figure 6: Qualitative visualization of method and state-of-the-art methods. The blue line is the observed trajectory. The red and green lines show the predicted and ground-truite trajectories.

---

4.5 Qualitative Results.
Figure ppresents the qualitative results of SHENet and other methods. By comparison, we surprisingly notice that in an extremely challenging situation where person walks to the curb and turns back (green curve), all other methods do not deal with it well, while our proposed SHENet can still handle it. This should be attributed to the effect of our dedicatedly designed historical group trajectory bank
module. In addition, compared to the mem- ory -based method MANTRA (20] we search trajectories of groups, instead of just individ-
Prodetion
Ground Ind
Ground Iruht
uals. This more versatile and can be ap- plied in more challenging scenes. Figure 7 includes the qualitative results of YNet and our SHENet without/with curve smoothing (CS). The first row presents the results of us- ing MSE loss Cira. Influenced by the past trajectories with some noise (e.g., abrupt and sharp turns), the predicted trajectory points of YNet are gathered together, and cannot present a clear direction, while our method can provide a potential path based the his- torical group trajectories. The two predic- tions are visually distinct, however, the nu- merical errors (ADE/FDE) are approximately the same. In contrast, the qualitative results of our proposed CS loss Les are shown in the second row of Figure 7 We can see that our proposed CS significantly reduces the impact
YNet
Ours
(a) Results of using Ltra-
Prodictoon
Predktion
Ground
YNet
Ours
(b) Results of using fess
Figure 7: Qualitative visualization without/with CS.
of randomness and subjectivity, and produces reasonable predictions by YNet and our method. More visualization results provided in supplementary material.
5
Conclusion
The paper presents SHENet, novel method that fully utilizes scene history for HTP. SHENet includes a GTB module to construct a group trajectory bank from all historical trajectories and retrieve a representative trajectory from this bank for an observed person, and also a CMI module (which interacts between human motion and environment information) to refine this representative trajectory. We achieve the SOTA performance on the HTP benchmarks and our method demonstrates significant improvements and generalizaibility in the challenging scenes. However, there still several unexplored aspects the current framework, e.g., the process of bank construction currently focusing only human motion. Future work includes further exploring the trajectory bank using the interact information (human motion and scene information).
Acknowledgments and Disclosure of Funding
This work is supported by the National Key R&D Program of China (202IZD011110) and sponsored by Shanghai Rising-Star Program.
References
[1] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 961-971, 2016.
[2] J.-w. Choi, R. Curry, and G. Elkaim. Path planning based on bézier curve for autonomous ground vehicles. In Advances in Electrical and Electronics Engineering-IAEIE Special Edition of the World Congress on Engineering and Computer Science 2008 pages 158-166. IEEE,
2008.
10

---

[3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04880 04005, 2018.
[4] A. Ellis and J. Ferryman. Pets2010 and pets2009 evaluation results using individual ground truthed single views. In 2010 7th IEEE international conference on advanced video and signal based surveillance, pages 135-142. IEEE, 2010.
[5] H. Girase, H. Gang. S. Malla, J. Li, A. Kanehara, K. Mangalam, and C. Choi. Loki: Long term and key intentions for trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9803-9811 2021.
[6] F. Giuliari, Hasan, M. Cristani, and F. Galasso. Transformer networks for trajectory forecasting. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 10335-10342. IEEE, 2021.
[7] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2255-2264, 2018.
[8] Y. Huang, H. Bi, Z. Li, T. Mao, and Z. Wang. Stgat: for human trajectory prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6272-6281, 2019.
Modeling spatial pempoal interactions
[9] B. Ivanovic and M. Pavone. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2375-2384, 2019.
[10] V. Kosaraju, A. Sadeghian, R. Martín-Martin I. Reid, H. Rezatofighi, and S. Savarese. Social- bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. Ad- vances in Neural Information Processing Systems, 32, 2019.
[11] L. Lexl-Taixé, A. Milan, I. Reid, S. Roth, and K. Schindler. Motchallenge 2015: Towards benchmark for multi-target tracking. arXiv preprint 150400/ 2015.
[12] N. Lee, W. Choi, P. Vernaza, C. B. Choy, P. H. Torr, and M. Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 336-345, 2017.
[13] A. Lerner, Y. Chrysanthou, and D. Lischinski. Crowds by example. In Computer graphics forum, volume 26, pages 655-664 Wiley Online Library, 2007.
[14] Li, H. Ma, and M. Tomizuka. Conditional generative neural system for probabilistic trajectory prediction. In 2019 IEEE/RSJ International Conference on Intelligent Robols and Systems (IROS), pages 6150-6156. IEEE, 2019.
[15] L. Li, M. Pagnucco, and Y. Song. Graph-based spatial transformer with memory replay for multi-future pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2231-22444 2022.
[16] J. Liang, L. Jiang, J. C. Niebles, A. G. Hauptmann, and L. Fei-Fei. Peeking into the future: Predicting future person activities and locations in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5725-5734, 2019.
[17] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[18] K. Mangalam, H. Girase, S. Agarwal, K.-H. Lee, E. Adeli, J. Malik, and A. Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. European Conference on Computer Vision, pages 759-776. Springer, 2020.
[19] K. Mangalam, Y. An, H. Girase, and J. Malik. From goals, waypoints & paths to long term human trajectory forecasting. Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 15233-15242, 2021.

---

[20] F. Marchetti, F. Becattini, L. Seidenari, and A. D. Bimbo. Mantra: Memory augmented networks for multiple trajectory prediction. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7143-7152, 2020.
[21] Marchetti, F. Becattini, L. Seidenari, and A. Del Bimbo. Multiple trajectory prediction of moving agents with memory augmented networks. IEEE Transactions Puttern Analysis and Machine Intelligence, 2020.
[22] A. Mohamed, K. Qian, M. Elhoseiny, and C. Claudel. Social-stgenn: A social Apatio-teppora graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14424-14432 2020.
[23] H.-S. Park and C.-H. Jun. A simple and fast algorithm for k-medoids clustering. Expert systems with applications, 36(2):3333-333 2009.
[24] Pellegrrni, A. Ess, and L. V. Gool. Improving data association by joint modeling pedestrian trajectories and groupings. In European conference on computer vision, pages 452-465. Springer, 2010.
[25] A. Rasouli, I. Kotseruba, T. Kunic, and J. K. Tsotsos. Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages
2271,2011 2019.
[26] A. Robicquet, Sadeghian, A. Alahi, and S. Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In European conference on computer vision, pages 549-50 565.. Springer, 2016.
[27] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-fessible trajectory forecasting with heterogeneous data, In European Conference on Computer Vision, pages 683-700. Springer, 2020.
[28] Sun, Q. Jiang, and C. Lu. Recursive social behavior graph for trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 660-669, 2020.
[29] J. Sun, Y. Li, H.-S. Fang, and C. Lu. Three steps to multimodal trajectory prediction: Modal- ity clustering, classification and synthesis. In Proceedings of the IEEÉ/CVF International Conference Computer Vision, pages 13250-13259, 2021.
[30] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[31] A. Vemula, K. Muelling, and J. Oh. Social attention: Modeling attention in human crowds. In 2018 IEEE international Conference on Robotics and Automation (ICRA), pages 4601-4607. IEEE, 2018
[32] C. Xu, W. Mao, W. Zhang, and S. Chen. Remember intentions: Retrospective- herooee--mmmrr--bbaaaaabea trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6488-6497, 2022.
[33] Xu, Z. Piao, and S. Gao. Encoding crowd interaction with deep neural network for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5275-5284 2018
[34] Y. Xu, L. Wang, Y. Wang, and Y. Fu. Adaptive trajectory prediction via transferable gnn. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6520 6531, 2022.
[35] H. Xue, D. Q. Huynh, and M. Reynolds. Ss-lstm: A hierarchical Istm model for pedestrian trajectory prediction. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1196.1194. IEEE, 2018.
12

---

[36] H. Xue, D. Q. Huynh, and M. Reynolds. Poppl: Pedestrian trajectory prediction by lstm with automatic route class clustering. IEEE transactions on neural networks and learning systems, 32(():77-90, 2020.
[37] Y. Yuan, X. Weng, Y. Ou, and K. M. Kitani. Agentformer: Agent-aware transformers for socio- temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 98 183-9823,, 2021.
[38] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302-33) 1200002--11122 2019.
[39] Z. Zhou, L. Ye, Wang, K. Wu, and K. Lu. Hivt: Hierarchical vector transformer for multi- agent motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8823-8833 2022.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] . You strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes]
Did you include the license the code and datasets? [No] proprietary.
The code and the data
Did you include the license to the code and datasets? [N/]]
Please do not modify the questions and only use the provided macros for your answers. Note that the
Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as a URL)? [Yes] All data used in the paper is public; our code are provided
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
13

---

(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] public datasets
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] public datasets
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "wFnrE9EUdJ-", "reviewer": "Reviewer_XkVC", "summary": "The authors tackle the problem of human trajectory forecasting with the help of a novel architecture. The architecture has two components, the group trajectory bank module to extract representative group trajectories as the candidate for future path, and the cross-modal interaction module to model the interaction between individual past trajectory and its surroundings for trajectory refinement, respectively. The authors construct a new challenging dataset, PAV for long-term prediction. The article demonstrates the superiority of the proposed model on ET/UCY and the new PAV dataset over state-of-the-art baseline methods. A new set of metrics, curve smoothing ADE/FDE have also been introduced here.", "questions": "As also previously mentioned:\n- What are the typical values of the trajectory threshold, beta and distance threshold, theta, used as hyper-parameters in the construction of group trajectory bank? Are the hyper-parameter values specific to a dataset?\n- What is the exact contribution of the PAV dataset? What are the criteria for the chosen subset of the MOT15 videos? What is the reason for introducing this dataset instead of experimenting on the existing and used datasets, such as Stanford Drone Dataset (SDD), Intersection Drone Dataset (inDD)?\n- The reported performance of some SOTA works on the ETH/UCY datasets is much worse than the results of their original papers. Why are the original results not reported in the comparison?", "limitations": "The authors did not discuss any limitations or potential negative social impact of the work.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The work is of moderate originality, with a fair presentation and writing quality. The proposed ideas of group memory and cross-modality fusion are interesting.\n- The proposed models seems to be able to improve the prediction performance concerning the reported state-of-the-art results. Nevertheless, there is some lack of clarification in the experiments and evaluation part, especially about the reported SOTA results and how they were obtained.\n\nWeaknesses:\n- It is not clear the exact contribution of the PAV dataset; what is the criteria for the chosen subset of the MOT15 videos? What is the reason for using this dataset instead of existing and used datasets, such as NuScenes, Oxford RobotCar, and Cityscapes?\n- It makes sense to enhance the loss function with curve smoothing during training; however, is it fair to use curve smoothing only on the ground truth during evaluation, especially for the state-of-the-art works that did not apply CS-loss? Actually, it is not mentioned if the SOTA works were retrained with CS-loss or not, and maybe it would be more convincing to include an evaluation with the default metrics as well as the new metrics.\n- The reported performance of some SOTA works on the ETH/UCY datasets is much worse than the results of their original papers. Why are the original results not reported in the comparison?\n- It is not mentioned anywhere in the paper what are the typical values of the trajectory threshold, beta and distance threshold, theta, used as hyper-parameters in the construction of group trajectory bank? It is also not said anywhere if the hyper-parameter values are specific to a dataset? \n- A comparative study could be performed with different types of clustering methods, used for constructing the group trajectory bank.\n\nMinor comments:\n- One minor mistake in row 108: t should start from -T_{pas+1}, instead of -T_{pas}.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "vkHT3qcLrc0", "reviewer": "Reviewer_tAHR", "summary": "This paper proposes to use the \"scene history\" for the human trajectory prediction. The \"scene history\" consists of two parts: historical group trajectories and individual-surroundings interaction. This paper proposes a SHENet to jointly use these two \"scene history\" clues.\n\nExtensive experiments are conducted on ETH, UCY, and a new, challenging benchmark dataset PAV.\n", "questions": "See [Weakness1] and [Weakness2].", "limitations": "No clear limitations and potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strength1] The proposed method which uses the \"scene history\" (especially historical group trajectories) is well-motivated. \n\n[Strength2] The basic idea makes sense. Historical group trajectories and individual-surroundings interaction are helpful to understand human behavior. \n\n[Strength3] This paper provides a new challenging dataset.\n\n[Strength4] The proposed method achieve good performance.\n\n[Weakness1] It is suggested to add the experiments on the SDD dataset, which is also an important benchmark for human trajectory prediction.\n\n[Weakness2] For most real-world applications, we can not obtain the scene in which the model will be used. In the proposed method, the bank is fixed for inference after training. How to guarantee the learned bank with training scene can be used for the new environment. \n\n[Weakness3] The reproduction is difficult since no source code is attached in the supp.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Nd0oyqhrMbI", "reviewer": "Reviewer_bcbq", "summary": "This paper focus on the task of human trajectory prediction. The authors stress the importance of scene history, including historical group trajectories and individual-surroundings interaction, and design a clustering & update procedure to maintain a group trajectory bank, which is used to generate trajectory prediction proposals. A cross modal transformer fuse the feature of observed trajectory and scene information, which is then used to predict the offset from the trajectory proposals. The authors conduct intensive experiments on ETH, UCY, and PAV datasets.", "questions": "None", "limitations": "About the ethical impact: acquiring human trajectory dataset may violate privacy under some circumstances. The authors may consider to add the potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-organized and easy to follow. The authors state their motivation quite clear.\n2. The novelty and originality are good. The group trajectory bank module is innovating.\n3. The ablative study is comprehensive and convincing. The results shows that the main module - group trajectory bank, improve the performance by a non-trivial margin.\n\nWeaknesses:\n1. In the experiment section of ETH/UCY dataset, the authors didn't states the setting clear: what is the sampling protocol when calculating ADE/FDE? Standard protocol is best-of-20. Also, as the architecture itself is not stochastic, it'll be better if deterministic prediction ADE/FDE is added.\n2. The performance of previous work in the table is different from the results from the original paper. For example, Y-net achieve 0.18/0.27 according to the original paper, but in table 2 it becomes 0.24/0.39. Is it due to different experiment setting or different implementation?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N7VNW5XXS1K", "reviewer": "Reviewer_ZK4Z", "summary": "In this paper, a method named Scene History Excavating Network (SHENet) is proposed for pedestrian trajectory prediction. The history information is divided into two categories: historical group trajectories and individual-surroundings interaction. Specifically, a group bank module and an interaction module are designed to explore these two information. In addition, the curve smoothing strategy is utilized to alleviate the individual bias. The proposed method can achieve SOTA performance on ETH/UCY as well as a new PAV dataset.", "questions": "Concerns:\n1. Memory Network is well used in many relevant tasks, i.e., trajectory prediction [1-3], question-answering, anomaly detection, etc, a subsection of related work should be included. Only one paper [4] is mentioned to tell the differences. The approach of trajectory clustering (group idea) is also not new [5-8] here. The similarity function (Eq. (1)) is the same as [4], it should be noted that it is adopted from other papers. In addition, the Transformer structures, including cross-attention and self-attention are also widely used in the trajectory prediction task. Thus, in my opinion, the novelty is not significant.\n\n2. Some key details are missing.  \n(a). Based on my knowledge, the original videos of ETH/UCY are not aligned with the pedestrian coordinates, so how to align the scene with the trajectory? Or is it directly operated in the feature-level for feature fusion?  \n(b). In Line 4 of Algorithm 1, the K is initialized as 32 (Line 211), what's the reason/insight of setting this variable equals to 32. It's a really important hyper-parameter since it represents the initial cluster number. I couldn't see any analysis or experiments about this variable.  \n(c). In Line 155 of Sec. 3.3, the Swin Transformer is applied for extracting scene semantic features, but the number of semantic classes is 150. I assume it is pre-trained here, but why use150 in the trajectory prediction task? I don't think it is reasonable to directly use150 semantic classes here. Details and explanations are missing here.  \n(d). As for the results of ETH/UCY, some of these baselines are stochastic, the results are usually the best of 20. I couldn't see any stochastic properties in this proposed method, so are the results deterministic or stochastic?\n\n3. Other questions.  \n(a). The bank is fixed while inference, how to deal with the situation when there is a novel/unseen trajectory (pattern) when testing?  \n(b). In Eq. (1), the Cosine Similarity is used there, the range of coordinates is not mentioned, usually we normalized the coordinates of ETH/UCY into $[-1, 1]$, which makes the range of $s_{i}$ is $[-1, 1]$. When $s_{i}=-1$, these two trajectories are totally opposite, how to define this kind of similarity? Is it similar or different? I think more explanations should be included here.  \n(c). What's the meaning/insights of introducing curve smoothing (CS)? If using CS loss could have better performance, does it mean the trajectories in this dataset are more \"stable\"? Will using CS loss affect the generalization ability of the model itself, only making smooth predictions? Is that prior knowledge? \n\n4. Insufficient experiments.  \n(a).  No experiment to support the necessity of using CS loss.  \n(b).  When memory mechanism is used, I think the analysis of complexity/cost is necessary.  \n(c). The study of Size($Z_{bank}$) is missing, how to initialize the memory bank, the changing process of this memory bank is necessary to support the whole method.\n\n\nMinors:  \n1. Line 2, Line 6 and Line 14 of Algorithm 1, $/*$ may need to adjust.\n2. Line 7 of Algorithm 1, I believe Size($Z_{bank}$) is K.\n3. In References, the names of conferences or journals, i.e. [1] IEEE or IEEE/CVF, are not consistent.\n\n[1] Remember Intentions: Retrospective-Memory-based Trajectory Prediction. CVPR 2022  \n[2] Graph-Based Spatial Transformer With Memory Replay for Multi-Future Pedestrian Trajectory Prediction. CVPR 2022  \n[3] Multiple Trajectory Prediction of Moving Agents with Memory Augmented Networks. IEEE TPAMI 2020  \n[4] Mantra: Memory Augmented Networks for Multiple Trajectory Prediction. CVPR 2020    \n[5] Recursive Social Behavior Graph for Trajectory Prediction. CVPR 2020  \n[6] Group lstm: Group Trajectory Prediction in Crowded Scenarios. ECCV Workshop 2018  \n[7] Three Steps to Multimodal Trajectory Prediction: Modality Clustering,Classification and Synthesis. ICCV2021  \n[8] PoPPL: Pedestrian Trajectory Prediction by LSTM With Automatic Route Class Clustering. IEEE TNNLS 2020  ", "limitations": "No limitations described in the main body paper as well as the supplementary material.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Clear-written and well-organized.\n- Promising performance.\n- A new complex trajectory dataset.\n\n\nWeaknesses: (See Questions for detailed comments)\n- Lack of novelty.\n- Missing technique details.\n- Insufficient experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept