Neuralized Markov Random Field for Interaction-Aware Stochastic Human Trajectory Prediction
Authors: Zilin Fang
David Hsu
Gim Hee Lee
Venue: ICLR 2025 Poster (2025)
Keywords: human trajectory prediction
interaction modeling
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Interactive human motions and the continuously changing nature of intentions pose significant challenges for human trajectory prediction. In this paper, we present a neuralized Markov random field (MRF)-based motion evolution method for probabilistic interaction-aware human trajectory prediction. We use MRF to model each agent's motion and the resulting crowd interactions over time, hence is robust against noisy observations and enables group reasoning. We approximate the modeled distribution using two conditional variational autoencoders (CVAEs) for efficient learning and inference. Our proposed method achieves state-of-the-art performance on ADE/FDE metrics across two dataset categories: overhead datasets ETH/UCY, SDD, and NBA, and ego-centric JRDB. Furthermore, our approach allows for real-time stochastic inference in bustling environments, making it well-suited for a 30FPS video setting. We open-source our codes at: https://github.com/AdaCompNUS/NMRF_TrajectoryPrediction.git

Published as a conference paper at ICLR 2025
NEURALIZED MARKOV RANDOM INTERACTION-AWAT STOCHASTIC HUMAN TRA-
FIELD FOR
JECTORY PREDICTION
Zilin Fang', David Hsul.2, Gim Hee Leet
School of Computing, National University of Singapore (NUS) Smmrrt Systems Institute, NUS
ABSTRACT
Interactive human motions and the continuously changing nature of intentions pose significant challenges for human trajectory prediction. In this paper, we present a neuralized Markov random field (MRF)-based motion evolution method for probabilistic interaction-awan human trajectory prediction. We use MRF to model each agent's ' motion and the resulting crowd interactions over time, hence is robust against noisy observations and enables group reasoning. We approx- imate the modeled distribution using two conditional variational autoencoders (CVAEs) for efficient learning and inference. Our proposed method achieves state- of-the-art performance on ADE/FDE metrics across two dataset categories: over- head datasets ETH/UCY, SDD, and NBA, and ego-centric JRDB. Furthermore. our approach allows for real-time stochastic inference in bustling environments, making it well-suited for 30FPS video setting. We open-source our codes at:
INTRODUCTION
Trajectory prediction aims to estimate the most possible future states of one or multiple interacting agents conditioned on past observations. It also serves as the underlying foundation for various intel- ligent systems, including autonomous driving (Jiang et al., [203]] Girase et al. (02T). human-robot interaction systems such as robot guidance (Aghili, 201 2 Rudenko et al.) 2020). service robots (Lee) al. 2018 Sun al., 2018), etc. One of the key challenges in this task to capture the inher- ent multimodal behaviors of agents, which are influenced by environmental obstacles, social rules, self-intention changes, interactive dynamics, etc. Compared to vehicle trajectories, human motions exhibit more short-term uncertainties due to fewer lane constraints in open spaces, and smaller mo- mentum makes the movements more agile and flexible, rendering them difficult to predict.
Current prediction works mainly address such multi-agents entangling problems from two aspects: i) stochastic algorithms that generate multiple future trajectories simultancously (Gupta et al.) 2018 Salzmann et al.) 2020 Giu et al., 2022) cover possible intention changes; ii) interaction modeling
through graphs at various scales (Li et al. 2020) Xu et al., 2022a), socially-ane loware operations (Alahi
ctt al. 2016 Xu et al. 20118) attention mechanism (Yu et al. 22000] Girgis al. [002). et al. Further- more, these two directions are usually combined for further performance improvements (Mangalam al., 2020; Sadeghian et al., 2019 Yuan et al.) 202T). Most of these methods assume that the interaction pattern can be captured and applied to future time, and thus extract human interaction features only from history sequences. Some other works such as latent self-motion (Choi & Hebert 2006), intended actions with heuristics (Han et al.) 2023). crowds as dissipative systems model- ing (Bhaskara al. 2024), etc. apply the Markov property in their framework designs to construct temporal and spatial dependencies. Our method differs with explicit structured modeling of each agent's motion and crowd interactions throughout the entire sequences, essentially state dynamics, employing MRF to iteratively infer the stochastic distribution of future joint configurations.
In this work, we introduce an MRF realized by neural networks for interaction- aware human trajec- tory prediction, depicted in Figure I We first derive the crowd motion evolution as probabilistic distribution. which consists of: i) a Bayesian update term to predict the next states based on the given observations; and ii) transition term that is further factorized into self-evolution term to

---

Published as a conference paper at ICLR 2025
repeat K-1 times
Segment;
Neuralized MRF
*
Encoding 1-Strp Prediction
Figure 1: Neuralized MRF for motion and interaction trajectory prediction. Red boxes rep- resent computation modules. Our proposed method leverages Markovian motion assumption and explicitly models future transitions and interactions. Through iterative prediction, produces con- centrated distribution around the ground truth with more accurate best-of-20 predictions.
predict the next states based on their current states, and an interaction term to model the spatial relations between agents. Figure sketches the proposed MRF-based evolution. assumes that short-term human motions are Markovian up to certain frequencies. By dividing the future joint space configuration (i.e., the collection of individual trajectories) sequences into multiple periods, the resulting trajectory segments form Markov chain conditioned past observations. Within each segment, trajectory clips are treated as nodes in a dynamically constructed graph, with edges linked based distance threshold model interactions between nearby agents. Consequently, joint configuration state can be represented by either one undirected graph or a set of multiple undi- rected sub-graphs. The internal motion dynamics of the state are explicitly modeled by the MRF, which inherently captures the evolving pair-wise and group-wise relationships for the future.
We approximate the distribution using neural network framework with CVAEs, employing variational inference as exact inference is intractable. Our proposed framework is composed of three sub-networks: term comprising the self-evolution and interaction terms; and iii) neural sampler to generate multi- modal samples from the learned latent distributions. We conduct experiments across four datasets named ETH/UCY, SDD, NBA Vp and JRDB to validate the effectiveness of our approach. Through quantitative and qualitative results, we show that our proposed neural interaction-an MRF can capture dynamic and hierarchical interactions among agents via neuralized potentials for future behaviors. Our approach achieves state-of-theeart performance under ADE/FDE metrics, while yielding real-time inference speed across various environments at the same time. Furthermore, our method is robust across the simulated observation noise, and the learned potentials in MRF can be utilized for additional human-centered scene understanding tasks such group inference.
CVAE for the Bayesian update term; ii) a recurrent CVAE for the transition
Our contributions can be summarized as follows: i) A new MRF-based framework for the human trajectory prediction task. Our MRF explicity models the agent's motion dynamics and the resulting crowd interactions. ii) Tractable learning and inference of the MRF by introducing a novel and lightweight neural network. iii) The state-of-the-art and time-efficient prediction performance of our approach, along with its robustness under noise disturbance, is demonstrated through evaluations on interaction-rich datasets.
2 RELATED WORKS
Interaction Modeling, Previous works have shown various approaches to model the interactions
within crowds, utilizing RNNs (Vemula al. 2018 Zhang et al.) 2019),, graph networks (Mohamed
et al., 2020) (Sllzaann et al.) |020)] Xu et al. 2022a) attention mechanisms (Kosaraju et al.) 2019
Xu
Mangalam et al.| [2220] Kamra al. (2020). and transformers to capture spatial and temporal depen- dencies together Yu et al. 2020 Giuliari et al. 2021; Yuan et al. 2021 (Saadatnejad et al.) 2023
Shi et al. 2023). Specifically, graph-based networks usually aggregate edge information from neigh- boring agents and conduct message passing to obtain the final ego-node representations (Salzmann et al. 2020 Hu et al., 2020); while attention and transformer structures attribute varying importance levels for spatially proximal agents and focus on the most relevant ones assist social- inffuenee counting. Our approach explicitly models the full dynamics of joint space state transitions and interactions with an MRF, and attentions employed to extract features from the past.
2

---

Published as a conference paper at ICLR 2025
Bayesian Update
Self-evolution + Interaction
Populate
8m
Sa
O
S1:K
:
S
S
6n
0 {0u: mm: 05}
S₂ X++(k-Dr+1+ttt
Figure 2: Overview of our MRF-hased human motion evolution. Given the history sequence, O1:s+ the first future segment S1 is estimated via the Bayesian update module, and future segments S2.K are generated from the self-evolution and interaction module. The Bayesian update, self-evolution and interaction modules are parameterized by 0 {0., Om, 0z}-
Stochastic Trajectory Prediction. Aside from deterministic methods (Alahi et al. 2016 Vemula
jet al. 2018: Zhang al. 2019). more stochastic trajectory prediction works with deep generative
models have been proposed to manage the multi-modalities of human motions. Generative Adversar-
ial Network (GAN) (Gupta al. [2018] Sadeghian et al.. 2019 Dendorfer al., (021), Variational
Auto-Encoder (VAE) (Mangalam et al. 2020 Lee et al., 2022, Xu et al. 2022c), and Conditional Variational Autoencoder (CVAE) (Lee al. 2017; Salzmann et al. 2020; Cheng et al. 2021; Xu
et al.) [022aa: Yue et al.] 2022 Mohamed et al.) (2200) are several lines. with some variants inte-
grating normalizing flow to implement non- Gaussian priors (Bhattacharyya et al. 2019) or heatmap to utilize rasterized images (Mangalam et al. (021). Recently, denoising diffusion probabilistic
models have also been applied into time series forecasting (Rasul et al. 2021), thereby expanding to trajectory prediction. MID (Gu et al.) (022) formulates the trajectories generation process reverse diffusion process by Markov chain with Gaussian transitions. To tackle the sample gener- ation efficiency, LED (Mao et al. 2023) utilizes learned initializer to produce correlated samples in the reverse process, replacing the Gaussian prior and skipping numerous denoising steps. Singu- lar Trajectory (Bae al., (022bb embeds all types of motion patterns into a Singular space for the generality across benchmarks and uses cascaded denoising in its diffusion-based predictor, signifi- cantly surpassing MID.
Markov Random Field Applications. As popular tool for modeling dependent distributions with an undirected graph, the Markov random field (MRF) has been widely applied in tasks such multi-target tracking (Khan et al. 2004) and location inferring (Mattyus al., 2015), among others. Recently, NMRF (Guan et al. 2024) has also utilized MRFs to capture complicated pixel relationships for stereo matching. In trajectory prediction tasks, the Markov property-i.e., pedes- trian's 's future state primarily depends only on its current state-has been implemented for agent-wise self-motion modeling. from early piecewise segment approach (Choi & Hebert 2006) to cent FlowMNO (Bhaskara et al. 2024). By using a heuristic that relates coordinates and actions, such turning right or left, S-T CRF (Han al., (023) employs CRF to model intended actions for future timesteps. Rather than applying the Markov property only to sub-components, we use the MRF to model the full dynamics of state transitions and crowd interactions, providing a neural network-based method for learning and inference.
3 CROWDS MOTION EVOLUTION WITH MARKOV RANDOM FIELD
3.1 PROBLEM FORMULATION
Given frames of observations O A (0,, 02. O.} up to the current time i, the goal human trajectory prediction to determine the future joint configuration space of the humans X(+1.44 & (X++1, X X in the next T time steps. We further define the joint space configuration M X, A [ = consist of M individual states.
Although it is possible to solve the human trajectory prediction problem with a deterministic func- tion map f Out X model probabilistic distribution p( p(X++1:4 O1a) to
3

---

Published as a conference paper at ICLR 2025
hance the learning of human interactions under noisy observations for future trajectory prediction. Formally, we define the evolution of joint configuration segments S {S Sk Sx - S is a Markov chain parameterized by to indicate that the future prediction segments with an assumed period are conditionally independent of the history segments given the current state and transition dynamics syper-parametee As a result, we get:
p(X++144+ 01:, 0)
p(X++1, X Xe- 01:1, 0)
= p(Si, S2,.., SK 01:
p(Sk - O1.,, ()ppp) ( 01:,,
p(Sk 11:K-1, 01:0 0) ............ p(S2 S1, Or., 0)p(S1 | O1:, 0)
p(Sk
1k- O1e ; ... .......pp)) SS22 O1,, ()p(S1 01a, 0)
(I)
K-1
p(Si 1:t, 0)
II p( + Sk. O),
Bayesian update
self-cvolution
interaction
where K |S| / is the set size of configuration segments determined by period 7 (also referred to as stride in network implementation). 0 is typically estimated from observations which encodes Bayesian update 6₂ based on past measurements, and system dynamics that include indi- vidual predictive motion models 6n and pairwise interactions f5. with visualizations in Figure
3.2 REALIZATION WITH CONDITIONAL VARIATIONAL AUTOENCODER
We propose to realize the posterior distribution p(X++1:++7 O1., 0) with conditional variational autoencoders (CVAEs) since exact inference computationally intractable. Let Z represent the latent embedding of the CVAE, We define the log posterior as: log log.(S...... ,Sk Oz 0) e log PS O) and maximize the log-posterior with the evidential lower bound:
log (a() [S| O) 2 LELBO
Egg ( -Ku (2)
where (.).) is the decoder and (-) is the encoder parameterized by Φ and vh. respectively. The first term is the reconstruction loss and the second term enforces the latent space to be close to the normal distribution. is the reconstructed output of the decoder and is the condition of the CVAE. We further assume that the latent embedding Z is conditionally independent of O given S, iee, ( ZZ 0,5) = ge bound on an n-item dataset (0, X). where |O| |X| n, 0 10 is the set of observations and X = {X++1144] } is the set of ground truth predictions (cf. Section more the training details).
S). The CVAE can be trained by minimizing the evidential lower
3.3 NETWORK ARCHITECTURE
Figure 3 shows an illustration of our network architecture that consists of two CVAFs: 1) Bayesian Update and 2) MRF-based Evolution to approximate the two factors of the posterior distribution derived in Equation] I
Bayesian Update CVAE. We design the Bayesian Update CVAE approximate the Bayesian up- date factor P(S} Oz:,, 0) p(S₁ O2:; 0w) in the posterior distribution pa(S O). The given observations O are first encoded into dimendepnnennt features using the History Encoder con- structed from convolutional layers and the gated recurrent units (GRU), and then fed into a single self-attention layer to output socially-dependent features. We use the socially-dependent features as the condition for our Bayesian Update CVAE. We introduce a Future Encoder which acts as the CVAE encoder to encode the given predicted trajectory X(+14++ into the latent embedding Z ~ (u,, o²) during CVAE training. Finally, we concatenate the socially-dependent features with the latent embedding as the input to multilayer perceptron-basee based Update Decoder that reconstructs the first segment S,. Note that the Future Encoder and the Update Decoder play the role of the CVAE encoder go (Z S) and decoder (S 7, O) in Equation 2 respectively.
4

---

Published as a conference paper at ICLR 2025
Stage CVAE Training
Stage Sampler Training
Future Encoder
Oy
O
Configuu Encoder
Configuration Encoder
Potentials
Potencals Update
History Sampler
+
State Sampler
Decoder
Update Decoder
Dynomics Decoder
Raymuu ltater
1+b--4
ME Lound Pudloker
100 Sucer Feudutiin
8
SAX
only IvUAAA Indiiig
- 21/3 Wrildaag inlence Karth- Kusth-testtaate
(5)
Figure 3: Our network architecture. The Bayesian Update module generates predictions S1 for the first defined sequence segment, which then functions as the input into the MRF-based Evolution module to produce succeeding samples $2 Sx iteratively.
MRF-based Evolution CVAE. We propose the MRF-based Evolution CVAE to approximate the self-evolution and interaction factor ITK-1 p(Sk+1 Sk, 0) in the posterior distribution po(S) O)-
We further factorize p(Sk+1 Sk, 0) into two parts:
The space-level distribution p(Sk 0j) which models the k-th joint space configuration. We inspired by (Khan et al.] 2004 (005) to construct a Markov random field (MRF) with an undirected graph G (V. F.). where the segments S are the nodes V and the edges E represent the interaction between the segments, i.e.:
SS () p(Se (,) II
(S.. 0g).
1EE
7(-) is the pairwise potential of the MRF. We link the edges dynamically according to pairwise distance in each computation step.
ii) The agent-level distribution p(S S. 6m) which models the agent-wise oofftrransitonn Putting the space-level and agent-level distributions together, the self-evolution and interaction fac- tor in Equation becomes:
p(S+11 Sk, 0) x [((SA S., 0m) II 7SS, Sy, 0g),
(4)
GEE
which we approximate with the MRF-based Evolution CVAE. To this end, we propose a Configu- ration Encoder which predicts S..++1 from S., approximate p(S.,t+1 S.,,, 0m). We further introduce a Potential Update module which calculates pairwise spatial distances by using the last frame of the configuration segment, links edges according to a defined distance threshold d, and ac- cumulates edge features into the connected nodes to approximate the MRF IIyEE (SS,,, SJ.x 0y)).
The two features from the Configuration Encoder and Potential Update module are the condition of the CVAE, which then concatenated with the latent embedding Z and fed into the Dynamics Decoder output the joint space configuration Sk+1 for the next timestep. These procedures are looped until the final segment prediction is obtained. All sub-modules are implemented using mul- tilayer perceptions. Note that the Future Encoder and the Dynamics Decoder play the role of the CVAE encoder go [Z] 5) and decoder (S Z, 0) in Equation 2 respectively.
3.4 NETWORK TRAINING
We train our network in two stages as illustrated in Figure 3 In Stage 1, we train the CVAEs using the evidential lower bound loss derived in Equation 2 The reconstruction loss is defined as the
5

---

Published as a conference paper at ICLR 2025
minimum error across N samples and the KL-divergence is summed over all segments as follows:
Estagel = a. min [iin|lx441444 - Xx+144rle + B. Σ DK (O),
(5)
N
Instead of directly sampling from the standard normal distribution (0 7), we further train two samplers: a History Sampler and a State Sampler with other modules frozen in Stage for purpo- sive sampling illustrated in the right-half of Figure 3 where we replace the KL-divergence with discrepancy loss (Bae et al. 2022b) samples over the encoded joint configuration state:
Estage2 min [ X4414+ll2 +
Σ log min
||,; - Egll (6)
JE
j7i
where Y, the i-th sampler output among N samples before feature fusion in Stage 2. a, 3 and À are hyperparameters to balance the loss terms during training. After training, we only apply the components in Stage 2 for inference.
EXPERIMENTS
Datasets. We evaluate methods on four trajectory prediction datasets: ETH/UCY (Pele-
grini et al.] 2009 Lerner et al., 2007). Stanford Drone Dataset (SDD) (Robicquet et al. 2016), NBA VU Dataset (NBA), and the JackRabbot Dataset and Benchmark (JRDB) Martin-Martin
et al. 2021). covering interaction-rich indoor and outdoor scenarios. ETH/UCY dataset contains five subsets: ETH. HOTEL, UNIV, ZARAI, and ZARA2. We follow the leave-one-out approach validation and the remaining subset for test- ing, predicting the future 12 frames (4.8s) using 8 frames observations (3.2s). SDD is a bird's eye view pedestrian dataset collected on a university campus. It is originally annotated in pixels without providing precise projection matrices. We predict the future 12 frames (4.8s) on frames (3.2s) his- tory and report the metrics on both pixel and meter units. NBA contains the trajectories of 10 players and the ball in real basketball games collected by the Vp tracking system. We predict the fu- ture 20 frames (4.0s) based on 10 frames (2.0s) history, similar to most other works. The agent's s intention changes suddenly and frequently in this dataset, resulting in zigzags, sharp turnarounds, etc which increases difficulties compared the pedestrian datasets. JRDB is large-scale egocentric dataset recorded by a social robot in indoor and outdoor environments with stationary and mov- ing behaviors. We follow the vrainvallddttion-tes split applied in Social-Transmttion (Saadatnejad et al. 2023) for deterministic prediction. For stochastic situations, splits are in accordance with the official JRDB detection and tracking challenge. Since all labels contained are in camera view and the robot is moving for several scenarios, we utilize the provided odometry information in rosbags to transform all trajectories into global world coordinates. As officially suggested, the trajectory of future 12 frames is predicted over the past 9 frames with 2.5 frames per second frequency.
from (Gupta et al. 2018) with four subsets for training-validation
Implementation details. We calculate displacement for each configuration segment instead of ab- solute quantities for robustness: S = Sg + A.K, where AS, network output. As mentioned in Section the History Encoder comprises a ID convolution layer with a kernel size of three and an output channel of 32, alongside a GRU layer with hidden embedding dimension of 256, similar to LED (Mao et al. 2023). All the other modules are built using multi-layer perceptions, and the distance threshold d for the potential update module is set to 3m. We set sample number N to 20 in both training and testing stages for stochastic prediction, and train the entire model with AdamW optimizer and StepLR scheduler on one Quadro RTX 8000 GPU. The latent vectors Z, and Z₂ have 16 dimensions and the embedding dimension of the configuration encoder is 32. Other network details and sapaset- hyperparameters are attached in the appendix.
Evaluation metrics. We evaluate our proposed method with ADE and FDE for single sample in the deterministic case, and with minADE and Fin in the stochastic cases following the previous works (Mangalam et al. 2020: Mao et al. 2023). ADE quantifies the average displacement error between predictions and ground truth across all time steps, while FDE measures the displace- ment error from the final timestep to the current time. The minADEg0 and minFDEgo metries cal- culate the minimum error among 20 predictions, and it is commonly used in evaluating generative models' ability to generate high-quality samples.

---

Published as a conference paper at ICLR 2025
Table I: minnADeao/minnDEgg in meters on ETH/UCY datasets. Bold/underline indicate the best/second-ber results. denotes results reproduced from official codes. marks results where the best model was chosen directly based on the test set. We address this issue by following the standard train-val-test split in Social-GAN (Gupta et al.) 2018). selecting the best model via the validation set. $ for the future information leakage issue fixed. Inference speed is measured for generating 20 samples under a scene with 57 pedestrians.
Salme
Proredd 114
1-00
T
1 10
Pay
Ours
ETA
4-51.40
0,011
0.441 12
015.3
0.411 60
0-600:1
6.600 34
0.50446
352044
UNI
$:441.00
0449133
0008 55
0388
0.21/44
8350.9
038255
6250-4
8 126014
027047
0.50135
8.36072
631944
018923
0229 46
053044
818924
015-3
0.8924
4156.2
8149.35
0.30122
11072
0138
6767
77914
13391
18 p00
-3520
-7.00
254
-3500
1288
3200
4.09
50
15.3m
-174m
Table 2: minADEgg/minDDEg in pixels and meters (if available) on SDD dataset. Bold/underline indicate the best/second best results. denotes results reproduced from official codes. Train-test splits are the same as the baseline Social-VAR (Xu et al. 2022c).
4.1 TRAJECTORY PREDICTION RESULTS
ETH/UCY. Table T shows the comparison between our model and recent state-of-the-art bench- marks on the ETH/UCY dataset. Since several baselines report their models' test performances by confounding the validation set with the test split, we correct these numbers by re-training with offi- cial codes and selecting the best model based on the validation set to ensure fairness. Additionally, recent work such as LMTraj (Bac et al.) 2024a) utilizes large language models (LLMs) to capture the multi-modality of human trajectories. However, we omit LMTraj from the baseline since its inference speed is heavily constrained by the response times of models such as GPT-3.5 and GPT-4, and its prediction performance does not surpass Singular Trajeccoryy (Bae et al., 2024b). As shown in the table, our method balances the prediction accuracy and computational efficiency with the best and an inference speed faster than 100Hz under 57-pedestrian sce- nario in the ETH/UCY dataset. It outperforms diffusion- methods like MID (Gu et 2022) and SingularTrajectory (Bae et al. 2024b), and the previous SOTA SocialCirele (Wong et al. 2024).
average A
SDD. Table X shows the comparisons on the SDD dataset with our proposed model maintaining the best performance. We transform the pixel-unit annotations into meter-unit following the base- line SocialVAE (Xu et al. 2022c) since our method uses meter-basee based distance threshold important to note that due the lack of precise projection matrices in the SDD dataset, the pixel-to- meter scale is estimated with large variation and unreliability with some values derived from Google Maps and others based on rough guesses (Amirian et al.) 2020). We include comparisons on this dataset because it is widely used in the community. However, we must emphasize that pixel errors would only be meaningful if a pixel corresponds to fixed physical distance across all scenes, which unfortunately is not guaranteed by SDD.
NBA. As presented in Table 3 the proposed method is then evaluated against 10 baselines the NBA dataset. Our method maintains ofate-of-the-ar accuracy to the best-of-20 samples. We com- pare the inference time with several baselines where official implementations for this dataset are provided, showing that our model infers trajectories less than one-third of the time while preserving the best prediction performance. Figure visualizes the estimation results from the current best
Table mininnDD/ meters on NBA dataset. Bold/underline indicate the best/second- best results." denotes results reproduced from official codes.
Time
Soca ma pac
Mangcaan
Yamial bux
STAR
Triestion
MerroNet
Auaiia
NPPNN
Namaa pc..
CoooeNet
MHD
Max pwzz
LED
Ours
Life
0.41M
0.34/1
0-40N1.71
0.451.55
0.3100.55
0.3531.5
0.26/1.37
0.183227
6.1691.4
2006
0,81/ 32
0,71.194
0.831
0,751
0.549142
14
06611
0.691120
0.510172
0.34M.50
3.6.
1.191 94
1,081 77
1.37/2 44
1155151 1.13201
0.5511 24
1 57
1.11/ 76
0.7341 02
0.130.9
0.5840.55
0.530.75
1.592-41
1.532 26
1,6M2.95 95
1.181.57
1 47
1.31/ 79
09961 30
01981 27
0.551.97
Spa par)
NA
NA
NA
NA
Ams
3-102m
19.3ma

---

Published as a conference paper at ICLR 2025
Samples generated with LED MMa et
2023
Samples generated with our method
Figure 4: Generated prediction samples with N = 20 on NBA dataset. The light blue and dark blue represent the history and future ground truth respectively, and white curves are samples. Mean estimation is highlighted by violet.
approach and our model. The prediction samples from LED (Mao et al. 2023) are noisy with sev- eral sampled positions deviating significantly from the real potential distribution human motions. This noise can negatively impact downstream tasks requiring estimated distributions with confidence since is impractical to measure the ground truth future and report the minimum error in such cases.
Table Comparisons for ADE and FDE on JRDB dataset with deterministic trajectory prediction, in meter unit. In input modality, "T" represents trajectory only, and '2d BB' means the bounding box of the person in 2D images. highlights previously reported best performance.
Time
Suciald STM
SuaNa--IAN
Dristtoral STM
Nenusiiaall pell
Trajerrone
Astrivees
ErMesion
Sucini-Treeenn Teanemution
Basianie paxt
Sorist-TTrmmutinn bey
FD Nell
Ouns (Nel)
Stage J Only
Inpor Modality
T+
ADE/DDE
0.400995
0.500199
0:4500677
0.50R073
Table 5: Stochastic trajectory prediction comparisons on JRDB dataset. minADE and min minFDEzo meters are reported. Numbers are related a constant reference frame.
Method
1.2s
2.48
3.65
Total(4.8s)
Speed (80 por.)
LED Mao et 2023
0.05/0.02
0.09/0.14
0./4/1
0.18/0.28
118ms
Ours (Stage Tmly
0.05/0.06
0.09/0.13
0.13/0.20
0.17/0.27
6.7ms
Ours
0.04/0.05
0.08/0.11
0.11/0.17
0.15/0.23
6.8ms
JRDB. We finally explore the performance of our method on JRDB, which contains diverse captured indoor and outdoor environments close to humans by robot. To ensure a fair comparison with other state-of-thee ahr methods, many of which provide deterministic results, degenerate
method to output only one sample at a time. Throughout training, validation, and testing, we set N to for the sampler, and the resulting ADE/FDE metrics (in the world frame) are detailed in Table Leveraging trajectory-only inputs, our approach outperforms the previous best model Social-Transmmon Transmotion (Saa-) datnejad al. 2023) that combines 2D bounding boxes extracted from images by approximately 29% and 34% in terms of ADE and FDE, respectively. Note that trajectory prediction is usually conducted in consistent reference frame, i.e., the world frame. However, since JRDB is captured by moving robot, baseline methods may directly use its instantaneous coordinate frame, causing human motion to be coupled with robot movement. Nonetheless, the numbers related to the moving camera frame are attached for reference: ADE/FDE on LED is 0.36/0.69, and ours is 0.33/0.63.
We also conduct experiments under stochastic conditions to further investigate the effectiveness of our proposed approach with results presented in Table Our model significantly outperforms the second-best model LED in deterministic cases in terms of minADE and minFDE2o metrics. More- over, the inference time of our method is over 17 times faster in a bustling environment involving 80 persons, which indicates it can applied for the standard video setting. and enables real-time ca- pabilities physical robotic systems desktop GPUs. Figure 5 iillsstrates eight examples of our prediction results, demonstrating the ability our learned distribution to capture human intention changes like sharp turns, sudden stops, etc.
8

---

Published as a conference paper at ICLR 2025
Table Robustness for imperfect past trajectories on JRDB dataset. History trajectory includes 9 frames with scale equal to one, and the percentage stands for the ratio of test data with noise.
Type
Gaussian Noise (100%, scale 0.1)
0.27 (4 frames)/ / 0.28 (9 frames)
0.49 (4 frames) / 0.500 frames)
Gaussian Noise (100%, scale 0.2)
0.32 (4 Irumes) / 0,34 frumes)
0.55 (4 frumes) / 0..5 frames)
Gaussian Noise (100%, scale 0.5)
0.71 (4 frames) / 0.88 (9 frumes)
1.18 (4 frumes) / 1.44 (9 frames)
Dropped History (20%, frames)
0.26
0.48
Dropped History (50%, frames)
0.49
Dropped History (50%. 7 frames)
0.29
0.52
Dropped History (80%, 7 frumes) Ours (full performance)
0.30
0.54
0.26
0.48
Table 7: Ablations on sampler types (Left) and different strides (Right). minADEz0 and minFDEzo for the entire trajectory are reported. Influences from stride 7 are shown with ZARAI subset, and they differ across datasets. 'AVG.' is for the average of distance errors on 20 samples and 'SD.' represents the standard deviation. Speed is measured for an 11 -erson scenario.
Dataset
Stage I Only
Stage 2. Only
Two Siages
Stride
minADF/ FDE20
AVG.
SD.
Speed
(~)
ETII
0.28/0.46
0.24/0.40
0,26/0.37
HOTEL
0.13/0.21
D.11//
0.1 17
0,28/0.35
UNIV
0.32/0.58
0,28/0.48
0,28/0.49
0.74/1.52
8.23ms
ZARA1
0,20/0.36
0.20/0.33
0.17/0.30
ZARA2
0.16/0.30
0,14/0.25
0,17/0.30
0.73/1.50
5.93ms
SDD (pixels)
8,16/13.37
7.83/12.59
7.20/11.29
0/18/0.30
0.71/1.46
0.39/0.81
4.73ms
NBA
0.83/1.16
0.77/0.98
0.75/0.97
0.18/0.30
0.73/1.50
0.39/0.81
3.56ms
4.2 ROBUSTNESS AGAINST OBSERVATION NOISE
In real-world scenarios, historical trajectories are often imperfect due to tracking errors and incom- plete detection. It may contain truncated and broken trajectories, mismatched detection candidates, etc. To validate the robustness, we introduce two types of noise to simulate detection and tracking errors, as shown in Table 6 and noisy frames randomly selected. The test is conducted on the JRDB dataset as it is egocentric and naturally prone to issues when obtaining history sequences from noisy tracklets. To deal with noise type Dropped History, left pad by repeating the first available frames to buffer the dropped observations, which is akin to a stationary person. Since there is no design bottleneck in our framework that precludes stationary states, we consistently produce high- quality predictions across different noise levels. Empirically, we note that minimum of 2 moving history frames are needed for our method to capture the motion effectively.
4.3 ABLATIONS
We further explore the effects of sampler types and different strides with results reported in Table 7 When replacing the sampling from standard normal distribution with purposive sampling, the performances improve from around 7% to 20% on different datasets. Even with Stage 1 solely, achieve comparable prediction performances with other baselines. Experiment results also validate that the Markovian assumption human motion usually holds up to defined observation frequency intervals with a stride reach the best performance.
4.4 GROUP REASONING VIA POTENTIALS
Potentials learned in the trajectory prediction network can also be utilized in group inference. Lever- aging group labels from the JRDB-Act (Ea (Ehsanpour et al. (022) dataset, we perform group clustering without prior assumptions or maximum group number constraints. Concatenated with history im- age features extracted via CLIP (Radford et al. 2021), we build binary edge classifier supervised by ground truth groups. The resulting group inference, portrayed in Figure 0 counts individuals part of group if there is edge between them and at least other agent within the current group. Despite some errors, our method effectively captures scenario relationships, beneficial for pedestrian-friendly navigation tasks adhering social norms.

---

Published as a conference paper at ICLR 2025
maass
16
12
11
10
25
30 21
24
27 28
20 30
12
34
Past trococtory
Grouctruth future tralectory
Prodiction samples
Average of samples
Eest estimattio
Figure 5: Generated prediction samples with N 20 JRDB dataset. highlighted with orange curves
Best-of-20 samples
Figure 6: Group inference with learned potentials for JRDB dataset. Matching capital letters rep- resent identical group labels, while ellipses the same color denote estimated groups. Errors may arise due to low-quality visual cues (ef Person I is misassigned to group in the green ellipse).
CONCLUSION
In this paper, propose neuralized Markov random field (MRF)-based method for human tra- jectory prediction. Specifically, we introduce an interaction-aware MRF that models agent motion and crowd interactions over time. We then design neural network framework consisting of two CVAEs to approximate the posterior distribution for efficient learning and inference. Our method achieves state-of-the-art performance across four benchmark datasets also enables group rea- soning. Furthermore, it is robust against noisy observations and allows for real-time stochastic inference, demonstrating its feasibility on real-world downstream systems.
Limitations. We learn the potential function applied to the MRF graph from data; if the training data only exhibits low graph complexity, the model's 's ability to capture complex interactions will be limited. This complexity generalization issue be further explored in the future. Moreover, the current prediction pipeline uses only coordinates as input; incorporating environmental contexts, such obstacles and traversable paths, could also be considered.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research project supported by Agency for Science, Technology & Research (A*STAR), Sin- gapore under its National Robotics Program (No. M23NBK00S3), and the National Research Foun- dation (NRF) Singapore, under its IRE-nnvesigatorship Programme (Award ID. NRF-NRFI09- 0008).
REFERENCES
Farhad Aghili. A prediction and motion-pland motiinn-panning scheme for visually guided robotic capturing of free-floating tumbling objects with uncertain dynamics. IEEE Transactions on Robotics, 28(3): 634-649, 2012.
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961-971, 2016.
Javad Amirian, Bingqing Zhang, Francisco Valente Castro, Juan Jose Baldelomar, lean-Bernard Hayet, and Julien Pettré. Opentraj: Assessing prediction complexity in human trajectories datasets. In Proceedings of the Asian Conference on Computer Vision, 2020.
Inhwan Bae, Jin-Hwi Park, and Hae-Gon Jeon. Learning pedestrian group representations for multi- modal trajectory prediction. In European Conference on Computer Vision, pp. 270-289. Springer, 2022a.
Inhwan Bae, Jin-Hwi Park, and Hae-Gon Jeon. Non-probability sampling network for stochastic human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6477-6487, 2022b.
Inhwan Bae, Junoh Lee, and Hae-Gon Jeon. Can language beat numerical regression? language- based multimodal trajectory prediction. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 753-766, 2024a.
Inhwan Bae, Young-Jae Park, and Hae-Gon Jeon. Singulartrajectory: Universal trajectory predictor using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Rashmi Bhaskara, Hrishikesh Viswanath, and Aniket Bera. Trajectory prediction for robot nav- igation using flow-guided markov neural operator. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 15209-152161 IEEE, 2024.
Apratim Bhattacharyya, Michael Hanselmann, Mario Fritz, Bernt Schiele, and Christoph-Nikolas Strachle. Conditional flow variational autoencoders for structured sequence prediction. arXiv preprint arXiv:1908.0900 2019.
Hao Cheng, Wentong Liao, Xuejiao Tang. Michael Ying Yang, Monika Sester, and Bodo Rosen- hahn. Exploring dynamic context for multi-path trajectory prediction. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 12795-12801 IEEF, 2021.
Patrick Pakyan Choi and Martial Hebert. Learning and predicting moving object trajectory: a piece- wise trajectory segment approach. Robotics Institute, 337:1-17, 2006.
Patrick Dendorfer, Sven Elfiein, and Laura Leal-Taixé. Mg-gan: A multi-generator model preventing out-of-distribution samples in pedestrian trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3158-13167. 2021.
Mahsa Ehsanpour, Fatemeh Saleh, Silvio Savarese, Ian Reid, and Hamid Rezatofighi. Jrdb-act: A large-scale dataset for spatio-temporal action, social group and activity detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20983-20992, 2022.
Harshayu Girase, Haiming Gang, Srikanth Malla, Jiachen Li, Akira Kanehara, Karttikeya Man- galam, and Chiho Choi. Loki: Long term and key intentions for trajectory prediction. In Pro- ceedings of the IEFF/CVF International Conference Computer Vision, pp. 9803-9812, 2021.
11

---

Published as a conference paper at ICLR 2025
Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D' D'Souza, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Latent variable sequential set trans- formers for joint multi-agen, motion prediction. In International Conference Learning Repre- sentations, 2021.
Francesco Giuliari, Irtiza Hasan, Marco Cristani, and Fabio Galasso. Transformer networks for trajectory forecasting. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 10335-10342. IEEE, 2021.
Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17113-17122, 2022.
Tongfan Guan, Chen Wang, and Yun-Hui Liu. Neural markov random field for stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2024.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: So- cially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2255-2264, 2018.
Pengqian Han, Jiamou Liu, Jialing He, Zeyu Zhang, Song Yang. Yanni Tang, and Partha Roop. St. crf: Spatial-temporal conditional random field for human trajectory prediction. arXiv preprint arXiv:2311.18196 2023.
Yue Hu, Siheng Chen, Ya Zhang, and Xiao Gu. Collaborative motion prediction via neural motion message passing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6319-6328, 2020.
Chiyu Jiang, Andre Cormman, Cheolho Park, Benjamin Sapp, Yin Zhou, Dragomir Anguelov, et al. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9644-9666 1653,9953, 2023.
Nitin Kamra, Hao Zhu, Dweep Kumarbhai Trivedi, Ming Zhang, and Yan Liu. Multi-agent trajectory prediction with fuzzy query attention. Advances in Neural Information Processing Systems, 33: 22530-22544 2020.
Zia Khan, Tucker Balch, and Frank Dellaert. An meme-based particle filter for tracking multiple interacting targets. In Computer Vision-ECCV 2004: 8th European Conference on Computer Vision, Prague, Czech Republic, May 11-14, 2004. Proceedings, Part IV 8, pp. 279-290. Springer, 2004.
Zia Khan, Tucker Balch, and Frank Dellaert. Meme-based particle filtering for tracking variable number of interacting targets. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(11):1805-1819 2005.
Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. Advances in Neural Information Processing Systems, 32, 2019.
Parth Kothari, Sven Kreiss, and Alexandre Alahi. Human trajectory forecasting in crowds: A deep learning perspective. IEEE Transactions Intelligent Transportation Systems, 23(7):7386-7400. 2021.
Beom-Jin Lee, Jinyoung Choi, Christina Back, and Byoung- Tak Zhang. Robust human following by deep bayesian trajectory prediction for home service robots. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7 7189-7195. IEEE, 2018
Mihee Lee, Samuel Sohn, Seonghyeon Moon, Sejong Yoon, Mubbasir Kapadia, and Vladimir Pavlovic. Muse-vae: multi-scale vae for environment-awarc long term trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2221-2230 2022.
12

---

Published as a conference paper at ICLR 2025
Namhoon Lee. Wongun Choi, Paul Vernza,, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 336-345, 2017.
Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by example. In Computer Graph- ics Forum, volume 26, pp. 655-664. Wiley Online Library, 2007.
Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajec- tory prediction with dynamic relational reasoning. Advances in Neural Information Processing Systems, 33:19783-19794, 2020.
Kartrikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned tra- jectory prediction. In Computer AitinnECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 759-776, Springer, 2020.
Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals, waypoints & paths to long term human trajectory forecasting. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 15233-15242, 2021.
Weibo Mao, Chenxin Xu, Qi Zhu, Siheng Chen, and Yanfeng Wang. Leapfrog diffusion model for stochastic trajectory prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 5517-5526, 2023.
Roberto Martin-Martin, Mihir Patel, Hamid Rezatofighi, Abhijeet Shenoi, Jun Gwak, Eric Frankel, Amir Sadeghian, and Silvio Savarese. Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Gellert Mattyus, Shenlong Wang, Sanja Fidler, and Raquel Urtasun. Enhancing road maps by pars- ing aerial images around the world. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1689-1697, 2015.
Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgenn: A so- cial
graph convolutional neural network for human trajectory prediction. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14424- 14432, 2020.
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll never walk alone: Modeling social behavior for multi-targe tracking. In 2009 IEEE 12th International Conference on Computer Vision, pp. 261-268. IEEE, 2009.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-878666 8763. PMLR, 2021.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising dif- fusion models for multivariate probabilistic time series forecasting. International Conference on Machine Learning, pp. 8857-8868. PMLR, 2021.
Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social eti- quette:
Human trajectory understanding in crowded scenes. In Computer Vision-ECCC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pp. 549-565. Springer, 2016.
Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. The International Journal of Robotics Research, 3988)5855-335 2020.
Saeed Saadatnejad, Yang Gao, Kaouther Messaoud, and Alexandre Alahi. Promptable human trajectory prediction. In The Twelfth International Conference on Learning Representations, 2023.
13

---

Published as a conference paper at ICLR 2025
Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical con- straints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. I 1349-1358, 2019.
Tim Salzmann, Boris Ivanovic. Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasiin trajectory forecasting with heterogeneous data. In Computer Vision-ECCO 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pp. 683-700. Springer, 2020.
Liushuai Shi, Le Wang, Chengjiang Long. Sanping Zhou, Mo Zhou, Zhenxing Niu, and Gang Hua. Sgen: Sparse graph convolution network for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8994-9003, 2021.
Liushuai Shi, Le Wang, Sanping Zhou, and Gang Hua. Trajectory unified transformer for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9675-9684 2023.
Li Sun, Zhi Yan, Sergi Molina Mellado, Marc Hanheide, and Tom Duckett. 3dof pedestrian trajec- tory prediction learned from long-term autonomous mobile robot deployment data. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 5942-5948. IEEE, 2018.
Anirudh Vemula, Katharina Muelling, and Jean Oh. Social attention: Modeling attention human crowds. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 4601- 4607. IEEE, 2018.
Chuhua Wang, Yuchen Wang, Mingze Xu, and David J Crandall. Stepwise goal-driven networks for trajectory prediction. IEEE Roboties and Automation Letters, 7(2):2716-273)) 2022.
Conghao Wong, Beihao Xia, Ziqian Zou, Yulong Wang, and Xinge You. Socialcirele: Learning the angle-based social interaction representation for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hyper- graph neural networks for trajectory prediction with relational reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6498 0607- 2022a.
Chenxin Xu, Weibo Mao, Wenjun Zhang, and Siheng Chen. Remember intentions: Retrospective- memory-based trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6488-6497, 2022b.
Chenxin Xu, Robby T Tan, Yuhong Tan, Siheng Chen, Yu Guang Wang, Xinchao Wang, and Yan- feng Wang. Eqmotion: Equivariant multi-agent motion prediction with invariant interaction rea- soning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 4410-1420, 2023.
Pei Xu, Jean-Bernard Hayet, and Ioannis Karamouzas. Socialvae: Human trajectory prediction using timewise latents. In European Conference on Computer Vision, pp. 511-528. Springer, 2022c.
Yanyu Xu, Zhixin Piao, and Shenghua Gao. Encoding crowd interaction with deep neural network for pedestrian trajectory prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5275-5284, 2018.
Yu Yao, Ella Atkins, Matthew directional pedestrian trajectory prediction with multi-modal goal estimation. IEEE Robotics and Automation Letters, 6(2): 2021.
Ram Vasudevan, and Xiaoxiao Du. Bitrap: Bi-
Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temports graph transformer networks for pedestrian trajectory prediction. In Computer Mision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pp. 507-523. Springer, 2020.
14

---

Published as a conference paper at ICLR 2025
Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transform- ers for socio-temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9813-9823, 2021.
Jiangbei Yue, Dinesh Manocha, and He Wang. Human trajectory prediction via neural social physics. In European Conference on Computer Vision, pp. 376-394. Springer, 2022.
Pu Zhang, Wanli Ouyang, Pengfei Zhang, Jianru Xue, and Nanning Zheng. Sr-Istm: State refinement for lstm towards pedestrian trajectory prediction. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12085-12094, 2019.
A APPENDIX
We attach network design details for each sub-component in CVAEs, and data-specific hyperparam- eters for training and inference in Table 8 and Table 9 respectively. In the training phase, we set hyperparameters a, /3 and loss terms as 1, x 10 and x 10-1
Table 8: Network design details.
Module Name
Compunents
History Eacoder
Conv-ld
(32
Update Decoder
MLP (256+16 106-166+5500 512
512
256
2x stridee)
Configuration Encoder
MLP (2xstride -2 32 32)
Dynamics Decoder
MLP (32+16 128
128
2Xsiride)
Potentials Update
(3
History Sampler
MLP (256 + 128 512 16xN)
State Sampler
Future Encoder
MLP (2xstride - 64 + 256 256 + 2x16)
Table 9: Hyperparameters for different datasets.
Dataset
Stride
Batch Size
Learning Rate
Step Size
Gamma
Epoch (CVAF)
Epoch (sampler)
ETH
10
16
200
60
HOTEL
3
50
x 10
16
0.5
200
60
UNIV
x 10
32
0.9
200
ZARAI
2
32
200
SDD
500
200
NBA
10
32
x
32
0.6
150
50
JDDB
3
1 10
32
0.6
200
In Figure we also compare the initialization and final prediction results for our proposed model and the previous state-of-the model LED (Mao al. 2023) on the NBA dataset. LED starts from a noisy initialization, and it can ignore other candidates when optimizing via the best sample, which results in distribution with multiple outliers as shown in Figure (c). Conversely, our method produces trajectory-like samples initially due to the modeling of crowd motion and interaction, and converges to a more plausible distribution around the ground truth after optimization.
Furthermore, more visualization results on ETH/UCY datasets for baseline methods NPSN (Bae al. 2022b), SingularTrajectory (Bae et al. 2024b), and our proposed model, are provided in Fig-
ure g
15

---

Published as a conference paper at ICLR 2025
-2
-3
-3 -2 -1 3
-2
LED initialization
Ours initialization
LED prediction
Ours prediction
Figure 7: Initialization and prediction results of LED and our method. Trajectories are normalized so that the coordinates of the last frames are located at the origin.
Groudouth
NPSN
Singubrrraeettory
Figure 8: Visualization the best predictions among generated samples (N 20) with NPSN, SingularTrajectory, and method.
16


Reviews:
[{"review_id": "ycxXwiP3p4", "reviewer": "Reviewer_o4s5", "summary": "The paper proposes a method for human trajectory prediction that incorporates interaction-awareness through a neuralized Markov Random Field (MRF) framework. By integrating MRF with neural networks, the model captures both individual movement patterns and inter-agent interactions over time, effectively addressing uncertainties in predicting human motion. This design is particularly well-suited for real-time trajectory forecasting.", "questions": "- Figure 1 was never referred to in the text. It will be good to mention it in your text.\n- In table 4, it seems there is a gap between the proposed method and the previous works on JRDB dataset. Please double-check if there is something wrong. (e.g., same data splits? data samples?)", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Novel integration of Markov Random Fields with neural networks for human trajectory prediction.\n- Advances the field by combining structured probabilistic modeling with deep learning to address noisy, dynamic environments.\n- Achieves strong performance on multiple datasets with short inference time.\n- Transparent documentation of training and hyperparameter choices ensures reproducibility.", "weaknesses": "- Important citations about the dataset used in this paper are missing. Eg., ETH, UCY, Stanford Drone Dataset (SDD)...\n- In table 1, it will be interesting to add the previous model Y-net [1] as a baseline since that model had very impressive performance on the datasets used in this paper. \n- The robustness experiment is not clear. E.g., when simulating noisy tracklets by adding Gaussian Noise, which 4 frames out of 9 observed frames are selected to add noise? Are they randomly selected or some specified 4 frames? In addition, adding noise to all 9 observed frames is more convincing to simulate sensor noise. \n\n\n\n[1] Mangalam, Karttikeya, et al. \"From goals, waypoints & paths to long term human trajectory forecasting.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V8jvNozIpa", "reviewer": "Reviewer_cAi6", "summary": "This work introduces a multi-person tracking approach. It summarizes per-frame tracks into short segments and uses a MRF to forecast those segments, rather than per-frame tracks. To initialize the first trackless, a VAE based “Bayesian update” is performed. Surprisingly, only this first step requires knowledge of the input motion, while the MRF factors out any of the input conditions. The method produces SOTA results on some benchmarks while being significantly faster than most other SOTA methods.", "questions": "In Figure 5 the method seems to predict left/right off-shoots - can the authors comment on why those are happening? They seem like very unlikely predictions.\n\nL472: what is a “standard normal distribution”?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The method produces SOTA results on standard benchmarks while greatly outperforming SOTA on runtime too.", "weaknesses": "I have two main concerns: [A] the method description is unclear, and [B] the two-stage approach is not well-motivated and not well-ablated. \n\n[A] I find the data description in LL158-161 confusing: could the authors clearly state what the input modalities are, i.e. are those per-person trajectories in 2D, are they collections of 2D per-person trajectories, how are they normalized/standardized - in what coordinate frame are they? \nThis also makes it difficult to follow the rest of the method description as it is unclear what the inputs/outputs are.\nThe authors make repeated use of the word “joint configuration space” (i.e. L165) - could they elaborate what that means?\n\nL161: what are those “M individual states”?\n\nThe authors use various words to describe there tracklets and the length, i.e. “stride”, “chunk”, “period”, “time chunk” - I would suggest to stick to one to make it easier to read.\n\nL182-L183: It is unclear here what the parameterizations are, i.e. what are the differences between \\theta_u, \\theta_m and \\theta_j —> this needs more details here and not just in the following chapters (Maybe also reference Figure 2 here..).\n\nCan the authors clarify what they mean by L070: “Human movements are Markovian up to certain frequencies” ?\n\n\n\n[B] Concerns wrt Two-stage approach\nI wonder if the split into Stage 1 (Bayesian Update) and Stage 2 is necessary. My main concern is that Stage 2 has no information about motion of the past beyond the previous segment S_{t-1}), while the segments seem to be rather short, as indicated in Table 7 (right side). I find it surprising that the method does not need to rely on past motion beyond a few frames to make more accurate predictions.\n\nFor example, just recursively applying the “Bayesian update” step (Stage 1) should perform better, as it has access to more historic information. I believe this is what the authors evaluated in Table 4 and 5 (“Ours (Stage 1)”) - could the authors confirm that this is indeed the case? They should more clearly describe this baseline / ablation in text. Also, it is surprising that this performs worse than the MRF as this method can exploit past motion better than the MRF. Could the authors elaborate what causes the Bayesian update step to be outperformed here?\n\nOn a similar note, if Stage 2 (self-evolution + interaction) seems to perform better, I wonder if the Stage 1 is even necessary as the initial Segment S_1 can be obtained from the historic motion sequence. I wonder how “Ours (Stage 2 only)” would perform in this case.\n\nA two-stage approach is more complicated and brittle and thus it is important to clearly motivate each part, which has not been done in the ablation.\n\n*Suggestions*\n\nAdd relative speeds to Table 3 where the fastest method is 1 and the others are multiples of it.\n\nFor the stride size in Table 7 (right) I would suggest to add the time in seconds as well", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g86m6ucMTB", "reviewer": "Reviewer_rTU5", "summary": "This paper presents a neutralized Markov random field-based motion evolution method for stochastic human trajectory prediction, which explicitly models the agent’s motion dynamics and crowd interactions, with lightweight and efficient learning as well as inference. Experimental results show that the proposed method is effective on multiple datasets, with robustness under noise disturbance.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes to use MRF and iteratively infer the stochastic distribution, which is different from existing explicit structured models.\n\n2. The numerical results are good, and the inference speed is promising.\n\n3. The paper is clearly written and easy to follow.", "weaknesses": "I put both of the Weaknesses and Questions here:\n\n1. The authors should elaborate more on why MRF that iteratively infers the stochastic distribution of future motions could achieve better performance, especially compared to existing methods.\n\n2. It appears to me that this paper clearly express what it does and how it does, but the motivation behind is not very clear, especially when readers are not in the same field, which makes it somewhat confusing why each proposed component would work.\n\n3. I think more visualized results of more baselines and the proposed method are needed for better comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oz2agJTRy4", "reviewer": "Reviewer_m1hq", "summary": "Modeling interactions and continuously changing intentions in trajectory prediction is challenging.\nTo address this, the paper proposes a Markov Random Field (MRF)-based motion evolution method. The trajectory is divided into time chunks, with the assumption that each trajectory segment follows a Markov chain sequentially.\nThe authors derive crowd motion evolution as a probabilistic distribution, which consists of a Bayesian update term, predicting the next state from each given observation, and a transition term. The transition term is further divided into a self-evolution term, which predicts the next state from each agent’s current state, and an interaction term, modeling relations between agents.\nAll of these terms are implemented using Conditional Variational Autoencoders (CVAEs).\nExperiments are conducted extensively on multiple pedestrian trajectory datasets, achieving state-of-the-art prediction performance and real-time inference speed.", "questions": "* On line 465, what does \"precludes stationary state\" mean? How is that related to robustness against noisy input? \n    - In addition, perturbation from Gaussian noise doesn’t seem related to stationarity.\n* The model uses CVAE in many parts, and CVAE is known to be vulnerable to posterior collapse in autoregressive methods, which could weaken stochastic prediction. What solutions were implemented to address this issue?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is easy to follow, and the proposed method is technically well-implemented.\n* The method is validated through extensive benchmark experiments. Its fast inference speed and high accuracy on the NBA and JRDB benchmarks are impressive, especially in JRDB-based experiments from an ego-centric view where noisy input was tested, and the prediction results appear plausible.\n* The experiment applying the proposed method to group reasoning is intriguing, and the visual results are very plausible.", "weaknesses": "* It seems difficult to claim state-of-the-art prediction performance. There \balready are some prediction models showing better performance on ETH-UCY and SDD than the baselines compared here. More convincing reasoning is needed to explain why the proposed method is necessary compared to these models.\n    - View Vertically (ECCV’22): 0.18/0.28, SICNet (ICCV’23): 0.19/0.33, Socialcircle (CVPR’24): 0.17/0.27  (: ADE/FDE on ETH-UCY benchmark)\n* I find it hard to fully grasp the novelty of the proposed method. As mentioned in the related work, there are already trajectory prediction methods that use the Markov property. Additionally, there are existing methods that construct a dynamic graph based on agents' positions to account for interactions between them (e.g., HiVT (CVPR’22), QCNet (CVPR’23)). The proposed approach seems like a combination of these two approaches, which might not meet the standards for ICLR.\n* I initially thought that using CVAE would introduce stochasticity, but it’s odd to see deterministic samplers are used during stage 2. In that case, why split it into two stages? Couldn’t deterministic sampling alone without CVAE sampling be trained and used for inference? The referenced Non-probability Sampling (Bae, CVPR’22) also suggests that deterministic sampling is preferable to random sampling, so I don’t understand why both random and deterministic sampling were used here. The ablation only compares with stage 1 alone; ablation results of state 2 only without CVAE sampling are needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)