ASTRA: A Scene-aware Transformer-based Model for Trajectory Prediction
Authors: Izzeddin Teeti
Aniket Thomas
Munish Monga
Sachin Kumar Giroh
Uddeshya Singh
Andrew Bradley
Biplab Banerjee
Fabio Cuzzolin
Venue: Accepted by TMLR ()

Abstract: We present ASTRA (A Scene-aware Transformer-based model for trajectory prediction), a light-weight pedestrian trajectory forecasting model that integrates the scene context, spatial dynamics, social inter-agent interactions and temporal progressions for precise forecasting. We utilised a U-Net-based feature extractor, via its latent vector representation, to capture scene representations and a graph-aware transformer encoder for capturing social interactions. These components are integrated to learn an agent-scene aware embedding, enabling the model to learn spatial dynamics and forecast the future trajectory of pedestrians. The model is designed to produce both deterministic and stochastic outcomes, with the stochastic predictions being generated by incorporating a Conditional Variational Auto-Encoder (CVAE). ASTRA also proposes a simple yet effective weighted penalty loss function, which helps to yield predictions that outperform a wide array of state-of-the-art deterministic and generative models. ASTRA demonstrates an average improvement of 27%/10% in deterministic/stochastic settings on the ETH-UCY dataset, and 26% improvement on the PIE dataset, respectively, along with seven times fewer parameters than the existing state-of-the-art model (see Figure 1). Additionally, the model's versatility allows it to generalize across different perspectives, such as Bird's Eye View (BEV) and Ego-Vehicle View (EVV).

Published in Transactions on Machine Learning Research (03/2025)
ASTRA: A Scene-aware TRAnsformer-based model for tra- jectory prediction
Izzeddin Teeti
izzeddin. detiadiintteii@gma.. com
Visual Artificial Intelligence Laboratory (VAIL), Oxford Brookes University
Aniket. Thomas* Indian Institute of Technology Bombay
andket.thommssWiib..ct
Munish Monga*
munisshSSmmongg@@maall..omm
Indian Instätute of Technology Bombay
Sachin Kumar'
sgiroh@ggaii.com
Indian Institute of Technology Bombay
Uddeshya Singh"
ud. udlxuddeshyy166@@gmaa com
Indian Institute of Technology Bombay
Andrew Bradley
abrudley@brookess.ccuu
Oxford Brookes University
Biplab Banerjee
getbiplab&gmail.com com
Center of Machine Intelligence & Data Science, Indion Institute of Technology Bombay
Fabio Cuzzolin
fabio. wwzzolin Strookks..rbrok
Visual Artificial Intellgence Laboratory (VAIL), Oxford Brookes University
Reviewed on OpenReview:
Abstract
We present ASTRA (A Scene-aware TRAnsfrrme--baad model for trajectory prediction), a light-weight, pedestrian trajectory forecasting model that integrates the scene context, spatial dynamics, social inter-agent interactions and temporal progressions for precise fore- casting. We utilised a U-Net-based feature extractor, via its latent vector representation, to capture scene representations and graph-aware transformer encoder for capturing social interactions. These components enabling the model to learn spatial dynamics and forecast the future trajectory of pedes- trians. The model is designed to produce both deterministic and stochastic outcomes, with the stochastic predictions being generated by incorporating a Conditional Variational Auto- Encoder (CVAE). ASTRA also proposes a simple yet effective weighted penalty loss function, which helps to yield predictions that outperform a wide array of saate-of-theart determin- istic and generative models. ASTRA demonstrates an average improvement of 27% 100 in deterministic/ sttccastic settings on the ETH-UCY dataset, and 26% improvement on the PIE dataset, respectively, along with seven times fewer parameters than the existing state- of-the-art model (see Figure D. Additionally, the model's versatility allows it to generalize across different perspectives, such as Bird's Eye View (BEV) and Ego- Vehicle View (EVV).
integrated to learn agent-scene aware embedding,
* Authors listed in alphabetical order and contributed equally.

---

Published in Transactions on Machine Learning Research (03/2025)
Avg minADE and minFDEw
0.4
(ScNet, 4.36)
(MID, 9.04)
(AgentFormer, 6.78)
(MemoNet, 5.281
(GroupNet. 3.14)
(Eqmotion, 5.54)
(Leapfrog, 11.2)
0.2
0.0
Number of Parameters (in Millions)
Figure 1: Comparison of average SeiinADDe//minnDDFiii against the number of parameters for various models on the ETH-UCY dataset. Each point represents a different model, with the model name and number of parameters millions indicated. Our model, ASTRA, achieves stale-of-the-arl results with the least number of parameters, demoustrating its efficiency and effectiveness in pedestrian trajectory forecasting.
1 Introduction
Motivation & Importance The pursuit of forecasting human trajectories is central, acting as cor- nerstone for devising secure and interactive autonomous systems across various sectors. This endcavour is crucial in a wide array of applications, encompassing autonomous vehicles, drones, surveillance, human-robot interaction, and social robotics. Furthermore, it crucial for predictive models to strike a balance between accuracy, dependability, and computational efficiency, given the imperative for these models function on in-vehicle processing units with limited capabilities. The challenge of trajectory prediction involves esti- mating the future locations of agents within a scene, given its past trajectory. This estimation task can be tackled either through Bird's Eye View (BEV) or comprehensive understanding of the scene, in addition to spatial, temporal, and social aspects that govern human movement and interaction.
View (EVV) perspectives. This demands a
Limitations in Existing Models To solve the prediction problem, various building blocks, including RNNs, 3D-CNNs, and transformers, have been employed address the temporal dimension, with trans- formers demonstrating superior efficacy Giuliari al. (2021a); Rasouli Kotserubal (2023). However, temporal modelling alone is unaware of the social behaviour of the agents within the scene, i.e. how agents interact with one another. In addressing the social dimension, methods such as Social Pooling [ang et al. (2021) h and Graph Neural Networks Kipf & Welling 20166) (GNNs) have been explored, with GNN emerging as the most effective Xu el al. (2023). Some researchers have integrated both transformers and GNN, either sequentially or in parallel, to refine the prediction paradigm Li et al. (2022) Chen et al. 2023); Jia et al. (2023): Liu et al. (2022). However, these approaches entail heightened computational burdens due to the resource-intensive nature of both GNNs and transformers. Furthermore, transformers, by their inherent design, may pose potential challenges in preserving information, they do not inherently accommodate the graph structure in their input. On the other hand, scene dimension, or scene embedding, delves into the interaction between an agent and its surroundings. Rasouli et al. (2021) and Mangalam et al. (2021) utilised semantic segmentating ion maps which enhanced the model's grasp of environmental context. Another aspect across all surveyed papers, however, is their tendency to focus exclusively on either BEV or EVV, rarely considering both like Yao et al. (2021). This narrow focus becomes particularly problematic in, e.g., unstructured environments where BEV might be available, limiting the applicability of these methods.
Contributions of ASTRA In light of these challenges, this paper introduces a lightweight model, coined ASTRA (A Scene-aware TRA TRAnsformer-basee model for trajectory prediction). By integrating U-Net-
2

---

Published in Transactions on Machine Learning Research (03/2025)
based key-point extractor Ribera et al. 2019): ASTRA captures essential scene features without relying explicit segmentation map annotations and alleviates the data requirements and preprocessing efforts highlighted earlier. This method also synergises the strengths of GNNs in representing the social dimension of the problem and of transformers in encoding its temporal dimension. Crucially, our approach processes spatial, temporal, and social dimensions concurrently, by embedding the graph structure into the token's 's sequence prior to the attention mechanism, rendering the transformer graph-aware. The model does so while keeping the complexity of the model minimal. To refine the model's ability to accurately learn trajectories, we implemented modified version of the trajectory prediction loss, incorporating penalty component (detailed in Section 4.5 This is in contrast to Yuan et al. (2021) which does not build graph and does not preserve the social structure; they distinguish between self-agent and all other agents, then they treat all other agents the same without encoding the positional or structural encodings.
Furthermore, distinct from the vast majority of models in this domain, our model demonstrates generalis- ability by being applicable to both types of trajectory prodiction datasets, BEV and EVV.
Our methodology underwent evaluation using renowned benchmark trajectory prediction datasets ETH Pellegrini et al. (2009a), UCY Lerner et al. (2007), and the PIE dataset Rasouli et al. (2019). The empirical findings highlight ASTRA's 's outperforming the latest state-of-the-art methodologies. Notably, our method showcased significant. improvements of 27% on the deterministic and 10% on the stochastic settings of the ETH and UCY datasets and 26% on PIE.
While maintaining high accuracy, ASTRA also features a significant reduction in the number of model's
parameters (Figure D FL.OPs (Floating Point Operations), inference running time & MACs (Multiply- Accumulate Operations) than the existing competing statc-of-the-art model Mao et al. 2023 Yuan et al.
p2021b.
The paper's highlights are as follows:
1. A lightweight model architecture that is seven times lighter than the existing state-of-the-art model, tailored for deployment on devices with limited processing capabilities while maintaining state-of- the-art predictive performance.
2. A loss-penalization strategy that enhances trajectory prediction, featuring weighting trajectory loss function that dynamically adjusts penalty progression in response to prediction challenges.
Utilisation of the Scene-aware embeddings with a U-Net-based feature extractor to encode scene representations from frames, addressing critical aspect often overlooked in recent works.
graph-aware trausformer eucoder that contributes to generating Agent-Scene aware embedding for improved prediction accuracy, ensuring informed inter-agent interaction capture.
Paper Organisation The paper is structured as follows: Section reviews related work trajectory prediction. Sections and defines the problem and details ASTRA's architecture and key innovations, re- spectively. Section describes the experimental setup. Section presents empirical findings and performance comparisons. Finally, Section summarizes contributions, limitations, and future directions.
2
Related Work
2.1
Stochastic vs. Deterministic Approaches in Trajectory Prediction
The trajectory prediction problem is usually addressed in two ways: stochastic predictions Mao al. (2023 Yuan et al. (2021): Xu et al. (2022b) Yao et al. (0221) and deterministic predictions Helbing & Molnar (1955]; Pellegrini et al. (2009b); Alahi et al. (2016). ) A deterministic approach assumes a fixed future and produces only single, most probable trajectory per input motion, making it suitable for scenarios where uncortainty is minimal or a single best estimate required. In contrast, a stochastic approach acknowlodges uncertainty in future motion and generates multiple possible trajectories for the same input, capturing the in- herent variability in human movement. Stochastic approaches utilizes generative techniques like Conditional
3

---

Published in Transactions on Machine Learning Research (03/2025)
Variational Auto-Encodees (CVAEs) Yuan et al. 22021 y Yao et al. (2021) p. Generative Adversarial Networks (GANs) Huang et al. 20011) Normalizing Flows Bhattacharyya et al. (2020). or Denoising Diffusion Prob- abilistic Models Mao et al. (2023) to introduce randomness into the prediction process, thereby generating diverse future trajectorics that better represent the distribution of possible pedestrian movements. ASTRA, like some prior works Xu et al.
(2023) Yao et al. (2021 Salzmann el al. (2020). supports both deterministic
and stochastic predictions.
2.2 Social & Scene-aware Modelling
Approaches to Agent-Agent Interaction Modelling The social dimension focuses on capturing agent- agent interactions, emphasizing how individuals or objects influence each other's movements within a shared space. Notably, some methodologies incorporate social pooling, concurrently with attention mechanisms Pang et al. (2021). Algorithms in this domain predominantly leverage various forms of Graph Neural Networks (GNNs) to encapsulate the social dynamics among agents. Some methods employ fully connected undirected graph, encompassing all scene agents Xu et al. (2023 Gilles et al. 22021) hs Kosaraju et al. (2019)
This approach, albeit comprehensive, escalates exponentially with the number of agents (nodes). Conversely, other methods opt for sparsely connected graphs, establishing connections solely among agents within a
proximal range, thereby reducing the linkage count substantially FFang et al. (2021) ; Girase et al. (2021)
Salzmann ct al. (2020); WWeng et al. 2021). In a similar vein, Yuuan et al. (2021); Salzmann ct al. (2020)
proposes sparse, directional graphs, predicated on the premise that different agent types possess varying perceptual ranges. Regarding the optimal depth of GNN layers, Addanki et al. (2021) advocate for deeper graphs to enhance performance. This stands in contrast to the findings of Weng et al. (2021) and Liu et al.
(2020), who posit that two layers are optimal. Nevertheless, this depth increases computational demands, particularly when agent nodes are numerous, posing challenges for autonomous vehicle applications reliant on edge devices for processing.
Scene Embeddings The scene dimension, extracted from the video frames, includes the low-level rep- resentation of the physical enviromment, obstacles, and any elements that could affect the agent's path, ensuring a comprehensive understanding of both social and environmental factors in predicting movement trajectories. Rasouli et al. (2021) and Mangalam el al. (2021) capture scene dimension with the help of semantic segmentation to delineate visual attributes of varied classes, subsequently elucidating their interre- lations via attention. However, obtaining a panoptic segmentation mask, might not be always feasible. Also, this approach introduces considerable dependency the availability of additional segmentation maps, presenting a challenge in terms of data requirements and preprocessing efforts.
2.3 Temporal Dimension
Understanding the trajectory history of an agent significantly augments the predictive accuracy regarding
its potential future path. Predominantly, ego-camera-basee models are tailored to shorter temporal horizons
and employ 3D Convolutional Neural Networks (3D-CNNs) Fang et al. (2021); |; Kotseruba et. al.
(2021
Some research, instead, adopts Hidden Markov Models (HMMs) for temporal analysis Cai et al. 2020
For scenarios necessitating extended time horizon considerations, more intricate structures are proposed.
including Transformers Yuan et al. h0021; Xu et al. (2023); Giuliari et al. (2021b) Chen et al.
2022 and
various forms of Recurrent Neural Networks (RNNs) Girase et al. (2021), including Long Short- Term Memory networks (LSTMs) Rasouli & Kotseruba (2023); Bhattacharyya et al. (2021): ; Fang et al. (2021) and Gated
Recurrent Units (GRUs) Gilles et al. (2021). Both Transformer and RNN-based models have exhibited
superior performance, often achieving state-of-the-art results in this domain. However, some of these models tend to address the temporal dimension in isolation from the social context. This segregated approach potentially results in information loss and contributes to an increased computational load, necessitated by the addition of separate components to handle the social dimension. Consequently, there emerges a pressing demand for integrated models capable of concurrently processing both temporal and social dimensions. A promising direction in this regard is the development of graph-aware transformers, which encapsulate the essence of both temporal dynamics and social interactions within unified framework.
4

---

Published in Transactions on Machine Learning Research (03/2025)
2.4 Graph-aware Transformers
Graph-aware transformers aim to compound the benefits of graphs (with their associated social embeddings) and of transformers, with their acclaimed attention mechanism and temporal embeddings. Notably, these advancements have predominantly catered to graph-centric datasets like ACTOR (Tang et al. (2009) and
CHAMELEON SQUIRREI. Rozemberezki et. al. (202T). Direct. application of graph-aware transformers remains untouched in pedestrian trajectory forecasting, with prevalent methodologies leaning towards trans- formers processing embeddings emanating from graphs Li et al. (2022) Chen et al. (2023); |; Jia et al. (2023)
There has been a discernible preference for using GNN and transformer blocks, rather than fully-integrated graph-aware transformers.
A comprehensive evaluation of numerous contemporary graph-transformer models across three graph-centric datasets is conducted in Müller et al. (2023). The analysis reveals consistent pattern: models employing Random Walk for structural encoding exhibit superior performance across all tested datasets. Building on this empirical evidence, our approach utilizes Random Walk to encode the pedestrian graph, which is then seamlessly integrated into the transformer architecture. This integration is designed to yield a graph- aware transformer, thereby enhancing the model's capability to effectively capture and interpret complex pedestrian dynamics within various environments. To the best of the authors' knowlodge, this the first work towards utilising graph-aware transformer solve the trajectory prediction problem, opposing many methods which graphs along with transformers.
3 Problem Formulation
The objective of trajectory prediction is to forecast a pedestrian's future position based on their observed historical trajectory. Formally, the historical trajectory of A larget agents is given as a sequence of coordi-
nates:
X= (X E (1, Tobs): a (1,2..... A)).
where Xf represents the position of agent at time t over the past Tots time steps. For bird's-eye view (BEV) datasets, X; consists of 2D coordinates {xf vf). For egocentric view (EVV) datasets, it consists of bounding box coordinates {-f vi, xf,,, v2,)) Additionally, the corresponding visual input frames/imager are given as = (Te E (1,2, Tobs)).
Deterministic Setting The goal of ASTRA to prodict deterministic future trajectorics for the pedes- trians. the deterministic setting, the model predicts future trajectory coordinates for each of the A agents over the next Tpred time steps:
8 8 tee (1,2, (1, Tprea); E (1,
(2)
where Pre represents the predicted position of agent at future time t, and the ground truth trajectory is denoted Y (Y? t (p,ea... (1.
Stochastic Setting In the stochastic setting, the goal is to learn generative model parameterized by 0 as XI) which, given the historical observations X and I, generates K possible future trajectories:
y= (1 (() ..... ((K))
(3)
This enables the model to capture the inherent uncertainty in pedestrian motion by producing diverse trajectory hypotheses.
5

---

Published in Transactions on Machine Learning Research (03/2025)
Sinosuidal Temporal Encoding
Frezen
Trainable
ENC
DEC
Latent Representaion
Sorn
3 Pemyaraa
Scene-sware Transformer Encoder
v Sieee
Frames
Keypoint Extractor
T
Concatenate
Imagea
Predicted Trajectories
X 22
Spettel
detorministic predictions
$ Hosiel
diaatol
Agent-aware Transformer Encoder
Apenis
Serne Agents
Opettall
Deadder
Yosgond
Concatenate
Concaterate
CVAE
LowP
coordinates
Foses
for stochasee predictions
Binosuidal Temporal
Food Sequaenca Coordinates
Encoding
Extracted Future
Trajectories
Fess coordinates
Actual Trajactories
Figure 2: Model Architecture. Overview of ASTRA model architecture for pedestrian trajectory forecasting.
4
ASTRA Model
4.1
High-Level Overview
The encoder part of our model consists of two main components: A scene-aware component and an agent- awarc component. While the former is dodicated to encoding the scene, and encapsulating the contextual details, the latter focuses on encoding the spatial, temporal, and social dimensions of the agents, as shown Figure 2 The output from these two components is aggregated before being decoded generate predicted future trajectories for deterministic stochastic predictions.
4.2 Scene-aware Transformer Encoder
ENC
DEC
U-Net Based
Tobs
Keypoint Extractor
Tobs
Frames
Corresponding Keypoints
Figure 3: Pretraining U-Net based keypoint extractor.
In order to learn essential information about the scene's spatial layout and the positional dynamics of agents within it, the U-Net Ronneberger et al. (2015) is pre-trained to predict the location of pedestrians in the frame using the method detailed in Ribera et al. (2019), which utilizes specialized loss function, Weightod Hausdorff Distance. This, in turn, helps the model learn a representation of the scene context. (Figure B. focusing the pedestrians in the scene. This method of pre-training method allows the extraction of essential scene features involving pedestrians ssnnee keypoints or locations of the pedestrians are always available in form of trajectories without relying on explicit segmentation map annotations. This not only alleviates the need for labor-intensive annotation processes but also enables more efficient training on datasets where only pedestrian locations are provided.
A latent representation of pedestrian characteristics obtained using a pre-trained U-Net encoder (Figure 2 this latent vector include some crucial characteristies like spatial groupings and interactions with

---

Published in Transactions on Machine Learning Research (03/2025)
the environment. The Grad-CAM visualizations (Figure 4 highlight this capability, showing that the pre- trained model pays attention to regions with high likelihood of pedestrian activity. This step is crucial as the U-Net extractor possesses the proficiency to discern both labelled and unlabelled pedestrians, depicted in green and red, respectively in Figure 4a
More sophisticated schemes generate the scene representation, like transformer-basee architectures, and fusing social representations via gated cross-attention can also be considered but we leave exploring possibly more effective and sophisticated architecture designs as future work. The U-Net-based keypoint extractor is frozen after the pretraining (Figure 2 when used in ASTRA model architecture.
(a) Original Image
(b) Grad-CAM Image
(c) Overlayed Grad-CAN
Figure 4: Grad-CAM visualizations: In (a); the red circle indicates unlabelled pedestrians, while the green square highlights labelled pedestrians. In (b), the U-Net-based keypoint extractor focuses on unlabelled pedestrians as well, thereby capturing scene context from them too.
The latent representation of all frames (Vescene) treated as input tokens to the scene-aware single-layer transformer encoder (Tseene-aware), which in turn generates sceno-aware emboddings (Doonne) for cach frame. The single-layer transformer encoder architectural choice significantly contributes to the lightweight nature of our model. The resulting scene embedding is:
V Scene I Scene (T Encoder (I))
(4)
Scenne Tseene-aware S[eee PTmmporal)
(5)
where the latent representation past input frame images (I) projected using a Multi-Layer Perceptron (MLP) layer (T ([sscnn), and T (.) denotes the encoder part of the U-Net.
The temporal encoding PTemporal, crucial for capturing the temporal dynamics within the observed frames, adopts the design of the traditional positional encoding Vaswani et al. (017) and follows the work of Yuan
al. 2021
4.3
Agent-aware Transformer Encoder
The second component is dedicated to encoding the different dimensions of each agent for all agents in the scene.
Spatial Dimension The spatial coordinates of each agent are linearly projected using an MLP layer (Tspatial) to obtain the spatial encoding (Pspatial). The spatial encoding belongs to R. Ax where A is the number of agents and Dsp is the spatial feature dimension.
Pspatial = Spattill (X) E BAx
(6)
Temporal Dimension Relying solely on spatial embeddings insufficient, as agents occupying the same location in different frames would have identical spatial representations. To address this, we incorporate temporal encoding to distinguish agents across time.

---

Published in Transactions on Machine Learning Research (03/2025)
To model temporal dependencies within pedestrian trajectories, introduce temporal encodings for both agents and the scene. These encodings guide the network in capturing sequential informat ion, similar to positional encodings in the Transformer architecture Vaswani et al. (2017). The temporal encoding Temmprra belongs to IR Ax Pr, where A the number of agents and Dr the temporal feature dimension.
sin 10000
if i is even,
"Temporal(t
i is odd. E RA
cos
10000
Social Dimension Having the spatial and temporal dimensions of the agents is still not enough to un- derstand their interaction in the scene. To caplure the social dimension this multi-agent environment, we generate a fully connected undirected graph between agents, which the nodes are the agents' locations, and the edges between agents are the reciprocal of the distance. Consequently, the closer the agents are to each other, the stronger the link between them. Formally, we represent the social dimension using a graph G F),, where V is the set of agents and E is the collection of edges, with weights
eij d(us, UT))
(8)
where d(u;, 1z) is the distance between agents and l/y.
Subsequently, Random Walk Positional Encodings (RWPEs) Dwivedi et al. (2021) are used to capture the structural relationships between nodes in the graph, such as their proximity and connectivity patterns. RWPE leverages random walks to encode positional information, allowing the model incorporate relational dependencies between agents in daiven manner. By considering the number of paths and transition probabilities between nodes, RWPE provides meaningful representation of spatial interactions in dynamic environment.
These RWPEs are then projected using a separate MLP (Toociia) to obtain social encodings (Dsocial). The social encoding belongs to Ax Dao, where A the number of agents and Dso is the social feature dimension:
Social = I'social (RWPE(G)) E E AxDs.
(9)
This preserves the graph structure of the agents while making the transformer encoder graph-aware. Fur- thermore, our method empowers the network to determine the significance of each agent relative to others autonomously.
While we do not claim be the first use transformers or graphs, we claim that we are the first integrate RWPE directly into transformer tokens, creating graph-aware transformer in the context of trajectory prediction.
Aggregating After computing the spatial, temporal, and social representations for each agent, our model concatenates them along the agent dimension. This ensures that each agent has a single feature vector con- sisting of timestamp (temporal encoding), social feature vector, and spatial feature vector. The con- catenated representation is then processed by an agent-aware single-layer transformer encoder (T (TAgent-aware), which generates the final agent-aware embedding ( Agegente).
Agents T aggent-aware (Deppttill: Pempporll: Social]) E ( Dr
(10)
4.4 Decoder
Stochastic Decoding To generate multiple stochastic trajectorics, we learn a generative model, ( (o()) X) for which we adopted CVAE (Conditional Variational Auto Encoder). We train CVAE, sim- ilar to Yuan et al. (2021) and Yao et. al. (202)) to learn the inherent distribution of future target trajectories

---

Published in Transactions on Machine Learning Research (03/2025)
conditioned on observed past trajectories, by utilizing a latent variable Z. CVAE consists of three compo- nents prior network (po(2(X)), recognition network (or posterior) (no X)) and generation network (Y1()))) parameterized by 0, @ and respectively. Here X is the latent representation of X and I, obtained after concatenating & Aggents and PSeene and Y is the output of the generation network and are the predicted future trajectories.
For CVAE, prior distribution (po(=[X)) is parameterized by (u (o²)²). The approximate posterior distri- bution (Q(() is parameterized by Mi(ug (oz)²), where μe and (o?)² represent the mean and variance of the prior distribution and μº and (ag)² represent the mean and variance of the posterior distribution. During training, we sample latent variable (z) from the recognition network (posterior distribution) and feed it to the generation network (Y1)), whereas during lesting we sample z from the prior network (prior distribution). We use the reparameterization trick to sample through the mean and variance pairs of (ng, (o and (ng, (o?)²), respectively. KL divergence Loss (LKL) help in minimizing the difference between the distribution of latent variable(z) of prior and recognition work.
K samples are drawn from the learned distribution and decoded using an MLP to obtain future trajectories. We optimize the parameters of the networks using the KI. divergence which ensures that the prior network implicitly learns the dependency between future trajectories (Y) and past trajectories (X)
LKI. = Dxrlga(Z XXX) (2 X)) = ([LN((H,,, [| )1)
(11)
Deterministic Decoding For deterministic predictions, CVAE is skipped and the outputs of both the scene transformer (DScene) and the agents' transformer (0xgens) are concatenated and directly passed
through an MLP decoder (I Decoder) to produce future trajectories (Y) of the agents in the future frames as shown in Figure 2 namcly:
Y Decoder ( [PSoone; & Agents])
(12)
4.5 Weighted Loss Function
We introduce a weighted-penale strategy that can be applicd to common loss functions used in trajectory prediction such as MSE and Smooth L1 Loss. The application of this strategy is through a dynamic penalty function w(t), designed to escalate or de-escalate the significance of prediction errors as one moves further into the future. The definition of the weighted loss function is given by:
Tpred
weiightd (,,))
Σ w(t) -(Y,,
(13)
Where ý and Y are the predicted and actual trajectories respectively, Tpred denotes the number of prediction timesteps, w(() represents the dynamic weighting function at time i, and L(Y,, Y.) is the predefined loss function (e.g., MSE or SmoothL1 Loss) applied to the predicted and true positions at each time step t.
In time series data, predictions tends to increase, leading to higher errors. Motivated by this intuition, we initially penalized the predictions using linear and quadratic loss functions. These approaches showed improvements in overall prediction accuracy. However, upon closer analysis of the results, we observed that in some trajectorics, there was a noticeable offset in the earlier parts of the predictions. To address this issue, we experimented with a parabolic weighting function for the penalty. Empirically, this approach outperformed the lincar and quadratic strategies, yielding the most balanced and accurate predictions across the trajectories.
move further into the future relative to the last observed data, the drift in
The weight function w(t) (generally defined as w(t)= f(t, ((red, a, B)) is designed to be versatile, accommo- dating a spectrum of mathematical formulations that align with the specific needs of the predictive model. We used the parabolic weighted penalty, defined as:
w(t) = (a- - B) (2 Tpred (1-1) -1)
1) 2 +
-1)
+ B === 3 2.
(14)
T Tpred
9

---

Published in Transactions on Machine Learning Research (03/2025)
where and 3 are parameters that establish the bounds of the weighting function (during our experimen-
tations and hyperparameter tuning, we found that = 4 and B 1, as the optimal values), and f is an adaptable function that governs the progression of weights at each timestep t.
In particular, the function w(t) may be selected from various mathematical forms, such as linear, parabolic, or quadratic (discussed more in the supplementary materials). The choice of function enables the model adjust the penalty progression in alignment with the anticipated prediction challenge at each timestep.
Hence, in the stochastic setting, the final loss consists of the weighted loss and the KL divergence loss from the CVAF. In contrast, in the deterministic setting, since the CVAE is not used, the KT. term is omitted from the final loss:
Lfnal Leeighted (Y,Y) + CKL.
(15)
5 Experiments
5.1 Datasets & Evaluation Protocols
For comprehensive evaluation, benchmarked our model on three trajectory prediction datasels; namely, ETH Pellegrini et al. (009aa), UCY Lerner et al. (2007). and PIE dataset Rasouli et al. (2019).
ETH-UCY (Bird's Eye View) ETH and UCY offer bird's-eye view of pedestrian dynamics in urban settings, including five datasets with 1,536 pedestrians across four scenes. For evaluation, we used their standard protocol; leave-one-out strategy, observing eight time steps (3.2s) and predicting the following steps (4.8s).
PIE (Ego-Vehicke Vehicle View) In contrast, the PIE dataset provides an Ego-vehicle perspective, containing 6 hours of ego-centric driving footage, along with bounding box annotations for traffic objects, action labels for pedestrians, and ego-vehicle sensor information Rasouli et al. (2019). A total of 1,842 pedestrian samples are considered with the following split: Training 50%), Validation(40% and Testing(10%) Rasouli let al. 2019 Model performance is evaluated based on a shorter observational window of 0.5 seconds and a prediction window 1 second, providing insights into the model's capability in rapidly evolving traffic scenarios Rasouli & Kotseruba 20223
5.2 Evaluation Metrics
We used the standard evaluation metrics of ADE and FDE for ETH-UCY deterministic settings and minADE and minFDE for their stochastic setlings. CADE, CFDE, ARB and FRB for PIE dalaset. The supplementary material explains these metrics.
5.3 Hyperparameter Tuning & Hardware Settings
The hyperparameters were initially tuned on the ETH subset the ETH-UCY dataset and subsequently applied across all subscts. The key architoctural hyperparameters used our model are as follows: spatial embedding dimension (Pspatial E R)) U-Net scene latent representation (eeeeee R,,,, temporal embed- ding dimension (#remporal R²), and random walk embedding (osocial E R$). The transformer encoder consists of a single layer with two attention heads and dropout rate of 0.2. To keep the model lightweight, MobileNet v2 Sandler et al. 22018) encoder layers are used for U-Net to reduce computational overhead while maintaining feature extraction efficiency.
For training, we employ the AdamW optimizer with weight decay of 10-4 over 200 epochs. A cosine annealing scheduler is used, starting with an initial learning rate of 1 10-3, All experiments were conducted an NVIDIA DGX A 1100 system with 8 GPUs, each equipped with 80 GB of memory.
10

---

Published in Transactions on Machine Learning Research (03/2025)
6 Results & Discussion
6.1 Quantitative Results
ETH-UCY For ETH-UCY, compared our model results against several baselines. These comparisons are presented in Table 1 and Table 4 which contains results primarily sourced from the EqMotion (CVPR
2023 for stochas- 2023) Xu et al. (2023) for deterministic predictions and LeapFrog (CVPR 2023) Mao et al.
tic predictions respectively. It is important to note that to provide a thorough comparative framework, we independently computed and included additional models Wang et al. (2022); Yao et al.
(2021 to their re-
spective tables as they were not originally part of the Eq or LeapFrog analysis. Our model advances the state-of- the-arrt on ETH-UCY, outperforming the EqMotion Xu al. (2023) by improving predictive accuracy by approximately 27% on average for deterministic predictions as shown in Table | and approxi- mately 10% on average improvement over LeapFrogMao al. (2023) for stochastic predictions as shown in highlighting the efficacy of our approach in diverse scenarios. While ASTRA maintains superior performance across most benchmarks, there are, however, some exception cases, like in the Hotel and Univ scenes, where a notable proportion pedestrians remain largely stationary throughout both the observation and prediction windows, resulting in slightly inferior performance in these scenarios. We also highlight the effectiveness of utilizing frame encodings from U-Net, demonstrated in Table
Table !
PIE Similarly, we benchmarked our model against various established models for the PIE dataset. The comparative analysis summarized Table ? with the reference results taken from PedFormer Rasouli Kotseruba 2023), demonstrating an average improvement of 26%.
Table 1: Deterministic Results: ADE/FDE results for ETH-UCY baselines. Best in bold, second best underlined.
Model
ETH
Hotel
Univ
Zaral
Zara2
Average
Linear
1.33/2.94
0.39/0.72
0.82/1.50
0.62/1.21
0.77/1.48
0.79/1.59
S-LSTM Alahi et al.
2016
1.09/2.35
0.79/1.76
0.67/1.40
0,47/1.00
0.56/1.17
0,72/1.55
S-Attention Vemula al.
2018
1.39/2.39
2.51/2.91
1.25/2.54
1.01/2.17
0.88/1.75
1.41/2.35
SGAN-ine ind Gupta et al. 2018
Gupta
1.13/2.21
1.01/2.18
0.60/1.28
0.42/0.91
0.52/1.11
0.74/1.54
Traj Salzmann et
1.02/2.00
0.33/0.62
0.53/1.19
0.44/0.99
0.32/0.73
0.53/1.11
Trans Giuliari et al.) 2021
1.03/2.10
0.36/0.71
0.53/1.32
0.44/1.00
0.34/0.76
0.54/1.17
MemoNe Xu et al. 1202bb
1.00/2.08
0.35/0.67
0.55/1.19
0.46/1.00
0.37/0.82
0,55/1.15
SGNet Wang al. 2022
0.81/1600
0.41/0.87
0.58/1.24
0.37/0.79
0.31/0.68
0.56/1.04
Eq Xu et al.
2023
0.96/1.92
0.30/0.58
0.50/1.10
0.39/0.86
0.30/0.68
0.49/1.03
ASTRA (Our))
0.47 0.47/0.82 00.82
0.29/0.55 0.56
0.55/1.00
0.34/0.71
0.24/0.41
0.38/0.70
Table 3: Ablation: ADE/FDE for penalised vs. umpenalised loss functions on UNIV dataset us- ing SOTA ASTRA's configuration
Table 2: Rosults for PIE dataset.
Model
CADE
CFDE
FRB
FOD
Yao al.
164.53
143.69
FP[
Yagg al
2013
36.06
132.23
Loss
Normal
Penalised
B-LST%
M Biattschaarva aL.
2018
27.09
66.74
37.
75.87
PIE...
Rasoull et al.
2019
21.82
53.63
27.16
55.39
MSE
0.58/1.13
0.57/1.00 2.55/1000
PIE
2019
19.50
45.27
24.40
49.09
SmoothL1
0.57/1.15
BiPed
Rasouii et
A120
15.21
35.03
19.62
39.12
PedForrmed Rasoul: & Kolseruba
2023
13.08
30.33
15.27
32.79
ASur)
9.91
22.42
18.32
17.07
6.2 Efficiency
Regarding the computational side, ASTRA has seven times fewer trainable parameters than the existing SOTA model LeapFrog Mao et al. (2023) shown in Figure It important note that the parameter count reported for ASTRA in Figure includes the parameters from the U-Net, which otherwise actually
11

---

Published in Transactions on Machine Learning Research (03/2025)
Table 4: Stochastic Results: mo and minFDE20 results for ETH-UCY baselines, Best in bold,
second best underlined. NP- means umpenalised.
Model
ETT
Hotel
Univ
Zaral
Zara2
Average
Social-GAY Gupta et
(2018)
0.87/1.62
0.67/1.37
0.76/1.52
0.35/0.68
0.42/0.84
0.61/1.21
NMMP Hu et a.. [2020]
0.61/1.08
0.33/0.63
0.52/1.11
0.32/0.66
0.43/0.85
0.41/0.82
STAR Yu al. [2020a]
0.36/0.65
0.17/0.36
0.31/0.62
0.29/0.52
0.22/0.46
0.26/0.53
PECNet Mangalam et al.
2020
0.54/0.87
0.18/0.24
0.35/0.60
0.22/0.39
0.17/0.30
0.29/0.48
Trajectron
Salzmann el.
2020
0.61/1.02
0.19/0.28
0.30/0.54
0.24/0.42
0.18/0.32
0.30/0.51
BiTrap-NP Yao et al.] 2021
0.55/0.95
0.17/0.28
0.25/0.47
0.22/0.44
0.16/0.33
0.27/0.49
MetoNNt Xu et al.
(2022b)
0.40/0.61
0.11/0.17
0.24/0.43
0.18/0.32
0.14/0.24
0.21/0.35
GroupNet Xu a..
2202a
0,40/0.76
0.12/0.18
0.22/0.41
0.17/0.31
0.12/0.24
0.21/0.38
SGNet Wang et al.
2022
0.47/0.77
0.20/0.38
0.33/0.62
0.18/0.32
0.15/0.28
0.27/0.47
MID Gu et al. (2022)
0.46/0.73
0.15/0.25
0.26/0.49
0.21/0.39
0.17/0.33
0.25/00.44
Agentformer Yuan et al.
(2021]
0.45/0.75
0.14/0.22
0.25/0.45
0.18/0.30
0.14/0.24
0.23/0.39
EqMotion Xu et. (2023
0.40/0.61
0.12/0.18
0.23/0.43
0.18/0.32
0.13/0.23
0.21/0.35
Leapfrog Mao et al.
2023
0.39/0.58
0.11/0.17
0.26/0.43
0.18/0.26
0.13/0.22
0.21/0.33
Fine Wang a..
2024
0.35/0.60
0.11/0.18
0.22/0.48
0.15/0.30
0.12/0.25
0.19/0.36
ASTRA(Non Penalised)
0.37/0.49 /0.49
0.24/0.34
0.37
0.23/0.32
0.16/0.23
00.3/0.38
ASTRA (Without Image Input)
ASTRA(Withord
0.29/0.30 /0.39
0.18/ 0.29
0.29/ 0.43
0.17/0.26
0.14/0.2
0.21/ 0.31
ASTRA(With Image Input)
0.27 /0.36
0.17/0.25
0.28/0.41
0.25
0.16
0.20/0.28
frozen during the trajectory prediction phase. The complete ASTRA architecture, including all compo- nents, comprises 1.56M parameters, 652M FLOPs (floating-point operations), and 326M MACs (Multiply- Accumulate Operations). Even in configurations without U-Net pretraining, the deterministic variant has 13.52K parameters, 15.87K FLOPs, and 7.935K MACs, while the stochastic variant has 141.4 18. parame- ters, 165.416K FLOPss, and 82. 70.708K MACs. This flexibility allows ASTRA to scale efficiently under strict resource constraints.
Additionally, we computed the model's running time, which takes 19.23ms to generate predictions for future trajectorics. To further assess ASTRA's 's efficiency, we compare rumning time, FLOPs, MACs, and number of parameters with three other models: AgentFormer, transformer-based architecture similar to ours; Leapfrog, state-of-the-art approach; and GroupNet, which has the fewest parameters among all compared models. As shown in the Table 5 ASTRA demonstrates strong efficiency compared to these alternatives.
Table Model Parameters and Inference Time comparison. (M denotes Million and ms denotes Millisecond)
Model
Parameters (M)
FLOPS (M)
MACs (M)
Inference Time (ms)
GroupNet
3.14
806
403
105.78
Agentformer
6.78
3084
1542
521.34
Leapfrog
11.2
5695
2848
28.91
Astra (Ours)
1.56
652
326
19.23
6.3
Ablations
In the ablation study presented in Table we evaluated the contribution of each component in our trajectory prediction model to ascertain their individual and collective impact the performance metrics the ETH-UCY (UNIV) dataset. Initially, the model incorporated only spatial information, which served bascline for subsoquent enhanccments. The sequential integration of temporal and social components yiclded successive improvements, demonstrating their respective siguificance in capturing the dynamics of agent movement. The addition the data augmentation technique (detailed in the supplementary material) further refined the model's performance, illustrating the value of varied training samples in enhancing generalization capabilities. Moreover, the incorporation of U-Net features contributed the importance of context-aware embeddings in accurately forecasting agent trajectories. This progression
lepp forward, highlighting
12

---

Published in Transactions on Machine Learning Research (03/2025)
emphasizes the synergistic effect of combining heterogencous data representations to capture the manced patterns of movement within scene.
We also investigate the effect of using & Single Transformer Encoder versus Dual Transformer Encoders (Table 6 last two rows). For the Single Encoder setup, we merge Tsceue-aware and TAgeut-aware into a unified transformer encoder. While this configuration captures a joint representation of scene features and agent dynamics, it inherently limits the model performance, as can be inferred from Table 6
Table 6: Ablation: ADE/FDE & minADEgg/minFDEa /iinFD22 with variations in ASTRA's model components UNIV dataset (where V: Component enabled, Component disabled).
U-Net
Transformer Encoder
mi
Spatial
Temporal Augmentation Social Features
ADE/FDE
minFDE20
Single
1.05/1.66
0.43/0.63
X
X
Single
0.86/1.47
0.39/0.54
Single
0.67/1.17
0.31 0.31/0.48
v
V
X
Single
0.66/1.12
0.29/0.43
Single
0.74/0.61
0.98/1.41
V V
Dual
0.55/1.00
0.28/0.41
The ablation study also extended to the evaluation of loss functions, comparing the effects of penalised versus umpenalised approaches. Penalized loss functions, designed to focus the model's attention on more critical prodiction horizons, proved to be more effective in refining the predictive accuracy, as outlined Table 3 and Table (ASTRA(NP)) for deterministic and stochastic setting respectively and the same can be observed in Figure
Additionally, we compared the CLIP encoder -with added lincar layer for finc-tuning in an end-to-end fashion -with our U-Net-based encoder, as shown in Table 7 Our model consistently performed on par with or better than CLIP while using significantly fewer parameters, MACs, and FLOPs, highlighting its efficiency. This can be attributed to the pre-training, which enables the model to focus specifically on pedestrians.
Table Efficiency and quantitative (minADE2o and minFDF20) results for ETH-UCY baselines for different image encoders. (Million denotes M)
Parameter
FLOPs
MACs
Encoder
Count (M)
(M)
(M)
ETH
Hotel
Univ
Zaral
Zara2
Average
CLIP Radford
2021
149.77
22540
11270
0.26/0.37
0.18/0.28
0.34/0.48
0.18/0.29
0.15/0.21
0.22/0.324
UNET (Ours)
1.56
652
326
0.27/0.36
0.17/0.25
0,28/0.41
0.15/0.23 0.13/0.16
0.2/0.282
6.4
Qualitative Analysis
We can clearly see the results of our prediction from Figure for deterministic predictions and Figure F for stochastic prediction. Figure 7 exemplifies the proximity of our model's results to the ground truth, it also shows how using the weighted penalty strategy has yielded better results than the unpenalised one, highlighting the improved effectiveness of our strategy.
7 Conclusion & Future Work
We presented ASTRA, model in the domain of pedestrian trajectory prediction, that outperforms the cxisting statc-of-the-art models. This advancement renders ASTRA particularly suitable for deployment on devices with limited processing capabilities, thereby broadening the applicability of high-accuracy trajectory prediction technologies. ASTRA's adeptness in handling both BEV and EVV modalities further solidifies its applicability in diverse operational contexts. With the ability to produce deterministic and stochastic results, it enhances the predictive robustness and situational awareness of autonomous systems. Moving forward, we aim to extend the capabilities of the ASTRA model beyond pedestrian trajectory prediction to encompass
13

---

Published in Transactions on Machine Learning Research (03/2025)
Past Tejectory Acuual Traectory Predicted Tryectory
(a) BEV
(b) EVV
Figure 5: Sample images of the deterministic prediction from BEV datasets (a.) (ETH and UCY) and EVV dataset (b.) (PIE). The Red and Yellow bounding box indicates the ground-truth and predicted final position respectively and the Blue bounding box indicates the start position.
Part Trajectory
Trajectory
Trajectrr
Predicted Trajectory (Pacaline)
Trajectory (Unpenelise)
Figure 6: Sample images of the stochastic pre- dictions from ETH-UCY Dataset.
Figure 7: Visual comparison penalised and unpenalised loss on ETH-UCY, showing the en- hanced performance of the former.
broader range of non-human agents. This expansion will involve adapting the model to understand and predict the movements of various entities within shared environments using more sophisticated architectural design choices to encode the seene and its fusion with social dimension. By broadening our focus, we hope to contribute to the development of truly comprehensive and adaptive systems capable of navigating the complexities real-world interactions among a wide array of agents.
14

---

Published in Transactions on Machine Learning Research (03/2025)
References
Ravichandra Addanki, Peter W Battaglia, David Budden, Andreea Deac, Jonathan Godwin, Thomas Keck, Wai Lok Sibon Li, Alvaro Sanchez-Gonzalee Jacklynn Stott, Shantanu Thakoor, et al. Large-scale graph representation ion learning with very deep gnns and self-supervision. orXiv preprinl or wwXiv:2107.09.22 2021.
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robiequet, Li Fei-Fei, and Silvio Savarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
Apratim Bhattacharyya, Mario Fritz, and Bernt Schicle. Long-term on-board prodiction of people in traffic scenes under uncertainly, 2018.
Apratim Bhattacharyya, Christoph-Nikolaa Strachle, Mario Fritz, and Bernt Schiele. Haar wavelet based block auloregressive flows for trajectories. CoRR, hts//2009009875 2020. URL https:///arai: http///aaxvv...rabbaaaaaae/a/
2009.09878
Apratim Bhattacharyya, Daniel Olmeda Reino, Mario Fritz, and Bernt Schiele. Euro-pvi: Pedestrian vehi- cle interactions in dense urban centers. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6404-6413, 2021.
Wenqi Cai, Ganglei He, Jianlong Hu, Haiyan Zhao, Yuhai Wang, and Bingzhao Gao. A comprehensive intention prediction method considering vehicle interaction. 2020 4th CAA International Conference on Vehicular Control and Intelligence, CVCI 2020, pp. 204-209, 2020. doi: 9338520.
Weihuang Chen, Fangfang Wang, and Hongbin Sun. S2tnet: Spatio-temporal transformer networks for trajectory prediction in autonomous driving. In Asian Conference on Machine Learning, pp. 454-469. PMLR, 2021.
Xiaobo Chen, Huanjia Zhang, Yu Hu, Jun Liang, and Hai Wang. Vnagt: Variational hon-autoregressivve graph transformer network for multi-agent trajectory prediction. IEEE Transactions Vehiculor Technology, pp. 1-12, 2023. doi: D0.109/TVTT wtt...2222.32222..
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. arXiv preprint arXiv:2110.07875 2021.
Jianwu Fang, Dingxin Yan, Jiahman Qiao, Jianru Xue, and Hongkai Yu. Dada: Driver attention prediction in driving accident scenarios. TEFE Transactions on Intelligent. Transportation Systems, 2021. ISSN 15580016. doi:
Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkon, Bogdan Stanciulescu, and Fabien Moutarde. Gohome: Graph-oriented heatmap output for future motion estimation. arXie preprinl arXiv:2109.014 2021.
Harshayu Girase, Haiming Gang, Srikanth Malla, Jiachen Li, Akira Kanehara, Karttikeya Mangalam, and Chiho Choi. Loki: Long term and key intentions for trajectory prediction. In Proceedings of the TEFF//C /CVF International Conference on Computer Vision, pp. 9803 9812, 2021.
Francesco Giuliari, Irtiza Hasan, Marco Cristani, and Fabio Galasso. Transformer networks for trajectory forecasting. In 2020 25th international conference on pattern recognition (ICPR), pp. 10335 10342. IEEE, 2021a.
Francesco Giuliari, Irtiza Hasan, Marco Cristani, and Fabio Galasso. Transformer networks for trajectory forecasting, In 2020 25th international conference pattern recognition (ICPR), pp. 18835-10342. IEEE,
2021b.
Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17113 17722,, June 2022.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially accept- able trajectories with generative adversarial networks. In Proceedings of the IEEE conference compuler vision and pattern recognition, pp. 2255 2264, 2018.
Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics. Physical review E, (5)):4288, 1995.
Yue Hu, Siheng Chen, Ya Zhang, and Xiao Gu. Collaborative motion prediction via neural motion message passing. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 6319 6328, 2020.
Lei Huang, Jihui Zhuang, Xiaoming Cheng, Riming Xu, and Hongjic Ma. Sti-gan: Multimodal pedestrian trajectory prediction using spatiotemporal interactions and a generative adversarial network. IEEE Access, 9:50846-50856, 2021.
Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE transactions on pattern analysis and machine intelligence, 2023.
Thomas N. Kipf and Max Welling. Semi-suppervised classsssciiiiiccsaaassccattctio classification with graph convolutional networks. CoRR, abs/1609.02900 2016. URL http://aaxiiv..og//ab///66 02907
Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín, Ian D. Reid, Seyed Hamid Rezatofighi, and Silvio Savarese. Social- bigat:: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. In NeurIPS, 2019.
Tuliia Kotseruba, Amir Rasouli, and John K. Tsotsos. Benchmark for evaluating pedestrian action prediction. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1257 1267, 2021. doi: 10.1109/WACV
Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by example. Computer Graphics
Forum, 26(3):655-664 2007. doi: //onlinelibrary.oom wiley. comm/doi/ab////occoococcoo//aa /1000/0///1..1111147444888555 200.70000990000
w00..900000000.... URL https:
Lihuan Li, Maurice Pagnucco, and Yang Song. Graph-based spatial transformer with memory replay for multi-future pedestrian trajectory prediction. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2231-2241, 2022.
Bingbin Liu, Ehsan Adeli, Zhangjie Cao, Kuan-Hui Lee, Abhijeet Shenoi, Adrien Gaidon, and Juan Car- los Niebles. Spatiotemporal relationship reasoning for pedestrian intent prediction. IEEE Robotics and Aulomation Lellers, 5(2):3485 3492, 2020.
Yao Liu, Lina Yao, Binghao Li, Xianzhi Wang, and Claude Sammut. Social graph transformer networks for pedestrian trajectory prediction in complex social scenarios. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CTKM '22, pp. 1339-1349, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392365. doi: 10. 011145/5111808.5555 35554455. URL https://doi.org 10. 1.0.1145//7 1141//511800..... 3557455
Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Huu Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory predic- tion. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK August 23 23-28, 2020, Proceedings, Part II 16, pp. 750-776. Springer, 2020.
Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals, waypoints & paths to long term human trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 15233 15242, 2021.
Weibo Mao, Chenxin Xu, Qi Zhu, Siheng Chen, and Yanfeng Wang. Leapfrog diffusion model for stochastic trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5517-5526, June 2023.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgenn: A social spatio- temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14424 14432, 2020.
Lais Müller, Mikhail Galkin, Christopher Morris, and Ladislav Rampášek. Attending to graph transformers.
arXiv preprint ar xrx:223002.44888 2023.
Bo Pang, Tianyang Zhao, Xu Xie, and Ying Nian Wu. Trajectory prediction with latent belief energy- based model. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11809 11899, 2021. doi: C4.109/CVPR44437.2000 20..01164.
S. Pellegrini, A. Ess, K. Schindler, and L. van Gool. You'll never walk alone: Modeling social behavior for multi-target tracking. In 2009 IEEE 12th International Conference on Computer Vision, pp. 261-268, 2009a. doi: 10.1109/ICCV. 2459264
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Lue Van Gool. You'll never walk alone: Modeling social behavior for multi-target tracking. In 2009 IEEE 12th international conference on computer vision pp. 261-268. IEEE, 2009b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Amir Rasouli and Iuliia Kotseruba. Pedformer: Pedestrian behavior prediction via cross-modal attention modulation and galed multitask learning. In 2023 IEEE International Conference on Robolics and Au- tomation (ICRA), pp. 9844 9851. IEEE, 2023.
Amir Rasouli, Iuliaa Kotseruba, Toni Kunic, and John Tsotsos. Pie: A large-scale dataset and models for pedestrian intention cstimation and trajectory prediction. In 2019 IEEE/CVE /CVF International Conference on Computer Vision (ICCV), pp. 6261 6270, 2019. doi: 10.1109/ICCV. 2019.00636..
Amir Rasouli, Mohsen Rohani, and Jun Luo. Podestrian behavior prediction via multitask learning and categorical interaction modeling. CoRR, abs/2012.03298, 2020. URL https://arxiv http:///aaxxv..og/aa//////// wrgg
03298
Amir Rasouli, Mohsen Rohani, and Jun Luo. Bifold and semantic reasoning for pedestrian behavior predic- tion. In Proceedings of the IFEE/CVF International Conference on Computer Vision, pp. 15600-15610, 2021.
Javier Ribera, David Guera, Yuhao Chen, and Edward J Delp. Locating objects without bounding boxes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1479-6489, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241. Springer, 2015.
Assisted Intervention Iatervention-MICCAA MICCAI 2015: 18th
Benedek Rozemberekki, Carl Allen, and Rik Sarkar. Multi-scale altributed node embedding, Journal of Complex Networks, (2)ccca,,,, 2021.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++3 Dynamically- feasible trajectory forecasting with heterogeneous data. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XVIIII 16, pp. 683 700. Springer, 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp.
2018.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-bssed localization. Tn Proceedings of the IEEE international conference on computer vision, pp. 618 626, 2017.
Jie Tang, Jimeng Sun, Chi Wang, and Zi Yang. Social influence analysis in large-scale networks. In Proceed- ings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD *09, pp. 807-816, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605584959. doi: 10.1145/ (1555511.15577717
URL https: ptttps://oi... http://doo.oogg1/////o 10. 11111111111111 11157701..115115
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpl::ttt:/ppptt htttpss:/pppcceeaaggsss... neurips.. wwt.ppp.ccppppp.pppp files/ /ppper/// 0017/4/lle/5fee22237766622212233 Paperr
Anirudh Vemula, Katharina Muelling, and Jean Oh. Social attention: Modeling attention in human crowds. In 2018 IEEE international Conference on Robolics and Aulomalion (ICRA), pp. 4601-4607. IEEE, 2018.
Kira Vinogradova, Alexandr Dibrov, and Gene Myers. Towards interpretable semantic segmentation via gradient-weighted class activation mapping (student abstract). Proceedings of the AAAI, 34(10): 13943-1399 April 2020. ISSN 2159-5399, doi: 10.1609/aama v34i1. URL http:://xttt.///////////////tttt doi org/10, 1609/aaai. v34110. 7244
Chuhua Wang, Yuchen Wang, Mingze Xu, and David J Crandall. Stepwise goal-driven networks for trajectory prediction. IEEE Robotics and Automation Letters, 7(2):2716 2723, 2022.
Kuan-Lin Wang, Li-Wu Tsao, Jhih-Ciang Wu, Hong-Han Shuai, and Wen-Huang Cheng. Trajfine: Predicted trajectory refinement for pedestrian trajectory forecasting. In 2024 IEEE/CV /CV Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2024.00451.
2024. doi: 10.1109/CVPRGGG20
Xinshuo Weng, Ye Yuan, and Kris Kilani. Pip: Parallelized tracking and prediction with graph neural networks and diversity sampling. IEEE Robotics and Automation Letters, 14640046477 7 2021. ISSN 23773766. doi: ...1099LRR..021.... 2021.306892..
Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 6498 6507, June 2022a.
Chenxin Xu, Weibo Mao, Wenjun Zhang, and Siheng Chen. Remember intentions: Retrospective-memoory based trajectory prediction, June 2022b.
Chenxin Xu, Robby T Tan, Yuhong Tan, Siheng Chen, Yu Guang Wang, Xinchao Wang, and Yanfeng Wang. Eqmotion: Equivariant multi-agent motion prediction with invariant interaction reasoning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1410-1420, 2023.
Takuma Yagi, Karttikeya Mangalam, Ryo Yonetani, and Yoichi Sato. Future person localization in first- person videos, 2018.
Yu Yao, Mingze Xu, Chiho Choi, David J Crandall, Ella M Atkins, and Behzad Dariush. Egocentric vision- based future vehicle localization for intelligent driving assistance systems. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9711-9717 IEFE, 2019.
Yu Yao, Ella Atkins, Matthew Jlanson-RRbbesoee Ram Vasudevan, and Xiaoxiao Du. Bitrap: Bi-directional pedestrian trajectory prediction with multi-modal goal cstimation. IEEE Robotics and Automation Letters, 6(2):1463 1470, 2021.
Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for 2020: 16th European Conference, Glasgow,
pedestrian trajectory prediction. In Computer UK, August 23 28, 2020, Proceedings, Part XII 16, pp- 507-523. Springer, 2020a.
18

---

Published in Transactions on Machine Learning Research (03/2025)
Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In Computer Dision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XII 16, pp. 507 523. Springer, 2020b.
Ye Yuan, Xinshuo Weng. Yanglan Ou, and Kris Kitani. Agentformer: Agent-aware transformers for socio- temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021.
Simone Zamboni, Zekarias Tilahun Kefato, Sarunas Girdzijanskas, Christoffer Norén, and Laura Dal Col. Pedestrian trajectory prediction with convolutional neural networks. Pallern Recognition, 121:108252, 2022.
19

---

Published in Transactions on Machine Learning Research (03/2025)
A Weighted Loss Function Details
A.1 Loss Function Formulation
The loss for stochastic predictions is given in equations equation 16 and equation 17 I weightted (Vis, Y) is calculated similar to Equation 18.
weightted (o, (Y.Y) min (weighted Y)
(16)
k=l,,.,K
Liual weightted (Y, Y) DKL(N((z,, || (zp))
(17)
For deterministic predictions, the final loss is the samee the weighted loss:
Tored
Limal weiighted (Y
w(t) L(Y,, Yi),
(18)
where wtt) represent the weighted penalty function (section A.2 and L(Y,, Y2) is the predefined loss function: MSE or Smooth L1 loss (discussed below).
Mean square error (MSE)
1 N Σ (y 0;)²
MSE
(19)
where Yi and i represent, the actual and predicted coordinates, respectively. MSE penalises larger trajectory prediction errors more heavily, ensuring model accuracy in critical scenarios.
Smooth L1 loss (SL1)
0.5 x (y 02)
if 1/4 0o
1
SLI 9.)
(20)
1y: Del - 0.5 otherwise.
Unlike MSE, SL1 effectively balances the treatment of small and large errors. This loss is also less sensitive to outliers, due to its combination of L1 and L2 loss properties.
A.2 Weighted Penalty Function
In many trajectory prediction tasks, errors at different time steps may have varying importance. For instance, predictions further into the future might be more uncertain, whereas errors in early predictions could be more critical for subsequent decisions. To address this, we introduce weighted penalty function w(f) that adjusts the loss function's sensitivity to prediction errors over time.
Our ablation focuses on three distinct penalty functions, w(tt Tpred, a, 9): Linear, Quadratic, and Parabolic, which are parameterized by and 3, determined empirically as shown in Figure g Here, represents the current time step, and Tpred denotes the total prediction horizon. Once the values of and are determined, w(t, pred a, 3) can be simply denoted w()).
Table g presents quantitative analysis comparing the three penalty strategies as applied to the ETH-UCY (UNTV) dataset using SL1 loss. It. can be observed that the Parabolic penalty yields better results compared to the other penalization strategies.
Figure 8 illustrates comparison of the three weighted penalty strategies for a prediction window of frames. The subsequent sections provide detailed explanation of each of these penalty strategies.
20

---

Published in Transactions on Machine Learning Research (03/2025)
Comparison of Penalty Weight Strategics
Parabolic Weighted Penalty (a 4, 1)
Lineer Weighted Penalty 0
Quadiratic Weighted Penalty
Penalty We Weightt (w(t))
2.0
Figure 8: Comparison of various weighted penalty strategies
A.2.1 Linear Weighted Penalty
The Linear Weighted Penalty employs a weight function, w(f), that linearly increases from a start weight (a) to an end weight (5), over the prediction period. This approach aims to progressively increase the penalty for prediction inaccuracies, particularly toward the latter part of the prediction horizon.
The weight function w(t) defined
wtt) = a + Tpred (3 -0-a). a),
(21)
where a and B are the weights assigned to the initial and final predicted time steps, respectively.
A.2.2 Quadratic Weighted Penalty
The quadratic weighted penalty strategy intensifies the penalty in quadratic manner as the difference between the prediction time and the past frames increases. This approach is more aggressive than the linear strategy, applying an exponentially increasing weight to errors in later prediction frames. The weight function wo(t) in this case defined as:
w(()= = a
(B - a))
(22)
T
Trred
A.2.3 Parabolic Weighted Penalty
The Parabolic Weighted Penalty assigns the maximum weight, to both the initial and final predicted time steps, highlighting their significance. Meanwhile, the minimum weight, 3 (8 < a), is allocated to the midpoint of the prediction interval. This distribution forms a parabolic trajectory (shown in Figure g of weights across the prediction period, as defined by:
w(t) = (a 33) (2. (
Tpred "
t -1) + ,,
(23)
A.2.4 Augmentation
To enhance the robustness and generalization of our trajectory prediction model, we implement data augmentation strategy inspired by Zamboni et. al. (0022 This strategy applies random rotation and trans-
21

---

Published in Transactions on Machine Learning Research (03/2025)
Actual Triectory
Actuall Injerter for
(a) Univ (No Penalty)
(b) Univ (Penalty)
Fust Sarccrry
Fost Trpes tuty Predictio Trajectory
Wedeited souectory
Zara01 (No Penalty)
(d) Zara0l (Penalty)
Figure 9: Qualitative comparison of unpenalised vs. penalised trajectories ETH-UCY dataset in stochastic setting.
Table 8: Ablation:
Comparing penalization strategies with SL1 loss on ETH-UCY (UNIV) dataset using
ASTRA's SOTA configuration
Loss
miinADD22//iiFDDer
Unpenalised
0.37/0.52
Linear
Quadratic
0.30/0.46
Parabolic 0.28/0.41
lation transformations to the trajectory sequences with a probability of 0.4, as illustrated in Fig. I1 introducing these augmentations, the model becomes orientation-agnostic and better equipped to handle po- sitional shifts in agents. The increased diversity training data enables the model learn more generalized representations of agent movements, improving its adaptability real-world scenarios.
By
B
Evaluation Metrics
B.1 ETH-UCY
To evaluate our model on ETH-UCY, we used commonly employed evaluation metrics Xu et al. (2023 Mohamed el al. 2020); Chen et al. 2023); Yu et al. (2020b): ADE/FDE and minADEx DDD/kminFDDE DDi Average Displacement Error (ADE) computes the average Euclidean distance between the predicted trajectory and the true trajectory across all prediction time steps for each agent. minADE refers the minimum ADE out of K randomly generated trajectories and ground truth future trajectories. We also used the Final Displacement, Error (FDE), which focuses the prediction accuracy at the final time step. It computes the
22

---

Published in Transactions on Machine Learning Research (03/2025)
Actual Triectory
Actuall bojerter fort
(a) Univ (No Penalty)
(b) Univ (Penalty)
Pust Sarccrryy
Fost Traes tuty Predictio Trajectory
Wedected souectory
(c) Zara01 (No Penalty)
(d) Zara0l (Penalty)
Figure 10: Qualitative comparison of unpenalised vs. penalised trajectories ETH-UCY dataset in stochas- tic setting.
Trunslation & Rotation
Translation
Translation
Figure 11: Illustration of data augmentation to trajectory sequences
Euclidean distance between the predicted and actual positions of each agent at the last prediction time step. minFDEx refers to the minimum FDE out of K randomly generated trajectories and ground truth future trajectories. For stochastic setting, minADE and minFDFg metrics used for evaluation.
23

---

Published in Transactions on Machine Learning Research (03/2025)
Treed
ADE
|| 1,1122
(24)
Tpred
1=1
Tored
minADEx min
| Yr, 11
(25)
T ppred
1=1
FDE
pred
De
T Tprce
122
(26)
(1Y Dra (Ppreddell2)
min minFDER
(27)
k
B.2 PIE
For the PIE Dataset, the ADE and FDE metrics are calculated based on the centroid of the bounding box Rasouli et al. (2019) Rasouli & Kotscruba (2023), denoted as Centre average displacement error for the bounding box (CADE) and Centre final displacement error for the bounding box (CFDE). In addition, we reported the average and final Root Mean Square Error (RMSE) of bounding box coordinates, denoted ARB and FRB, respectively Rasouli et al. 20021.
C Grad-CAM visualizations
Grad-CAM images were obtained by generating heatmaps overlaid onto the original image aid in vali- dating the relevance of highlighted regions. To obtain the Grad- C-- visualization, single-channel output segmentation map
obtained from the pre-trained U-Net network, representing the probability of each pixel location being keypoint Ribera et al. (2019). Probabilities were aggregated across all pixels, by comparing them with Irue keypoints and gradients of activation for the initial layer were extracted, similar to the approach taken by /inogradova et al. (2020). Utilizing these gradients, a weighted average of the ac- tivation maps of the initial layer was computed to reconstruct the heatmap, similar to the method described in Selvaraju al. (2017), for the Grad-CAM visualization. Overlaying this heatmap onto the original image highlights the regions that contribute significantly to the keypoint predictions made by the model.
D Abbreviations and Mathematical Notation
To ease reading the paper, Table and 10 list the abbreviations and the mathematical symbols mentioned in the paper, respectively.
24

---

Published in Transactions on Machine Learning Research (03/2025)
Trajecory
Nctual Traaectory Predicted Trajectory
Accumll
Prodectos Wrajeztary
(a) ETH
(b) Hotel
Paw: Thalectory Actual Trkerta
Act.al Traiectory
(c) Univ
(d) Zaral
Trar Mar Actue Predeced tea moroctory
(e)
Zara2
Figure 12: Trajectory visualizations for stochastic setting on ETH-UCY dataset (BEV)
25

---

Published in Transactions on Machine Learning Research (03/2025)
Past Trajectory
Actual Trodectory
Predicted Thajectory
dicted Taajectoyy
(a) ETH
(b) Hotel
Past Thajectory Actial Trajectory
Rud Teturdery Actual Treletory
(c) Univ
(d) Zaral
Past Trajectory Predictec Trajectory
(e)
Zara2
Figure 13: Deterministic trajectory visualizations on ETH-UCY dataset (BEV)
26

---

Published in Transactions on Machine Learning Research (03/2025)
(b)
Figure 14: Trajectory Visualizations on PIE Dataset (EVV) where the red and cyan bounding box indicates the ground-truth and predicted final position respectively and the blue bounding box indicates the start position.
Table 9: Table of Abbreviations Used
Sebreviation/Teri
Description
ASTRA
Agent-Scene aware model for pedestrian trajectory forecasting
BEV
Bird's Eye View
EVV
Ego-Vehicle View
AV
Autonomous Vehicle
MLP
Multi- Perceptron
CVAE
Conditional Variational
GNN
Graph Neural Network
RWPE
Random Walk Positional Encoding
MSE
Mean Square Error (Loss Function) Smooth L1 Loss (Loss Function) Average Displacement Error Final Displacement Error
SL1
ADE
FDE
CADE
Centre average displacement error for the bounding box Centre final displacement error for the bounding box
CFDE
ARB
Average Root Mean Square Error for the bounding box Final Root Mean Square Error for the bounding box
FRB
27

---

Published in Transactions on Machine Learning Research (03/2025)
Table 10: Table of Mathematical Symbols Used
Symbols
Description
N
Total number of predictions in MSE calculation
X
Observed trajectories of agents
Y
Groundtruth future trajectories of agents
Y
Predicted trajectories of agents
Tobe
Number of past time instants for observation Number of future time instants for prediction Sequence of past input frame images Observed coordinates for agent at time Predicted coordinates for agent a at time t Number of target agents
Tpred
I1=1TTW
Xf
A
eg
Edge weight. graph G between nodes and j Distance between agents v and By Weight function in weighted-penalty strategy Start weight in weighted-penan End weight in weighteed-penatty strategy
d(v,,)))
w())
Wstart
strategy
Wend
L Scene
Latent representation of scene(past frame) obtained from U-Net encoder Scene-aware embeddings
Scone
Scene-aware
Scene-aware Transformer encoder
Y Encoder
U-Net Encoder
Scone
Multi-layer rercept layer for Scene embeddings
Temporal
Temporal encoding
T Spatial
Multi-layer Perceptron layer for Spatial embeddings
$ Spatial
Spatial embeddings
Social
Multi-layer Perceptron layer for Social embeddings Social Embeddings
Social
T ggntt-ware
Agent-aware Transformer encoder
Agenta
Agent-awas embeddings
weiggtted (Y,y, Y)
Weighted-penalle Loss Function
28


Reviews:
[{"review_id": "DA65fnM8J6", "reviewer": "Reviewer_Vspv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strength:* \n- The paper shows strong performance compared to previous works on three human trajectory datasets.\n\n*Weaknesses:*\n- The main weakness of the paper, in my opinion, is that it’s not clear what the main factor contributing to the performance is. Despite the ablation in section 4.4, some of the technical decisions are not tested, i.e., for instance, what is the effect of using a ‘graph-aware’ transformer instead of using a streamlined transformer architecture?\n\n- Some of the claims could be better supported:\n    - Authors claim that their model is lightweight but only focus on parameter count. What is the advantage of Astra in terms of training and testing FLOPs?\n    - Authors hypothesize that “transformers, by their inherent design, may pose potential challenges in preserving information, as they do not inherently accommodate the graph structure in their input,” but don’t verify it empirically.\n    - Authors propose a scene-aware embedding using a UNet as a contribution but don’t compare it with other ways to extract scene information. One could potentially use DINOv2 or CLIP features to encode visual information\n    -  It would be nice to validate the graph-aware architecture empirically, compared to a streamlined transformer architecture.\n\n- The clarity of the paper could also be improved:\n   - What is the main contribution of the paper? Is it a novel way to combine existing architecture components, or is it a specific architecture component like the graph-aware transformer or the UNet scene encoder?\n   - The model description could use more details. For instance, it’s not clear how you extract a fixed-length representation from the transformer scene and agent representation.\n   - While the author propose loss penalty as a contribution, there is no description of this part in the main text, only in appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Astra, a new model that focuses on human trajectory forecasting. The main contribution are a novel model architecture dedicated to human trajectory prediction and a penalization scheme for the training loss function.\n\nThe author applies their Astra model to the ETH, UCY, and PIE datasets. Their empirical evaluation, both in deterministic and stochastic settings, shows the advantage over previous work on the proposed approach.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Ablation to validate the different design choices of the model, graph-aware transformer, compare different scene embedders…\n- Add new results to better show the advantage of Astra in terms of training/inference runtime.\n- Improve the clarity of the paper (see weaknesses).", "broader_impact_concerns": "No concen", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kAKTXJDRJe", "reviewer": "Reviewer_ytj4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n* The model incorporates scene embeddings from a pretrained keypoint detector which is interesting.\n* The proposed architecture appears simple which is a positive.\n* The reported results are quite strong.\n* The same architecture + training seems to transfer from the ETH dataset (Birds Eye View) to the PIE Dataset (Ego Vehicle)\n\n### Weaknesses\n\n* The clarity of the paper can be significantly improved.\n* After reading the architecture, it's unclear what elements are based on prior work and what are the authors' novel contributions.\n* Some important details of the paper are relegated to the supplementary or not clearly explained.\n\nI request the authors to improve the paper based on my suggestions below", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a “lightweight” model for trajectory prediction. At each timestep, trajectories are represented by matrices of shape $A \\times 2$ (coordinates) or $A \\times 4$ (bounding boxes) where A is the number of agents (pedestrians). In addition to spatial embeddings, of these trajectories, The model incorporates\n\n* Scene embeddings from a pretrained UNeT trained on keypoint detection that encodes the raw pixels.\n* Social embeddings using an agent-to-agent graph based representation.\n* Ttemporal position embeddings” in addition to “Spatial embeddings”. \n\nA conditional VAE is also trained to output stochastic trajectories. Further, the authors introduce a timestep dependent weighting function in the loss which can upweight/downweight the loss in the future.\n\nQuantitative results are reported on the ETH-UCY benchmark where the model outperforms all other baselines with lower parameter count on the average score.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clarity\n-----\n\n* Please use notation such as $\\phi \\in R^{T \\times D}$ to denote various embeddings such as the spatial, social and temporal in Section 3. This would make it clearer if these are global (1-D) or include spatial information (2-D).\n* Along what axis does the self-attention in the transformer encoder operate? Is it the temporal dimension? Again, clarifying the dimensions of different vectors will be super helpful?\n* How are the $T_{obs}$ dimensions to the inputs of the transfomer folded into a single output vector?\n* Augmentation seems very important in Table 3 but is mentioned in Sec 4.4.3. What sort of augmentation is this and do other works use this as well?\n* How are the hyperparameters  tuned?\n* What is the precise $f$ used in Section 3.1.7? How are alpha and beta selected? Is it tuned for every dataset? Please highlight the main $f$ used in the paper.\n* Eq 6): What are the dimensions of the graph representation G and the output of RWPE(G)? \n* A high level description of RWPE would be helpful.\n* Please describe the architecture of the CVAE in the main paper. How are the posterior, prior and decoder parametrized? What are the inputs to these different networks and what architectures do they follow?\n* Page 3: Alleviates the data requirements and preprocessing efforts highlighted earlier. It’s unclear what this means and why using a keypoint model instead of a segmentation model helps to do this?\n* Page 3: incorporating a penalty component is described here, but what this actually means is described as late as Section 3.1.7.\n* This is in contrast to Yuan et al. (2021) which does not build a graph and does not preserve the social structure; they distinguish between self-agent and all other agents, then they treat all other agents the same without encoding the positional or structural encodings. Does this mean Yuan et. al (2021) do not have a graph representation for the input coordinates?\n* For ablations in Table 3, the corresponding features are removed but the transformer Encoders are retrained. Is that correct?\n* How was the keypoint model pretrained?\n* It seems two separate models are needed for deterministic and stochastic predictions?\n\nNovelty\n--------\n\nI understand that the loss weighting and keypoint latent representations are contributions of this paper? It’s a bit unclear what the other contributions are and which were based on prior work. Some concrete examples include the usage of graph representations to output $\\phi_{social}$, usage of CVAE to model stochastic trajectories, sinusoidal temporal embeddings. The paper can state this upfront and provide citations.\n\n\nArchitecture\n--------\n\n* Are two transformer encoders (Scene-aware and Agent-aware) necessary with separate parameters? What happens if you concatenate all features and feed them to a single transformer? This ablation has to be compute adjusted so once has to train a bigger single transformer to make a fair comparison\n* Have the authors considered using the keypoints itself rather than the latent representation?", "broader_impact_concerns": "noo", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nNeU8V8PAU", "reviewer": "Reviewer_muft", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. **Novelty**: The paper introduces a new architecture that incorporates multiple data sources. This is an interesting approach that attempts to resolve some practical challenges, such as data limitations in certain real world scenarios. \n2. **Empirical results**: The experiment section suggest that this model outperforms almost all baseline models across a wide range of datasets. \n\n\n**Weaknesses**:\n1. **Claims of suitability for on-device processing**: The paper emphasizes the \"number of parameters\" as a key reason for being more suitable for on-device processing. However, parameter count is a major consideration from a training perspective, rather than a deployment perspective. To address this claim, the paper should include experiments and discussion on *inference speed* on a resource-limited device and *memory footprint* of ASTRA in comparison to other models. These are more important from a deployment perspective. \n2. **Claims of usefulness during data absence**: The authors claim that prior works are problematic where data is unavailable, such as BEV. The experiment section does not provide evidence that ASTRA can perform in the absence of either data source.  \n3. **Stochasticity and multi-modal prediction**:  In Sec 2.1, the definition of stochastic prediction should not be equivalent to multi-modal prediction. Multi-modal prediction refers to predicting multiple modes or distributions (e.g. Gaussians) and can be used without the addition of randomness or generative models. Take for instance, the Gaussian Mixture Model (GMM). This section is also missing related lines of work on mixture density networks and multiple choice learning that are used in trajectory prediction.  See [1] as an example.  \n4. **Description of weighted trajectory loss function**: The description of the weighting function in Equation 10 on page 8 is extremely vague and could encapsulate almost anything. I do not think this description is sufficiently concise for something that is claimed to be a key contribution in the introduction. The exact functions are actually in the appendix, however the discussion is poor and mostly empirical. \n5. **Description of KL Divergence for CVAE**:  There is a misunderstanding of the KL divergence term in Equation 11. It seems  that the KL divergence term arises due to the use of a CVAE, as per Figure 3 and Appendix A. There is a sentence that implies that the KL divergence helps the model learn the \"dependency between Y and X\". This is not true. The KL divergence term regularizes the latent distribution by encouraging the learned latent distribution to approximation a prior distribution (usually a Gaussian). The KL divergence term does not enable the generation of multiple plausible trajectories as suggested. Also the \"contrasting\" of $\\mu_{z_q}$ and $\\mu_{z_p}$ does not guide the model towards accurate predictions either. These descriptions mischaracterized the role of the KL divergence and should be removed. \n\n**Reference**:\n1. Makansi, Osama, et al. \"Overcoming limitations of mixture density networks: A sampling and fitting framework for multimodal future prediction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a neural network architecture, ASTRA, for forecasting the future trajectories of pedestrians from a bird's eye or vehicle perspective. ASTRA combines information from two sources of data, past pedestrian trajectories (X) and past images (I), to improve prediction performance. The model extracts features from these two sources of data (X and I) through two different neural networks: (1) U-Net + Transformer and a (2) Graph Neural Network + Transformer. These features are concatenated and passed through a decoder which predicts future trajectories. ASTRA can output a single prediction or multiple predictions through the use of a conditional variational auto-encoder (CVAE). \n\nThe paper claims the following contributions:\n1) The two feature extraction heads are able to encode two different data sources to improve overall performance. The author's claim that this is an advantage where one source is unavailable.\n2) ASTRA is trained with a weighted trajectory loss function that improves performance.\n3) ASTRA contains 7 times fewer parameters than prior models which is more suitable for on-device processing.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Experiments showing inference speed and memory footprint on a resource-limited device \n2. Experiments showing performance without either data source (images or trajectories)\n3. Address the definition of stochastic and multi-modal prediction as per above section.\n4. Improve description of the weighted trajectory loss function in main paper. \n5. Correct the description of KL divergence term and its role in the CVAE. Provide details on how the multi-modal sampling occurs through the CVAE.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "w5TH2Zra5z", "reviewer": "Reviewer_1y6m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The proposed method is evaluated on several datasets and compared against different baseline models (up to 2023), achieving the best performance on most datasets.\n2. Compared to state-of-the-art methods, ASTRA’s reduced number of parameters represents a step toward making these models applicable to real-world applications.\n\nWeaknesses\n1. The results section would be improved by providing a more detailed explanation of why ASTRA outperforms state-of-the-art methods on most datasets but falls short on the Hotel, Univ, and Zara2 datasets.\n2. The paper does not clearly explain how the network's hyperparameters were selected.\n3. The model is claimed to be suitable for deployment on devices with limited processing power, however, the paper lacks a table showing performance metrics like floating-point operations per second (FLOPS) to support this claim.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces ASTRA, a trajectory forecasting model that can produce both deterministic and stochastic predictions, with seven times fewer parameters than the current state-of-the-art. The model leverages a U-Net-based feature extractor to generate scene-aware embeddings from input frames and employs a graph-aware transformer encoder to capture relationships between agents. Additionally, the authors propose a novel loss function that improves prediction accuracy by dynamically adjusting penalties based on prediction errors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. (Critical) Given the paper’s focus on building a more efficient model for trajectory forecasting, the authors should (i) provide a breakdown of the number of parameters for each component of the network, and (ii) report performance metrics like floating-point operations per second (FLOPS) or MAC (Multiply-ACCumulate) operations.\n2. (Critical) The authors should also specify which dataset was used to tune the network’s hyperparameters and report these hyperparameters for easier reproducibility.\n3. (Critical) The authors should consider including more recent trajectory forecasting models that were published before the paper’s submission (e.g., [1]).\n4. (Would strengthen the work) The authors should expand the results section to explain why the model underperforms on certain datasets, such as Hotel and Univ.\n5. (Would strengthen the work) Figure 3 includes a legend for frozen and trainable components, but the symbol is only used for the frozen components of the U-Net. The symbol for trainable components is missing, such as in the transformer encoders.\n\n[1] Wang, Kuan-Lin, et al. \"TrajFine: Predicted Trajectory Refinement for Pedestrian Trajectory Forecasting.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "broader_impact_concerns": "No concern about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although overall the reviewers see value in the contribution to the research community, some details and polish are still missing from the paper, hindering its reproducibility and impact. The following are comments from the reviewers that should be addressed in the final paper version.

- See Reviewer ytj4 comments on combination of different dimensions.
- Sec 3 (Deterministic vs Stochastic Setting) - a deterministic and stochastic setting could be both multi-modal but this section seems to imply that only the deterministic setting is multi-modal. The same assumption is made in the introduction of Section 4.1.
- Sec 4.4 (Decoder) - "Training" is capitalized in the middle of a sentence. "fed" --> "feed".
- Sec 4.5 (Loss) - The final sentence in this section needs to be rewritten. For example, the paper is referring to KL divergence as "the stochastic".
Decision: Accept (poster)