Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction
Authors: M. Eren Akbiyik
Nedko Savov
Danda Pani Paudel
Nikola Popovic
Christian Vater
Otmar Hilliges
Luc Van Gool
Xi Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Ego-trajectory prediction
driver attention
multimodal learning
field-of-view
gaze fixations
deep learning
autonomous driving
driver behavior modeling
dataset creation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Introduces RouteFormer, a multimodal ego-trajectory prediction model leveraging driver attention and field-of-view data, along with the GEM dataset and Path Complexity Index, significantly improving prediction accuracy over state-of-the-art methods.

Abstract: Understanding drivers’ decision-making is crucial for road safety. Although predicting the ego-vehicle’s path is valuable for driver-assistance systems, existing methods mainly focus on external factors like other vehicles’ motions, often neglecting the driver’s attention and intent. To address this gap, we infer the ego-trajectory by integrating the driver’s gaze and the surrounding scene. We introduce RouteFormer, a novel multimodal ego-trajectory prediction network combining GPS data, environmental context, and the driver's field-of-view—comprising first-person video and gaze fixations. We also present the Path Complexity Index (PCI), a new metric for trajectory complexity that enables a more nuanced evaluation of challenging scenarios. To tackle data scarcity and enhance diversity, we introduce GEM, a comprehensive dataset of urban driving scenarios enriched with synchronized driver field-of-view and gaze data. Extensive evaluations on GEM and DR(eye)VE demonstrate that RouteFormer significantly outperforms state-of-the-art methods, achieving notable improvements in prediction accuracy across diverse conditions. Ablation studies reveal that incorporating driver field-of-view data yields significantly better average displacement error, especially in challenging scenarios with high PCI scores, underscoring the importance of modeling driver attention. All data and code are available at meakbiyik.github.io/routeformer.

Published as a conference paper at ICLR 2025
LEVERAGING DRIVER FIELD-OF-VIEW FOR MULTIMODAL EGO-TRAJECTOR PREDICTION
M. Eren Akbiyik', Nedko Savov?, Danda Pani Paudel', Nikola Popovic Christian Vater', Otmar Hilliges', Luc Van Gool & Xi Wang 1.4
ETH Zürich
INSAIT, Sofia University "St. Kliment Ohridski"
University of Bern
4 TTU Munich
ABSTRACT
Understanding drivers' decision-making is crucial for road safety. Although pre- dicting the ego-vehicle's path is valuable for driver-assistance systems, existing methods mainly focus external factors like other vehicles' motions, often ne- glecting the driver's attention and intent. To address this gap, we infer the ego- trajectory by integrating the driver's gaze and the surrounding scene. We intro- duce RouteFormer, a novel multimodal ego-trajectory prediction network com- bining GPS data, environmental context, and driver field-of-views weteoo--iewww-opriiiiiiiei- comprising first-person video and gaze fixations. We also present the Path Complexity Index (PCT), metric for trajectory complexity that enables ation of challenging scenarios. To tackle data scarcity and enhance diversity, we introduce GEM, comprehensive dataset of urban driving scenarios enriched with synchronized driver field-of-vier and gaze data. Extensive evaluations on GEM and DR(eye) VE demonstrate that RouteFormer significantly outperforms state-of- the-art methods, achieving notable improvements in prediction accuracy across diverse conditions. Ablation studies reveal that incorporating driver field-of-view data yields significantly better average displacement error, especially in challeng- ing scenarios with high PCI scores, underscoring the importance of modeling driver attention. All data and code is available at meakbiyik, wtthhb.io/rouueoommee
nuanced evalu-
INTRODUCTION
Understanding the perception and decision-making process of drivers crucial for road safety in autonomous and assisted driving. Statistics reveal that 42% of car-bicycle collisions result from driver inattention (Allenbach et al., 202T). In response, a driver-assistance system capable of inte- grating the driver's perspective into its decision-maken making would enable vehicles to make informed decisions (Schwarting et al.] 2018). Although predicting the driver's path is critical for such sys- tems, traditional methods focus mainly inferring the paths of other vehicles (Salzmann et al.) 2020 Gu et al. 202I; Ngiam et al. [021]: Varadarajan et al.) 2022: Nayakanti et al. (003)). In contrast, we seek predict the drivers' ego-trajectory by observing both the surrounding scene, and the perception of the ego-vehicle ' drivers via their frontal view and gaze fixations.
The central idea of this work builds on the insight that the perception of person is tightly inter- twined with their imminent and long-term goals, influenced by the surrounding traffic (Triesch et al., driver can reveal other-
2003 Hayhoe al., 2003 Argyle et al. 1994). In driving, the attention wise inaccessible information about their decisions on the road. Complementing scene information with these cues enables the vehicles to resolve ambiguous situations. For instance, the head move- ments of the driver might be highly informative in determining the turn direction in junction. The value predicting ego-trajectories emerge ambiguous, complex road paths, e.g., cases where the road is not trivially straight, when predicting where the driver would turn at intersection.
Our focus is thus twofold: firstly, to develop a framework capable of predicting the drivers' future ego-trajectories in various driving scenarios, and secondly, create a method for identifying com- plex yet rare trajectories and quantifying their complexity. Despite the presence of other driving
datasets with gaze data (Gopinath al. 2021
Xia et al. 2017 Fang et al. 2023). to the best of

---

Published as a conference paper at ICLR 2025
Driver FOV (video gaze)
Multimodal encoder
Pust trajetoor
Predicted trajectory
Model loss
Time-series transformer
Surrounding scene us video
Predicted visual fenturee
Figure I: RouteFormer framework. Using the past GPS and the scene together with driver field- of-view, we predict the future ego-trajectory and visual features in driving with a novel loss scheme.
knowledge, only one public benchmark VE contains both ego-drivers' locations and their field-of-view (Palazzi et al. 2018). The scarcity of available data poses another challenge for our task. To this end, we present an end-to-end framework that predicts future ego-trajectories, novel metric to measure the complexity of road scenarios, and new dataset of real-world driving recordings that include drivers' field-of-view changes.
First, we build multi-modal framework, RouteFormer, for egocentric trajectory estimation that in- tegrates scene data, driver FOV, and past trajectory, as shown in FFiurre RouteFormer crafted based insights from time-series forecasting literature, offering the flexibility to incorporate ad- ditional data from other modalities, such as navigation paths and LIDAR. We train Roumer with auxiliary tasks of predicting future gaze and scene features for regularization, using novel future-dist diuur-discounted loss formulation. To better quantifyy the complexity of various driving scenarios, propose new metric named Path Complexity Index (PCI), which measures the divergence of driving trajectory from an extrapolation of the current path. It also indicates the difficulty the cor- responding prediction tasks and allows us analyze model performance under different scenarios. Finally, we introduce a Gaze-assisted Ego Motion (GEM) dataset that captures diverse urban driv- ing scenarios across 10 subjects, integrating GPS, high-resolution scene data from front-view compiinnn first- fist-person video and gaze fixations.
cameras, and driver field-of-view (FOV) comprising
Experimental results show that RouteFormer outperforms state-of-the-art (SOTA) methods on the GEM dataset. To test its generalization capabilities, we also evaluate RouteFormer on the DR(eye)VE dataset (Palazzi lation studies show that incorporating drivers' field-of-view information improves the prediction quality, particularly in high-PCC -CI situations.
al., 2018), and our method surpasses the SOTA approaches. Ab-
In summary, our contributions are:
RouteFormer, an end-to-end multimodal ego-motion prediction network that effectively utilizes FOV data with novel loss design.
PCI, metric that measures the trajectory complexity.
GEM, an ego-motion dataset with driver positions and perspective.
2 RELATED WORK
Ego-trajectory prediction. Predicting the future sequence of locations of a moving entity has been explored for humans (Lyu et al.
2022 Rodin et al., 2021 Singh et al.] 2016)] Park et al. 2016).
road agents (Huang et al. 2022
Gulzar et al. 2021) and aircraft (Zeng et al. (022). Autonomous
driving has largely focused on estimating the trajectories of the many agents surrounding the ego-
vehicle (Salzmann et al.) 2020 Gu et al. 202 Nayakanti et al. 2023 Varadarajan et al. 2022
Ngiam et al. (022)) Recent advances in multi-agent prediction leverage specialized modalities
like HD maps (Shi al. 2022a) and interaction graphs (Girgis et al.
2021 Vishnu et al. 2023
2023 Zheng et al.) 2024 and human-inspired perception (Liao et al. 2024a b). The fusion of these diverse modalities has shown
while emerging approaches exploit large language models (Mao et al.
2

---

Published as a conference paper at ICLR 2025
promise in improving prediction accuracy (Choi al. 2021 Li et al. 2024a). However, predicting the ego- ego-vehicles's s trajectory, which is crucial for driving assistance systems (Schwarting et al., 2018
Jain et al., 2015 Vellenga et al., 20224 Kung al. 2024), remains less explored. Kinetic models assuming constant velocity and turn rate have been utilized (Ammoun & Nashashibi] 2009). Kim et al. (202T) predict the ego-vehicle's 's trajectory with VAE by conditioning
predicted driving style, while [Baumann et al. (2018) use observations of the static environment, albeit neither with visual information, similarly with Kim et al. (2017) and Feng et al. (2019). Only |aall et al. (2020) proposes vision module but requires hand-annotated actions. Taking a different direction, we focus on leveraging drivers' gaze and the scene predict future ego-vehicle locations.
Driving and attention. Individuals tend to focus on relevant objects in situations with particular goal view, acquiring necessary information even before they are needed (Land & Lee 1994
Underwood et al. 1999. Zhang et al., 2022 Fathi et al. 2012 Admoni & Srinivasa, [006)] Argyle
al. 1994). In navigation tasks, individuals often fixate on an intended path or destination well before initiating the movement (Hayhoe et al. (000)). For example, Zheng al. (2022) successfully shows that the gaze is a high-quality indicator for human motion with GIMO, Similarly, numerous studies have also employed eye tracking to study the driving behavior of individuals, focusing on aspects such as attention (Ahlström et al. (2021) cognitive load (Engström et al.) Kountoouuut ett al. 2015), and levels of fatigue (Heitmann et al. 2001 Gao et al. 2015). while many of them conducted in a driving simulator. For driver maneuver classification, recent work has shown relative success with an in-cabin camera observing the driver (Jain al., 2015 Ma al. 2023 Vellenga
iet al., 2024 Li ct al. 2024b), with some works exploring using gaze and scene information for ego-action prediction. c.g. turns or lane changes (Lee et al.) 2021 Amadori al.) 2020 Wu al. Yi et al.
2019
2023). Han et al. (2023) predict maneuvers based on rider sensor head motion
data. Yan et al. (2023) use head motion and headset gaze to forecast the driver's future path by fitting polynomial, but their non-robust gaze data limits its effectiveness. Finally, Ma et al. (2023) propose a transformer model, which integrates driver behavior information for maneuver prediction. In contrast, use driver's 's field of view together with the surrounding scenes for the trajectory prediction task, inspired by similar works in human motion such as GIMO (Zheng et al., 2022).
Driving datasets with driver FOV. In the context of driving, there datasets that provide in- vehicle footage of the driver's 's behavior as an indication of driver's 's attention (Ortega et al. 2020). zone-binned gaze (Ghosh et al. 2021 Ribeiro & Costa 2019) (stationary), head pose (Schwarz et al. 2017]. Jain et al.] (2013) provide synchronized footage of the driver and the surrounding scene, yet without gaze data. Palazzi et al. (2018) provide dataset of driving under various conditions, with gaze data from a tracking headset and synchronized GPS locations, however, with limited urban scenes. Given the limited amount of driving datasets with gaze, we introduce a new righuresouution dataset, where accurate eye tracker gaze location and vehicle GPS information are available. In contrast to others, we focus on city scenes with many agents where complex situations can arise.
EGO-MOTION PREDICTION WITH DRIVER FIELD-OF-VIEW
Driving is an interplay of short and long-term goals, influenced by exogenous factors. Although such external effects can be understood to some level through computer vision, drivers' behavior is mostly driven by self-established targets, which complicates the task of ego-motion prediction. To address this challenge, we propose a new framework named RouteFormer. At its core, our method is designed to exploit visual information from multiple cameras, the past trajectory of the vehicle, and, most notably, the driver's held-of-view--ooprrie modality contributes to understanding different aspect: the surrounding scene, the motion of the vehicle, and the targets of the driver. Regure 2 for an overview.
first-person video and gaze fixations. Each
RouteFormer offers two key contributions: (1) novel architecture that fuses multimodal inputs via self- and cross-attention mechanisms for time-sent series prediction, and (2) an auxiliary loss and future-discounting to regularize long-term forecasting that is otherwise prone to over-fitinng
3.1 TASK DEFINITION
In the ego-trajectory prediction task, given measurements of the vehicle and the environment for T time steps, aim anticipate the future locations of the vehicle for each of the next Tprd time
3

---

Published as a conference paper at ICLR 2025
FOV
Staclaed actoes time dimenséon
dow
No graffect
Future FOV
Vilen F2
Modial emllddiings Eg
Fir
Croesmodal transfommer
Mar
Multimodal encoder 8u
Felifaane
Fir
soeno
Sorne videos
Video cneoder
Self attention
Time trunsforteer
Vinoal
by
Vidao
Trajectory
cacodes
Maltinodal eacoder
Fye
(a) Multimodal encoder for FOV and scenes
(b) Time-series module and loss configuration
Figure 2: Framework details. The multimodal architecture fuses FOV, scene, and motion data for ego-trajectory forecasting. (a) The videos are encoded with the scene encoder Eg frame-wise using pre-trained vision backbone. FOV data is then encoded via a cross-modal transformer. The resulting tensors, all in the image feature domain, are stacked across time, self-attended, and concatenated with motion features for forecasting. (b) RouteFormer predicts the trajectory, as well as features from visual modalities concurrently use them as auxiliary losses for regularization.
steps (our prediction horizon). For the first T steps, we use as input the vehicle's motion M and scene representations in the form of video from K cameras of the surroundings S17 Notably, in addition, we incorporate FOV data Fi.r- We build model (M,, Sl.K Fir X) M1 +1: predict the future motion, where Ả are the model's parameters.
We represent vehicle location with two GPS coordinates using the EPSG:3857 coordinate system (in meters) (Pridal & Pohanka 2024). The t'th element of the input motion MI.7 TT corresponds to the relative change of the coordinates compared to t 3 1, as in Salzmann et al. (2021). For each time step E [0, T) the scene S RK xxx3x H5 xw contains images from K external vehicle cameras, with width W and height H FOV data has two components Fir {PY representing a frontal video feed from camera fitted to the driver's head F.T R T Tx XWF, and the high-frequeng, gaze positions, respectively. The second component, F E Tx2 consists of (x, y) coordinates of points gazed in the image VV relative to the bottom-left of the image.
(.T. FL
3.2 ROUTEFORMER ARCHITECTURE
In our architecture, scene and FOV modalities are processed by separate encoders Es(S): s and (F(r)))= f1.T, producing sequences of compact feature vectors of the same size. Next, all scene features are fused with FOV features using a self-attention module Ey (s fI.) = e RTxD, Finally, a time-series forecasting module T predicts the future Tpred data points the future motion, in addition to feature vectors of other modalities used in training. We design two versions of this architecture: RouteFormer- Baas,, V/ (M,. S1.K (A), which does not use the FOV features, and RouteFormer, (M S1:K Fir (A), which incorporates all the available modalities, including the driver's perspective.
Scene Module. Our shared scene module Es encodes each frame in S1:K using a pre-trained vision backbone, in our case Swin (Liu et al. 2021). The features are then reduced through self- attention layer to produce $1.7 E R X (see Figure 2 a). These features are used in subsequent layers for temporal reasoning with inter-frame attention.
FOV Module. The frames FY, of the FOV data Fi.r are encoded using the same scene encoder Es. in order to share the representation space and allow for implicit registration between the images. To account for high frame rate of gaze positions and refine the potentially noisy gaze positions, we perform multi-head self-attention of E following the decoder design in Zhou et al. (2021). After achieving two encoded feature vectors of the same shape for gaze and frontal video, we perform fuze-frame cross-attentio allows the model to "pick" the relevant scene features using the gaze positions across frames, thus accounting for the highly dynamic scene in moving vehicle. The vector produced is fir E D
between visual and gaze location features. The attention mechanism
4

---

Published as a conference paper at ICLR 2025
Modality Fusion. The next step is using encoder Ey to fuse scene $1. and optionally FOV fir features. To this end, we concatenate s and fı. across the time dimension, sum with the modality embeddings Eas. and apply a self-attention mechanism. EM encodes the feature source (e.g. one of the scenes (A E [0, K) or FOV). This self-attention layer performs implicit image registration and feature selection within and between the scene and FOV modalities. More precisely, we take the concatenation [ Ji. E IR (1) and produce its fused encoding v E IR''D In the case of RouteFFormer-Bas, where gaze data is not available, we only fuse the scene vectors.
Time-Series Forecasting. Next, we merge (along the feature dimension) the motion features mi:T and the aforementioned fused features V1.T obtain [M1:T, U1:T] g This forms the input for our time-series forecasting component T, which takes the merged features [M,1T 11:T| and produces prediction of the future motion MT:Tue' For this task, experiment with the state-of- the-art long-term time-series prediction methods Informer (Zhou al. 2021), Transformer (Vaswani et al. 2017). PatchT'T (Nie et al., (023)) and NI LinearrDD Linear (Zeng et al. 2023). While the focus on future trajectory prediction, we also predict future visual features Vrr: The auxiliary outputs are essential for regularization during training by enforeing intermediate features to remain rich in scene- and FOV-related information, whenever available.
3.3 TRAINING OBJECTIVES
Trajectory prediction models are prone to over-fitting due to the inherent difficulty of the training regime: as the subsequent predictions depend the correctness of the previous forecasts, models might learn from spurious correlations of the later sections of the trajectory. To tackle this problem, RouteFormer employs novel loss formulation which we term as dfsture-discoutted loss". The principle behind this loss is to consider predictions for future time steps with diminishing weight, particularly the early stages of the training, making predictions for the immediate future more influential on the overall loss than those for the distant future. This also follows real-world use cases where immediate predictions often have higher impact than long-term predictions.
distor-discounted loss. Given a predicted sequence Ypred and a ground truth Ygt. the future- discounted loss Li over N time steps is then defined as:
Ca Ngc)=
(V ygt,,)²
where y is the discount factor with 0 < < 1. This is inspired by reward discounting in reinforce-
ment learning literature, where the discount rate is often in [0.9, 0.99). On 6s at 5fps, the choice of 0.97, as used in our experiments, gives 0.9730 = 0.4 weight in the final time step.
Loss composition. Both the primary trajectory loss and the auxiliary losses are subject to the future- discounting. Let LT denote the discounted trajectory loss and Ly the discounted video visual. See Appendix A.1 for detailed definitions. The combined loss Lcombined can be expressed as:
Leombined = Lr + ov Ly.
The coefficient av dynamically balances the primary and auxiliary losses. It is determined by:
||CT||
cty = PV x
Lv
Here, PV is a scalar hyperparameter and || represents the parameter's scalar value detached from gradients. This ensures auxiliary losses maintain a consistent proportion to Lr, stabilizing training while preserving trajectory prediction as the primary objective.
3.4 IMPLEMENTATION DETAILS
RouteFormer is trained using the AdamW optimizer with linear warm-up of 2 epochs and cosine annealing, over a total of 200 epochs. Maximum learning rate x 10-5 and weight decay of 1 x 10 used with batch size 16. A full set of hyperparameters can be found in Appendix
Regularization. We employ no dropout inside the transformers, but apply aggressive within- modality dropouts (e.g., dropping one of the two scene videos) to improve validation scores.
5

---

Published as a conference paper at ICLR 2025
Caching. To benefit from pre-trained vision backbones while keeping the training duration under control, we have implemented module-level caching that utilizes intermediate features extracted from these backbones. Details of the caching mechanism explained in Appendix
PATH COMPLEXITY INDEX
critical challenge in autonomous driving is tackling the long tail of driving scenarios situations that are less frequent but potentially high-impact. These scenarios often encompass diverse range of unpredictable driving conditions, such as navigating complex intersections or road obstructions. Traditional approaches often view the task tions of hand labeling such events (Teeti et al. 2022 Girase et al. 2021; Palazzi et al. 2018). To automatically identify and quantify the complexity of driving situations, introduce novel met- ric termed Path Complexity Index (PCI). This metric evaluates the deviation from simple driving patterns, thereby providing robust tool for assessing and improving trajectory prediction models.
classification problem, constrained by the limita-
Given an input trajectory, the PCI metric computes the Fréchet distance between the target trajectory and a hypothetical simple trajectory in which the driver maintains the speed and direction exhib- ited in the last segments of the input trajectory. Formally, the PCI of an observed target trajectory x2, T x2 given an input trajectory Tinput T"2x, is given by:
Trarget
PCI(Trarges Timputt)
inf max 0.8E
[[ - Tsimpte (3(t)) [E(0,1]
sup
where [0 [0 and [0. [0. 1] are continuous non-decreasing functions that represent all path reparameterizations (Fiter & Mannila, 1994). and Teimple E Tx2 is the simple
trajectory derived by following the final velocity vector of Timputi
Tsmmple (t) Tinppe (T) + Ufinal t, t2
Here, Ufinal Timpue (T*) Timgue (T" 1) represents the velocity vector estimated from the final two points of Timput. and the time parameter extending beyond the duration of the input trajectory T'. ||-|| denotes the Eu- clidean distance in the 2D plane.
PC10-20 2C120-10
PCI 40-60
PCI 60-80
PC] 80+
high value of PCI indicates a significant deviation from the simple trajectory. Some generated examples of deviations from a straight input trajectory and their respective PCI range can be seen in Figure 3] A sample of real trajectories for these ranges are pro- vided in In(Fiuure 4] We propose that a high PCI indicates intriguing events, which are often characterized as sudden changes in direction or speed in the literature (Girase al. 202T). We provide more insights into this behav- ior in Appendix with a comprehensive break- down of the PCI statistics in GEM dataset.
Path Complexity Index
86
100
Figure 3: Generated trajectories and their values. The black paths to the left are inputs, and the colored paths are targets generated exhaustively by varying the speed, turning angle, and turn curvature.
P0
3D
50 PCI
70
90 PCT
Are all curved paths interesting? An es- sential task for trajectory prediction models is learning the road curvature from scene infor- mation. PCI is simplistically designed to not bias towards a particular type of curvature to encourage that while allowing eliminate instances of straight cruising that may domi- nate the training and evaluation phases.
Figure 4: Example trajectories with varying PCI. White the input and red is the target trajectory.

---

Published as a conference paper at ICLR 2025
Table 1: Contribution of GEM. Our dataset designed for ego-trajectory prediction with noise- free GPS positions and gaze data, in an entity-rich urban setting.
(a) Comparison of existing trajectory prediction datasets with ours. Only public datasets are considered. *: hand- annotated.
(b) GEM vs. DR percentage
frames with other agents, and total entity
counts, both detected by YOLOVS at fps.
Dotneet
Gaze
Naise free pose
Scene cameras
Duration
GEM
DRCeye/VE
Waymo Eminger all
2021
574h
% of frames with other drivers
89.7%
38.1%
nuScener
5.5h
% of frumes with pedestriams
31.2%
22.5%
Argovets Wilsor o
30Sh
LOKI (Ginn il mi MP41
2.3h
pedestrians
3640
297
DRiegSTVE (PilaZ) ara 2018
(+1 headeam)
6h
of buses
1215
155
(Oirs)
2(E hasdEaans
"5h
bicyckes
357
18
GAZE-ASSISTED EGO MOTION (GEM) DATASET
The driver's gaze plays a significant role during driving, providing insights about their decisions on the road. Understanding this role may allow However, training models in the gaze domain is constrained by data scarcity. The sole public dataset featuring ego-driver positions and gaze, DR(eye)VE, contains only hours of data, only one-third of which is urban driving, hindering the capture of rare gaze-based cues. (Palazzi et al. 2018).
build systems to improve vehicular safety.
We, therefore, propose GEM, a multimodal ego-motion with gaze dataset with synchronized multi- cam video footage of the road scene and FOV data, with precise gaze locations from an eye-tras eye-tracking headset, and GPS ego-vehicle locations, manually corrected for further accuracy. By coupling gaze data with the car's s motion and the driving scene, our dataset provides rich source of information for ego-trajectory prediction and modeling of drivers' behavior. The primary objective of our dataset is to enable the development and validation of models that predict future ego-vehicle locations based on the driver head/eye movements and surrounding scenes.
5.1 HARDWARE SETUP
We mount high-resolution cameras on the vehicle to ensure high-quality recordings of the surround- ing environment and record drivers' eye gaze using a hean-mounted eye tracker.
Frontal cameras and GPS tracking. To record the driving scene and car motion, we use two GoPro cameras with a 4K resolution and a 30Hz sampling rate. These cameras record the car's 's GPS positions in real-time and embed this information into the video streams. However, GPS data can have discrepancies due to various factors such as satellite interference or obstructions, which we further correct.
Gaze tracking. We employ the Pupil Invisible glasses from Pupil Labs to track the driver's gaze. This device captures eye movements at 200Hz using frontal video camera of the scene. The gaze tracker and GoPro cameras are fully synchronized in post-processing using the timestamp metadata.
two near-eye infrared cameras and includes
5.2 DATA COLLECTION
10 participants were instructed drive naturally on variety of road types with focus on urban areas, during different times of the day to capture comprehensive range of driving conditions. Each participant drove for approximately 30 minutes, resulting in a total of 5 hours of collected data. See Appendix D.I for breakdown of the dataset.
Procedure. Upon arrival, participants were briefed about the study and given tutorial on the equipment. Before commencing the driving sessions, participants went through a gaze calibration process. They were asked to look at specific points on a board, ensuring that the eye-tracking device was accurately capturing their fixations. After fitting the Pupil Invisible glasses and ensuring calibration, participants were asked to start their driving session. A researcher was present in the car to oversee the equipment and address any unexpected situations.
Tech Specs.
7

---

Published as a conference paper at ICLR 2025
RouteFormer (ours)
GIMO
+ Multimodal Transformer
Ground Truth
RouteFormer
RouteFormer Base
Figure 5: Qualitative examples. RouteFormer shows higher confidence sharp turns than other SOTA models using gaze, which tend to prefer the mean of the past trajectory (left). The turn confidence lower when no driver FOV information used (right).
Post-processing. GPS technology inherently exhibits limitations, including potential positional in- accuracies up to 1.82m at the 95th percentile, as reported by the U.S. FAA (U.S. FAA NSTB/WAAS) T&E Team 2021). These inaccuracies are notably magnified in occluded environments, making the resulting data unsuitable for ground truth applications (see Appendix for examples). To mitigate this, developed user-operated application that allows for the manual correction of GPS markers to more accurate positions. This correction tool has been instrumental in enhancing the positional accuracy of all samples within the GEM dataset. The tool's source code will be released alongside the dataset, We provide the data split and detailed description of the dataset the Appendix.
5.3 COMPARISON WITH EXISTING DATASETS
Most driving datasets to create road assistance systems lack either gaze data, noise-free vehicle tra- jectory, or sufficient quality scene information, as seen Table I (a). To the best of the authors' knowledge, the only other public multimodal driving dataset with gaze data is DR, as pre- sented by Palazzi al. (2018). However, this dataset includes mostly rural/highway driving (63.5% of the data), while GEM is designed for complex urban trajectory prediction with many interacting agents. Our comparison in Table (b), using YOLOV8 (Jocher et al.) (023). shows GEM's signif- icant lead in terms of traffic-reaated content: it has other vehicles in 89.7% of frames (compared to DR(sye)VE's 38.1 1%), six times more cars, over ten times more pedestrians, and twenty times more bicycles than (eyee))V.. For GEM, further ensured the high quality of the GPS informa- tion, fixing the inherent GPS limitations through hand annotation, especially in occluded areas like tunnels (see in Appendix C for examples, and the dataset noise comparison).
EXPERIMENTS
We evaluate our method RouteFormer by measuring the predicted motion trajectories using a set of standard metrics. We compare against several state-of-the-art methods and baselines. See Appendix B for more results and examples.
6.1 SETUP
Datasets. We evaluate our method on our proposed GEM dataset. To demonstrate that the benefit from gaze extends beyond our dataset, we additionally train and evaluate our framework on a second dataset. DR (Palazzi et al.) 2018) has 6 hours of driving along with gaze input, however, with a smaller number traffic entities than GEM, and lower data quality.
Task. Following the task setting in nuScenes (Caesar et al. 2020), our task uses 8-second input trajectories to predict the next 6-second motion. We sample synchronized sequences of location, videos, and gaze data with sliding window of 2-second strides, resulting in approximately 16 hours of input for the training split. We choose different drivers for training and evaluation according to the data splits of either dataset (see Appendix D) to better demonstrate the benefit of FOV.
8

---

Published as a conference paper at ICLR 2025
Table Comparison to baselines and SOTA. We use the test sets both the proposed dataset and
the DR dataset. Our model RouteFormer achieves the best performance on both datasets.
GEM
DR
Method
ADE (m)
ADE+20PCI (m)
ADE (m)
2 (m)
Stationary Baseline
32.04
30.22
64.11
61.41
Linear Baseline
7.37
13.37
10.16
15.54
GIMO (Zheng el al., 2022
TO.03
13.97
18.68
Multimodal Transtormer LL
2021
6.07
8.41
10.67
14.65
Ours - RouteFormer- Base (video only)
6.13
8.14
9.95
13.15
Ours RouteFormer (vidco+gaze)
5,99
7.70
8.75
12.26
Table 3: Varying the prediction horizon. Average final displacement errors GEM for predicting 1-6s the future for test set samples with 20+ PCI, demonstrating RouteFormer's increasingly higher accuracy in long-term forecasting for complex samples, compared to RouteFormer-Baat
Method
FDE6
FDE@3s
FDE@4s
FDE@
RouleFormer Base (video only)
1.74
4.19
7.08
10.56
14.45
18.45
RouteFormer (video=gaze)
1.68
4.04
6.72
9.93
13.61
17.37
Improvement
3.57%
3.71%
5.36%
6,34%
6.17%
6.21%
Baselines. We two simple baselines. Stationary baseline assumes that the vehicle remains stationary throughout the prediction window. Linear baseline assumes that the vehicle maintains a linear trajectory, following the vehicle's final direction and speed from the input.
Motion prediction models. We compare our model with two gaze-assisted human motion predic- tion models adapted from the literature. Multimodal Transformer (Li et al. (22TT) encodes the input modalities with linear layers and predicts the future trajectory with an encoder/decoder ar- chitecture. In addition, RouteFormer's own video encoder instead of 3D point clouds and predict the future trajectory with cross-modal encoder architecture. The adaptations follow Zheng et al. (2022)'s adjustments in GIMO to compare with their baseline Multimodal Transformer (referred to in Tab. 2).
implement GIMO (Zheng al.] 2022) to utilize scene videos using
Metrics. We use the standard evaluation metrics Average Displacement Error (ADE) and Final Displacement Error (FDE). ADE measures the average eтror between the ground truth and predicted trajectory across all time steps. FDE measures the displacement error the final prediction time step, emphasizing the model's accuracy in predicting the endpoint of trajectory.
6.2 EVALUATION OF ROUTEFORMER
Comparison to baselines and SOTA. We report results on the test sets both our proposed dataset GEM and the VE dataset in Table We train 8 participants and test 2 others to explore generalizability across people. Our model RouteFormer outperforms all baselines across both evaluation metries and, therefore achieving state-of-the-as performance on both datasets.
we display the behavior of the proposed model compared to SOTA and baselines. RouteFormer shows better confidence in sharper turns compared other mod- els, predicts right exits in roundabouts, and overall has better performance in high-PCI situations.
Qualitative evaluation. In Figure 5,
Evaluating the benefit of driver FOV. We compare RouteFormer with RouteFormer-Be Base, its ver- sion without driver FOV (Table 2}. The additional modality in RouteFormer significantly improves performance, especially for later points in the prediction horizon (Talle 3). This confirms the im- portance of driver field-of-vier information in ego-trajectory prediction, particularly for long-tern term forecasts. Notably, RouteFormer achieves lane-level accuracy for shorter horizons even challeng- ing GEM samples, demonstrating potential for industrial short-term forecasting applications.
Path complexity evaluation. We evaluated RouteFormer and RouteFormer- levels of path complexity using the PCI metric. Results in Figure 6 show that gaze data becomes increasingly valuable for more irregular trajectories. This demonstrates RouteFormer's ability to generalize driver FOV learning, improving predictions for complex paths across different drivers.
across different
Gaze fixation importance. Throughout this study, we consider driver's 's FOV as a composition of a first-person video feed and gaze fixations, as they highly interconnected and novel remote track- ing systems (such as Smart Eye Pro) can capture them together (Smart Eye AB 2024). To verify the importance of gaze fixations beyond first-person video, we analyzed their attribution using Inte-

---

Published as a conference paper at ICLR 2025
Rauteforrmrr -rizae-gaze)
RoutePormer (vidno)
Improvenio ADE(M)
60-80
15
9:54
1.53
2:22
improvement with Gaze
40-60
3
ADE (m)
10
9.5
10.0
0.4
PCI Range
6.27
0.72
1.29
2.04
2.91
3.83
7.5
20-40
2
0.2
0.76
2.20
3.17
0-20
0
0.0
6.24
0.67
1.19
1.84
2.59
3.39
0-20
20-40
40-60
60-80
Path Complexity Index
2
Prediction Horizon (seconds)
Figure 6: The effect of gaze over different PCI. Lower better for ADE. We bin GEM's test set by PCI. displaying average ADE per group. The line represents the ADE difference between RouteFormer-Bil Base and RouteFormer, with higher values favoring RouteFormer.
Figure 7: Attribution to gaze fixations across different PCI bins and prediction horizons. Gaze fixations increasingly affect the prediction for further away horizons and medium/high PCI values. Estimated using In- tegrated Gradients (Sundararajan et al. 2017).
grated Gradients (Figure 7). Fixations significantly impact the final result, with increased reliance for later prediction points and higher PCI samples. This confirms that RouteFormer effectively utilizes both video and gaze data, despite having the same parameter count RouteFormer- BBse.
6.3 ABLATION STUDIES
Table Loss ablations. Results over GEM vali- dation set for 20+ PCI samples.
We performed set of ablation studies to choose the right vision backbones and time se- ries modules for RouteFormer, and justify the discounted/auxiliay losses.
Method
Val. ADE (m)
Val. FDE (m)
Vanilla RouteFormer
8.85
58.54
+ auxiliary losses
8.68 (-0.17)
$7.09 (-1.4)
Auxiliary and discounted losses. As one of the major contributions of our framework, we assess effect of the future-discounted loss Lya and the auxiliary losses. Both additions show significant improvement over the standard architecture on the validation set (Table 4).
+ future-discounted loss
8.27 (-0.41)
54.38 (-2.71)
Table 5: Visual encoder and time-series mod- ule ablations. Results over GEM val. set for 20+ PCI samples. Time-series modules (top) perform prediction without scene gaze data. Pretrained vision modules (bottom) use Informer.
Time-series module. Informer (Zhou et al. 2021 is Transformer-basee based model and lever- ages the ProbSparse attention mechanism. TT-
Mishod
Val. ADE (m)
Val. FDE (m)
DLinear Zeng 2023
10.67
$1.40
N1.ncar
7air B.
10.49
ble shows that Informer consistently outper-
PatchTST
Ni 20/3
10.23
79 80
forms later models such as Autoformer (Wu
Transformer
Zhio
2017
9.97
67.88
al. 202T). LTSF-Linear (Zeng et al., (003)) and PStchTST (Nie et al.] 2023). We used In-
InformasAA Kinllow al 2023
9.08
$7.89
Informer:Dino DinoV2
8.69
60.83
former in RouteFormer.
Informer-Swin2
ara
Pretrained vision backbone. In Table 5| we evaluate the impact of various visual encoders, includ- ing SAM (Segment Anything) Kirillov et al., 2023), DinoV2 (Oquab al. 2023) and SwinV2 (Liu et al. 202T). SwinV2 consistently demonstrated superior performance on the validation set with embedding sizes smaller than its counterparts.
7 CONCLUSION
Our results highlight the significance of incorporating driver FOV in predicting ego-vehicle trajec- tories. Through our innovative multimodal method, RouteFormer, which integrates FOV, scene and motion data, we achieve superior performance over existing baselines, pushing the boundaries of state-of-the-an achievements this domain. Specifically, our approach demonstrates that incor- porating first-person video and gaze fixations data enhances the prediction of complex non-lineat trajectories. By establishing a new benchmark with our GEM dataset, we pave the way for further intuitive and human-centric developments in assisted driving technologies.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
INSAIT, Sofia University "St. Kliment Ohridski". Partially funded by the Ministry of Education and Science of Bulgaria's ss support for INSAIT as part of the Bulgarian National Roadmap for Research Infrastructure. This project was supported with computational resources provided by Google Cloud Platform (GCP). This project is also partially funded by the German Federal Ministry of Education and Research through the ExperTeam-KI funding program for UDance (Grant No. 01IS24064).
REFERENCES
Henny Admoni and Siddhartha Srinivasa. Predicting user intent through eye gaze for shared auton- omy. In 2016 AAAI Fall Symposium Series, 2016.
Christer Ahlström, Katja Kircher, Marcus Nyström, and Benjamin Wolfe. Eye tracking in driver attention research- how gaze data interpretations influence what we learn. Frontiers in neuroer- gonomics, 2:778043, 2021.
Roland Allenbach, Steffen Niemann, Yvonne A. Stürmer, Patrizia Hertach, Karin Huwiler, and Jasmin Zimmermann. Sinus 2021 - sicherheitsniveau und unfallgeschehen im strassenverkehr 2020. Bern: BFU, Beratungsstelle für Unfallverhitungg 2021. doi: 10. 10.13100/FU..2403.0000 2021.
Pierluigi Vito Amadori, Tobias Fischer, Ruohan Wang, and Yiannis Demiris. Decision anticipa- tion for driving assistance systems. 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1-7, 2020. doi: 10.
Samer Ammoun and Fawzi Nashashibi. Real time trajectory prediction for collision risk estimation between vehicles. In 2009 IEEE 5Th international conference on intelligent computer communi- cation and processing, pp. 417-422. IEEE, 2009.
Michael Argyle, Mark Cook, and Duncan Cramer. Gaze and mutual gaze. The British Journal of Psychiatry, 165(6))848--) 850, 1994. doi: 00.1017/0007125000007
Ulrich Baumann, Claudius Guiser, Michael Herman, and J Marius Zollner. Predicting ego-vehicle paths from environmental observations with a deep neural network. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 4709-4716. IEEE, 2018.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020.
Chiho Choi, Joon Hee Choi, Jiachen Li, and Srikanth Malla. Shared cross-modal trajectory predic- tion for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 244-253, 2021.
Thomas Eiter and Heikki Mannila. Computing discrete fréchet distance. 1994. URL https://api. wwwp://hoooooooogooooosDDDDD.d.o CorpussD:16000056666
semanticscholar.orgg
Johan Engström, Emma Johansson, and Joakim Östlund. Effects of visual and cognitive load in
real and simulated motorway driving. Transportation research purt traffic psychology and behaviour, 8(2):97-220, 2005.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Benjamin Sapp, Charles R. Qi, Yin Zhou, Zoey Yang, Aurelien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving The waymo open motion dataset. CoRR, abs/2104. 10133, 2021. URL https///aariivvoo///abbe// 10133
Jianwu Fang, Dingxin Yan, Jiahuan Qiao, Jianru Xue, and Hongkai Yu. Dada: Driver attention prediction in driving accident scenarios, 2023. URL https://ariivorg/abb//111....
Alireza Fathi, Yin Li, and James M Rehg. Learning to recognize daily actions using gaze. In Computer Vision ECC--ECCV-CCC 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part I 12, pp. 314-327. Springer, 2012.
11

---

Published as a conference paper at ICLR 2025
Xidong Feng, Zhepeng Cen, Jianming Hu, and Yi Zhang. Vehicle trajectory prediction using intention-basee conditional variational autoencoder. In 2019 IEEE Intelligent Transportation Sys- tems Conference (ITSC), pp. 3514-3519. IEEE, 2019.
Xiang- Yu Gao, Yu-Fei Zhang, Wei-Long Zheng, and Bao-Liang Lu. Evaluating driving fatigue de- tection algorithms using eye tracking glasses. In 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 767-770. IEEE, 2015.
Shreya Ghosh, Abhinav Dhall, Garima Sharma, Sarthak Gupta, and Nicu Sebe. Speak2label: Using domain knowledge for creating large scale driver gaze zone estimation dataset. In Proceedings of the IEEE/CVÉ International Conference on Computer Vision, pp. 2896-2905, 2021.
Harshayu Girase, Haiming Gang, Srikanth Malla, Jiachen Li, Akira Kanehara, Karttikeya Man- galam, and Chiho Choi. Loki: Long term and key intentions for trajectory prediction. In Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pp. 9803-9812, 2021.
Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D' D'Souza, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Latent variable sequential set trans- formers for joint multi-agent motion prediction. arXiv preprint arXiv:2104.0083 2021.
Deepak Gopinath, Guy Rosman, Simon Stent, Katsuya Terahata, Luke Fletcher, Brenna Argall, and John Leonard. Maad: A model and dataset for "attended awareness" in driving, 2021. URL
https:/aarxivorr/bb//11111.000
Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15303-15312, October 2021.
Mahir Gulzar, Yar Muhammad, and Naveed Muhammad. A survey on motion prediction of pedes- trians and vehicles for autonomous driving. IEEE Access, 9: 91137957-137969, 2021.
Zengyi Han, Liqiang Xu, Xuefu Dong, Yuuki Nishiyama, and Kaoru Sezaki. Headmon: Head dy- namics enabled riding maneuver prediction. In 2023 IEEE International Conference Pervasive Computing and Communications (PerCom), pp. 22-31. IEEE, 2023,
Mary M. Hayhoe, Anurag Shrivastava, Ryan Mruczek, and Jeff B. Pelz. Visual memory and motor planning in a natural task. Journal of Vision, 3(1):6-6, 02 2003. ISSN 1534-7362, doi: 10.1167/ 3.1.6. URL
Anneke Heitmann, Rainer Guttkuhn, Acacia Aguirre, Udo Trutschel, and Martin Moore-Fde, Tech- nologies for the monitoring and prevention of driver fatigue. In Driving Assessment Conference, volume University of Iowa, 2001.
Yanjun Huang, Jiatong Du, Ziru Yang, Zewei Zhou, Lin Zhang, and Hong Chen. A survey on prajectory-prediction methods for autonomous driving. IEEE Transactions on Intelligent Vehicles, 7(3):652-674, 2022.
Ashesh Jain, Hema S. Koppula, Bharad Raghavan, Shane Soh, and Ashutosh Saxena. Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models, In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3182-3190, Santiago, Chile, December 2015. IEEF. ISBN 978-1-4673-8391-2. doi: 10. 2..119//CCCV.2015..666
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics YOLO, January 2023. lttp::///g/g com//utrallttes/uutaaayyiy
ByeoungDo Kim, Chang Mook Kang, Jaekyum Kim, Seung Hi Lee, Chung Choo Chung, and Jun Won Choi. Probabilistic vehicle trajectory prediction over occupancy grid map via recur- rent neural network. In 2017 IEEE 20Th international conference on intelligent transportation systems (FTSC), pp. 399 404. IEEF, 2017.
Dongchan Kim, Hyukju Shon, Nahyun Kweon, Seungwon Choi, Chanuk Yang, and Kunsoo Huh. Driving style- conditional variational autoencoder for prediction of ego vehicle trajectory. IEFE Access, -199388-693566 2021.
12

---

Published as a conference paper at ICLR 2025
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02644 2023.
Georgios K Kountouriotis, Richard M Wilkie, Peter H Gardner, and Natasha Merat. Looking and thinking when driving: The impact gaze and cognitive load on steering. Transportation re- 2015.
search part F: traffic psychology and behaviour,
Yung-Cur Chi Kung, Arthur Zhang, Junmin Wang, and Joydeep Biswas. Looking inside out: Anticipat- ing driver intent from videos. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 8608-5614 IEEE, 2024.
M. Land and David Lee. Where look when steer. Nature, 369:742-744, 07 1994, doi: 10. 10.1038/369742a0
Michael Land. Eye movements and the control of actions everyday life. Progress in retinal and eye research, 25(3):296-324, 2006.
Sukhan Lee, Muhammad Qasim Khan, and Mohd Nizam Husen. Continuous car driving intent detection using structural pattern recognition. IEEE Transactions on Intelligent Transportation Systems, 1001):100 2021. doi: 10.1 109/TITS SETTTS.2019.20 2019.2961928.
Chao Li, Zhanwen Liu, Shan Lin, Yang Wang, and Xiangmo Zhao. Intention-convolution and hybrid-attention network for vehicle trajectory prediction. Expert Systems with Applications, 236: 121412, 2024a.
Jingyuan Li, Titong Jiang, He Liu, Yingbo Sun, Chen Lv, Qingkun Li, Guodong Yin, and Yahui Liu. Lane changing maneuver prediction by using driver's ppapooltemporal gaze attention inputs for naturalistic driving. Advanced Engineering Informatics, 61:102529, 2024b.
Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa. Ai choreographer: Music condi- tioned 3d dance generation with aist++, 2021.
Haicheng Liao, Yongkang Li, Zhenning Li, Chengyue Wang. Chunlin Tian, Yuming Huang, Zilin Bian, Kaiqun Zhu, Guofa Li, Ziyuan Pu, Jia Hu, Zhiyong Cui, and Chengzhong Xu. Less is more: Efficient brain-inspired learning for autonomous driving trajectory prediction. In Proceedings of the European Conference on Artificial Intelligence (ECAT), pp. 4361-4368, 2024a.
Haicheng Liao, Shangqian Liu, Yongkang Li, Zhenning Li, Chengyue Wang, Yunjian Li, Shengbo Eben Li, and Chengzhong Xu. Human observation-inspired trajectory prediction for autonomous driving in mixed-autonomy traffic environments. 2024 IEEE International Con- ference Robotics and Automation (ICRA), pp. 44212-14219 IEEE, 2024b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Kedi Lyu, Haipeng Chen, Zhenguang Liu, Beiqi Zhang, and Ruili Wang. 3d human motion predic- tion: A survey. Neurocomputing, 489:345 365-335--6,,-0 2022.
Yunsheng Ma, Wenqian Ye, Xu Cao, Amr Abdelraouf, Kyungtae Han, Rohit Gupta, and Ziran Wang. Cemformer: Learning to predict driver intentions from in-cabin and external cameras via spatial- temporal transformers. In 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), pp. 4960-4966. IEEE, 2023.
Srikanth Malla, Behzad Dariush, and Chiho Choi. Titan: Future forecast using action priors. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11 186 11196, 2020.
Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415, 2023.
13

---

Published as a conference paper at ICLR 2025
Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple efficient attention networks. In 2023 IEEE International Conference Robotics and Automation (ICRA). pp. 2980-2987. IEEE, 2023.
Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelof', Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified architecture for predicting multiple agent trajectories. arXiv preprint arXiv:2106.08417, 2021.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference Learning Representations, 2023.
OpenStreetMap contributors. https//wwww.peentteeemmgg org, 2017.
Planet dump retrieved
from htps://plantt.csm.oorg
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-War Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Juan Diego Ortega, Neslihan Kose, Paola Cañas, Min-An Chao, Alexander Unnervik, Marcos Nieto, Oihana Otaegui, and Luis Salgado. Dmd: A large-scale multi-modal driver monitoring dataset for attention and alertness analysis. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16. pp. 387-405. Springer, 2020.
Andrea Palazzi, Davide Abati, Simone Calderara, Francesco Solera, and Rita Cucchiara. Predicting the driver's focus of attention: the dr(eye)ve project. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2018.
Hyun Soo Park, Jyh-Jing Hwang, Yedong Niu, and Jianbo Shi. Egocentric future localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4697- 4705, 2016.
Petr Pridal and Tomas Pohanka. Wgs 84 pseudo-mercator spherical mercator, 2024. URL https: //epsg.io/3851
Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary R. Bradski. Kornia: open source diffcrentiable computer vision library for pytorch. CoRR, abs/1910.02190, 2019. URL http://aaxxv.orrg/////9901000000
Rafael F, Ribeiro and Paula D. P. Costa. Driver Gaze Zone Dataset With Depth Data. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1-5,
May 2019. doi: 10. L00/FF.2019.8755577
Ivan Rodin, Antonino Furnari, Dimitrios Mavroeidis, and Giovanni Maria Farinella. Predicting the future from first person (egocentric) vision: survey. Computer Vision and Image Understanding, 211:103253 2021.
Tim Salzmann, Boris Ivanovic. Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasihle trajectory forecasting with heterogeneous data. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pp. 683-700. Springer, 2020.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasihee trajectory forecasting with heterogeneous data. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1122-1 130, 2021.
Wilko Schwarting, Javier Alonso-Mora, and Daniela Rus. Planning and decision-making for autonomous vehicles. Anmual Review of Control, Robotics, and Autonomous Systems, 1(1): URL https://do..or//0..14466
187-210, 2018. doi: 10 17t
14

---

Published as a conference paper at ICLR 2025
Anke Schwarz, Monica Haurilet, Manuel Martinez, and Rainer Stiefelhagen. Driveahead-a large- scale driver head pose dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1-10, 2017.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems, 35:6531-6543, 2022a.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems, 2022b.
Krishna Kumar Singh, Kayvon Fatahalian, and Alexei A. Efros. KrishnaCam: Using longitudinal, single-person, egocentric dataset for scene understanding tasks. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1-9, Lake Placid, NY, USA, March 2016. IEEE.
ISBN 978-1 1-0----090000411--0 doi: 10. 109//ACVV2016.777777
Smart Eye AB. Smart Eye Pro remote eye tracking system, 2024. URL https//www.smmrreee..es Smart-eye--roo)
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowci Zhou. LoFTR: Detector-free local feature matching with transformers. CVPR, 2021.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. CoRR, 0ts/1703.013655 2017. URL http://arri..ogg/ab/1111.......1
Izzeddin Teeti, Salman Khan, Ajmal Shahbaz, Andrew Bradley, and Fabio Cuzzolin. Vision-based intention and trajectory prediction in autonomous vehicles: A survey. In Lud De Raedt (cd.), Pro- ceedings of the Thirty-Firs International Joint Conference on Artificial Intelligence, IJCAI-22, International Joint Conferences Artificial Intelligence Organization, 7 2022. doi: 10. /tt.966//jaii22222/77. URL httppp://bo..rrg/00.29.3... 2022/7778 Survey Track.
pp.
Mare Tonsen, Chris Kay Baumann, and Kai Dierkes. high-level description and performance evaluation of pupil invisible. arXiv preprint arXiv:2009.00550 2020.
Jochen Triesch, Dana Ballard, Mary Hayhoe, and Brian Sullivan. What you see is what you need. Journal of vision, 3:86-94,00 02 2003. doi: 10. 167/3.....
Michal J. Tyszkiewiez, Pascal Fua, and Eduard Trulls. Disk: Learning local features with policy gradient, 2020.
Geoffrey J Underwood, Peter Chapman, David Crundall, Sarah Cooper, and Robert Wallen. The visual control of steering and driving: Where do we look when negotiating curves? Vision in vehicles, 7:245-255 1999.
U.S. FAA NSTB/WAAS T&E Team. Global Positioning System Standard Positioning Ser- vice Performance Analysis Report. January 2021. URL https:/wwww.nsb.cc.aa.aa gov//rporrs
PAN PAN.v...p
Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Mul- tipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In 2022 International Conference on Robotics and Automation (ICRA). pp. 7814-7821. IEEE, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30, 2017.
Koen Vellenga, H. Joe Steinhauer, Göran Falkman, and Tomas Björklund. Evaluation of video masked autoencoders' performance and uncertainty estimations for driver action and intention recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 7429-7437, January 2024.
15

---

Published as a conference paper at ICLR 2025
Chalavadi Vishnu, Vineel Abhinav, Debaditya Roy, C Krishna Mohan, and Ch Sobhan Babu. Im- proving multi-agent trajectory prediction using traffic states on interactive driving scenarios. IEEE Robotics and Automation Letters, 8(5):2708-271)) 2023.
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandel- wal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021). 2021.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- with Cuto-Correlltion for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.
Min Wu, Tyron Louw, Morteza Lahijanian, Wenjie Ruan, Xiaowei Huang, Natasha Merat, and Marta Kwiatkowska. Gaze-based intention anticipation over driving manoeuvres in semi-autonomous vehicles. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2210-6216, 2019. doi: 10.1109//0040
Ye Xia, Danqing Zhang, Alexei Pozdnoukhov, Ken Nakayama, Karl Zipser, and David Whitney. Training network to attend like human drivers saves it from common but misleading loss func- tions. CoRR,
2017. URL http://aarxi..or//bbb///11 http://bb///111.100.00
Zhanhong Yan, Bo Yang, Zheng Wang, and Kimihiko Nakano. A predictive model of a driver's target trajectory based estimated driving behaviors. Sensors, 23(3):1405, 2023.
Yangtian Yi, Chao Lu, Boyang Wang, Long Cheng, Zirui Li, and Jianwei Gong, Fusion of gaze and scene information for driving behaviour recognition: A wraph-neural-netweee based framework. IEEE Transactions on Intelligent Transportation Systems, 24(8):8109 8120, 2023. doi: 10.1 109/ TTTS.2023.32638755
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? 2023.
Weili Zeng, Xiao Chu, Zhengfeng Xu, Yan Liu, and Zhibin Quan. Aircraft 4d trajectory prediction in civil aviation: A review. Aerospace, 9(2)991, 2022.
Mingfang Zhang. Yunfei Liu, and Feng Lu. GazeOnee: Real-Time Time Multi--Prronn Gaze Estimation. In 2022 IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4187- 4196, New Orleans, LA,, USA, June 2022. IEEE. ISBN 978-1-66546-946-3. 0ERR5268..0222.04422
doi: 10.1 109/
Xiaoji Zheng, Lixiu Wu, Zhijie Yan, Yuanrong Tang, Hao Zhao, Chen Zhong, Bokui Chen, and Jiangtao Gong. Large language models powered context-aware motion prediction. arXiv preprint arXiv:2403. 1057, 2024.
Yang Zheng. Yanchao Yang. Kaichun Mo, Jiaman Li. Tao Yu, Yebin Liu, Karen Liu, and Leonidas human motion prediction in context. In ECCV, 2022.
Guibas. Gimo:
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. The Thirty- Fifth AAAI Conference Artificial Intelligence, AAAI 2021, Viriual Conference, volume 35, pp. 11106- 1066-1111 11115. AAAI Press, 2021.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
RouteFormer Implementation Details
17
A.
RouteFormer Notation
17
Module-Level Caching with TORCHCACHE
18
Experiments
B.
Evaluation Metrics
19
B.2
Hyperparameters
19
B. 3
Analysis of Failure Cases
19
Ablation Studies on Modality Contributions
20
B.5
Extended Baseline Selection Discussion
21
GEM Dataset
22
Ground Truth Correction with TRAJAPP
22
C.2
GEM vs. DR(cye)VE Noise Comparison
22
Gaze Fixation Detection
22
C.4
Gaze Alignment and Robustness
24
Exploring Image Homography and Stitching
24
PCI Metric
27
D.I
Breakdown of GEM Dataset and PCI Insights
ii..............................
27
D.2 Visualizing Path Complexity Index
27
D33 PCI Analysis Urban Driving
28
A ROUTEFORMER IMPLEMENTATION DETAILS
All experiments are carried out machine with an NVIDIA GPU with > 10 GB memory, 64 GB of RAM, and an Intel CPU. The framework is implemented using PyTorch 2.0.
A. ROUTEFORMER NOTATION
Using the notation we have established, full model can be described as:
(Mrr, s1:K S1:K Firl
T (MrT, Ep(ss EF (Fi:))))))
[M'T UTTTTpa
Loss definitions. For the model outputs V (M S1ik FirTA) {MT-Tma Traa let Lr denote the discounted trajectory loss, and Lv the discounted visual loss. The auxiliary losses defined as:
Lx = Lis (MrTTma Mr.1Fpo))
Lv = Les (oT:ITar Ep (EsEEE(S (5:()) Er EF (FFTTT)a)))
17

---

Published as a conference paper at ICLR 2025
A.2 MODULE-LEVE CACHING WITH TORCHCACHE
Caching is a crucial component of our training regime. During the initial epochs, the intermediate representations from the pre-trained vision backbones computed for each input frame and stored in an efficient cache memory structure. When subsequent epochs require the same input data, the pre-computed representations are fetched from the cache, thereby eliminating the need for repetitive and costly forward passes through the vision backbones.
To make this process repeatable, implemented the TORCHCACHE utility to provide a generic caching mechanism. The TORCHCACHE library offers an easy way to cache the outputs of PyTorch modules, making particularly suited for our need to efficiently cache the outputs of large, pre-trained vision transformers. The package of this mechanism is open-sourced at gihub..om/mnakkbyi/oooccaaaa
A.2.1 KEY FEATURES
The TORCHCACHE library offers several notable features:
Allows caching of PyTorch module outputs in two formats: in-memory for fast retrieval, and persistently on the disk for larger datasets.
Adopts decorator- based interface that simplifies its integration with PyTorch modules.
Implements an MRU ( most-recently-usee) cache policy to manage and limit memory or disk consumption.
A.2.2 USAGE
The library's decorator-based design ensures minimal modifications to the existing Py PyTorch code- base. By adding the @ stochcach cached, illustrated in the provided basic usage example:
() decorator, the output of the module's 's forward method gets
from torchcache import torchcache
@torchcache ()
class MyModule Monuuee) def init (self):
super init (n)
self. self.linear = I.. (10, 10) def (sel,, x):
This output will be cached return self. linear (x)
More detailed examples, advanced usage patterns, and further documentation will be accessible at the official TORCHCACHE documentation.
A.2.3 OPERATIONAL ASSUMPTIONS
For the TORCHCACHE library to operate seamlessly, there are certain assumptions that the Py Torch modules should adhere to:
1. The module should be subclass of nn. Module.
2. The forward method of the module should be capable of accepting any number of positional arguments with shapes denoted by (B, +), where B stands for the batch size, and + denotes any other dimensions.
3. All input tensors should be located the computational device and should share the same data type (dtype).
4. The method should return a single tensor with the shape (B, +).
The TORCHCACHE library has proven invaluable in experiments by allowing to efficiently utilize the outputs our pre-trained vision transformers without incurring significant computational overhead.
18

---

Published as a conference paper at ICLR 2025
(=) Os
(b) 1s
(c) 1 2s
(d) L t= 3s
(e) 1 = 4s
14
Speed (m/s)
10
6
0.0
1.0
2.0
3.0
4.0
Time (8)
Figure 8: A sample from the GEM dataset. The top shows frames with gaze heatmap overlay on top, at Hz. The bottom left image displays the trajectory of the vehicle, while the bottom right plot shows the speed changes. This is example of gaze-indicated short-tern intent: the driver decelerates as soon as their gaze meets with the brake lights of the vehicle in front.
B EXPERIMENTS
B.1 EVALUATION METRICS
The Average Displacement Error (ADE) is formally defined as:
T ADE
- Σ
lp,
peed
P 1l
where 7 is the total number of prediction time steps, p. is the predicted position at time I, and p the ground truth position the same time.
The Final Displacement Error (FDE) is the displacement error at the final prediction time step:
FDE
pred
B.2 HYPERPARAMETERR
The hyperparameters for the experiments can be found in Table Table 7
Table 8
and Table
B.3 ANALYSIS FAILURE CASES
We observe two primary categories of prediction failures:
Occlusions in right turns: As shown in Figure 9a and 9b trees and parked vehicles occlude crucial parts of the upcoming path during right turns. While human drivers naturally compensate for such occlusions through experience, our model shows increased uncertainty in these scenarios. Reliance additional information from the car, such turn signals or LIDAR, can improve driver behavior prediction for such instances.
2. Complex intersections: Figure 9c and 9d demonstrate varying accuracy in predicting motion at complex intersections and turns. The model particularly struggles determining when the driver going to stop resume in complex intersections (Figure 9c) and when multiple path options available (Figure 9d). This limitation suggests an opportunity to incorporate past driver behavior to model driver inirrative/aggressonn.
19

---

Published as a conference paper at ICLR 2025
Table Training configurations
Table 7: Framework configurations
Parameter
Value
Parameter
Value
Optimizer
AdamW
Encoder heads
Learning rate
1e-5
Encoder layers
8
Weight decay
le-4
Feature dropout
0.05
Epochs
200
Encoder hidden size
64
Epsilon
Gaze decoder layers
Visual epsilon
0.3
Gaze dropout
0.2
Scene video FPS
View dropout
0.6
Gaze video FPS
Front video scaling factor
0.3
Batch size
16
Scene video scaling factor
0.1
Output FPS
Gaze decoder dropout
0.05
Loss discount factor
0.97
Image embedding size
64
Dense visual loss ratio
0.5
Table 8:
GPS backbone (Informer) configura-
Table 9: Video backbone (SwinV2) configura-
tions
tion
Parameter
Value
Parameter
Value
Distil
true
Model Type
swinv2 window 12to16 true
Factor
Pad to square
Model dimension
832
Train backbone
false
Dropout
0.0
Number Heads
Sequence Length
40
Decoder layers
Encoder layers
Prediction Length
30
Label Length
40
Activation
relu
Individual false
Moving average 25
B.4 ABLATION STUDIES ON MODALITY CONTRIBUTIONS
Following our architecture in Figure 2(a), we conducted ablation experiments by selectively re- moving modality branches from RouteFormer during inference. For FOV removal, we bypass the cross-modal transformer and use only scene features for self-attention. For scene removal, we retain only the FOV branch. When testing motion-only performance, we replace the entire visual feature tensor UTr with zeros.
Method
ADE (m)
A ADE (%)
ADE+20PCI (m)
A ADE+20PCI (%)
RouteFormer (full)
5.99
Baseline
7.70
Baseline
motion+scene (w/ FOV)
6.60
+10.2
8.58
+11.4
Motion+FOV (w/o scene)
7.83
+30.7
11.17
+45.1
motion only
8.19
+36.7
11.22
+45.7
Table 10: Ablation study results showing degraded performance in removing different modalities.
The results, show in Tab. 10 demonstrate the clear benefits of each modality. Notably, our scene- only variant maintains performance consistent with RouteFormer Base (6.13m), and our FOV-only variant remains competitive with GIMO (7.61m). This stability across modality configurations is particularly valuable for real-world deployment, where sensors or gaze tracking might be temporar- ily unavailable.
20

---

(d) Driver does not stop while models do Published as a conference paper at ICLR 2025
An unanticipated right
(b) right occluded by
(c) Driver stops while model predict right turn
Figure 9: Failure cases analysis showing different scenarios where the model prediction de- viates from ground truth (green). Blue RouteFormer, red Routeformer- Base (without FOV modality), and white is the input trajectory.
B.5 EXTENDED BASELINE SELECTION DISCUSSION
We detail here our approach to baseline selection and the challenges in adapting vehicle-specific models to our task. As the recent state-of-the-arn methods work with multi-agent related traffic information, we consider such models (Autobots Girgis et al. (2021), MTR (Shi et al. (022b)) as potential baselines.
We adapted Autobots Girgis et al. (2021) and setting other inputs (agents_ing ji. road graph) to zeros/placeholders. The model's performance was significantly worse than baselines (ADE of 10.87m GEM). This underperformance stems from the fundamental mismatch between the input modalities and architectural design of these mod- els, which are optimized for rich environmental context from multi-agent interactions rather than driver-centric prediction. The key architectural differences between ego-trajector and multi-agent prediction approaches summarized in Tab.
our datasets by providing past ego-trajectory (ego in)
Given the constraints, we focused on baselines that enable fair evaluation of our core contribu- tion integrating driver FOV data for trajectory prediction. GIMO and Multimodal Transformer
21

---

Published as a conference paper at ICLR 2025
Feature
MTR/Autobots
RouteFormer (Ours)
Primary Input
BEV, HD maps, LIDAR Multi-agent trajectories
Egocentric video, FOV data Ego-vehicle trajectory
Target Output
Table 11:
Architectural differences between ego-motion and multi-agent prediction ap-
proaches.
were selected because they support similar input modalities while representing the state-of-the-an in attention-based trajectory prediction.
C GEM DATASET
In this work, we proposed GEM, multimodal ego-motion with gaze dataset with synchronized multi-cam video footage of the road scene, precise gaze locations from an eye-tracking headset and GPS ego-vehicle locations, manually corrected for further accuracy. We provide an additional sample scene from the dataset in Figure 8, demonstrating the connection between driver FOV and the short-term intentions of the driver.
C.1 GROUND TRUTH CORRECTION WITH TRAJAPP
GPS data, particularly those collected using devices like GoPro cameras, can often be noisy and potentially inaccurate due various reasons, ranging from satellite interference to obstructions like tall buildings or tunnels. Such inaccuracies in geospatial data can severely hamper the reliability and utility of dataset, especially when the data is intended for detailed analysis and modeling. Examples of such cases, as well as milder issues, can be seen in Figure 10
In response this challenge, we have developed the desktop tool TRAIAPP, designed to facilitate the correction of noisy GPS data. The primary strength of TRAJAPP lies in its unique user interface which juxtaposes the video feed and the associated GPS location points on map. This side-by-side presentation enables users to cross-verify the video content with the GPS coordinates visually.
Using the application intuitive and straightforward, an example of which can be seen in Figure 11 As users watch the video playback, they can simultancously observe the GPS path on the map. If discrepancies are noted, users can manually re-lahel the GPS data by simply clicking on the map and dragging the points to the desired accurate location. One key feature of TRAJAPP is its inter- polation mechanism. Once users adjust certain key points on the map, the application automatically interpolates the GPS data between these points, ensuring smooth and consistent transitions.
Designed with user-friendliness mind, TRAJAPP is cross-platform, with versions available for Windows, MacOS, and Linux. This ensures that researchers and dataset curators across diverse technological environments can access and utilize the tool to enhance the accuracy of their geospatial data. The application can be downloaded for free ggithbb.com/meakkbyii/rrgg.grr
C.2 GEM vs. DR(EYE)VE NOISE COMPARISON
To demonstrate the difference in degree noise between the proposed GEM dataset and the driv- ing gaze saliency dataset from the literature, DR(cye)VE, we plotted the distribution of the GPS distances the nearest road center for each dataset in Figure 12] The road centers are gathered from the public data in OpenStreetMap. For GEM, the distance distribution is centered around 1.25 meters, which is consistent with the average lane width of meters in urban settings, while the distribution is considerably more heavy-tailed in the case of DR(eye) indicating significant GPS noise.
C.3 GAZE FIXATION DETECTION
For our fixation detection, we adopted the methodology from Pupil Labs, the creators the glasses embedded with gaze detectors (Tonsen et al. 2020 Our criteria define gaze fixations instances
22

---

Published as a conference paper at ICLR 2025
Leafer Opedirelligg
(a) A bad GPS sample from scene where the car drives through tunnel.
Consistent offset while driving straight.
234
Loaftct Dean Seruttion
Left turn with slow-down.
(d) Lane change and stop.
Figure 10: Example trajectory fixes with TRAJAPP. The original GPS points are in gray, and the hand-placed markers are blue. The new markers are interpolated with a cubic spline with a color that represents the instantaneous speed: red for slow, and green for fast movement.
InsiApo
Ready
Save Data
Seek to start
Seek to end
Load from file
Walt until the GPS points are randered and the app atate is Ready.
Red markers
original. reisy GPS points. Red markers
regresent points before the cumeent video haam. The wowe of the markers represont absziute speed charge. more markers
1425
points.
Right-clas Gray marker go to video
Click on the map to add Blue marker for the current video frome.
Deshed
Ideally, these lines should be short!
adding enouggh merkeers, CSV file.
sove
Clcea Video
D Strocts Sataline Alt. Satel lEar'i
Figure 11: User interface for GPS refinement. We show a video and its corresponding GPS locations map concurrently. An annotator places correct markers on the map such that the GPS locations align with the video observations.
23

---

Published as a conference paper at ICLR 2025
Density
0.05
Dr(eye)ve
GEM
0.00
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Distance (m)
Figure 12: GPS distances to the nearest road center. Estimated via OpenStreetMap. DR(eye)VE's GPS points are further away from road centers, indicating significant noise.
where the gaze remains relatively stationary for a duration ranging from 80ms to second. The upper limit of this range, second, could also characterize smooth pursuit, where the eyes smoothly track a moving object. Furthermore, we account for dispersion, which represents the spatial movement permitted within single fixation. A gaze group is considered fixation the dispersion is within limit of 1.5 degrees, ensuring that minor involuntary or random gaze movements do not undermine the detection of genuine fixations.
A set of example fixations can be seen in Figure 14 Note that there are frames without any detected fixations, which is explained by saccades and blinks. For others, the accuracy is high: note how the gaze meets exactly with the brake light of the bus in front, in Scene 1, Frame 9. Similarly, the gaze over the motorcycle in front well-centered in Scene 5 for most frames.
C.4 GAZE ALIGNMENT AND ROBUSTNESS
Our framework processes driver gaze data following established practices in eye-tracking re- search (Hayhoe et al. 2003; Land 2006). To align the high-frequency gaze measurements (200 Hz) with video frames (5 Hz), use median downsampling. This approach ensures robust position estimates while preserving the temporal characteristics of driver attention patterns.
To evaluate our model's robustness potential measurement errors, conducted additional exper- iments on the GEM dataset. As shown in Table [2] artificially introducing gaze position noise of +:50 pixels (approximately 5% error) results in minimal degradation of prediction accuracy (0.01m increase ADE). This resilience can be attributed to our "fell-offviee approach, which treats gaze supplementary the driver's head-mounted video feed rather than primary input.
Configuration
ADE (m)
ADE+20PCI (m)
RouteFormer without noise RouteFormer with =:50px noise
5.99
7.70
6.00
7.72
RouteFormer-Bate Base (without FOV)
6.13
8.14
Table 12: Impact gaze measurement noise on prediction accuracy. The minimal change in ADE demonstrates our model's robustness to gaze uncertainty. Top and bottom rows from Table
These results demonstrate that our processing pipeline, built on standard gaze analysis techniques, effectively handles both natural gaze patterns and potential measurement uncertainties.
C.5 EXPLORING IMAGE HOMOGRAPHY AND STITCHING
In our dataset, provide synchronized videos from different viewpoints. In attempt provide fully registered view from multiple cameras to our model, we have explored approaches for stitching them. However, have discovered that using the separate videos brings benefits:
Using the current state of the art, not feasible to register the driver headcam and the scene videos within acceptable error margin. We experimented with classical feature extractors such as DISK (Tyszkiewicz et al.) (0020) and transformer based detector- -fee local feature matching (Sun al. 2021). using Kornia library (Riba al. 2019)
24

---

Published as a conference paper at ICLR 2025
We have observed that the redundancy from having multiple scene camera feeds can be exploited for training regularization. By dropping out one the views at each frame during training, improved the validation performance.
Figure 13: Successful and failed stitching examples. We used LoFTR for feature- scene videos and the head-cam. Example frames from the stitching experiments can be seen in Figure 13
feature-matching matching across
25

---

Published as a conference paper at ICLR 2025
Figure 14: Visual representation of gaze heatmaps across different scenes in the dataset. Each scene row is visualized with a sample of 10 frames with one one- second time difference, illustrating gaze patterns over time. Note that fixations are not always available due blinks and saccades.
Frame 10
(i) Frame 9
(h) Frame 8
g Frame 7
Frame
Frame 5
(d) Frame
Frame
Frame
Frame
Sceme
Scene 5
26

---

Published as a conference paper at ICLR 2025
300
Min. distance to an intersection (m)
250
Speed range [m/s]
200
150
100
50
20
40 60 80
100 120
140
PCI
Figure 15: Function of PCI with respect to the distance to an intersection. Speed range-the difference between the highest and lowest instantaneous speed throughout the trajectory-is is chosen instead speed as high yet constant speed without change direction (e.g. cruising on a highway) is low-PCI event. Bins with multiple samples are displayed, with a median speed range bin. PCI is highly correlated with distance to an intersection, but only if a vehicle changes its speed throughout the trajectory, which discards stoppings and straight driving cases.
Table 13: Breakdown of our GEM dataset. We report duration, weather condition, and PCI for each participant.
Subject
Total Distance (m)
Rainy
PCI (mean)
PCI (max)
Split
001
13,112
V
12.37
92.99
train
002
12,824
/
10.76
81.22
val.
003
12,997
X
20.09
94.87
train
004
13,064
X
14.59
79.30
val.
005
12,872
X
17.29
86.39
train
006
13,127
X
13.82
88.00
train
007
12,973
X
16.42
77.47
train
008
13,700
V
17.02
89.48
test
009
12,958
X
15.30
103.96
test
010
13,048
X
14.63
81.42
train
D PCI METRIC
D. BREAKDOWN OF GEM DATASET AND PCI INSIGHTS
For the information on the dataset split and the behavior of each participant, see Table 13
deeper dive into our dataset, combined with synchronization with OpenStreetMap (Open-) StreetMap contributors 2017), reveals that the high PCI events indeed happen either close to the intersections, or on trajectories with a high-speed change, as observed Figure
D.2 VISUALIZING PATII COMPLEXITY INDEX
Figure 16 illustrates the key components of PCI computation. Given an input trajectory (gray), extrapolate a simple trajectory (dashed) by maintaining the final velocity vector. PCI measures the Fréchet distance (red) in meters between this simple trajectory and the actual target path (blue), effectively quantifying how much the actual path deviates from the simple projection. This deviation captures the complexity the maneuver higher deviations indicate more complex trajectories, requiring additional context to predict accurately. Unlike alternative metric like mean-squared
27

---

Published as a conference paper at ICLR 2025
Input Trajectory
Simple Trajectory Target Trajectory
Figure 16: A simple representation of PCI as a concept.
error, Fréchet distance is more stable against speed changes, better capturing differences in trajectory rather than point-by-poing error.
D.3 PCI ANALYSIS ON URBAN DRIVING
Figure 17 visualizes PCI values across representative urban drive in GEM. High PCI regions (bright yellow) consistently align with challenging scenarios: roundabout exits (top-right, PCI > 80), lane merges after intersections (bottom-left. PCI 60), and sharp turns exceeding 45 degrees (top-left and bottom-right). Notably, slower 90-degree turns exhibit lower PCI values due to re- duced divergence from simple trajectories, demonstrating PCT's ability to distinguish the degree of challenge to predict each target curve.
28

---

Published as a conference paper at ICLR 2025
(C) OpenStreetHäp contributors
Figure 17: Average PCI values of a full trajectory video in GEM. Estimated by running sliding window of I second across the whole trajectory with 6s-8s input-target length, and averaging the PCI values assigned to each point. Brightest yellow parts map to ~ 80 PCI while the dark blue is at 0 PCI.
29


Reviews:
[{"review_id": "nAGQWWhBLx", "reviewer": "Reviewer_U1qP", "summary": "The paper introduces a multimodal ego-trajectory prediction model that utilizes driver field-of-view (FOV) data—including first-person video and gaze fixations—along with environmental and GPS information to enhance the accuracy of vehicle trajectory forecasting. The method surpasses conventional models through the implementation of a Path Complexity Index (PCI), which assesses the complexity of diverse driving scenarios. The authors present a novel dataset that provides comprehensive urban driving data enriched with driver FOV information.", "questions": "Please refer to the weaknesses above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- New dataset: the authors contribute GEM, a high-quality dataset focused on urban driving, which includes synchronized FOV and gaze data, filling a critical gap in available resources for trajectory prediction. It will be good to have this dataset for the autonomous driving community. \n- The proposed RouteFormer introduces a unique approach by integrating driver field-of-view (FOV) data, such as first-person video and gaze fixations, with GPS and environmental data, enhancing the accuracy of ego-trajectory prediction.\n- Introduction of Path Complexity Index (PCI). The paper proposes a novel PCI metric to quantify the complexity of driving scenarios, offering a more nuanced assessment of prediction challenges compared to traditional metrics.\n- The proposed method outperforms the other baselines on both datasets.", "weaknesses": "- To fully understand how much each modality contribute to the improvement, it will be interesting to add the ablations studies below and compare the ADE and ADE+20PCI.\n   1. RouteFormer without video and gaze (Motion only).\n   2. RouteFormer without surrounding scenes (Motion + Gaze).\n\n- Stronger baselines are needed. The current baselines (GIMO, Multimodal Transformer) used in this paper are mainly designed for human motion. Compared with at least one vehicle motion predictor is more convincing (e.g., MTR [1], Autobots [2], ...). Or are there any potential reasons that the authors thought the human motion baselines were more appropriate than vehicle baselines?\n\n[1] Shi, Shaoshuai, et al. \"Motion transformer with global intention localization and local movement refinement.\" Advances in Neural Information Processing Systems 35 (2022): 6531-6543.\n[2] Girgis, Roger, et al. \"Latent variable sequential set transformers for joint multi-agent motion prediction.\" International Conference on Learning Representations (ICLR), 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IAJ6vEBFIC", "reviewer": "Reviewer_15rW", "summary": "The paper introduces RouteFormer, a multimodal ego-motion prediction network that integrates driver field-of-view (FOV) data with scene and GPS information to improve trajectory prediction. RouteFormer is the best solution on the market for this problem. Its Path Complexity Index (PCI) metric assesses trajectory difficulty and enhances model evaluation in complex scenarios. Additionally, the paper presents GEM, a new dataset containing synchronised driver FOV, gaze, and GPS data, capturing diverse urban driving conditions. RouteFormer significantly improves prediction accuracy, especially in high-complexity scenarios, advancing safety in driver-assistance systems.", "questions": "1. I have a couple of concerns regarding the definition of corner-case scenarios. The authors propose a model that essentially employs a contrastive learning approach—predicting a trajectory and then categorizing scenarios by comparing it with the ground truth. I believe this method introduces bias, as predictions differ across models. What evidence suggests that the prediction model used here is unbiased? While many recent works adopt similar methods to address long-tail issues, in this particular field, I find the approach still somewhat subjective.\n\n2. The authors have not clearly defined what constitutes the long-tail phenomenon. It’s unclear why this method is capable of defining rare scenarios. I believe that this method, as described, does not adequately capture or define the essence of long-tail events. Could the authors provide further clarification?\n\n3. The manuscript is inspired by human visual attention during driving, so it should cite earlier works that introduced similar ideas in trajectory prediction. These are:\n(1) Human Observation-Inspired Trajectory Prediction for Autonomous Driving in Mixed-Autonomy Traffic Environments, IEEE International Conference on Robotics and Automation (ICRA 2024)\n(2) Less is More: Efficient Brain-Inspired Learning for Autonomous Driving Trajectory Prediction, European Conference on Artificial Intelligence (ECAI 2024)\n\n4. The related work section could benefit from a broader discussion of recent research. Currently, I find the discussion of the latest studies to be rather limited.\n\n5. The paper would benefit from a discussion section that addresses its limitations and future challenges. This would provide a more comprehensive perspective.\n\n6. Could the authors consider open-sourcing the project code and dataset? This would significantly contribute to the community.\n\nOverall, I believe this paper meets the publication standards for ICLR.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper has done an impressively solid and hardcore job—it's a pleasure to read, providing a refreshing clarity that’s truly satisfying. It offers genuinely unique insights and, in my opinion, fully meets the standards for conference publication.\n\n1. RouteFormer effectively combines driver FOV, scene, and GPS data, resulting in enhanced accuracy for predicting complex, non-linear trajectories. This multimodal integration marks a notable advancement over existing models.\n\n2. The introduction of the Path Complexity Index (PCI) provides a novel way to quantify trajectory difficulty, allowing for better evaluation of model performance in challenging scenarios, which is often overlooked in traditional metrics.\nThe GEM dataset fills a significant gap by providing synchronized driver gaze, field-of-view, and GPS data, particularly suited for urban driving conditions with various traffic elements. This dataset enables more nuanced model training and testing, contributing to the broader research community.\n\n3. Extensive testing demonstrates that RouteFormer performs robustly in real-world scenarios, highlighting its potential for practical applications in driver-assistance and AD systems.", "weaknesses": "Please refer to the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4O5Qz0JatZ", "reviewer": "Reviewer_H88J", "summary": "In this paper, the authors propose an end-to-end multimodal ego-motion prediction network that utilizes driver’s field-of-view data. Specifically, the proposed model predicts egocentric trajectory by taking scene videos, driver field-of-view videos, driver gaze position, and past trajectory as its input. Besides, the authors also propose a metric to measure the trajectory complexity, and an ego-motion dataset with driver positions and perspective. The experimental results show some improvements in the task of trajectory prediction task.", "questions": "1. What if the estimated gaze positions move quickly (either caused by wrong prediction or the driver moves his/her gazes quickly)? Is there any negative impact on the final results?\n2. Line 190, the gaze positions are in the shape of Tx2. Does it mean we only use one gaze for each time stamp?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed solution sounds solid in theory. Specially, considering driver’s attention and intent as well as surrounding environment is convincing. The driver’s attention and intent is a key impact factor of future trajectory.  \n2. The proposed dataset is helpful for other researchers to do more related research on that. \n3. The proposed metric of trajectory complexity is helpful. It is a good indicator while analyzing the performance of a self-driving algorithm.\n4. The ablation study and supplementary material are helpful. Readers can get more information from the ablation study results.", "weaknesses": "1. It will be better if the authors could report some failure cases. Especially for those cases which the predictions are totally opposite of the ground truth. \n2. It will be better if the authors could compare their solution with latest schemes published in Year 2023 or 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)