DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images
Authors: Ozgur Kara
Harris Nisar
James Matthew Rehg
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion
Eye Tracking
Scanpath Prediction
Saliency Prediction
Primary area: deep_learning
TL;DR: We propose DiffEye, a diffusion-based training framework designed to model continuous and diverse eye movement trajectories during free viewing of natural images.

Abstract: Numerous models have been developed for scanpath and saliency prediction, which are typically trained on scanpaths, which model eye movement as a sequence of discrete fixation points connected by saccades, while the rich information contained in the raw trajectories is often discarded. Moreover, most existing approaches fail to capture the variability observed among human subjects viewing the same image. They generally predict a single scanpath of fixed, pre-defined length, which conflicts with the inherent diversity and stochastic nature of real-world visual attention. To address these challenges, we propose DiffEye, a diffusion-based training framework designed to model continuous and diverse eye movement trajectories during free viewing of natural images. Our method builds on a diffusion model conditioned on visual stimuli and introduces a novel component, namely Corresponding Positional Embedding (CPE), which aligns spatial gaze information with the patch-based semantic features of the visual input. By leveraging raw eye-tracking trajectories rather than relying on scanpaths, DiffEye captures the inherent variability in human gaze behavior and generates high-quality, realistic eye movement patterns, despite being trained on a comparatively small dataset. The generated trajectories can also be converted into scanpaths and saliency maps, resulting in outputs that more accurately reflect the distribution of human visual attention.
DiffEye is the first method to tackle this task on natural images using a diffusion model while fully leveraging the richness of raw eye-tracking data. Our extensive evaluation shows that DiffEye not only achieves state-of-the-art performance in scanpath generation but also enables, for the first time, the generation of continuous eye movement trajectories. Project webpage: https://diff-eye.github.io/

DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images
Ozgur Kara" Harris Nisar* James M. Rehg {ozgurk2, nisar2, jrehg) wilginii.lii.......mm edu University of Illinois Urbana- Chammpaan
Abstract
Numerous models have been developed for scanpath and saliency prediction, which are typically trained on scanpaths, which model eye movement as a sequence of dis- crete fixation points connected by saccades, while the rich information contained in the raw trajectories is often discarded. Moreover, most existing approaches fail capture the variability observed among human subjects viewing the same image. They generally predict a single scanpath of fixed, pre-defined length, which conflicts with the inherent diversity and stochastic nature of real-world visual attention. To address these challenges, we propose DiffEye, a diffusion-based training framework designed model continuous and diverse eye movement tra- jectories during free viewing of natural images. Our method builds on a diffusion model conditioned on visual stimuli and introduces novel component, namely Corresponding Positional Embedding (CPE), which aligns spatial gaze information with the patch-based semantic features of the visual input. By leveraging raw eye-tracking trajectories rather than relying on scanpaths, DiffEye captures the inherent variability in human gaze behavior and generates high-quality, realistic eye movement patterns, despite being trained on a comparatively small dataset. The generated trajectories can also be converted into scanpaths and saliency maps, resulting in outputs that more accurately reflect the distribution of human visual attention. DiffEye the first method tackle this task on natural images using a diffusion model while fully leveraging the richness of raw eye-tracking data. Our extensive evaluation shows that DiffEye not only achieves state-of-the-ar perfor- mance in scanpath generation but also enables, for the first time, the generation of continuous eye movement trajectories. Project webpage: https://diff-eye github.io/
Introduction
Humans obtain visual information via sequence of eye movements that direct the fovea analyze important elements of the visual scene. The resulting pattern of fixations and saccades' is dependent upon the task hand (e.g., free exploration, visual search, etc.) and determines the visual elements that receive our attention [1, 2]. The gold standard for quantifying visual attention is eye-tracking, which records subject's eye movements as they view an image (stimulus) on monitor. Eye tracking produces a trajectory of eye movements, consisting of a sequence of (x,, 3) pairs, which is processed to identify the fixations and saccades. Eye tracking provides valuable insights into the properties of the visual scene that drive human attention, and is widely-used for applications in virtual reality [3, 4. 5], foveated rendering [6. 7J, and advertising [8. 9]. Eye tracking also widely used in psychology to identify differences in how visual information is processed by individuals [10]. For example, eye tracking has been used study social cognition in conditions such as autism by identifying differences in how social cues processed [11, 12] (c.g., individuals with autism have been shown
"Equal contribution
Fixatiins occur when the eye remains relatively still and attention is focused on specific location. Saccades rapid eye movements between fixations that enable quick shifts of focus between regions of interest.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Stimulus
(b) Saliency Map
(c) Scanpaths
Trajectories
Figure 1: Comparison of different eye-tracking data types. (a) Original visual stimulus. (b) Saliency maps highlight regions of interest but do not capture the temporal dynamics of human attention. (c) Scanpaths offer a compressed representation of eye movement trajectories. (d) Full eye movement trajectories, recorded via eye trackers, provide detailed insights into attention dynamics. This example is from the MIT1003 dataset [18]; each color represents a different subject, emphasizing the importance of modeling inter-subject variability.
to exhibit atypical patterns of attention to faces [13, 14)). Although eye-tracking studies provide high-quality data, they are costly and time-consuming to conduct, and require significant expertise for proper experimental design [10]. As result, there has been growing interest in developing computational models that can predict visual attention to images.
Modeling human visual attention has traditionally focused on saliency prediction [15, 16]. In this task, given an image, the goal is to produce a heat map, known as saliency map, where high values correspond to regions that are more likely to receive fixations. However, these spatial summaries neglect the temporal structure of eye movements, which is critical for applications in virtual reality and psychology research. For example, recent work in developmental science has shown that children with and without autism demonstrate different spatio-temporal gaze behaviors [17]. To capture the temporal dimension, researchers have investigated the scanpath prediction task, which involves predicting a sequence of fixations for a given image. Scan path prediction has historically involved generating a single deterministic output, which is insufficient to capture the inherent variability of human attention. Figure illustrates the variation in both scanpaths (c) and eye movement trajectories (d) for population of viewers, with each color representing different subject. As a result, recent work has shifted towards generative modeling to better reflect the stochastic nature of eye movements. However, existing approaches to the generative modeling of visual attention suffer from two major limitations: First, most prior methods operate directly on scanpath data rather than on eye movement trajectories, resulting in the loss of valuable information for spatio-temporal modeling. For example, in the MIT1003 dataset [18), the average scanpath consists of 8.4 + 2.6 timesteps, while the raw trajectories average 723.7 + 13.4 timesteps, indicating substantial reduction in spatio-temporal information. Second, many existing methods model scanpath variability either autoregressively by sampling intermediate outputs, such saliency maps derived from fixation history [19, 20, 21 or perform deterministic scanpath prediction [22], both of which fall short in accurately capturing the true distribution of gaze behavior.
In this paper, we hypothesize that generative modeling using the full eye movement trajectories produced by eye tracking can yield more effective representations of visual attention dynamics, in comparison to prior works that operate on scanpath data. In particular, we demonstrate that training on eye movement trajectories results in increased accuracy in scanpath prediction. In addition, we argue that generative models are better suited than discriminative models for learning from this rich temporal information, particularly given the variability present in both visual tasks and data collection processes. To evaluate these hypotheses, we introduce DiffEye, a diffusion-based framework that conditioned on given visual stimulus and trained directly on raw eye movement trajectories to generate realistic sye-tracking samples. These generated trajectories can then be transformed into scanpaths or saliency maps. To enhance stimulus conditioning and improve the interaction between trajectories and the image via cross- attesnatten,oon we propose novel positional embedding strategy called Corresponding Positional Embedding (CPE). This method strengthens conditioning by aligning gaze information with the semantic features of the stimulus. In addition, we use high resolution image features to improve semantic precision, leading to more accurate trajectory generation.
In summary, our key contributions are as follows:
We present the first diffusion-basee training framework that directly utilizes the raw eye movement trajectories obtained from standard eye tracking datasets. Our method outper-
2

---

forms existing approaches in scanpath generation on unseen datasets, verifying its generaliz- ability, and enables the synthesis of continuous eye-tracking data for natural images.
We introduce novel positional embedding strategy, Corresponding Positional Embedding (CPE), which improves conditioning on visual stimuli by aligning spatial locations in both the image and trajectory space. Notably, our model achieves strong results even when trained on relatively small datasets.
Unlike recent models that rely on autoregressive sampling to capture scanpath variability produce single deterministic prediction, our approach leverages diffusion model the inherent variability in human gaze behavior across subjects. As a result, can generate eye movement trajectories that can be converted into scanpaths of variable lengths, producing diverse and plausible patterns that reflect the natural variability in human attention.
2 Related Works
Eye Movement Trajectory Generation The closest related work to this paper is DiffGaze [23], which adopts a diffusion-based approach to generating eye movement trajectories in 360 images. While the focus on 360 images is innovative, the resulting models cannot be applied the analysis of the standard RGB images used in eye tracking research. Moreover, the conditioning approach in DiffGaze relies on single global feature vector obtained via spherical convolution. In contrast, our approach enhances conditioning through the use of CPE and patch level image features. Our ablation studies (see Table 2) demonstrate the importance of these elements for achieving strong performance. Unfortunately, direct comparison to DiffGaze is not possible, since the code and pretrained models are not publicly available. We hope that the release of our code and models, along with training and testing splits for eye movement trajectory data from standard datasets, will spur additional research on this understudied problem.
Scanpath Modeling Works on scanpath generation operate directly on the sequences of fixations that are extracted by postprocessing eye-tracking data [24]. Given an input image, these models predict the location and sequence of fixations comprising the generated scanpath. Previous works have explored variety of methods and problem formulations. FOR-ROT [25] uses LSTMs predict scanpaths but outputs scanpaths of fixed length which limits its generalizability. DeepGaze III [19] samples from saliency maps using the fixation history, and result it cannot utilize the global image representation in predicting each fixation. Gazeformer [22] introduces ZeroGaze for unseen targets, but their deterministic method cannot produce a distribution of trajectories for given stimulus. Some works on scanpath generation have addressed the fact that gaze behavior is dependent upon the task. Similar us, PathGAN [26] uses GANs [27] for generative modeling in the FV case, though training is unstable. At present, the only available datasets containing eye movement trajectories address the standard free viewing (FV) task. Therefore, an investigation of the performance of our DiffEye approach on non-FV tasks will depend on future data collection efforts. In contrast, Chen et al. [21] use reinforcement learning for target present (TP) and visual QA tasks. Other methods [28, 29] support TP and the related target absent (TA) condition, but not FV. HAT [20] is unique in covering the three tasks FV, TP, and TA with a single model via task query, but follows a similar sampling strategy as DeepGaze III to generate scanpaths. In summary, no prior scanpath generation work has taken advantage of the benefits of diffusion models, and competing methods for the FV task have a variety of limitations. Our experiments (see Table 3) compare scanpaths extracted from fiffEye to prior scanpath generation methods and demonstrate superior performance across a variety of metrics. We also note that our approach of directly generating eye movement trajectories gives end users the flexibility to define and extract fixations using any desired approach.
There have been a number of prior scanpath generation works that target 360 images. Similar to the discussion of DiffGaze above, these methods are not directly comparable to ours and are included here for the sake of completeness. SaltiNet [30] samples from saliency volumes; Pathformer3D [31] uses transformers for autoregressive prediction. ScanGAN360 [32] and Scantd [33] apply GANs and diffusion to the 360 modeling task. Finally. we note that most prior works including ours have
used aggregate eye tracking data across multiple users to model attention at the population level.
In contrast, a recent work from Chen et. al. [34] develops an approach to scanpath prediction for single user. The specialization of DiffEye to the 360 image and individual prediction tasks remain as interesting subjects for future research.
3

---

Saliency Modeling Prior works on saliency prediction have benefited from a deep connection to research in visual perception [35, 36, 37, 1, 38]. Itti's ' seminal work [39] later drew interest from the computer vision community [40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50]. As saliency methods capture only spatial attention and ignore the temporal dynamies central to eye movements, they are not able to address the needs of the diverse applications that motivate this work. Saliency maps can be derived post hoc from the output of our DiffEye method via fixation extraction [51] and spatial aggregation. In this regard, DiffEye can be viewed as alternative approach to saliency prediction based on the detailed modeling of gaze behavior. However, methods that are designed to directly optimize saliency prediction performance
capable of higher accuracy than an indirect approach based on fixations. Further, the saliency baselines we compared against [47, 48, 50] were trained on combinations of various saliency datasets [18, 43, 52, 53, 54, 55], including MITI003, while our method is only trained on the relatively smaller MIT1003 dataset.
3 Methods
3.1 Preliminaries
Given dataset D {(T', ( where each pair consists of visual stimulus represented by an RGB image I E #x x3 and a set of fixed length sequences R2 E LR2222 of continuous eye movement trajectories collected from K* different subjects for the 7th stimulus, our goal is to two- model the joint distribution of stimuli and trajectories. Each time step in Rý corresponds dimensional coordinate (x, y). representing the spatial location of gaze at that timestep. For simplicity, we denote the set of stimuli as S and the set of eye movement trajectories as R, and use R to refer to a single eye movement trajectory and I for a single visual stimulus. We model the distribution over R conditioned on S using a diffusion model. This framework enables learning the underlying generative process of eye movement behavior in response to complex visual stimuli, capturing both temporal dynamics and spatial variability. To achieve this, Di DiffEye adopts the denoising diffusion probabilistic (DDPM) model [56] to learn a distribution po (R. | S) that approximates the true conditional distribution q(R S) of eye-tracking data. DDPM consists of a forward process that progressively adds Gaussian noise to the trajectories over Tain diffusion steps, and a reverse process that gradually denoises samples to recover realistic trajectories. The diffusion model is trained by minimizing the denoising objective ming R 12 [|le c R(tarr) lain, T)]] where (O( I) is the ground-truth noise and eø (( Lairr, 1) is the model's 's prediction of noise given the noised trajectory R(tas), diffusion timestep lain, and conditioning image T.
3.2 Our Approach
Base Model We use a U-Ner-based architecture [57] as the noise prediction model in our diffusion pipeline. The model consists of downsampling, mid, and upsampling blocks, where 1D convolution layers temporally downsample or upsample the trajectories while increasing or decreasing their number of features. Self-attention layers are applied after the convolutions capture temporal dependencies in the eye-tracking data at multiple resolutions throughout the network. In addition to the original U-Net design, we incorporate the current diffusion timestep fair by passing through a sinusoidal positional embedding module [58], followed by fully connected layer, a SiLU activation [59], and another fully connected layer. The resulting timestep embedding is added to the output of the first convolution layer in each block, allowing the network to condition on tai and enabling effective diffusion training.
Stimuli Conditioning An essential component of our pipeline is conditioning the model on a given stimulus in the form of an image. To achieve this, first, the input trajectory is passed through peLx 1D convolution layer, projecting it from R E Lx2 to Reroj E LLD where I. is the number of trajectory timesteps and D the embedding dimension.
Initially, we attempt to condition the model using the global feature vectors produced by the DI- NOv2 [60] Vision Transformer foundation model, chosen for its strong and well-established semantic representation capabilities. This global vector, originally of size IR [x Paaa, is projected via a fully connected layer to R xD and then concatenated with the projected trajectory tokens to form input sequence Egiobal E IR (4+1))xD, However, this approach alone leads to poor generation quality, as

---

(a) End tond-to-Be to-End Training
Editf
U-Net
R
Forward Process
Trajectory Pasitional
R
Embedding
FeatUp & Interpolation
Parch
KV
R0
K00
Posisional
Embedding
FerR
parch
Rel Fam.F11 FII
(b) Correspanding Positional Emhedding (CPE)
(c)
Inference
Solmalus
Trajectory Poshional Embedding
Parch Positional Eachedding
Eye Minementt
P
2
Linaa
Fiee
Forrd
Mppz
Linzar
Searplatio
Scapaaho
Salancy
Map
Figure 2: An illustration of DiffEye. (a) End-to-end Training. Given an initial trajectory R(0) and image I, noise is added to produce R FeatUp extracts patch features Fouthh and both inputs are passed to the (b) CPE module, which aligns trajectory and patch positions. The resulting representations CC and FCPF are processed by U-Net with cross-attention at each block and optimized via diffusion loss. (e) Inference. Starting from noise, the model denoises for T steps to generate an eye movement trajectory, which can be used to produce scanpaths or saliency maps.
Table 1: Training data comparison across methods. DiffEye is trained solely on the MITI003 dataset using full eye trajectory data, while other methods are typically trained on scanpaths and rely on larger or combined datasets.
Method
Eye Trajectories
Scampaths
Images
IOR-RO1 ROI
OSIE [S3]
8,400
560
DeepCazell
1IT [18) and SALICON (43]
615,0415
11,003
Chen et al. GazeFommer
AiR [62]
$9,080
1,454
COCO- Search18 COCO-Seurch88 [63]
62,000
6,202
COCO-Search FOeO-FFeeView (64].
[18), [18)
7,905
DirrEy
8,934
1,003
the global feature lacks localized spatial semantics necessary for effective conditioning. To better incorporate localized spatial information, we utilize patch-level features from DINOv2. Let the patch features be denoted as Fpuch E H x Das, where (H', W') are the spatial dimensions of the patch grid. These features are projected via linear transformation to Fproj E H x concatenating these directly with the trajectory, we adopt a cross-attention mechanism that enables interaction between the projected image patch tokens and the trajectory tokens Roroj E Lx Cross- attention is performed between the L trajectory tokens and the N, H' W image patch tokens (flattened from F. ). Initial attempts with a single cros-aatenntion layer before passing the trajectory the U-Net proved insufficient for effective conditioning. To address this, we add cross-attention layers at the end of each U-Net block, which is empirically proven to enhance the performance. Additionally, to improve spatial precision, we replace DINOv2 patch embeddings with those from FeatUp, model-agnostic framework that restores fine-grained spatial details in deep features [61] J. FeatUp outputs 224 x 224 x Dn feature map, which we interpolate to H W Da, where Hí = W/ = 32 effectively doubling the spatial resolution relative to standard DINOv2 features.
xW' xD Rather than
Corresponding Positional Embedding (CPE) To further improve spatial alignment, introduce novel positional embedding method, Corresponding Positional Embedding (CPE). Let the faur-tepp noisified input trajectory Ri be denoted R E R lxx where each represents a 2D coordinate (I,, y:). We first project the trajectory into the hidden dimension, resulting in the embedded trajectory Rpnij E R.... Next, construct 2D positional embedding grid P R wxw>X based on the original image resolution (H, W), using sinusoidal encodings [58]. For each timestep i E {1, L.).
5

---

Stimiuu
Groun Trith
JOR-
Chen etall
GaedPormer
DoepGaoe III
HAT
Diilgge
Figure 3: Qualitative comparison of scanpath generation. Scanpaths generated by DiffEye and baseline models are shown alongside ground truth annotations across four different scenes. Each row represents unique stimulus, and each column shows the corresponding scanpaths generated from specific method.
we retrieve the corresponding positional embedding vector from the grid as pı = Plys, 23 E DR where R. (ra; y4) is the i-th coordinate in the original trajectory R. The final CPE-augmented trajectory token is then computed as:
RC
Rpmuj Pr-
The same positional grid P is also applied to the image features. Let the projected image features be F Froj E BR xW" We interpolate the positional grid P to the patch resolution, yielding p' E HR
We then add this the image features:
FCPE
Pmnj P'.
By sharing and aligning positional information across both trajectories and image patches, CPE enables effective spatial correspondence and interaction between gaze behavior and visual content. Figure 2 summarizes our approach, including end-to-end training (a), the CPE module (b), and inference (c).
Data Preprocessing We use the MITI003 dataset [18] 2 to train and evaluate model. To the best of our knowledge, it is the only publicly available dataset that provides raw eye-tracking data for natural images obtained during a free-viewing task. The dataset contains recordings from 15 subjects who free-viewed 1,003 images for seconds each, resulting in a total of 15,045 eye-tracking sequences. Data was collected using the ETL 400 ISCAN system at sampling rate of 240 Hz. During preprocessing, we removed all blinks and NaN values, and retained only the sequences with at least 720 timesteps. After filtering, 8,934 trajectories remained. All remaining trajectories were truncated or downsampled to a uniform length of 720 samples, as our model is designed to generate fixed-length eye movement trajectories and the U-Net architecture requires several downsampling steps which is possible with a sequence length of 720. Additionally, all stimuli were resized to 224 224 pixels. The dataset is split into training (90%) and testing (10%) sets based on stimuli, ensuring that images used for evaluation were not seen during training.
4 Experimentation
Implementation Details We train our model using the Adam optimizer [65] for 3000 epochs with fixed learning rate of 10-4, During training, we utilize the DDPM scheduler with linear noise schedule ranging from 10-4 to 2 x 10-2, For sampling, we adopt Denoising Diffusion Implicit Models (DDIM) [66]. which enable high-quality sample generation in significantly fewer steps without compromising performance. We set the number diffusion steps to Tain 1000 during training and reduce it to 50 during sampling for improved efficiency. Throughout both training and inference, we apply classifier-fre free guidance (CFG) to effectively condition the model on the
hnttps://peppee csail mit. hatp://jjjd//hheroooopoeooooo//t
csail,

---

Table 2: Ablation study for scanpath and eye movement trajectory generation. We evaluate the con- tribution of key components in Diff by removing modules such as FeatUp, CPE (Corresponding Positional Embedding). U-Net Pross-attention and patch-level features. Results are reported for both scanpath and continuous trajectory generation using four trajectory- based evaluation metries.
Tiok
Centgurstios
Draonce (x11P)
Lownebsen
Tikcoee Tricker Dieence 1- (II))
Dynomi: Time Warping C1171)
Irehwidiing
Thme Dalay
14ean
Btal
MBea
F1er"
N0er
MBros
Beat
Dicerge
0.139
11.87
2,449
0.157
0.117
80.661
53.894
Seangath
Generaation
Wee U1 Net Cooss Allosiin
CT
13163
11151
13
0.307
3.703
2.557
0.159
0.190
02125
107.962
100/792
69:523
61.353
Pacc Level Fosturcs
0.259
0.147
DiddZye
10:853
0,200
3,001
2
11.334
3.212
38.225
28.968
Ese Movermnnt Iragectrry
Wes FoalUp
101255
8.
3.544
2,02
12.513
3.545
41.224
25,43
11 Nat Criss Arsenioon
10:571
114
437
14 14.214
10 180000
50.71
16-364
Wis Plachl and Franess
11.791
1.088
7.761
13.107
13.12
77.00
47:354
Stimulas
Ground Truth
Didllyye
Stimulus
Ground Truth
Diflly
3
wio U-Ner Cross-Attn
wio Pasch-Level
Stimulos
Ground Truth
DEDEE
FeaUUp
Features
Figure 4: Qualitative analysis and ablation study of continuous eye movement trajectory generation. (a) Multiple eye movement trajectories generated by DiffEye) alongside ground truth annotations across four different scenes. (b) Ablation study showing the impact of removing individual architec- tural components (FeatUp, CPE, cross-attention, and patch-level features) on continuous trajectory generation.
visual stimulus [67]. This encourages the model to learn both conditional and unconditional denoising behavior, supporting effective guidance at inference time. See Supplementary for the complete details. All experiments were conducted using an NVIDIA RTX A6000 GPU.
Baselines and Datasets We compare our method against several baseline models trained for the scanpath prediction/generation tasks. The baselines include the Human Attention Transformer (HAT) [20] and GazeFormer [22]. both of which are transformer-based architectures, DeepGaze III [19], a convolutional network, Chen et al. [21], which is a reinforcement learning model that mainly focuses on visual question answering, and IOR-ROI [25]. Note that GazeFormer can not generate a distribution but can only predict scanpaths. DeepGaze III and IOR-ROI requires the number of fixations per scanpath as input, which we set 10 for all experiments. Each bascline is used to generate scanpaths for all test stimuli. Scanpath baselines were trained on scanpaths coming from various combinations of datasets shown in Table 1. In contrast, our method is trained solely on a subset of the raw eye movement trajectories from the MIT1003 dataset which is relatively very small dataset with fewer pairs of stimuli and eye movement trajectories, yet achieves competitive performance, demonstrating robustness under limited data. We evaluate our results on the test sets of the MIT1003 and OSIE [53] 3 datasets. It is important note that the MIT1003 dataset does not provide official train-test splits; therefore, we created a random split of 100 images with eye-tracking data from 15 subjects each as described in Section 3.2. As a result, methods such as HAT and DeepGazellI. whose training data included MITI003 have already been exposed to the evaluation data. The OSIE test set contains 70 images with eye-tracking data from 15 subjects.
Evaluation Metrics For scanpath generation, follow the evaluation protocol proposed in [33, 23). For each image in the test set, we generate 15 eye-tracking trajectories per model, extract fixation points using the conversion method described in [18], and construct the corresponding scanpaths. For models supporting multiple visual tasks, we ensured that scanpaths were generated specifically for the Free- Viewing task. We evaluate the results using four standard metrics commonly used in
htttps://gg www.tt/b...m/teeeee/////aatteete http://ttttteeeeee//////////
7

---

Table 3: Quantitative comparison of scanpath generation on the MITI003 and OSIE datasets. We report the performance of each method using four commonly used trajectory-batt based metrics: Levenshtein Distance, Discrete Fréchet Distance, Dynamic Time Warping (DTW). and Time Delay Embedding (TDE). The label (seen) indicates that the model's 's training set includes the test set images. Bold values indicate the best scores, while underlined values denote the second- -best scores.
Test Dataset Method
Levenehtin Distance
Distance (x10P)
Divertte Friciict
Dyuaii Time Warping 1 (xx10")
Embediing
Time Delay
Masxx
Mess
Bexl
Meon
Best
IOR-ROI
11:092
2.460
1.834
1.317
108:
DasyGaaze III (sean)
11. 11.856
3,553
2.160
1.757
1.141
96.456
05.408
MIT
GuaeFommer
Chess ot
14887
12.943
2,6122
3,553
1.851
1.545
02.10K0
92.751
74.212
HAT (sars)
18.440
14,645
2060
2.680
1.862
13,009
2.449
1,573
1.067
88.661
ROI
12:152
1,099
1.167
70,624
DeepCaze
3,077
1,705
1.166
57.756
OSIE
Chen et GuaeFommer
15.
15.320
1,772
1.667
1.276
51.509
HAT DiffBye
15
3.712
2.598
2.501
1.757
111.413
53.
14.771
12.077
3,068
2.238
1.552
1,089
81.925
54.347
Levenshtein Distance, which measures the sequence similarity based on insertion, deletion, and substitution operations; Discrete Fréchet Distance (DFD), which captures the similarity between curves while considering their sequential nature; Dynamic Time Warping (DTW), which aligns sequences that may vary in speed length; and Time Delay Embedding (TDE), which assesses the temporal structure of trajectories [68]. Each test image is associated with 15 ground truth scanpaths. We generate 15 scanpaths per model; for deterministic models such as Gazeformer, only a single prediction is evaluated. For each metric, we report two scores: best and mean. The best score obtained by computing the evaluation metric between each ground truth scanpath and all generated scanpaths, selecting the most similar (or least distant) one, and then averaging over all scanpaths and images. The mean score is obtained by averaging the metric across all pairwise comparisons between ground truth and generated scanpaths for each image, followed by averaging across all test images (see Algorithm in Supplementary.).
the eye-tracking literature:
Scanpath Generation Results As shown in Table 3, the mean scores highlight our model's superior ability to generate trajectories that align closely with the overall distribution of human fixations. This is demonstrated by the lowest Levenshtein and TDE values, which capture spatial and temporal deviations across full scanpaths. Furthermore, the DTW and DFD metric, sensitive to the shape and curvature of the trajectories, indicates that our model better preserves the sequential flow of eye movements. Figure 3 visually reinforces these findings. In the second row (the image of the climbing human), the model by Chen et al. overly concentrates predictions around the low-resolution human head, while Di Dif distributes gaze more naturally across salient regions, including the background, showing stronger alignment with the ground truth. In contrast, DeepGaze III and HAT produce diffuse trajectories, suggesting a failure learn coherent attention patterns. IOR-ROI performs slightly better, but still deviates from the true distribution. Gazeformer, meanwhile, does not generate a distribution, but performs prediction, resulting in higher Levenshtein and TDE mean scores, indicating weaker alignment with human gaze distributions. Note that our model is trained on the MIT dataset and evaluated on both the MITI003 test set and the entirely unseen OSIE dataset, demonstrating strong generalization. Additional qualitative results are included in Supplementary.
Ablation Study We conduct an ablation study to assess the importance of key architectural com- ponents in DiffEye. Specifically, we compare the full model, comprising FeatUp high-resolution patch features, the proposed CPE, and cross-attention at all U-Net layers, against variants where one of these components was removed. As shown in Table 2, ablating any of these components degrades performance across both scanpath and continuous eye movement trajectory tasks. Removing FeatUp and replacing it with DINOv2 features leads to a consistent drop in performance. This confirms the benefit of FeatUp. which provides higher resolution and spatially structured visual features essential for precise modeling. The CPE module, a novel contribution of our work, also significantly boosts performance. By aligning eye trajectories with the spatial layout of the visual stimulus, CPE enhances the model's ability to localize attention signals effectively, especially evident in the Levenshtein and Fréchet distances, which are sensitive to spatial alignment. We further evaluate the effect of distributing cross-attention across all U-Net blocks versus using a single cross-attention layer at the beginning. Disabling full cross-attention notably worsens performance, suggesting that deep conditioning through the network is crucial to preserve and propagate stimulus-related signals.
8

---

Stimalus
Ground Touth
DeepCiaze
DeepCiaze IIF
TranSaiNer
SUM
Diffaye
mit
Figure 5: Qualitative comparison of saliency map predictions. Saliency maps generated by DiffEye and baseline models are shown alongside ground truth maps for four different scenes. Each row cor- responds to different stimulus, with columns displaying the stimulus, ground truth, and predictions.
Finally, we test using only global token instead of patch-level features which is concatenated an additional token to the trajectory tokens before passing it through the U-Net. This setup parallels DiffGaze [23], a model designed for 360" images, which also uses conditioning mechanism with global token. Our results show that such global conditioning is insufficient for natural images, where fine-grained patch-level information is needed to localize saliency and highlight the value of our patch-based strategy. Please refer to Figure for qualitative comparisons that further illustrate the contributions of each component. See Supplementary for additional qualitative results for the eye movement trajectory generation.
Saliency Prediction We compare our method against the SUM model [50]. TranSalNet [49], DeepGaze I [47], and DeepGaze IIE [48] which are all trained specifically for the saliency pre- diction task. Each baseline model is used to generate saliency maps for the MITI003 test split used evaluate scanpaths. We convert the continuous eye movement trajectories generated from DiffEye to saliency maps as described in the Supplementary materials. Figure presents qualitative results for the saliency prediction task comparing DiffEye with baseline models. Despite not being explicitly trained for saliency map generation, our model produces visually competitive outputs. For instance, in the second row, while DeepGaze IIE and SUM primarily focus on the subject's s face, DiffEye captures broader set of salient regions, including the face, the bicycle, and the left hand, closely resembling the ground truth distribution. These results demonstrate our model's capacity to generalize beyond its original training objective. Additional evaluations and qualitative results are provided in Supplementary.
Discussion & Limitations
We present DiffEye, a novel diffusion- based model for generating eye movement trajectories on natural images. Unlike prior methods that rely deterministic architectures, autoregressive sampling, and training with compressed representations like saliency maps or discrete scanpaths, DiffEye captures the variability of human visual behavior using state-of-the-art generative modeling and trains directly on raw eye-tracking data. To support this, we introduce CPE, a mechanism that aligns spatial gaze patterns with semantic content via cross-attention between image patches and trajectory timesteps. By modeling scanpaths as a generative process, DiffEye produces realistic and diverse gaze behaviors and achieves state-of-the-art results. Beyond technical performance, has potential applications in developmental science by simulating population-specific gaze patterns. This may support the diagnosis of conditions such as autism by generating stimuli that maximize group-level distinctions in eyc- eye-tracking studies. Although DiffEye achieves strong results using only the MITI003 dataset-the the only known dataset offering raw eye trajectories alongside scanpaths for natural images the dataset size remains a key limitation and scaling its performance will require access additional data. Future work will explore transfer learning from datasets containing saliency maps and scanpaths, and aim to include data from both neurotypical and developmentally diverse populations. We also advocate for the release of raw eye-tracking data. At present, fiffEye generates only fixed-length outputs at 240 Hz; broader data availability would enable support for variable-length sequences and diverse sampling rates. Ultimately, DiffEye could also be used synthesize training data, helping to address data scarcity in this field.

---

Acknowledgments and Disclosure of Funding
Portions of this research were supported in part by the Health Care Engineering Systems Center in the Grainger College of Engineering at UIUC, and the National Institutes of Health (NIH) under award P41 P41EB028242.
References
[1] Alfred L Yarbus. Eye movements and vision. Springer, 2013.
[2] Constantin A. Rothkopf, Dana H. Ballard, and Mary M. Hayhoe. Task and context determine where you look. Journal of Vision, 7(14):16, July 2016.
[3] Viviane Clay. Peter König, and Sabine U. König. Eye tracking in virtual reality. Journal of Eye Movement Research, 12(1), April 2019.
[4] Rehma Razzak, Yi (Joy) Li, Jing (Selena) Hc, Sungchul Jung, Chao Mei, and Yan Huang. Using virtual reality to enhance attention for autistic spectrum disorder with eye tracking. High-Confidence Computing, 5(1):100234, March 2025.
[5] Xiaojun Ren. Jiluan Fan, Ning Xu, Shaowei Wang. Changyu Dong. and Zikai Wen. Dpgazesynth: Enhancing eye-tracking virtual reality privacy with differentially private data synthesis. Information Sciences, 675:120720. July 2024.
[6] Jihwan Kim, Jejoong Kim, Myeongul Jung. Taesoo Kwon, and Kwanguk Kenny Kim. Individualized foveated rendering with eye-tracking head-mounted display. Virtual Reality, 28(1), January 2024.
[7] Wenxuan Liu, Budmonde Duinkharjav, Qi Sun, and Sai Qian Zhang. Fovealnet: Advancing ai ai-driven gaze tracking solutions for efficient foveated rendering in virtual reality. IEEE Transactions on Visualization and Computer Graphics, page 1-, 2025.
[8] Jiaying Liu, Joe Phua, Dean Krugman, Linjia Xu, Glen Nowak, and Lucy Popova. Do young adults attend to health warnings in the first iqos advertisement in the u.s.? an eye-tracking approach. Nicotine &amp; Tobacco Research, 23(5):815 11-8815-822 822, November 2020.
[9] Xuebai Zhang and Shyan-Ming Yuan. An eye tracking analysis for video advertising: Relationship between advertisement elements and effectiveness. IEEE Access, 6:10699-10777 2018.
[10] Kenneth Holmqvist, Marcus Nyström, Richard Andersson, Richard Dewhurst, Haiszka Jarodzka, and Joost Van de Weijer. Eye tracking: A comprehensive guide to methods and measures. oup Oxford, 2011.
[1I] Kritika Nayar, Frederick Shic, Molly Winston, and Molly Losh. A constellation of eye-tracking measures reveals social attention differences asd and the broad autism phenotype. Molecular autism, 13(1):18, 2022.
[12] Ryan Anthony de Belen, Hannah Pincham, Antoinette Hodge, Natalie Silove, Arcot Sowmya, Tomasz Bednarz, and Valsamma Eapen. Eye-tracking correlates response to joint attention preschool children with autism spectrum disorder. BMC psychiatry, 23(1):211, 2023.
[13] James C. McPartland, Sara Jane Webb, Brandon Keehn, and Geraldine Dawson. Patterss of visual attention faces and objects autism spectrum disorder. Journal of Autism and Developmental Disorders. 41(2):148 119-177,,., May 2010.
[14] Chawarska Katarzyna, Volkmar Fred, and Klin Ami. Limited attentional bias for faces in toddlers with autism spectrum disorders. Archives of general psychiatry, 67(2):178-185, 2010.
[15] Zoya Bylinskii, Adrià Recasens, Ali Borji, Aude Oliva, Antonio Torralba. and Frédo Durand. Where should saliency models look next? In Computer Vision-ECCV 2016: 14th European Conference, Amsterilam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 809-824. Springer, 2016.
[16] Ali Borji. Saliency prediction in the deep learning era: Successes and limitations. IEFEE transactions on pattern analysis and machine intelligence, 43(2):67-7700, 2019.
[17] Jason W. Griffin, Adam Naples, Raphael Bernier, Katarzyna Chawarska, Geraldine Dawson, James Dziura, Susan Faja, Shafali Jeste, Natalia Kleinhans, Catherine Sugar, Sara Jane Webb, Frederick Shic, and James McPartland. Spatiotemporal eye movement dynamics reveal altered face prioritization in early visual processing among autistic children. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 10(1):45-51 57, January 2025.
10

---

[18] Tilke Judd. Krista Ehinger. Frédo Durand, and Antonio Torralba. Learning to predict where humans look. In IEEE International Conference Computer Vision (ICCV), 2009.
[19] Matthias Kümmerer, Matthias Bethge, and Thomas S. A. Wallis, Deepgaze ii:: Modeling free-viewing human scanpaths with deep learning. Journal Vision, 22(5):7, April 2022.
[20] Zhibo Yang, Sounak Mondal, Seoyoung Ahn, Ruoyu Xue, Gregory Zelinsky. Minh Hoai, and Dimitris Samaras. Unifying top down and bottom-up scanpath prediction using transformers. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 1683 1693. IEEE, June 2024.
[21] Xianyu Chen, Ming Jiang, and Qi Zhao. Predicting human scanpaths in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10876- 10885, 2021.
[22] Sounak Mondal, Zhibo Yang. Seoyoung Ahn, Dimitris Samaras, Gregory Zelinsky, and Minh Hoai. Gazeformer: Scalable, effective and fast prediction of goal-directed human attention. In 2023 JEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 1441-1450. IEFF, June 2023.
[23] Chuhan Jiao, Yao Wang. Guanhua Zhang, Mihai Bâce, Zhiming Hu, and Andreas Bulling. Diffgaze: diffusion model for continuous gaze sequence generation on 360 images, 2024.
[24] Matthias Kümmerer and Matthias Bethge. State-of-ther ofttre-rtttttttt- human scanpath prediction. arXiv preprint urXiv:2102.12233 12239, 2021.
[25] Wanjie Sun, Zhenzhong Chen, and Feng Wu. Visual scanpath prediction using ior-roi recurrent mixture density network. IEFE transactions on pattern analysis and machine intelligence, 43(6):2101-21 18, 2019.
[26] Mare Assens, Xavier Giro-i Nieto, Kevin McGuinness, and Noel E Connor. Pathgan: Visual scanpath prediction with generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[27] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139.
144, 2020.
[28] Shima Rashidi, Krista Ehinger, Andrew Turpin, and Lars Kulik. Optimal visual search based on a model of target detectability in natural images. Advances in Neural Information Processing Systems, 33:9288 9-929,, 2020.
[29] Zhibo Yang. Sounak Mondal, Scoyoung Ahn, Gregory Zelinsky, Minh Hoai, and Dimitris Samaras. Target-absent human attention. In European Conference on Computer Vision, pages 52 68. Springer, 2022.
[30] Mare Assens Reina, Xavier Giro-i Nieto, Kevin McGuinness, and Noel E O' O'Connor Saltinet: Scan prediction on 360 degree images using saliency volumes. In Proceedings of the IEEE international conference on computer vision workshops, pages 2331-2338, 2017.
[31] Rong Quan, Yantao Lai, Mengyu Qiu, and Dong Liang. Pathformer3d: A 3d scanpath transformer for 360 images. In European Conference on Computer Vision, pages 73-90. Springer, 2024.
[32] Daniel Martin, Ana Serrano, Alexander W Bergman, Gordon Wetzstein, and Belen Masia. Scangan360; generative model realistic scanpaths 360 images. IEEE Transactions on Visualization and Computer Graphics, 28(5):2033-2013, 2022.
[33] Yujia Wang, Fang-Lue Zhang, and Neil A Dodgson. Scantd: 360 scanpath prediction based on time-series diffusion. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7764-7773, 2024.
[34] Xianyu Chen, Ming Jiang, and Qi Zhao. Beyond average: Individualized visual scanpath prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 25420- 25431, 2024.
[35] Anne M. Treisman and Garry Gelade. A feature- intturrinnttgatioon theory of attention. Cognitive Psychology. 12(1):97-1 136, January 1980.
[36] Christof Koch and Shimon Ullman. Shifts in Selective Visual Attention: Towards the Underlying Neural Circuitry, page 115-141. Springer Netherlands, 1987.
[37] John M Findlay and lain D Gilchrist. Visual attention: The active vision perspective. In Vision and attention, pages 83- 103. Springer, 2001.

---

[38] Gregory Zelinsky, A theory of eye movements during target acquisition. Psychological review, 115(4):787. 2008.
[39] L. Itti, C. Koch, and E. Niebur. model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(1 ():1112244-125) 1998.
[40] David J Berg. Susan E Boehake, Robert A Marino, Douglas P Munoz, and Laurent Itti. Free viewing of dynamic stimuli by humans and monkeys. Journal of vision, 165):1--9, 2009.
[41] Ali Borji and Laurent Iti. Sfate-of-the-art in visual attention modeling. IEEE transactions pattern analysis and machine intelligence, 35(1):185 207, 2012.
[42] Yin Li, Xiaodi Hou, Christof Koch, James M. Rehg, and Alan L. Yuille. The secrets of salient object segmentation. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 280-287, 2014.
[43] Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. Salicon: Saliency in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1072-1080, 2015.
[44] Saumya Jetley, Naila Murray, and Eleonora Vig. End-to-end saliency mapping via probability distribution prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5753-5761, 2016.
[45] Srinivas SS Kruthiventi, Kumar Ayush, and R Venkatesh Babu. Deeplix: A fully convolutional neural network for predicting human eye fixations. IEEE Transactions on Image Processing, 26(9):4446-4566 2017.
[46] Marcella Comia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara. Predicting human eye fixations via an lstm-based saliency attentive model. IEEE Transactions on Image Processing. 27(10)55142- 2018.
[47] Matthias Kümmerer, Lucas Theis, and Matthias Bethge. Deep gaze i: Boosting saliency prediction with feature maps trained on imagenet. In ICLR (Workshop). 2015.
[48] Akis Linardos. Matthias Kümmerer, Ori Press, and Matthias Bethge. Deepgaze iie: Calibrated prediction in and of for state-of art saliency modeling. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 12899-12908, 12908, 2021.
[49] Jianxun Lou, Hanhe Lin, David Marshall, Dietmar Saupe, and Hantao Liu. Transalnet: Towards perceptu- ally relevant visual saliency prediction. Neurocomputtig, 494:455-465 467, July 2022.
[50] Alireza Hosseini, Amirhossein Kazerouni, Saeed Akhavan, Michael Brudno, and Babak Taati. Sum: Saliency unification through mamba for visual attention modeling. In 2025 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), page 1597- 1607. IEEE, February 2025.
[51] Dario D Salvucci and Joseph H Goldherg. Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the 2000 symposium on Eye tracking research & applications, pages 71-78, 2000.
[52] Ali Borji and Laurent Itti. Cat2000: A large scale fixation dataset for boosting saliency research. arXiv preprint arviv: 3555::5555.03888 2015.
[53] Juan Xu, Ming Jiang. Shuo Wang. Mohan S Kankanhalli, and Qi Zhao. Predicting human gaze beyond pixels. Journal of vision.
28, 2014.
[54] Lai Jiang, Yifei Li, Shengxi Li, Mai Xu, Se Lei. Yichen Guo, and Bo Huang. Does text attract attention on e-commerce images: A novel saliency prediction dataset and method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2088-2097, 2022.
[55] Yue Jiang, Luis A Leiva, Hamed Rezazadegan Tavakoli, Paul RB Houssel, Julia Kylmälä, and Antti Oulasvirta. Ueyes: understanding visual saliency across user interface types. In Proceedings of the 2023 CHI conference on human factors in computing systems, pages 1-21, 2023.
[56] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
[57] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation, page 234-241. Springer International Publishing. 2015.
12

---

[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, Vishwanathan, and Garnett, editors, Advances Nearal Information Processing Systems. volume 30. Curran Associates, Inc., 2017.
[59] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv:600600415. 2016.
[60] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy V. Vo, Mare Szafranicc, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacidin El-N El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Yao Huang, Shang Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Rojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. Featured Certification.
Shang-Wen
[61] Stephanie Mark Hamilton, Laura Brandt, Axel Feldmann, Zhoutong Zhang, and William Freeman. Featup: A model-agnostic framework features at any resolution. The Twelfth International Conference on Learning Representations, 2024.
[62] Shi Chen, Ming Jiang, Jinhui Yang, and Qi Zhao. Air: Attention with reasoning capability, In ECCV, 2020.
[63] Yupei Chen, Zhibo Yang. Seoyoung Ahn, Dimitris Samaras, Minh Hoai, and Gregory Zelinsky. Coco- search18 fixation dataset for predicting goal-directed attention control. Scientific reports, 11(7) 2021.
[64] Yupci Chen, Zhibo Yang, Souradeep Chakraborty, Sounak Mondal, Scoyoung Ahm, Dimitris Samaras, Minh Hoai, and Gregory Zelinsky. Characterizing target-absent human attention. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 5031 5031-5040, 2022.
[65] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[66] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[67] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021.
[68] Ramin Fahimi and Neil D. B. Bruce. On metrics for measuring scanpath similarity. Behavior Research Methods, 53(2):609-628, August 2020.
[69] Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Fredo Durand. What do different evaluation metries tell us about saliency models? IEEE Transactions Pattern Analysis and Machine Intelligence, 41(3):40--75, March 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have explicitly explained our paper's contribution and scope in the Abstract and Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Our limitations are discussed in the paper in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have describe all information needed to reproduce the experimental results in Sections 3.2, 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We are going to release the code upon acceptance of the paper. The datasets used are publicly available and we describe how used them in Section 3.2.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
httpps.ccppublic//ee /eoeed//eoomdbmiiiiccce ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation and dataset details are described in Sections 4. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: To compute error bars, we would need to perform computationally intensive simulations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss the compute resources used in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::::////uuuss....o https///ecpppcctccececeeeeeaeed
Answer: [Yes]
Justification:
It conforms the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The main motivation of this work is for the use of this technology to better diagnose and understand developmental conditions such autism and this is discussed in Sections 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited the owners of the assets.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method does not involve any LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: /ttpas/:///uus..//eeee /00p:////c.....eeeeee/e////////0 for what should or should not be described.
20

---

A Appendix supplemental material
A.1 Classifier Free Guidance
Given a noised eye movement trajectory R(far) at diffusion timestep faift, we perform two forward passes through the noise prediction network eg: conditioned the visual stimulus I, yielding co( (R(Rfan), tair I), and another using an unconditional input Tuc. defined as zero matrix, yielding ea (R(tar) lair !. The final guided noise prediction ég is computed as:
Eg (1 c) co(R tun !uc) cg(R(fan) taar
(3)
where e is the classifier-free guidance scale, which we set to 4 during inference. To enable this, we simulate the unconditional setting during training by randomly replacing the conditioning input / with a zero matrix in 10% of the training samples. This encourages the model to learn both conditional and unconditional denoising behavior, supporting effective guidance inference time.
A.2 Evaluation Algorithm
Below is the algorithm used to compute the mean and best scores for each of the scanpath and continuous trajectory metrics.
Algorithm Evaluation of Scanpath and Continuous Trajectory Generation Metrics
Require: Test images ground truth scanpaths G. {gt...., }: generated scanpaths S; {s1...., SM y evaluation metric :)
Ensure: Overall best and mean scores across test images
I: Initialize mast_scores +- [1] and mean_scores +- []
2: for each image I do
Initialize image_best bestttt- + [ and image mean +
4: for each g E G; do
5:
Compute distances (d(y, 8) s E S.}
6: best_g minles, dig, 8) mean_g
E.es, d(g,s) s)
Append best_g to image_ best 10: end for
8:
9:
Append mean to image mean
11: Append
image_bes best to best_ 12: Append image_ e_mean to scores
13:
end for
14:
return Overall Best
Σ best_scores Overall Mean
Σ
mean_ scores
A.3 Additional Scanpath Distributions and Evaluations
Please see additional examples scanpaths generated by DiffEye for the MIT1003 dataset in Fig.
In addition to the wajecttry-basee metrics, we evaluate our model using Sequence Score (SS) and Semantic Sequence Score (Sem SS). The results, presented Table 4, show that our model, DiffEye, achieves state-of-the-art performance on the MIT1003 dataset, outperforming all baselines in both SS and Sem SS. Furthermore, our model generalizes effectively by remaining highly competitive on the unseen OSIE dataset. This robust performance is particularly notable given that DiffEye was trained on significantly less data than competing models. For instance, while DeepGazelll was trained on approximately 600,000 scanpaths, our method achieved these results with only 8,900 trajectories, demonstrating our approach's 's ability to produce high-quality scanpaths by leveraging rich. trajectory information.
A.4 Additional Continuous Eye Movement Trajectory Distributions
Please additional examples of continuous eye movement trajectories generated by DiffEye for the MITI003 dataset in Fig. 7.
A.5 Analysis of Saliency Prediction
21

---

Stimulus
Ground Truch
IOR-ROI
Chen
GazeForner
DeepGaze III
HAT
DiftEye
Figure 6: Additional scanpath generation results. Scanpaths generated by DiffEye and baseline models are shown alongside ground truth annotations across four different scenes. Each row represents a unique stimulus, and each column shows the generated scanpaths for each method.
Table 4: Quantitative comparison using Sequence Score (SS) and Semantic Sequence Score (Sem SS) on the MIT1003 and OSIE datasets. We report the mean performance for each method. Bold values indicate the best scores, while underlined values denote the second-best scores.
MIT1003
OSIE
Model
SS (t)
Sem SS (1)
SS (t)
Sem SS (t)
DiffEye (Ours)
0.4782
0.6611
0.4371
0.5837
HAT
0.4079
0.5794
0,4002
0.5791
Gazeformer
0.3531
0.4522
0.2713
0.3602
DeepGazelll
0.4440
0.6604
0.4623
0.6459
ROI
0.4506
0.6603
0.4404
0.6110
Chen et al.
0.4237
0.6397
0.4333
0.5711
To evaluate the spatial realism of our generated eye-tracking sequences, we compared our method against several models trained specifically for the saliency prediction task. The baselines include: the SUM model [50], which integrates the Mamba architecture with a U-Net to output saliency maps across diverse image types; TranSalNet [49], which leverages transformers for saliency prediction; and DeepGaze [47] and DeepGaze IIE [48], which are based on pretrained convolutional neural networks.
Saliency Prediction
Table 5: Saliency prediction comparison. Bold is best and underline is second best.
Method
AUC-Judd f AUC-Borji * NSS 1
SIM
CC
KL J
DeepGaze I
0.883
0.766
2.306
0.484
0.580
0.980
DeepGaze IE
0.923
0.830
3.321
0.618
0.794
0.552
TranSalNet
0.896
0.874
2.443
0.508
0.658
6.369
SUM
0.931
0.8458
3.611
0.727
0.878
1.438
DiffEye
0.832
0.737
1.991
0.447
0.527
1.991
For each image in the test set, we generated 15 eye-tracking trajectories using our method. From each trajectory, we extracted fixation points using script provided by [18], and used these to create individual fixation maps. These maps were aggregated and convolved with a Gaussian kernel to produce single saliency map per image. For the baseline methods, which directly output saliency maps, we passed the same test images through each model. We then compared all predicted saliency maps, including ours, to the ground truth saliency maps provided in the dataset using six standard metries: AUC-Judd, AUC-Borji, Normalized Scanpath Saliency (NSS), Similarity (SIM), Pearson's Correlation Coefficient (CC), and Kullback-Leibler Divergence (KL). Please refer to [69] for comprehensive detailing of the saliency metrics used. Fig. shows additional examples of saliency maps generated by DiffEye and the baselines for the MIT 1003 dataset and Table 5 reports the quantitative results.
22

---

Stimulus
Ground Truth
DiffEye
XV
XV
Figure 7: Additional qualitative results of continuous eye movement trajectory generation. Addi- tional eye movement trajectories generated by DiffEye alongside ground truth annotations across four different scenes.
Stimulus
Ground Truth
DecpKisee
DeepCiue
TranSalNet
DiflEye
Figure 8: Additional qualitative results for saliency prediction. Saliency maps generated by DiffEye and baseline models are shown alongside ground truth maps for four different scenes. Each row corresponds to different stimulus image, with columns displaying the stimulus, ground truth saliency map, and predictions.
23

---

A.6 Analysis of Statistical Properties of Scanpath Generation
To evaluate the plausibility of the generated scanpaths, we compare their statistical properties against those of ground truth human eye movements. Figure 9 shows the distributions of three key metries: saccade amplitude, saccade direction, and inter-saccade angle for both the MITI003 and OSIE datasets.
Our model's performance (blue line) demonstrates strong alignment with the ground truth distribu- tions (black dashed line) across all three metrics. the saccade amplitude distribution, our model successfully captures the peak lower pixel values. For saccade direction, our model accurately reflects the horizontal bias present in human vision (peaks at O and 180 degrees). Finally, in the inter-saccade angle distribution, our model correctly shows strong tendency for forward movements (peak near O degrees) and return saccades (smaller peak near +180 degrees). These results are consistent across both datasets, confirming that our approach generates statistically more realistic scanpaths than the compared methods.
1.0
Secralo soeid
Sucodd Deaccan bul22
Saccale Diaecio Diniludi
3.00
300
100
(a) MIT1003
(b) OSIE
Figure 9: Comparison of statistical properties for generated scanpaths on the (a) MITI003 and (b) OSIE datasets. The distributions for saccade amplitude, saccade direction, and inter-saccade angle of model (blue) compared against ground truth human scanpaths (black, dashed) and several other methods. Our model consistently provides the closest fit to the ground truth distributions.
24


Reviews:
[{"review_id": "V36QM7aiVh", "reviewer": "Reviewer_ftE1", "summary": "The authors devise a system based on diffusion, specifically the denoising diffusion probabilistic (DDPM) model, that takes an image as input and generates eye movement trajectories. The system is trained end-to-end on the raw eye-tracking data from the MIT1003 dataset. The authors introduce several new architectural and algorithmic components, in that image patch features are computed and trajectories are conditioned on partial past trajectories. Importantly, the authors report that the embedding of the gaze positions within a trajectory has to be aligned with the features of the image patch to achieve the observed performance in prediction. As the system generates raw eye movement trajectories, scanpaths and saliency maps can be derived from them, and the authors show improved performance on several benchmarks.", "questions": "Can the authors give the reader an intuitive explanation for the necessity of the CPE alignment beyond the statement: \"Corresponding Positional Embedding (CPE) To further improve spatial alignment, we introduce a novel positional embedding method, Corresponding Positional Embedding (CPE)\"?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The manuscript is clearly written, and the figures very clearly help in evaluating the system’s architecture and simulation results.\n\nIt is a good idea to train models on the eye movement trajectories instead of the scanpaths or empirical saliency maps. The empirical evaluations in this manuscript that generate scanpaths and saliency maps from the generated trajectories support this conclusion. \n\nIt is also a good idea to use a probabilistic model as one of the key observations in human eye movement sequences has been that they are highly variable. \n\nI would be critical about the claim that the system helps “to better understand how the complex human attention system operates on images”. This is one big data fitting exercise. Scientifically, we do not understand the human attention system better. This is also not relevant here, because there will be ample interest in this system simply to be able to better predict human gaze.", "was_revised": true, "final_justification": "I have now not only checked the rebuttal from the authors but also read through the answers to the other reviewers' comments. As I said, I think that this is a valuable contribution and therefore, I maintain my overall positive score. Maybe a few comments.\n\nThe authors use the term scanpath as it is used in the computer vision community which sometimes differs from the use in the cognitive science & neuroscience communities. The term is used in the CV field to distinguish the succession of fixation locations from the full and raw gaze data.\n\nAs for information not being completely shut out during saccades and at least being processed to some degree, see e.g. the work by Martin Rolfs.\n\nMany of the scanpath metrics that are still being used are fundamentally flawed, see e.g. \nMatthias Kuemmerer and Matthias Bethge. State-of- the-art in human scanpath prediction.\n\nThe model indeed does not model individual observers but the distribution across observers. I find this a plus. \n\nIn my view, this manuscript contains more scientific content than other manuscripts that seem on a trajectory of being accepted at Neurips, at least in my batch of papers.", "rebuttal": "Dear Reviewer ftE1,\n\nThank you for your thoughtful and constructive review of our manuscript. We appreciate your valuable feedback.\n\nWe are encouraged that you recognized the core strengths of our work, including:\n* The clarity of the manuscript and the effectiveness of the figures.\n* The value of our approach in training directly on raw eye-movement trajectories.\n* The appropriate use of a probabilistic model to account for the inherent variability in human gaze.\n\n***\n> **I would be critical about the claim that the system helps “to better understand how the complex human attention system operates on images”. This is one big data fitting exercise. Scientifically, we do not understand the human attention system better.**\n\nWe agree with this statement, and will remove this sentence from the final paper and focus on positioning our work in the context of human gaze trajectory generation.\n\n***\n> **Can the authors give the reader an intuitive explanation for the necessity of the CPE alignment beyond the statement: \"Corresponding Positional Embedding (CPE) To further improve spatial alignment, we introduce a novel positional embedding method, Corresponding Positional Embedding (CPE)\"?**\n\nThanks for this useful question, we provide an in-depth explanation below, and we will add this discussion to the final version of the paper. The core challenge in generating a realistic eye-tracking trajectory is to teach the model not just what is in an image, but precisely where the eye should look in relation to that content. Our initial attempts to condition the model using a single global image feature vector led to poor generation quality (see Table 2 w/o Patch Level Features), as the global feature lacks localized spatial semantics necessary for effective conditioning. The model understood the image's global context but could not ground the gaze path in specific local regions. CPE solves this by providing a common spatial \"address system\" for both the visual input and the gaze trajectory.\n\n1. **Creating the \"Map\"**: We first construct a 2D positional embedding grid, denoted as $P \\in \\mathbb{R}^{H \\times W \\times D}$, which acts as a universal map based on the original image's resolution ($H \\times W$). This grid assigns a unique positional vector—an \"address\"—to every coordinate in the image space.\n\n2. **Tagging the Trajectory**: For each coordinate $(x_i, y_i)$ in the eye-tracking trajectory $R$ , we retrieve its corresponding positional address\n $p_i$ from the map $P$ and add it directly to that trajectory point's embedding. The final augmented trajectory token is computed as:\n $R_{i}^{CPE} = R_{proj}[i] + p_{i}$. The trajectory is now a sequence of coordinates, each tagged with its absolute position on the shared map.\n\n3. **Tagging the Image Content**: Crucially, we use the exact same map $P$ for the image's patch-level features, $F_{proj}$. We interpolate the map\n $P$ to the patch resolution, yielding $P' \\in \\mathbb{R}^{H' \\times W' \\times D}$, and add these positional addresses to their corresponding image patch features. This is computed as:\n $F_{CPE} = F_{proj} + P'$. Now, each piece of visual content is also tagged with its absolute position on the same shared map.\n\n\"By sharing and aligning positional information across both trajectories and image patches, CPE enables effective spatial correspondence and interaction between gaze behavior and visual content\". Because both the gaze location and the image content now speak the same spatial language, our model's cross-attention mechanism can effectively associate a gaze point with the visual content at that exact location.\n\nThe importance of this component is empirically demonstrated in our ablation study (Table 2). When CPE is removed (w/o CPE), the model's performance degrades significantly across all metrics, especially those sensitive to spatial alignment like Levenshtein and Fréchet distances. This confirms that CPE's explicit spatial alignment is a critical contribution that enhances the model's ability to effectively localize attention signals.\n***"}, {"review_id": "Ldns0Pguj5", "reviewer": "Reviewer_4EVS", "summary": "Paper proposes a diffusion-based for the novel problem of raw trajectories prediction of gaze. The model was trained a single dataset, MIT1003, as it contains the raw trajectories data. The predictions were converted to scanpaths and saliency maps for direct comparisons with existing scanpath and saliency prediction models.", "questions": "The inadequate experiments in terms of datasets and use of non-standard metrics should be explained and strongly justified.\n\nThe claim that the prediction of the raw trajectories is better than scanpath prediction is not well supported by eye movement studies. The experimental results while somewhat supportive of this claim, was not comprehensive in size (only 2 datasets) and metric (using trajectories-based instead of visual attention metrics) to be convincing.", "limitations": "No. The authors did not adequately address the limitations as explained in the previous review sections.\n\n[After rebuttal]\nAuthors had sufficiently addressed my concerns regarding the positioning and motivation of the work. This is especially that they had evaluated their work with the standard visual attention metrics.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\nS1. The hypothesis that raw trajectories better capture the richness of eye movements/gaze is novel. This is somewhat supported by the experimental results.\n\nS2. The application of diffusion method to visual attention seems novel. \n\nWeaknesses\nW1. The claim by the paper than scanpath is a \"compression of trajectories\" is flawed. Instead, fixations are critical for the brain to consolidate the information in the foveated vision, as shown by various cognitive studies. These works show that information is processed by the brain during fixations and not saccades (trajectories). As such, it is not clear what is the actual value proposition of raw trajectories prediction only (without fixations/saliency). \n\nRucci M., Poletti M. (2015). \"Control and function of fixational eye movements\". Annual Review of Vision Science. 1: 499–518\n\nW2. The comparison of the scanpath and saliency prediction results are limited to 2 datasets. In [20], the COCO-FreeView and COCO-Search18 were included. The calls into the question of the generalization of the proposed method as COCO-FreeView is a large dataset which contains scanpath data.\n\nW3. The evaluation metrics are non-standard for visual attention community. There are standard scanpath metrics like sequence score (SS), Semantic Sequence Score (SemSS). For saliency prediction, AUC (Area Under Receiver Curve) and NSS (Normalized Scanpath Saliency) and Correlation Coefficient (CC). But the paper only uses trajectories-based metrics.\n\nW4. (minor) Paper claims that the current models are unable to capture the diversity of individual viewers. But there is no attempt to model individual differences in their approach. Hence, this point, while true, is irelevant to their work.", "was_revised": true, "final_justification": "Authors had sufficiently addressed my concerns regarding the positioning and motivation of the work. This is especially that they had evaluated their work with the standard visual attention metrics.", "rebuttal": "We sincerely thank the reviewer for their detailed and constructive feedback. We appreciate the acknowledgment of our work's novelty in its hypothesis (S1) and methodology (S2). Below, we address the identified weaknesses and questions by quoting the specific concerns and providing our clarifications. \n***\n### W1: Value Proposition of Raw Trajectories\n> **\"The claim by the paper than scanpath is a 'compression of trajectories' is flawed... it is not clear what is the actual value proposition of raw trajectories prediction only (without fixations/saliency).\"**\n\nWe used \"compressed\" in the sense that scanpaths, while capturing the most critical information for cognitive processing, are a lower-dimensional summary of the full spatio-temporal eye movement data. In the final version of the paper we will use more appropriate language.\n\nOur motivation for modeling raw trajectories is twofold. First, it provides a more fundamental and flexible representation of the gaze data produced by eye tracking. By generating the foundational raw signal, our approach allows end-users the \"flexibility to define and extract fixations using any desired approach\". This is beneficial because different eye tracking studies and products might use different approaches to extract the scanpath. Our model provides the underlying data, making it adaptable to any specific post-processing pipeline. From the modeling perspective, our approach avoids the need for arbitrary decisions that have plagued prior scanpath-based methods.  For example, DeepGaze III and IOR-ROI require the number of fixations to be specified as a fixed input for scanpath synthesis, and this may not be straightforward to specify for a given stimuli. In contrast, DiffEye generates a continuous trajectory. Consequently, the number of fixations emerges naturally as a property of the generated data. This allows the model to produce scanpaths with an adaptive number of fixations for each image, which better reflects the natural diversity of human viewing patterns. As shown in the table below, mean number of fixations generated by our model is closer to the original statistics.\n\n## Number of Fixations\n\n| Model | mean (MIT1003) | mean (OSIE) |\n| :--- | :--- | :--- |\n| Ground Truth | 8.40 | 9.40 |\n| Ours | 8.48 | 8.08 |\n| HAT | 12.04 | 12.37 |\n| Gazeformer | 3.98 | 4.09 |\n| Chen et al. | 10.04 | 12.00 |\n\nSecond, raw trajectories provide a **richer signal for generative models**. They contain fine-grained spatio-temporal information, like microsaccades and saccadic curvature, which is lost during scanpath extraction. As noted in our paper, raw trajectories in the MIT1003 dataset average 723.7 timesteps, while scanpaths average only 8.4, indicating a \"substantial reduction in spatio-temporal information\". Diffusion models have proven highly effective at learning complex, high-dimensional data distributions, making them a natural and powerful choice for modeling the intricate patterns found in raw eye-tracking data. By learning from this richer data space, our model can produce more realistic and nuanced gaze behaviors, which is supported by our superior performance on trajectory-based metrics.\n***\n### W2 & W3: Datasets and Evaluation Metrics\n> **\"The comparison of the scanpath and saliency prediction results are limited to 2 datasets... The calls into the question of the generalization of the proposed method...\"**\n> **\"The evaluation metrics are non-standard for visual attention community... But the paper only uses trajectories-based metrics.\"\nWe understand the reviewer's concerns regarding the scope of our evaluation. Our choices were guided by our core hypothesis and significant practical data constraints.**\n\n**On Datasets:** Our method's primary requirement is the availability of **raw eye-movement trajectory data**, not just scanpaths. The MIT1003 dataset is \"the only publicly available dataset that provides raw eye-tracking data for natural images obtained during a free-viewing task\". While datasets like COCO-FreeView are larger, they only contain scanpath data and thus cannot be used to train our model, which operates on the raw signal. Our purpose in this work is not to claim state-of-the-art scanpath prediction across all datasets, but to introduce and validate a novel methodology for generating continuous eye-tracking data, a task no prior work has addressed for natural images. We demonstrated strong generalization by evaluating on \"the entirely unseen OSIE dataset\", where our model performed competitively without any fine-tuning. We will modify the final version of the paper to clarify our claim for performance.\n\n**On Evaluation Metrics For Scanpaths:** We chose trajectory-based metrics (Levenshtein, DTW, DFD) because our model is inherently generative, and these metrics excel at comparing the similarity between entire sequences, capturing both spatial and temporal structures. This aligns with our goal of evaluating the model's ability to replicate the distribution of human gaze behavior. However, based on the reviewe’s comment we added additional SS and SemSS metrics. Please find SS and SemSS results below:\n\n## MIT1003 Results\n\n| Model | SS (↑) | Sem SS (↑) |\n| :--- | :--- | :--- |\n| Ours | **0.4782** | **0.6611** |\n| HAT | 0.4079 | 0.5794 |\n| Gazeformer | 0.3531 | 0.4522 |\n| DeepGazeIII | 0.4440 | 0.6604 |\n| ROI | 0.4506 | 0.6603 |\n| Chen et al. | 0.4237 | 0.6397 |\n\n## OSIE Results\n\n| Model | SS (↑) | Sem SS (↑) |\n| :--- | :--- | :--- |\n| Ours | 0.4371 | 0.5837 |\n| HAT | 0.4002 | 0.5791 |\n| Gazeformer | 0.2713 | 0.3602 |\n| DeepGazeIII | **0.4623** | **0.6459** |\n| ROI | 0.4404 | 0.6110 |\n| Chen et al. | 0.4333 | 0.5711 |\n\nOur model achieves state-of-the-art performance on the MIT1003 dataset, excelling in both Sequence Score and Semantic Sequence Score. Furthermore, the model generalizes effectively by remaining competitive on the unseen OSIE dataset, validating that the rich signal from raw trajectories produces high-quality scanpaths. This strong generalization stands in sharp contrast to models like ROI, which, having been trained on the OSIE dataset, performs well there but fails to produce competitive results on the MIT dataset. Notably, our model’s robust performance was achieved with significantly less data than competing models. While DeepGazeIII was trained on approximately 600,000 scanpaths from 11,000 images, our model used only 8,900 trajectories from 1,000 images. We attribute any minor performance differences to this vast disparity in data scale, demonstrating that by using rich trajectory information, our method can achieve competitive, and even superior, results with less than 10% of the training images.\n\n**On Evaluation Metrics For Saliency:** We would like to clarify that we did perform a full evaluation using standard saliency metrics (AUC-Judd, AUC-Borji, NSS, SIM, CC, KL). These results are presented in Table 4 in the supplementary material. \n***\n### W4: Modeling Diversity\n> **\"(minor) Paper claims that the current models are unable to capture the diversity of individual viewers. But there is no attempt to model individual differences in their approach. Hence, this point, while true, is irrelevant to their work.\"**\n\nWe thank the reviewer for this clarifying comment.\n\nOur point was simply that stochastic generative models are advantageous relative to deterministic synthesis approaches that  produce a single, \"average\" scanpath for a given image, which fails to capture the natural diversity in gaze trajectories arising when different subjects attend to different regions.\n\nOur diffusion-based approach inherently addresses the stochasticity of eye tracking data across subjects. By learning the entire distribution of trajectories from the population data, our model can be sampled to generate a diverse set of plausible outputs for a single input image. Each generated sample is a trajectory that could have been produced by a human viewer, thereby capturing the variability observed across the population. We will revise the phrasing in the abstract and introduction to make this important distinction clear, and mention the characterization of individual differences as a topic for future work"}, {"review_id": "A8gx37vuM5", "reviewer": "Reviewer_6vm6", "summary": "The work proposes DiffEye, a model that generates fixation trajectories conditioned on the regions inducing the fixations. These trajectories are learned from MIT1003 [18], a dataset that provides the stimuli, the images used for tracking observers' gaze, and the vectors of fixations at (x, y) locations on each image for fifteen observers. \n\nThe model proposed is based, according to the paper's claims,  on DDPM [56], a diffusion model that progressively adds noise to data and trains a U-Net for the reverse process. The work introduces a corresponding positional embedding (CPE) to associate fixation trajectory tokens with image patch tokens. \n\nComparisons are made with state-of-the-art representatives by testing the competitor methods on the MIT1003 selected test set and the OISE test set. \n\nFour metrics have been chosen from  [68]. Based on the chosen metrics and tests run on the competitor's algorithms and code, the results show a significant improvement in scan path generation by the DiffEye model.", "questions": "1. In the implementation paragraph, it is written \"We set the number of diffusion steps to Tdiff = 1000\nduring training and reduce it to 50 during sampling for improved efficiency.\" How are the diffusion steps applied to the input trajectories?\n\n2. What is backpropagated through F^CPE and R^CPE?\n\n3. What kind of temporal information is effectively obtained? Taking the derivative of the fixation locations reveals significant jumps, indicating saccades. How do you use this information? If the temporal aspect is considered only for diffusion, but there is no effect on the prediction, why should it be useful?\n\n4. Have you made some statistics on the MIT1003 trajectories?\n\n5. It is unclear whether you associate class tokens or feature maps with the trajectories. Furthermore, the resizing operations performed by convolution and interpolation are not described, and some examples would be helpful.\n\n6. With Dinov2 and FeatureUp, a specific number of feature maps is obtained. How is this number reduced in the process? Which loss affects the reduction process?", "limitations": "The authors have addressed the limitations, though they appealed mainly to the lack of eye trajectory datasets.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**: \n\n- The primary objective of the paper is to model eye movements with a temporal structure and learning to generate these trajectories conditioned on image semantics. This idea is of great interest, despite not being new in the field of eye movement studies.\n\n- The novelty of the paper lies in highlighting that the dynamics of eye trajectories can provide a better structure for interpreting saccades and what drives them, making it possible to understand priority relations in image semantics. \n\n- Using a diffusion-based model ensures that the trajectory dynamics are based on the Markovian relation between two successive steps. Also, it allows the introduction of a distribution over trajectories, enabling the comparison of different observers' behaviour. \n\n- The paper makes an effort to provide qualitative images of the trajectories of all the observers, as learned from the MIT 1003 fifteen observers. The paper discusses an ablation of the constituent components of the model, focusing on the component that connects region semantics with the locus attended by the gaze in the scan path. \n- Two metrics, namely Dynamic Time Warping and Time Delay Embedding, suggested in [68], highlight the temporal relevance of the predictions. The obtained results, compared with a representative set of competitors and under four metrics, are promising.\n\n**Weaknesses**\n\nDespite the interest in the idea, the model, and the observations made by the authors, as well as the results exhibited, the article struggles to achieve its objectives.\n\n1. The bias reported by the authors of MIT 1003 [18], which is very significant and strongly reduces the possibility of learning general trajectories, is never mentioned. \n\n2. In MIT 1003, we have the locations of fixations. However, one can deduce the saccades (in principle, also the microsaccades) as arc length, as they are the relevant characteristic of the temporal structure of gaze trajectories. For example, in Figure 3 of DeepGazeIII [19], a statistical analysis of saccades and fixations is made, which is missing here. Therefore, the concept of temporal structure is not demonstrated, and it is unclear how it could emerge.\n\n3. The model is based on DDPM [56], utilising the U-Net and the Transformer's sinusoidal position embedding [58]. The denoising objective, however, lacks several aspects that are not discussed and left without rationale: How is $t_{diff}$ distributed? What happens at $t_0$? Which function is $\\epsilon_{\\theta}$ approximating? How is the forward variance set? \n\n4. The section on stimulus conditioning is too vague; several lines are used to explain why Dinov2 is not exploited, and it is ultimately concluded that FeatureUP is chosen. However, for example, the trajectory passed to the U-Net is not described in terms of sampling. The loss function does not consider $R^{CPE}$ or $F^{CPE}$; therefore, it is challenging to understand how the U-Net, whose role in [56] is to learn to remove the scheduled noise, learns to associate feature patches with trajectory steps here. How is CPE included in backpropagation? Note also that the forward noise implies sampling, and how the input trajectories are sampled is particularly difficult, as it needs to avoid losing essential information. Information on this aspect would be made credible using diffusion. \n\n5. In the paragraph Data Preprocessing, it is written (line 222) that trajectories are downsampled. Like image resizing, downsampling affects the significance of the input trajectories, especially being biased at the origin and being such a small number. \n\n6. The reviewer appreciates the effort of providing qualitative images of the gaze trajectories. However, it would have made sense to compare single observers to understand if at least one of the fifteen predicted trajectories is similar to one of the fifteen ground-truth trajectories. Unfortunately, the images are not very explanatory.\n\n7. Concerning the experiments, note that most of the referred works do not use the chosen metrics; the advantage of the chosen metrics against the area under the ROC curve is not explained.", "was_revised": true, "final_justification": "- I have read the author's rebuttal and discussed with the authors the aspect that I considered unclear. \n\n- The authors have made a strong effort to reply to all my concerns. \nHowever, as I have written in my final considerations, their answers, especially on the learning aspects, were contradictory. \n\n- The authors have used a single small biased dataset for training and a minimal dataset for testing. \n\n- I understand that the other reviewers were particularly interested in the fact that the authors declared they can elicit special information on eye motion from the scanpath dataset. Also, one reviewer considered a plus that the scan paths are given as a set. \n\n- I appreciated the insight of the other reviewers, but I remain firm on my criticisms.", "rebuttal": "We sincerely thank the reviewer for their feedback. Below, we address each of the weaknesses and questions.\n***\n**W1**\n\nWe acknowledge that all datasets contain biases, including the well-known center bias in many visual attention datasets. However, our model is designed to mitigate this specific issue. Because DiffEye is conditioned on patch-level image features, it learns to associate gaze with specific semantic content within the image, rather than simply memorizing a spatial prior like a center bias.\n\nOur strong performance (see Table 3 in the original paper) on the **entirely unseen OSIE dataset** serves as direct evidence of this capability. This result demonstrates that our model can generalize beyond the specific characteristics and any potential biases of the MIT1003 training set. We will add a discussion to the limitations section.\n***\n**W2 & Q3 & Q4**\n\nOur core hypothesis is that while fixations are where visual information is primarily processed, the full trajectory—which includes saccade velocity, curvature, and microsaccades—contains a richer, continuous spatio-temporal signal. Modeling this complete signal allows for a more faithful representation of the underlying dynamics of visual exploration, which might be useful for certain studies, and there is no cost for this flexibility as we can always extract the corresponding scanpath if needed.\n\nDiffusion models are well-suited for modeling such continuous, high-dimensional data. The smooth nature of raw trajectories is a better representational match for these models than discrete, variable-length scanpaths. This allows DiffEye to learn the nuanced patterns of human eye movements, which is reflected in our strong performance on metrics that explicitly evaluate temporal sequence similarity, like DTW and TDE. We have extracted additional statistics from the eye tracking trajectories based on the reviewer’s suggestion. While we can’t include a figure in the rebuttal, please find the statistics in the tables below:\n\n### Saccade Amplitude (x10² pixels)\n\n| Model | mean | std | 25% | median | 75% | skew |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| **Ground Truth**| 1.90 | 1.32 | 0.88 | 1.55 | 2.58 | 1.27 |\n| **Ours** | 1.76 | 1.26 | 0.79 | 1.40 | 2.44 | 1.30 |\n| **HAT** | 2.78 | 1.65 | 1.49 | 2.45 | 3.74 | 0.96 |\n| **Gazeformer** | 0.99 | 0.82 | 0.35 | 0.77 | 1.41 | 1.36 |\n| **DeepGazeIII** | 1.89 | 1.37 | 0.84 | 1.51 | 2.62 | 1.23 |\n| **ROI** | 2.54 | 0.92 | 1.95 | 2.42 | 2.94 | 1.51 |\n| **Chen et al.** | 1.49 | 1.84 | 0.26 | 0.57 | 2.34 | 1.64 |\n\n### Saccade Direction (degrees)\n\n| Model | mean | std | 25% | median | 75% | skew |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| **Ground Truth**| 2.5 | 108 | -87.2 | -0.7 | 94.5 | 0.03 |\n| **Ours** | -1.9 | 107 | -93.3 | -1.4 | 88.5 | 0.05 |\n| **HAT** | 0.9 | 107 | -90.5 | 1.1 | 91.1 | -0.01 |\n| **Gazeformer** | 9.0 | 101 | -66.1 | 4.5 | 92.6 | -0.08 |\n| **DeepGazeIII** | -0.02 | 108 | -87.8 | -1.4 | 90.7 | 0.03 |\n| **ROI** | 10.8 | 113 | -47.2 | 1.9 | 111 | -0.09 |\n| **Chen et al.** | 13.2 | 97.3 | -59.0 | 0.0 | 90.0 | 0.06 |\n\n### Inter-Saccade Angle (degrees)\n\n| Model | mean | std | 25% | median | 75% | skew |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| **Ground Truth**| 95.9 | 61.0 | 35.2 | 102 | 157 | -0.13 |\n| **Ours** | 93.7 | 59.7 | 35.8 | 95.6 | 153 | -0.06 |\n| **HAT** | 107 | 51.6 | 66.2 | 116 | 152 | -0.44 |\n| **Gazeformer** | 48.0 | 42.3 | 15.6 | 33.6 | 71.9 | 1.11 |\n| **DeepGazeIII** | 101 | 57.7 | 47.3 | 109 | 156 | -0.24 |\n| **ROI** | 111 | 65.1 | 42.9 | 142 | 170 | -0.50 |\n| **Chen et al.** | 119 | 54.9 | 82.9 | 135 | 169 | -0.67 |\n\n### Number of Fixations\n\n| Model | mean (MIT1003) | mean (OSIE) |\n| :--- | :--- | :--- |\n| Ground Truth | 8.40 | 9.40 |\n| Ours | 8.48 | 8.08 |\n| HAT | 12.04 | 12.37 |\n| Gazeformer | 3.98 | 4.09 |\n| Chen et al. | 10.04 | 12.00 |\n\nWhile we cannot put the statistics for OSIE dataset due to the character limitation, and can't upload figure, we will put them to the final version. This analysis demonstrates that our approach generates scanpaths whose statistical properties closely mirror the ground truth. \n***\n**W3 & W4 & Q1 & Q2 & Q5 & Q6**\n\nWe will significantly expand our methods section to provide the following clarifications.\n\n* **Diffusion Process Details:**\n    * **Forward Process:** For each training sample, a diffusion timestep $t_{diff}$ is uniformly sampled from {1, ..., T_diff}, where $T_{diff} = 1000$. At $t=0$, we have the original, noise-free trajectory $R^{(0)}$. Noise is added to $R^{(0)}$ according to a pre-defined linear noise schedule with variance ranging from $1 \\times 10^{-4}$ to $2 \\times 10^{-2}$, producing the noised trajectory $R^{(t_{diff})}$.\n    * **Reverse Process:** The U-Net, $\\epsilon_\\theta$, is trained to predict the total noise $\\epsilon$ that was added to the original trajectory $R^{(0)}$ to obtain $R^{(t_{diff})}$. For inference, standard diffusion models (DDPMs) can be very slow, as they require reversing the process through all 1000 steps. To improve efficiency, we use a DDIM [66] scheduler. DDIM introduces a more flexible, non-Markovian diffusion process that allows the model to take larger \"jumps\" during the reverse denoising phase. Instead of taking 1000 small steps, DDIM enables high-quality sample generation in significantly fewer steps. This allows us to approximate the reverse process in just 50 steps at inference time, greatly speeding up generation without a major compromise in quality.\n\n* **Stimulus Conditioning and CPE Details:**\n    * **Conditioning Mechanism:** We use feature maps, not class tokens, for conditioning. FeatUp provides a high-resolution feature map for a given image. We interpolate this map to a $32 \\times 32$ grid, resulting in 1024 patch tokens. These patch tokens serve as the key and value inputs to cross-attention layers located within each block of our U-Net. The trajectory tokens (augmented with positional information) serve as the query.\n    * **Backpropagation and CPE:** The loss function is the L2-norm between the ground-truth noise $\\epsilon$ and the U-Net's predicted noise $\\epsilon_\\theta$. Our proposed CPE module is a simple, parameter-free addition operation. Because it is just an addition, gradients flow directly through it to the layers that precede it. This end-to-end process forces the U-Net to learn the correspondence between the visual content (from the patch features) and the gaze location (from the trajectory) to successfully predict and remove the noise. The backpropagated gradients update all trainable parameters, including the U-Net weights, the cross-attention modules, and the initial linear layers that project the raw inputs into the shared embedding dimension.\n***\n**W5**\n\nWe will clarify this phrasing. The vast majority of trajectories in the dataset were shorter than our target length and were padded. A small fraction of sequences that were slightly longer than 720 timesteps were minimally downsampled to fit the fixed-length requirement of the model architecture. Most importantly, the core temporal structure was preserved. We will make it clear that most trajectories were truncated or padded, and only a few were lightly downsampled, with a negligible effect on the signal.\n***\n**W6**\n\nOur intent in the main paper (e.g., Figure 3) was to illustrate that the entire set of our generated trajectories successfully captures the overall distribution of the ground-truth trajectories from all 15 observers. This demonstrates our model’s ability to learn population-level variability, not just replicate a single average path.\n\nHowever, your point about showing a direct one-to-one similarity is crucial. Quantitatively, the\"Best\" score in Tables 2 and 3 is designed to capture exactly this: it reflects the average distance between each ground-truth scanpath and its single closest match among all generated scanpaths. To make this more intuitive, we will add a new qualitative figure to the appendix showing these one-to-one \"best match\" comparisons to visually demonstrate our model's ability to replicate specific human gaze patterns.\n\n***\n**W7**\n\nThe metrics were chosen to fit our primary task of generating dynamic sequences, which differs fundamentally from static saliency prediction. Metrics like the AUC are excellent for evaluating static saliency maps, but they are agnostic to temporal order, path shape, and the dynamics of how a scene is explored. They cannot distinguish between two different scanpaths that happen to cover the same salient regions in a different order.\n\nIn contrast, metrics like DTW and Levenshtein Distance are specifically designed to measure the similarity between two entire sequences.\n* DTW is particularly valuable as it aligns sequences that may vary in speed, capturing similarity in the overall shape and flow of a path.\n* Levenshtein Distance measures the \"edit distance\" between two fixation sequences, providing a robust measure of similarity in both the spatial locations and the order of fixations.\n\nBy using these sequence-based metrics, we perform a \"distribution-wise\" evaluation. A low average score signifies that the distribution of paths produced by our model is structurally and temporally similar to the human distribution. While we evaluate the secondary task of saliency prediction using standard metrics like AUC (provided in the supplementary material), our primary contribution is best assessed by these sequence-aware metrics.\n\nWe provide results on Sequence Score (SS) and Semantic Sequence Score (SemSS) on MIT1003, which further validate our choice of sequence-based evaluation, demonstrating our model's superior ability to capture the spatio-temporal dynamics of human gaze.\n\n| Model | SS (↑) | Sem SS (↑) |\n| :--- | :--- | :--- |\n| Ours | **0.4782** | **0.6611** |\n| HAT | 0.4079 | 0.5794 |\n| Gazeformer | 0.3531 | 0.4522 |\n| DeepGazeIII | 0.4440 | 0.6604 |\n| ROI | 0.4506 | 0.6603 |\n| Chen et al. | 0.4237 | 0.6397 |"}, {"review_id": "0KFS6xTbe7", "reviewer": "Reviewer_A95t", "summary": "This paper concerns the modelling of human overt visual attention (gaze position) when free-viewing images. \nThe paper identifies two problems in the existing literature: first, that models usually target fixations and scanpaths rather than raw eye trace data, and second, that models ignore individual differences.\nThe paper presents a generative model of raw human gaze trace data, using an image-conditioned diffusion model and an alignment procedure \"Corresponding Positional Embedding\".\nThe resulting gaze samples are compared qualitatively and quantitatively to existing scanpath models and saliency models.\nThe authors also conduct a series of ablation studies to quantify the contribution to model performance of various mechanisms or design decisions.", "questions": "### 1. Why does this work?\n\nA main take-home message of the paper is that using the full gaze data samples rather than fixation or fixation sequences allows learning more effective representations.\nWhy is this? \nThe paper currently does not seem to answer this question.\n\nI could imagine that one possibility is that there is simply more data.\nBut this seems unlikely, if we consider that the sample numbers are relatively small compared to larger eyemovement or related datasets (e.g. Salicon).\n\nInstead, I assume the reason is because the model can learn more about the spatial and temporal characteristics of eye movements from the raw gaze traces (for example, the curved trajectories of saccades, which the model seems to produce, or the acceleleration of the eye, which predicts saccade amplitude (the main sequence)).\nIt would be nice to know the relative contribution of these aspects to the model's performance.\n\nTo do this, the authors could train on modified datasets, in which the gaze samples are modified to linearly interpolate between sequential fixation positions in space, time or both. \nThis would selectively remove the saccade curvature and speed information, respectively.\nAdding models trained on this \"ablated\" data would offer an explanation for what information is specifically missing from the previous training sets (that only include sequences of fixations), which would potentially inform how experimentalists collect the most \"useful\" eye movement data.\n\n### 2. Choice of 10 fixations per scanpath for other models\n\nline 244: DeepGaze III and IOR-ROI are set to 10 fixations per scanpath. How do the results depend on this setting? If you classify DiffEye's trajectories into fixations, how many fixations are produced on the same stimuli?\n\n### 3. Predictions too concentrated\n\nSome of DiffEye's predictions seem *too* concentrated (for example: Figure 3, first two images; DiffEye samples contain no fixations on the rear of the wagon or the second person). Which of the considered metrics are most sensitive to the overall spread of predictions, and are these qualitative observations consistent with differences in these metrics?", "limitations": "As above, clarify whether the paper does in fact include individual-level modelling / conditioning, or whether that is only a possible extension.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is good quality and generally clearly-written, except for some details in the methods that are unclear to me. I find the approach potentially significant in advancing modelling of human gaze behavior in free viewing tasks, and the diffusion approach is sufficiently novel.\nThe results compare generally favourably to other methods.\n\nI think the biggest weakness of the paper is that it's not clear to me what it is about modelling the gaze trace data that seems to help (see Questions section below).\nIn addition, while the model is motivated by the need to account for individual differences in gaze behavior, I see no evidence that this model currently actually does this.\nThe data is described in general as collected from $K^i$ different subjects, but then as far as I can tell the model is only conditioned on the stimulus image, not the subject.\nAssuming I have not misunderstood this, then the individual differences aspect must be clearly described as a possible future extension, rather than something the model currently allows.", "was_revised": true, "final_justification": "Given only responses to my comments, I would keep my score as is (4). However, I find the authors' responses to the other reviewers to be quite convincing, especially the inclusion of gaze statistics and the new sequence scores in response to `6vm6`. (Note however that I suspect there might be something off with the distribution summaries for the direction and angle, which are circular variables). Therefore, I raise my score to 5.", "rebuttal": "We sincerely thank the reviewer for their detailed assessment and insightful questions. We are glad they recognize the *potential significance* and *novelty* of our diffusion-based approach for modeling human gaze. Below, we address the weaknesses and questions raised in the review.\n\n***\n\n> **I think the biggest weakness of the paper is that it's not clear to me what it is about modelling the gaze trace data that seems to help (see Questions section below). In addition, while the model is motivated by the need to account for individual differences in gaze behavior, I see no evidence that this model currently actually does this. The data is described in general as collected from \n different subjects, but then as far as I can tell the model is only conditioned on the stimulus image, not the subject. Assuming I have not misunderstood this, then the individual differences aspect must be clearly described as a possible future extension, rather than something the model currently allows.**\n\nThe reviewer correctly points out that our model is conditioned on the stimulus image and not on a specific subject ID. We apologize for any ambiguity in our initial framing. Our primary motivation was not to model *specific individuals*, but rather to capture the **inherent variability** observed across different human subjects when viewing the same image, and we will clarify this in the final version of the paper\n\nTraditional deterministic models predict a single \"average\" scanpath, which fails to represent the stochastic nature of scanpaths. Our generative approach, by training on raw trajectories from multiple subjects, learns a distribution of plausible gaze behaviors for a given image. The goal is to generate samples that reflect the diversity seen in the human population, not to replicate a particular person's gaze pattern. As the reviewer suggests, conditioning on subject-specific information is an interesting and important direction for future research, building upon the foundation we establish here for modeling population-level gaze dynamics.\n\n***\n\n> **Question 1: Why does modeling raw gaze traces work better?**\n\nThis is an excellent question. The reviewer rightly intuits that the benefit goes beyond simply having \"more data.\" Our hypothesis, supported by our results, is that raw eye-tracking trajectories contain a wealth of spatio-temporal information that is discarded when input data only is scanpaths. For instance, in the MIT1003 dataset, the average raw trajectory has about 724 timesteps, while the average scanpath has only 8.4. This lost information includes:\n\n* **Saccadic Dynamics:** The precise curvature, velocity, and acceleration profiles of saccades are lost in scanpath representations. Our model can learn these subtle but important characteristics from the continuous trajectory data.\n* **Suitability for Diffusion Models:** Diffusion models excel at learning complex distributions over continuous, high-dimensional data. Raw gaze trajectories, which are essentially smooth time-series data, are a natural fit for this modeling paradigm. In contrast, scanpaths`are discrete, lower-dimensional sequences, and modeling them directly with diffusion is less straightforward.\n\nThe reviewer's suggestion to train on linearly interpolated trajectories is a very insightful way to perform an ablation study. Such an experiment would allow us to precisely disentangle the contributions of saccade curvature and timing. It is a fantastic idea for a follow-up study. Our paper's primary contribution is to demonstrate that using a diffusion framework on the full, raw trajectory data is not only feasible but also **outperforms** methods that rely on only scanpaths, with the caveat that our experimental findings are limited to datasets containing the raw eye tracking data (a limitation we will make clear in the final paper)..\n\n***\n> **Question 2: Choice of 10 fixations for other models.**\n\nWe thank the reviewer for this important question. The decision to set the number of fixations to 10 for models such as DeepGaze III and IOR-ROI was not an arbitrary choice, but rather a constraint imposed by the models themselves. These specific baselines are designed to generate scanpaths of a fixed, predetermined length and cannot produce variable-length outputs.\n\nWhile we acknowledge that evaluation metrics can be sensitive to scanpath length, our empirical tests showed that the performance of these baselines did not change significantly when this parameter was varied. We therefore adopted a length of 10 for consistency across these models.\n\nThis constraint highlights a key advantage of our method. DiffEye generates a continuous, fixed-duration trajectory (720 timesteps), from which fixations are extracted via post-processing. Consequently, the number of fixations is an emergent property of the generated trajectory, allowing it to vary naturally for each sample, even for the same image. This behavior more closely mirrors that of human observers, whose scanpath lengths are not predetermined.\n\n### Number of Fixations\n\n| Model | mean (MIT1003) | mean (OSIE) |\n| :--- | :--- | :--- |\n| Ground Truth | 8.40 | 9.40 |\n| Ours | 8.48 | 8.08 |\n| HAT | 12.04 | 12.37 |\n| Gazeformer | 3.98 | 4.09 |\n| Chen et al. | 10.04 | 12.00 |\n\nResults are showing the average number of fixations in the generated/ground truth scanpaths for both datasets. As the results show, our model's ability to generate variable-length scanpaths allows it to better match the statistical properties of human gaze across different datasets. For the MIT1003 dataset, the average number of fixations in the ground truth is 8.40, while our model produces scanpaths with 8.48 fixations. Similarly, for the OSIE dataset, our model generates 8.08 fixations per scanpath, closely approximating the ground truth of 9.40. This consistent alignment with human data demonstrates a clear advantage over models that cannot produce variable-length scanpaths.\n\n***\n> **Question 3: Predictions seem too concentrated.**\n\nWe thank the reviewer for this keen qualitative observation. It is true that in some instances, as shown in Figure 3, the model's generated scanpaths can be more concentrated than the ground truth distribution.\n\nThis observation touches on the known challenge of evaluating generative models, where no single metric tells the whole story. The metrics we used—Levenshtein Distance, Discrete Fréchet Distance , DTW, and TDE—are standard in the literature and capture different aspects of sequence and temporal similarity. While our model performs very well on these metrics on average, they may not perfectly capture the *\"overall spread\"* or diversity of fixations across all salient objects.\n\nThis is precisely why we included extensive qualitative results, to provide a more holistic view of the model's performance. The reviewer's point is well-taken, and developing more robust metrics that specifically evaluate the spatial distribution and spread of generated fixations against the ground truth is a valuable avenue for future work.\n***"}]

Author rebuttal:

Meta-review: The reviewers appreciated the novelty of the proposed idea and the strong experimental results. Overall, the ratings from the reviewers are positive. Some concerns were raised regarding certain claims in the paper, as well as the learning aspects, and the limited scope of training and testing on minimal datasets. These issues were addressed to a reasonable extent during the rebuttal. I encourage the authors to further clarify these points and strengthen the experimental validation in the final version.
Decision: Accept (poster)