Title: DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion

Eye Tracking

Scanpath Prediction

Saliency Prediction
Abstract: Numerous models have been developed for scanpath and saliency prediction, which are typically trained on scanpaths, which model eye movement as a sequence of discrete fixation points connected by saccades, while the rich information contained in the raw trajectories is often discarded. Moreover, most existing approaches fail to capture the variability observed among human subjects viewing the same image. They generally predict a single scanpath of fixed, pre-defined length, which conflicts with the inherent diversity and stochastic nature of real-world visual attention. To address these challenges, we propose DiffEye, a diffusion-based training framework designed to model continuous and diverse eye movement trajectories during free viewing of natural images. Our method builds on a diffusion model conditioned on visual stimuli and introduces a novel component, namely Corresponding Positional Embedding (CPE), which aligns spatial gaze information with the patch-based semantic features of the visual input. By leveraging raw eye-tracking trajectories rather than relying on scanpaths, DiffEye captures the inherent variability in human gaze behavior and generates high-quality, realistic eye movement patterns, despite being trained on a comparatively small dataset. The generated trajectories can also be converted into scanpaths and saliency maps, resulting in outputs that more accurately reflect the distribution of human visual attention.
DiffEye is the first method to tackle this task on natural images using a diffusion model while fully leveraging the richness of raw eye-tracking data. Our extensive evaluation shows that DiffEye not only achieves state-of-the-art performance in scanpath generation but also enables, for the first time, the generation of continuous eye movement trajectories. Project webpage: https://diff-eye.github.io/


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper concerns the modelling of human overt visual attention (gaze position) when free-viewing images. \nThe paper identifies two problems in the existing literature: first, that models usually target fixations and scanpaths rather than raw eye trace data, and second, that models ignore individual differences.\nThe paper presents a generative model of raw human gaze trace data, using an image-conditioned diffusion model and an alignment procedure \"Corresponding Positional Embedding\".\nThe resulting gaze samples are compared qualitatively and quantitatively to existing scanpath models and saliency models.\nThe authors also conduct a series of ablation studies to quantify the contribution to model performance of various mechanisms or design decisions.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is good quality and generally clearly-written, except for some details in the methods that are unclear to me. I find the approach potentially significant in advancing modelling of human gaze behavior in free viewing tasks, and the diffusion approach is sufficiently novel.\nThe results compare generally favourably to other methods.\n\nI think the biggest weakness of the paper is that it's not clear to me what it is about modelling the gaze trace data that seems to help (see Questions section below).\nIn addition, while the model is motivated by the need to account for individual differences in gaze behavior, I see no evidence that this model currently actually does this.\nThe data is described in general as collected from $K^i$ different subjects, but then as far as I can tell the model is only conditioned on the stimulus image, not the subject.\nAssuming I have not misunderstood this, then the individual differences aspect must be clearly described as a possible future extension, rather than something the model currently allows.\n\nQUESTIONS:\n### 1. Why does this work?\n\nA main take-home message of the paper is that using the full gaze data samples rather than fixation or fixation sequences allows learning more effective representations.\nWhy is this? \nThe paper currently does not seem to answer this question.\n\nI could imagine that one possibility is that there is simply more data.\nBut this seems unlikely, if we consider that the sample numbers are relatively small compared to larger eyemovement or related datasets (e.g. Salicon).\n\nInstead, I assume the reason is because the model can learn more about the spatial and temporal characteristics of eye movements from the raw gaze traces (for example, the curved trajectories of saccades, which the model seems to produce, or the acceleleration of the eye, which predicts saccade amplitude (the main sequence)).\nIt would be nice to know the relative contribution of these aspects to the model's performance.\n\nTo do this, the authors could train on modified datasets, in which the gaze samples are modified to linearly interpolate between sequential fixation positions in space, time or both. \nThis would selectively remove the saccade curvature and speed information, respectively.\nAdding models trained on this \"ablated\" data would offer an explanation for what information is specifically missing from the previous training sets (that only include sequences of fixations), which would potentially inform how experimentalists collect the most \"useful\" eye movement data.\n\n### 2. Choice of 10 fixations per scanpath for other models\n\nline 244: DeepGaze III and IOR-ROI are set to 10 fixations per scanpath. How do the results depend on this setting? If you classify DiffEye's trajectories into fixations, how many fixations are produced on the same stimuli?\n\n### 3. Predictions too concentrated\n\nSome of DiffEye's predictions seem *too* concentrated (for example: Figure 3, first two images; DiffEye samples contain no fixations on the rear of the wagon or the second person). Which of the considered metrics are most sensitive to the overall spread of predictions, and are these qualitative observations consistent with differences in these metrics?\n\nLIMITATIONS:\nAs above, clarify whether the paper does in fact include individual-level modelling / conditioning, or whether that is only a possible extension.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes DiffEye, a model that generates fixation trajectories conditioned on the regions inducing the fixations. These trajectories are learned from MIT1003 [18], a dataset that provides the stimuli, the images used for tracking observers' gaze, and the vectors of fixations at (x, y) locations on each image for fifteen observers. \n\nThe model proposed is based, according to the paper's claims,  on DDPM [56], a diffusion model that progressively adds noise to data and trains a U-Net for the reverse process. The work introduces a corresponding positional embedding (CPE) to associate fixation trajectory tokens with image patch tokens. \n\nComparisons are made with state-of-the-art representatives by testing the competitor methods on the MIT1003 selected test set and the OISE test set. \n\nFour metrics have been chosen from  [68]. Based on the chosen metrics and tests run on the competitor's algorithms and code, the results show a significant improvement in scan path generation by the DiffEye model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**: \n\n- The primary objective of the paper is to model eye movements with a temporal structure and learning to generate these trajectories conditioned on image semantics. This idea is of great interest, despite not being new in the field of eye movement studies.\n\n- The novelty of the paper lies in highlighting that the dynamics of eye trajectories can provide a better structure for interpreting saccades and what drives them, making it possible to understand priority relations in image semantics. \n\n- Using a diffusion-based model ensures that the trajectory dynamics are based on the Markovian relation between two successive steps. Also, it allows the introduction of a distribution over trajectories, enabling the comparison of different observers' behaviour. \n\n- The paper makes an effort to provide qualitative images of the trajectories of all the observers, as learned from the MIT 1003 fifteen observers. The paper discusses an ablation of the constituent components of the model, focusing on the component that connects region semantics with the locus attended by the gaze in the scan path. \n- Two metrics, namely Dynamic Time Warping and Time Delay Embedding, suggested in [68], highlight the temporal relevance of the predictions. The obtained results, compared with a representative set of competitors and under four metrics, are promising.\n\n**Weaknesses**\n\nDespite the interest in the idea, the model, and the observations made by the authors, as well as the results exhibited, the article struggles to achieve its objectives.\n\n1. The bias reported by the authors of MIT 1003 [18], which is very significant and strongly reduces the possibility of learning general trajectories, is never mentioned. \n\n2. In MIT 1003, we have the locations of fixations. However, one can deduce the saccades (in principle, also the microsaccades) as arc length, as they are the relevant characteristic of the temporal structure of gaze trajectories. For example, in Figure 3 of DeepGazeIII [19], a statistical analysis of saccades and fixations is made, which is missing here. Therefore, the concept of temporal structure is not demonstrated, and it is unclear how it could emerge.\n\n3. The model is based on DDPM [56], utilising the U-Net and the Transformer's sinusoidal position embedding [58]. The denoising objective, however, lacks several aspects that are not discussed and left without rationale: How is $t_{diff}$ distributed? What happens at $t_0$? Which function is $\\epsilon_{\\theta}$ approximating? How is the forward variance set? \n\n4. The section on stimulus conditioning is too vague; several lines are used to explain why Dinov2 is not exploited, and it is ultimately concluded that FeatureUP is chosen. However, for example, the trajectory passed to the U-Net is not described in terms of sampling. The loss function does not consider $R^{CPE}$ or $F^{CPE}$; therefore, it is challenging to understand how the U-Net, whose role in [56] is to learn to remove the scheduled noise, learns to associate feature patches with trajectory steps here. How is CPE included in backpropagation? Note also that the forward noise implies sampling, and how the input trajectories are sampled is particularly difficult, as it needs to avoid losing essential information. Information on this aspect would be made credible using diffusion. \n\n5. In the paragraph Data Preprocessing, it is written (line 222) that trajectories are downsampled. Like image resizing, downsampling affects the significance of the input trajectories, especially being biased at the origin and being such a small number. \n\n6. The reviewer appreciates the effort of providing qualitative images of the gaze trajectories. However, it would have made sense to compare single observers to understand if at least one of the fifteen predicted trajectories is similar to one of the fifteen ground-truth trajectories. Unfortunately, the images are not very explanatory.\n\n7. Concerning the experiments, note that most of the referred works do not use the chosen metrics; the advantage of the chosen metrics against the area under the ROC curve is not explained.\n\nQUESTIONS:\n1. In the implementation paragraph, it is written \"We set the number of diffusion steps to Tdiff = 1000\nduring training and reduce it to 50 during sampling for improved efficiency.\" How are the diffusion steps applied to the input trajectories?\n\n2. What is backpropagated through F^CPE and R^CPE?\n\n3. What kind of temporal information is effectively obtained? Taking the derivative of the fixation locations reveals significant jumps, indicating saccades. How do you use this information? If the temporal aspect is considered only for diffusion, but there is no effect on the prediction, why should it be useful?\n\n4. Have you made some statistics on the MIT1003 trajectories?\n\n5. It is unclear whether you associate class tokens or feature maps with the trajectories. Furthermore, the resizing operations performed by convolution and interpolation are not described, and some examples would be helpful.\n\n6. With Dinov2 and FeatureUp, a specific number of feature maps is obtained. How is this number reduced in the process? Which loss affects the reduction process?\n\nLIMITATIONS:\nThe authors have addressed the limitations, though they appealed mainly to the lack of eye trajectory datasets.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper proposes a diffusion-based for the novel problem of raw trajectories prediction of gaze. The model was trained a single dataset, MIT1003, as it contains the raw trajectories data. The predictions were converted to scanpaths and saliency maps for direct comparisons with existing scanpath and saliency prediction models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nS1. The hypothesis that raw trajectories better capture the richness of eye movements/gaze is novel. This is somewhat supported by the experimental results.\n\nS2. The application of diffusion method to visual attention seems novel. \n\nWeaknesses\nW1. The claim by the paper than scanpath is a \"compression of trajectories\" is flawed. Instead, fixations are critical for the brain to consolidate the information in the foveated vision, as shown by various cognitive studies. These works show that information is processed by the brain during fixations and not saccades (trajectories). As such, it is not clear what is the actual value proposition of raw trajectories prediction only (without fixations/saliency). \n\nRucci M., Poletti M. (2015). \"Control and function of fixational eye movements\". Annual Review of Vision Science. 1: 499–518\n\nW2. The comparison of the scanpath and saliency prediction results are limited to 2 datasets. In [20], the COCO-FreeView and COCO-Search18 were included. The calls into the question of the generalization of the proposed method as COCO-FreeView is a large dataset which contains scanpath data.\n\nW3. The evaluation metrics are non-standard for visual attention community. There are standard scanpath metrics like sequence score (SS), Semantic Sequence Score (SemSS). For saliency prediction, AUC (Area Under Receiver Curve) and NSS (Normalized Scanpath Saliency) and Correlation Coefficient (CC). But the paper only uses trajectories-based metrics.\n\nW4. (minor) Paper claims that the current models are unable to capture the diversity of individual viewers. But there is no attempt to model individual differences in their approach. Hence, this point, while true, is irelevant to their work.\n\nQUESTIONS:\nThe inadequate experiments in terms of datasets and use of non-standard metrics should be explained and strongly justified.\n\nThe claim that the prediction of the raw trajectories is better than scanpath prediction is not well supported by eye movement studies. The experimental results while somewhat supportive of this claim, was not comprehensive in size (only 2 datasets) and metric (using trajectories-based instead of visual attention metrics) to be convincing.\n\nLIMITATIONS:\nNo. The authors did not adequately address the limitations as explained in the previous review sections.\n\n[After rebuttal]\nAuthors had sufficiently addressed my concerns regarding the positioning and motivation of the work. This is especially that they had evaluated their work with the standard visual attention metrics.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors devise a system based on diffusion, specifically the denoising diffusion probabilistic (DDPM) model, that takes an image as input and generates eye movement trajectories. The system is trained end-to-end on the raw eye-tracking data from the MIT1003 dataset. The authors introduce several new architectural and algorithmic components, in that image patch features are computed and trajectories are conditioned on partial past trajectories. Importantly, the authors report that the embedding of the gaze positions within a trajectory has to be aligned with the features of the image patch to achieve the observed performance in prediction. As the system generates raw eye movement trajectories, scanpaths and saliency maps can be derived from them, and the authors show improved performance on several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nThe manuscript is clearly written, and the figures very clearly help in evaluating the system’s architecture and simulation results.\n\nIt is a good idea to train models on the eye movement trajectories instead of the scanpaths or empirical saliency maps. The empirical evaluations in this manuscript that generate scanpaths and saliency maps from the generated trajectories support this conclusion. \n\nIt is also a good idea to use a probabilistic model as one of the key observations in human eye movement sequences has been that they are highly variable. \n\nI would be critical about the claim that the system helps “to better understand how the complex human attention system operates on images”. This is one big data fitting exercise. Scientifically, we do not understand the human attention system better. This is also not relevant here, because there will be ample interest in this system simply to be able to better predict human gaze.\n\nQUESTIONS:\nCan the authors give the reader an intuitive explanation for the necessity of the CPE alignment beyond the statement: \"Corresponding Positional Embedding (CPE) To further improve spatial alignment, we introduce a novel positional embedding method, Corresponding Positional Embedding (CPE)\"?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}