Title: What Moves the Eyes: Doubling Mechanistic Model Performance Using Deep Networks to Discover and Test Cognitive Hypotheses
Venue: NeurIPS 2025
Accepted: yes
Keywords: eye movements

scanpaths

mechanistic modelling

benchmarking

cognitive

data-driven

behaviour
Abstract: Understanding how humans move their eyes to gather visual information is a central question in neuroscience, cognitive science, and vision research. While recent deep learning (DL) models achieve state-of-the-art performance in predicting human scanpaths, their underlying decision processes remain opaque. At an opposite end of the modeling spectrum, cognitively inspired mechanistic models aim to explain scanpath behavior through interpretable cognitive mechanisms but lag far behind in predictive accuracy. In this work, we bridge this gap by using a high-performing deep model—DeepGaze III—to discover and test mechanisms that improve a leading mechanistic model, SceneWalk. By identifying individual fixations where DeepGaze III succeeds and SceneWalk fails, we isolate behaviorally meaningful discrepancies and use them to motivate targeted extensions of the mechanistic framework. These include time-dependent temperature scaling, saccadic momentum and an adaptive cardinal attention bias: Simple, interpretable additions that substantially boost predictive performance. With these extensions, SceneWalk’s explained variance on the MIT1003 dataset doubles from 35% to 70%, setting a new state of the art in mechanistic scanpath prediction. Our findings show how performance-optimized neural networks can serve as tools for cognitive model discovery, offering a new path toward interpretable and high-performing models of visual behavior. Our code is available at https://github.com/bethgelab/what-moves-the-eyes


Peer reviews:
comments: SUMMARY:
The authors propose to improve a mechanical model of eye movements on static images by using a data-driven model and to analyse the controversial predictions between both models in order to improve the mechanical model. The results show a significant improvement in the predicting eye movements.

STRENGTHS AND WEAKNESSES:
The strength of the model presented here is that it presents a mechanical model, and is therefore much more realistic, biologically, which reduces the gap between models that perform well, but are unrealistic and are based on mechanical rules. The inferred statistics of the different characteristics of eye movements during the exploration of an image has particular significance for the scientific community studying  biological movements.

A weakness of the paper is that it is relatively incremental and shows improvement only mainly on the evaluation criteria that were used to learn the data-driven model.

QUESTIONS:
A major contribution of your model and establish a function of temperature and momentum that is introduced into the mechanistic model. Can you relate the parameters you changed to some biological data, whether mechanistic for example by considering the system of muscles that allow eyeball movement.

A major feature of the interaction between eye movements and scene understanding is that the image on the retina is foveated, thus allocating more ressources (photorecpetors) around the axis of sight. Could you relate your model to such models that would use foveated retinotopy?

LIMITATIONS:
One limitation of the paper is its relatively incremental advancements, which, while valuable, do not significantly depart from existing methodologies. Interpretability would be a huge bonus.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses a deep neural network (DNN) model of human gaze behavior called DeepGaze III (DG III) to improve a mechanistic model called SceneWalk. In particular, the authors applied both models to the MIT1003 dataset of images and determined specific cases where the scanpath predictions of DG III and SceneWalk differed. Then, the authors used those \"controversial fixations\" to come up with extensions and modifications to SceneWalk that would make the model perform more similarly to DG III. (In other words, they identified common patterns that explained the different fixation predictions made by the DG III and SceneWalk so that they could generate new mechanisms to add to SceneWalk that fit those patterns.) In particular, they added the following features to SceneWalk: time dependent temperature scaling (favoring high-saliency targets early in scanpaths), saccadic momentum (from an oculumotor bias map), and a horizontal and left-ward bias. These modifications brought SceneWalk's performance from 0.15 bit/fix to about 0.3 bit/fix, compared to DG III's 0.45 bit/fix.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper tackles a central question in cognitive science and computational modeling: what drives visual attention? This question is critical for understanding vision writ-large because what we attend to becomes the training curriculum for visual learning in general. The authors engaged with the literature on scanpath modeling and positioned their work in relation to both cognitive models and deep learning approaches. This is a promising direction, but I'm not yet convinced by the current version.\n\nWeaknesses:\n\n1) The paper sets up a false dichotomy between mechanistic models and DNNs. Deep networks are not inherently non-mechanistic; their internals can embody explicit cognitive hypotheses (e.g., energy-based surprise models for gaze—Aakur & Bagavathi 2020). By equating “mechanistic” with “non-DNN\" the authors ignore the possibility of leveraging deep models that they already recognize as more powerful models. Ironically, the paper is able to generate 3 mechanistic insights from the performance of DG III, and the original DG III paper (Kummerer et al., 2022) includes multiple mechanistic insights like interplay of scene content and scanpath history.\n\nMore broadly, DNNs are becoming increasingly popular as formal, task-performing models of cognition. If a DNN can perform the same task as the brain, then this serves as evidence that the brain may solve the task in a similar way. The brain is a complex system of billions of neurons, and its operations may very well be better characterized by an optimization process (ala deep learning) than by simple cognitive rules. Moreover, while the authors are correct that most deep learning approaches to scanpaths (e.g., DG III) learn to predict human fixation directly, DNNs do not have to be trained this way by necessity. It's possible to train DNNs with the same data as humans (i.e., without supervised human behavior like image labels and scanpaths) to test if they can learn the same skills from the same types of data.\n\nThe result of this false dichotomy is that the authors acknowledge how much more powerful DNN models are, but rather than embracing DNNs and using DNNs directly to learn about scanpath mechanisms, they use DG III to bootstrap SceneWalk. This is analogous to having access to a Ferrari (DNN) and choosing to retrofit a horse-and-buggy cart (cognitive model) to follow its tracks. Rather than embracing the capabilities of the more powerful vehicle, they attempt to enhance the older model—ultimately limiting the practical gains that could be realized by simply using the better tool.\n\n2) It's not clear why the paper does not include a human upper bound on performance. The only explanation offered is that \"estimates of inter-observer consistency are hard to derive\" \"in high-dimensional distributions.\" However, I don't understand why that difficulty would not apply equally to comparing any two high-dimensional distributions (including model-vs-model and human-vs-model). Instead the paper uses DG III as gold standard / ground truth, but without evidence that DG III performance should be trusted as ceiling.\n\n3) Relatedly, by using DG III both as the gold standard and to find controversial fixations, the method becomes somewhat circular. The overall approach is: (1) contrast SceneWalk with DG III, (2) hand-engineer changes to reduce the gap, (3) measure whether SceneWalk now more closely resembles DG III. Because performance is never validated against human behavior, the exercise risks becoming self-referential. Instead, it would help for the authors to show that changes which shrink the SceneWalk-DG III gap also shrink the gap to humans on a held-out images. \n\nAnother change that is also needed to deal with possible circularity issues is to specify (in SI) whether the data used to diagnose SceneWalk-DG III discrepancies is disjoint from the data used to evaluate the final, tuned model. There is some limited discussion about crossvalidation folds in SI, but it's not detailed enough for me to tease apart the stages of training, testing, initial SceneWalk-DG III comparison, and final SceneWalk-DG III evaluation.\n\nQUESTIONS:\nTo make the rebuttal process more constructive, I’ve aligned my questions below with the corresponding numbered weaknesses in my review.\n\n1a. Would the authors consider revising their conceptual framing to reflect the growing use of task-performing DNNs as mechanistic models (e.g., surprise-driven DNNs, goal-directed architectures, or task-constrained pretraining)? This would allow their discussion to better integrate and contextualize recent trends in computational cognitive modeling.\n\n1b. Can the authors justify their choice to retrofit SceneWalk using DG III outputs, rather than analyzing DG III itself to extract mechanistic principles (which they show is possible)? What is gained by working through a weaker model rather than analyzing the stronger one directly?\nThe paper generates mechanistic insights using discrepancies between SceneWalk and DG III, but the original DG III paper also reports its own mechanistic insights. Could the authors clarify why their methodology brings something unique that mechanistic analyses of DG III don't?\n\n1c. Could the insights the authors take from this paper ultimately be applied to mechanistic DNN models? Could we aim for an explanatory model that exceeds DG III?\n\n2. Can the authors provide any estimate of human inter-observer consistency as a comparison point? Or, alternatively, justify in more detail why DG III should be treated as a stand-in for human scanpaths?\n\n3. Could the authors clarify the data partitioning at each stage of their pipeline? Specifically:\n- What data is used for initial SceneWalk vs. DG III comparisons?\n- How is the final performance comparison between SceneWalk vs. DG III done: is it on held-out data?\n- Is there any data from human performance to demonstrate that SceneWalk is now more like human scanpaths and not just more like DG III?\n\nA note on limitations of possible revisions:\nFor the sake of transparency, I want to be clear that even if the authors address all the questions above, my core concern may still remain. The central methodological premise — retrofitting a simpler model to approximate a stronger one that already exists and performs better — seems conceptually inefficient and hard to justify. Unless the authors can articulate why this retrofit strategy is necessary or uniquely revealing, my evaluation may not shift substantially, even if surface-level issues are resolved.\n\nLIMITATIONS:\nPotential limitations are already addressed above. I see little (if any) potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this manuscript the authors improve the mechanistic SceneWalk model which predicts scan paths of humans free viewing natural scenes. They base these improvements on comparisons to DeepGaze III, a deep learning model for the same task, which yields “controversial fixations”, I.e. fixations where the deep learning model still performs much better than the mechanistic one. Concretely they add time dependent temperature scaling to make the predictions higher entropy later in the trial, saccadic momentum to implement return saccades and the tendency to continue moving in the same direction, and an attention bias towards the cardinal directions and horizontal directions in particular. Overall these additions improve the prediction quality of the SceneWalk model by a substantial 0.2 bits/fixation on the DAEMONS and MIT1003 data sets.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, this is solid work. The authors clearly improve the model by comparing to deep learning predictions and also implement the SceneWalk model in JAX which doubtlessly will improve speed, too. Also, the additions to the model are all sensible and correspond to regularly observed data patterns in scan paths. However, this does imply that the comparison to DeepGaze might not have been strictly necessary and comparing to deep learning models to improve mechanistic ones is also not a new idea. To their Credit the authors acknowledge and cite the earlier occurrences of these ideas. Thus, I overall like this work, but I do not believe it is a big breakthrough.\n\nIn this manuscript the authors add a number of interpretable new mechanisms to the state of the art mechanistic model of eye movements. While I think specialist experimentalists in this particular subfield could know about the existence of the modelled phenomena, there are no mechanistic models that contain the added effects. Thus, this work really moves the state of the art forward for this type of mechanistic eye movement model. Also the path chosen to implement this improvement is a fairly recent addition in cognitive modelling toolkit, which I have not seen applied to eye movement models anywhere before and the authors promise to make their expanded model and a faster version of the original one public. So this paper is---in my view---a substantial and noteworthy step forward for models of eye movement trajectories.\n\nAlso, I like that this article does actually appear solid. For each of the new mechanisms there is evidence in the data and the authors know of corresponding literature and cite it. This makes this an exceptionally clear and solid paper. This solidness is also the reason for my fairly short review. I read through the descriptions of all the different mechanisms and found them to be generally sensible. If they were exceptional ideas or questionable, I would have had much more to write about, but they are all fairly straightforward implementations of the observed phenomena.\n\nI value solid research highly and believe this paper does well on this front, which is why I vote for acceptance even though I have to acknowledge that the parts that the authors put together here have been presented elsewhere before, such that an argument for originality or novelty is hard to make.\n\nQUESTIONS:\nThe weighted log-likelihood difference is an unusual measure for the difference between predictions. Why do the authors prefer it over a more classical measure like the KL-divergence?\n\nWouldn’t it be interesting to evaluate the model additions on the data used originally to develop the SceneWalk model? They are available (https://osf.io/n3byq/)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate how a high-performing deep learning model (DeepGaze III) can be used to improve a more interpretable one (SceneWalk) for human gaze fixation prediction. The authors compare the predictions of DeepGaze III and SceneWalk on empirical gaze data, identify discrepancies between them, and use insights from this analysis to revise the SceneWalk model. The proposed modifications, such as incorporating saccadic momentum and time-dependent temperature scaling, lead to notable performance improvements, bringing SceneWalk closer to DeepGaze in predictive power.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents an interesting idea: leveraging a highly-performing but difficult to interpret model to guide improvements to a more interpretable one. However, several aspects of the execution limit the paper’s impact and clarity.\n* It is unclear why the use of a generative model like DeepGaze is useful to improve a more interpretable model like SceneWalk, beyond what could be achieved from the empirical data alone. The authors assert that this type of analysis is difficult with empirical data but provide no quantitative result to support this claim.\n* The authors mention that the modifications to SceneWalk were already known in the literature. If these mechanisms were previously hypothesized, it is unclear why they had not been implemented in SceneWalk before. The paper would benefit from clarifying how their approach identifies these mechanisms in a more data-driven or systematic way, rather than simply reintroducing known components.\n* Another unclear point is why the authors don't consider DeepGaze as a mechanistic model. While it is indeed more difficult to interpret a deep learning model, it does not necessarily mean that it doesn’t capture mechanisms of the human visual system. The main difference between DeepGaze and SceneWalk seems more to be in how easily interpretable they are.\n* The authors mention that “these controversial fixations can be classified into few categories”. A quantitative analysis of these categories of prediction differences would be that claim stronger.\n* The cross-validation method is crucial to evaluate the risk of overfitting in the results, but it is only briefly mentioned in the appendix and lacks detail.\n* Overall, the approach presented by the authors is interesting but lacks systematicity and generality. Making the method more general and less reliant on manual analysis and insight would make it more impactful.\n\nQUESTIONS:\n* How exactly is the data split for the cross-validation? Is it only split across fixations? Or are some of the stimuli and/or participants held-out as well?\n* In the controversial stimuli analysis, it would help to have a quantitative analysis of the different types of prediction differences between the models.\n* Overall, is there a way to make the method more systematic and less manual? The proposed method would be more impactful if it would be applicable to more models and datasets.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}