What Moves the Eyes: Doubling Mechanistic Model Performance Using Deep Networks to Discover and Test Cognitive Hypotheses
Authors: Federico D'Agostino
Lisa Schwetlick
Matthias Bethge
Matthias Kuemmerer
Venue: NeurIPS 2025 poster ()
Keywords: eye movements
scanpaths
mechanistic modelling
benchmarking
cognitive
data-driven
behaviour
Primary area: neuroscience_and_cognitive_science
TL;DR: A systematic fixation-level comparison of a performance-optimized DNN scanpath model and a mechanistic cognitive reveals behaviourally relevant mechanisms that can be added to the mechanistic model to substantially improve performance

Abstract: Understanding how humans move their eyes to gather visual information is a central question in neuroscience, cognitive science, and vision research. While recent deep learning (DL) models achieve state-of-the-art performance in predicting human scanpaths, their underlying decision processes remain opaque. At an opposite end of the modeling spectrum, cognitively inspired mechanistic models aim to explain scanpath behavior through interpretable cognitive mechanisms but lag far behind in predictive accuracy. In this work, we bridge this gap by using a high-performing deep model—DeepGaze III—to discover and test mechanisms that improve a leading mechanistic model, SceneWalk. By identifying individual fixations where DeepGaze III succeeds and SceneWalk fails, we isolate behaviorally meaningful discrepancies and use them to motivate targeted extensions of the mechanistic framework. These include time-dependent temperature scaling, saccadic momentum and an adaptive cardinal attention bias: Simple, interpretable additions that substantially boost predictive performance. With these extensions, SceneWalk’s explained variance on the MIT1003 dataset doubles from 35% to 70%, setting a new state of the art in mechanistic scanpath prediction. Our findings show how performance-optimized neural networks can serve as tools for cognitive model discovery, offering a new path toward interpretable and high-performing models of visual behavior. Our code is available at https://github.com/bethgelab/what-moves-the-eyes

What Moves the Eyes: Doubling Mechanistic Model Performance Using Deep Networks to Discover and Test Cognitive Hypotheses
Federico D'Agostino
Lisa Schwetlick EPFL
Tübingen AI Center, University Tübingen federico. wwg.sttnoobeehhheeeeeee org
lisa. Alaasschwetlickkeppll.ch
Matthias Bethge*
Matthias Kümmerer'
Tübingen Al Center, University of Tübingen matthias@bethgelabb
Tübingen Al Center, University Tübingen matthias kuemmeree@@ethheel... org
Abstract
Understanding how humans move their eyes gather visual information is central question in neuroscience, cognitive science, and vision research. While recent deep learning (DL) models achieve state-of-the-ar performance in predict- ing human scanpaths, their underlying decision processes remain opaque. At an opposite end of the modeling spectrum, cognitively inspired mechanistic mod- els aim to explain scanpath behavior through interpretable cognitive mechanisms but lag far behind predictive accuracy. this work, we bridge this gap by using segg-perforiig deep podell-DeepGaan III to discover and test mech- anisms that improve a leading mechanistic model, SalkeWalk. By identifying individual fixations where DeepGaze IIII succeeds and SceneWalk fails, we iso- late behaviorally meaningful discrepancies and use them to motivate targeted extensions of the mechanistic framework. These include time-dependent tem- perature scaling, saccadic momentum and an adaptive cardinal attention bias: Simple, interpretable additions that substantially boost predictive performance. With these extensions, Salk''aak'ss explained variance on the MIT1003 dataset doubles from 35% 70%, setting prediction. Our findings show how performance-optimnzzee neural networks can serve tools for cognitive model discovery, offering pretable and high-performing models of visual behavior. Our code is available at https: htaps:///gthuble wtt.ggtt....o.beeeeeeeeeeeeeeeee www.mmeeegeeaa//eeeeoeeeooaaaaee what moves moves-the-eyes
state of the art in mechanistic scanpath
path toward inter-
Introduction
Every time we view scene, our eyes produce a sequence of rapid movements (saccades) and brief pauses (fixations), selectively directing high-resolution fovea to parts of the visual world. This active sampling process-capturee in so-called ccanpatss-reveal we perceive, attend to, and make sense of our surroundings. Understanding what governs fixation selection is therefore a key question in visual neuroscience and cognitive science and also receives substantial interest from computer vision due to many applications ranging from compression [3 to design and layouting [15]
great deal about how
Indeed, fixation selection has been extensively studied in variety of scenarios, from reading 65
23
Shared senior authorship
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

to scene viewing 13 specifically on modeling free viewing of natural scenes.
93
80 45
to performing real-world tasks
33
53
64
Here we focus
Fixation data
b
3.0
DeepGaze
2.4-
Potentially
Propose validata mechanistta
2.9
SceneWalk-X
explainable
Loogs hood (onts))))
Machanisticc model
model
2.0-
DospGaze
Explained
insights from DeepGaze
2.7
ScenWWalk
2.6
Controvensial lisations
33.4%
Explained by previously- known mochanlsms
2.6
4
2.5
Spatia/
Model
Figure I: (a) We systematically compare prediction performances of mechanistic scanpath model (SleneWalk) to a nigh-performing DNN model (DeepGaze III) to find situations that could be predicted very well but are not by the mechanistic model. Inspecting these extreme cases results in ideas for effects than can be confirmed through further analyses and give rise to mechanisms that are added to the mechanistic model. (b) This process yields three new mechanisms that double Scene predictive performance on MIT1003, substantially narrowing the explainability gap.
Approaches to this question in the modeling literature can be viewed as occupying different points on a Pareto front trading off predictive performance with direct mechanistic interpretability. At one end, models like SceneWalk (24] 68 770 are constructed from a small set of interpretable components that directly reflect cognitive hypotheses about attentional dynamics, oculomotor constraints, and perceptual processing. These approaches align with the view of scientific understanding as form of compression [87] the goal is to explain complex phenomena through a small set of generalizable principles, a critical aim even when performance is not maximized. At the other end of the spectrum, ceriormance-orienteed deep neural networks (DNNs) like DeepGaze III (5I] take a data-driven approach, learning to predict human fixations from large-scale data without explicitly modeling the underlying mechanisms. Their strength lies in establishing a high mark for predictive accuracy, revealing the extent of predictable structure in the data.
As common for the prediction of complex behavior, deep scanpath models indeed outperform mechanistic ones in predicting where people look [44). But this predictive success comes at a cost: While DNNs can embed mechanisms and offer scientific insights [1), they often capture complex behavioral patterns in -igh-dimensional parameter spaces that are not immediately transparent 27). Conversely, mechanistic models, while more transparent, typically under-perform due their parsimonious parametrization and due to the same strong prior assumptions that make them great testbeds for formulating and evaluating theoretical principles. This leaves a gap: We know that certain behaviors are robustly predictable by data-driven models, but we lack compact, mechanistic account of them. One flourishing line of research addresses this by starting from the predictive end of the modeling Pareto front, developing more transparent, hybrid, or task-constrained DNNs. In this work, however, we pursue a complementary strategy embracing the mechanistic modeling philosophy: starting from the interpretable end, we ask how much of the performance gap can be closed by systematically improving the fully interpretable model.
A principled way to close this gap, then, is to focus on the mechanistic model's largest prediction errors. In the domain of static spatial saliency, model failures are often assessed by comparing model predictions to inter-observer consistency, which serves as an empirical upper bound on pre- dictability 44 14 12 45). However, in high-dimensional distributions such as scanpaths, estimates of inter-observer consistency have been so far intractable: Each fixation is sampled from a condi- tional distribution I[f++///s 7) that depends on the unique history of previous fixations. Because scanpaths are highly idiosyncratic, the same fixation history is almost never observed more than once, leaving us with single empirical sample for each high-dimensional distribution. This sparse coverage makes it challenging to form stable, non non-parametric estimates of inter-observer agreement.
2

---

Here is where a perffornnn DNN becomes indispensable. We propose to use DeepGaze III not just as benchmark, but as a dense. queryable approximation of the true genteranng process [41 2] By directly comparing the predictions of DeepGaze III and SceneWalk at the level of individual fixations, we identify controversial fixations: fixatiin Cases where the models strongly disagree. These disagreements highlight specific behavioral patterns captured by the deep model but missed by the mechanistic one. We use them as entry points for analysis, confirming with empirical data whether the behavior is human-like, and then reformulating the missing tendencies explicit mechanisms within the Scene framework. A key question at the outset was whether the discrepancies between the models would reveal few systematic, missing principles or a myriad of subtle, complex effects. Interestingly, as we will show, the disagreements clustered into clear, actionable categories, validating the approach. Subsequent model comparisons between versions of the mechanistic model provide a strong test for the newly added mechanisms. This iterative process repurposes the deep model as a tool for mechanistic discovery, helping quantify how much specific mechanisms contribute to scanpath behavior and bridging the gap between prediction and understanding. As such, our primary contribution is not the discovery of entirely novel phenomena- would also have been possible-, but a systematic, model-guided methodology for prioritizing which of the many plausible mechanisms most improve prediction on naturalistic data.
We evaluate our method on four free-viewing datasets and demonstrate that augmenting Scene Walk with mechanisms inspired by DeepGaze leads to consistent performance improvements, bringing interpretability closer to predictive performance by halving the explanatory gap on most datasets. In doing so, we offer roadmap for principled model development that leverages the complementary strengths of hypothesis- and data-driven approaches.
2 Related Work
Scanpath Modeling While the field of eye movement prediction has been focused for long time
mainly on spatial fixation density prediction ("saliency prediction", [36] [1] 38
47
56
22 see
45 for an overview), the problem of predicting the dependencies within scanpaths of fixations
has also attracted substantial interest. The seminal model of Itti and Koch [36] already predicted
scanpaths, and since then a range of different mechanistic models have been proposed. Many models take direct inspiration from neuroscience and biology (MASC [1, LATEST [8]), Star-FC
[89]
40 or build on statistical ideas such as CLE (5] SaccadicFlow [19], WALD-EM 43 and
others 110 79
55 54 21 90 144, and some models implement ideas from cognitive science and
attention research, such as Walk [24 68 70). pepploratton-xxpoittaati
[59] and ROI-LSTM
(78). Finally, the progress in machine learning has brought deep-learning based models that either only partially interpretable or complete black boxes such as SaltiNet (4. PathGAN 4. DeepGaze III 11 GazeFormer (60), ScanDMM (77) and HAT [92] Many scanpath models take given spatial saliency map of the image, however, there are also models that incorporate more complex interactions of scene content and scanpath dynamics. An extensive comparison 44 51 showed that Scene Walk is the best mechanistic model in terms of explained information, which is why we are using it as the starting point of our project. While here we focus on free-viewing behaviour, other models also take into account task dependency, such as visual search 1 91 or inter-subject
differences 17 37
DNNs as scientific tools One conceptual foundation of our work is the idea that DNNs can serve as queryable empirical stand-ins for Bayesian ideal observers in domains where the analytical solution is intractable 21I. A direct extension of this idea is Scientific Regret Minimization (SRM (2):: using expressive machine learning models as powerful approximators of the data gentaagnnrating process, leveraging their predictions to identify systematic failures in simpler. hypothesis-driven models. Crucially, this emphasizes cases where we because the DNN predicts accurately- -that the behavior should have been predictable, but the mechanistic model fails. This reframes the role of DNNs not merely as predictive tools, but as guides for scientific inquiry.
Model comparisons for insight and hypothesis generation Our method also relates to work com- paring model predictions to extract insights or derive interpretable mechanisms. Classic approaches include some early work in model distillation, where the goal is to extract simpler, interpretable models from complex DNNs e.g., distilling neural networks into decision trees (26). Other work frames model comparison way to generate maximally informative contrasts between hypotheses
3

---

or models. An example of this is the Maximum Differentiation (MAD) Competition for perceptual models (35, or the use of controversial stimuli to pit vision models against each other in regions of disagreement [30] These approaches resonate with our method of using disagreement between models to uncover where new mechanisms may be required.
Finally, it is important to distinguish our approach from different line of research in post hoc mechanistic interpretability methods, such feature attribution 42 95], which aim to explain black-box models' internals. In contrast, we treat the DNN as an opaque but predictive reference, and focus instead on refining an interpretable model by learning from its disagreements with the DNN.
3 Methods and Experimental Setup
Modeling Framework and Evaluation Our models and evaluations are formulated in the frame- work of cit-fiiiaaioo-prdcttio
45:: Eye movements can be considered a sequential decision process in which each decision is about where to look next. This means that given an image I and a sequence of previous fixations fo..... fv. models predict the next fixation fi+1 as a conditional probability distribution p(fe+1 fa...., I). or for short p(f++1 fge, I). Compared to predicting whole scanpaths, the framework of to inspect model predictions for each individual fixation in the dataset and is compatible with classic spatial fixation density prediction: it constitutes the special case p(fi+1 fsi, = p(f++1 | T) where each fixation is predicted independently of previously made fixations. Model predictions can naturally be scored via log- ligllikel log 1(fe(f1+ Si.... fo. 1) which we report in bits and relative to the uniform baseline model as log p(f++
comes with the advantage that it allows
Ji: Ji..... So, 1) - log Puniform (fe+1 () (sometimes also termed information gain, IG (48). Information gain has been shown to be a preferable metric due to its sensitive nature [50] and we use it as our main metric of predicton quality. The traditionally often used scanpath similarity metrics [3] are not applicable in our case because they don't 't allow
Lingle-fiation analyses and have additional conceptual problems
Explainable Information Gain Because information gain constitutes ratio scale 766.. differences and relative changes are meaningful. This forms the basis of explainable information gain [48], which quantifies how much of the theoretically possible performance a model achieves. In spatial saliency, this is measured between a lower bound (i.e. center bias) and an upper bound (i.e. gold-standard empirical model), with the difference defining the explainable information gain log log 7) p(f)- In our setting, we are interested in modeling the dependency of fixations on scanpath history. Therefore, the static gold standard from spatial saliency becomes our lower baseline p(f++1 7). The upper bound, a principled gold standard model of p(fi+1 fsi, 1). is intractable to estimate non-parametrically from empirical data due to the sparsity of unique scanpath histories. Since each trial generates a unique fixation history, we are typically left with only a single data point for each nigh-dimensional conditional distribution, precluding direct estimation of inter-observer consistency. We therefore approximate this ceiling with what is to our knowledge the best coltommancc-opimiizeee model for scanpath prediction, DeepGaze III [5I]. which serves queryable proxy for human predictability. As a model, it also enables controlled analyses of the data distribution that sparse empirical observations alone do not permit (see Fig. 5
Controversial Fixations At the heart of our analysis technique is what we call controversial fixa- tions, based the concept of controversial stimuli [30). A controversial fixation is a fixation for which the predictions from DeepGaze III and SceneWalk, given the previous scanpath history, differ sub- stantially. We use two different measure to quantify these differences: First, we use the likeiiood difference (LJD) log PDGS (J Sgi, 7)- log (fe+1 JKt, I). Large values mean that DeepGaze III assigns much more probability to given fixation than SW. c.g. times as much for a difference of 3 bit. Such differences suggest that while the fixation is predictable, the mechanisms currently imple- mented by SceneWalk do not capture well. Additionally, we use the weighted log-likelihood differ- ence (WLLD) PuGs (ftt1 fsa, I) (log Poca (fit1 fgi. I)- log (f fs;, I)). By weighting with DeepGaze's 's probability, we select fixations with large LLD that are additionally considered very likely by DeepGaze. We chose these fixation centric metrics over full-distribution measures like KL--dvergeence because they focus the analysis on the actually observed fixation and are substantially more computationally efficient. This means that we focus more on cases that DeepGaze considers easy and which might hence showcase effects more clearly.
These controversial fixations serve entry points for further analysis. Rather than proposing mecha-

---

early saccades
continuing saccades
10 B3%
longer saccades
shorter saccades
Drafioa burail
3.3 30%b
2.53 Dith
Eng Derery
2005
Figure 2: Controversial Fixations: Those fixations in the dataset where Scene Walk loses most performance compared DeepGaze III in terms of LLD or WLLD. Numbers on top of predictions indicate the achieved log-likelihood relative to uniform model for the fixation in question.
nisms ad hoc from single examples, use these disagreements to guide targeted comparisons across models and empirical data. Patterns that generalize across fixations are formalized and integrated into the Scene Walk model. This provides principled alternative to intuition-driver model development, using prediction disagreement to identify which candidate mechanisms are most impactful.
Experimental Setup
Datasets We use the MITI003 dataset (38] for our controversial fixations experiments. It well established in the gaze prediction community since it is the training set for the MIT300 benchmark dataset (39] of the MIT/Tübingen Saliency Benchmark 46). It features broad range of mostly photographic images, including indoor, urban, and natural scenes, which are viewed by 15 observers for 3 seconds each. All training and evaluation on this dataset employs 10-fold cross-validation scheme across images (with separate training, validation and test splits, see Appendix D), ensuring that models are always tested on stimuli (and their associated fixations) held out during training.
We additionally use the DAEMONS 72), COCO Freeview [18] and Potsdam corpus [69] datasets for additional evaluations of the mechanisms added through our controversial fixations analyses on MIT1003.
Spatial Baseline Our spatial baseline model tries predict the average spatial fixation density p(J 7) on images, i.e. without taking scanpath dependencies into account. We require it for two reasons: Firstly need it for putting scanpath model performances into perspective: performance gains compared to the spatial bascline show how well models exploit the scanpath dynamics. Secondly, as is the case for many scanpath models, the SceneWalk model requires input spatial saliency map: a map encoding which image areas are interesting independent of scanpath dynamics. Consistent with previous applications of the model 224 70 we use our spatial baseline model to that end. Our spatial baseline model is mixture model of KDE of empirical fixations and predictions from SOTA saliency model, where the parameters are fitted per image. For details see Appendix C
SceneWall The SceneWalk model [24 68 70 is mechanistic scanpath model inspired by results from cognitive science. It implements an activation map and inhibition map, which evolve over time depending on the fixated image locations. The activation map is updated for each fixation with Gaussian window on the saliency map and decays over time, encoding what the model knows about relevant parts of the image. The inhibition map is updated for each fixation with a Gaussian at the fixation location and also decays over time, encoding which image areas the model has already explored. To predict the next saccade target, activation and inhibition maps are postprocessed with an exponential nonlinearity and combined to yield the fixation selection map. Here, we use the latest version of Scene Walk [70] which additionally includes pari-saccadic attention shift mechanism:
5

---

Prior to saccade, attention is already shifted towards the saccade target and hence, after the saccade, small area the direction of the saccade is uncovered. This mechanism results higher prediction performance and explains the directional statistics of scanpaths better. SceneWall Walk has only about 20 parameters, which we fit using MAP estimation (for details see Appendix E.
DeepGaze III [51] is a deep learning based scanpath model. For predicting the next fixation in a scanpath, it takes as input the viewed image and the last four fixation locations. The input image is encoded with pretrained backbone and processed by priority network that decodes a spatial priority map. This spatial priority map is then combined with the information about previous fixation locations in fixation selection network to output priority map that is conditioned on the previous scanpath history. This map is blurred, combined with a center bias and passed through a softmax to yield the probability distribution for the upcoming fixation location. For better comparability with Scene Walk, here we removed the priority network and instead used the same empirical densities that are also used by Scene Walk in the fixation selection network. This serves to make sure that all differences between the two models are necessarily due to how they combine the spatial priority map with the fixation history and avoids confounders. We fit DeepGaze IIII in this setting on the MIT1003 dataset using 10-fold crossvalidation across images (see Appendix E
DeepGaze III
Experiments and Results
a
b
Entropy
12.0
Expected Sa
11.5
DeepGaze
DeepGaze SceneViblk amprical
11.0
SceneWalk
11
4
12
Scarpath
Scanpath
C
early saccade
late saccade
Gernma
2.4
Procotype version
Gamma for base model
B-11 NoNt
1.2
Emaiiiaa feaa Derett
3.51 Norty
Erightcn Samrtyy
3.65 B81
Fixation index
d
Log Log-llkeslhh.om (bitsthx]
temp, scaling
DeepGaze
7 Nettr
3. BBy
Fixation
index
Figure 3: Time dependent temperature scaling. We found that DeepGaze shows higher confidence (less entropy) early on (a) and predicts fixations more salient locations (b) agreement with the empirical data. SceneWalt is not showing this effect. To address this, we introduced fixation-index. dependent temperature scaling (c), modeled with exponential decay (d), which improves predictions for both early and late fixations (c).
5.1 Model Comparison
Explainable information By evaluating our spatial baseline, DeepGaze and SceneWalk on the held- out test cross- cross-validan validation splits of the 1001003 dataset, we find that DeepGaze explains 0.43 bit/fix more than the spatial baseline. From estimate of the explainable information gain, the SceneWall model explains with 0.15 about one third, leaving explanatory gap of two thirds.
Controversial Fixations After the full-dataset level analysis, we now go to the level individual fixations and inspect those fixations where SceneWalk shows the largest failures. For both log- likelihood difference and weighted logelikel difference, we select the top six images with largest mean score and from them visualize the six fixations with highest individual scores. We find that

---

nearly all (92%, see Appendix D of these controversial fixations can be classified into four categories (Figure 2] DeepGaze III seems to gain by predicting that (1) early saccades go exclusively to the most salient targets and that (2) often saccades tend to continue in the direction of the previous saccade. In addition, DeepGaze gains by having a better understanding of when saccades are (3) longer or (4) shorter.
5.2 New Mechanisms in Scene Walk
Time Dependent Temperature Scaling Controversial fixations revealed that DeepGaze favors high-saliency targets early in scanpaths, with this effect diminishing time (Fig. 2 We confirmed that DeepGaze indeed shows higher confidence (less entropy) early on (Fig. 31) and predicts fixations in more salient locations in agreement with the empirical data (Fig. Bp) Salk partially captures this effect thanks to the buildup of the inhibitory stream over time which forces the model to increasingly attend less salient locations. However, the effect in Walk clearly weaker than in DeepGaze and very off for the first fixation. The original Scene Walk model already uses exponents to shape the activation and inhibition maps to be more or less deterministic, i.e. emphasize highly activated areas. This essentially regulates the temperature of the distributions. To model the change in the focus on salient locations, we now made this exponent in SceneWalk dependent on the fixation index in the scanpath (Fig. Bb). We first used different independent exponents for earlier and later fixations ("prototype version" in Fig. 3 These exponents closely resembled an exponential decay and hence for the final version we fitted the exponent an exponential decay over the course of a scanpath (details in Appendix B.I). The result is a strong early preference for salient locations, with the exponent decaying below the baseline model's value around the third fixation, allowing more exploratory behavior later in viewing. This improves prediction performance both for early and late fixations (Fig- (i) and also visibly improves the behavior on the corresponding controversial fixations (Fig. 3b). Interestingly, it has been hypothesized before that the temperature of the fixation distribution might change over presentation time (69].
a
frequency
0.4
pINN vacrade c deg
prev vaccade > deg
b
frequency
0.4
eafly ABccade
saccade
Scerwilek
DGI 6AG7
Scenewiiic
0.2
0.2
0.0
0.0
back right tum none left tur saccade direction change
back
back right turn none left turn back saccade direction change
Scaling factor Combined preference
Fixation Ind-1 à saccade len-1 deg
Fixation ind=5.A saccade len-5 deg
0.9
0.8
0.7
back
right turn
none
turn
back
Direction change
d
1.0
6 BRTK
9:30 Befe
2.5
5.0 7.5
5
10
Prev. saccade longth [deg]
Fixation index
Figure 4: Saccadic momentum: Controversial fixations suggested that DeepGaze sometimes prefers saccade continue in the same direction. We confirmed such a saccadic momentum effect especially after long saccades (a) and later in scanpaths (b). These effects remain even after controlling for the distribution of salient objects (dashed lines), indicating genuine directional bias. We added a saccadic momentum and return mechanism to Scene Walk (c) modulated by previous saccade length (d) and fixation index (e), improving predictions for "ongoing saccades" controversial fixations (f).
Saccadic Momentum second prominent cluster of controversial fixations clearly indicated that
DeepGaze more accurately predicted short, continuing fixations mostly following long saccades. We found the effect be consistent with known phenomena of saccadic momentum 77 75 88], which have also been previously used in scanpath models [40]. We noticed that this behavior is
7

---

particularly pronounced after long saccades [75] and during later stages of the scanpath (..). While some of this structure could be explained by saliency alone, control analyses using DeepGaze evaluated on uniform empirical fixation densities (i.e., with saliency effectively removed) revealed that the momentum effect largely persists (Fig dashed lines), suggesting reflects some form of oculomotor bias rather than image content. Besides momentum, both DeepGaze and the empirical data exhibited a balance between returning to previously attended regions and continuing in the same direction, especially during early viewing and following short saccades.
SceneWalk's 's perisaccadic attention shift can partially explain such behavior, but much more weakly than in the data (see Appendix Fig for a larger figure). To address this, we introduced dynamic oculomotor bias map that captures both saccadic momentum and return tendencies. The mechanism modulates directional preferences based on the length of the previous saccade and the position within the scanpath and applies the resulting bias map to SceneWalk's internal probability map (details in Appendix B.2). Following short saccades early in a scanpath, the model learns to balance return saceades and momentum, while after long saccades and later in viewing, it favors continued movement in the same direction (Fig. 4c). This extension significantly improves SceneWalk s ability to capture directional patterns in controversial fixations, bringing its predictions closer to those of DeepGaze (Fig. 4). Again, this mechanism aligns with prior literature suggesting that saccadic momentum and return effects may jointly reflect oculomotor constraints or
mechanisms 74 88 677. and even more complex dynamics beyond muscle physics 662
Saccadas Going
b 000oo00
DeepGaze
DeepGaze saliency)
SceneWalk
mequee
0.4
0.4
0.4
50%
02
0.
0.2
0.2
40%
right Saccade direction
right up Saccade direction
left
left down right up Saccade direction
Fixation Index in Scanpath
Attention map Cardinal leftward bias
Attention map ibles appliedi
h
3.10 tirk
(base)
horizontal
g
left-ward
blas
balanced
balanced
Fixation Index
Fixation Index
Figure 5: Horizontal and left-wards attention bias: Controversial fixations suggested that DeepGaze predicts early saccades to go the left. This is indeed the case in the data (a) and the DeepGaze predictions (b). The effect persists when DeepGaze is run with uniform saliency maps (c) while SclneWalk (d) shows too weak effect that increases over time. Therefore, attention bias to SceneWalk, than can additionally have a left-asymmetry (c) and can adapt over time (f. g). This improves predictions the relevant controversial fixations (f).
added cardinal
horizontal and left-wards attentional bias Especially for early controversial fixations, DeepGaze scemed very confident about left-wards saccades (Figure 2). We found his effect in- deed present in the data (Fig. 55)), as previously reported elsewhere (25). We tested how well the models reflected the effect by computing, for each individual fixation, histogram of expected saccade directions and averaged them for earlier and later fixations. This confirmed that DeepGaze learned a left-warde and more generally horizontal bias (Fig. 5b)) Importantly, the bias is still there when we evaluate DeepGaze with uniform saliency maps 5p), confirming that the effect is not not purely saliency-driven Interestingly, DeepGaze's leftward bias fades over time, while its horizontal bias builds up across early fixations. Scene Walk, in contrast, shows only weak directionality and an incorrect increase in leftward bias over time (Fig. H),, possibly reflecting artifacts from the empirical densities.
8

---

We decided to account for these directional biases by applying variable cardinal prior to Scene Walk's 's attention map before saccade selection. This modification is motivated by two complementary observations. First, purely oculomotor bias-like the one implemented by 40 is unlikely to explain the found preference for early saccades to the left or the weaker horizontal bias for early fixations even in the absence of saliency. Second, beyond known effects of cardinal oculomotor biases, empirical evidence suggests that attentional processes themselves exhibit anisotropics across the cardinal axes parsiiaaiiullarlyy favoring horizontal over vertical directions [83
57
82 Notably,
attentional modulation has been observed as early as the lateral geniculate nucleus (57 lending
biological plausibility to attentional bias operating in concert with oculomotor constraints.
Concretely, our cardinal attention bias is implemented as a smooth function of saccade angle, with preference peaking along the cardinal axes and decaying with distance from the current fixation. Additionally, we model dynamic asymmetry between horizontal and vertical directions, with the horizontal bias gradually increasing over the course of trial (Fig. 5D Finally, to account for the early leftward bias, we also introduced an asymmetry that favors leftward over rightward saccades during early fixations, with strength that decays over time (Fig. 5gg The resulting cardinal attention bias map is combined multiplicatively with the attention map (Fig. Se). For details see Appendix B.3
5.3 Results
A new SOTA for mechanistic scanpath prediction On the MITIO03 dataset, our full extended SclneWalk model (which we term SceneWal--X for short), doubles the performance of the original Scene Walk perisaccadic model compared to the spatial baseline. It further closes 56% of the remaining gap in explainable performance as estimated by DeepGaze. Since Scene was the previous SOTA on MIT1003 in mechanistic scanpath prediction [5I], this sets a new SOTA in mechanistic scanpath prediction. Notably, we evaluated SceneWalks X also on the DAEMONS dataset [72]. which is specifically designed for scanpath prediction and includes longer scanpaths on high- resolution natural scenes. Scene performs better when trained and evaluated on DAEMONS, where it already explains more than 60% of the explainable information relative to the baseline. SceneWalk-X closes 40% of the residual gap in explainable performance between DeepGaze and the base SceneWall model. While this is smaller than the gains observed MITI003, it is worth noting that DAEMONS contains longer, more complex scanpaths where the spatial baseline alone is less predictive, and the original Walk model already performs well cenl-eonsistent with its design focus on extended viewing behavior. Nonetheless, SceneWalk Walk-X still nearly halves the remaining gap to DeepGaze, demonstrating its robustness and leaving room for future improvements specific to this richer dataset.
More evaluations In Appendix A we confirm our results hold on two more datasets (COCO Freeview [18 and the Potsdam Scene Viewing Corpus (69] and compare with other mechanistic models. We further report AUC scores for all datasets to show that our results do not depend on the chosen evaluation metric.
Relevance Assessment To quantitativly assess the overall relevance of each added effect, we evaluated different extensions SceneWalk and also compared to the earlier "classic" SceneWalk 24 68) without perisaccadic attention shifts (Fig. (). We find that on MIT1003, each of the three mechanisms contributes about equally to the overall increase in explanatory performance. On the DAEMONS dataset, the performance gain is mostly due to momentum and cardinal biases. while temperature scaling contributes little. We hypothesize that this might be because DAEMONS features much longer scanpaths, and the temperature stays mostly constant after the first few fixations.
Reconciling controversial fixations In Appendix D we show all examined controversial fixations together with the predictions from the fully extended Scene model and find that many of them are now predicted substantially better.
Discussion
Data-driven (DNN) and hypothesis-driven (mechanistic) approaches to modelling cognitive and perceptual processes are usually painted as being contrast with predict, the other primarily to understand and confirm hypotheses. In this work we demonstrate how
each other. One aims primarily to

---

MIT1003
DAEMONS
2.9
Spatial
3.0
Perfommnnce Rxpple Fredd EM4
Log Liveeeooooo lettstttt
OeepGaze
80%
-00%
Srenwnkk (perisccade)]
2.8
Individual component contribut
80%
-60%
2.4
canoinal bas
perka cedde
scaling
SW
ScalTi
209
-00%
2.0
Figure 6: Contributions of the different added mechanisms on the MIT1003 and DAEMONS datasets.
both techniques can be used in concert to improve understanding of the mechanisms underlying fixation selection. By focusing on fixations where the predictions a deep model (DeepGaze III) and mechanistic model (SceneWalk) diverged, isolated specifie mechanisms implicitly captured by the deep model but absent in Scene These mechanisms were then formalized and incrementally implemented into SceneWalk. Each addition tested via its impact predictive likelihood, thereby providing a strong test of its behavioral relevance.
The conceptual cornerstone of our method is to a DNN model as stand-in for predictability. This turns black-box model into a scientific instrument: not something we trust blindly, but something can probe, compare against, and learn from. This shift circumvents many of what the classic weaknesses of deep learning and focuses instead on the strenghts of such models. For example, DeepGaze III allows to test whether certain spatial temporal biases affect behavior beyond what is already explained by bottom-up saliency- a manipulation that would be extremely difficult, if not impossible, novel ways to look at the underlying data distribution. When combined with direct comparisons to mechanistic models, this enables more flexibly look data and formulate hypotheses.
portrayed as
achieve with empirical data alone. In this sense, deep models offer
Importantly, the main contribution of our method not necessarily in discovering novel mechanisms. Many of the effects we identify, have been observed before, and we mainly add nuance to the way they might be operating. which will require future work to confirm. Rather, our main contribution here lies in offering a principled procedure for identifying which among the many plausible mechanisms are most useful for improving behavioral predictions. By translating prediction errors into candidate explanations and rigorously testing them in mechanistic framework, our method strengthens the link between predictive performance and theoretical insight.
Limitations and outlook A few potential limitations and subtleties of our approach warrant further discussion. First, our method assumes that DeepGaze provides a reasonable proxy for human fixation behavior, but, like any DNN, it may learn dataset-specific biases or shorteuts (28] 27 Second, we acknowledge that adding a mechanism which improves the mechanistic model's fit does not necessarily imply that the same mechanism is used by humans, it could be correlate or computational proxy. While mechanistic models aim for interpretability. they are still approximations and subject to their own inductive biases. Grounding mechanisms in empirical findings helps address this limitation. Finally, our current pipeline relies on manual identification and implementation of mechanisms, a deliberate choice to validate the methodology We view this work as a foundational proof of concept. Automating this search over candidate model extensions, especially for more complex interactions across time and context, remains an open and exciting challenge (29) that points toward generalizable, semi-automate
transparent, end-to-end fashion.
modeling pipeline- step towards what could be seen as an "automated scientist" [e-g., 58 for behavioral modeling. Similarly, our results are based on scanpaths over natural images; applying this framework to other domains such as reading, video, or real- wal-world interactions is an exciting direction for future research.
Broader impact Accurate scanpath models have practical relevance in fields such as visual design, education, and assistive technology. However, there are ethical considerations: predictive models of attention could be used to manipulate user behavior, for example in advertising or UX design. It is therefore important that such models are developed transparently and deployed with caution.
10

---

Acknowledgments
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Federico D' Agostino. This work was supported by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, project number: 276693517.
References
[1] H. Adoli, F. Vitu, and G. J. Zelinsky. A Model of the Superior Colliculus Predicts Fixation Locations during Scene Viewing and Visual Scarch. The Journal of Neuroscience, 37(6):1453-1445 Feb. 2017. ISSN 0270-6474, 1529-2401. doi: 1005E///UE 16.2016.
[2] M. Agrawal, J. C. Peterson, and I. L. Griffiths. Scaling psychology via Scientitic Regret Minimization. Proceedings of the National Academy of Sciences, 1915841117.
Apr. 2020. doi: 10. 1073//a/ss
[3] N. C. Anderson, F. Anderson, A. Kingstone, and W. F. Bischof. A comparison of scanpath comparison methods. Behavior Research Methods, 3744):1377-1392 Dec. 2015. ISSN 1554-3528. doi: 10.3758/ s13428-014 070888014-555---
[4] M. Assens, X. Giro-i-Nieto, K. McGuinness, and N. E. O'Connor. PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks. In L. Leal-Taixé and S. Roth, editors, Computer Vision ECCV 2018 Workshops, Lecture Notes in Computer Science, pages 406 422, Cham, 2019. Springer International Publishing. ISBN 978-3-030- 110115.. doi: 10.1007/978- 3-030-11021-3 11021-5_25.
[5] G. Baccignone and M. Ferraro. Modelling gaze shift constrained random walk. Physica A: Statistical Mechanics and its Applications, 331())2007-218, Jan. 2004. ISSN 0378-4371, doi: 10.1016/.physa.200. 09.011.
[6] G. Boccignone and M. Ferraro. Ecological Sampling of Gaze Shifts. IEEE Transactions on Cybernetics, 24(2):266-27, Feb. 2014. ISSN 2168-2275. doi: 10.1 1L..LOCY...0011..2222
[7] A. Borji and L. Itti. State-of-the-Art in Visual Attention Modeling. IEEE Transactions Pottern Analysis and Machine Intelligence, 35(1) 1155)27777 Jan. 2013. ISSN 0162-8828. doi: 2..1099TPAMI.2012
[8] A. Borji and L.. Itti. CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research. CVPR 2015 workshop "Future of Datasets" 2015.
[9] J. Bradbury, R. Frostic, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke,
J. VanderPlas, S. Wanderman-MMlne, and Q. Zhang. JAX: Composable transformations of Python+NumPy programs. jax-ml, May 2025.
[10] D. Brockmann and T. Gcisel. The ecology of gaze shifts. Nearocomputing, 32-33:643-650, June 2000. ISSN 0925-2312. doi: 10.1016/50925- 2000000925-232(0000) 20000000277
[11] N. D. B. Bruce and J. K. Tsotsos. Saliency, attention, and visual search: An information theoretic approach. Journal of Vision, -33):-5,, Mar. 2009. ISSN 1534-7362. doi: 10.1167/9.3.5
[12] N. D. B. Bruce, C. Wloka, N. Frosst, S. Rahman, and K. Tsotsos. On computational modeling of visual saliency: Examining what's right, and what's 's left. Vision Research, 116, Part B:95-112, Nov. 2015. ISSN 0042-6989. doi:
2015.00..
[13] G. Buswell. How People Look at Pictures. The University Chicago Press, 1935.
[14] Z. Bylinskii, Recasens, A. Borji. A. Oliva, A. Torralba, and F. Durand. Where Should Saliency Models Look Next? In Computer Vision ECCV 2016, Lecture Notes in Computer Science, pages 809-824. Springer, Cham, Oct. 2016. ISBN 978-3-319-46453-4 47644-1. 978-3-319-4666 999-3-319-44454-14499
doi: 10.1007/
[15] Z. Bylinskii, N. W. Kim, O'Donovan, Alsheikh. S. Madan. Pfister, F. Durand, Russell, and A. Hertzmann. Learning Visual Importance for Graphic Designs and Data Visualizations. In Proceedings of the 30th Annual ACM Symposiun on User Interface Software ond Technology, UIST '17, pages 57-69, New York, NY, USA, Oct. 2017. Association for Computing Machinery. ISBN 978-1-45033-488-9 doi: 10.1145/312654433 116665..
[16] G. Cartella, M. Comia,, V. Cuculo, A. D' A'melo,, D. Zanca, G. Boccignone, and R. Cucchiara. Trends, Applications, and Challenges in Human Attention Modelling. Apr. 2024.
11

---

[17] X. Chen, M. Jiang, and Q. Zhao. Beyond Average: Individualized Visual Scanpath Prediction. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 25420- 25431, 2024.
[18] Y. Chen, 7. Yang, S. Chakraborty, S. Mondal, S. Ahn, D. Samaras, M. Hoai, and G. Zelinsky. Characterizing Target-Ahsent Human Attention. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5031-5040, 2022.
[19] A. D. F. Clarke, M. J. Stainer, B. W. Tatler, and A. R. Hunt. The saccadic flow baseline: Accounting image-independent biases in fixation behavior. Journal of Vision, 17(11):12 12, Sept. 2017. ISSN 7362. doi: 10.1167/17.11.1..
[20] D. Cousineau. Confidence intervals in within-subjeen designs: simpler solution Loftus and Masson's method. Tutorials in Quantitative Methods Psychology, 1(1):22 45, Sept. 2005. ISSN 1913-4126. doi: 10.209882hmm..01.....
[21] A. Coutrot, H. Hsiao, and B. Chan. Scanpath modeling and classification with hidden Markov models. Behavior Research Methods, 50(1))362-379 Feb. 2018. ISSN 1554-3528. doi: 10.3758/ $13428-017-0876-8
[22] R. Droste, Jiao, and J. A. Noble. Unified Image and Video Saliency Modeling. H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision ECCV 2020, Lecture Notes in Computer Science, pages 419-435, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58558-7. doi: 01177797883-00-5855555 25.
Vedaldi,
[23] R. Engbert, A. Longtin, and R. Kliegl. A dynamical model of saccade generation in reading based on spatially distributed lexical processing. Vision Research, 42(5):621-66 6661, Mar. 2002. ISSN 0042-6989. doi:
(0006//8044-69899011)000
[24] R. Engbert, H. A. Trukenbrod, S. Barthelmé, and F. A. Wichmann. Spatial statistics and attentional dynamics in scene viewing. Journal of Vision, 15(1):14-14, Jan. 2015. ISSN 1534-7362. doi: 10.1167/15. 1.14.
[25] T. Foulsham, A. Gray, E. Nasiopoulos, and A. Kingstone. Leftward biases in picture scanning and line bisection: A gaze-contingent window study. Vision Research, 78:14-25, Jan. 2013. ISSN 0042-6989. doi: 10. 1..1016).visrrs.22111
[26] N. Frosst and G. Hinton. Distilling a Neural Network Into Soft Decision Tree, Nov. 2017.
[27] C. Funke, J. Borowski, T. Wallis, W. Brendel, A. Ecker, and M. Bethge. Comparing the ability of humans and DNNs recognise closed contours cluttered images. Journal of Vision, 18 18100)8000800 Sept. 2018. ISSN 1534-7362. doi: 10. 10.1167/18.10.800
[28] R. Geirhos, J. Jacobsen, C. Michaclis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shorteut learning deep neural networks. Nature Machine Intelligence, 2(16):6556773 Nov. 2020. ISSN 2522-5839, doi: 10. (0.1008//42256-000 -00-22660-00-0055222
[29] A. Ghosh, S. Dziadzio, A. Prabhu, V. Udandarao, S. Albanie, and M. Bethge. ONEBench to Test Them All: Sample-Level Benchmarking Over Open- Open-Ended Capabilities. Dec. 2024.
[30] T. Golan, P. C. Raju, and N. Kriegeskorte. Controversial stimuli: Pitting neural networks against each other as models human cognition. Proceedings of the National Academy of Sciences, 117(47)29330-2993) 29337, Nov. 2020. doi: 10.1073/pnas. 19121912222 23341177.
[31] R. Gupta, M. Thapar Khanna, and S. Chaudhury. Visual saliency guided video compression algorithm. Signal Processing: Image Communication, 100- Oct. 2013. ISSN 0923-5965. doi: 10.1016/j. image.201 201ge.2013.07..0 3.07.003.
[32] C. R. Harris, K. J. Millman. S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, K. Sheppard, T. Reddy, W. Weekesser, H. Abbasi, C. Gohlke, and T. E. Oliphant. Array programming with NumPy. Nature, 585(7825)3357 362, Sept. 2020. ISSN 1476-4687. doi:
J. F. del Río, M. Wiebe, P. Peterson, P.
[33] M. Hayhoe and D. Ballard. Eye movements in natural behavior. Trends in Cognitive Sciences, 9(4): 188-194, Apr. 2005. ISSN 1364-6613. doi: 10.1016/j. wtt.:iisss.....00000
[34] D. Hoppe and C. A. Rothkopf. Multi-step planning of eye movements in visual search. Scientific Reports. 9(1: 12, Jan. 2019. ISSN 2045-2322, doi: 10.1038//41598-01 018--3533---0000
12

---

[35] D. Hunter. Matplolli:: A 2D Graphics Environment. Computing in Science & Engineering, 93):90-95.
2007. ISSN 1521-9615. doi: 10.1 1109/CCEE.2000.555.
[36] L. Itti, C. Koch, and E. Niebur. A model of saliency- based visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence, 20(1
1998.
[37] Y. Jiang. Z. Guo, H. Rezazadegan Tavakoli, L. A. Leiva, and A. Oulasvirta. EyeFormer: Predicting Personalized Scanpaths with Transformer-Guitn Reinforcement Learning. Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, UIST '24, pages 1-15, New York, NY, USA, Oct. 2024. Association for Computing Machinery. ISBN 379-8-4007-000 10. 00.144/3654477.36777777
doi:
[38] T. Judd, K. Phinger, F. Durand, and A. Torralaa. Learning predict where humans look. In Computer Vision, 2009 IEFE 12th International Conference On, pages 2106-2113. IEEE, 2009.
[39] T. Judd, F. Durand, and A. Torralba. A Benchmark of Computational Models of Saliency Predict Human Fixations. MIT Tech Report, Jan. 2012.
[40] F. Kadner, T. Thomas, D. Hoppe, and C. Rothkopf. Improving Saliency Models' Predictions of the Next Fixation With Humans' Intrinsic Cost of Gaze Shifts. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2104-21 14, 2023.
[41] A. J. Kell and J. H. McDermott. Deep neural network models of sensory systems: Windows onto the role of task constraints. Current Opinion in Neurobiology, 55:121 Apr. 2019. ISSN 0999943888. doi: 10. 10016/.comb.2219.02..0
[42] P.-. Kindermans, K. T. Schütt, M. Alber, K.-R. Müller, D. Erhan, B. Kim, and S. Dähne. Learning how to explain neural networks: PatternNet and PatternAtribttio. In ICLR 2018, Feb. 2018.
[43] S. Kucharsky, D. van Renswoude, M. Raijmakers, and I. Visser. WALD-EM: Wald accumulation for locations and durations of eye movements. Psychological Review, 128(4):667-689) 2021. ISSN 1939-1471. doi: 10. 00.007//ew0000292.
[44] M. Kümmerer and M. Bethge. State-of-the-Art in Human Scanpath Prediction. arXiv:2102. [csl, Feb. 2021.
[45] M. Kümmerer and M. Bethge. Predicting Visual Fixations. Annual Review of Vision Science, (1))269-291, Sept. 2023. ISSN 2344-4642, 2374-4650.
doi: 10.1 (0.1116//nnurevvvistt vision-120022-07777017 120822-072528.
[46] M. Kümmerer, Z. Bylinskii, T. Judd, A. Borji, L. Itti, F. Durant, A. Oliva, A. Torralba, and M. Bethge. MIT/Tuebingen Saliency Benchmark. tibbigg....
[47] M. Kümmerer, L. Theis, and M. Bethge. Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet. ICLR Workshop Track, 2015.
[48] M. Kümmeree, T. S. Wallis, and M. Bethge. Information theoorettthcreiccccc model comparison unifies salieney metrics. Proceedings of the National Academy of Sciences, 1 112(52):10054-166 Dec. 2015. ISSN 0027-8424, 1091 16490. doi: 10. 10.1073/pnas. 1510393112.
[49] M. Kümmerer, T. S. A. Wallis, I., A. Gatys, and M. Bethge. Understanding Low- and High-L Level Contributions to Fixation Prediction. In The IEEE International Conference on Computer Vision (ICCV). pages 4789-4798. 2017.
[50] M. Kümmerer, T. S. A. Wallis, and M. Bethge. Saliency Benchmarking Made Easy: Separating Models, Maps and Metrics. In Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, editors, Computer Vision ECCV 2018, Lecture Notes in Computer Science, pages 798-814. Springer International Publishing. 2018. ISBN 978-33-330-01270-0
[51] M. Kümmererr M. Bethge, and T. S. A. Wallis. DeepGaze III: Modeling free-viewing human scanpaths with deep lcarning. Journal of Vision, 22(5):7, Apr. 2022. ISSN 1534-7362, doi: 10.1 1671166...2....77
[52] M. Kümmeree, H. Khanuja, and M. Bethge. Modeling Saliency Dataset Bias, May 2025.
[53] M. F. Land and B. W. Tatler. Steering with the head: The visual strategy of a racing driver. Current Biology, 11155:1215-112 Aug. 2001. ISSN 0960-9822. doi: 10. (0000000000000000000000000
[54] O. Le Meur and A. Coutrot. Introducing context-depender and spatially variant viewing biases in saccadic models. Vision Research. 121:72- 84, Apr. 2016. ISSN 0042-6989, doi: 10. vistes/ivsses.2016...
13

---

[55] O. Le Meur and Z. Liu. Saccadic model of eye movements for free-viewing condition. Vision Research, 11::152-164, Nov. 2015. ISSN 0042-6989. doi: 1i.0016.viiees.2214.22...20
[56] A. Linardos, M. Kümmerer, O. Press, and M. Bethge. DeepGaze IE:: Calibrated Prediction in and Out-of-Domain for State-of-the-Art Salieney Modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12919-12928, 2021.
[57] S. Ling, S. Pratte, and F. Tong. Attention alters orientation processing in the human lateral geniculate nucleus. Nature Neuroscience, 18(4):49644)) 498, Apr. 2015, ISSN 1726.1726 doi: 10.1038/nn.366.
[58] C. Lu. C. Lu, T. Lange, J. Foerster. J. Clune, and Ha. The Al Scientist: Towards Fully Automated Open-Ender Scientific Discovery. Sept. 2024.
[59] N. model of local and global attention in natural scene viewing. PLOS Computational Biology, 16(12): e e1007880, Dec. 2020. ISSN 1553-7358. doi: 10.137 1.o33a/journal.pche 1007880.
M. Opper. S. Reich, L. Schwetlick, S. A. Seelig, and R. Engbert. A mathematical
[60] S. Mondal, Z. Yang, S. Ahn. D. Samaras, G. Zelinsky, and M. Hoai. Gazeformer: Scalable, Effective and Fast Prediction of Goal- horee-Die Human Attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1441-1450, 2023.
[61] R. D. Morey. Confidence Intervals from Normalized Data: A correction to Cousineau (2005). Tutorials in Quantitative Methods for Psychology, 4(2))61-66 64, Sept. 2008. ISSN 1913-4126. doi: 10.20982/tqmp.04.. p061.
[62] B. C. Motter. Saccadic momentum and attentive control V4 neurons during visual search. Journal of Vision, 18(11):1,, Oct. 2018. ISSN 1534-7362. doi: 10.1 1167/18.1111 1..16
[63] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library, Dec. 2019.
[64] J. B. Pelz and R. Canosa. Oculomotor behavior and perceptual strategies in complex tasks. Vision Research, 41(25):5877 (1996 Nov. 2001. ISSN 0042-6989. doi: (101////0002-668900100000000
[65] K. Rayner. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3):372-422 1998. ISSN 1939-1455, doi: 0103770033-299.124..3
[66] Reback, jbrockmendel. W. McKinney, J. den Bossche, Augspurger, Cloud, S. Hawkins, M. Roeschke, gfyoung, Sinhrks, A. Klein, Hoefler, Petersen, J. Tratner, C. She, W. Ayd, Naveh, J. H. M. Darbyshire, M. Garcia, R. Shadrach, J. Schendel, A. Hayden, D. Saxton, M. E. Gorelli, F. Li, M. Zeitlin, V. Jancuskas,, A. McMaster, P. Battiston, and S. Seabold. Pandas-dev/pandas: Pandas 1.4.1. Zenodo, Feb. 2022.
[67] L. O. M. Rothkegel. H. H. Schütt, H. A. Trukenbrod, F. A. Wichmann, and R. Engbert. Searchers adjust their eye-movement dynamics target characteristies in natural scenes. Scientific Reports, 9(1):1635, Feb. 2019. ISSN 2045-2322. doi: 10. 51888444158--08--755555
[68] H. Schütt, L. M. Rothkegel. H. A. Trukenbrod. S. Reich, A. Wichmann. and R. Engbert. Likelihood- based parameter estimation and comparison of dynamical cognitive models. Psychological Review, 124(4); 505-524, 2017. ISSN 1939-1471, 0033-295X. doi: 10. 1037/rev00000068..
[69] H. H. Schütt, I.. O. M. Rothkegel. H. A. Trukenbrod, R. Engbert, and F. A. Wichmann. Disentangling bottom-up versus top-down and low-level versus high-level influences eye movements over time. Journal Vision. 19(3):1-1, Mar. 2019. ISSN 1534-7362. doi: 10.11.7/1..31.
[70] L. Schwetlick, L. O. M. Rothkegel, H. A. Trukenbrod, and R. Engbert. Modeling the effects of perisaccadic attention gaze statistics during scene viewing. Communications Biology, 3(1):-111, Dec. 2020. ISSN 2399-3642, doi: 10. 1033//420003000-0042222
[71] L. Schwetlick, D. Backhaus, and R. Engbert. dynamical scan-path model for task-dependence during scene viewing. Psychological Review, 130(3):807-840, Apr. 2023. ISSN 1939-1471, 0033-295X, doi: 10. 10.1037/re00000379
[72] L. Schwetlick, M. Kümmerer, M. Bethge, and R. Engbert. Potsdam data set of eye movement on natural scenes (DAEMONS). Frontiers in Psychology, 15, May 2024, ISSN 1389609.
doi: 10.3389//ssy..224..
14

---

[73] L. N. Smith and N. Topin. Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates, May 2018.
[74] T.J. Smith and J. M. Henderson. Facilitation of return during scene viewing. Visual Cognition, 17(6-7): 1083-1108, Aug. 2009. ISSN 1350-6285. doi: 00.1000/13502280006675533
[75] T. J. Smith and J. M. Henderson, Looking back at Waldo: Oculomotor inhibition return does not prevent return fixations. Journal of Vision, 11(1):3 Jan. 2011. ISSN 1534-7362. doi: 10.1167/11.133
[76] S. S. Stevens. On the Theory of Scales of Measurement. Science, 103(26664)6677-680 June 1946. ISSN 0036-8075, 1095-9203, doi: 10. 126/scecce. 10r.ee.......... 10.. 2684.677.
[77] X. Sui, Y. Fang. H. Zhu, S. Wang, and Z. Wang. ScanDMM: A Deep Markov Model of Scanpath Prediction for 360deg Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6989-6999, 2023.
[78] W. Sun, Z. Chen, and F. Wu. Visual Scanpath Prediction using IOR-ROT Recurrent Mixture Density Network. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2019. ISSN 1939-3539, doi: 10.T..// 299.99............
[79] X. Sun, H. Yao, R. Ji, and X.-M. Liu. Toward Statistical Modeling of Saceadic Mov-move and Visual Saliency. IFFE Transactions Image Processing, 23(1 (6644946694466,, Nov. 2014. ISSN 1941-0042. doi:
10. 1110/TTP.201..233777
[80] B. W. Tatler, M. M. Hayhoe, Land, and D. H. Ballard. Eye guidance in natural vision: Reinterpreting salience. Journal of Vision, 11(555-5, May 2011. ISSN 1534-7362. doi: 10. 10.1167/1..555
[81] B. W. Tatler, J. R. Brockmole, and R. H. S. Carpenter: LATEST: A model of saccadic decisions in space and time. Psychological Review, 124(3):267-300, 2017. ISSN 1939-1471, 0033-295X. doi: 10. 1000000/
[82] I. M. Thornton, Q. C. Vuong, and K. S. Pilz. A Search Advantage for Horizontal Targets in Dynamic Displays. i-Perception, (222:2041669521100400 Apr. 2021. ISSN 2041-6695, doi: 00.1177/0416665211
[83] Y. Tsal. Attending to horizontal, diagonal, and vertical positions in space. Bulletin of the Psychonomic Society, 27(2)) 134, Feb. 1989. ISSN 0090-5054. doi: 10.3558/B003329900.
[84] P. Virtanen, R. Gommers,
Oliphant, M. Haberland, T. Reddy, D. Coumapeau, F. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, 1. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde. J. Perktold, R. Cimrman, Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro. F. Pedregosa, and P. van Mulbregt. SciPy 1.0: Fundamental algorithms for scientific computing in Python. (138//415992-11 010-0099---6966
Nature Methods, pages 12, Feb. 2020. ISSN 1548-7105. doi:
[85] Z. Wang and E. P. Simoncelli. Maximum differentiation (MAD) competition: A methodology for comparing computational models of perceptual quantities. Journal of Vision 8(12):8, Sept. 2008. ISSN 1534-7362. doi: 10.1 16.71........
[86] M. Waskom. Seaborn: Statistical data visualization. Journal of Open Source Software, 6(60))302,, Apr. 2021. ISSN 2475-9066. doi: 10.21105/joss.0300 03001.
[87] D. A. Wilkenfeld. Understanding compression. Philosophical Studies, 176(10):2007-281) Oct. 2019. ISSN 1573-0883, doi: 10.1007/sl
10900///111111111 1098-019 01-8-1188111 1152-
[88] N. Wilming, S. Harst, N. Schmidt, and P. König. Saccadic Momentum and Facilitation of Return Saccades Contribute to an Optimal Foraging Strategy. PLOS Computational Biology, 1002871, Jan. 2013. ISSN 1358-7358. doi: 10.1371/jjummal.pcl pcb.il00871...... 1002871.
[89] C. Wloka, I. Kotseruba, and J. K. Tsotsos. Active Fixation Control to Predict Saccade Sequences. In 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3184-318 3193, Salt Lake City, UT, USA, June 2018. IEEE, ISBN 978- 1--1---566662205 6640099 doi:
[90] Y. Xu, Y. Dong, J. Wu, Z. Sun, Z. Shi, J. Yu, and S. Gao. Gaze Prediction in Dynamic 360 Immersive Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5333-5342, 2018.
[91] Z. Yang. L. Huang. Y. Chen. Z. Wei, S. Ahn, G. Zelinsky, D. Samaras, and M. Hoai. Predicting Goal- Directed Human Attention Using Inverse Reinforcement Learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 193-202, 202, 2020.
15

---

[92] Z. Yang. S. Mondal, S. Ahn, R. Xue. G. Zelinsky, M. Hoai, and D. Samaras. Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers. In Proceedings of the IEEE/CVE Conference Computer Vision and Puttern Recognition, pages 1683-1693, 2024.
[93] A. L. Yarbus. Eye Movements and Vision. Plenum Press, 1967.
[94] D. Zanca, S. Melacci, and M. Gori. Gravitational Laws of Focus of Attention. IEEE Transactions on Puttern Analysis and Machine Intelligenee pages 1-1, 2019. ISSN 0162-8828, doi: 10. 10.100//PAM..211..222 201...9206666.
[95] R. S. Zimmermann, J. Borowski, R. Geirhos, M. Bethge, T. Wallis, and W. Brendel. How Well do Feature Visualizations Support Causal Understanding of CNN Activations? In Advances in Neural Information Processing Systems, volume 34, pages 11730-11744, Curran Associates, Inc., 2021.
16

---

A Comparative evaluation on more models and datasets
We evaluated our models on four datasets: MITIO03 [38.. DAEMONS 722,. COCO Freeview [88 and the Potsdam Corpus [69]. On all four datasets, we construct common spatial baseline model and then train DeepGaze as well the different SceneWalk variants using the spatial baseline model. For better comparison, we also evaluate the Constrained Levy Exploration model [CLE; G and the model of Kadner al., 2023 40) For all models we evaluate information gain relative the uniform
baseline (LL) and AUC 44 We find (Table 1 that on all datasets, SceneWalk- X closes substantial part of the performance gap between the perisaccadic Scene and DeepGaze, both LL and in AUC.
As note, temperature scaling has only a very small effect in AUC: this is to be expected, AUC is invariant to temperature changes. The marginal score increase is likely an indirect effect, as having temperature free parameter during training gives the optimizer more flexibility, allowing the other model parameters to converge to slightly better solution.
Model
MITI003 AUC
DAEMONS AUC
COCO Freeview AUC
Potsdam Corpus
LL
LL
LL
LL
AUC
Spatial Baseline
2.52
0,9223
1.88
0.8743
2.08
0,9034
1.81
0.8768
Kadner et al. 2023 40
0,9237
0.8806
0.9113
CLE (Boccignone al. 2004, 6
2.18
0,9240
2.42
0.9118
2.34
0,9212
SceneWalk (perisaccadic)
2,66
0,9304
2.71
0.9207
2.53
0,9240
2.32
0.9129
temp. scaling
2.72
0,9305
2.73
0,9209
2.54
0.9241
2.33
0,9130
momentum
2.77
0,9321
2.82
0.9235
2.58
0.9254
2.38
0.9147
card. bias WalkeWal--X
2.82
0,9343
2.88
0.9267
2.64
0,9275
2.42
0.9164
DeepGaze
2.94
0.9386
3.18
0.9365
2.87
0,9343
2.54
0,9194
Table I: Performance of our models and other previous scanpath models on MIT1003, DAEMONS, COCO-Freeview and the Potsdam Scene Viewing Corpus
B
Implementation details of the new mechanisms
B.1 Time dependent temperature scaling
To capture the observation that early fixations tend to be more strongly biased toward salient image regions, we introduce a time-dependent temperature scaling mechanism into the SceneWalk model. This mechanism dynamically adjusts the exponents used shape the attention and inhibition maps during scanpath generation based on the fixation index. The original Scene model already applies fixed exponent to these maps to regulate their sharpness, acting similarly to a softmax temperature. Our extension generalizes this by making the exponents decay over time, allowing for sharper attention selection early the trial and broader selection later.
Concretely, in the original Scene model, the priority map (t) at each location computed as subtractive combination of normalized, exponentiated attention Aig and inhibition (t)(t) maps:
(F)) (Ag)(t)) Cp Em(FR((t))* Usg (t) (m(An(()))
Here, Cr is weight parameter for inhibition, and, importantly, > controls the sharpness of both the attention and inhibition distributions and was constant across fixations.
We replace the constant 7 with fixation- iirtoon-innxx-eee madex-dependem value. Specifically, the exponent y, for the nith fixation given by an exponential decay from an initial high value 7 toward base value Thase:
7/n Tbase + (> Tbase) (a-1))
(2)
Note that here the fixation index in the scanpath n follows a I b-baeed indexing.
The new trainable parameters we introduce with this formulation are:
1. Tase: minimum exponent for inhibition
2. dy; decay rate for
17

---

This enables the model to express strong preference for high-saliency targets early on (large %n) and gradually reduce this preference (lower 7n) over time, converging Thase.
B.2 Saccadic momentum
To better capture directional tendencies that DeepGaze can capture in human caanpahs--specifcee the tendency continue in the same direction (momentum) and to return previously attended regions (return saccades) we augment the SceneWalk Walk model with dynamic oculomotor bias map Mare. The map is computed over all possible gaze targets each fixation, and combines two angular Gaussian components: A momentum component favoring forward continuation and a return component favoring 180" reversals. These components blended using a dynamic weighting function, which depends both on the current fixation index and the length of the preceding saccade.
Specifically, let
n be the current fixation index (with 1-based indexing),
s be the length of the previous saccade, in degrees of visual angle,
Byy be the angular difference between the current fixation and candidate saccade directions, normalized between and π.
Then define the fixation index weight
Winx
(3)
exp
n-His
and the saccade length weight as:
Wlen
+ exp
Salatt
Sucde
The fixation index weight and the saccade length weight are then multiplied get the combined direction weight:
Wdr
Wix
Wlen
(5)
We compute the angular preference fields
02
Gromennum (Bay) exp
2o Trum
(6)
(0.. )²)²
(_(0p) + m)²
(0i;
Greum (0ig) exp
+ exp
2n feturn
2n feturn
These are then combined as follows, using the combined direction weight:
M dir (1 Wdir) Greturn (0:j) Widir Gmomentum (0:z).
Finally, the map is combined with Salk's priority map u.j limal before saccade selection:
final 1ag
(wbins
Mdir)
funl
(8)
The new trainable parameters introduce with this mechanism are:
1. mmom: width of the angular Gaussian for saccadic momentum (same-direction preference)
2. Oretum: width of the angular Gaussian for return saccades (opposite (opposite-direction preference)
3. Mixx fixation index midpoint for the sigmoid controlling the momentum-retum tradcoff
4. drxx steepness of the sigmoid over fixation index
5. Sshont midpoint of the sigmoid controlling sensitivity to saccade length
6. Swate: scale parameter of the sigmoid over saceade length
7. Whiss: scaling factor controlling the influence of the directional prior in the final priority map.
18

---

B.3 Cardinal attention bias
Following comparisons with DeepGaze and consulting the literature, capture anisotropies in visual attention and oculomotor behavior we introduce a mechanism that biases Scene Walk's 's attention map toward cardinal (horizontal and vertical) saccade directions. Additionally, we model leftward bias for early fixations. These components are combined into single potential map that modulates the attention map before normalized and combined with inhibition. As for saceadic momentum and temperature, the mechanism is dependent on fixation index.
Let:
n be the current fixation index (with 1-based indexing),
Oig be the angular difference between the current fixation and candidate saccade target,
di be the Fuclidean distance of candidate saccade target to the current fixation.
Then we formulate the cardinal bias as:
dij
Quardinal cos(40l)) exp
where 7 x is a decay constant.
We further define the vertical/horrizontal strength modulation as scalar that increases fixations:
(n)) = 1 + (Aiven 1) -0 Pn),
(10)
and the leftward bias as:
(1)) = (Oigh -(n-11)
(11)
The biases are then applied as follows. First, for the horizontal/vertical bias, let m.y = location (i.j) lies the vertical direction relative to the current fixation (i.c., |dx| < [dyl), otherwise m, The vertical bias is applied
guardinal <-
Quardinal
[mpg (n)r + (1 m, m 1]
(12)
Similarly, for the leftward bias, we define binary masks for horizontal directions:
mlfi 1 if dx < 0 and [dx] Idyl. 0 otherwise;
mij right if dr 0 and |dx| |dyl. otherwise;
and then apply:
Q cardinal
Q
cardinal
lell mig
(n) + mi
+ (1 - mij
mm
right
1
(13)
Onen (n)
Finally, the resulting potential map is normalized to sum to 1, and applied multiplicatively to the attention map:
Aij (1 + Wmix (uardinal)
A
mod
(14)
Z
The new trainable parameters we introduce with this mechanism are:
1. x= controls the spatial decay rate of the cardinal potential with distance
2. Olyert maximum vertical vs horizontal bias ratio
3. pt growth rate vertical/horizootal anisotropy over fixations
4. alen initial strength of the leftward bias
5. A: decay of the leftward bias fixations
6. Wmix: global scaling factor for the oculomotor potential's influence on the attention map.
19

---

b
earty
0.4
prev: saccade deg
late saccade
prev. saccade deg
frequency
DG3
frequency
DG3 wio saliency
wio salienry ScaneWalk
0.2
0.1
0.1
0.0
back
right
back
back
right
back
saccade direction change
saccade direction change
Figure 7: relative saccade direction distribution predicted by DeepGaze, DeepGaze without saliency maps and Scene Walk for early and late saccades (a) as well as after short or long previous saccades (b).
C Spatial baseline
Commonly, empirical fixation densities are estimated with Gaussian kernel density estimates. How- ever, recent progress in spatial saliency modeling [52] has produced models which predict fixations from new observers better than the empirical density. While this feels counterintuitive, it is possible
because saliency datasets typically have data from only 10-20 subjects per image [8 g ) Subopti- mal saliency maps could lead to confounders in our study, e.g., models might use previous fixation
locations simply for estimating a better saliency map. Hence estimate the average fixation density for each subject on an image as a mixture four distributions: first, KDE over the fixations from all other subjects, secondly the centerbias, i.c. KDE over the fixations from all other images, thirdly uniform component, and in addition as fourth component we add the predicted fixation density from the state-of-the-art saliency model [52]. The last component guarantees that the spatial baseline model at least as good as the used saliency model. For a discussion of the other three components, see 45
extend the approach by [52 and
The parameters of the model, i.e. bandwidth and mixture weights, are optimized individually on each image for maximum likelihood. This results in prediction for each subject, using only other subject's 's fixations. To compute the final fixation density for an image, we average over the predicted fixation densities for all subjects with weights that are location dependent: for each true fixation location, only the components not using this fixation their KDE are switched on. This defines weighting coefficients for all fixated pixels such that the prediction for that pixel only comes from subjects which didn't fixate the pixel. For all other pixels, the weights are interpolated linearly, and the resulting spatial average is re- re-normalized. This approach removes visual artifacts due to single, potentially random fixations and gives more interpretable results.
D Details about the datasets
The MIT1003 dataset is accessed via the pysaliency python library htttp://githhr http///tthhas-k/yyaallleee
DAEMONS and COCO Freeview come with an off- using. MITI003 and the Potsdam Cor- pus do not have official validation splits. Instead, we use 10-fold crossvalidation us- ing pysaliency. filter dateredattttss..t train, validation, test folffol((( test_fold (stimull,,, fixations, crossval [rossval_folds=10 folds=10, test folds=flds=1, val fol_folds=1). On MIT1003, the validation split has been used for tuning model hyperparameters and learning rate schedules, which have then also been used on the other datasets. All reported evaluations use the test split on MIT1003 and the Potsdam Corpus, and the official validation split for DAEMONS and COCO Freeview.
cial training/validation split which
E Details about models and training
DeepGaze DeepGaze III was originally trained using four stage training paradigm including spatial pretraining different dataset, spatial training the target dataset, scanpath training
20

---

the target dataset and final joint training phase. This procedure was necessary for not overfitting the spatial priority network while training the scanpath parts. Since we removed the spatial priority network, we only need the scanpath training phase. Since DeepGaze III was originally trained on MIT1003 dataset, which we are also using, we can apply exactly the same hyperparameters: DeepGaze is trained using the Adam optimizer with a batchsize of 4 and an initial learning rate of 0.001. In addition, we use a learning rate schedule consisting decays of the learning rate by factor of 10 after 10, 20, 30 and 31 epochs. After the 32nd epoch, training is stopped. Consistent with the original training, we also use 10-fold crossvalidation. For all results this paper use for each image the model weights which have not this image in training.
SceleWalk Originally, SceneWalk was trained fully Bayesian using MCMC. Here, we instead re-implemented the model in Jax and train it using MAP estimation, resulting in substantially faster fitting times. Scene Walk trained via gradient descent with the Adam optimizer Parameter prior ranges are chosen to be broad, and taken mostly from Schwetlick et al. [70]. The hyperparameter configuration for the base perisaccadic model is also taken from Schwetlick al. [70] We use "OneCycle" learning rate schedule [73] over 30 epochs for the MITIO0 1003 dataset, and 50 epochs for DAEMONS, with an initial learning rate 0.001, peak learning rate of 0.01 and a batch size of 32.
MAP objective.
Error bars
All reported error bars bootstraped 95% confidence intervals for the mean log- logelikeldoood per image using the normalization method of Cousineau (20] for paired comparisons with the correction of Morey [61].
G Assets
DeepGaze is implemented in python using pytorch 663
using the official implementation from SceneWalk was reimplemented in jax 99| (Apache 2 li-
github. com/mat chias-k/Deeppazee
cense) based on the official implementation from gi github. www.///wwwoocooiooeeooooocc Swwee/alleeeeoeeeeeeeeeeee Model
Model evaluations and saliency metrics were using the public pysaliency toolbox (github com/
matthias- x/pysaliency MIT license). Also used were scipy [84] and numpy [32] for computa- tions, pandas [66] for statistics and data handling as well matplotlib [35] and seaborn [86
for plotting.
H Code Availability
The Scene Walk-X X implementation in jax, along with code to reproduce our main findings can be found at httpt:tt/ttttttiae com/bethgelab/ wwwaaattwwaaaaaaaeeaaeaeeaaaaaaa wwt----ooveeeeeeeee
I Compute Resources
All main experiments were conducted on 2080Ti GPUs. Training the DeepGaze model took about days for all 10 splits and was done four times until the final model setup evolved. Training Scene Walk took about 30 minutes, while developing the mechanisms we trained about 50 model iterations. DAEMONS, we used A100 GPUs and DeepGaze took about 4 days to train, and each SceneWalk model about two hours. Overall, we used about 18 days of compute.
Controversial Fixations
Here we show the full set controversial fixations that we inspected and classified as basis for our mechanistic extensions. We examined a total of 72 fixations, defined the top 6 fixations from the top images in both conditions (LLD and WLLD). For all controversial fixations, we show predictions from DeepGaze, SceneWalk and all mechanistic extensions of Scene Walk, including the fully extended SceneWalk-X model.
Of the four categories we detail in Figure
2 found:
21

---

Long saccades: 17 clear, 3 plausible
Short saccades: 8 clear, I plausible
Continuing: 13 clear, 2 plausible
Early fixations: 30 clear
66 of 72 fixations (92%) could be clearly assigned of the four categories, and the remaining 6 (8%) still plausibly belonged to a known type. We see this as a strong indicator that the analysis indeed revealed failure cases that are not idiosyncratic, but rather fall into meaningful, repeatable patterns that are interpretable and mechanistically actionable.
Englnne
Figure Controversial Fixations selected for maximum difference in log-likelihood between DeepGaze and SceneWalk, Part 1. We first select the top images with mean log-likelihood dif- ference and then show for each image the 6 fixations with maximum log- lileeiildoood difference. We visualize the image, the empirical densities and the predictions of DeepGaze, SceneWalk and all extension versions that we introduce. On top of each prediction, we show the log- ligelioel of how well the fixation predicted. For the DeepGaze prediction, between DeepGaze and Scene that resulted in this fixation being picked as controversial. On top of the image show the mean score difference that resulted in this image being picked.
additionally show the score difference
22

---

LGS
Figure Controversial Fixations selected for maximum difference in log-likelihood between DeepGaze and SclneWalk, Part 2.
Ery Draly
RADIO
17720
SETARO prreet
TIENE
Figure 10: Controversial Fixations selected for maximum difference in eoelikklihood between DeepGaze and Scene Walk, Part 3.
23

---

Per Caaryy
C
I8MN
1880
KI1PP goranic
40 ME
1030
Figure 11: Controversial Fixations selected for maximum difference in log-likellnn ligliiklihhodd between DeepGaze and SceneWall, Part 4.
e 286 1
TATARO (onntr
Figure 12: Controversial Fixations selected for maximum difference in likelkkli between DeepGaze and SceneWalk Part 5.
24

---

36 Sity
Figure 13: Controversial Fixations selected for maximum difference in ligliiklihood between DeepGaze and SceneWall, Part 6.
EST MUT
LB abbeiii
12
LA Bih C.ccaaay
STANCO
Figure 14: Controversial Fixations selected for maximum difference in log- lieliiel between DeepGaze and Scene weighted by the probability assigned by DeepGaze, Part 1.
25

---

ESC MATx
Figure 15: Controversial Fixations selected for maximum difference in log-likon ligliiil between DeepGaze and SceneWalk weighted by the probability assigned by DeepGaze, Part 2.
1.2
THE ELSCAINYS
LYAN
Figure 16: Controversial Fixations selected for maximum difference in logllikelihood between DeepGaze and Scene weighted by the probability assigned by DeepGaze, Part 3.
26

---

LB situ Stamroryy
Figure 17: Controversial Fixations selected for maximum difference in log-likell ligeliil between DeepGaze and Scene Walk weighted by the probability assigned by DeepGaze, Part 4.
Deg Dvety
PecTe 1.33
A
Lia seth Measaays
Figure 18: Controversial Fixations selected for maximum difference in log- logllikel between DeepGaze and Scene weighted by the probability assigned by DeepGaze, Part 5.
27

---

75 15..C
6
63,
ca,
LG www.etiooo wot.rtaaarii
18:0%
18.0%
$2
69,
6
63,
63,
10
12.20
1805
+
Figure 19: Controversial Fixations selected for maximum difference in log- log-likelihood between DeepGaze and Saek weighted by the probability assigned by DeepGaze, Part
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims in the abstract and the introduction supported by the results in the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work as far as applicable already in the section on the new mechanisms, and more broadly in the discussion including a specific section on limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper doesn't 't contain any theoretical results.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We specify details about used dataset splits and training methods for our models in the appendix, as well as the used metrics and selection methods. The base models we're using are public models, so there is no need to specify details about the architecture. We specify details for the new mechanisms we have implemented in Walk-XX in the appendix.
28

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: httpa////wwrao.o/.oooooo////aaaa -heetatt---moo--------e-eeeee-ee Experimental setting/details
6.
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We specify details about used dataset splits and training methods for our models in the Appendix, well as the used metrics and selection methods. The models we'ne re using are public models, so there is no need to specify details about the architecture.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We display error bars for all our results and give details about their computation in the Appendix.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We detail the used resources in the Appendix and include estimate of the total compute needed.
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwrrppppppppppteeeeeeeee
Answer: [Yes]
Justification: We made sure to adhere to the NeurlPS code of ethics in each step of our experiments.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss positive and negative broader impact of our research in the discus- sion
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We do not release any data a Responsible AI Licence (RAIL, licenses. ai) i to prohibit misuse.
and our models will be released under
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
29

---

Answer: [Yes]
Justification: All authors of used assets are referenced in the paper and licenses, as far as specified, are mentioned and respected. However, for some software libraries couldn's find license details.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce new assets. We will make model and checkpoints available in the future.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with humans subjects. We only use previously published datasets.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with humans subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: This paper does not use LLMs core part of the research.
30


Reviews:
[{"review_id": "T9Abgb6WaK", "reviewer": "Reviewer_3tRx", "summary": "The authors investigate how a high-performing deep learning model (DeepGaze III) can be used to improve a more interpretable one (SceneWalk) for human gaze fixation prediction. The authors compare the predictions of DeepGaze III and SceneWalk on empirical gaze data, identify discrepancies between them, and use insights from this analysis to revise the SceneWalk model. The proposed modifications, such as incorporating saccadic momentum and time-dependent temperature scaling, lead to notable performance improvements, bringing SceneWalk closer to DeepGaze in predictive power.", "questions": "* How exactly is the data split for the cross-validation? Is it only split across fixations? Or are some of the stimuli and/or participants held-out as well?\n* In the controversial stimuli analysis, it would help to have a quantitative analysis of the different types of prediction differences between the models.\n* Overall, is there a way to make the method more systematic and less manual? The proposed method would be more impactful if it would be applicable to more models and datasets.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper presents an interesting idea: leveraging a highly-performing but difficult to interpret model to guide improvements to a more interpretable one. However, several aspects of the execution limit the paper’s impact and clarity.\n* It is unclear why the use of a generative model like DeepGaze is useful to improve a more interpretable model like SceneWalk, beyond what could be achieved from the empirical data alone. The authors assert that this type of analysis is difficult with empirical data but provide no quantitative result to support this claim.\n* The authors mention that the modifications to SceneWalk were already known in the literature. If these mechanisms were previously hypothesized, it is unclear why they had not been implemented in SceneWalk before. The paper would benefit from clarifying how their approach identifies these mechanisms in a more data-driven or systematic way, rather than simply reintroducing known components.\n* Another unclear point is why the authors don't consider DeepGaze as a mechanistic model. While it is indeed more difficult to interpret a deep learning model, it does not necessarily mean that it doesn’t capture mechanisms of the human visual system. The main difference between DeepGaze and SceneWalk seems more to be in how easily interpretable they are.\n* The authors mention that “these controversial fixations can be classified into few categories”. A quantitative analysis of these categories of prediction differences would be that claim stronger.\n* The cross-validation method is crucial to evaluate the risk of overfitting in the results, but it is only briefly mentioned in the appendix and lacks detail.\n* Overall, the approach presented by the authors is interesting but lacks systematicity and generality. Making the method more general and less reliant on manual analysis and insight would make it more impactful.", "was_revised": true, "final_justification": "Thank you to the authors for clarifying my questions and providing more quantitative analyses. I raised my rating accordingly.", "rebuttal": "We thank you for providing a thoughtful review of our work. \nWe appreciate your positive assessment of the core idea and the constructive feedback: We agree that certain aspects of our methodology and contributions require clearer exposition. Your comments helped us identify clear areas for improvement in the manuscript.\n\n### [Weakness] On the use of DeepGaze III vs analyzing empirical data directly\n\nThe central difficulty lies in the structure of scanpath prediction itself. In spatial saliency modelling, it is common to compare model predictions with empirical fixation scatter plots or KDE estimates of the underlying ground truth distribution for finding the largest remaining model errors. This works well because for each image, we typically have hundreds of fixations and can form a relatively stable nonparametric estimate of the underlying spatial distribution $p(x, y)$.\n\nIn scanpath prediction, the modelled distribution is more complex. Each fixation is assumed to come from a conditional distribution $p(x_i, y_i \\mid  x_0, y_0, \\dots, x_{i-1}, y_{i-1})$ which depends on all previous scanpath fixations. Because scanpaths are highly individual, the same fixation history is never observed more than once in practice. This means we typically have one sample per conditional distribution, making a KDE impossible. One could instead try to estimate the whole function $p(x_i, y_i \\mid  x_0, y_0, \\dots, x_{i-1}, y_{i-1})$ nonparametrically, e.g. with Gaussian Processes. But for typically 10 fixations in a scanpath (on MIT1003), this is a function of already 2x10=20 variables, which we would need to fit given only about ~100 datapoints (fixations) for an image. This very sparse coverage of the relevant input space is further complicated by the varying lengths of scanpath histories. In such a setting, there is no straightforward empirical baseline that allows us to say whether a given fixation is surprising or expected given prior context.\n\nHere is where DG becomes indispensable. As a high-performing, sequence-aware model trained on large-scale scanpath data, it provides a dense, queryable approximation of the true distribution $p(x_i, y_i | ...)$, allowing us to go beyond what the empirical data alone permits. For example, in Figure 2 of the paper, we highlight controversial fixations where SceneWalk performs poorly compared to DG. Without DG’s predictions as a proxy distribution, we would be left with only a single empirical datapoint to assess SW’s failure, without knowing whether that fixation was actually predictable from context.\n\n**Planned revisions:** We will update paragraphs in the introduction and Section 3 to more explicitly state this advantage, and better explain why inter-observer consistency cannot be estimated in the usual ways. We will emphasize that using a model as a \"queryable stand-in\" (as mentioned on lines 105-106) allows for controlled experiments on the data distribution itself, providing unique insights that guide the search for new mechanisms.\n\n### [Weakness] On the novelty of the added mechanisms\n\nYou correctly note that the effects we find (e.g., saccadic momentum) have been discussed in prior literature. However, the literature contains a myriad of reported effects that will influence fixation selection: oculomotor properties (including momentum, but also hypometric saccades, corrective saccades and saccadic adaptation), the covert attention system (pre-saccadic attention shifts, retinotopical attentional traces, spatial attention biases, attention dynamics, inhibition of return with all the different sub types and temporal dynamics, different kinds of feature attention, belief cuing,...), foveal remapping and remapping errors, different memory systems and their decay times, anatomical findings on circuit timings (e.g., cortex–colliculus loops), and cognitive influences like exploration–exploitation trade-offs, coarse-to-fine strategies or task structure.\n\nImplementing all these effects would be infeasible, even more so, because many of them are just that: effects, not yet plausibly constructed mechanisms. It is necessary to decide which of these effects might be most promising to address next. Historically, scientific intuition was used for that. Here, our approach goes one step further and provides a principled way to select those effects that seem to result in the largest prediction errors — not on datasets that have been carefully designed to maximally evoke a certain hypothesized effect, as it is often the case in the literature, but on large datasets of free-viewing behaviour on natural scenes.\n\nThis is the key contribution of our work: the use of prediction disagreement on unconstrained, ecologically valid data as a systematic way to triage candidate mechanisms for inclusion in interpretable models. While the specific components we implement may not be new in themselves, our method provides a quantitative, model-based framework for assessing their individual contributions, revealing, in some cases, subtleties such as time dependencies or asymmetries in the effect shapes that had not previously been reported or appreciated.\n\n**Planned revisions:** We will revise our Methods and Introduction sections to emphasize the systematic nature of our framework. In particular, we will clarify how prediction disagreement is used to prioritize which mechanisms to implement, and how this represents a principled alternative to intuition- or experiment-driven model construction.\n\n### [Weakness] On deep vs mechanistic models\n\nThis appears to be primarily a wording issue, and we are happy to clarify our intent. Our use of the term “mechanistic model” follows a long tradition in psychology, neuroscience, and cognitive science, where “mechanistic” typically refers not just to models that implement mechanisms in some abstract sense, but to those that are explicitly constructed from interpretable components aligned with hypothesized cognitive or neural processes. However, we agree that the concept of mechanistic models in machine learning is seen as broader. We will detail the range of mechanistic models and will make clear where our work stands within that range.\n\nFor a more detailed answer, please see our response to reviewer i2u5.\n\n### [Weakness + Question] On the classification of controversial fixation\n\nFor our controversial fixation analysis, we inspected a total of 72 fixations, defined as the top 6 fixations from the top 6 images in both conditions (LLD and WLLD), all shown in Appendix H. Of the four categories in Figure 2, we found:\n\n* Long saccades: 17 clear, 3 plausible\n* Short saccades: 8 clear, 1 plausible\n* Continuing: 13 clear, 2 plausible\n* Early fixations: 30 clear\n\n66 out of 72 fixations (92%) could be clearly assigned to one of the four categories, and the remaining 6 (8%) still plausibly belonged to a known type. We see this as a strong indicator that the analysis indeed reveals failure cases that are not idiosyncratic, but rather fall into meaningful, repeatable patterns that are interpretable and mechanistically actionable.\n\n**Planned revision:** We will include a short quantitative summary in the main text (Section 5.1) and have a full breakdown and classification details in Appendix H.\n\n### [Weakness + Question] On Cross-validation details\n\nAll evaluations on MIT1003 (both for DeepGaze III and for the various SceneWalk versions) are conducted using 10-fold cross-validation *across images*. That is, images are partitioned into 10 folds, and models are always evaluated on images that were not seen during training. Both the stimuli and the associated fixations from all observers are held out during testing. This is consistent with standard practice in the saliency and scanpath prediction literature, where cross-validation across images is the norm (e.g., as done in the MIT/Tuebingen saliency benchmark), since images typically provide the strongest source of variation. Additionally, with the DAEMONS (and now Potsdam) datasets, we also test our results on completely held-out data.\n\n**Planned revisions:** We will revise the paper to make this clearer, not only in the appendix but also the main text, space permitting.\n\n### [Weakness + Question] On “sistematicity and generality”\n\nWe thank the reviewer for highlighting this important and ambitious direction. We fully agree that increasing the level of automation and generality would enhance the power and applicability of the approach.\n\nAt the current stage, we intentionally chose a human-in-the-loop strategy to validate our methodology in a transparent, end-to-end fashion. Our goal was to demonstrate that targeted comparisons between a performance-driven model and a highly-interpretable one can surface actionable, theory-grounded mechanisms that measurably improve prediction. We see this as a necessary proof of concept before systematizing the pipeline further.\n\nThat said, we strongly share the reviewer’s vision. Automating the identification and characterization of controversial fixations (e.g., via large vision-language models) holds clear promise. We believe approaches like VisDIFF point in the right direction, but still need to mature before fully supporting this type of model-guided scientific discovery. For now, a manual loop ensures robustness and clarity while paving the way toward a more scalable framework.\n\n**Planned revisions:** We will revise the discussion to emphasize this outlook, framing the current work as a first step toward a generalizable, semi-automated modeling pipeline, ultimately aiming for what could be seen as an “automated scientist” for behavioral modeling.\n\n---\n\nWe sincerely thank the reviewer for their thoughtful and detailed feedback. We have taken your concerns seriously and hope that our clarifications and planned revisions make the contributions of the paper clearer. If our responses addressed your doubts, we would be grateful if you considered updating your evaluation accordingly."}, {"review_id": "7zeKBlOhbp", "reviewer": "Reviewer_DPB9", "summary": "In this manuscript the authors improve the mechanistic SceneWalk model which predicts scan paths of humans free viewing natural scenes. They base these improvements on comparisons to DeepGaze III, a deep learning model for the same task, which yields “controversial fixations”, I.e. fixations where the deep learning model still performs much better than the mechanistic one. Concretely they add time dependent temperature scaling to make the predictions higher entropy later in the trial, saccadic momentum to implement return saccades and the tendency to continue moving in the same direction, and an attention bias towards the cardinal directions and horizontal directions in particular. Overall these additions improve the prediction quality of the SceneWalk model by a substantial 0.2 bits/fixation on the DAEMONS and MIT1003 data sets.", "questions": "The weighted log-likelihood difference is an unusual measure for the difference between predictions. Why do the authors prefer it over a more classical measure like the KL-divergence?\n\nWouldn’t it be interesting to evaluate the model additions on the data used originally to develop the SceneWalk model? They are available (https://osf.io/n3byq/)", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Overall, this is solid work. The authors clearly improve the model by comparing to deep learning predictions and also implement the SceneWalk model in JAX which doubtlessly will improve speed, too. Also, the additions to the model are all sensible and correspond to regularly observed data patterns in scan paths. However, this does imply that the comparison to DeepGaze might not have been strictly necessary and comparing to deep learning models to improve mechanistic ones is also not a new idea. To their Credit the authors acknowledge and cite the earlier occurrences of these ideas. Thus, I overall like this work, but I do not believe it is a big breakthrough.\n\nIn this manuscript the authors add a number of interpretable new mechanisms to the state of the art mechanistic model of eye movements. While I think specialist experimentalists in this particular subfield could know about the existence of the modelled phenomena, there are no mechanistic models that contain the added effects. Thus, this work really moves the state of the art forward for this type of mechanistic eye movement model. Also the path chosen to implement this improvement is a fairly recent addition in cognitive modelling toolkit, which I have not seen applied to eye movement models anywhere before and the authors promise to make their expanded model and a faster version of the original one public. So this paper is---in my view---a substantial and noteworthy step forward for models of eye movement trajectories.\n\nAlso, I like that this article does actually appear solid. For each of the new mechanisms there is evidence in the data and the authors know of corresponding literature and cite it. This makes this an exceptionally clear and solid paper. This solidness is also the reason for my fairly short review. I read through the descriptions of all the different mechanisms and found them to be generally sensible. If they were exceptional ideas or questionable, I would have had much more to write about, but they are all fairly straightforward implementations of the observed phenomena.\n\nI value solid research highly and believe this paper does well on this front, which is why I vote for acceptance even though I have to acknowledge that the parts that the authors put together here have been presented elsewhere before, such that an argument for originality or novelty is hard to make.", "was_revised": true, "final_justification": "Unsurprisingly, the authors response did not change my opinion about this paper much.\n\nReading through the other reviews and responses also did not change my rating. I would now give the authors a little more novelty credit, because this way of using DNNs to improve models clearly had not reached all the reviewers of this paper yet. At the same time, some of the misunderstandings hint at the paper seeming more clear to me than it actually was due to my prior knowledge. Keeping things as they are seems fitting to me.", "rebuttal": "Thank you for your thorough and positive review. We are very grateful for your support of our work and for your encouraging assessment of its solidity and contribution to the field. We also appreciate the questions you raised, which will help us further refine the paper.\n\n### [Question] On the use of Weighted Log-Likelihood Difference (WLLD)\n\n> The weighted log-likelihood difference is an unusual measure for the difference between predictions. Why do the authors prefer it over a more classical measure like the KL-divergence?\n\nYou raise an important point. Indeed, KL divergence is a widely used measure for comparing distributions, which we have considered as it could in principle be applied here. However, we chose the Weighted Log-Likelihood Difference (WLLD) for two main conceptual and practical reasons.\n\nConceptually, WLLD better reflects our analysis goals: It emphasizes how well a model predicted the actual observed fixation, rather than comparing the full predicted distributions. This aligns with our focus on understanding behaviorally relevant errors. By weighting the log-likelihood difference by DeepGaze’s predicted probability, we highlight fixations that DeepGaze deems highly likely but where SceneWalk fails. It is these cases that are especially informative for model improvement and interpretability purposes. This is more flexible than what standard KL divergence prioritizes.\n\nPractically, WLLD is far more efficient to compute and store. Because we score models using log-likelihoods at the ground-truth fixation locations, we only need to store one scalar per fixation per model. In contrast, computing KL divergence between the full distributions would require saving the complete per-fixation prediction maps, which can be prohibitive in terms of memory: over 300 GB for MIT1003 alone if saved without compression (100,000 fixations * 1024px * 768 px * 4 bytes). Since our method involves computing differences across many models and fixations, this would have been less manageable at scale.\n\nOn a conceptual level, WLLD and KL-Divergence are actually closely related: KL-Divergence in our case would be a pixelwise sum over weighted differences in log probability. Our WLLD essentially only takes the value from the pixel that actually has been fixated next.\n\n**Planned revision:** We will clarify this rationale in the final version, and we agree that exploring alternative divergence measures could be an interesting future extension.\n\n### [Question] On evaluating on the Potsdam dataset\n\n> Wouldn’t it be interesting to evaluate the model additions on the data used originally to develop the SceneWalk model?\n\nWe thank the reviewer for this thoughtful suggestion. We agree that evaluating on the original dataset used to develop the SceneWalk model provides a valuable reference and an additional test of generalization. Following this suggestion, we ran the updated SceneWalk-X model and individual new mechanisms on the Potsdam corpus.\n\n| Model                    |   Information Gain |\n|:-------------------------|-------------------:|\n| Spatial baseline         |            1.80648 |\n| SceneWalk (classic)      |            2.23223 |\n| SceneWalk (perisaccadic) |            2.32155 |\n| SceneWalk-X              |            2.42571 |\n| DeepGaze                 |            2.54    |\n\nThe results confirm the same qualitative pattern shown when evaluating on DAEMONS in Figure 6: temperature scaling yields modest improvements when averaged across the dataset due to the longer viewing times in this corpus compared to MIT1003, but the overall SceneWalk-X model improves predictive performance over the original perisaccadic SceneWalk by 0.1 bits/fixation (a 20% improvement with the static baseline as a reference), and closes approximately 47% of the remaining performance gap to DeepGaze III (reaching 82% “performance explained” in absolute terms as used in Figure 6). These findings further support the generality and robustness of the model extensions we propose.\n\n**Planned revision:** We will include these additional evaluation results in the appendix of the revised version of the paper.\n\n### [Formatting concern] On the readability of figures\n\nThank you for pointing this out. We will carefully review all figures, especially Figures 3, 4, and 5, and increase font sizes and adjust layouts to ensure all text and axes are clearly legible in the final version.\n\n---\n\nOnce again, we thank you for your thoughtful and constructive review. We are especially grateful for your recognition of the paper’s solidity and for highlighting its relevance to the field of scanpath modeling. We hope our responses have addressed your remaining questions and clarified our methodological choices. Your feedback helped us strengthen the manuscript further, and we look forward to incorporating these refinements in the final version."}, {"review_id": "3uf9iBhSSA", "reviewer": "Reviewer_i2u5", "summary": "This paper uses a deep neural network (DNN) model of human gaze behavior called DeepGaze III (DG III) to improve a mechanistic model called SceneWalk. In particular, the authors applied both models to the MIT1003 dataset of images and determined specific cases where the scanpath predictions of DG III and SceneWalk differed. Then, the authors used those \"controversial fixations\" to come up with extensions and modifications to SceneWalk that would make the model perform more similarly to DG III. (In other words, they identified common patterns that explained the different fixation predictions made by the DG III and SceneWalk so that they could generate new mechanisms to add to SceneWalk that fit those patterns.) In particular, they added the following features to SceneWalk: time dependent temperature scaling (favoring high-saliency targets early in scanpaths), saccadic momentum (from an oculumotor bias map), and a horizontal and left-ward bias. These modifications brought SceneWalk's performance from 0.15 bit/fix to about 0.3 bit/fix, compared to DG III's 0.45 bit/fix.", "questions": "To make the rebuttal process more constructive, I’ve aligned my questions below with the corresponding numbered weaknesses in my review.\n\n1a. Would the authors consider revising their conceptual framing to reflect the growing use of task-performing DNNs as mechanistic models (e.g., surprise-driven DNNs, goal-directed architectures, or task-constrained pretraining)? This would allow their discussion to better integrate and contextualize recent trends in computational cognitive modeling.\n\n1b. Can the authors justify their choice to retrofit SceneWalk using DG III outputs, rather than analyzing DG III itself to extract mechanistic principles (which they show is possible)? What is gained by working through a weaker model rather than analyzing the stronger one directly?\nThe paper generates mechanistic insights using discrepancies between SceneWalk and DG III, but the original DG III paper also reports its own mechanistic insights. Could the authors clarify why their methodology brings something unique that mechanistic analyses of DG III don't?\n\n1c. Could the insights the authors take from this paper ultimately be applied to mechanistic DNN models? Could we aim for an explanatory model that exceeds DG III?\n\n2. Can the authors provide any estimate of human inter-observer consistency as a comparison point? Or, alternatively, justify in more detail why DG III should be treated as a stand-in for human scanpaths?\n\n3. Could the authors clarify the data partitioning at each stage of their pipeline? Specifically:\n- What data is used for initial SceneWalk vs. DG III comparisons?\n- How is the final performance comparison between SceneWalk vs. DG III done: is it on held-out data?\n- Is there any data from human performance to demonstrate that SceneWalk is now more like human scanpaths and not just more like DG III?\n\nA note on limitations of possible revisions:\nFor the sake of transparency, I want to be clear that even if the authors address all the questions above, my core concern may still remain. The central methodological premise — retrofitting a simpler model to approximate a stronger one that already exists and performs better — seems conceptually inefficient and hard to justify. Unless the authors can articulate why this retrofit strategy is necessary or uniquely revealing, my evaluation may not shift substantially, even if surface-level issues are resolved.", "limitations": "Potential limitations are already addressed above. I see little (if any) potential negative societal impact of their work.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThis paper tackles a central question in cognitive science and computational modeling: what drives visual attention? This question is critical for understanding vision writ-large because what we attend to becomes the training curriculum for visual learning in general. The authors engaged with the literature on scanpath modeling and positioned their work in relation to both cognitive models and deep learning approaches. This is a promising direction, but I'm not yet convinced by the current version.\n\nWeaknesses:\n\n1) The paper sets up a false dichotomy between mechanistic models and DNNs. Deep networks are not inherently non-mechanistic; their internals can embody explicit cognitive hypotheses (e.g., energy-based surprise models for gaze—Aakur & Bagavathi 2020). By equating “mechanistic” with “non-DNN\" the authors ignore the possibility of leveraging deep models that they already recognize as more powerful models. Ironically, the paper is able to generate 3 mechanistic insights from the performance of DG III, and the original DG III paper (Kummerer et al., 2022) includes multiple mechanistic insights like interplay of scene content and scanpath history.\n\nMore broadly, DNNs are becoming increasingly popular as formal, task-performing models of cognition. If a DNN can perform the same task as the brain, then this serves as evidence that the brain may solve the task in a similar way. The brain is a complex system of billions of neurons, and its operations may very well be better characterized by an optimization process (ala deep learning) than by simple cognitive rules. Moreover, while the authors are correct that most deep learning approaches to scanpaths (e.g., DG III) learn to predict human fixation directly, DNNs do not have to be trained this way by necessity. It's possible to train DNNs with the same data as humans (i.e., without supervised human behavior like image labels and scanpaths) to test if they can learn the same skills from the same types of data.\n\nThe result of this false dichotomy is that the authors acknowledge how much more powerful DNN models are, but rather than embracing DNNs and using DNNs directly to learn about scanpath mechanisms, they use DG III to bootstrap SceneWalk. This is analogous to having access to a Ferrari (DNN) and choosing to retrofit a horse-and-buggy cart (cognitive model) to follow its tracks. Rather than embracing the capabilities of the more powerful vehicle, they attempt to enhance the older model—ultimately limiting the practical gains that could be realized by simply using the better tool.\n\n2) It's not clear why the paper does not include a human upper bound on performance. The only explanation offered is that \"estimates of inter-observer consistency are hard to derive\" \"in high-dimensional distributions.\" However, I don't understand why that difficulty would not apply equally to comparing any two high-dimensional distributions (including model-vs-model and human-vs-model). Instead the paper uses DG III as gold standard / ground truth, but without evidence that DG III performance should be trusted as ceiling.\n\n3) Relatedly, by using DG III both as the gold standard and to find controversial fixations, the method becomes somewhat circular. The overall approach is: (1) contrast SceneWalk with DG III, (2) hand-engineer changes to reduce the gap, (3) measure whether SceneWalk now more closely resembles DG III. Because performance is never validated against human behavior, the exercise risks becoming self-referential. Instead, it would help for the authors to show that changes which shrink the SceneWalk-DG III gap also shrink the gap to humans on a held-out images. \n\nAnother change that is also needed to deal with possible circularity issues is to specify (in SI) whether the data used to diagnose SceneWalk-DG III discrepancies is disjoint from the data used to evaluate the final, tuned model. There is some limited discussion about crossvalidation folds in SI, but it's not detailed enough for me to tease apart the stages of training, testing, initial SceneWalk-DG III comparison, and final SceneWalk-DG III evaluation.", "was_revised": true, "final_justification": "The rebuttal was exceptionally clear, resolved my methodological concern about circularity, and clarified the authors’ position on the continuum between interpretable and performance-driven models. My main reservations remain philosophical rather than technical — I continue to view directly analyzing or refining high-performing DNNs as a more promising path than retrofitting simpler mechanistic models. Having said that, I do see merit in different modeling philosophies coexisting in the field. Overall, my score reflects my evaluation of the paper's alignment with my acceptance threshold, not the quality of the exchange, which was excellent.", "rebuttal": "Thank you for your deeply engaged and challenging review. We appreciate the time you took to analyze our work from a conceptual standpoint. Your feedback has pushed us to realize that our framing of the relationship between mechanistic models and DNNs was not as nuanced as it should have been, and we are grateful for the opportunity to clarify our methodology and contributions. We will start with your main concern about the usefulness of “retrofitting” simple models, and then continue with the other questions and issues.\n\n### [Q1b] Why simple models?\n\nThe question touches on fundamental differences in modeling philosophy: namely, what counts as scientific understanding. We respectfully argue that even in the eara of of powerful function approximators, simple mechanistic models are still essential in cognitive science, and dismissing their widespread and long-standing role across the sciences altogether reflects a perspective that is not representative of the field as a whole.\n\nAt the core of our perspective is the idea that scientific understanding essentially is a form of compression: expressing complex phenomena, like gaze behavior, in terms of a smaller set of testable and generalizable principles, typically using language (natural or optimally mathematical). From this view, shared by many in philosophy of science (e.g., Wilkenfeld, 2019), a DNN with millions of entangled parameters constitutes a much less efficient description than a mechanistic model with a handful of interpretable components. All else being equal, the latter better supports explanation, communication, and theory-building, in our case being clearly positioned in the algorithmic level out of Marr's (1982) famous three levels (computational, algorithmic, and implementational). Of course, simple explanations are not helpful when they make terrible predictions: Ultimately, there is a pareto front along the tradeoff between simplicity and predictive power. We consider it worthwhile to also (not only!) strive for the most simplistic but still predictive account of behaviour, which forces us to explain how different interpretable mechanisms interact. This is in line with long standing and still active traditions in the field (Craver et al 2024’s “Mechanisms in Science” in the Encyclopedia of Philosophy; Bechtel’s mechanistic philosophy, 2011).\n\nWe understand the concern about “retrofitting” a simpler model to match a more powerful one. However, our goal is not to retrofit in a superficial sense, but to use the predictive success of DG as a scaffold for structured hypothesis generation. Rather than digging into the internals of the DNN – which, when dealing with cognitive phenomena, can be prone to pitfalls (e.g. Geirhos et al., 2020) – we ask: What kinds of behavioral errors remain in the mechanistic model, and do they point to missing cognitive components? The value of this approach is backed by recent publications in top‐tier venues like Nature, which, like the Centaur foundation model—explicitly position computational predictions as stepping stones toward theory: they argue that predictive models offer ‘tremendous potential for guiding the development of cognitive theories’. This synergy is also strongly endorsed in the philosophical literature (e.g. Harbecke 2021) and empirical practice in cognitive science.\n\nAs a final point, we’d like to playfully return to the car analogy. SceneWalk-X achieves over 70% of DeepGaze III’s performance using fewer than 1% of its parameters (31 vs. thousands) and compute. In that light, it is at least intriguing, we argue, that a horse-and-buggy can reach 70% of the Ferrari’s top speed while using <1% of the fuel. And while you might pick a Ferrari to go fast, if your goal is to teach someone how a car works you might prefer the old-timer with the hood open and the individual components clearly visible.\n\n### [Q1a] On the continuum between mechanistic models and DNNs\n\nWe agree that our framing should more clearly reflect the continuum of approaches between classic cognitive models and modern deep networks. While DNNs are indeed computational mechanisms, and can instantiate cognitive hypotheses or be trained with elegant constraints, their complexity typically prevents full interpretability. Our use of the term “mechanistic model” (grounded in the modelling literature in neuroscience and cognitive science) refers specifically to models built with transparent, modular components that directly reflect hypotheses. However, we acknowledge that between the extremes, a continuum of hybrids and more interpretable DNNs has evolved with different tradeoffs between performance and simplicity. We will revise our terminology to reflect this continuum more explicitly and justify our position therein.\n\nFurther, we fully agree with the reviewer that DNNs can be integrated into scientific explanations when their structure is designed with clear cognitive hypotheses in mind. The scene-history interaction component of DeepGaze III is indeed such an example and shows that DNNs can yield individual mechanistic insights. \n\nYet, we consider these orthogonal but related approaches at different ends of the above-mentioned pareto front: In DeepGaze III, a high-performing model is made more interpretable, while we aim at making a fully interpretable model more high-performing.\nThe former establishes that a mechanism exists by asserting that we can't predict behaviour better by loosening constraints. The latter establishes how much of the overall behaviour we can explain via only fully interpretable mechanisms. \n\n**Planned revision (Q1a and Q1b):** We will revise and extend our introduction and discussion to clearly reflect the continuum between interpretability-first and performance-first models. Specifically, we will (1) clarify our use of “mechanistic,” (2) acknowledge the increasing role of hybrid and task-constrained DNNs in cognitive modeling, and (3) better articulate why interpretable models remain crucial for scientific understanding, even when performance is not maximized.\n\n### [Weakness] On concerns of circularity\n\nWe agree that care must be taken to avoid circular reasoning when using one model to guide improvements in another. However, we believe the core of the issue raised might stem from a misinterpretation of how our evaluation is performed.\n\nWhile we do use DeepGaze III to identify controversial fixations, we do not measure the success of our method by checking whether SceneWalk becomes more similar to DG. Instead, we evaluate all model variants using log-likelihood scores *against human fixations*, i.e., real behavioral data. Our metric directly reflects how well the model predicts actual human eye movements, not whether it agrees with DG III. In this context, DG, in the form of its performance, just provides an estimate for the log-likelihood on the true data that *could* be achieved.\n\nFurthermore, we believe the inclusion of a second, completely held-out dataset (DAEMONS), where SceneWalk-X also improves, provides strong evidence that our methodology leads to genuine improvements in human behavioral prediction.\n\n### [Q1c] \n> Could the insights the authors take from this paper ultimately be applied to mechanistic DNN models? Could we aim for an explanatory model that exceeds DG III?\n\nWe agree this is a compelling direction. As long as the DNN model is sufficiently powerful and there is enough data to train it, it should always perform at least as well as a more constrained model. Given that data is not infinite, though, it might be possible to find mechanisms that given the possible DNN size are hard to approximate and where a direct implementation could perform better, especially on OOD data.\n\n### [Weakness + Question 2] On upper bound explainability estimates\n\nWe appreciate this important point, which was also raised by reviewer 3tRx. Please check our response there for a more detailed explanation and our revision plan for addressing this clarity issue.\n\nBriefly, each fixation must be predicted from a unique and growing history of previous fixations, so the relevant distribution depends on up to ~20 input variables per step. With only about 100 fixations per image, this leads to extremely sparse coverage, making empirical or nonparametric baselines infeasible.\n\n### [Question 3] Data partitioning and evaluation\n\nFor model training and evaluation, we used 10-fold cross-validation on MIT1003 over images, for each fold using 8 parts for training, one part for analysis and one part for final performance evaluation, ensuring that both DG and SW were always evaluated on held-out data with respect to fitting.\n\n* The initial DG vs SW comparisons and the identification of controversial fixations were done across the full MIT1003 dataset, looking at conditional densities on *validation splits* for a particular image.\n* The final performance on MIT1003 (Fig. 1b and 6) is reported aggregated over *test splits*. For DAEMONS, which was fully held out during development, it is on the dataset’s provided public validation split.\n* We evaluate models using log-likelihood scores against human fixations, i.e., real behavioral data. Our metric directly reflects how well the model predicts actual human eye movements, not whether it agrees with DG III.\n\n---\n\nWe thank you once more for your review, which prompted us to sharpen our conceptual distinctions and more explicitly justify key design choices. We also appreciate the opportunity to clarify that our evaluation is always against human data, and that we avoid the risk of circularity through cross-validation and out-of-distribution testing.\n\nIn light of these clarifications, we hope you might view the work not only as an interesting idea, but also as a carefully executed and generalizable contribution. In spite of your confidence in the original judgment, we respectfully invite you to reconsider whether the manuscript might warrant a more favorable assessment."}, {"review_id": "OHOj3Z2IvT", "reviewer": "Reviewer_SwGg", "summary": "The authors propose to improve a mechanical model of eye movements on static images by using a data-driven model and to analyse the controversial predictions between both models in order to improve the mechanical model. The results show a significant improvement in the predicting eye movements.", "questions": "A major contribution of your model and establish a function of temperature and momentum that is introduced into the mechanistic model. Can you relate the parameters you changed to some biological data, whether mechanistic for example by considering the system of muscles that allow eyeball movement.\n\nA major feature of the interaction between eye movements and scene understanding is that the image on the retina is foveated, thus allocating more ressources (photorecpetors) around the axis of sight. Could you relate your model to such models that would use foveated retinotopy?", "limitations": "One limitation of the paper is its relatively incremental advancements, which, while valuable, do not significantly depart from existing methodologies. Interpretability would be a huge bonus.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The strength of the model presented here is that it presents a mechanical model, and is therefore much more realistic, biologically, which reduces the gap between models that perform well, but are unrealistic and are based on mechanical rules. The inferred statistics of the different characteristics of eye movements during the exploration of an image has particular significance for the scientific community studying  biological movements.\n\nA weakness of the paper is that it is relatively incremental and shows improvement only mainly on the evaluation criteria that were used to learn the data-driven model.", "was_revised": true, "final_justification": "While the responses to my concerns were clearly expressed, the proposed changes are too minor to reevaluate my score.", "rebuttal": "We thank you for your positive review and for your appreciation of our work’s goal.\nWe are grateful for your feedback and your insightful questions, which will help us better connect our findings to the broader scientific context. Let us respond point by point below.\n\n### [Question] On relating new mechanisms to biological data\n\n> A major contribution of your model and establish a function of temperature and momentum that is introduced into the mechanistic model. Can you relate the parameters you changed to some biological data, whether mechanistic for example by considering the system of muscles that allow eyeball movement.\n\nThis is an excellent point. While our modeling is at a functional level, the mechanisms we've added are indeed motivated by and consistent with known biological and cognitive phenomena. However, exactly matching the parameter values is likely nontrivial at this point. For saccadic momentum, Wilming et al., (2013) show that muscle properties alone are not enough to explain the observed saccadic momentum and conclude that central oculomotor planning likely also plays a role. For the case of the temperature change, we expect a dependency on dataset statistics, task instructions and subject motivation.\n\n**Planned Revision:** We will expand our discussion of the different mechanisms to make these links more explicit. For saccadic momentum (lines 322, 341-342), we will reinforce its connection to established findings on oculomotor dynamics and facilitation-of-return. For the cardinal attention bias (lines 462-467), we will further highlight the literature suggesting such anisotropies may originate as early as the superior colliculus and reflect fundamental properties of attentional processing.\n\n### [Question] On the relation to foveated retinotopy\n\n> A major feature of the interaction between eye movements and scene understanding is that the image on the retina is foveated, thus allocating more ressources (photorecpetors) around the axis of sight. Could you relate your model to such models that would use foveated retinotopy?\n\nThis is a very insightful question. While SceneWalk does not simulate a foveated retina at the pixel level, its core principles are compatible with the functional consequences of foveation. The model does not operate directly on the image itself, but rather on a static saliency map derived from the image. This map is initially hidden and we uncover it locally by looking somewhere, which opens a Gaussian window on the saliency map: This resembles how visual acuity is concentrated at the center of gaze. In this sense, SceneWalk shares important functional similarities with models that explicitly incorporate foveated retinotopy (such as through image-space transformations) though it operates at a higher level of abstraction, dynamically allocating resources over a saliency map rather than a pixel-level retinal representation. \n\nGoing one step earlier and exploring in more detail how scene content is integrated into a latent state that is then used to select fixations would be a very exciting next step. A conceptually simple approach would be one one from MASC (Adeli et al., 2017), where peripherally blurred images are passed to a saliency backbone to update an internal saliency map. More appropriate but also more involved would be a backbone that directly models foveation, e.g., FLIP (Traub et al., 2025), and could include object attention, semantic attention, scene relationships and other effects. \n\n**Planned Revision:** We will add a sentence to the discussion acknowledging this strong conceptual link. We will state that our model captures the functional dynamics of foveated vision and suggest that incorporating an explicit foveated input representation is a promising avenue for future work, both for SceneWalk and for DeepGaze III.\n\n### [Weakness] On scope of the contributions\n\n> A weakness of the paper is that it is relatively incremental and shows improvement only mainly on the evaluation criteria that were used to learn the data-driven model.\n\nWhile we agree that our approach builds on existing components, we believe its value lies in demonstrating how these components can be systematically integrated to advance both mechanistic modeling and interpretability in deep learning. Our framework is inspired by the idea of scientific regret minimization (which we cite and build upon), but we go beyond prior work by applying it in a concrete setting where we substantially improve a mechanistic model and simultaneously gain insight into the inductive biases of a deep network.\n\n\nWe view this as a contribution of both methodological and conceptual significance: it shows how existing ideas can be operationalized to generate interpretable, testable refinements to biologically inspired models. While indeed the improvements made to SceneWalk may be incremental in a narrow technical sense, we believe such work is crucial for bridging the gap between high-performing performance-driven models and mechanistic understanding, a central challenge in both neuroscience and AI.\n\nFinally, we want to emphasize that the performance gain we achieve is not small, but closes a substantial part of the gap between purely mechanistic performance and DNN performance.\n\n---\n\nThank you again for the constructive review. We hope our responses have clarified how this work, while building on prior ideas, provides a concrete path toward improving mechanistic models by leveraging insights from deep networks. Beyond predictive performance, we believe the framework contributes to a growing effort to build models that are both interpretable and grounded in biological principles—an important step for bridging data-driven and theory-driven approaches to understanding behavior. We hope our response has helped support your positive assessment or even strengthened your confidence in the paper’s value."}]

Author rebuttal:

Meta-review: This paper aims to provide a model of eye fixation that both accurately describes how humans move their eyes and also maintains interpretability, by using a deep model (DeepGaze III) to inform a mechanistic model (SceneWalk). This approach, clearly described in the paper, leads to improvements on multiple datasets, while providing mechanistic interpretability. The authors provided extensive responses to the concerns of the reviewers, who were satisfied with many of them. While the work is somewhat incremental, it still generated excitement and promises to be useful for the field. An interesting discussion occurred between a reviewer and the authors about the tradeoff between interpretability and predictive performance, which should be integrated in the paper, along with the other responses to the reviewers.
Decision: Accept (poster)