Behind the Machine’s Gaze: Neural Networks with Biologically-inspired Constraints Exhibit Human-like Visual Attention
Authors: Leo Schwinn
Doina Precup
Bjoern Eskofier
Dario Zanca
Venue: Accepted by TMLR ()

Abstract: By and large, existing computational models of visual attention tacitly assume perfect vision and full access to the stimulus and thereby deviate from foveated biological vision. Moreover, modeling top-down attention is generally reduced to the integration of semantic features without incorporating the signal of a high-level visual tasks that have been shown to partially guide human attention.
We propose the Neural Visual Attention (NeVA) algorithm to generate visual scanpaths in a top-down manner. With our method, we explore the ability of neural networks on which we impose a biologically-inspired foveated vision constraint to generate human-like scanpaths without directly training for this objective. The loss of a neural network performing a downstream visual task (i.e., classification or reconstruction) flexibly provides top-down guidance to the scanpath.
Extensive experiments show that our method outperforms state-of-the-art unsupervised human attention models in terms of similarity to human scanpaths. Additionally, the flexibility of the framework allows to quantitatively investigate the role of different tasks in the generated visual behaviors. Finally, we demonstrate the superiority of the approach in a novel experiment that investigates the utility of scanpaths in real-world applications, where imperfect viewing conditions are given.

Published in Transactions on Machine Learning Research (10/2022)
Behind the Machine's Gaze: Neural Networks with
Biologically-inspired Constraints Exhibit Human-like Visual Attention
Leo Schwinn Doina Precup2344
leo. soo.schrinnn Saww dprecup Ges. wes.mooill..o co. bjuern, eskofier Cerrrrrua.uu... de dario. zanca &8 de (corr. author)
Bjoern M. Eskofier
Dario Zanca!
Uniiosstt Universität Brlanggn-Nürnbery
Mila McGill University
4 Deep Mind
Reviewed on OpenReview: httpes//ppeemmeeeenc///oaouuut
Abstract
By and large, existing computational models of visual attention tacitly assume perfect vi- sion and full access to the stimulus and thereby deviate from foveated biological vision. Moreover, modeling top-down attention generally reduced to the integration of semantic features without incorporating the signal of high-level visual tasks that have been shown to partially guide human attention. We propose the Neural Visual Attention (NeVA) algo- rithm to generate visual scanpaths in top-down manner. With method, explore the ability of neural networks on which we impose biologically-nnsireed foveated vision constraint to generate human-like scanpaths without directly training for this objective. The loss of a neural network performing a downstream visual task (i.e., classification or re- construction) ficxibly provides top-down guidance to the scanpath. Extensive experiments show that our method outperforms state-of-the-art unsupervised human attention models in terms of similarity human scanpaths. Additionally, the flexibility of the framework allows to quantitatively investigate the role of different tasks in the generated visual behaviors. Fi- nally, demonstrate the superiority of the approach in novel experiment that investigates the utility of scanpaths in real-world applications, where imperfect viewing conditions are given.
1 Introduction
Vision enables humans and other animals the detection of conspecifics, food, predators, well as other vital information even at considerable distances. However, huge amount of information of about 107 to 10 bits reaches the optic nerve every second (Itti & Koch 2001): to process it all would be impossible for biological hardware with limited computational capacity. The mechanism visual attention allows to select only a subset of the visual information for further processing by means of eye movements. This breaks down the problem of scene understanding into a scries of less demanding and localized visual analysis. The extracted subset of information mostly corresponds populated with photoreceptors, which allows for fine vision (Bringmann et al. 2018). Fast eye movements called saccades serve to rapidly shift the fovea to regions of interest in the visual field (Purves et al. 2019). Between saccades, the direction of gaze remains still, during the so-called fization where visual information is collected. The sequence of fixations, also called scampalh, determines the flow of incoming information and is therefore critical for the effectiveness of human vision.
the fovea, cireunscribed area of the retina highly
The selection performed by attention can be characterized by two distinct mechanisms: bottom-up and top-down (Conoor et. al. 2004). The bottom-up mechanism operates in the raw sensory input, rapidly

---

Published in Transactions on Machine Learning Research (10/2022)
Stimulus
Perceived
Signal propagation
Loss propagation
Stimulus sampling
Attention,
Perceived
aggregate
π
h1=2
T
Agent State
Figure 1: Illusraaionn of the proposed Neural Visual Attention (NeVA) algorithm. A given sttmulus is blurred and subsequently processed by the attention mechanism (a). The attention mechanism calculates an attention position for the foveation mechanism (σ). The foveation mechanism deblurss the image at the attention position and thus creates the perceived stimulus at time t ++1. The perceived stimulus is processed together with previously seen stimuli of the model to obtain the agent state of the image at time + 1. Lastly, the loss signal the lask model (r) for the agent state is used to assess the quality the attention position and thus guide and train the attention mechanism.
and involuntarily orienting attention towards visual features of potential importance (e.g., to a red spot in a green field). The top-down mechanism slower, task-driven, and generally aimed at implementing long-tern cognitive strategics (e.g., assess the age of person). Humans show specific temporal pattern (Tatler et al. 2005): their very first fixations are generally guided by bottom-up features and exhibit high correlation, while they diverge very soon due to individual goals and motivations (i.c., tasks). This suggests that bottom- up attention can be described as common mechanism among individuals, which makes it easier to be studied and characterized. For this reason, most recent studies have been oriented to bottom-up visual attention modeling.
Computational modeling of human visual atlention lies at the intersection of many disciplines such as neuroscience, cognitive psychology, and computer vision. The construction of models that formalize the mechanisms of attention is of profound importance both for understanding the mechanism itself, potentially allowing the derivation of its biological implementation in the form of neural hardware (Koch & Ullman
1987 Zanca et al. 2020a), and in the field of application such as surveillance (Yubing et. al. 2011), video- compression (Hadizadeh & Bajic 2013), virtual reality streaming (Rondon et al. 2021), image quality measure [Barkwssy et al.
2009 or question-answering (Das et al. 2017). Equipping artificial intelligence attention mechanisms could foster the robustness and interpretability of such Vuyyuru al.] 2020). In the last. three decades, many efforts have been made models of visual attention. Based on the seminal work by Treisman describing the feature integration theory (Treisman & Gelade 1980), current approaches assume centralized role of the saliency map, i.e., spatial map expressing the conspicuity of the information for each location in the visual field. Within this theory, attention shifts would be generated from the saliency map by the winner-take-all algorithm (Koch & Ullman 1987), which selects the position associated with the highest saliency as the first fixation, then inhibits this location and moves to the second highest saliency location to generate the subsequent fixation, and so on. For this reason, the vast majority of studies have focused on improving saliency map estimation (Borji Itti 2012 Riche et al. 2013). Currently, the development of increasingly large eye-tracking datasets (Judd et al. 2009 Borji & Ittii 2015 Jiang al. 2015), together with the
with Mooooicalllyiisspieed
systems (Luo et al. 2015
developing compulational
2

---

Published in Transactions on Machine Learning Research (10/2022)
of transfer learning techniques (Huang et al. 2015 Kümmerer et al. 2016 Borji 2019). have made models based on deep convolutional neural networks the state-ol-the-art, in the saliency prediction task.
Following the trend of these works, the vast majority of eye-tracking datasets were collected under free- viewing conditions. This implies that subjects are required to "watch the images freely during the time they are presented' to them. This should eliminate the influence of any tasks and make the assumption of a bottom-up-driven vision (and attention) legitimate. However, this assumption drastically simplifies the problem of vision to that of a process of discovering what is where in the world. Instead, vision can not be thought of just internal representations which serve as basis for future actions. This duality (information processing and representation learning) inextricably links vision to the concept of intent or, similarly, lask. For example, even when looking at stimuli under free- viewing conditions, we we still observe that objects drive attention better than bottom-up saliency (Einhäuser et al.) 2008), or that emotional content influences attention dynamics already in the beginning of the visual exploration (Pilarczyk& & Kuniccki 2014). This leads to the conclusion that eye-tracking data collected under free-viewing conditions may not be free of lask-driven components and the development of purely task-driven methods may help to quantify this phenomenon.
iifformation-proceeee
task (Marr 2010): our brains must be able to develop
Most datasets and methods are designed for the purpose of saliency prediction (Boccignone et al. 2019b; Zanca et al. 2020b). However, saliency maps are static and do not encode the temporal aspect of visual exploration. Saliency maps do not take into account temporal dynamics, i.e., how fixations are ordered in a sequence. This implies that once collected, fixations are inherently interchangeable with respect to their temporal order. Consequently, the temporal order of fixations is largely ignored in experimental evaluations, with important consequences: 1) two scanpaths visiting the exact same points in stimulus produce the same saliency map but may describe very different dynamics (e.g., some plausible, and some very unnatural). 2) Modelling these dynamics correctly can lead to more interpretable algorithms. To solve this issue, methods have been proposed that try to model the human scanpaths directly. Scanpath models can be separated into unsupervised, i.c., those that do not use any eyo-tracking data to learn attentional shifts and supervised approaches. Since supervised methods (Jiang et al., 2015 Assens et al. 2018) cannot provide an explanation about the mechanism underlying human attention, they tend to overfit on the experimental sctup of the available data, and hence generalise poorly across domains (i.c., roquire retraining) (Arjovsky et al. Ullman 1987 coupled with the mechanism of inhibitiion-f-returr.. can be combined with any unsupervised saliency estimation method, such as Itti's saliency map (Itti et al.) 1998), to generate scanpaths in an unsupervised manner. Boccignone & Ferraro (2004) describe gaze shifts as a realization of a stochastic process with non-local transition probabilities defined a saliency field, that represents a landscape upon which a constrained random walk is performed. Attention then simulated as a search mechanism driven by a Langevin equation whose random term is generated by Levy distribution. In (Zanca & Gori, 2017; Zanca et al.) 2019a), attention is described as dynamic process where eye movements are driven by three basic principles: boundedness of the retina, attractiveness of brightness gradients, and brightness invariance. Similarly, (Zanca et al.) 2019b) proposes a mechanism of attention that emerges as the movement of a unitary mass in a gravalational field, where specific visual features (intensity, optical flow and face masks) are extracted for each location and compete to attract the focus of attention. A drawback of current approaches in computational modcling of visual attention is assuming perfect information (Bringmann et al. 2018)). It is well known that human vision operates on imperfect (i.e., loveated) information and how actions (e.g., choosing the location of the next focus of attention) are taken this basis. In contrast, all the proposed models tacitly assume that the visual input in its full resolution is available at each time. Wang et al. (2011), for example, design mechanism of foveated vision but, to predict the new location of interest, they compare the foveated input with the original stimulus and select the point of greatest. discrepancy. Gravitational models of attention (Zanca et al. 2019bb extract a gravitational field from feature maps that depends on the distance from the focus of attention in a similar way to the distribution of photoreceptors in the human retina, but the calculation of such fields is still done on the original stimulus at each time. In contrast, testing models that rely solely on imperfect vision could have a twofold ad advantage: highlighting how the implementation of biologically-inspree constraints can aid human-like scanpath simulation and developing methods that can actually work under lisited conditions.
2019 We focus on unsupervised methods in what follows. The circuitry of winner-tako-all (Koch &
3

---

Published in Transactions on Machine Learning Research (10/2022)
While imperfect vision and task guidance have not been explored in the modeling of human visual attention, they have been successfully studied in other application areas. [arochelle & Hinton (2010) described a model based on a Boltzmann machine with third-order connections that can learn to accumulate information from different fixations, showing that it is possible to achieve performance at least on a par with models trained on whole images for certain classification lasks. In (Denil et al. 2012), a model constrained with foveated vision trained to predict where to look for a task of image tracking. Different from previous proposals, the reward function is modeled as Gaussian process expand the action space from the discrete space of fixations to continuous space. Mnih et al. (2014) propose recurrent neural network model that is capable of extracting local information from an image or video by adaptively selecting sequence of regions to be processed at high resolution. The resulting model is not differentiable, and it is trained to maximize performance on rean tasks by a reinforcement learning procedure. In (Xu et al. 2015) an attention model learned for image captioning, where qualitative evaluation shows that the model is able to attend to relevant objects while generating the corresponding words.
In summary, we observe two main limitations in current approaches for computational modeling of visual attention. 1) All mentioned approaches are based on raw sensory signals (bottom-up) and do not incorporate any task guidance. 2) All of the mentioned methods still assume perfect information, mcaning that the original stimulus in its full resolution is needed to perform any computations.
this paper, we propose the Neural Visual Attention (NeVA) algorithm, a purely task-driven mechanism of visual attention which operates with imperfect (i.e., foveated) information to generate scanpaths. NeVa utilizes differentiable deep neural networks (i.e., pre-trained for image classification or image reconstruction) which convey the concept of visual task in the framework. We define differentiable layer that simulates the human-like foveation mechanism, which allows for end-to-end training of an attention mechanism guided exclusively by the error signal of the reference task. Our goal is not to develop a biologically plausible model of human attention but rather to observe whether imposing biologically-inspiee networks will lead to more human-like image explorations. Morcover, we show that a top-down signal can be effectively incorporated to condition the generaled scanpaths. An illustration of the NeVA method given in Fig. I We emphasize that eye-tracking data is not used during training, but only for the evaluation of the proposed methods. Our contributions can be summarized as follows:
constraints on neural
We use common metrics measure similarity between simulated and human scanpaths, and in- troduce and motivate the Baring-Based Time-Delay Embeddings (SBTDE), an alternative metric to account for stochastic components in human behavior.
In an extensive evaluation with multiple well-established eye-tracking datasets, we show that scan- paths generated by NeVA exhibit the highest similarity to humans compared to stato-of-the-art unsupervised methods, as measured by several metrics.
The flexibility of the proposed approach allows to quantify the contribution of different tasks (classification or reconstruction) to the generated scanpaths, and the importance of partial vision in generating plausible scanpaths (by comparing NeVA with its optimization-base version which assumes perfect vision).
Lastly, propose novel experiment where the utility of scanpaths to solve downstream visual task on unseen data under the constraint of imperfect information is assessed. In contrast to prior results that mainly focus on similarity to human scanpaths, this experiment aims to assess the usability of the simulated scanpaths for practical applications. Here, we show that NeVA-generated scanpaths are more effective at solving downstream visual tasks on unseen datascts, opening up possibilities for using NeVA models in practical applications where agents have to operate with constrained resources.
4

---

Published in Transactions on Machine Learning Research (10/2022)
2 Methods
2.1
Notation
Here we introduce the notation necessary for the following sections. In our experimental sctup, an agent (human or artificial) is presented with a static stimulus for a certain period of time T. Let. S denole a visual stimulus,
E Rd
where d indicates the dimensionality of the visual field. In the case digitized stimulus, i.e.; organized as two-dimensional grid of equally spaced pixels, d equals the number of pixels and their respective color channels. Suppose the agent's perception is foveated and let & denote its scanpath, which is defined sequence of fixations over period of time T,
{= (Ether
such that & I² represents the fixation location at time t E T. Finally, we denote by the perceived stimulus, which is obtained, at each time t T, by means of an attention-dependen foveation mechanism that is applied to the original stimulus S, i.e.,
R RR + R
(S, + (S,
The foveation mechanism should be designed to reflect the distribution of photoreceptors around the fovea. We provide differentiable implementation of a foveation mechanism in the following sections. In biological systems, the agent does not have direct access to the original stimulus S, mainly due to limited resources. All of the agent's computations, including any attentional behavior, are based on the sole imperfect infor- mation carried by the perceived stimulus क (S, Lt). This means that, we denote by a the agent's 's atlention mechanism responsible of generating scanpaths &, we can generally formalize it
(α (S, h1-1) +> E1+1.
where h4-1 denotes the agent's internal state at time t - 1. This assumption is often violated in most approaches of computational modeling of visual attention, where perfect vision assumed, for example when predicting saliency (Borji & Itti 2012) or scanpaths (Kümmerer & Bethge (022)).
2.2 Neural Visual Attention (NeVA)
We propose an algorithm, which we call NeVA (Neural Visual Attention), to generate visual scanpaths in a purcly driven approach. This is achieved by exploiting the loss signal of differentiable vision models (task models), such as convolutional artificial neural networks (CNNs), to guide visual attention. NeVA imposes the constraint of foveated vision the perception of these task models. This is achieved by inserting differentiable foveation mechanism before the first layer of the respective model. The foveation mechanism filters the original input and preserves only a small area of high acuity at the currently attended position. Thus, the performance of the task model degrades, which increases the loss. We exploit the loss signal of the task model to define a mechanism of selective attention that optimally solves the visual task at hand (i.e., which regions of the image are most important solve the given task and should be altended). The algorithm consists of the following three main components which are described in what follows. Fig. [ provides an illustration of the overall process.
2.2.1
Task model
The task model r is a differentiable model trained to solve a visual downstream task. Let S (SI, be collection of input stimuli. We specifically consider two different downstream tasks: a supervisod classification and an unsupervised reconstruction task. In the case of supervised classification, a set Y (v), (y1,..,yN) UN) of semantic labels is available, where y, represents the main class of objects contained in S,.
5

---

Published in Transactions on Machine Learning Research (10/2022)
Labels are generally provided by human supervisor and T is trained to learn mapping from S to Y, such that
T(S.) yı, Vi [J,
In the case of reconstruction, r is trained to compress the information S; to a smaller dimensional space (encoding) and to reconstruct S; from this compressed representation (decoding). Therefore, the target domain matches the input domain, i.e., = This process can be formally described as Tdec o Teno: where denotes the function composition operator.
Tdoc (en S,
where (S) E Rdd d < d to avoid trivial solutions and induce the learning of meaningful features. Hereby, we denote by (()(S), Y) the loss function evaluating how well the model T solves the given task. This information will be used as a signal to guide the attention mechanism. In our experiments, we train one neural network for each task and use common loss functions. In this context, we use the categorical-cross entropy loss to train the classification model and the mean-squared error loss to train the reconstruction model.
2.2.2 Foveation mechanism
To simulate human perception with any artificial system, we design a mechanism that is inspired by the biological constraint of foveated vision (i.e., a center of high visual acuity and coarse resolution in the periphery)- Given stimulus S, the foveation mechanism computes the perceived stimulus (8,&) foveated rendering of S centered at the current focus of attention & To ensure the mechanism is differen- tiable and, therefore, can be integrated with any differentiable models (e.g., neural networks) for end-to-end learning, we define it as follows.
Let Š be a coarse version of S, obtained by applying a convolution to the original stimulus S to suppress any frequecyy components, i.e.,
S=S Gop:
where Go, is a Gaussian kernel with zero mean and standard deviation op. From a biological viewpoint, $ can be regarded as the gist (Sampanes al.J 2008 Oliva & Torralba (006)) that is perceived by peripheral simulate how the visual acuity
vision in the first milliscconds of stimulus prosentation. Furthermore, decreases from the fovea center towards the periphery by means of Gaussian blob (4), with mean & and standard deviation de. The value of de is chosen to correspond with the average radius of the fovea region in humans' eyes, i.e., about 2 degrees of visual angle (Bringmann et al. 2018). Finally, the perceived stimulus can be obtained as a linear combination the original stimulus and its coarse version,
x (S = Go (1) S ())
which can also be interpreted as the operation of partially removing the smoothing effect (5 (5-S) S) from the original stimulus S, based on Gog
ग (S, ()) 5- (t) (5-5).
Here, denotes clement-wise multiplication. It worth noting that there is no Icakage of information from the original stimulus to subsequent layers, as the G-matrices are fixed.
2.2.3
attention mechanism
We consider an agent that soquentially explores the stimulus through fovcation mechanism. The agent's atlention a is responsible of selecting the next location to attend depending on the current perceived stimulus a (.) In addition, we assume the agent's decision to be based on the past perceived stimuli. To incorporate past information, we define the internal agent's state h4 h (S, to express the cumulative perceived information, or memory of the system. This state could be modeled, e.g., as recurrent neural network. However, since aim analyze the effect of foveated vision the visual exploration, keep this

---

Published in Transactions on Machine Learning Research (10/2022)
component simple possible. Thus, simply aggregate past Gaussian blobs, which effectively increases the area of high visual acuity after each fixation:
(8. - S (Ja - Gr(t))
with
0:1 Yo (t -))
0 Gs(t)= Σ
where [..]0:1 is element-wise clipping operator with minimun 0 and maximum 1, and Ja is d x d dimensional unit matrix (matrix only filled with ones). We notice that
h (S, Et) (S, c.) h (S, EI-1
and the computation can be made more efficient by using the following iterative update rule
G5:(1) = (1-
The parameter y E [0,1] can be regarded as forgetting coefficient: the closer y is to 1, the faster past information is dissipated. Finally, we model an attention mechanism as a function of the cumulated perceived stimulus, such that
(h (S) E1+1-
As we aim at defining a purely task-driven mechanism of attention, we determine the new fixations as the optimal ones for the resolution of the given task. We propose an approach that uses the loss signal of the task model t as an auxiliary signal to guide the attention mechanism. At each time step 4, we calculate
C(r (h (S)
and train a neural network (attention model) to minimize it by unblurring the relevant parts of the input using the foveation mechanism. Here, the input of the attention model is the internal agent state h(-) and the output is attention position &++1 R². Subsequently, the attention position is used by the foveation mechanism to calculate the next perceived stimulus T++1- At the first step, no attention position is available and we present the fully blurred stimulus to the attention model to generate the first attention position & The individual steps are shown in Fig. [ Here, the attention model is presented with the internal agent state ((())=1 (here h(-)t=1 = Tt=1 since the model has only performed a single fixation). Subsequently, the attention position ε== and perceived stimulus æt = are calculatod. Next, the internal representation is updated by linearly combining the old internal representation with the newly perceived stimulus.
During training of the attention model, h(-); is forwarded to the task model. The task model calculates loss value with respect to the downstrcam task. Lastly, the loss is backpropagated through the differentiable loveation mechanism to the allention model to update its weights. This solution is very efficient at inference time since the task model can be discarded and a scanpaths can be generated by iteratively predicting the next fixation using only the attention model. Additionally, this approach benefits from efficient implementations of neural networks on GPUs, which enable fast and parallelizable processing.
3 Experimental Setup
3.1 Datasets
To provide broad evaluation of the proposed method against baselines and competitors, selected collection of three different and well-established eye-tracking datasets. They differ in input characteristics and resolution, eye-tracking device, and stimulus semanttic. All three datasets were collected under the free-viewing condition in order to minimise the top-down (lask-driven) influence of the subjects.
The MIT1003 (Judd et al. 2009) datasct consists of 1003 natural indoor and outdoor scenes that include 779 landscape images and 228 portrait images with horizontal and vertical resolution of up 1024 pixels. The dataset. contains fixations of 15 subjects during 3 seconds of free-viewing.
7

---

Published in Transactions on Machine Learning Research (10/2022)
The Toronto Bruce & Tsotsos 2007) datasets consists of 120 images of outdoor and indoor scenes wit resolution of 681 x 511 pixels. In contrast to the other two datasets, large portion of the images in the Toronto datasets have no particular region of interest. The dataset contains fixations of 20 subjects during seconds of troevviewing.
The KOOTSTRA (Kootstra et al. 2011 dataset contains 99 photographs from categories with reso- lution of 1024 768 pixels. This includes symmetrical natural objects, animals in a natural setting, street scenes, buildings, and natural environments. The category of natural environment is over-represented, with respect to the other categories. The dataset contains fixations of 31 subjects during 5 seconds of free-viewing
3.2 Baselines and competitors
We restrict our choice of competitors to other unsupervised models, i.e., those models that do not learn attention mechanism directly from human data. We point out here that in fact, in this work, eye-tracking datasets were only used for evaluation purposes and not during the learning of model parameters.
Constrained Levy Exploration (CLE) (oocignooe & Ferraro 2004p. Scanpath is described a realization of a stochastic process. We used saliency maps by IItti et al. 1998) to define non-local transition probabilities. Langevin equation whose random term is generated by a Levy distribution, and a Mctropolis algorithm, are used to generate scanpaths. We use the python implementation provided by the authors in (Boccignone et al.) 2019a).
Gravitational Eye Movements Laws (G-EYMOL) Zanca et al. 2019b). Scanpath is obtained as the motion of a unitary mass within gravitational fields generated by salient visual features. We use the python implementation provided by the authors the original paper.
Winner-take-all (WTA) (Koch & Ullman 1987). We use saliency maps by (Itti et al. 1998) and implement the algorithm as follows. We generate the first fixation at the location of maximum saliency. We apply inlibitionn-o-return by canceling the saliency in a neighborhood of the attended location in ray of one degree of visual angle. The second fixation is selected as the location of maximum saliency in the updated map, and so on.
Additionally, some baselines were defined to better position the effectiveness of the approaches under inves- tigation against, e.g., well-known human vision biases.
Random baseline. Scanpaths are generated by subsoquently sampling fixation points from a uniform probability distribution defined over the entire stimulus area. At each time step, each pixel of the digitized stimulus has the same probability of being attended to, regardless of its position content.
Center bascline. Scanpaths are randomly sampled from 2-dimensional Gaussian distribution cen- tered in the image. We used the center matrix provided in (Judd al. 2009).
Human baseline. This baseline is used to assess how well a human subject is predictive of another subject's atnaltional dynamics. Thus, for all metrics and datasets considered this study, this baseline calculated as the average obtained by considering each of the subjects relative to the rest of the population.
Lastly, we compare all approaches to the proposed NeVA (Neural Visual Attention) method. We test two different NeVA configurations. For the first configuration, we classification task model to supervise the NeVA-based attention model during training (NeVAc). Here, use wide ResNet that was trained on the CIFAR10 dataset (Krizhevsky) 2009) from the Robust library. For the sccond configuration we use a reconstruction-tass model (NeVAR). In this case, we train a denoising autoencoder on the CIFAR10 dataset using the implementation proposed in (Zhang et al.) 2017) and visually inspected the results to ensure that the training was conducted correctly. For both attention models we use wide ResNets (Zagoruyko & Komodakis 2016). More details about the model configurations and training are given in Section of the supplement.
8

---

Published in Transactions on Machine Learning Research (10/2022)
3.3 Metrics
To obtain a quantitative measure of the similarity between two scanpaths, use two different metrics. Let A and B denote two agents, observing the same stimulus S during a certain period of time, and let 5 = .E......N and & = , be their respective scanpaths. We only consider scanpaths of the same length N.
3.3.1 String-edi edit distance (SED)
The string-edit distance (SED) (Jurafsky & Martin 2000), also known as Levenshtein distance, allows to quantify the distance between two strings (i.e., sequences of letters) as the number of operations needed (deletion, insertion and substitution) to transform one string into the other one. It has been adapted in the domain of visual attention analysis compare visual scanpaths (Brandt Stark 1997, Foulsham & Underwood] 2008). The stimulus is divided into n n equally spaced regions, each of them uniquely labeled with letter,
== CUS Sat
iE(I....a2)
where 5d: C S, 5 USC = 0i * j. Here, aj is the letter associated with the subset So: Let string be the function that converts scanpath to string by assigning each of the fixations the corresponding letter associated with the region the fixation belongs, i.e.
string(E) + (an Din);
where & E S wk E (1,..,N Then, we can compute the string-edit distance between two scanpath
SED (string($**) strringg
The string-edit distance has proven to be robust with respect to image resolution (provided the dictionary length fixed for all stimuli) (Choi et al. 1995..
3.3.2 String-based time-delay embeddings (SBTDE)
Previous research suggested as human responses to stimuli are not deterministic and people generally attend to different locations on the same stimulus. To take into account the stochastic component the attention process (Pang al. 2010), we would like incorporate in our evaluations invariance with respect to delays. To this end, we define metric that is based the concept of time-delay embeddings (Abarbanel et al. 1994), widely use in physics to compare stochastic and dynamic trajectories. Time-delay delay embeddings have already been adopted for the analysis of visual attention scanpaths (Wang et al. 2011 Zanca et al. 2019b By computing time delay embeddings, we can normalize the metrics with respect to the scanpath length. In order to make results robust with respect to changes in both stimulus resolution and scanpath length, we propose a modified version called string- based time-delay embeddings (SBTDE) that computes the time delay embedding the string domain. Again, let 5* and e be two scanpath of the same length to compared. Let [I.....N) N) be reference sub-sequence length, we denote with
= = [(S),.... (:)))))
the set of all possible sub-sequences of length contained in the scanpath gA. Analogously, we define E for t Then, we compute
de (x( min SSDDF((rrin((t))strinn) string(y)))
y
In other words, given any sub-sequence x of the scanpath gA, we look for the sub-sequence of t minimal string-edit distance. Finally, we compute the SBTDE, of order & as the average minimal distance, i.e.
SBTDE& 55 = Σ d(x,
9

---

Published in Transactions on Machine Learning Research (10/2022)
3.3.3 Metrics visualization
To better understand the predictive power of the models and baselines under examination, we plot metrics at all possible sub-sequence length.
To take into account the large variability between human subjects, we calculate two types of averages for the metries. For the first, which we will simply refer to the score mean, for each stimulus, we compute the mean score of the scanpath under examination relatively to all human scanpaths available for that stimulus and then compute the mean for all stimuli in the dataset. In the second case, we make use of the recently proposed ScanPath Plausibility SPP (Fahimi & Bruce, 2021). which considers only the human scanpath with the minimum distance for each stimulus, and then computes the mean for all stimuli the dataset. This metric demonstrates to be particularly useful cases where a large number of different strategics is implemented by different subject (e.g,, complex images with many candidate focal points).
Since longer scanpaths are more difficult to predict, the metries might vary lot with respect to the considered sub-sequence length. For this reason, we visualize the results with respect to the worst and best references, i.e., the Random and Humen baselines.
3.4 NeVA forgetting hyperparameter
In a preliminary experiment, we analyzed the influence of the forgetting parameter on the properties of the generated scanpaths on the MIT1003 dataset. For both NeVA configurations (classification and recoustruction supervision) setting 7 = 0.3 let to the best results in all metrics. Thus, we set y = 0.3 for the remaining experiments. A more detailed analysis is given in Section C of the supplement.
4 Results
In the following, we summarize the results of five different experiments used to evaluate the proposed NeVA method.
4.1
Simulating human visual attention under biologically-inspired constraints
The proposed NeVA attention model only considers the currently perceived stimulus π (c) to generate the next attention position. To evaluate the effect of this property we propose an alternative approach, where optimize the attention position & directly to minimize the loss of the task model and consider multiple foveation positions simultaneously to generate the next attention position. In this case, we first calculate the loss of the task model for the currently perceived stimulus. Then, we compute the gradient with respect to the associated attention position & and itcratively update the position using gradient descont. An efficient implereettation of this optimization algorithm is described in Section D of the supplement.
To now asses the impact of using only imperfect information we compared the scanpaths generated by the scanpath model with those generated by the scanpath optimization algorithm on the MIT1003 dataset. We use the two different downstream-lask models that were also used to train the NeVA attention models (NeVAc and NeVAR) for the NeVA optimization algorithm described above. Additionally, we tested classification and reconstruction task models that were trained on ImageNet for the optimization-bused NeVA algorithm (for all classification tasks we use pretrained models from the RobustBench library). We refer to these approaches as NoVA-Oc and NeVA-OR- Detailed information about the downstream-task models are given in Appendix D In our experiments on MIT1003, the three classification-bssed NeVA approaches achieve the lowest SBTDE distance for all scanpath lengths. Nevertheless, the trained attention model demonstrates approaches. The two CIFAR10 reconstruction-
better performance than the two optimization- approaches show similar metrics, while the ImageNet reconstruction-based approach is considerably worse than all other approaches. We refrained from training ImaggeNel-based NeVA attention models for the remaining experiments due to the following two reasons. Scanpaths that were generated with ImageNet- basod supervision showed a higher mean and SPP-SBTDE distance in our experiments than those created with CIFAR10-based supervision. Additionally, CIFAR10-base attention models are substantially faster at the inference time, since we can re-scale images to the relatively low training resolution of 32r32 before
10

---

Published in Transactions on Machine Learning Research (10/2022)
The scanpath metrics of the different configurations summarized in Figure & in Appendix 4.1 We discuss possible explanations for the differences wetween CIFAR10 and ImageNet-based models and the disadvantages of the optimization-basee approach in Section
processing.
4.2 Scanpath plausibility
We evaluated the similarity of scanpaths generated by NeVA to those of human subjects on three different datasets. For each stimulus, NeVA is used to generate a task-driven visual exploration scanpath. The generated scanpaths were compared with human scanpaths collected by eye-tracking. The performance of NeVA was compared with those of three state-of-the-ar unsupervised scanpath models (Boccignone & Ferraro 2004; Zanca et al.) 2019b Koch & Ullman 1987) of visual attention and three baselines (random, center, and human baselines). We generate scanpaths of length 10 (i.e., sequences of 10 fixations) with all models and baselines. We only consider the first 10 fixations of human scanpaths and discard recordings with less than 10 fixations. Nevertheless, we investigate the behavior of early fixations by analyzing the similarity of scanpaths over time. Table | summarizes the results for all metrics and datascts. Scanpaths generated by the NeVAc are best, on average over all metrics for all datasels. For individual metrics NeVAc is the pest-performing method in 11/12 metrics and the second-bes in the remaining 1. NeVAR is the second best method in 7/12 metries. G-Eymol is the second-best method in 4 metrics, while CLE is the best method in metric and the second-best method in metric (same score as G-Eymol). Figure 2 shows the similarity of artificially generated and human scanpaths for the three different datasets.
For the MIT1003 dataset, scanpaths generated by the NeVAc approach demonstrate the overall highest similarity to human scanpaths. Specifically for long scanpaths, the mean distance and SPP distance (distance to the scanpath of the closest human subject) are considerably lower than those of all other competitors. NeVAR is the second-best approach in terms of the SBTDE distance and is comparable with G-Eymol in the SED. The bottom-up models (G-Eymol and CLE) perform particularly well in the very first fixations. The scanpaths generated by all competitor approaches exhibit considerably higher similarity to human scanpaths than the two simple baselines (random, center) and the basic approach of WTA.
For the Toronto dataset, the results for the SED are similar to MIT1003. However, for this dataset CLE and G-Eymol show similar performance. The SBTDE distance is substantially lower for both NeVA approaches for all scanpath lengths compared to all competitors. Furthermore, for subsequence lengths of & < 9 the mean SBTDE distance of the NeVA approaches is smaller than the SBTDE distance between the human subjects. This might be due to the fact that being composed of generic outdoor and indoor images, the scanpaths in the Toronto dataset exhibit high variability. Hence a single subject is generally bad predictor of the remaining population, especially in the short term.
For the Kootstra dataset, NeVAc does not achieve the lowest SPP-SED for small number of fixations (< 8) but shows the best performance for long scanpaths compared to the other methods. Both NeVA approaches achieve a considerably lower mean SBTDE distance than all other approaches. In contrast to the other results, CLE demonstrates the lowest SPP-match SBTDE distance followed by the two NeVA-basod approaches with considerable gap to the other methods.
Example scanpaths generated with NeVA are shown in Fig. 3 approaches is given in Appendix H
A qualitative comparison with existing
4.3 Loss behavior during visual explorations
To determine the cause of the difference between classification and reconstruction based supervised models, we further analyzed the loss behavior of the downstream-tase -task models during scanpath generation. Note that we do not investigate the training loss, but rather how the loss changes for an already trained model during scanpath generation al inference time. Figure 4 demonstraaes that the loss of the classification-task model rapidly decreases after the first fixation and remains largely the same for the remaining fixations. In contrast, the loss of the reconstruction task model docreases more consistently. We argue that the considerable decrease in the loss of classification task models after the first fixation leads less exploration and shorter scanpaths. Equivalently, the slower decay of the loss in the reconstruction model leads to more exploration and longer
11

---

Published in Transactions on Machine Learning Research (10/2022)
NeVAc
NeVAR Center
G-Eymol
CLE
WTA
Random
Human
8
Mcan SED
Mean SED
Mean SED
6
6
4
2
2
2
3
7
8
9
10
4
5
6
7
8
9
10
2
3
4
5
10
Fixations
Fixations
Fixations
SPP SED
Best. SED
Best SED
O
3
2
1
2
3
6
7
8 9
10
4
5
6
7
8
9
10
2
3
1
5
9 10
Fixations
Fixations
Fixations
Mean SBTDE
0.9
Mean SBTDE
0.9
Mean SBTDE
0.9
0.8
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.6
0.4
0.4
0.5
0.3
0.3
0.4
2
8
9
10
6
3
10
Subsequence length (k)
Subsequence length (k)
Subsequence length (k)
0.8
0.7
SPP SBIDE
Best SBTDE
0.8
Best SBTDE
0.6
0.6
0.6
0.5
0.4
0.4
0.4
0.3
0.2
0.2
0.2
0.1
0.0
0.0
0.0
2
3
8
9
10
5
6
8
2
3
4
5
10
Subsequence length (k)
Subsequence length (k)
Subsequence length (k)
(a) MIT1003
(b) Toronto
(c) Kootstra
Figure 2: Similarity of human and artificially generated scanpaths from different methods assessed with 4 different distance metrics for the MIT1003 dataset. A lower score corresponds to a higher similarity to human scanpaths for each metric. The human baseline is calculated by comparing scanpaths between different subjects and can be considered as a gold standard.
12

---

Published in Transactions on Machine Learning Research (10/2022)
Table 1: Similarity of scanpaths generated by the proposed NeVA method and other competitors to those of humans for several metrics. A lower score in each metric corresponds to higher similarity to human scanpaths. The best results shown in bold and the second best results are underlined. The human column denotes the intra-scanpath distance between humans.
Datasets
NeVAc
NeVAn
G-Eymol
CLE
WTA
Center
Random
Human
MIT1003
Mean SED
4.3
4.49
4.48
4.60
4.90
4.99
5.09
3.74
SPP SED
3.15
3.49
3.39
3.41
4.15
4.26
4.44
1.65
Mean SBTDE
0.62
0.67
0.68
0.72
0.76
0.78
0.81
0.57
SPP SBTDE
0.51
0.57
0.59
0.60
0.67
0.71
0.74
0.23
Average
2.14
2.30
2.28
2.33
2.62
2.68
2.77
1.55
Toronto
Mean SED
4.22
4.42
4.52
4.56
4.74
5.05
5.19
3.72
SPP SED
3.35
3.71
3.79
3.74
4.17
4.51
4.71
2.28
Mean SETDE
0.58
0.64
0.69
0.72
0.73
0.82
0.85
0.64
SPP SBTDE
0.58
0.64
0.68
0.72
0.73
0.82
0.84
0.30
Average
2.18
2.35
2.42
2.44
2.59
2.80
2.90
1.74
Kootstra
Mean SED
4.75
4.67
4.99
5.08
4.26
SPP SED
2.98
3.26
3.12
3.12
3.66
3.75
3.88
1.16
Mean SBTDE
0.71
0.73
0.74
0.76
0.78
0.80
0.68
SPP SBTDE
0.43
0.48
0.51
0.41
0.53
0.58
0.60
0.20
Average
2.19
2.31
2.21
2.30
2.49
2.53
2.59
1.57
Figure 3: Example scanpaths generated by NeVA. The first row shows the original stimulus, the second row shows the loveation heatmaps and fixations, and the last row shows the internal representation after the last fixation.
13

---

Published in Transactions on Machine Learning Research (10/2022)
-10-2
30
0.030
25
0.025
Task model loss
20
0.020
15
0.015
0.010
5
0.005
0
0.000
0
2
8
2
4
8
Fixations
Fixations
(a) Classification task
(b) Reconstruction task
Figure 4: Box plots the task model loss during different fixations averaged over all datasets. The loss of a classification (a) and a reconstruction (b) task model shown. Note that the difference in loss magnitude between the two tasks is caused by the two different loss functions (categorical cross entropy for classification and mean square error for reconstruction).
scanpaths. It important to note that we do not expect decrease in loss necessarily correspond more human-like scanpaths. Otherwise, the aptimization-basee approach would generate more human-like scanpaths.
4.4 Application in downstream tasks
In real-world scenarios, scanpaths can be used to reduce computational overhead of downstream applications (e.g., by reducing the resolution of foveated image regions). To assess the usability of the different scanpath methods for possible application scenarios we explored their ability to generate scanpaths for classification downstream task on an unknown dataset with unseen classes. We evaluate the performance the different methods on subsets of the CIFAR100 and ImageNet datasets (1000 images each). To measure the accuracy, we use two different ResNet classifiers trained on the complete CIFAR100 and ImageNet datasets, respec- tively. Both datasets trained CIFAR10. The results summarized in Table 2 The overall best performance is achieved by the two NeVA-based algorithms followed by the G-Eymol method for both datasets. These methods also showed the best performance for creating human- scanpaths in Section 4.2) Note that NeVA achieves higher accuracy than all prior approaches, although it has only access to the current perceived stimulus while creating the scanpaths, while other methods use the real stimulus. This property makes NeVA more suitable in real-world applications, where only an imperfect version of the stimulus might be available to save computational resources. The bonsiderable accuracy degradation for the CIFAR100 dataset from clean images to partly blurred images indicates that the blurring of the images has considerable effect even for low-resolution images. We observed that the overall accuracy of the classifier monotonically increased with the number of fixations (see Appendix D. However, since the CIFAR100 and ImageNet classifier were not trainod to classify partially blurred images the accuracy converges only slowly to the value observed on clean images.
not by the downstream-task models used in the NeVA method that were
4.5 Influence of the predicted label
In the previous section, we observed quantitative differences between scanpaths generated with classification and reconstruction supervision. To further investigate the influence of the task model on the scanpaths, explored the impact of the predicted label of classification task model the quality of the generated
14

---

Published in Transactions on Machine Learning Research (10/2022)
Table 2: Accuracy of a classification task, where the input is a blurred version of the original input. The imput is soquentially deblurred using scanpaths from different methods and the accuracy is averaged over the scanpath. Both subsets of the CIFAR100 and the ImageNet dataset (1000 images each) are considered. Best results are shown in bold and second-best results are underlined. For the proposed NeVA method C indicates an attention model trained on classification downstream task and R an attention model trained on a reconstruction downstream task.
Dalasets
Standard
NeVAc
NeVAR
G-Eynnl
CLE
WTA
Center
Random
CIFAR100
57.40
39.68
38.23
37.95
30.99
25.92
33.41
25.66
ImageNet
75.50
61.03
58.89
55.95
52.96
47.65
55.72
54.07
scanpaths. It not immodiately clear how the predicted label influences the scanpath as the task model that was used for most experiments was trained on the CIFARIO dataset, which does not contain most of the objects present in the different evaluation datasets (MIT1003, Toronto, and Kootstra). First, we assessed how less or more fine-grained labels of a classification task model affect the scanpaths generated by the NeVA method. Therefore, we generated scanpaths with two different NeVA configurations for two different images of the MIT 1003 dataset. For the first configuration, we trained a NeVA al altetion model using a CIFARIO classifier as task model. For the second configuration, we used the NeVA optimization algorithm to generate scanpaths directly with an ImageNet classifier. Figure Jillutrrates that more fine- fine-grained labels can lead to less human-like scanpaths. The ImageNet, classifier predicts very specific labels for the objects in the scene. This leads to a focus of the attention on these models (i.e., "baseball player' in (b) and "dumbbell" in (d)).
Next, we assessed how changing the label predicted by the classifier affects the generated scanpaths of the NeVA optimization algorithm. SSegedy et al. (2014) demonstrated that neural networks exhibit the intriguing property that their predictions can be casily influenced by so-called adversarial perturbations that are not visible to human perception. We utilized adversarial perturbations to change the predicted label of the task model by applying them to the original stimulus. We then compared the scanpaths generated under adversarial attacks to normally generated scanpaths. Additionally, we investigate if the resulting scanpaths achieve worse performance in the classification downstream task proposed in Section 4.4 Note that we generale the scanpaths using the NeVA oplimization algorithm and not a NeVA-based attention model. This was done as the attention model is not dependent on the labels predicted by the task model during inference.
The adversarial perturbation is generated as follows. We aim to find a perturbation y that when added to the pixel values of the original stimulus S, lets the task model predict. a different label for the adversarial stimulus Sade = S.7. Here, we constrain the foo norm of gamma to || < E, so that it does not influence the human perception. We set 4/255 which was sufficient to reduce the accuracy of the classifier to below random guessing. To solve this optimization problem, use the Jitter attack with 100 iterations (Schwinn et al. 2021b).
Scanpaths that are generated for adversarial attacked stimuli demonstrate less similarity to human scanpaths in both analyzed metrics. The analysis of the scanpaths in the presence of adversarial attacks is shown in Figure 10 of Appendix F Moreover, the label change has a negative influence on the performance of generated scanpaths in an unknown classification downstream task. The accuracy is considerably reduced for both the subset of CIFAR100 (4.86%) and the subset of ImageNet (5.19%). After the attack, the simple center baseline achieves higher performance than NeVA on both datasets, although it does not consider any image information. The results indicate that the label has a considerable influence on the scanpath even if the originally predicted label does not make sense (this is the case for most of the MIT1003 images since the image labels do not overlap with the labels of CIFAR1O, which the NeVA models were trained on). This suggests that the model encodes label-spocific concepts in its predictions which affect the exploration in a substantial manner even for unknown objects. We further discuss this observation in Section An overview is given in Table 4 of Appendix F
15

---

Published in Transactions on Machine Learning Research (10/2022)
(a) CIFARIO
(b) ImageNet
(c) CIFARIO
(d) ImageNet
Figure 5: Scanpaths generated from the MIT1003 datasets with different NeVA configurations. For the first NeVA configurations attention model trained under the supervision of CIFAR10 classification model. For the second configuration scanpaths are directly generated using the NeVA optimization algorithm and an ImageNet classifier. The greedy optimization of scanpaths with the ImageNet model leads to locally constrained scanpaths. This is caused by the labels predicted by the ImageNett classifier. For the scanpath generated in (b) the model predicts "baseball player" and focuses only on the person with the hat. For the scanpath shown in (d) the model predicts 'dumbbell" and focuses mainly on the arm and object in the hand of the woman.
16

---

Published in Transactions on Machine Learning Research (10/2022)
4.6 Additional analysis
To gain additional quantitative insights into the scanpaths generated with NeVA, we conducted further analyses. In section G we observe that the NeVA algorithm exhibits greater similarity to humans in terms of the distribution of saccade amplitudes, followed by G-Eymol. Note that none of the models directly enforce a preference for the distance between successive fixations. NeVA models supervised by the reconstruction tasks tend to have wider explorations, as they manage to decrease the task model loss progressively with each fixation. A different observation is made for classification task-driven models, which minimizes the loss
already at the first fixation (see Figure 4 These loss results are shown in sections G and 4.3
5 Discussion
In this section, we discuss results from the previous section and relate them to existing work from the literature.
Top-down signal guidance explains human attention better than any existing bottom-up ap- proach. We conducted extensive experiments to compare NeVA with competitors (WTA, CLE, and G- Eymol) and baselines (Random, Center, and Humans). Unlike competing approaches that are mainly bottom-up, NeVA is driven solely by the signal obtained from the task model and scanpaths are gener- aled in order to maximize performance in a top-down manner. We found that lop-down guidance explains human scanpaths in terms of scanpath similarity considerably better than bottom-up approaches. However, in line with previous rescarch demonstrating bottom-up guidance in the very first phase of visual exploration, in the first few fixations, NeVA is often worse than existing bottom-up approaches. This observation moti- vates the combination of top-down guidance by neural networks and bottom-up up mechanisms in future work. The Center baseline did not perform much better than Random. This is surprising result if we consider that a center blob can predict saliency very well (Judd et al. 2009 Borji & Itti 2012). Additionally, the result reinforces the notion, that compared to saliency prediction, the modeling of a scanpath mechanism is a more complex problem. On the other hand, we could observe that Human baseline (obtained by compar- ing one subject each time with the rest of the population) largely outperforms each model, especially in the SPP-metrics. These observations have two important implications. First, there is still room for improvement in the computational modelling of visual exploration dynamics. Second, to some extent, it is possible to identify within the human set clustered diversity: the fact that the SPP-metric is significantly better than the mean metric suggests that, for every human, there is at least one human which explores the scene in a comparable way. A similar result was obtained by (Zangrossi et al. 2021). This clustered diversity needs to be further analyzed to lead to better and more diverse models. One solution could be the introduction of non-deterministic components into the models. Another approach could consist in modeling the mechanisms that lead to the diversity within humans with hyperparameters (e.g., exploration tendency, task) to simulate different persons and behaviors. The classification task consistently led to better per- formance in terms of scanpath similarity compared to the reconstruction task. This could strengthen the hepothesis that, even in free-viewing conditions, humans tend to be guided by objects (Finhäuser et. al. 2008 before considering any other information in the scene.
underlying
Imperfect information benefits the generated visual behaviors. In this work, we took special care on modeling imperfect vision, i.e., to ensure that the attention model never has direct access the original stimulus in its full resolution (but only through foveation). This is because, in addition to not having biological validity, the assumption of perfect vision could actually disfavor modeling by placing the artificial system in conditions different from the human ones it aims to imitate. The results presented in Figure g suggest that imposing biologically-inspired constraints, in fact, benefits the resulting generated behaviors. The optimized version of the model, ie., NeVA-O, selects the next fixation after having actually observed all possible locations the visual field. This resulted in a less plausible behavior when compared with humans. We claim that the optimized version of the model will avoid purely exploratory fixations which are needed by a system with no perfect information to minimize its uncertainty toward the environment.
17

---

Published in Transactions on Machine Learning Research (10/2022)
Top-down features versus task-driven approaches. The contribution of top-down features to at- tention modeling has been extensively investigated in the literature (Kummerer et al. 2017 Borji 2012). Independently, models have been proposed that instead implicitly learn an attention mechanism in order to solve a given task (Dosovitskiy et al. 2020 Xu et al., 2015) (with no relation to human attention modeling). To the best of our knowledge, however, NeVA is the first allempt in which the latter is designed to incorpo- rate biologically-insprred constraints, simulate human attention, and evaluated the mechanism in terms of similarity to human scanpaths.
Performance on resource-constrained real-world applications. The goal of computational modeling of visual attention is not only to improve our understanding of human mechanisms but to develop algorithms that can equip any measurement--onstrainne autonomous exploration system with similar capabilities, e.g., a space rover which needs to perform region-of-interes prioritize sampling (Bhattacharjee et al. 2021
In order to use an artificial attention mechanism in real-world applications, it has to be computationally efficient, i.c., by being subject to the iotogically-inspired constraint of fovcated vision or usable with low- resolution data. Otherwise, the computational cost. of generating the scanpath may negale any potential savings in processing resources. In contrast to existing work (Borji & Itti, 2012, Kümmerer & Bethge 2021 NeVA generates fixations by only considering the currently perccived foveated stimulus. To evaluate the usability of scanpaths in real-world (asks we propose a novel experiment where the utility of scanpaths to solve a downstream classification task is analyzed. This form of evaluation differs notably from previous work in the area of human visual attention, in which scanpaths were evaluated primarily on their similarity to human scanpaths. However, similar approaches can be found in the domain of photon-counting sensors where systems need to make decisions based on a partial version of the input. (Chen & Perona 2016). Our experiments showed that, although NeVA does not use the original stimulus for scanpath generation, it still outperforms prior methods in the downstream task of object classification with unknown object categories. Moreover, since the used NeVA model was trained with CIFAR10- based supervision it takes low-resolution images in 3R32x32 as an input instead of the full-resolution images. This further considerably reduces the computational requirements. In future work, joint training of both the task model and the NeVA attention model in an end-to-end fashion might further increase the utility of the generated scanpaths. This would prompt the task model to focus on important features in the image since would have to make predictions based solely on the currently perceived stimulus. In turn, attending the correct positions would become even more crucial for solving the task, which might improve the allention model.
On the relationship between deep neural networks, gradients, and human attention. Additional motivation for the proposed NcVA algorithm can be found in the rescarch area of explainable machine learning. Prior work showed that the gradients of machine learning models with respect to the input can be used to identify important features the data (Bachrens et al.] 2010 Simonyan et al. 2014 Sundararajan let al.] 2017 Furthermore, gradient-based attribution methods have shown to be aligned with human semantics (Selvaraju et al. 2017 Smilkov et al.] 2017). Besides, another line of rescarch showed that gradients of neural networks can be used to identify untrustworthy data (Schwinn et al.] 2021a) (e.g., semantically meaningless data such as noise or out-of-distribution data). These findings further motivate the use of gradients from a task model to guide visual attention in our approach.
Trade-off between correct and too specific (downstream) classification. First and foremost, cur- rent literature suggosts that a purely top-down description of visual attention is not biologically plausi- ble (Treisman & Gelade 1980 Duan & Wang 2015). This is not, in fact, goal of this paper, which is only aimed at demonstrating how, within the proposed evaluation scheme, top-down components have a promi- nent role in explaining the vast majority of attentional shifts. As we already suggested, an integration of bottom-up components in the current method is worth investigating and could bring a more comprehensive explanation of the phenomenon of attention. In fact, our experiments already highlight some limitations of top-down attention. Namely, as the proposed attention mechanism is highly dependent on the task model that guides it, it also inhorits its limitations. In our experiments, the supervision of an ImageNet- basod classifier (i.e., trained to recognize objects among 1000 classes) lead to highly class-specific scanpaths that explored only highly specific regions of the respective images. An ImageNet classifier must learn to distin-
18

---

Published in Transactions on Machine Learning Research (10/2022)
guish 1000 different classes that may have semantic similarities (e.g., multiple dog breeds). We argue that the classifier thus learns to focus on descriptive local features and the scanpath becomes highly specific. In contrast, CIFAR10-based classifiers learn more high-level features as the classes in CIFARIO are semantically more different. This problem might be circumvented by considering task models that can learn more compre- hensive representations Lions of the world. This includes object delectors like YOLO (Redmon et al.) 2016) that can detect multiple objects at the same time and thereby should not saturate as fast as classifier- based task models (see Figure D.. Other options could be multi-label classification tasks (Dembczynski et al.) 2010), models that perform hierarchical classification (Marszalek & Schmid, 2007) and can attend broader and fine- grained concepts at the same time, or a weighted combination of different tasks. Lastly, we demonstrated that attention models can generalize well even if the supervising task model does not contain the concepts present in the image. This hypothesis was further strengt by the fact, that changing the predicted label of the task model had a considerable influence on the quality of the generated scanpaths in terms of similarity to human scanpaths and applicability in downstream tasks.
Study implications and limitations. The key takeaway of this work is that adding biologically-inspired constraints to neural networks makes them explore images in more human-like manner. We believe that this finding can be integrated into existing approaches that model human attention. Moreover, it motivates further studies on other biological constraints and if adding them to neural networks further improves the alignment between artificial and human scanpaths. In this first work, we focus on foveated vision as it is present at the very first steps of the vision 'pipeline'. We believe that having limited access to visual input is crucial to develop a plausible attention mechanism. For the sake of a clear interpretation of the results, we have limited ourselves to this component in the first experiments. However, it is of great interest to extend it to other biological constraints (c.g., eye-fixation duration) in future work.
References
Henry DI Abarbanel, TA Carroll, LM Pecora, JJ Sidorowich, and L Sh Tsimring. Predicting physical variables in time-delay embedding. Physical Review E, 49(3):1840, 1994.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz Paz. Invariant risk minimization. CoRR, abs/1907.02880 2019.
Mare Assens, Xavier Giro-i Nieto, Kevin McGuinness, and Noel E O'Connor. Pathgan: Visual scanpath prediction with generative adversarial networks. In Proceedings of the European Conference Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 11:
1803-1831, 2010.
Marcus Barkowsky, Jens Bialkowski, Björn Eskofier, Roland Bitto, and André Kaup. Temporal trajectory aware vidco quality measure. IEEE Journal of Selected Topics in Signal Processing, 266) 2009.
Protim Bhattacharjee, Martin Burger, Anko Boerner, and Veniamin I Morgenshtern. Region-of-interest prioritised sampling for constrained autonomous exploration systems. arXiv preprint 217.:2207.071866
2021.
Giuseppe Boceignone and Mario Ferraro. Modelling gaze shift a constrained random walk. Physica A: Stalistical Mechanics and ils Applications, 331 (1-2):22007722 201-218,, 2004.
Giuseppe Boccignone, Vittorio Cuculo, and Alessandro D' Ammelo. How to look next? data-driven approach for scanpath prediction. In International Symposium Formal Methods, pp. 131-145. Springer, 2019a.
Giuseppe Boccignone, Vittorio Cuculo, and Alessandro D'Amelio. Problems with saliency maps. In Inter- national Conference on Image Analysis and Processing, pp. 35-46. Springer, 2019b.
Ali Borji. Boosting bottom-up up and top-down visual features for saliency estimation. In 2012 IEEE conference on computer vision and poltern recognition, pp. 438 445. TEFE, 2012.
19

---

Published in Transactions on Machine Learning Research (10/2022)
Ali Borji. Saliency prediction in the deep learning era: Successes and limitations. IEEE transactions pollern analysis and machine intelligence, 43(2):679-70M) 2019.
Ali Borji and Laurent Itti. Stale-of-the-ar. visual attention modeling. IEEE trunsactions on paltern analysis and machine intelligence, 35(1):185 207, 2012.
Ali Borji and Laurent Itti. Cat2000: A large scale fixation dataset for boosting saliency research. arXiv preprint Xrx: 500.00881, 2015.
Stephan A Brandt and Lawrence W Stark. Spontancous eye movements during visual imagery reflect the content of the visual scene. Journal of cogreilive neuroscience, 9(1):27 38, 1997.
Andreas Bringmann, Steffen Syrbe, Katja Görner, Johannes Kacza, Mike Francke, Peter Wiedemann, and Andreas Reichenbach. The primale fovea: structure, function and development. Progress in retinal and eye research, 66:49 84, 2018.
Neil Bruce and John Tsotsos. Attention based on information maximization. Journal of Vision, 7(9):950-995 2007.
Bo Chen and Pictro Perona. Vision without the image. Sensors, 16(4):484, 2016.
Yun S Choi, Anthony D Mosley, and Lawrence W Stark. String editing analysis of human visual search. Optometry and vision science: official publication of the American Academy of Optometry, 72(7):439 451, 1995.
Charles E Connor, Howard E Egeth, and Steven Yantis. Visual attention: bottom-up versus top-down. Current biology, (4(11):8800-FF)) R880, 2004.
Abhishek Das, Harsh Agrawal, Larry Zitnick, Devi Parikh, and Dhruv Batra. Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding, 163:90 100, 2017.
Krzysztof Dembezynski, Weiwei Cheng, and Eyke Hüllermeier. Bayes optimal multilabel classification via probabilistic classifier chains. In Proceedings of the 27th International Conference on Machine Learning (ICML), pp. 279 286. Omnipress, 2010.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255 2009.
Misha Denil, Loris Bazzani, Hugo Larochelle, and Nando de Freitas. Learning where to attend with deep architectures for image tracking. Neural computation, 24(8):2151 2184, 2012.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXio:2010. 11929, 2020.
Haibin Duan and Xiaohua Wang. Visual attention model based on statistical properties of neuron responses. Scientific reports, 5(1):1-10, 2015.
Wolfgang Finhäuser, Merrielle Spain, and Pietro Perona. Objects predict fixations better than early saliency. Journal of vision, 8(14):18 18, 2008.
Ramin Fahimi and Neil DB Bruce. On metrics for measuring scanpath similarity. Behavior Research Methods, 53(2):609-626 2021.
Tom Foulsham and Geoffrey Underwood. What can saliency models predict about eye movements? spatial and sequential aspects of fixations during encoding and recognition. Journal of vision, 8(2):6-6, 2008.
Hadi Hadizadeh and Ivan V Bajić. Saliency-aware video compression. IEEE Transactions on Image Pro- cessing, 23(1):19 10-13, 2013.
20

---

Published in Transactions on Machine Learning Research (10/2022)
Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao. SALICON: reducing the semantic gap in saliency prediction by adapting deep neural networks. In 2015 IEFE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 262 270. IEEE Computer Society, 2015. doi: URL
2015. 38
Laurent Itti and Christof Koch. Computational modelling of visual attention. Nature reviews neuroscience, 2(3):194-28 203, 2001.
Laurent Iui, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on pottern analysis and machine intelligence, 20(11):1254 1259, 1998.
Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. Salicon: Saliency in context. Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 1072-1080, 2015.
Tilke Judd, Krista Ehinger, Frédo Durand, and Antonio Torralba. Learning to prodict where humans look. In 2009 IEEE 12th international conference computer vision, pp. 1106-2113, IEEE, 2009.
Daniel Jurafsky and James Martin. Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition, 2000.
Christof Koch and Shimon Ullman. Shifts in selective visual attention: towards the underlying neural circuitry. In Malters of intelligence, pp. 115 141. Springer, 1987.
Gert Kootstra, Bart de Boer, and Lambert RB Schomaker. Predicting eye fixations complex visual stimuli using local symmetry. Cognitive computation, 3(1):223-24) 2400, 2011.
Alex Krizhevsky. Learning multiple layers features from tiny images. Technical report, 2009.
Matthias Kümmerer and Matthias Bethge. State-of-the in human scanpath prediction. arXiv preprint arXiv:2002.m 12239 2021.
Matthias Kümmerer, Thomas SA Wallis, and Matthias Bethge. Deepgaze ii: Reading fixations from deep features trained on object recognition. aix preprint 1rXi::1610.0163, 2016.
Matthias Kummerer, Thomas SA Wallis, Leon Gatys, and Matthias Bethge. Understanding low-and high-level contributions to fixation prediction. In Proceedings of the IEEE international conference on computer vision, pp. 4789-4798, 2017.
Hugo Larochelle and Geoffrey E Hinton. Learning to combine foveal glimpses with a third-order boltzmann machine. Advances in neural information processing systems, 23, 2010.
Yan Luo, Xavier Boix, Gemma Roig, Tomaso Poggio, and Qi Zhao. Foveation-based mechanisms alleviate adversarial examples. arXiv preprint xxX1v:1511.06292, 2015.
David Marr. Vision: A computational investigation into the fuman representation and processing of visual information. MIT press, 2010.
Marcin Marszalek and Cordelia Schmid. Semantic hicrarchics for visual object recognition. In JEEE Com- puter Sociely Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, 2007.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. Advances in neural information processing systems, 27, 2014.
Aude Oliva and Antonio Torralba. Building the gist of a scene: The role of global image features in recognition. Progress in brain research, 155:23-30 36, 2006.
Derek Pang, Tatsuto Takeuchi, Kouji Miyazato, Junji Yamato, Kunio Kashino, al. A stochastic model of human visual attention with a dynamic bayesian network. Xiv preprint ar 100 004.0005.. 2010.
21

---

Published in Transactions on Machine Learning Research (10/2022)
Joanna Pilarczyk and Michal Kuniecki. Emotional content of an image attracts attention more than visually salient, features in various signal-to-noise ratio conditions. Journal of Vision, 14(12):4-4, 2014.
Dale Purves, George J Augustine, David Fitzpatrick, William Hall, Authony-Samuel LaMantia, and Leonard White, Neurosciences. De Boeck Supérieur, 2019.
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 779-788. IEEE Computer Society, 2016.
Nicolas Riche, Matthicu Duvinage, Matei Mancas, Bernard Gosselin, and Thierry Dutoit. Saliency and hu- man fixations: State-of-the-arl and study of comparison metrics. In Proceedinggs of the TEFE international conference on computer vision, pp. 1153- 1160, 2013.
Miguel Fabian Romero Rondon, Dario Zanca, Stefano Melacci, Marco Gori, and Lucile Sassatelli. Hemog: A white-box model to unveil the connection between saliency information and human head motion in virtual reality. In 2021 IEEE International Conference Artificial Intelligence and Virtual Reality (AIVR), pp. 10-18. IEFE, 2021.
Anthony Chad Sampanes, Philip Tseng, and Bruce Bridgeman. The role of gist scene recognition. Vision research, 48(21):2275 2283, 2008.
Leo Schwinn, An Nguyen, René Raab, Leon Bungert, Daniel Tenbrinck, Dario Zanca, Martin Burger, and Björn M. Eskofier. Identifying untrustworthy predictions in neural networks by geometric gradient analy- sis. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, UAI 2021,
volume 161 of Proceedings of Machine Learning Research, pp. 554-864, AUAT Press, 2021a.
Leo Schwinn, René Raab, An Nguyen, Dario Zanca, and Bjoern M. Fsskffir. Exploring misclassifications of robust neural networks to enhance adversarial attacks. CoRR, abs/2105. abs/2105.10304, 2021b.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Pro- ceedings of the IEEE international conference computer vision (ICCV), pp. 118-626. IEEE Computer Society, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference on Learning Representa- lions, ICLR 2014, Workshop Track Proceedings, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, abs/1706.00825, 2017. URL httphtttp//axiii http://xww...oogab////////////// arg/ 03825
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, 1CML, volume 70 of Proceedings of Machine Learning Research, pp. 3319-3328. PMLR, 2017.
Christian Szegedy, Wojeiech Zaremba, Nya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR, 2014.
Benjamin W Tatler, Roland J Baddeley, and Tain D Gilchrist. Visual correlates of fixation selection: Effects of scale and time. Vision research, 45(5):6433655 659, 2005.
Anne M Treisman and Garry Gelade, A feature-inteeraation theory of attention. Cognitive psychology, 12(1): 97-136, 1980.
Manish Reddy Vuyyuru, Andrzcj Banburski, Nishka Pant, and Tomaso Poggio. Biologically inspired mech- anisms for adversarial robustness. Advances in Neural Information Processing Systems (NeurIPS), 33: 2135-22 1135-2146, 2020.
22

---

Published in Transactions on Machine Learning Research (10/2022)
Wei Wang, Cheng Chen, Yizhou Wang, Tingting Jiang, Fang Fang, and Yuan Yao. Simulating human saccadic scanpaths on natural images. In CVPR 2011, pp. 441-448. IEER, 2011.
Zifeng Wu, Chunhua Shen, and Anton van den Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. CoRR, abs/1611.10000, 2016. URL http://arxiv http://wwrr/abb/111....
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption gemeration with visual attention. Tn International conference on machine learning (ICML), pp. 2048 2057. PMLR, 2015.
Tong Yubing, Faouzi Alaya Cheikh, Fahad Fazal Elahi Guraya, Hubert Konik, and Alain Trémeau. A spatiotemporal saliency model for video surveillance. Cognitive Computation, 3(1):241 2011.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference, BMVC. BMVA Press, 2016.
Dario Zanca and Marco Gori. Variational laws of visual attention for dynamic scenes. Advances in Newral Information Processing Systems (NeurIPS), 30, 2017.
Dario Zanca, Marco Gori, and Alessandra Rufa. A unified computational framework for visual attention dynamics. In Progress in Brain Research, volume 249, pp. 183-188. Elsevier, 2019a
Dario Zanca, Stefano Melacci, and Marco Gori. Gravitational laws of focus of attention. IEEE transactions on pattern analysis and machine intelligence, 42(12))2983-29)) 2995,, 2019b.
Dario Zanca, Marco Gori, Stefano Melacci, and Alessandra Rufa. Gravitational models explain shifts on human visual attention. Scientific Reports, 10(1):1-9, 2020a.
Dario Zanca, Stefano Melacci, and Marco Gori. Toward improving the evaluation of visual attention models: a crowdsourcing approach. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. TEFF, 2020b.
Andrea Zangrossi, Giorgia Cona, Miriam Celli, Marco Zorzi, and Maurizio Corbetta. Visual exploration dynamics are low-dimensional and driven by intrinsic factors. Communications biology, 4(1):1 14, 2021.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang, Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising. IEFF Transactions Image Processing, 26(7):3142-3155) 2017.
A Model Training
We trained multiple different neural networks in the scope of this work. The training hyperparameters and choices are given in the following sections.
A.1 Denoising autoencoder
We trained two denoising autoencoders (Zhag et al. 2017 using the implementation proposed (Zhang et al. 2017) on the CIFAR10 (Krizhevsky) (2099) and ImageNet datasets (Deng et al. 2009). These models were used to supervise the training of attention models described in Section 2.2.3 for the optimization algorithm described in Section 4.1 We used 17 layers and set the number of output channels of the first layer to 64. We used an Adam optimizer with learning rate of 1 = 0.001 for both datascts. This was found to be the best. value for CIFARIO, as found by a small grid search for I E {0.0001, 0.001, 0.013. We trained the networks for 100 and 5 epochs for CIFAR10 and ImageNet, respectively. This was sufficient for the convergence of the loss in both cases. For both datasots, we used batch size of 100. For CIFAR10 we injected noise from a Gaussian distribution with standard deviation of 0.15 and for ImageNet we used a standard deviation of 0.3.
23

---

Published in Transactions on Machine Learning Research (10/2022)
Table 3: Similarity scanpaths generated by the proposed NeVA method and other competitors to those of humans for several metrics. Results are normalized such that the average score between humans equals and the average score between humans and random scanpaths equals 0. A lower score in each metric corresponds to higher similarity to human scanpaths. The best results are shown in bold and the second best results are underlined.
Datasets
NeVAc
NeVAR
G-Eymol
CLE
WTA
Center
MIT1003
Mean N-SED
0.41
0.55
0.50
0.58
0.88
0.92
SPP N-SED
0.51
0.63
0.57
0.57
0.90
0.93
Mean N-SBTDE
0.22
0.41
0.45
0.64
0.77
0.87
SPP N-SBTDE
0.55
0.67
0.70
0.73
0.86
0.93
Average
0.42
0.57
0.55
0.63
0.85
0.91
Toronto
Mean N-SED
0.30
0.42
0.44
0.49
0.67
0.89
SPP N-SED
0.42
0.56
0.56
0.55
0.78
0.90
Mean N-SBTDE
-0.4
0.0
0.14
0.35
0.39
0.85
SPP N-SBTDE
0.48
0.61
0.69
0.76
0.77
0.95
Average
0.20
0.37
0.46
0.53
0.65
0.90
Kootstra
Mean N-SED
0.50
0.60
0.47
0.78
0.89
0.88
SPP N-SED
0.63
0.73
0.65
0.69
0.94
0.93
Mean N-SBTDE
0.28
0.44
0.50
0.69
0.77
0.87
SPP N-SBTDE
0.55
0.68
0.76
0.51
0.80
0.94
Average
0.49
0.61
0.59
0.67
0.85
0.91
A.2 Attention models
We choose wide ResNets with a width of 10 and depth of 28 as the architecture for the attention models (Wu et al., 2016). All models were trained for 200 epochs with a batch size of 256. In our experiments, the training of all attention models converged before the maximum amount of epochs was reached. We trained all models with an Adam optimizer and multiple learning rates E (0.0001 0.001, 0.01) and used the models with the lowest loss achieved in training for the remaining experiments. However, we observed only negligible differences in the different learning rates for all attention models. For all models, we used 10 fixation steps during training and updated the weights for every fixation.
B Scanpath Generation Normalized Results
Throughout the paper unnormalized results are shown for all metrics that compare human and artificial scanpaths. This was done to make the results comparable to other work. In Figure b and Table 3 we show normalized results, which are easier to interpret. The metrics are normalized such that the intra-human distance equals and the distance between human and random scanpaths equals 1. More specifically, for each score(m, k) obtained by the model for sub-sequence length k, we consider the normalized score
N-score(m, k)
score(m, k) score uman, k) score( Random, k) score( Maore(Huane Human, k)
We denote the normalized SDE by N-SDE and the normalized SBTDE by N-SBTDF.
24

---

Published in Transactions on Machine Learning Research (10/2022)
NeVAc WTA
NeVAR Center
G- G-Eymol
CLE
Random
Human
1.0
1.0
1.0
Mean N-SED
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
2
3
4
5
6
10
5
6
7
8
10
1
2
3
4
5
10
Fixations
Fixations
Fixations
1.0
1.0
1.0
SPP N-SED
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
2
3
4
5
6
9
10
5
6
7
8
10
2
3
4
5
10
Fixations
Fixations
Fixations
Mean N-SBTDE
1.0
1.00
1.0
0.75
0.8
0.8
0.50
0.6
0.25
0.6
0.4
0.00
0.4
0.25
0.2
0.50
0.2
0.0
0.75
0.0
2
4
5
6
10
5
6
7
8
10
2
3
4
5
10
Subsequence length (k)
Subsequence length
(k)
Subsequence length (k)
1.0
1.0
1.0
SPP N- 1-SIIDE
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.1
0.2
0.2
0.2
0.0
0.0
0.0
2
3
8
9
10
1
2
3
4
9 10
2
6
7
8
9
10
Subsequence length (k)
Subsequeut uence length (k)
Subsequence length (k)
(a) MIT1003
(b) Toronto
(c) Kootstra
Figure 6: Similarity of human and artificially generated scanpaths from different methods assessed with different distance metrics on 3 datasets. Each row corresponds to different metric and each column to a different datasct. A lower score corresponds to higher similarity to human scanpaths for each metric. The human baseline is calculated by comparing scanpaths between different subjects and can be considered as gold standard (metrics are normalized such that the average distance is 0 and 1 for humans and random fixations, respectively).
25

---

Published in Transactions on Machine Learning Research (10/2022)
C Influence of the Forgetting Hyperparameter
We investigated the inftuence of the forgetting hyperparamoter y the similarity between the generated scanpaths and human scanpaths and the distribution of the saccade amplitudes. Figure 7 summarizes the influence of on the normalized mean SED. Note that all configurations lead to the same first two fixations as NeVA docs only have access to the blurred stimulus for the first fixation and only uses the currently perceived stimulus for the second fixation. Afterward, past information is considered and the value y has an influence on the generated fixations. For the classification supervised NeVAc attention model the influence of y considerably increases in the later fixations. For the NeVAR configuration this effect is less distinct. Furthermore, the best value for y changes during the scanpath for both configurations. Averaged over all metries, 0.3 resulted in the highest similarity between NeVA and human scanpaths in our experiments.
0.0
0.1
== 0.2
7= 0.3
7 0.4
7 0.5
0.6
0.7
7= = 0.8
7
= 0.9
7= 1.0
0.500
0.60
Mean N N-SED
0.475
Mcan N-SED
0.58
0.450
0.56
0.425
0.54
0.400
0.52
0.375
0.50
0.350
0.48
0.325
0.46
0.300
0.44
1
7 8 90
2
3 4
5 6 7 910
Fixations
Fixations
(a) NeVAc
(b) NeVAR
Figure 7: SED of different NeVA configurations for the MIT1003 dataset. A lower score corresponds to a higher similarity human scanpaths. Metrics are normalized such that the average distance and 1 for humans and random fixations, respectively.
D NeVA optimization algorithm
Instead of training an allention model to predict scanpaths for given stimulus NeVA can be implemented as an optimization algorithm. Here, the scanpaths are directly calculated by using the loss of the task model and iteratively updating the attention position for each fixation. The optimization for each attention position can be solved quickly by proposing several possible attention position candidates evenly distributed over the image at each step. For each candidate, the loss value of the task model is then computed to find the best attention position. Given a sufficient number of candidates, this implementation eliminates the need for iterative updates. As a result, only one forward pass is required to compute an atlention position for a given task model. Quantitative differences between optimization- NeVA-O and training- based NeVA are given in Figure
E Application in downstream tasks
Complementing the results in Table 2 Figure 9 illustrates the accuracy over the number of fixations for the different datasets and attention models.
F Influence of the predicted label
Figure 10 and Table Table4summn summarrze the influence of the predicted label on the quality of the generated scanpath. In both cases, the label is changed with adversarial attack the clean image. Figure 10 illustrates that
26

---

Published in Transactions on Machine Learning Research (10/2022)
CIFAR1O NeVAC
CIFAR10 NeVAR
CIFAR1O NeVA-Oc
CIFAR10 NeVA-OR
ImageNet NeVA-Oc
ImageNet NeVA-OR
0.7
0.8
Mean N-SBTDE
0.6
SPP K-SBTDE
0.7
0.5
0.4
0.6
0.3
0.5
0.2
0.4
0.1
0.3
0.0
0.2
6
10
8
10
Subsequence length (k)
Subsequence length (k)
Figure 8: SBTDE distance of human and artificially generated scanpaths of different NeVA configurations for the MIT1003 dataset. A lower score corresponds to a higher similarity to human scanpaths. Metrics are normalized such that the average distance is 0 and 1 for humans and random fixations, respectively,
NeVAc
NeVAR Center
G-Eymol Random
CLE
WTA
60
40
Aceuracy [%)]
Accuracy [%)]
40
20
20
2
4
8
10
2
4
10
Fixations
Fixations
(a) CIFAR100
(b) ImageNet
Figure 9: Accuracy of a classification task, where the input a blurred version the original input. The input is sequentially deblurred using scanpaths from different methods and the accuracy shown over the different fixations of the scanpath. Both subsets of the CIFAR100 and the ImageNet dataset (1000 images each) are considered. For the proposed NeVA method C indicates attention model trained classification downstream task and R an attention model trained on reconstruction downstream task.
27

---

Published in Transactions on Machine Learning Research (10/2022)
Human
Random
NeVA-Oc
NeVA-Oc Adversarial
Mean N-SBTDE
1.0
SPP N- N-SBTDE
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
1
2
3
5
6
7
8
9
10
2
3
4
5
6
7
9 10
Subsequence length (k)
Subsequence length (k)
Figure 10: Similarity of human and artificially generated scanpaths for adversarially attacked images and normal images for the MIT1003 dataset. A lower score corresponds to higher similarity to human scanpaths for the two metrics. The human bascline is calculated by comparing scanpaths between different subjects and can be considered as a gold standard (metrics are normalized such that the average distance is 0 and 1 for humans and random fixations, respectively).
Table 4: Accuracy of a classification task, where the input is blurred version of the original input. The input is sequentially deblurred using scanpaths from the NeVA method and the accuracy is averaged over the scanpath. Both subsets of the CIFAR100 and the ImageNet dataset (1000 images cach) are considered. For the column NeVA attacked, the images where attacked by an adversary to change the class label used for supervision during scanpath generation.
Datasets
NeVA
NeVA Attacked
Center
CIFAR100
35.79
30.94
33.41
ImageNet.
56.12
50.93
55.72
the similarity to human scanpaths decreases if the label is altered, while Table a decline in accuracy in a classification downstream task for the attacked NeVA algorithm.
G Saccade amplitude analysis
The scanpath metrics used in the paper do not directly convey information about the nature similaritics and differences between the scanpaths of the different models (e.g., do they generally attend a variety of objects, perform high amplitude saccades etc.). To explore the priors that the different scanpath methods have on the properties of the generated scanpaths, we performed additional analysis investigate the saccade amplitudes of the generated scanpaths and compared them to the saccade amplitude distribution of humans. The saccade amplitude is defined the angular distance traveled by the eye during saccadic movement. To compute it, we convert the distance in pixels to centimeters, by considering resolution and size of the presentation screen used during data collection. Then, taking into account the distance of the subject from the screen, we convert the distance in centimeters to degrees visual angle by applying trigonometric rules. This analysis is particularly interesting as all methods do not explicitly constrain the scanpaths to have hunan-like characteristics. Thus, this experiment offers insights into what mechanisms lead to human-like saccades. Figure [1 compares the saccade amplitude of the scanpaths of the different approaches. Scanpaths of trained attention models show higher total saccade amplitudes than scanpaths that are directly optimized with the task-models for both tasks and all datascts. Furthermore, scanpaths generated under reconstruction- lask supervision are generally longer than scanpaths generated by classification-las supervision. In addition, we calculated the distribution saccade amplitudes between different fixations for all models. The results are summarized in Figure [2 Humans exhibit. a peak around 2 degrees of visual angle, in accordance with prior work (Tatler et al. (2005). The NeVA configurations also generate saccades in the same range of amplitudes, but clearly show second peak around the value of 10 degrees. Among competitors, G-Eymol
28

---

Published in Transactions on Machine Learning Research (10/2022)
NeVAc Ne
NeVAn
NoVA-On
125
Total saccade amplitude [9]
100
75
50
25
0
MIT1003
Kootstra
Toronto
dataset
Figure 11: Boxplots of total saccade amplitudes different NeVA configurations for the three considered datasets.
exhibits a similar pattern as humans, in accordance with prior results (Zanca et al. 2020bb. CLE and WTA algorithms, instead, produce wider saccades, and saccade amplitudes appear be normally distributed, Random and Center baselines have quite different and unnatural behaviors, with the first one having high prevalence of shorter saccades.
In Figure the the saccade amplitude distributions for different values of are shown. For both NeVA attention models (trained with classification and reconstruction supervision) lower values of gammaa lead to a higher similarity to the human saccade amplitude distribution. We observed that setting gamma to values close to zero will substantially reduce the saccade amplitude between later fixations. Thus, the number of saccades between 10 and 15 degrees is reduced, which leads a higher similarity to the human saccade amplitude decrease in saccade amplitude between later fixations for humans with scanpaths of length 10. We will explore this phenomenon in more detail in future work.
distribution. We could not
Overall, the optimized version of NeVA using classification as a downstream task exhibited the lowest Kull- back Leibler (KL) divergence between its distribution of saccade amplitudes compared to those of humans. However, the other approaches also showed a considerably lower KL divergence than the center and random bascline. This result surprising as none of the approaches contains priors that explicitly induce this be- havior. We further did not observe correlation between the similarity of saccade amplitudes and the other metrics used to assess the similarity of the scanpaths. Both aspect needs to be investigated further in future work, for example by studying the dynamics of saccade amplitudes over time.
H Qualitative analysis
Here we report a qualitative comparison between different methods. Figure 14 exemplifies how NeVA im- proves upon prior approaches. In the first example, the bird is hardly visible against the background.
29

---

Published in Transactions on Machine Learning Research (10/2022)
NeVAc
NeVA-Oc
NeVAR
NeVA-OR
G-Eymol
CLE
WTA
Center
Random
Human
-10
-10-2
-10-2
6
6
Probability
Probability
Probability
4
4
4
2
2
2
0
0
0 5 10
15
20
25 30
0
10
15
20
25 30
0 5 10
15
20
25
30
Saccade amplitude [9]
Saccade amplitude [9]
Saccade amplitude []]
(a) Proposed NeVA
(b) Prior work
(c) Baselines
Proposed
NeVAc 0.38
NeVA-Oc 0.143
NeVAR 1.1
NeVA-OR
0.52
Prior work
G-Eymol 0.73
CLE 0.65
WTA 0.32
Basclines
Center 1.16
Random 1.3
(d) Kullback-Labbee--divvrgeee
Figure 12: (a-c) Saccade amplitude distributions of scanpaths generated by different methods (c) Kullback- Leiblee-divergence between the saccade amplitude distributions of scanpaths generated by different methods and the saccade amplitude distribution of human scanpaths. Shown results are averaged over all datasets.
30

---

Published in Transactions on Machine Learning Research (10/2022)
Human
7 1.0
? 0.9
y 0.8
? 0.7
7 0.6
7 7= 0.5
7
0.4
7 0.3
7 0.2
7 0.1
7.0.0
-002
-10
6
6
Probability
4
Probability
4
2
2
0
0
10
15 20
30
10
15 20
30
Saccade length
Saccade length
(a) Classifier supervision
(b) Reconstruction supervision
Figure 13: Saccade amplitude distributions of scanpaths generated with different values of the forgetting hyperparameter Shown results are averaged over all datasets.
(a) NeVA
(b) G-Eymol
(c) CLE
(d) NeVA
(e) G-Eymol
(f) CLE
Figure 14: Example scanpaths generated by NeVA and prior approaches. The first row shows the original stimulus, the second row shows the foveation heatmaps and fixations, and the last row shows the internal representation after the last fixation.
Bottom-up approaches thus have hard time detecting relevant features. In the second example, existing approaches fail to attend the head of the foreground object.
31


Reviews:
[{"review_id": "tppKZhzo9Q", "reviewer": "Reviewer_uam2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\nThe paper addresses a very interesting problem of task guided visual attention, which links scanning patterns of foveated regions to a high level task such as image recognition or reconstruction. The experiments are extensive and the results are encouraging. Further, the paper is reasonably well written, with key ideas and technical components well illustrated. \n\n**Weakness**\n\nMy main concern is the similarity to a prior work from Mnih et al., which is unfortunately not discussed in the paper. Mnih et al. also presented a visual attention model that (1) considers task guidance; and (2) uses foveated local regions (i.e., imperfect information as described in this paper). More specifically, their model builds a recurrent neural network that learns to attend to a sequence of foveated local regions guided by the task of image recognition, where visual attention can be learned without eye tracking data. While their paper focused on recognition, their model is very similar (both conceptually and operationally) to the proposed method. For example, their method also generates scanpaths that can be compared against eye tracking data. It remains unclear to me how the proposed method is different to Mnih et al., diminishing the key technical contribution of this paper. \n\n\nIndeed, I have to point out that the paper missed several related works in computer vision. The authors motivated the work by pointing out two major limitations of existing models on visual attention: (1) the lack of top-down task guidance; and (2) the assumption of full resolution image inputs (i.e., perfect information). However, both have been addressed in prior works. For example, Gao et al. presented a top down visual attention model guided by scene recognition. Mnih et al. and Xu et al. presented models that attend to a series of foveated local regions for image recognition / image captioning. Again, it is not entirely clear how the proposed method adds value to those existing approaches. \n\n\nWhile the experiments are extensive, the design does not seem to align with the motivation of the paper. And some of the results are confusing. \n* The main experiments are carried out using eye tracking data during free-viewing of images. Task-guided and free-viewing scanpaths are known to have different characteristics. Free-viewing scanpaths can focus on different regions of the same image, and have a much larger variance across subjects, as also recognized by the authors.  Comparing task guided scanpaths (e.g., recognizing objects / scenes) from a model to free-viewing scanpaths from humans seems rather problematic. A higher similarity in the scanpaths does not necessarily suggest the effectiveness of the method. For example, in one of the experiments (Figure 2 row 3 column 2), the model generated scanpaths show higher similarity to humans than other human scanpaths. It seems to me that these evaluations are insufficient to justify the proposed method. \n* The visual recognition results are confusing. The text suggested that “the downstream task model is trained on CIFAR10” and “Both (ImageNet and CIFAR100) are not seen by the task model.\" It seems that the task / NVA model has to be fine-tuned on ImageNet and CIFAR100, as the object categories are different and the image resolutions vary (between CIFAR and ImageNet). However, there is no clear description of the fine-tuning process. Further, the drastic performance drop between (a) using full resolution image and (2) using a sequence of foveated local regions is not well justified. As NVA will deblur the image along the scanpaths, with sufficient fixations, the model could see all pixels in the full resolution image. Further, if the scanpaths did capture the main object in the first few fixations, the model might as well make a correct prediction. It is not clear why such a performance drop still exists. It will be interesting to report the recognition accuracy as a function of the number of fixations. \n\n*References*\n* Gao, D., Han, S., & Vasconcelos, N. (2009). Discriminant saliency, the detection of suspicious coincidences, and applications to visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(6), 989-1005.\n* Mnih, V., Heess, N., & Graves, A. (2014). Recurrent models of visual attention. Advances in neural information processing systems, 27.\n* Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., ... & Bengio, Y. (2015, June). Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning (pp. 2048-2057). PMLR.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presented a computational model of top down visual attention, realized as a deep neural network. Inspired by the human vision system, the proposed model – Neural Visual Attention (NVA), selectively attends to a sequence of foveated, local image regions, and extracts information necessary for a high level task (e.g. recognition or reconstruction). In doing so, NVA produces a scanpath of an input image guided by the target task, without using the supervision of eye tracking data. To evaluate the resulting scanpaths, the authors also presented a new metric called String-based time-delay embedding (SBTDE), extending the string editing distance and accounting for scanpaths with varying lengths. The proposed method was evaluated on three public image datasets, where the model generated scanpaths are compared against eye tracking data. The generated scanpaths can be further used for visual recognition and demonstrate promising results. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The main requested changes include\n* It is suggested to draw a clear boundary to Mnih et al. and compare the results to Mnih et al. in experiments.\n* Please discuss prior works in computer vision \n* It will be great to evaluate the proposed model on eye tracking datasets where the subjects are performing a set of tasks.\n* Please clarify the visual recognition experiments and justify the performance drop.\n\nHere are some minor comments regarding the presentation and organization of the paper. \n* Abstract: Extensive experiments show that outperforms … -> Extensive experiments show that our method outperforms …\n* Intro (Page 3 second paragraph): , tend to overfit … ->, they tend to overfit …\n* Intro (Page 3 second paragraph): , we focus on unsupervised ->. We focus on …\n* Intro (Page 3 last paragraph): Differently by previous proposal -> Different from previous proposals, …\n* Intro (Page 3 last paragraph): to expands the action space -> to expand the action space\n* Intro: It is suggested to separate out the introduction and the related work section. \n", "broader_impact_concerns": "There is no ethical concerns for this paper. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MdaKJjt2KI", "reviewer": "Reviewer_qd44", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n* The introduction, motivations and discussions of the work are well written.\n* The proposed model seems to be sound and correct (apart some clarifications to be dealt with).\n* In-depth experimental analysis, comparing with other methods for different datasets and evaluation metrics.\n\nWeaknesses\n* Almost all reported results are in relative format. To encourage other scientists to use this method, I'd suggest to use absolute numbers.\n* The model might be computationally expensive, since the modified version of the full image is processed multiple times (as many times as fixations).\n* Some parts of the paper are hard to follow (especially how the new fixation point is generated).\n* Results using ImageNet models are not convincing.\n\nPlease see \"requested changes\" for more information about how to improve the weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method to generate visual scanpaths in a top-down matter which is focus on incorporating the signal from high-level visual tasks such as image classification and reconstruction.\n\nContributions:\n* Incorporate the task guidance into the attentional modeling \n* Use partial information (e.g., foveated) to generate scanpaths\n* Extension of existing metrics to account for stochastic components in human behaviour ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Clarity of the paper\n* Abstract. “[…] without incorporating the signal of a high-level visual tasks that have shown to partially guide human attention.“ I’d suggest to add some motivation about why it’s important to have visual tasks to guide attention. It’s written in the introduction already, but it’s also important to have a sentence here in the abstract too.\n* Introduction. “When predicting saliency, the temporal dimension is lost, and the scanpath generation problem still remains unsolved.” It’s not clear why temporal order is important for this task. One could consider the points of scan as independent points and compute saliency maps. Could you please provide an example to motivate this claim?\n* Fig 1 in page 2 is distracting, when reading the introduction. Please move after Sec. 2.2. \n* Sec. 2.2.3, first equation. It’s a bit hard to imagine the effect of this equation. I’d suggest to add a short description of what intuitively is happening and what is the outcome of such operation. \n* Page 5. “the categorical-cross-entropy loss for the classification task and the mean-squared error loss for the reconstruction task.” Please be clear that these are two different networks, otherwise the reader can thing that you trained for both combined loss.\n* Sec. 3.3.2. “To take into account the stochastic component of the attention process” I suggest the authors to expand a bit the meaning of this sentence. I know that there is a reference, but it’d be great to have a sentence describing what stochastic complement means.\n* Sec. 3.3.2. “to make them robust with respect to changes in stimulus resolution and scanpath length” To me it’s not clear how the proposed metric is more robust wrt (Wang et al., 2011; Zanca et al., 2019b). Can you provide more insight and maybe an example?\n\nTechnical correctness\n* Introduction. The authors claim 2 limitations of current approaches. It would be interesting to see a comparison with some of existing works that incorporate task guidance and use partial information (foveation). Some examples are [A1, A2], but there might be others that are more recent.\n* Second equation in page 5 and last of page 6. What’s \\alpha? Is it the agent a? If so, please fix notation. Or clarify it’s definition.\n* Sec 2.2.2. The method at every step t works on the full smoothed image. However in practice, humans do not have access to the “full image” when looking at something (given the limited field of view). I’m wondering if it would be possible to extend the method to deal with sub-regions of the image instead of full images. This is also important from the computational perspective, because now the method have less pixels to compute at each time t. I’d like to either see a discussion about this and/or an experiment showing that the method works under these more realistic conditions.\n* Page 7. It’s quite difficult to understand how the attention model is defined and how the next fixation is generated. Is there a way to expand this part and have it in math form? Alternatively, it would be appreciated if the authors try to explain what model it is and how the outputs are generated for a given input. It’s also confusing because training and prediction part are mixed. So it’s unclear what happens in the two cases.\n\n[A1] Larochelle, Hugo, and Geoffrey E. Hinton. \"Learning to combine foveal glimpses with a third-order Boltzmann machine.\" Advances in neural information processing systems 23 (2010).\n\n[A2] Denil, Misha, et al. \"Learning where to attend with deep architectures for image tracking.\" Neural computation 24.8 (2012): 2151-2184.\n\nExperiments\n* Page 8. “we use a wide ResNet that was trained on the CIFAR10 dataset“ Isn’t  this already at very low resolution (32x32)? What is the effect of blurring if the images are already quite low resolution?\n* Sec. 4.1. It’s unclear why the CIFAR10 models are much better than ImageNet models. It makes sense to me that higher resolution models should be more beneficial to capture details in the fovea, when compared to the low resolution of CIFAR10. Maybe the ImageNet model did not converge well during training? Or maybe there is a big domain shift between datasets? I’d like to see an argumentation about this point. \n* Sec. 4.2. “We only consider the first 10 fixations of human scanpaths and discard recordings with less than 10 fixations”. Why can’t we use this data with less fixations? One can run the evaluation by varying the length of fixations in the test set (e.g., only data with 1, with 2, …). This would give us a better understanding what happens with examples with shorter scanpaths too.\n* Sec. 4.2 and Table 1. “To simplify the interpretation of the results all metrics are normalized such that the intra-human distance equals 0 and the distance between human and random scanpaths equals 1.” I’d really suggest the authors to provide absolute numbers instead of relative. This is to encourage comparisons with future paper and make it easier. \n* Table 2. Even if it is not very relevant to this experiment, I’d suggest to add the best accuracies on the datasets of the current state of the art methods. It’s not for direct comparison, but to show the reader where we are with attentional methods compared the best results for the task.\n* Fig. 4. It’d be interested to see the qualitative results of NeVA scanpaths compared to the second best performing method.", "broader_impact_concerns": "No concerns about broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eVATbgDKNV", "reviewer": "Reviewer_n3FM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is strong in the following aspects:\n- The model is more biologically plausible than previous models in two key aspects: modeling the availability of visual information available to the model during the sequence of eye-fixations, and also, the eye-fixations are driven by top-down signals related to a visual task.\n- Simplicity of the model. While the model is trying to mimic in more detail the human visual system than previous works, the model remains intuitive and allows to be effectively trained.\n- Experiments not only try to show improvements from previous works, but also investigate properties of the model such as the effect of the specificity of the label and the task that the model is trained on.\n\nThe main weaknesses at a general level are the following (below in \"requested changes\" I explain them in more detail):\n- Positioning of the work: It is unclear why the focus of this work is on the biologically plausibility of the partial information available during the sequence of eye-fixation and the top-down signal that drives them, and not other key aspects of visual attention (eg. eye-fixation duration, inter-subject variability, free viewing vs other tasks, etc.). In other words, the aspects that this work focuses are interesting, but it is not clear why the authors have chosen to study these aspects and not other aspects.\n- Implications of the work: While developing accurate models of the brain is important for understanding human vision and for a number of applications, it is unclear what can be derived from this model besides that it works better and that is more biologically plausible than previous models. \n- Internal representations: The proposed internal representation to accumulate the observed image scan-path is not biologically plausible. The integration is done at the stimulus level rather than with some form of working memory neural representation. Thus, while the paper makes progress introducing more biologically plausible aspects to the model, it also introduces some components that are not biologically plausible.\n- Partial information vs full: It is unclear why the proposed model with partial image information performs better than models with access to the full image. \n- Some technical aspects of the experiments are unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new model to predict the sequence of human eye-fixations in an image. The model predicts the position of the next eye-fixation by using only the previously foveated image regions, unlike previous works that use the information of the entire image, even from image regions that have not been foveated. Also, the model uses a visual task such as object recognition or image reconstruction as training objective, while previous works used the prediction of the sequence of eye-fixations as the model's objective. In this way, the model introduced in this paper models how top-down information drives the sequence of eye-fixations. Comparisons with several baselines demonstrate the effectiveness of the model.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Positioning of the work: I think the paper would benefit from mentioning what aspects remain not biologically plausible. This could done  by motivating why the paper focuses on the partial information and the top-down signals, and by explaining that there is a number of aspects that remain unexplored and why would be interesting to explore them in the future building from this work.\n- Implications of the work: Some discussion of whether something new about human vision has been learned from this work and how it is expected that this model will help understanding human vision in the future or developing new applications.\n- Internal representations: A discussion about the biologically plausibility of the internal representation would be beneficial to understand this critical component of the model. It seems to me that the proposed internal representation is not biologically plausible. It would be awesome to see some experiments on different alternatives for this component. For example, what is the relation with [1].\n- Partial information vs full: What is the intuition behind the result that shows that a model with partial image information obtains better results than a model with full image information? The latter has access to the same information as the former, so in principle the latter has the capability of making the same predictions as the former. I may be not fully understanding the baselines in section 4.1, and some clarification may be necessary.\n\n-----------\n\nTechnical aspects: I think the following points require the attention of the authors:\n\n* Page 3, end of first paragraph: Zhao et al., 2015 introduces a dataset not a model, the right citation is [2].\n* Perfect information / Imperfect information: These terms are not precise.\n* I think \\alpha is never defined\n* Is the pre-training of ResNet used in the classification task an advantage over the ResNet trained from scratch in the reconstruction task? Also, is the pre-trained ResNet from RobustBench somehow trained to be robust to adversarial examples, or what is the reason to use RobustBench?\n* Loss behavior in section 4.3: would increasing the learning rate improve the convergence speed of the reconstruction task? If so, would the authors expect that the eye-fixation scan-path be more accurate? It seems that the answer would be 'yes' after reading section 4.3, but this is unclear, why would convergence speed be related to final accuracy?\n* The accuracy of the best model in ImageNet is 61.03%. Is this top-1? I think some comment comparing this with the accuracy of ResNet would help understanding this result in a broader context.\n* In the paragraph before 4.6 there is a typo: \"W we\"\n* Analysis of the scan-paths in the presence of adversarial examples: the paper misses the take-away of the experiment. The paper explains some of the technical details, but I am not sure what is the take-away of the result in Figure 7. I think it would be best to briefly mention the take-away in 4.5.\n* I think it would be helpful to provide some comparison with a method that directly trains to predict the sequence of eye-fixations, in order to assess the impact of training the model to classification or reconstruction, specially given that this is one of the main novelties of this work.\n* In A.1 it is mentioned that the training is stop when the loss converges, while in A.2 the models are trained for 200 epochs. I think it would be best to explain these differences.\n\n[1] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. In NIPS’14\n\n[2] Xun Huang, Chengyao Shen, Xavier Boix, Qi Zhao, SALICON: Reducing the semantic gap in saliency prediction by adapting deep neural networks. ICCV'15\n\n\n", "broader_impact_concerns": "I think there are no broader impact concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Official recommendations are split. Reviewer n3FM remains unclear about the central claims of the work and recommends rejection. Reviewer qd44 leans towards accept but recognizes n3FM's concerns -- especially regarding the positioning of the work. I agree the positioning could be stronger, but do not think it constitutes a unsubstantiated claim. It is the AE's opinion that the primary claims are clearly stated and sufficiently supported by experiments and that the issues with positioning could be addressed with a minor revision.

For me to recommend acceptance, the claims on biologic plausibility or the model having a "biological constraint" will need to be weakened to reflect that these are not claims about the inner workings of biological systems or that the proposed methods are apt analogs to biological systems. I point authors to the last paragraph of the claims section for further advice on the requested changes. 
Decision: Accept (poster)