FoveaTer: Foveated Transformer for Image Classification
Authors: Aditya Jonnalagadda
William Yang Wang
B.S. Manjunath
Miguel Eckstein
Venue: ICLR 2022 Submitted (2022)

Abstract: Many animals and humans process the visual field with varying spatial resolution (foveated vision) and use peripheral processing to make eye movements and point the fovea to acquire high-resolution information about objects of interest. This architecture results in computationally efficient rapid scene exploration. Recent progress in vision Transformers has brought about new alternatives to the traditionally convolution-reliant computer vision systems. However, the Transformer models do not explicitly model the foveated properties of the visual system nor the interaction between eye movements and the classification task. We propose foveated Transformer (FoveaTer) model, which uses pooling regions and eye movements to perform object classification tasks using a vision Transformer architecture. Our proposed model pools the image features using squared pooling regions, an approximation to the biologically-inspired foveated architecture, and uses the pooled features as an input to a Transformer Network. It decides on subsequent fixation locations based on the attention assigned by the Transformer to various locations from previous and present fixations. The model uses a confidence threshold to stop scene exploration, dynamically allocating more fixation/computational resources to more challenging images. After reaching the stopping criterion, the model makes the final object category decision. We construct a Foveated model using our proposed approach and compare it against a Full-resolution model, which does not contain any pooling. On the ImageNet-100 dataset, our Foveated model achieves the accuracy of the Full-resolution model using only 35% transformer computations and 73% overall computations. Finally, we demonstrate our model's robustness against adversarial attacks, where it outperforms the full-resolution model.

Under review as a conference paper at ICLR 2022
FOVEATER: FOVEATED TRANSFORMER FOR IMAGE CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Many animals and humans process the visual field with varying spatial resolu- tion (foveated vision) and use peripheral processing to make eye movements and point the fovea acquire high-resolution information about objects of interest. This architecture results in computationally efficient rapid scene exploration. Re- cent progress in vision Transformers has brought about new alternatives the traditionally convolution-reliant computer vision systems. However, the Trans- former models do not explicitly model the foveated properties of the visual sys- tem nor the interaction between eye movements and the classification task. We propose foveated Transformer (FoveaTer) model, which uses pooling regions and eye movements to perform object classification tasks using vision Transformer architecture. Our proposed model pools the image features using squared pool- ing regions, an approximation the biologically-insiree foveated architecture, and uses the pooled features as an input to a Transformer Network. It decides on subsequent fixation locations based on the attention assigned by the Trans- former to various locations from previous and present fixations. The model uses a confidence threshold to stop scene exploration, dynamically allocating more fix- ation/computationall resources to more challenging images. After reaching the stopping criterion, the model makes the final object category decision. We con- struct a Foveated model using our proposed approach and compare it against a Full-resolution model, which does not contain any pooling. On the ImageNet- 100 dataset, our Foveated model achieves the accuracy of the full-resolution model using only 35% transformer computations and 73% overall computations. Fi- nally, demonstrate our model's robustness against adversarial attacks, where it outperforms the full-resolution model.
INTRODUCTION
Many mammals, including humans, have evolved a locus (the fovea) in the visual sensory arry with increased spatial fidelity and use head and eye movements (Land) 2012 Marshall et al. 2014) orient such locus regions and objects of interest. The system design allows visual-sensing organisms to accomplish two objectives: fast target detection crucial for survival and savings in the computational cost. Computational savings are accomplished by limiting the number of units with high computational costs (i.e., higher spatial resolution processing) to the fovea's 's small spatial region. Fast target detection is achieved by distributing the remaining computational power across a much larger area in the periphery, with a lower spatial resolution with increasing distance from the fovea. Critical to the design is an efficient algorithm to guide through eye movements the high- resolution fovea to regions of interest using the low-resolution periphery (Hayhoe & Ballard (005) Strasburger et al. 2011 Ludwig et al., 2014) and allow optimizing the target detection and scene classification. Various computational models were proposed to model the search using foveated visual system (Yamamoto et al. 1996; Prince et al., (2005).
Computer vision has evolved from using hand-crafted features data-driven features in modern CNNs. Due to their computational limitations, the objectives of the computer vision systems align well with those of human visual system: optimize visual detection and recognition with effi- cient computational and metabolic footprint. Approaches towards saving computational power can be seen; for example, computer vision systems evolved from using sliding windows RCNN's Gir-

---

Under review as a conference paper at ICLR 2022
shick et al. 2014) use of selective search and Faster-RCNN's (Ren et al. 2015) use of Region Pro- posal Network (RPN).
A system that mimics the human vision by processing the scene with foveated system and rational eye movements has also been proposed. This approach to exploring the scene can be seen in models like RAM (Mnih et al. 2014) for recognizing handwritten single-digits or detecting objects (Akbas & Eckstein (017)) where they sequentially process the image and decide what to process next by using the peripheral information. These foveated models approach that of full-resolution models but using a fraction of the computations. Foveated systems have also shown to result in more robust-
ness
(Luo al. 2015 Deza et al. 2019) Deza & Konkle 2020 Kiritani & Ono. 2020 Vuyyuru
et al.
2020 against adversanal attacks.
There has been recent innovation in computer vision of using Transformers TTuvron al. 2020 Dosovitskiy et al. 2020) for object classification tasks which departs from the traditional over- reliance on convolutions. Even after replacing the convolutions with attention modules and mul- tilayer perceptrons, visual Transformers (Dosovitskiy et al. 2020)] Touvron al. (020) achieve close to state-of-the-art performance on the ImageNet dataset and provide better robustness against adversarial attacks (Shao et al.) (20TT)
Due to the flattened architecture of the transformers, it easier for multi-resolution features to share the same feature channels. Transformers (Vaswani et al. 2017) have the added benefit of self-attention, which facilitates the interaction of various parts of the image irrespective of distance. No papers have evaluated the additional potential gains of incorporating foveated architecture into Vision transformers for the task of ImageNet classification.
Here, evaluate the effect of a foveated architecture and sequential eye movements on state of the art transformer model sitting on a convolutional backbone. We compare the foveated transformer relative to the full-resolution model in terms of classification accuracy and robustness adversar- ial attacks. Furthermore, we extend previous papers by investigating the separate contributions to
robustness related to the foveated processing (Luo et al. 2015 Deza et al. 2019 Deza & Konkle
2020 Kiritani & Ono. 2020 Vuyyuru et al. 2020) and that of the process of fixating images at different locations.
To this end, we perform an object classification task using multiple fixations, moving foveal attention across different parts of the image, and using only limited portion of the image information at each fixation, thereby reducing the input to the transformer by many folds. The model decides on subsequent fixation locations using the learned self-attention weights and fixation locations of all the fixations until the current step. Finally, the model integrates information across fixations using average-pooling to make the final classification decision.
2 RELATED WORK
Transformers have achieved great success in the field of Natural Language Processing since their introduction by Vaswani et al. (2017) for machine translation. Recently, the application of Trans- former models in computer vision has seen tremendous success. Vision Transformer (ViT) model introduced by Dosovitskiy et al. (2020) achieved remarkable performance on ImageNet (Deng et al., 2009) by using additional data from JFT 300M (Sun et al.) 2017) private dataset. Subsequently, the DeiT model (Touvron et al.) 2020) introduced knowledge transfer concepts in transformers to lever- age the learning from existing models. Using augmentation and knowledge transfer, the DeiT model achieved close to state-of-the the-art performance using training data from the ImageNet dataset alone.
Sequential processing provides three main advantages in computer vision. Larochelle & Hinton (2010) proposed a model based on the Boltzmann machine that uses foveal glimpse and can make eye movements. First, can limit the amount of information to be processed at given instant to be constant, i.e., the ability to keep computations constant irrespective of the input image size. Second, sequential models can help model human eye movement strategies and help transfer that information to build better computer vision systems. RAM (Mnih et al.) 2014) introduced sequential model capable of making a sequence of movements across the image to integrate information before clas- sification. In addition, the hard-attention mechanism, implemented using reinforcement learning, was used to predict the sequence fixation locations. Ba and Minh (Ba et al.) (013)) extended these ideas to recognize multiple objects in the images dataset constructed using MNIST. Third,
2

---

Under review as a conference paper at ICLR 2022
quential processing requires fewer parameters than model using full-resolution image input. Other models (Xu et al. 2015) have proposed image captioning models based on both hard-attention and soft-attention. Additionally, the spatial bias introduced into CNNs due to padding (Alsallakh et al. 202T can be overcome using sequential models (Tstssss
2011
Computational models of categorization and eye movements have been proposed for rapid cat- egorization in terms of low-level properties such as spatial envelopes (Oliva & Torralba (2001)
and texture summary statistics (Rosenholtz al., 2012). Saliency-based models (Koch & Ullman
1987 [tti et al. 1998) Iuti & Koch| (000) traditionally tried to model eye movements by identifying
bottom-up properties in the image that will capture attention. Torralba et.al (Torralba et al., 2006)
showed how saliency could be combined with contextual information to guide eye movements.
Low-resolution periphery and high-resolution central fields are integrated with saliency to predict human-like eye-movements (Wloka et al. (018). Akbas (Akbas & Eckstein 2017) implemented mologically-innpiree foveated architecture (Freeman & Simoncelli, 201 1D with a deformable parts model to build foveated object detector which accuracy was close to full-resolution model but using a fraction of the computations. Spatial transformer networks (Jaderberg et al.] 2013) were used with foveation to improve object localization using foveated convolutions (Harris et al. 2019) and achieve better eccentricity performance (Dabane et al. [021).
FoveaTer combines an approximation to pallooiicall-innsiiee foveated architecture with Vision Transformer Network. We apply our model to real-world images from the ImageNet dataset for the task of image classification. We pool the input the transformer using the pooling architecture described in the following sections based on the fixation location, which reduces the number of inputs to the transformer. The subsequent fixation location given by confidence map, constructed using the attention weights from the last transformer block. Feature vectors corresponding to the class token from different fixations are averaged using an average- saraageppooling layer followed by a fully connected layer, resulting in the final classification decision.
novel aspect of the proposed work is that the model also learns that all images are not equally dif- ficult to classify adapting the exploration eye movements to different image classes and thus varying computational resources used to classify different images successfully. The model implements this idea using confidence threshold to restrict the scene exploration to the necessary fixations to clas- sify the image.
Also novel is an evaluation the adversarial robustness of our model to understand the separate con- tributions of the foveated architecture and that of sequential fixations towards defense against adver- sarial attacks. We use the fast gradient sign method (FGSM (Goodfellow et al. (2014)) which is an adversarial attack that computes the adversarial image by backpropagating the gradients once, and the projected gradient descent method (Kurakin al. 2017 Madry et al. 2018), which iteratively computes the adversarial image, as our adversarial attacks.
MODEL
DeiT-Tiny (Tuvron et al., 2020): The DeiT-Tiny architecture begins with a convolution embed- ding layer that transforms the [3, 224, 224] input image into a [192, 14, 14] representation that is input into series of twelve transformer blocks, each sized for a 192-dimensional embedding.
Full-resolution model: Our Full-resolution model refers hybrid adaptation, transformer net- work with convolutional backbone, of the DeiT-Tiny architecture using the same number of param- eters. This hybrid model replaces the single convolutional layer of the DeiT-Tiny model with a convolutional backbone. The convolutional backbone is composed of first two stages of the ResNet- 18 (He et al. (016) architecture, which transforms the input from [3. 224, 224] to [128, 28, 28), followed by three convolutional layers to acheive final feature map of size [192, 14, 14). The first convolutional layer transforms the input feature dimensionality from 128 to 192 using kernels of size 3 and stride 1, resulting in [192, 28, 28). The second convolutional layer downscales the fea- tures using a kernel of size 3 and a stride of 2, resulting in features of size [192, 14, 14]. The third convolutional layer uses a kernel size of and stride of 1, retaining the size of the feature map. The number of transformer blocks is reduced from 12 to 9 to match the number of parameters with the DeiT-Tiny model.
3

---

Under review as a conference paper at ICLR 2022
Foveated model: Extending the hybrid architecture described above, our Foveated model analyzes a pooled-version of the features from the convolution backbone at each time step and arrives at the final category decision after making the required number of time steps to satisfy the decision crite- rion. We show the architecture of our model in Figure 1. The input image is first passed through the convolutional backhone described above, resulting in size (192, 14, 14]. Then, dependent average-pooling, using the "Foveation module" (see next section), on the features from the convolutional backbone resulting in features of size [192, 29], where the fixation refers one of the 196 possible spatial locations in the input feature map. Under this non-uniform average- pooolingg model, locations closer the fixation location use smaller neighborhoods for pooling than loca- tions far from the fixation location. Pooled features of size [192, 29] along with the class token are passed through the nine transformer blocks, where class token refers to a learnable vector of size 192 values. Since the transformer blocks retain the dimensionality of the input, the output of the transformer blocks has the same size of [192, 30]. We use the self-attention weights, corresponding to the class token feature vector from the last transformer block, in the "Accumulator module" (see next section) to predict the following fixation location. Feature vector corresponding to the class token is given input to the average-poolin pooling layer, averaging across such vectors received from all the fixations. Finally, the classification layer transforms the average-pooled feature vector into logits vector.
perform fixation
Input Image
Class token
Convolution backbone
Concatenate
Position embeddings
Nth Fixation
Foveation module
Transformer blocks 0-7
Transforme block 8
Input
Attention Weights
Average pooling across fixations
Accumulator
Classification layer
Network output
(N+1)th fixation location
Figure I: FoveaTer architecture: Solid black arrows denote the flow of image-related features. The foveation module performs fixation-dependent pooling, the full-resolution features (1192, 14, 14]) output of the convolution backbone, resulting in pooled feature vectors ([192, 29)). During training, random location in the image serves as the initial fixation. Accumulator uses the attention weights from the last transformer block to predict the next fixation location. The Average-poolinn layer takes the feature vector corresponding to the class token input from each past fixation, followed by final classification layer.
3.1 FOVEATION MODULE
Inspired from the radial-polar pooling regions proposed in Freeman & Simoncelli (2011). we use squared pooling regions in this paper for computational speed-up. Each image in a mini-batch has a corresponding fixation location, where the fixation location represents the center of the visual field shown in Figure 2, thereby allowing us to align the input image/feature map with the visual field. After aligning the input feature map with the visual field, features falling within pooling region
4

---

Under review as a conference paper at ICLR 2022
Fixation locations
Feature máps
Pad & Align all feature maps
1x1 pooling
3x3 pooling
5x5 pooling
7x7 pooling
Extract pooled features at pooling centers
Remove outside pooling centers
Pad equal sequence length
Visual field
Transformer Input
Figure 2: Foveation module: Inputs to the module are aligned to the visual field and are zero-paceee pero-padded have the same size as the visual field. We pool the input features according the pre-defined pooling neighborhoods in the visual field, resulting in one feature vector for each of the 49 pooling regions in the visual field, colored blocks in the visual field figure correspond the centers of these 49 pooling regions. After ignoring the pooling centers falling in the padded region instead of on the original image, the remaining pooled features are zero- paro-padded a constant sequence length of 29.
average average-pooled, and the resultant pooled vector represents that pooling region. Highlighted regions in [ggure/[correspone to the centers of the pooling regions, 49 blocks.
We use pooling regions with receptive field sizes 1 x 1. 3 x 3, 5 x 5, 7 x 7 blocks on feature map of size 14 14 blocks, which is the output of the convolution backbone. Each block corresponds to [16, 16] pixel region in the input image of width and height 224. The visual field is of size 27 x 27 blocks. Central green average-pooling
block represents the high-resolution Fovea, where there is no The next ring of pooling regions, where the center of the pooling region is blue, has a receptive field of 3 3 which translates to average-pooling 9 feature vectors to generate the representative feature vector for that pooling region. Similarly, the rings of violet and red-colored pooling centers have receptive fields of 5 x 5 and 7 x blocks, respectively. Yellow squares are of size 14 14 blocks, i.e., equal to the feature map of the image, thereby illustrating the size of the image size to the visual field, which is approximately one-quarter.
We ignore the pooling regions falling outside the image, and the output of the foveation module is zero-padded to maintain the same length across all images present in a batch since different fixation locations give rise to an unequal number of active pooling regions. We use 29 the feature length as it is the maximum possible number of active pooling regions. The feature vector corresponding each pooling region is computed as follows,
M
P. = (1/M)
Σ E 1=0
......,,9) .49 corresponds to the pooling region index, Fay is the embedded feature vector present in that pooling region, M = {1, 9, 25, 49} corresponds to number of features inside the pooling region and P, is the mean feature vector corresponding to that pooling region.
Where i= (1....,
3.2 GUIDED FIXATION SEQUENCE PREDICTION:
Self-attention weights from the last transformer block are collected to get each pooling region's contribution to the feature vector corresponding to the class token. A confidence map is constructed based on the fixation point location by putting these weights back on 14 x 14 map at the corre- sponding pooling region's location, where 14 x 14 corresponds to the size of the input feature map. After each fixation, we degrade the attention weights in the confidence map at a rate of 50%. Inhibi- tion of return (IOR) (Dukewich & Klein 2015) refers at previously attended or fixated regions. We constructed an IOR map by applying an IOR of 3 x blocks around fixation location and degraded it by 50% after each fixation. After subtracting the
tendency in human observers not attend
5

---

Under review as a conference paper at ICLR 2022
IOR map from the confidence map, the following fixation location is extracted based on the location of the maximum in the confidence map.
Loss function: We Cross-entropy for computing the classification loss. Loss from all fixations is incorporated to get the mini-batch loss, as shown below.
loss ΣΣ ys log(p(u,)) + (1 リソ) log(1 1al1-p(y31 p(y)))
Where i corresponds the fixation index, N 5 for the Foveated model & N for Full- resolution model, i.e., single-pass, y corresponds to the target label, j corresponds to the class index, and p correspond the predicted probability.
RESULTS
We train the model weights from scratch and no pre-trainng. The Foveated model trained for five fixations to have a computational complexity lower than the Full-resolution model. In the following sub- sub-sections, we compare the performance and computational complexity of the Foveated model performance against the Full-resolution model. In addition, we compare the adversarial robustness of the Foveated and Full-resolution models. Appendix [6.] consists of ablation studies regarding architecture choice, contribution of the attention mechanism, and the importance of peripheral rings.
4.1 IMPLEMENTATION DETAILS
Training: We train all the 5.5M parameters of the model. Using initial learning of 1e - 5 and minimum learning rate of le 6, train both Foveated and Full-resolution models for 300 epochs. We use AdamW (Kingma & Ba 2014 Loshchilov & Hutter 2019) optimizer with a decay le and cosine learning rate schedule. We use the hyperparameters and augmentation settings from DeiT implementation (Touvron et al. 2220)). We use ImageNet- 100 (Tian et al.) 2020) and ImageNet (Deng et al. 2009) datasets for the results shown the following sub-sections. For ImageNet-100, We use the train validation split provided by the ImageNet and report results on the validation set. We use GeForce GTX 1080 Ti for training and testing purposes. Both Foveated and Full-resolution models take 25 hrs and 21 hrs respectively to train for 300 epochs on three GPUs. The Foveated model made five fixations on each input image during training, with the initial fixation starting at random location. Due to our model's flexibility, the same model can make any desired number fixations.
Dynamic-stop of Fixation Exploration: Due to various factors such occlusion, camera angle, brightness, the difficulty of making a classification decision varies across images. To achieve higher computational efficiency in our Foveated model during inference, we stop exploring the images with fixations when the predicted class with the highest probability reaches a pre-defined threshold corresponding to that class. The threshold equals the mean probability of all the correct predictions belonging to that class in the training dataset.
Images/Sec: We report the number of inferences completed by the GPU during one-second time interval to compare the computational complexity of different models during inference time.
Initial fixation for the Foveated model: The input feature map the Foveation module has spatial size of 14 x 14, and thus consider any one of these 196 fixation locations. During training, a random location selected as the initial fixation point, and the model guides subsequent fixations. During inference, for the results presented in Table and Table 2 the initial fixation is set at the center of the image to remove any variation in results arising from the randomness in the initial fixation location.
4.2 TOP-1 ACCURACY:
The full-resolution model allows the long-range full-resolution interactions between the feature vec- tors, whereas the Foveation model serially explores the image allowing only limited number of interactions between pooled feature vectors at each given fixation. Using the guided fixation predic- tion method described in the previous section, the Foveated model makes four guided fixations after

---

Under review as a conference paper at ICLR 2022
Table 1: Performance on ImageNet100 Tian et al., 2020) dataset: We find that the Foveated model outperforms the Full-resolution model both in terms of performance and computational com- plexity. Pure-Trane refers to the model with a single convolutional layer, and Hybrid refers to the models with convolutional backbone the start of the transformer as described the previous sec- tions. For evaluation, the initial fixation for the Foveated model is chosen at the center of the image to remove effects due to variations of the initial fixation location.
Model
Type
Params
Top-1
Conv Flops
Trans Flops
Total Flops
Deit- Deit-Tiny
Pure-Trans
5.5M
76.3
0.029
1.046
1.07
Full-resolution (Ours)
Hybrid Hybrid
5.3M
84.58
1.245
0.697
2.03
Fovcated (Ours)
5.3M
84.66
1,245
0.531
1.84
Foveated with Dynamic-Stop (Ours)
Hyhrid
5.3M
84.68
1.245
0.245
1.49
Table Performance on ImageNet dataset: We compare our models against the state-of-the models with a similar number of parameters using three metries: Top-1 accuracy, Image through- put, and Adversarial robustness. For reporting adversarial robustness, e set at 0.01, and the adversarial attack is computed for the transformer front-end only; therefore, we do not report com- Throughput for the dynamic-stop model
parisons to the convolutional models. DS:Dynamic-Sun computed as weighted combination of throughputs of the Foveated model with constant num- ber of fixations, i.e., to 5 fixations. DeiT-Ti (distilled) refers to the transformer architecture aided by convolution network.
Model
Type
Params
Images/sec
Top-1
Top-1-Ady (c 0.01)
FGSM
PGD
ResNet-18 18 [e al. 2016
Conv Conv
11.7M
1505
69.8
EfficienNNet.BU lan
2019
5M
720
77.1
Deit- li [Touvron al.
2020
Pure-Tram
5.7M
796
72.2
46.53
36.5
Deit- Ti (distiled) Touvron al.
Trans-Cowv
6M
796
74.5
Full-rcsolution (Ours)
Hyhrid
5.5M
652
73.6
39.37
28.98
Foveated (Ours)
Hybrid
5.5M
637
70,2
59.24
57.67
Foveated-DS (Ours)
Hybrid
5.5M
960
69.9
the initial random fixation. Sample fixations made by the model are shown in Figure 3a and average number of fixations made by the model per class are shown in Figure 3b
Additional visualization
of what is visible to the model at each fixation is shown in Appendix 6.2
4.2.1 RESULTS ON IMAGENET- 100
Due to the hybrid nature of the models proposed, incorporating CNN backbone and a transformer network, the Foveated and Full-resolution models outperform the DeiT-Tiny model, as shown in T The Foveated model's s accuracy is slightly above the Full-resolution model but critically has lower computational cost. Critically, the last row shows that with the addition of the dynamic-stop, the FoveaTer model can achieve close to 65% computational savings in transformer flops and 26% savings terms overall flops while retaining the performance achieved by the Full-resolution model.
4.2.2 RESULTS ON IMAGENET
We present the results the ImageNet dataset in Table 2 The Foveated model achieves accu- racy above the ResNet-18 model. For the dynamic-stop. we first compute the number of images falling in each of the five fixation bins and throughputs of the Foveated models with to five fixations. The throughput of the Dynamic-stop model is computed as the weighted average of the throughputs of individual fixation models using the bin count as the weights. Using the dynamic- stop, the throughput of the the Foveated model is improved (637 + 960), as shown in the last row. Appendix 6.3 shows the correlation between object complexity and the number of fixations made by the Dynamic-stop model.
4.3 ROBUSTNESS AGAINST ADVERSARIAL ATTACKS
We demonstrate the robustness of our foveated system this section. We consider the two adver- sarial attacks, Fast Gradient Sign Method (FGSM Goodfellow al. (2014) and Projected Gradient
7

---

Under review as a conference paper at ICLR 2022
Bveeraee predictions
lesser paaada
recreational vehide
couliffower
Labrador retriever
brown, beal
hatdag
whitte
broom
mailot ot
velvet
wall
class calegory
(a) Fixation sequences: A maximum of five fixations are (b) Average fixations per class: Average number made for each image with Dynamic-stop. The initial random fixations used by each class are plotted on y- fixation is followed by four guided fixations. Each subse- axis. Some of the class names are displayed on quent fixation center is represented with a bigger red circle. the x-axis.
Bluc arrows represent the transition from fixation to the next. Green and Red bars top correspond correct and wrong predictions respectively.
Figure 3: Fixation sequences and Average fixations per class
FGSM attack
PGD attack
80
80
Accuracy
60
60
Drveated-Guided Fovested-Guideat
Foveated-Staie
Full-resolution
20
20
0,000 0.025 0.050
0.100 0.000
0.025
0.050
0.100
Epsilon
Epsilon
Comparison of Foveated model (with guided fixations and two different initial fixations, RD initial random fixation. CT initial fixation at the image center), a foveated model that makes the same fixation sequence for every image (Foveated model with static fixations), and the Full-resolution model. Epsilon represents the strength of the attack. As the strength of the attack increases the accuracy of the Foveated model with guided fixations accuracy is greater than the Foveated model with static fixations while the Full-resolution model has the lowest accuracy. Raw values are available in Table in Appendix.
Figure 4: Adversarial robustness:
Descent (PGD Kurakin et al. (201); Madry et al, (2018), to compare the robustness of Foveated and Full-resolution models, We use Cleverhans library (Papernot et al.) 2018) for implementing the adversarial attacks.
To compute the adversarial robustness of the model, we separated the network into a convolutional backbone and Transformer front-end. The adversarial attack is computed only for the Transformer front-end, i.e., features from the convolutional backbone as network input and Transformer front-end as the model. Since model only affects the Transformer part of the network, this disassociates the interaction of the adversarial attack with the convolutional backend, which does not interact with the foveation module. For this reason, purely convolutional networks are not considered. We compare only the DeiT-Tiny model with Full-resolution and Foveated models. Epsilon corresponds to the strength of the attack.
8

---

Under review as a conference paper at ICLR 2022
Overall, Foveated model outperforms both DeiT-Tiny and Full-resolution models in terms of adver- sarial robustness.
4.3.1 RESULTS ON IMAGENET-I0N 100
Figure 4 shows that performance the Full-resolution model (84.58) and Foveated model (84.62) similar for zero epsilon value, but the strength of the attack increases, the Foveated model outperforms the rull-resolution model. The robustness of the Foveated model can be either due to the pooling operations that are part of the model or our model's inherent ability to dynamically decide fixation paths, thereby minimizing the effect of the attack by automatically following different path from that of the adversarial model.
To isolate effect of foveation from the influence of fixation exploration, We compare various con- figurations of the Foveated model. For the foveated guided fixations model, we explored version with random initial fixation and another version with constant initial fixation at the center of the image. The model then dynamically selected the subsequent four fixations, thereby allowing the possibility of different fixation paths for the adversarial image computation and model evaluation. The model with the random initial fixation results more variability in subsequent fixations. Criti- cally, we also used a pre-defined fixation sequence (center initial fixation followed by four fixations at the center of each quadrant), thereby ensuring that the adversarial image computation and the model evaluation that follows will use the same fixation locations. We refer to this model as the Foveated static.
Results in Figure show that the Foveated guided model with initial random fixation outperforms the static fixation model, suggesting that the variability in fixation sequences contributes towards adversarial robustness. The comparison the Foveated Guided model with the fixed initial fixation at the center of the image shows that a stochastic component to the initiation can be important to vary the fixation sequence and result in greater robustness.
Finally, we illustrate the contributions of the foveated nature of the model by comparing the Foveated model with the five pre-defined fixations (red curve, Foveated static) to the Full-resolution model (orange curve). This pre-defined fixation path Foveated model will also be the desirable solution when the computational time of serial fixation model a bottleneck, all the pre-defined fixations can run in parallel. Figure show that the Foveated model with static fixations outperforms the Full-resolution model illustrating the contributions of the foveated nature(L.uo al., 2015 Kiritani & Ono 2020: Vuyyuru et al. 2020) of the model to robustness to adversarial attacks.
4.3.2 RESULTS ON IMAGENET
We show the results of adversarial attacks in the last two columns of Table 2 Due to architec- tural changes from the DeiT-Tiny model, the Full-resolution model (FGSM-39.4: PGD-29) exhibits lower adversarial robustness compared to Deit-Tiny model (FGSM-46.5; PGD-36.5). On the other hand, Foveated model (FGSM-59.2) 2; PGD-57.7), which shares architectural similarity with the Full- resolution model, exhibits higher adversarial robustness than both DeiT-Tiny and Full-resolution architectures.
CONCLUSION
We provided a comprehensive framework for using foveal processing and fixation exploration on vision transformer architecture for image classification. The proposed architecture introduces a way to limit computations required to process an image by flexibly adjusting the required number of fixations, allowing larger batch sizes during training, providing robustness adversarial attacks, and giving us a model that can allocate computational resources based on the difficulty of an im- age. Our findings reveal that variability fixation sequences to images can contribute over foveated processing robustness to adversarial attaches. In conclusion, we leveraged the most recent Vi- sion Transformer architecture and combined it with ideas from foveated vision to come up with model which has multiple knobs in terms of the number of fixations to be made and limits the computations done so that the end-user will have the flexibility to mentunee depending on their needs.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Emre Akbas and Miguel P. Eckstein. Object detection through search with a foveated visual system. PLOS Computational Biology, 13(10) 10 2017. doi: 10.137 //o.ural. 1005743, URL
ht
1tw.:/...3111311 1tw.////o/.....//ouur....a..aaaa pcbi. .00.:.1000110 00057333
Bilal Alsallakh, Narine Kokhlikyan, Vivek Miglani. Jun Yuan, and Orion
Mind
the pad - {cnn)s can develop blind spots. International Conference on Learning Representa- net mettoooormmiiiio wow.forrrunnmmmmmmmmpoopoooo
tions, 2021. URL /ttttpp:://opnrrevveeee wwwpereeeeeeeeeeeerrrrrreee
Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. In ICLR (Poster), 2015. URL /t..o///////xx.xoooooo//////////
Ghassan Dabane, Laurent Perrinet, and Emmanuel Daucé. What You See Is What You Trans- form: Foveated Spatial Transformers
bio-inspired attention mechanism. 9 2021. doi: URL
10.3627//echnnivv165559111
preprint/Whht__oore You See Is What You_ Betee__sWttttto__Toeee___
Transformers_as_ bts.abiiiinnpppii--e. _td________ooonnnreredeeeeeeeeee ont http::/www.......oooooooooooooon 155n0991
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier-
archical image database. 2009 JEEE Conference on Computer Vision and Pattern Recognition,
pp. 248-255, 2009. doi: 10. 1LL1///CVR.200995206639
Arturo Deza and Talia Konkle. Emergent properties of foveated perceptual systems. abs/2006.072 079911 2020.
ArXiv,
Arturo Deza, Aditya Jonnalagadda, and Miguel P. Eckstein. Towards metamerism via foveated style transfer. ArXiv, 1ttp11005000000 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 100.//9922 /929, 2020.
Kristie R. Dukewich and Raymond M. Klein. Inhibition of return: A phenomenon in search of definition and a theoretical framework. Attention, Perception, Psychophysics, 1645) 1658, Jul 2015, ISSN 1943-393X. doi: 10.377866134144-05--0333- URL https://doi.
org/10. 375. 11111//11111111---111 013--11---03333
Jeremy Freeman and Eero P. Simoncelli. Metamers of the ventral stream. Nature neuroscience, 1195 1201, 2011.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 580-587, 2014. doi: 10. 109/CVPR.2014.8L.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv 1412.6572, 12 2014.
Ethan William Albert Harris, Mahesan Niranjan, and Jonathon Hare. Foveated convolutions: im- proving spatial transformer networks by modelling the retina. In Shared Visual Representa- tions in Human and Machine Intelligence: 2019 NewrIPS Workshop, December 2019. URL
https: /ttps://eprints... soton. ac wu/ 441204/
Mary Hayhoe and Dana Ballard. Eye movements natural behavior. Trends in cognitive sciences, 9()::188 194, April 2005. ISSN 1364-6613. doi: 10. tis. URL https: / h/tpi.org/1...... 00100
2005. 02.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
L. Itti, C. Koch, and E. Niebur. A model of saliency- based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 125544) 1259, 1998. doi: 0o.100934.730553
10

---

Under review as a conference paper at ICLR 2022
Laurent Itti and Christof Koch. A saliency-ber based search mechanism for overt and covert shifts of visual attention. Vision Research, 40(19))1489-11 1506, 2000. ISSN 0042-6989. doi: https: http:u/ww|....000666002220999990 sci httpe///rrrddcleeess////00222229
URL https:/ www. wttp://wwwiinccrrrrttcccoo com/ 637]
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial trans- former networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL htttps://prooeeeddngg..com wttpaarppp..cpppppr//eeee pctpsccaapep/201//1eeee (eeea6776644e333a338222222666683
.ddf
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
Taro Kiritani and Koji Ono. Recurrent attention model with log-polar mapping is robust against adversarial attacks. ArXiv, abs/2002.05388,, 2020.
Christof Koch and Shimon Ullman. Shifts in Selective Visual Attention: Towards the Underly- ing Neural Circuitry, pp. 115-141. Springer Netherlands, Dordrecht, 1987. ISBN 978-94- 009-3833-3
doi: 0000079978-4-009-3333 978 94 003-0333-3332 3833-5
URL /attps://aci. /Ettss///c...oo/11010000
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. ArXiv, 054/1607.025333 2017.
M. Land. Oculomotor behaviour in vertebrates and invertebrates. The Oxford Handbook of Eye Movements, 01 2012. doi: 10. 10051oofoodth/77339999999990.3.0
Hugo Larochelle and Geoffrey E Hinton. Learning to combine foveal glimpses with a third- order boltzmann machine. In J. Lafferty, C. Williias, Shawe-Taylor, R. Zemel, and A. Cu- lotta (eds.), Advances in Neural Information Processing Systems, volume 23. Curran Asso- ciates, Inc., 2010. URL httpps::/proccedinggg.. http./rrrrs.s.ccppppe///////e/e Pappee ppf
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Casimir J. H. Ludwig. J. Rhys Davies, and Miguel P. Eckstein. Foveal analysis and peripheral selection during active visual sampling. Proceedings of the National Academy of Sciences, 111 (2):E291-E299, 2014. ISSN 0027-8424. doi: 1..073//pas.1113555333 Utthttps: pnas wwt.:.rggccooooo /massorg/ccontent/11////22
Yan Luo, Xavier Boix, Gemma Roig. Tomaso A. Poggio, and Qi Zhao. Foveation-based mechanisms alleviate adversarial examples. ArXiv, abs/ 1.06222, 2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ArXiv, abs/1706.06083, 2018.
N Marshall, M. Land, and T Cronin. Shrimps that pay attention: Saccadic eye movements in stom- atopod crustaceans. Philosophical transactions of the Royal Society of London. Series B, Biolog- ical sciences, 369:2013004,, 02 2014. doi: 10. 1098//stth.b... 201...2111322
Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS' 14, pp. 2204-2211 Cambridge, MA, USA, 2014. MIT Press.
Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, (2(33)1455-17) May 2001. ISSN 1573-1405. doi: 10.1023/A:1011113993
org.
1011119631724

---

Under review as a conference paper at ICLR 2022
Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Ku- rakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy. Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang. Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, and Rujun Long. Technical report on the cleverhans v2.1.0 adversarial examples library. arXiv preprint arXiv:1610.000 00766,, 2018.
Simon Prince, James H. Elder, Yuqian Hou, Mikhail Sizintsev, and Yevgen Olevskiy. Statistical cue integration for foveated wide-field surveillance. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 00), 2:603-610 vol. 2, 2005.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and J. Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 39:1 137-11499 2015.
Ruth Rosenholtz, Jie Huang. and Krista Ehinger. Rethinking the role of top-down attention in vision: Effects attributable to lossy representation in peripheral vision. Frontiers in Psy-
chology, 3:13, 2012. ISSN 1664-1078. doi: 10.3389/fpsyy2221220001
URL https:
7 //www. www.ttrrnrrsst.oo wttp:///...g/a..teeeeeeeeea 3389/ Ipsyc 2012.22222 00013
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pu Chen, and Cho-Jui Hsich. On the adversarial robust- ness of visual transformers. ArXiv, abs/2 100.11037700 2021.
Hans Strasburger, Ingo Rentschler, and Martin Jüttner. Peripheral vision and pattern recognition: A review. Journal of vision, 11:13, 05 201 1. doi: 10.1 10.1167/11.5.13.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effec- tiveness of data in deep learning era. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 843-852, 2017. doi: 10. 10.100092.2
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114 PMLR, 2019. URL BRRLittt:////oceeet//
mlr www.pprrssvvvve. tan htmmll
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Jan-Michael Frahm (eds.). Computer Vision - ECCV 2020, pp- 776-794, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58621--8
Antonio Torralba, Aude Oliva, Monica Castelhano, and John Henderson. Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search.
Psychological review, 113:766-86, 86, 11 2006. doi: 10. 10330//00033220 21223--99XX11111 113.4.761..
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and
Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint 2rXiv:2012.12877 2020.
John K. Tsotsos. A computational perspective on visual attention. 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, un- definedukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 188150060964.
Manish Reddy Vuyyuru, Andrzej Banburski, Nishka Pant, and Tomaso Poggio. Biologically in- spired mechanisms for adversarial robustness. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2135-2146. Curran Associates, Inc., 2020. URL
igg neurips.coom Pat.44-aaperr......
Calden Wloka, Iuliia Kotseruba, and John K. Tsotsos. Active fixation control to predict saccade sequences. In 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 3184-3193, 2018. doi: 10.1009/CVPR.2218.00033
12

---

Under review as a conference paper at ICLR 2022
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual at- tention. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2048-2057, Lille, France, 07-09 Jul 2015. PMLR. URL http: htttp://prroeeeiingg ml mm..press/v37
xuc15. html
H. Yamamoto, Y. Yeshurun, and M. Levine. An active foveated vision system: Attentional mecha- nisms and scan path covergence measures. Comput. Vis. Image Underst., 63:50-65, 1996.
13

---

Under review as a conference paper at ICLR 2022
Table 3: Performance of the ImageNet pre-trained models the auxiliary task without any addi- tional training.
Pre-trained Model
Top-1
ResNet-18
1.3
EfficientNet-B0
1.8
Deit-Ti
7.8
Full-resolution (Ours)
10.9
Foveated (Ours)
7.5
Guided foatios
0.32
0.31
0.30
0.2
Number fixations
(a) Comparison between random fixations and guided fixations with crop lo- (b) Sample evaluation image cated at top-left corner and the initial fixation at bottom-right corner.
Figure Additional task to determine the effectiveness of guided fixations
APPENDIX
6.1 ABLATION STUDIES
6.1.1 ABLATION STUDY 1: ATTENTION MECHANISM
Objects the ImageNet dataset occupy a high percentage of the image area resulting in a dimin- ished difference between guided and random fixations. In order to illustrate the performance of the attention mechanism, we create this small extension task.
Task: 224x 224 image from the ImageNet dataset is down-scaled to a size of x6 and is randomly placed in 224 x 224 image with normal random noise (N(u, d²), where μ = 0 and a 0.5). Table 3 shows the performance of the ImageNet pre-trained models this task. Transformer- based models, including our hybrid models, easily outperform the CNN models. Sample image is shown in Figure 5b
Training: Model trained on ImageNet dataset fine-tuned for 30 epochs with a lower learning rate of 5e 5 on this auxiliary task. Initial fixation is random location during training.
Evaluation: Chance value 0.1 due to 1000 classes. The down-scaled image and the initial fixation at the top-left corner and bottom-right corner, respectively, to start at the maximum possible distance from the target for both guided fixations and random fixations conditions.
Comparison of the model with guided fixations is plotted against random fixation model in Figure 5a The guided model consists of one initial fixation at the bottom-right, followed by four guided fixations. The random fixation model consists of one initial fixation the bottom-right followed by four random fixations. The difference in performance illustrates the contributions of the fixation prediction component of our model.
Results:
6.1.2 ABLATION STUDY 2: INPUT FEATURES TO FOVEATION MODULE
Table consiiss of hybrid version of the Foveated model where a convolutional backbone provides input features to the foveation module. To assess the contribution of this convolution backbone,
14

---

Under review as a conference paper at ICLR 2022
Table 4
Model
Type
Params
Top-1
Foveated (Ours)
Hybrid
5.5M
70.2
Deit-Ti Touvron et al. 2020)
Pure-Trans
5.7M
72.2
Foveated-1 (Ours)
Pure-Trans
5.7M
68.5
replaced the convolution backbone with transformer layers, making the resultant pure-transformer model (Foveated-TT entirely equivalent for the DeiT-Ti model except for the presence of foveation module. In this model, the Foveation module preceded by four transformer layers followed by eight transformer layers, where the original DeiT-Ti model consists of Twelve transformer layers. Results are shown in T The hybrid version the Foveated model has a Top-1 I accuracy of 70.2 compared to 68.5 for the pure- Transformer version of the Foveated model. Therefore, the presence of the CNN backbone contributes to an improvement of 1.7%.
6.1.3 ABLATION STUDY 3: CONTRIBUTION PERIPHERALL FEATURES
As described in the Foveation module section, Fovea is 3 followed by three peripheral rings of pooled features. In this section, we look at the contribution of those peripheral rings towards the final performance. We reuse the task described in Section 6.1.T to get the contribution of those peripheral features. To this end, train four different models on this auxiliary task. Foveated-F consists of only Fovea and no additional peripheral features. Foveated-F33 consists of Fovea and one peripheral ring of x average pooled peripheral features. Foveated-F35 consists of Fovea and two peripheral rings, 3 x and 5 x 5 average pooled peripheral features. Foveated-F357 consists Fovea and three peripheral rings, x 3, 5 5, and 7 average pooled peripheral features. Foveated-F355 is equivalent to the model presented in Section 6.1.1
All four models with the weights of the Foveated model trained on ImageNet. Then the model is fine-tued for 30 epochs. Figure 6 compares the performance of all four models under random fixations condition described in Section 6.1.1] Models with more peripheral rings outperform the models with lower peripheral rings, and the models converge to saturation performance with an increase in the number of fixations.
0.25
Top-1 Accuracy
0.20
0.15
0.10
Foveated-F357
Fovested-F35
0.00
Foveated-F
4
Number of fixations
Figure 6: Model Comparison: Down-sized images are present at the top-left corner, and initial fixation is set at the bottom-right corner. Models with more periphery rings outperform the models with less number of periphery rings.
15

---

Under review as a conference paper at ICLR 2022
6.2 VISUALIZATION:
GT: chainlink,tenc
chainink Jence: 0.12
fence:
chainlink Jance: 0.13
mondday 0:13
chainlink fence: 0.14
chainlink Jence: 0.14
CT: beacon
beacon: 0.54
beacon: 0.63
beacon: 0.63
beacor: 0.63
beacon: 0.65
cricket:
cricket
spotted salamander
common, newt: 0.31
common, newt
common new!: spotted_ spotted_salamandee salamander
common_neew 0.21
craylish
centpece:
centipede:
cenipedd:
trailer
traile track: 0.78
trailer truck: 0.83
trailer, truck: 0.83
trailer truck: 0.83
traier, 0.84
hammerhhead
tiger_shari:
figer_thare chark:
tiger_sharic shark:
tiger sharks
liger_sharkc shark:
tiger
Siberian husky
Siberian, husky: 0.27
Siberian: husky: 0.
Siberian husig, 0.29
Siberian_huskkr huskys 0.3
Siberian,
Siberian, husky. 0.29
GT American.coom coot
American_coot 0.77
American_copt coot: 0.79
American_comm coot: 0.0
American, coricaanccoot:oooo 0.79
American, coet: 0,
Figure 7: What the model sees: Each row contains five images that correspond to the five fixations on a single image. Left-most the original image with the ground-truth touud-truth (GT) class.
16

---

Under review as a conference paper at ICLR 2022
Table 5: Adversarial robustness on ImageNet100: dataset. Fov:ated-S-RD: Foveated model with guided fixations and random initial fixation. Foveated-P: Foveated model with pre-defined fixations with parallel execution capability. Foveated-S-CT: Foveated model with guided fixations and image center initial fixation.
DeiT-TTny
Full-resolution
Foveated-S-Ri
Foveated-P
Foveated-S-
Epsilon (e)
FGSM
PGD
FGSM
PGD
FGSM
PGD
FGSM
PGD
FGSM
PGD
76.34
76.34
84.58
84.58
84.60
84.64
84.94
84.94
84.62
84.62
0.005
69.82
69.26
83.62
83.60
83.82
83.98
84.22
84.22
84.0
83.94
0.01
58.62
55.10
80.82
80.78
83.42
83.02
82.62
82.50
83.12
83.14
0.025
35.90
23.74
70.54
67.86
77.48
76.96
75.58
74.70
76.18
75.46
00.05
23.56
10.14
56.6
46.62
67.40
63.06
64.26
58.76
65.30
59.94
0.1
15.42
6.58
44.04
25.80
54.60
43.48
52.92
39.04
52.96
39.34
0.2
10.24
5.56
35.08
8.38
46.26
25.18
44.80
21.46
44.54
21.28
6.3
CORRELATION OF OBJECT COMPLEXITY AND NUMBER OF FIXATIONS:
Using the Foveated model trained ImageNet, compute entropy metric for each image quantify the object complexity. Using a x 4 grid, the foveated model was run for one fixation these 16 locations resulting in 16 probability values per image for the ground-truth class. The entropy of the image is computed using these values. For each of the thousand classes, the corre- lation between the number of fixations made using Dynamic-Stop and the entropy of those images is calculated. Results are shown in Figure 8 Out of the 1000 classes, 815 classes have a negative coefficient. When the object is small and occupies a small percentage of image area, the resulting entropy will be low the one- one-fixation model will not detect the target everywhere on the uniform grid. At the same time, when the object is small, the Dynamic-She more fixations to detect the target as it can not make the right decision by fixating anywhere on the image. A negative correlation from the plot captures the inverse relationship of entropy and the required number of fixations.
model will need
298
300
272
Number of classes
250
166
150
100
50
-1.00
0.75
-0.50
-0.55
0.00
0.25
0.50
0.75
1.00
Pearson correlation coefficient
Figure 8: Relationship between image entropy and the number of fixations made by the Foveated model with Dynamic-stop. Pearson correlation coefficient computed for each class separately, and the histogram displays the distribution of correlation coefficient value across classes.
17


Reviews:
[{"review_id": "TSQQQoYpK2", "reviewer": "Reviewer_gFeo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While recent success in convolutional processing has dominated the field, certain aspects of vision common to many biological vision systems have had limited treatment. The proposed work is interesting in this respect, as it combines a foveated vision model within the popular transformer model framework. It demonstrates a model for active vision with predicted eye movements using recurrent network to predict the next area of foveated attention as a simulated eye movement. \n\nThe introduction provides motivation for foveated vision and review of prior art in this area. However, the most relevant works are not cited and the paper falsely claims to be the first for combining transformer networks and foveated vision: \"No papers have evaluated the additional potential gains of incorporating a foveated architecture into transfomers.\" The authors should correct this claim, and take note of the following references to explain what is unique in their approach and how it differs from other foveated vision models:\n\n* Harris, Ethan William Albert, Mahesan Niranjan, and Jonathon Hare. \"Foveated convolutions: improving spatial transformer networks by modelling the retina.\" (2019).\n* Dabane, Ghassan, Laurent Perrinet, and Emmanuel Daucé. \"What You See Is What You Transform: Foveated Spatial Transformers as a bio-inspired attention mechanism.\" (2021).\n\nThe authors reference 2017 work of Akbas and Eckstein as another foveated object detector, but do not provide distinctives that set their model apart from the work. The current paper would benefit from direct comparison to these other foveated vision methods. The only comparisons to other methods are against full-resolution models.\n\nIt is unclear from the introduction why adversarial attacks are a focus of this paper. The motivation of the paper starts on the basis of biologically inspired vision systems, yet such systems do not have to address adversarial attack other than overcoming natural camouflage (which is not at all similar to the types of adversarial attacks compared in the paper involving gradient informed adversarial examples), and in any case there is not explanation of why adversarial robustness should be a feature of foveated vision, or why to evaluate foveated vision systems on this basis.\n\nSimilarly, while the connection between transformer networks and foveated vision systems seems more supportable, it is lacking. There is an obvious opportunity and need to provide some description of transformer networks and why they are particularly well suited a framework to support the foveated vision system the authors wish to demonstrate. \n\nEye movements seem limited by a 14x14 grid. Explanation of the foveation pooling and aggregation approach was a little confusing. Is the movement restricted to an inner region, or is the zero-padding extend to potentially half the visual field when the attention is placed near the edge of the image. It would be interesting to visualize the visual field for a given fixation. Given the biological motivation, I would be interesting to know how this compares to biological systems. \n\nClassification results seem a little underwhelming. While it barely beats ResNet-18, 70.2% vs 77.1% against EfficientNet-B0 is disappointing. Again, without direct comparisons to other foveated vision systems it is difficult to tell how much this improves (if at all) on current art.\n\nI was particularly intrigued by what could be learned from the saccades themselves, but relatively little was discussed in that regard. The examples in Fig 3 were helpful to see but limited to a few hand-picked examples, and difficult to know how well movements relate to object content. It would be informative to compare eye movements predicted by the model to biological movements in gaze-tracking datasets. \nSuch comparisons could support the biological motivation of the paper and show that the model is learning beneficial movements. Comparing the only to full resolution does not sufficiently answer this. Another comparison to illustrate this would be to compare the existing model against random movements. \n\nI think thresholding the classification probability is a good idea to determine a stopping point for the dynamic stopping of fixation exploration, and the average fixations per class is interesting but I would like to understand if this correlates in some way to actual object complexity. The conclusion claims the learned model can allocate computational resources based on the difficulty of an image, but I did not see any comparisons that defended this. Can you show that more fixations were used by the model for images that were more complex by some entropy or other complexity metric? Or compare to human gaze experiments?\n\nOther specific comments:\nPg. 2: \"reach that approaches\" -> \"approach\"\nPg. 2: \"using which locations far apart in the image will interact\" -> this sentence is unclear as to its meaning, but needs revision\nPg. 4: \"squared pooling regions...for computational speed-up\": what kind of speedup is obtained, and by how much, and compared to what?\nPg. 6: description of dynamic-stop of fixation exploration: Is this only for inference, or is dynamic stopping performed during training?\nFigure 4: Requires color to interpret. It may be helpful to use different marker and line styles in case this is viewed in black and white\n\n\n\n", "summary_of_the_paper": "The authors present an approach to computer vision combining Transformer networks and foveated vision. They contrast their approach, which uses a recurrent attention model to simulate eye saccades to explore a scene, against full resolution deep CNN models for classification problems to demonstrate reduced computation and greater resistance to adversarial attacks.", "summary_of_the_review": "This is an interesting area of research, but I do not think that the authors did a sufficient job of motivating the model and experiments. They did not motivate the use of transformer networks, or explain why adversarial attacks are an appropriate problem for comparing foveated vision systems. They also left some important questions answered or not fully answered (such as whether the fixation movements were actually benefiting the classification). Experiments provided do not do direct comparison with equivalent models with and without the predictive attention which could easily determine the contribution for this part of the model. Claims in the paper are not sufficiently supported. A priority claim made in the paper as the first work to combine foveated vision and transformer networks is false and ignores published work by Harris et al. 2019 that uses spatial transformer networks for foveated convolutions. Direct comparisons to other foveated vision systems are lacking, and without these it is not possible to assess how well their particular approach compares to the most similar prior art.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q6srjYRbkv", "reviewer": "Reviewer_EWqw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method presents a unique idea of computer vision tasks based on the human perceptional system. Since humans archive vision processings with limited computing resources, employing the idea may work for computing tasks with the limited resource, thus, the idea proposed in this paper may give an impact on the community. However, I still wonder about the effectiveness due to the lack of comparison with existing methods.\n\n- Performance of object recognition:\nThe performance of ImageNet task is far from SOTA. I understand the main claim of the paper is introducing a new scheme that increases the performance of the existing algorithm, but if so, the question is why the paper used Deit-Tiny for only the reference. It is highly desired to combine other methods as well as the proposed foveated scheme.\n\n- Attention model:\nDiscussion (and hopefully experimental validation) need about the difference between the attention model and the proposed gaze control. \n\n\n- Foveated transformer:\nI wonder which is better between the proposed foveated transformer v.s. dilated convolution since they have similar computational models. Rather, dilated convolution is much similar to the human vision for its gradient-density nature. ", "summary_of_the_paper": "The paper shows a method of Foveated Transformer motivated by a human foveal vision where the spatial resolution varies depending on the focused point. Namely, the method mainly consists of the spatial attention model that controls the gaze point and spatial-varying convolutional filter (dense pooling in center and sparse pooling in peripheral). The former simulates human gaze control and the latter mimics the human retina. The experimental validation is performed by the object recognition task (ImageNet) and adversarial attack tasks and performed better than the Deit-Tiny model that uses a similar number of parameters.", "summary_of_the_review": "The method presents an interesting and unique approach that is motivated by human vision perception system including gaze and attention control as well as the foveated vision. However, experimental validation is limited, and thus difficult to understand the pure effectiveness. I think other application scenarios such as real-time video recognition may be suit for the scheme than the object recognition tasks for its low-cost computing nature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GAHorSM4i8", "reviewer": "Reviewer_NG64", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- Overall, I found this paper both an interesting read and quite a novel approach. It is great to see practical sequential models being developed, and the inclusion of robustness evaluation via adversarial attacks was very helpful in bolstering the paper's claims.\n\n- I appreciate the effort made to make the model comparisons more fair (i.e. based on a similar number of parameters). \n\n- The components of the model, training steps, and experiments were clearly described.\n\nWeaknesses and Questions:\n\n- Why restrict the adversarial attacks to only the transformer component? We know adversarial attacks affect CNNs, too, so I was unclear on the rationale for this design choice. Also, since Deit-Ti is a purely transformer-based method, I found it unclear if this meant that the comparison of the adversarial attack on Deit-Ti against the two hybrid methods proposed in the paper was fully equivalent.\n\n- What is Deit-Ti (distilled)? I could not find an explanation for this model in the text, and it only appears in Table 2.\n\n- Why is the Dynamic Stop version of the foveated model not included in the adversarial robustness investigation?\n\n- In Table 1, one column uses \"PC\" as a column head, but I couldn't find an explanation of what this stood for. It is possible I missed it, though. I am assuming from context it is performance.\n\n- In Section 3.2, I found it somewhat unclear to follow. Are there two 50% decay mechanisms (1. the attention weights of the previous confidence map, 2. the IOR map of previously fixated locations)?\n\n- The line directly below equation (2) has a typo: \"fixatiothe n\"\n\n- In Section 2: Sequential processing, the way the three main advantages are interspersed with the rest of the paragraph text is a little hard to follow. Also, an additional advantage of sequential processing with eye movements is articulated by Tsotsos (2011) A Computational Perspective on Visual Attention, which is to compensate for the \"Boundary Problem\" introduced by the loss of a convolution kernel half-width of fully defined output for each layer of convolutional filtering. The relevance of this issue to CNNs was demonstrated by Alsallakh et al., Mind the Pad - CNNs Can Develop Blind Spots, ICLR 2021.\n\n- In Section 2: Computational models of categorization and eye movements, it is mentioned that saliency-based models do not incorporate foveal vision, but Wloka et al., Active Fixation Control to Predict Saccade Sequences, CVPR 2018 provides a foveated model of eye movements based on an underlying saliency map.", "summary_of_the_paper": "The paper introduces a fixation-based method based on a hybrid transformer-CNN architecture, which demonstrates many of the strengths of sequential processing, namely computational efficiency and adversarial robustness. This model largely outperforms a comparative full resolution model, and has a number of attractive behavioural qualities.", "summary_of_the_review": "This is an overall strong paper. It proposes an interesting new approach, and does a good job of exploring the behaviour of this approach with respect to alternative architectures. The weaknesses identified are largely minor, points of clarity, or easily fixable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FaCRBjJjow", "reviewer": "Reviewer_czVG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Paper strength]\n- The method is well-motivated. The fovea in computer vision can save the computation, improve the robustness of the prediction, and improve the model performance.\n\n- The proposed method achieves better performance than baselines in the adversarial robustness experiments. The author shows the results for the robustness experiments in table 2 that the proposed method achieves better performance than baselines. The author then shows the contributions of the foveated natural of the proposed model in figure 4 for the adversarial robustness.\n\n- The proposed architecture make sense to me. The proposed method uses the foveation module to extract coarse and fine information from the feature map, and then change the fovea locations with the fixation sequence prediction, and transformer in the middle to extract feature with attention. Essentially, there are three components of attention mechanisms: foveation module, sequential fixation, and transformer. \n\n- The author shows the proposed method can run faster than other baselines methods while maintaining a similar performance in terms of the accuracy of the classification tasks.\n\n- The author provides detailed information about the proposed network architecture that is possible for re-implementation.\n\n[Paper weakness]\n- The main weakness of this paper is the lack of technical novelty. Compared with the previous DeiT-Tiny, the main contribution of this paper is the foveation module. However, the foveation module is very much like the dilated convolutions [a] and the pyramid of dilated convolutions [b], except here the filter/kernel is average pooling. If the author wants to claim novelty over the family of dilated convolutions, they should provide more discussions, analysis and experiments to show the advantage of the proposed method. Current, I cannot judge from the paper. \nThe sequential fixation is the recurrent CNN, and there is no other technical contribution as far as I can see.\nEssentially, the proposed method is a combination of recurrent CNN, transformer, and dilated convolutions. If the author would like to claim this combination itself is the contribution, there should be more ablation experiments on the contribution of each component.\n\n    [a] Multi-scale context aggregation by dilated convolutions. ICLR 2016.\n\n    [b] ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation. ECCV 2018\n\n- It is not clear to me how to train the sequential fixations. Since the fixation point is a spatial location, directly retrieving it from the attention weights map in the paper is not differentiable. I am confused about how the author does the backpropagation, i.e. how to judge one fixation position is better than another one? As far as I know, this is hard attention implemented by recurrent neural networks output the location, which usually can be solved with spatial transformer[c], LSTM, or other reinforcement learning loss. The task itself is not easy to train.\n\n    [c] Spatial transformer networks. NeurIPS 2015.\n\n- There is no significant performance improvement. As shown in Tables 1 and 2, there is no improvement from the proposed method over other baselines in terms of the accuracy of the image classification task. Although the proposed method can run faster than baselines, I do not think the running speed performance is superior since it is actually slower than the ResNet-18 in table 2.\n\n- The initial fixation should start at the centre of the image. I can see from figure 4 that a random initial fixation point is helpful for the proposed method since it can have multiple choices to choose the ideal locations. However, it is unfair for the full-resolution baseline to have a random initial location since it cannot achieve good performance if the random location is not ideal. \n\n- It is not clear to me how much of the class token could be helpful for the final performance, or does it matter at all?\n\n- Equation 2 seems like the loss is averaged after the classification layer, which is a conflict with figure 1 that the averaging happens before the classification layer. Please ignore this comment if I am wrong.", "summary_of_the_paper": "This paper proposes a called Foveater which uses a foveated module to extract the information from the feature map with different levels of details and different locations. This proposed method has the architecture that makes sense for the image classification task. However, I found it is difficult to find significant technical novelty in this paper.", "summary_of_the_review": "I agree with the author that the foveation module could be helpful for the computer vision task. However, it is hard for me to see the technical novelty of the proposed method. Therefore, I lean to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an architecture that uses pooling regions and
eye movements to sequentially build up an object representation.  A 
confidence threshold is used to allow recognition in less time for
easier images.

There was a lot of disagreement on this paper.  Those in favor argued 
that it is a worthy endeavor to explore new biologically motivated
architectures and foveated eye movements are an important aspect of  
human vision that is worth exploring for computer vision.  Another pro
was the improved robustness to some adversarial attacks.  Those
arguing for not accepting the paper, argued that classification
performance is not improved over SOTA and that more ablation studies 
should be done to better understand the role and importance of the
various aspects of the model and how they differ from other
architectural designs with dilated convolutions instead of the 
foveation module.
 
I agree that more ablation studies would be useful to better
understand the role of the different model components. While I 
feel that this novel sequential processing algorithm is worth publishing to
increase activity in this area, I feel it would be best received after further 
studies help clarify the importance of different aspects of the model.
I recommend resubmission after further analysis.
Decision: Reject