FoveaTer: Foveated Transformer for Image Classification

1 INTRODUCTION . Many mammals , including humans , have evolved a locus ( the fovea ) in the visual sensory array with increased spatial fidelity and use head and eye movements ( Land , 2012 ; Marshall et al. , 2014 ) to orient such locus to regions and objects of interest . The system design allows visual-sensing organisms to accomplish two objectives : fast target detection crucial for survival and savings in the computational cost . Computational savings are accomplished by limiting the number of units with high computational costs ( i.e. , higher spatial resolution processing ) to the fovea ’ s small spatial region . Fast target detection is achieved by distributing the remaining computational power across a much larger area in the periphery , with a lower spatial resolution with increasing distance from the fovea . Critical to the design is an efficient algorithm to guide through eye movements the highresolution fovea to regions of interest using the low-resolution periphery ( Hayhoe & Ballard , 2005 ; Strasburger et al. , 2011 ; Ludwig et al. , 2014 ) and allow optimizing the target detection and scene classification . Various computational models were proposed to model the search using foveated visual system ( Yamamoto et al. , 1996 ; Prince et al. , 2005 ) . Computer vision has evolved from using hand-crafted features to data-driven features in modern CNNs . Due to their computational limitations , the objectives of the computer vision systems align well with those of human visual system : to optimize visual detection and recognition with an efficient computational and metabolic footprint . Approaches towards saving computational power can be seen ; for example , computer vision systems evolved from using sliding windows to RCNN ’ s ( Gir- shick et al. , 2014 ) use of selective search and Faster-RCNN ’ s ( Ren et al. , 2015 ) use of Region Proposal Network ( RPN ) . A system that mimics the human vision by processing the scene with a foveated system and rational eye movements has also been proposed . This approach to exploring the scene can be seen in models like RAM ( Mnih et al. , 2014 ) for recognizing handwritten single-digits or detecting objects ( Akbas & Eckstein , 2017 ) where they sequentially process the image and decide what to process next by using the peripheral information . These foveated models approach that of full-resolution models but using a fraction of the computations . Foveated systems have also shown to result in more robustness ( Luo et al. , 2015 ; Deza et al. , 2019 ; Deza & Konkle , 2020 ; Kiritani & Ono , 2020 ; Vuyyuru et al. , 2020 ) against adversarial attacks . There has been a recent innovation in computer vision of using Transformers ( Touvron et al. , 2020 ; Dosovitskiy et al. , 2020 ) for object classification tasks which departs from the traditional overreliance on convolutions . Even after replacing the convolutions with attention modules and multilayer perceptrons , visual Transformers ( Dosovitskiy et al. , 2020 ; Touvron et al. , 2020 ) achieve close to state-of-the-art performance on the ImageNet dataset and provide better robustness against adversarial attacks ( Shao et al. , 2021 ) . Due to the flattened architecture of the transformers , it is easier for multi-resolution features to share the same feature channels . Transformers ( Vaswani et al. , 2017 ) have the added benefit of self-attention , which facilitates the interaction of various parts of the image irrespective of distance . No papers have evaluated the additional potential gains of incorporating a foveated architecture into Vision transformers for the task of ImageNet classification . Here , we evaluate the effect of a foveated architecture and sequential eye movements on a state of the art transformer model sitting on a convolutional backbone . We compare the foveated transformer relative to the full-resolution model in terms of classification accuracy and robustness to adversarial attacks . Furthermore , we extend previous papers by investigating the separate contributions to robustness related to the foveated processing ( Luo et al. , 2015 ; Deza et al. , 2019 ; Deza & Konkle , 2020 ; Kiritani & Ono , 2020 ; Vuyyuru et al. , 2020 ) and that of the process of fixating images at different locations . To this end , we perform an object classification task using multiple fixations , moving foveal attention across different parts of the image , and using only a limited portion of the image information at each fixation , thereby reducing the input to the transformer by many folds . The model decides on subsequent fixation locations using the learned self-attention weights and fixation locations of all the fixations until the current step . Finally , the model integrates information across fixations using average-pooling to make the final classification decision . 2 RELATED WORK . Transformers have achieved great success in the field of Natural Language Processing since their introduction by Vaswani et al . ( 2017 ) for machine translation . Recently , the application of Transformer models in computer vision has seen tremendous success . Vision Transformer ( ViT ) model introduced by Dosovitskiy et al . ( 2020 ) achieved remarkable performance on ImageNet ( Deng et al. , 2009 ) by using additional data from JFT 300M ( Sun et al. , 2017 ) private dataset . Subsequently , the DeiT model ( Touvron et al. , 2020 ) introduced knowledge transfer concepts in transformers to leverage the learning from existing models . Using augmentation and knowledge transfer , the DeiT model achieved close to state-of-the-art performance using training data from the ImageNet dataset alone . Sequential processing provides three main advantages in computer vision . Larochelle & Hinton ( 2010 ) proposed a model based on the Boltzmann machine that uses foveal glimpses and can make eye movements . First , it can limit the amount of information to be processed at a given instant to be constant , i.e. , the ability to keep computations constant irrespective of the input image size . Second , sequential models can help model human eye movement strategies and help transfer that information to build better computer vision systems . RAM ( Mnih et al. , 2014 ) introduced a sequential model capable of making a sequence of movements across the image to integrate information before classification . In addition , the hard-attention mechanism , implemented using reinforcement learning , was used to predict the sequence fixation locations . Ba and Minh ( Ba et al. , 2015 ) extended these ideas to recognize multiple objects in the images on a dataset constructed using MNIST . Third , se- quential processing requires fewer parameters than a model using full-resolution image input . Other models ( Xu et al. , 2015 ) have proposed image captioning models based on both hard-attention and soft-attention . Additionally , the spatial bias introduced into CNNs due to padding ( Alsallakh et al. , 2021 ) can be overcome using sequential models ( Tsotsos , 2011 ) Computational models of categorization and eye movements have been proposed for rapid categorization in terms of low-level properties such as spatial envelopes ( Oliva & Torralba , 2001 ) and texture summary statistics ( Rosenholtz et al. , 2012 ) . Saliency-based models ( Koch & Ullman , 1987 ; Itti et al. , 1998 ; Itti & Koch , 2000 ) traditionally tried to model eye movements by identifying bottom-up properties in the image that will capture attention . Torralba et.al ( Torralba et al. , 2006 ) showed how saliency could be combined with contextual information to guide eye movements . Low-resolution periphery and high-resolution central fields are integrated with saliency to predict human-like eye-movements ( Wloka et al. , 2018 ) . Akbas ( Akbas & Eckstein , 2017 ) implemented a biologically-inspired foveated architecture ( Freeman & Simoncelli , 2011 ) with a deformable parts model to build a foveated object detector which accuracy was close to a full-resolution model but using a fraction of the computations . Spatial transformer networks ( Jaderberg et al. , 2015 ) were used with foveation to improve object localization using foveated convolutions ( Harris et al. , 2019 ) and achieve better eccentricity performance ( Dabane et al. , 2021 ) . FoveaTer combines an approximation to biologically-inspired foveated architecture with a Vision Transformer Network . We apply our model to real-world images from the ImageNet dataset for the task of image classification . We pool the input to the transformer using the pooling architecture described in the following sections based on the fixation location , which reduces the number of inputs to the transformer . The subsequent fixation location is given by a confidence map , constructed using the attention weights from the last transformer block . Feature vectors corresponding to the class token from different fixations are averaged using an average-pooling layer followed by a fully connected layer , resulting in the final classification decision . A novel aspect of the proposed work is that the model also learns that all images are not equally difficult to classify adapting the exploration eye movements to different image classes and thus varying computational resources used to classify different images successfully . The model implements this idea using a confidence threshold to restrict the scene exploration to the necessary fixations to classify the image . Also novel is an evaluation the adversarial robustness of our model to understand the separate contributions of the foveated architecture and that of sequential fixations towards defense against adversarial attacks . We use the fast gradient sign method ( FGSM ( Goodfellow et al. , 2014 ) ) , which is an adversarial attack that computes the adversarial image by backpropagating the gradients once , and the projected gradient descent method ( Kurakin et al. , 2017 ; Madry et al. , 2018 ) , which iteratively computes the adversarial image , as our adversarial attacks . 3 MODEL . DeiT-Tiny ( Touvron et al. , 2020 ) : The DeiT-Tiny architecture begins with a convolution embedding layer that transforms the [ 3 , 224 , 224 ] input image into a [ 192 , 14 , 14 ] representation that is input into a series of twelve transformer blocks , each sized for a 192-dimensional embedding . Full-resolution model : Our Full-resolution model refers to a hybrid adaptation , transformer network with convolutional backbone , of the DeiT-Tiny architecture using the same number of parameters . This hybrid model replaces the single convolutional layer of the DeiT-Tiny model with a convolutional backbone . The convolutional backbone is composed of first two stages of the ResNet18 ( He et al. , 2016 ) architecture , which transforms the input from [ 3 , 224 , 224 ] to [ 128 , 28 , 28 ] , followed by three convolutional layers to acheive a final feature map of size [ 192 , 14 , 14 ] . The first convolutional layer transforms the input feature dimensionality from 128 to 192 using kernels of size 3 and stride 1 , resulting in [ 192 , 28 , 28 ] . The second convolutional layer downscales the features using a kernel of size 3 and a stride of 2 , resulting in features of size [ 192 , 14 , 14 ] . The third convolutional layer uses a kernel size of 1 and stride of 1 , retaining the size of the feature map . The number of transformer blocks is reduced from 12 to 9 to match the number of parameters with the DeiT-Tiny model . Foveated model : Extending the hybrid architecture described above , our Foveated model analyzes a pooled-version of the features from the convolution backbone at each time step and arrives at the final category decision after making the required number of time steps to satisfy the decision criterion . We show the architecture of our model in Figure 1 . The input image is first passed through the convolutional backbone described above , resulting in size [ 192 , 14 , 14 ] . Then , we perform fixation dependent average-pooling , using the ” Foveation module ” ( see next section ) , on the features from the convolutional backbone resulting in features of size [ 192 , 29 ] , where the fixation refers to one of the 196 possible spatial locations in the input feature map . Under this non-uniform average-pooling model , locations closer to the fixation location use smaller neighborhoods for pooling than locations far from the fixation location . Pooled features of size [ 192 , 29 ] along with the class token are passed through the nine transformer blocks , where class token refers to a learnable vector of size 192 values . Since the transformer blocks retain the dimensionality of the input , the output of the transformer blocks has the same size of [ 192 , 30 ] . We use the self-attention weights , corresponding to the class token feature vector from the last transformer block , in the ” Accumulator module ” ( see next section ) to predict the following fixation location . Feature vector corresponding to the class token is given as input to the average-pooling layer , averaging across such vectors received from all the fixations . Finally , the classification layer transforms the average-pooled feature vector into a logits vector .

Summary: The paper shows a method of Foveated Transformer motivated by a human foveal vision where the spatial resolution varies depending on the focused point. Namely, the method mainly consists of the spatial attention model that controls the gaze point and spatial-varying convolutional filter (dense pooling in center and sparse pooling in peripheral). The former simulates human gaze control and the latter mimics the human retina. The experimental validation is performed by the object recognition task (ImageNet) and adversarial attack tasks and performed better than the Deit-Tiny model that uses a similar number of parameters.