Glance-and-Gaze Vision Transformer

1 Introduction . Convolution Neural Networks ( CNNs ) have been dominating the field of computer vision , which have been a de-facto standard and achieved tremendous success in various tasks , e.g. , image classification [ 16 ] , object detection [ 15 ] , semantic segmentation [ 5 ] , etc . CNNs model images from a local-to-global perspective , starting with extracting local features such as edges and textures , and forming high-level semantic concepts gradually . Although CNNs prove to be successful for various vision tasks , they lack the ability to globally represent long-range dependencies . To compensate a global view to CNN , researchers explored different methods such as non-local operation [ 36 ] , self-attention [ 33 ] , Atrous Spatial Pyramid Pooling ( ASPP ) [ 5 ] . Recently , another type of networks with stacked Transformer blocks emerged . Unlike CNNs , Transformers naturally learn global features in a parameter-free manner , which makes them stronger alternatives and raises questions about the necessity of CNNs in vision systems . Since the advent of Vision Transformer ( ViT ) [ 12 ] , which applied Transformers to vision tasks by projecting and tokenizing natural images into sequences , various improvements have been introduced rapidly , e.g. , ∗Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . better training and distillation strategies [ 32 ] , tokenization [ 41 ] , position encoding [ 7 ] , local feature learning [ 14 ] . Moreover , besides Transformers ’ success on image classification , many efforts have been made to explore Transformers for various down-stream vision tasks [ 35 , 24 , 13 , 3 , 46 ] . Nevertheless , the advantages of Transformers come at a price . Since self-attention operates on the whole sequences , it incurs much more memory and computation costs than convolution , especially when it comes to natural images , whose lengths are usually much longer than word sequences , if treating each pixel as a token . Therefore , most existing works have to adopt a compromised strategy to embed a large image patch for each token , although treating smaller patches for tokens leads to a better performance ( e.g. , ViT-32 compared to ViT-16 [ 12 ] ) . To address this dilemma , various strategies have been proposed . For instance , Pyramid Vision Transformer ( PVT ) [ 35 ] introduced a progressive shrinking pyramid to reduce the sequence length of the Transformer with the increase of network depth , and adopted spatial-reduction attention , where key and value in the attention module are down-sampled to a lower resolution . Swin-Transformer [ 24 ] also adopted the pyramid structure , and further proposed to divide input feature maps into different fix-sized local windows , so that self-attention is computed within each window , which reduces the computation cost and makes it scalable to large image scales with linear complexity . Nonetheless , we notice that these strategies have some limitations : Spatial-reduction attention can reduce memory and computation costs to learn high-resolution feature maps , yet with a price of losing details which are expected from the high-resolution feature maps . Adopting self-attention within local windows is efficient with linear complexity , but it sacrifices the most significant advantage of Transformers in modeling long-range dependencies . To address these limitations , we propose Glance-and-Gaze Transformer ( GG-Transformer ) , inspired by the Glance-and-Gaze human behavior when recognizing objects in natural scenes [ 11 ] , which takes advantage of both the long-range dependency modeling ability of Transformers and locality of convolutions in a complementary manner . A GG-Transformer block consists of two parallel branches : A Glance branch performs self-attention within adaptively-dilated partitions of input images or feature maps , which preserves the global receptive field of the self-attention operation , meanwhile reduces its computation cost to a linear complexity as local window attention [ 24 ] does ; A Gaze branch compensates locality to the features obtained by the Glance branch , which is implemented by a light-weight depth-wise convolutional layer . A merging operation finally re-arranges the points in each partition to their original locations , ensuring that the output of the GG-Transformer block has the same size as the input . We evaluate GG-Transformer on several vision tasks and benchmarks including image classification on ImageNet [ 10 ] , object detection on COCO [ 23 ] , and semantic segmentation on ADE20K [ 48 ] , and show its efficiency and superior performance , compared to previous state-of-the-art Transformers . 2 Related Work . CNN and self-attention . Convolution has been the basic unit in deep neural networks for computer vision problems . Since standard CNN blocks were proposed in [ 22 ] , researchers have been working on designing stronger and more efficient network architectures , e.g. , VGG [ 30 ] , ResNet [ 16 ] , MobileNet [ 29 ] , and EfficientNet [ 31 ] . In addition to studying how to organize convolutional blocks into a network , several variants of the convolution layer have also been proposed , e.g. , group convolution [ 21 ] , depth-wise convolution [ 6 ] , and dilated convolution [ 40 ] . With the development of CNN architectures , researchers also seeked to improve contextual representation of CNNs . Representative works , such as ASPP [ 5 ] and PPM [ 45 ] enhance CNNs with multi-scale context , and NLNet [ 36 ] and CCNet [ 20 ] provided a non-local mechanism to CNNs . Moreover , instead of just using them as an add-on to CNNs , some works explored to use attention modules to replace convolutional blocks [ 18 , 28 , 34 , 44 ] . Vision Transformer . Recently , ViT [ 12 ] was proposed to adapt the Transformer [ 33 ] for image recognition by tokenizing and flattening 2D images into sequence of tokens . Since then , many works have been done to improve Transformers , making them more suitable for vision tasks . These works can be roughly categorized into three types : ( 1 ) Type I made efforts to improve the ViT design itself . For example , DeiT [ 32 ] introduced a training scheme to get rid of large-scale pre-training and distillation method to further improve the performance . T2T-ViT [ 41 ] presented a token-to-token operation as alternatives to patch embedding , which keeps better local details . ( 2 ) Type II tried to introduce convolution back into the ViT design . E.g. , Chu et al . [ 7 ] proposed to use convolution for position encoding . Wu et al . [ 37 ] used convolution to replace the linear projection layers in Transformers . ( 3 ) Type III tried to replace CNNs by building hierarchical Transformers as a plug-in backbone in many downstream tasks . Wang et al . [ 35 ] proposed a pyramid vision Transformer , which gradually downsamples the feature map and extract multi-scale features as common CNN backbones do . However , applying self-attention on high-resolution features is not affordable in terms of both memory and computation cost , thus they used spatial-reduction attention , which downsamples key and value in self-attention as a trade-off between efficiency and accuracy . Later , Liu et al . [ 24 ] proposed a new hierarchical Transformer architecture , named Swin-Transformer . To handle the expensive computation burden incurred with self-attention , they divided feature maps into several non-overlapped windows , and limited the self-attention operation to be performed within each window . By doing so , Swin-Transformer is more efficient and also scalable to large resolution input . Besides , to compensate the missing global information , a shifted window strategy is proposed to exchange information between different windows . Our method differs from aforementioned works in the following aspects : Type I , II methods usually utilize a large patch size and thus incompatible to work with high-resolution feature map . Type III methods proposed new attention mechanism to handle the extreme memory and computation burden with long sequences , but they sacrifices accuracy as a trade-off with efficiency . In contrast , GGTransformer proposes a more efficient Transformer block with a novel Glance-and-Gaze mechanism , which not only enables it to handle long sequences and scalable to high-resolution feature maps , but also leads to a better performance than other efficient alternatives . 3 Method . The design of GG-Transformer draws inspiration from how human beings observe the world , which follows the Glance and Gaze mechanism . Specifically , humans will glance at the global view , and meanwhile gaze into local details to obtain a comprehensive understanding to the environment . We note that these behaviors surprisingly match the property of self-attention and convolution , which models long-range dependencies and local context , respectively . Inspired from this , we propose GG-Transformer , whose Transformer block consists of two parallel branches : A Glance branch , where self-attention is performed to adaptively-dilated partitions of the input , and a Gaze branch , where a depth-wise convolutional layer is adopted to capture the local patterns . 3.1 Revisit Vision Transformer . We start with revisiting the formulation of vision Transformer block , which consists of multi-head self-attention ( MSA ) , layer normalization ( LN ) , and multi-layer perceptron ( MLP ) . A Transformer block processes input features as follows : z′ℓ = MSA ( LN ( zℓ−1 ) ) + zℓ−1 , ( 1 ) zℓ = MLP ( LN ( z ′ ℓ ) ) + z ′ ℓ , ( 2 ) where zℓ is the encoded image representation at the ℓ-th block . MSA gives Transformers the advantages of modeling a global relationship in a parameter-free manner , which is formulated as : MSA ( X ) = Softmax ( QKT√ C ) V , ( 3 ) where Q , K , V ∈ RN×C are the query , key , and value matrices which are linear mappings of input X ∈RN×C , C is the channel dimension of the input , and N is the length of input sequence . Note that for simplified derivation , we assume the number of heads is 1 in the multi-head self attention , which will not affect the following complexity analysis and can be easily generalize to more complex cases . For vision tasks , N is often related to the input height H and width W . In practice , a 2D image is often first tokenized based on non-overlapping image patch grids , which maps a 2D input with size H ×W into a sequence of token embeddings with length N = H×WP 2 , where ( P , P ) is the grid size . In MSA , the relationships between a token and all tokens are computed . Such designs , though effectively capturing long-range features , incur a computation complexity quadratic to N : Ω ( MSA ) = 4NC2 + 2N2C . ( 4 ) For ViTs that only work on 16× down-sampled feature maps ( i.e. , P = 16 ) , the computation cost is affordable , since in this scenario N = 14 × 14 = 196 ( for a typical ImageNet setting with input size 224× 224 ) . However , when it comes to a more general vision scenario with the need of dense prediction based on high-resolution feature maps ( such as semantic segmentation ) , where the cost dramatically increases by thousands of times or even more . Naively applying MSA to such high-resolution feature maps can easily lead to the problem of out-of-memory ( OOM ) , and extremely high computational cost . Although some efficient alternatives [ 35 , 24 ] were brought up recently , accuracy is often sacrificed as a trade-off of efficiency . To address this issue , we propose a new vision Transformer that can be applied to longer sequence while keeping high accuracy , inspired by the Glance-and-Gaze human behavior when recognizing objects in natural scenes [ 11 ] .

Summary: The paper introduces a novel attention mechanism for the transformer architecture. Inspired by human gaze, authors propose to split the self-attention operation which is typically done across all patches in two parallel processes: one which reasons locally and another one that reasons globally. Authors evaluate their method in standard datasets for segmentation, classification and detection and ablate the main design decisions.