Glance-and-Gaze Vision Transformer

1 Introduction . Convolution Neural Networks ( CNNs ) have been dominating the field of computer vision , which have been a de-facto standard and achieved tremendous success in various tasks , e.g. , image classification [ 16 ] , object detection [ 15 ] , semantic segmentation [ 5 ] , etc . CNNs model images from a local-to-global perspective , starting with extracting local features such as edges and textures , and forming high-level semantic concepts gradually . Although CNNs prove to be successful for various vision tasks , they lack the ability to globally represent long-range dependencies . To compensate a global view to CNN , researchers explored different methods such as non-local operation [ 36 ] , self-attention [ 33 ] , Atrous Spatial Pyramid Pooling ( ASPP ) [ 5 ] . Recently , another type of networks with stacked Transformer blocks emerged . Unlike CNNs , Transformers naturally learn global features in a parameter-free manner , which makes them stronger alternatives and raises questions about the necessity of CNNs in vision systems . Since the advent of Vision Transformer ( ViT ) [ 12 ] , which applied Transformers to vision tasks by projecting and tokenizing natural images into sequences , various improvements have been introduced rapidly , e.g. , ∗Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . better training and distillation strategies [ 32 ] , tokenization [ 41 ] , position encoding [ 7 ] , local feature learning [ 14 ] . Moreover , besides Transformers ’ success on image classification , many efforts have been made to explore Transformers for various down-stream vision tasks [ 35 , 24 , 13 , 3 , 46 ] . Nevertheless , the advantages of Transformers come at a price . Since self-attention operates on the whole sequences , it incurs much more memory and computation costs than convolution , especially when it comes to natural images , whose lengths are usually much longer than word sequences , if treating each pixel as a token . Therefore , most existing works have to adopt a compromised strategy to embed a large image patch for each token , although treating smaller patches for tokens leads to a better performance ( e.g. , ViT-32 compared to ViT-16 [ 12 ] ) . To address this dilemma , various strategies have been proposed . For instance , Pyramid Vision Transformer ( PVT ) [ 35 ] introduced a progressive shrinking pyramid to reduce the sequence length of the Transformer with the increase of network depth , and adopted spatial-reduction attention , where key and value in the attention module are down-sampled to a lower resolution . Swin-Transformer [ 24 ] also adopted the pyramid structure , and further proposed to divide input feature maps into different fix-sized local windows , so that self-attention is computed within each window , which reduces the computation cost and makes it scalable to large image scales with linear complexity . Nonetheless , we notice that these strategies have some limitations : Spatial-reduction attention can reduce memory and computation costs to learn high-resolution feature maps , yet with a price of losing details which are expected from the high-resolution feature maps . Adopting self-attention within local windows is efficient with linear complexity , but it sacrifices the most significant advantage of Transformers in modeling long-range dependencies . To address these limitations , we propose Glance-and-Gaze Transformer ( GG-Transformer ) , inspired by the Glance-and-Gaze human behavior when recognizing objects in natural scenes [ 11 ] , which takes advantage of both the long-range dependency modeling ability of Transformers and locality of convolutions in a complementary manner . A GG-Transformer block consists of two parallel branches : A Glance branch performs self-attention within adaptively-dilated partitions of input images or feature maps , which preserves the global receptive field of the self-attention operation , meanwhile reduces its computation cost to a linear complexity as local window attention [ 24 ] does ; A Gaze branch compensates locality to the features obtained by the Glance branch , which is implemented by a light-weight depth-wise convolutional layer . A merging operation finally re-arranges the points in each partition to their original locations , ensuring that the output of the GG-Transformer block has the same size as the input . We evaluate GG-Transformer on several vision tasks and benchmarks including image classification on ImageNet [ 10 ] , object detection on COCO [ 23 ] , and semantic segmentation on ADE20K [ 48 ] , and show its efficiency and superior performance , compared to previous state-of-the-art Transformers . 2 Related Work . CNN and self-attention . Convolution has been the basic unit in deep neural networks for computer vision problems . Since standard CNN blocks were proposed in [ 22 ] , researchers have been working on designing stronger and more efficient network architectures , e.g. , VGG [ 30 ] , ResNet [ 16 ] , MobileNet [ 29 ] , and EfficientNet [ 31 ] . In addition to studying how to organize convolutional blocks into a network , several variants of the convolution layer have also been proposed , e.g. , group convolution [ 21 ] , depth-wise convolution [ 6 ] , and dilated convolution [ 40 ] . With the development of CNN architectures , researchers also seeked to improve contextual representation of CNNs . Representative works , such as ASPP [ 5 ] and PPM [ 45 ] enhance CNNs with multi-scale context , and NLNet [ 36 ] and CCNet [ 20 ] provided a non-local mechanism to CNNs . Moreover , instead of just using them as an add-on to CNNs , some works explored to use attention modules to replace convolutional blocks [ 18 , 28 , 34 , 44 ] . Vision Transformer . Recently , ViT [ 12 ] was proposed to adapt the Transformer [ 33 ] for image recognition by tokenizing and flattening 2D images into sequence of tokens . Since then , many works have been done to improve Transformers , making them more suitable for vision tasks . These works can be roughly categorized into three types : ( 1 ) Type I made efforts to improve the ViT design itself . For example , DeiT [ 32 ] introduced a training scheme to get rid of large-scale pre-training and distillation method to further improve the performance . T2T-ViT [ 41 ] presented a token-to-token operation as alternatives to patch embedding , which keeps better local details . ( 2 ) Type II tried to introduce convolution back into the ViT design . E.g. , Chu et al . [ 7 ] proposed to use convolution for position encoding . Wu et al . [ 37 ] used convolution to replace the linear projection layers in Transformers . ( 3 ) Type III tried to replace CNNs by building hierarchical Transformers as a plug-in backbone in many downstream tasks . Wang et al . [ 35 ] proposed a pyramid vision Transformer , which gradually downsamples the feature map and extract multi-scale features as common CNN backbones do . However , applying self-attention on high-resolution features is not affordable in terms of both memory and computation cost , thus they used spatial-reduction attention , which downsamples key and value in self-attention as a trade-off between efficiency and accuracy . Later , Liu et al . [ 24 ] proposed a new hierarchical Transformer architecture , named Swin-Transformer . To handle the expensive computation burden incurred with self-attention , they divided feature maps into several non-overlapped windows , and limited the self-attention operation to be performed within each window . By doing so , Swin-Transformer is more efficient and also scalable to large resolution input . Besides , to compensate the missing global information , a shifted window strategy is proposed to exchange information between different windows . Our method differs from aforementioned works in the following aspects : Type I , II methods usually utilize a large patch size and thus incompatible to work with high-resolution feature map . Type III methods proposed new attention mechanism to handle the extreme memory and computation burden with long sequences , but they sacrifices accuracy as a trade-off with efficiency . In contrast , GGTransformer proposes a more efficient Transformer block with a novel Glance-and-Gaze mechanism , which not only enables it to handle long sequences and scalable to high-resolution feature maps , but also leads to a better performance than other efficient alternatives . 3 Method . The design of GG-Transformer draws inspiration from how human beings observe the world , which follows the Glance and Gaze mechanism . Specifically , humans will glance at the global view , and meanwhile gaze into local details to obtain a comprehensive understanding to the environment . We note that these behaviors surprisingly match the property of self-attention and convolution , which models long-range dependencies and local context , respectively . Inspired from this , we propose GG-Transformer , whose Transformer block consists of two parallel branches : A Glance branch , where self-attention is performed to adaptively-dilated partitions of the input , and a Gaze branch , where a depth-wise convolutional layer is adopted to capture the local patterns . 3.1 Revisit Vision Transformer . We start with revisiting the formulation of vision Transformer block , which consists of multi-head self-attention ( MSA ) , layer normalization ( LN ) , and multi-layer perceptron ( MLP ) . A Transformer block processes input features as follows : z′ℓ = MSA ( LN ( zℓ−1 ) ) + zℓ−1 , ( 1 ) zℓ = MLP ( LN ( z ′ ℓ ) ) + z ′ ℓ , ( 2 ) where zℓ is the encoded image representation at the ℓ-th block . MSA gives Transformers the advantages of modeling a global relationship in a parameter-free manner , which is formulated as : MSA ( X ) = Softmax ( QKT√ C ) V , ( 3 ) where Q , K , V ∈ RN×C are the query , key , and value matrices which are linear mappings of input X ∈RN×C , C is the channel dimension of the input , and N is the length of input sequence . Note that for simplified derivation , we assume the number of heads is 1 in the multi-head self attention , which will not affect the following complexity analysis and can be easily generalize to more complex cases . For vision tasks , N is often related to the input height H and width W . In practice , a 2D image is often first tokenized based on non-overlapping image patch grids , which maps a 2D input with size H ×W into a sequence of token embeddings with length N = H×WP 2 , where ( P , P ) is the grid size . In MSA , the relationships between a token and all tokens are computed . Such designs , though effectively capturing long-range features , incur a computation complexity quadratic to N : Ω ( MSA ) = 4NC2 + 2N2C . ( 4 ) For ViTs that only work on 16× down-sampled feature maps ( i.e. , P = 16 ) , the computation cost is affordable , since in this scenario N = 14 × 14 = 196 ( for a typical ImageNet setting with input size 224× 224 ) . However , when it comes to a more general vision scenario with the need of dense prediction based on high-resolution feature maps ( such as semantic segmentation ) , where the cost dramatically increases by thousands of times or even more . Naively applying MSA to such high-resolution feature maps can easily lead to the problem of out-of-memory ( OOM ) , and extremely high computational cost . Although some efficient alternatives [ 35 , 24 ] were brought up recently , accuracy is often sacrificed as a trade-off of efficiency . To address this issue , we propose a new vision Transformer that can be applied to longer sequence while keeping high accuracy , inspired by the Glance-and-Gaze human behavior when recognizing objects in natural scenes [ 11 ] .

Summary: The paper presented a new Transformer-based model for visual recognition. The key idea is to use a checkerboard-like pattern (dilated splitting) to divide the 2D feature plan into multiple smaller partitions, where the standard self-attention can be efficiently computed on each of these partitions (glance branch). A local depthwise convolution was also added to link the partitions (gaze branch). To evaluate the proposed model, the authors presented extensive results across several visual recognition tasks, including image classification, semantic segmentation, and object detection. The results are promising.