ViDT: An Efficient and Effective Fully Transformer-based Object Detector

Transformers are transforming the landscape of computer vision , especially for recognition tasks . Detection transformers are the first fully end-to-end learning systems for object detection , while vision transformers are the first fully transformer-based architecture for image classification . In this paper , we integrate Vision and Detection Transformers ( ViDT ) to build an effective and efficient object detector . ViDT introduces a reconfigured attention module to extend the recent Swin Transformer to be a standalone object detector , followed by a computationally efficient transformer decoder that exploits multi-scale features and auxiliary techniques essential to boost the detection performance without much increase in computational load . Extensive evaluation results on the Microsoft COCO benchmark dataset demonstrate that ViDT obtains the best AP and latency trade-off among existing fully transformer-based object detectors , and achieves 49.2AP owing to its high scalability for large models . We will release the code and trained models upon acceptance . 1 INTRODUCTION . Object detection is the task of predicting both bounding boxes and object classes for each object of interest in an image . Modern deep object detectors heavily rely on meticulously designed components , such as anchor generation and non-maximum suppression ( Papageorgiou & Poggio , 2000 ; Liu et al. , 2020 ) . As a result , the performance of these object detectors depend on specific postprocessing steps , which involve complex pipelines and make fully end-to-end training difficult . Motivated by the recent success of Transformers ( Vaswani et al. , 2017 ) in NLP , numerous studies introduce Transformers into computer vision tasks . Carion et al . ( 2020 ) proposed Detection Transformers ( DETR ) to eliminate the meticulously designed components by employing a simple transformer encoder and decoder architecture , which serves as a neck component to bridge a CNN body for feature extraction and a detector head for prediction . Thus , DETR enables end-to-end training of deep object detectors . By contrast , Dosovitskiy et al . ( 2021 ) showed that a fully-transformer backbone without any convolutional layers , Vision Transformer ( ViT ) , achieves the state-of-theart results in image classification benchmarks . Approaches like ViT have been shown to learn effective representation models without strong human inductive biases , e.g. , meticulously designed components in object detection ( DETR ) , locality-aware designs such as convolutional layers and pooling mechanisms . However , there is a lack of effort to synergize DETR and ViT for a better object detection architecture . In this paper , we integrate both approaches to build a fully transformer-based , end-to-end object detector that achieves state-of-the-art performance without increasing computational load . A straightforward integration of DETR and ViT can be achieved by replacing the ResNet backbone ( body ) of DETR with ViT – Figure 2 ( a ) . This naive integration , DETR ( ViT ) 1 , has two limitations . First , the canonical ViT suffers from the quadratic increase in complexity w.r.t . image size , resulting in the lack of scalability . Furthermore , the attention operation at the transformer encoder and decoder ( i.e. , the “ neck ” component ) adds significant computational overhead to the detector . Therefore , the naive integration of DETR and ViT show very high latency – the blue lines of Figure 1 . Recently , Fang et al . ( 2021 ) propose an extension of ViT to object detection , named YOLOS , by appending the detection tokens [ DET ] to the patch tokens [ PATCH ] ( Figure 2 ( b ) ) , where [ DET ] tokens are learnable embeddings to specify different objects to detect . YOLOS is a neck-free architecture and removes the additional computational costs from the neck encoder . However , YOLOS shows limited performance because it can not use additional optimization techniques on the neck architecture , e.g. , multi-scale features and auxiliary loss . In addition , YOLOS can only accommodate the canonical transformer due to its architectural limitation , resulting in a quadratic complexity w.r.t . the input size . In this paper , we propose a novel integration of Vision and Detection Transformers ( ViDT ) ( Figure 2 ( c ) ) . Our contributions are three-folds . First , ViDT introduces a modified attention mechanism , named Reconfigured Attention Module ( RAM ) , that facilitates any ViT variant to handle the appended [ DET ] and [ PATCH ] tokens for object detection . Thus , we can modify the latest Swin Transformer ( Liu et al. , 2021 ) backbone with RAM to be an object detector and obtain high scalability using its local attention mechanism with linear complexity . Second , ViDT adopts a lightweight encoder-free neck architecture to reduce the computational overhead while still enabling the additional optimization techniques on the neck module . Note that the neck encoder is unnecessary because RAM directly extracts fine-grained representation for object detection , i.e. , [ DET ] tokens . As a result , ViDT obtains better performance than neck-free counterparts . Finally , we introduce a new concept of token matching for knowledge distillation , which brings additional performance gains from a large model to a small model without compromising detection efficiency . ViDT has two architectural advantages over existing approaches . First , similar to YOLOS , ViDT takes [ DET ] tokens as the additional input , maintaining a fixed scale for object detection , but constructs hierarchical representations starting with small-sized image patches for [ PATCH ] tokens . Second , ViDT can use the hierarchical ( multi-scale ) features and additional techniques without a significant computation overhead . Therefore , as a fully transformer-based object detector , ViDT facilitates better integration of vision and detection transformers . Extensive experiments on Microsoft COCO benchmark ( Lin et al. , 2014 ) show that ViDT is highly scalable even for large ViT models , such as Swin-base with 0.1 billion parameters , and achieves the best AP and latency trade-off . 2 PRELIMINARIES . Vision transformers process an image as a sequence of small-sized image patches , thereby allowing all the positions in the image to interact in attention operations ( i.e. , global attention ) . However , the canonical ViT ( Dosovitskiy et al. , 2021 ) is not compatible with a broad range of vision tasks due to its high computational complexity , which increases quadratically with respect to image size . The Swin Transformer ( Liu et al. , 2021 ) resolves the complexity issue by introducing the notion of shifted windows that support local attention and patch reduction operations , thereby improving compatibility for dense prediction task such as object detection . A few approaches use vision transformers as detector backbones but achieve limited success ( Heo et al. , 2021 ; Fang et al. , 2021 ) . 1We refer to each model based on the combinations of its body and neck . For example , DETR ( DeiT ) indicates that DeiT ( vision transformers ) is integrated with DETR ( detection transformers ) . Detection transformers eliminate the meticulously designed components ( e.g. , anchor generation and non-maximum suppression ) by combining convolutional network backbones and Transformer encoder-decoders . While the canonical DETR ( Carion et al. , 2020 ) achieves high detection performance , it suffers from very slow convergence compared to previous detectors . For example , DETR requires 500 epochs while the conventional Faster R-CNN ( Ren et al. , 2015 ) training needs only 37 epochs ( Wu et al. , 2019 ) . To mitigate the issue , Zhu et al . ( 2021 ) propose Deformable DETR which introduces deformable attention for utilizing multi-scale features as well as expediting the slow training convergence of DETR . In this paper , we use the Deformable DETR as our base detection transformer framework and integrate it with the recent vision transformers . DETR ( ViT ) is a straightforward integration of DETR and ViT , which uses ViT as a feature extractor , followed by the transformer encoder-decoder in DETR . As illustrated in Figure 2 ( a ) , it is a body–neck–head structure ; the representation of input [ PATCH ] tokens are extracted by the ViT backbone and then directly fed to the transformer-based encoding and decoding pipeline . To predict multiple objects , a fixed number of learnable [ DET ] tokens are provided as additional input to the decoder . Subsequently , output embeddings by the decoder produce final predictions through the detection heads for classification and box regression . Since DETR ( ViT ) does not modify the backbone at all , it can be flexibly changed to any latest ViT model , e.g. , Swin Transformer . Additionally , its neck decoder facilitates the aggregation of multi-scale features and the use of additional techniques , which help detect objects of different sizes and speed up training ( Zhu et al. , 2021 ) . However , the attention operation at the neck encoder adds significant computational overhead to the detector . In contrast , ViDT resolves this issue by directly extracting fine-grained [ DET ] features from Swin Transformer with RAM without maintaining the transformer encoder in the neck architecture . YOLOS ( Fang et al. , 2021 ) is a canonical ViT architecture for object detection with minimal modifications . As illustrated in Figure 2 ( b ) , YOLOS achieves a neck-free structure by appending randomly initialized learnable [ DET ] tokens to the sequence of input [ PATCH ] tokens . Since all the embeddings for [ PATCH ] and [ DET ] tokens interact via global attention , the final [ DET ] tokens are generated by the fine-tuned ViT backbone and then directly generate predictions through the detection heads without requiring any neck layer . While the naive DETR ( ViT ) suffers from the computational overhead from the neck layer , YOLOS enjoys efficient computations by treating the [ DET ] tokens as additional input for ViT . YOLOS shows that 2D object detection can be accomplished in a pure sequence-to-sequence manner , but this solution entails two inherent limitations : 1 ) YOLOS inherits the drawback of the canonical ViT ; the high computational complexity attributed to the global attention operation . As illustrated in Figure 1 , YOLOS shows very poor latency compared with other fully transformer-based detectors , especially when its model size becomes larger , i.e. , small→ base . Thus , YOLOS is not scalable for the large model . 2 ) YOLOS can not benefit from using additional techniques essential for better performance , e.g. , multi-scale features , due to the absence of the neck layer . Although YOLOS used the same DeiT backbone with Deformable DETR ( DeiT ) , its AP was lower than the straightforward integration . In contrast , the encoder-free neck architecture of ViDT enjoys the additional optimization techniques from Zhu et al . ( 2021 ) , resulting in the faster convergence and the better performance . Further , our RAM enables to combine Swin Transformer and the sequence-to-sequence paradigm for detection . 3 VIDT : VISION AND DETECTION TRANSFORMERS . ViDT first reconfigures the attention model of Swin Transformer to support standalone object detection while fully reusing the parameters of Swin Transformer . Next , it incorporates an encoder-free neck layer to exploit multi-scale features and two essential techniques : auxiliary decoding loss and iterative box refinement . We further introduce knowledge distillation with token matching to benefit from large ViDT models . 3.1 RECONFIGURED ATTENTION MODULE . Applying patch reduction and local attention scheme of Swin Transformer to the sequence-tosequence paradigm is challenging because ( 1 ) the number of [ DET ] tokens must be maintained at a fixed-scale and ( 2 ) the lack of locality between [ DET ] tokens . To address this challenge , we introduce a reconfigured attention module ( RAM ) 2 that decomposes a single global attention associated with [ PATCH ] and [ DET ] tokens into the three different attention , namely [ PATCH ] × [ PATCH ] , [ DET ] × [ DET ] , 2This reconfiguration scheme can be easily applied to other ViT variants with simple modification . and [ DET ] × [ PATCH ] attention . Based on the decomposition , the efficient schemes of Swin Transformer are applied only to [ PATCH ] × [ PATCH ] attention , which is the heaviest part in computational complexity , without breaking the two constraints on [ DET ] tokens . As illustrated in Figure 3 , these modifications fully reuse all the parameters of Swin Transformer by sharing projection layers for [ DET ] and [ PATCH ] tokens , and perform the three different attention operations : • [ PATCH ] × [ PATCH ] Attention : The initial [ PATCH ] tokens are progressively calibrated across the attention layers such that they aggregate the key contents in the global feature map ( i.e. , spatial form of [ PATCH ] tokens ) according to the attention weights , which are computed by 〈query , key〉 pairs . For [ PATCH ] × [ PATCH ] attention , Swin Transformer performs local attention on each window partition , but its shifted window partitioning in successive blocks bridges the windows of the preceding layer , providing connections among partitions to capture global information . Without modifying this concept , we use the same policy to generate hierarchical [ PATCH ] tokens . Thus , the number of [ PATCH ] tokens is reduced by a factor of 4 at each stage ; the resolution of feature maps decreases from H/4 ×W/4 to H/32 ×W/32 over a total of four stages , where H and W denote the width and height of the input image , respectively . • [ DET ] × [ DET ] Attention : Like YOLOS , we append one hundred learnable [ DET ] tokens as the additional input to the [ PATCH ] tokens . As the number of [ DET ] tokens specifies the number of objects to detect , their number must be maintained with a fixed-scale over the transformer layers . In addition , [ DET ] tokens do not have any locality unlike the [ PATCH ] tokens . Hence , for [ DET ] × [ DET ] attention , we perform global self-attention while maintaining the number of them ; this attention helps each [ DET ] token to localize a different object by capturing the relationship between them . • [ DET ] × [ PATCH ] Attention : This is cross-attention between [ DET ] and [ PATCH ] tokens , which produces an object embedding per [ DET ] token . For each [ DET ] token , the key contents in [ PATCH ] tokens are aggregated to represent the target object . Since the [ DET ] tokens specify different objects , it produces different object embeddings for diverse objects in the image . Without the crossattention , it is infeasible to realize the standalone object detector . As shown in Figure 3 , ViDT binds [ DET ] × [ DET ] and [ DET ] × [ PATCH ] attention to process them at once to increase efficiency . We replace all the attention modules in Swin Transformer with the proposed RAM , which receives [ PATCH ] and [ DET ] tokens ( as shown in “ Body ” of Figure 2 ( c ) ) and then outputs their calibrated new tokens by performing the three different attention operations in parallel . Positional Encoding . ViDT adopts different positional encodings for different types of attention . For [ PATCH ] × [ PATCH ] attention , we use the relative position bias ( Hu et al. , 2019 ) originally used in Swin Transformer . In contrast , the learnable positional encoding is added for [ DET ] tokens for [ DET ] × [ DET ] attention because there is no particular order between [ DET ] tokens . However , for [ DET ] × [ PATCH ] attention , it is crucial to inject spatial bias to the [ PATCH ] tokens due to the permutation-equivariant in transformers , ignoring spatial information of the feature map . Thus , ViDT adds the sinusoidalbased spatial positional encoding to the feature map , which is reconstructed from the [ PATCH ] tokens for [ DET ] × [ PATCH ] attention , as can be seen from the left side of Figure 3 . We present a thorough analysis of various spatial positional encodings in Section 4.2.1 . Use of [ DET ] × [ PATCH ] Attention . Applying cross-attention between [ DET ] and [ PATCH ] tokens adds additional computational overhead to Swin Transformer , especially when it is activated at the bottom layer due to the large number of [ PATCH ] tokens . To minimize such computational overhead , ViDT only activates the cross-attention at the last stage ( the top level of the pyramid ) of Swin Transformer , which consists of two transformer layers that receives [ PATCH ] tokens of size H/32×W/32 . Thus , only self-attention for [ DET ] and [ PATCH ] tokens are performed for the remaining stages except the last one . In Section 4.2.2 we show that this design choice helps achieve the highest FPS , while achieving similar detection performance as when cross-attention is enabled at every stage . We provide more details on RAM including its complexity analysis and algorithmic design in Appendix A .

Summary: The paper builds upon the recent advances in transformer based image classification methods (ViT variants) and detection methods (DETR variants). It argues that naively replacing the conv feature backbone in DETR with a ViT based one, is problematic due to (i) the quadratic complexity of the self attention module in ViT, and (ii) then again the attention module in the transformer encoder decoder part (which they call "neck"). To remedy, it proposes to build on top of Swin Transformer backbone, by proposing a novel reconfigured attention module (RAM), and further removing the encoder-decoder in the neck, replacing it with only a lightweight decoder.