You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection

1 Introduction . Transformer [ 58 ] is born to transfer . In natural language processing ( NLP ) , the dominant approach is to first pre-train Transformer on large , generic corpora for general language representation learning , and then fine-tune or adapt the model on specific target tasks [ 18 ] . Recently , Vision Transformer ( ViT ) 1 [ 21 ] demonstrates that canonical Transformer encoder architecture directly inherited from NLP can perform surprisingly well on image recognition at scale using modern vision transfer learning recipe [ 33 ] . Taking sequences of image patch embeddings as inputs , ViT can successfully transfer pre-trained general visual representations from sufficient scale to more specific image classification tasks with fewer data points from a pure sequence-to-sequence perspective . Since a pre-trained Transformer can be successfully fine-tuned on sentence-level tasks [ 7 , 19 ] in NLP , as well as token-level tasks [ 48 , 52 ] , where models are required to produce fine-grained output at the token-level [ 18 ] . A natural question is : Can ViT transfer to more challenging object- and region-level target tasks in computer vision such as object detection other than image-level recognition ? ViT-FRCNN [ 6 ] is the first to use a pre-trained ViT as the backbone for a Faster R-CNN [ 50 ] object detector . However , this design can not get rid of the reliance on convolutional neural networks ( CNNs ) ∗Yuxin Fang and Bencheng Liao contributed equally . †Xinggang Wang is the corresponding author . This work was done when Yuxin Fang was interning at Horizon Robotics mentored by Rui Wu . 1There are various sophisticated or hybrid architectures termed as “ Vision Transformer ” . For disambiguation , in this paper , “ Vision Transformer ” and “ ViT ” refer to the canonical or vanilla Vision Transformer architecture proposed by Dosovitskiy et al . [ 21 ] unless specified . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and strong 2D inductive biases , as ViT-FRCNN re-interprets the output sequences of ViT to 2D spatial feature maps and depends on region-wise pooling operations ( i.e. , RoIPool [ 23 , 25 ] or RoIAlign [ 27 ] ) as well as region-based CNN architectures [ 50 ] to decode ViT features for object- and region-level perception . Inspired by modern CNN design , some recent works [ 39 , 59 , 62 , 65 ] introduce the pyramidal feature hierarchy , spatial locality , equivariant as well as invariant representations [ 24 ] to canonical Vision Transformer design , which largely boost the performance in dense prediction tasks including object detection . However , these architectures are performance-oriented and can not reflect the properties of the canonical or vanilla Vision Transformer [ 21 ] directly inherited from Vaswani et al . [ 58 ] . Another series of work , the DEtection TRansformer ( DETR ) families [ 10 , 72 ] , use a random initialized Transformer to encode & decode CNN features for object detection , which does not reveal the transferability of a pre-trained Transformer . Intuitively , ViT is designed to model long-range dependencies and global contextual information instead of local and region-level relations . Moreover , ViT lacks hierarchical architecture as modern CNNs [ 26 , 35 , 53 ] to handle the large variations in the scale of visual entities [ 1 , 37 ] . Based on the available evidence , it is still unclear whether a pure ViT can transfer pre-trained general visual representations from image-level recognition to the much more complicated 2D object detection task . To answer this question , we present You Only Look at One Sequence ( YOLOS ) , a series of object detection models based on the canonical ViT architecture with the fewest possible modifications , region priors , as well as inductive biases of the target task injected . Essentially , the change from a pre-trained ViT to a YOLOS detector is embarrassingly simple : ( 1 ) YOLOS replaces one [ CLS ] token for image classification in ViT with one hundred [ DET ] tokens for object detection . ( 2 ) YOLOS replaces the image classification loss in ViT with the bipartite matching loss to perform object detection in a set prediction manner following Carion et al . [ 10 ] , which can avoid re-interpreting the output sequences of ViT to 2D feature maps as well as prevent manually injecting heuristics and prior knowledge of object 2D spatial structure during label assignment [ 71 ] . Moreover , the prediction head of YOLOS can get rid of complex and diverse designs , which is as compact as a classification layer . Directly inherited from ViT [ 21 ] , YOLOS is not designed to be yet another high-performance object detector , but to unveil the versatility and transferability of pre-trained canonical Transformer from image recognition to the more challenging object detection task . Concretely , our main contributions are summarized as follows : • We use the mid-sized ImageNet-1k [ 51 ] as the sole pre-training dataset , and show that a vanilla ViT [ 21 ] can be successfully transferred to perform the complex object detection task and produce competitive results on COCO [ 36 ] benchmark with the fewest possible modifications , i.e. , by only looking at one sequence ( YOLOS ) . • For the first time , we demonstrate that 2D object detection can be accomplished in a pure sequence-to-sequence manner by taking a sequence of fixed-sized non-overlapping image patches as input . Among existing object detectors , YOLOS utilizes the minimal 2D inductive biases . • For the vanilla ViT , we find the object detection results are quite sensitive to the pre-train scheme and the detection performance is far from saturating . Therefore the proposed YOLOS can be also used as a challenging benchmark task to evaluate different ( label-supervised and self-supervised ) pre-training strategies for ViT . 2 You Only Look at One Sequence . As for the model design , YOLOS closely follows the original ViT architecture [ 21 ] , and is optimized for object detection in the same vein as Carion et al . [ 10 ] . YOLOS can be easily adapted to various canonical Transformer architectures available in NLP as well as in computer vision . This intentionally simple setup is not designed for better detection performance , but to exactly reveal characteristics of the Transformer family in object detection as unbiased as possible . 2.1 Architecture . An overview of the model is depicted in Fig . 1 . Essentially , the change from a ViT to a YOLOS detector is simple : ( 1 ) YOLOS drops the [ CLS ] token for image classification and appends one hundred randomly initialized learnable detection tokens ( [ DET ] tokens ) to the input patch embeddings ( [ PATCH ] tokens ) for object detection . ( 2 ) During training , YOLOS replaces the image classification loss in ViT with the bipartite matching loss to perform object detection in a set prediction manner following Carion et al . [ 10 ] . Stem . The canonical ViT [ 21 ] receives an 1D sequence of embedded tokens as the input . To handle 2D image inputs , we reshape the image x ∈ RH×W×C into a sequence of flattened 2D image patches xPATCH ∈ RN× ( P 2·C ) . Here , ( H , W ) is the resolution of the input image , C is the number of input channels , ( P , P ) is the resolution of each image patch , and N = HWP 2 is the resulting number of patches . Then we map xPATCH toD dimensions with a trainable linear projection E ∈ R ( P 2·C ) ×D . We refer to the output of this projection xPATCHE as [ PATCH ] tokens . Meanwhile , one hundred randomly initialized learnable [ DET ] tokens xDET ∈ R100×D are appended to the [ PATCH ] tokens . Position embeddings P ∈ R ( N+100 ) ×D are added to all the input tokens to retain positional information . We use the standard learnable 1D position embeddings following Dosovitskiy et al . [ 21 ] . The resulting sequence z0 serves as the input of YOLOS Transformer encoder . Formally : z0 = [ x1PATCHE ; · · · ; xNPATCHE ; x1DET ; · · · ; x100DET ] +P . ( 1 ) Body . The body of YOLOS is basically the same as ViT , which consists of a stack of Transformer encoder layers only [ 58 ] . [ PATCH ] tokens and [ DET ] tokens are treated equally and they perform global interactions inside Transformer encoder layers . Each Transformer encoder layer consists of one multi-head self-attention ( MSA ) block and one MLP block . LayerNorm ( LN ) [ 2 ] is applied before every block , and residual connections [ 26 ] are applied after every block [ 3 , 61 ] . The MLP contains one hidden layer with an intermediate GELU [ 29 ] non-linearity activation function . Formally , for the ` -th YOLOS Transformer encoder layer : z′ ` = MSA ( LN ( z ` −1 ) ) + z ` −1 , z ` = MLP ( LN ( z ′ ` ) ) + z ′ ` . ( 2 ) Detector Heads . The detector head of YOLOS gets rid of complex and heavy designs , and is as neat as the image classification layer of ViT . Both the classification and the bounding box regression heads are implemented by one MLP with separate parameters containing two hidden layers with intermediate ReLU [ 41 ] non-linearity activation functions . Detection Token . We purposefully choose randomly initialized [ DET ] tokens as proxies for object representations to avoid inductive biases of 2D structure and prior knowledge about the task injected during label assignment . When fine-tuning on COCO , for each forward pass , an optimal bipartite matching between predictions generated by [ DET ] tokens and ground truth objects is established . This procedure plays the same role as label assignment [ 10 , 71 ] , but is unaware of the input 2D structure , i.e. , YOLOS does not need to re-interpret the output sequence of ViT to an 2D feature maps for label assignment . Theoretically , it is feasible for YOLOS to perform any dimensional object detection without knowing the exact spatial structure and geometry , as long as the input is always flattened to a sequence in the same way for each pass . Fine-tuning at Higher Resolution . When fine-tuning on COCO , all the parameters are initialized from ImageNet-1k pre-trained weights except for the MLP heads for classification & bounding box regression as well as one hundred [ DET ] tokens , which are randomly initialized . During fine-tuning , the image has a much higher resolution than pre-training . We keep the patch size P unchanged , i.e. , P ×P = 16× 16 , which results in a larger effective sequence length . While ViT can handle arbitrary input sequence lengths , the positional embeddings need to adapt to the longer input sequences with various lengths . We perform 2D interpolation of the pre-trained position embeddings on the fly2 . Inductive Bias . We carefully design the YOLOS architecture for the minimal additional inductive biases injection . The inductive biases inherent from ViT come from the patch extraction at the network stem part as well as the resolution adjustment for position embeddings [ 21 ] . Apart from that , YOLOS adds no non-degenerated ( e.g. , 3× 3 or other non 1× 1 ) convolutions upon ViT 3 . From the representation learning perspective , we choose to use [ DET ] tokens to bind objects for final predictions to avoid additional 2D inductive biases as well as task-specific heuristics . The performance-oriented design inspired by modern CNN architectures such as pyramidal feature hierarchy , 2D local spatial attention as well as the region-wise pooling operation is not applied . All these efforts are meant to exactly unveil the versatility and transferability of pre-trained Transformers from image recognition to object detection in a pure sequence-to-sequence manner , with minimal knowledge about the input spatial structure and geometry . Comparisons with DETR . The design of YOLOS is deeply inspired by DETR [ 10 ] : YOLOS uses [ DET ] tokens following DETR as proxies for object representations to avoid inductive biases about 2D structures and prior knowledge about the task injected during label assignment , and YOLOS is optimized similarly as DETR . Meanwhile , there are some key differences between the two models : ( 1 ) DETR adopts a Transformer encoder-decoder architecture , while YOLOS chooses an encoder-only Transformer architecture . ( 2 ) DETR only employs pre-training on its CNN backbone but leaves the Transformer encoder & decoder being trained from random initialization , while YOLOS naturally inherits representations from any pre-trained canonical ViT . ( 3 ) DETR applies cross-attention between encoded image features and object queries with auxiliary decoding losses deeply supervised at each decoder layer , while YOLOS always looks at only one sequence for each encoder layer , without distinguishing [ PATCH ] tokens and [ DET ] tokens in terms of operations . Quantitative comparisons between the two are in Sec . 3.4 .

Summary: This paper explores how to transfer the pure ViT that is pre-trained on ImageNet to the more challenging object detection task. To cope with the object detection task, the proposed YOLOS first drops the CLS tokens in ViT and appends learnable DET tokens. Besides, the bipartite matching loss is utilized to perform set prediction for objects. Extensive experiments are provided to analyze the impacts of pre-training schemes. The proposed YOLOS also shows good performance for object detection on COCO benchmark.