You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection

1 Introduction . Transformer [ 58 ] is born to transfer . In natural language processing ( NLP ) , the dominant approach is to first pre-train Transformer on large , generic corpora for general language representation learning , and then fine-tune or adapt the model on specific target tasks [ 18 ] . Recently , Vision Transformer ( ViT ) 1 [ 21 ] demonstrates that canonical Transformer encoder architecture directly inherited from NLP can perform surprisingly well on image recognition at scale using modern vision transfer learning recipe [ 33 ] . Taking sequences of image patch embeddings as inputs , ViT can successfully transfer pre-trained general visual representations from sufficient scale to more specific image classification tasks with fewer data points from a pure sequence-to-sequence perspective . Since a pre-trained Transformer can be successfully fine-tuned on sentence-level tasks [ 7 , 19 ] in NLP , as well as token-level tasks [ 48 , 52 ] , where models are required to produce fine-grained output at the token-level [ 18 ] . A natural question is : Can ViT transfer to more challenging object- and region-level target tasks in computer vision such as object detection other than image-level recognition ? ViT-FRCNN [ 6 ] is the first to use a pre-trained ViT as the backbone for a Faster R-CNN [ 50 ] object detector . However , this design can not get rid of the reliance on convolutional neural networks ( CNNs ) ∗Yuxin Fang and Bencheng Liao contributed equally . †Xinggang Wang is the corresponding author . This work was done when Yuxin Fang was interning at Horizon Robotics mentored by Rui Wu . 1There are various sophisticated or hybrid architectures termed as “ Vision Transformer ” . For disambiguation , in this paper , “ Vision Transformer ” and “ ViT ” refer to the canonical or vanilla Vision Transformer architecture proposed by Dosovitskiy et al . [ 21 ] unless specified . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and strong 2D inductive biases , as ViT-FRCNN re-interprets the output sequences of ViT to 2D spatial feature maps and depends on region-wise pooling operations ( i.e. , RoIPool [ 23 , 25 ] or RoIAlign [ 27 ] ) as well as region-based CNN architectures [ 50 ] to decode ViT features for object- and region-level perception . Inspired by modern CNN design , some recent works [ 39 , 59 , 62 , 65 ] introduce the pyramidal feature hierarchy , spatial locality , equivariant as well as invariant representations [ 24 ] to canonical Vision Transformer design , which largely boost the performance in dense prediction tasks including object detection . However , these architectures are performance-oriented and can not reflect the properties of the canonical or vanilla Vision Transformer [ 21 ] directly inherited from Vaswani et al . [ 58 ] . Another series of work , the DEtection TRansformer ( DETR ) families [ 10 , 72 ] , use a random initialized Transformer to encode & decode CNN features for object detection , which does not reveal the transferability of a pre-trained Transformer . Intuitively , ViT is designed to model long-range dependencies and global contextual information instead of local and region-level relations . Moreover , ViT lacks hierarchical architecture as modern CNNs [ 26 , 35 , 53 ] to handle the large variations in the scale of visual entities [ 1 , 37 ] . Based on the available evidence , it is still unclear whether a pure ViT can transfer pre-trained general visual representations from image-level recognition to the much more complicated 2D object detection task . To answer this question , we present You Only Look at One Sequence ( YOLOS ) , a series of object detection models based on the canonical ViT architecture with the fewest possible modifications , region priors , as well as inductive biases of the target task injected . Essentially , the change from a pre-trained ViT to a YOLOS detector is embarrassingly simple : ( 1 ) YOLOS replaces one [ CLS ] token for image classification in ViT with one hundred [ DET ] tokens for object detection . ( 2 ) YOLOS replaces the image classification loss in ViT with the bipartite matching loss to perform object detection in a set prediction manner following Carion et al . [ 10 ] , which can avoid re-interpreting the output sequences of ViT to 2D feature maps as well as prevent manually injecting heuristics and prior knowledge of object 2D spatial structure during label assignment [ 71 ] . Moreover , the prediction head of YOLOS can get rid of complex and diverse designs , which is as compact as a classification layer . Directly inherited from ViT [ 21 ] , YOLOS is not designed to be yet another high-performance object detector , but to unveil the versatility and transferability of pre-trained canonical Transformer from image recognition to the more challenging object detection task . Concretely , our main contributions are summarized as follows : • We use the mid-sized ImageNet-1k [ 51 ] as the sole pre-training dataset , and show that a vanilla ViT [ 21 ] can be successfully transferred to perform the complex object detection task and produce competitive results on COCO [ 36 ] benchmark with the fewest possible modifications , i.e. , by only looking at one sequence ( YOLOS ) . • For the first time , we demonstrate that 2D object detection can be accomplished in a pure sequence-to-sequence manner by taking a sequence of fixed-sized non-overlapping image patches as input . Among existing object detectors , YOLOS utilizes the minimal 2D inductive biases . • For the vanilla ViT , we find the object detection results are quite sensitive to the pre-train scheme and the detection performance is far from saturating . Therefore the proposed YOLOS can be also used as a challenging benchmark task to evaluate different ( label-supervised and self-supervised ) pre-training strategies for ViT . 2 You Only Look at One Sequence . As for the model design , YOLOS closely follows the original ViT architecture [ 21 ] , and is optimized for object detection in the same vein as Carion et al . [ 10 ] . YOLOS can be easily adapted to various canonical Transformer architectures available in NLP as well as in computer vision . This intentionally simple setup is not designed for better detection performance , but to exactly reveal characteristics of the Transformer family in object detection as unbiased as possible . 2.1 Architecture . An overview of the model is depicted in Fig . 1 . Essentially , the change from a ViT to a YOLOS detector is simple : ( 1 ) YOLOS drops the [ CLS ] token for image classification and appends one hundred randomly initialized learnable detection tokens ( [ DET ] tokens ) to the input patch embeddings ( [ PATCH ] tokens ) for object detection . ( 2 ) During training , YOLOS replaces the image classification loss in ViT with the bipartite matching loss to perform object detection in a set prediction manner following Carion et al . [ 10 ] . Stem . The canonical ViT [ 21 ] receives an 1D sequence of embedded tokens as the input . To handle 2D image inputs , we reshape the image x ∈ RH×W×C into a sequence of flattened 2D image patches xPATCH ∈ RN× ( P 2·C ) . Here , ( H , W ) is the resolution of the input image , C is the number of input channels , ( P , P ) is the resolution of each image patch , and N = HWP 2 is the resulting number of patches . Then we map xPATCH toD dimensions with a trainable linear projection E ∈ R ( P 2·C ) ×D . We refer to the output of this projection xPATCHE as [ PATCH ] tokens . Meanwhile , one hundred randomly initialized learnable [ DET ] tokens xDET ∈ R100×D are appended to the [ PATCH ] tokens . Position embeddings P ∈ R ( N+100 ) ×D are added to all the input tokens to retain positional information . We use the standard learnable 1D position embeddings following Dosovitskiy et al . [ 21 ] . The resulting sequence z0 serves as the input of YOLOS Transformer encoder . Formally : z0 = [ x1PATCHE ; · · · ; xNPATCHE ; x1DET ; · · · ; x100DET ] +P . ( 1 ) Body . The body of YOLOS is basically the same as ViT , which consists of a stack of Transformer encoder layers only [ 58 ] . [ PATCH ] tokens and [ DET ] tokens are treated equally and they perform global interactions inside Transformer encoder layers . Each Transformer encoder layer consists of one multi-head self-attention ( MSA ) block and one MLP block . LayerNorm ( LN ) [ 2 ] is applied before every block , and residual connections [ 26 ] are applied after every block [ 3 , 61 ] . The MLP contains one hidden layer with an intermediate GELU [ 29 ] non-linearity activation function . Formally , for the ` -th YOLOS Transformer encoder layer : z′ ` = MSA ( LN ( z ` −1 ) ) + z ` −1 , z ` = MLP ( LN ( z ′ ` ) ) + z ′ ` . ( 2 ) Detector Heads . The detector head of YOLOS gets rid of complex and heavy designs , and is as neat as the image classification layer of ViT . Both the classification and the bounding box regression heads are implemented by one MLP with separate parameters containing two hidden layers with intermediate ReLU [ 41 ] non-linearity activation functions . Detection Token . We purposefully choose randomly initialized [ DET ] tokens as proxies for object representations to avoid inductive biases of 2D structure and prior knowledge about the task injected during label assignment . When fine-tuning on COCO , for each forward pass , an optimal bipartite matching between predictions generated by [ DET ] tokens and ground truth objects is established . This procedure plays the same role as label assignment [ 10 , 71 ] , but is unaware of the input 2D structure , i.e. , YOLOS does not need to re-interpret the output sequence of ViT to an 2D feature maps for label assignment . Theoretically , it is feasible for YOLOS to perform any dimensional object detection without knowing the exact spatial structure and geometry , as long as the input is always flattened to a sequence in the same way for each pass . Fine-tuning at Higher Resolution . When fine-tuning on COCO , all the parameters are initialized from ImageNet-1k pre-trained weights except for the MLP heads for classification & bounding box regression as well as one hundred [ DET ] tokens , which are randomly initialized . During fine-tuning , the image has a much higher resolution than pre-training . We keep the patch size P unchanged , i.e. , P ×P = 16× 16 , which results in a larger effective sequence length . While ViT can handle arbitrary input sequence lengths , the positional embeddings need to adapt to the longer input sequences with various lengths . We perform 2D interpolation of the pre-trained position embeddings on the fly2 . Inductive Bias . We carefully design the YOLOS architecture for the minimal additional inductive biases injection . The inductive biases inherent from ViT come from the patch extraction at the network stem part as well as the resolution adjustment for position embeddings [ 21 ] . Apart from that , YOLOS adds no non-degenerated ( e.g. , 3× 3 or other non 1× 1 ) convolutions upon ViT 3 . From the representation learning perspective , we choose to use [ DET ] tokens to bind objects for final predictions to avoid additional 2D inductive biases as well as task-specific heuristics . The performance-oriented design inspired by modern CNN architectures such as pyramidal feature hierarchy , 2D local spatial attention as well as the region-wise pooling operation is not applied . All these efforts are meant to exactly unveil the versatility and transferability of pre-trained Transformers from image recognition to object detection in a pure sequence-to-sequence manner , with minimal knowledge about the input spatial structure and geometry . Comparisons with DETR . The design of YOLOS is deeply inspired by DETR [ 10 ] : YOLOS uses [ DET ] tokens following DETR as proxies for object representations to avoid inductive biases about 2D structures and prior knowledge about the task injected during label assignment , and YOLOS is optimized similarly as DETR . Meanwhile , there are some key differences between the two models : ( 1 ) DETR adopts a Transformer encoder-decoder architecture , while YOLOS chooses an encoder-only Transformer architecture . ( 2 ) DETR only employs pre-training on its CNN backbone but leaves the Transformer encoder & decoder being trained from random initialization , while YOLOS naturally inherits representations from any pre-trained canonical ViT . ( 3 ) DETR applies cross-attention between encoded image features and object queries with auxiliary decoding losses deeply supervised at each decoder layer , while YOLOS always looks at only one sequence for each encoder layer , without distinguishing [ PATCH ] tokens and [ DET ] tokens in terms of operations . Quantitative comparisons between the two are in Sec . 3.4 .

Summary: This paper presents the design and evaluation of a Transformer-based object detector which works in a pure sequence-to-sequence manner. The proposed YOLOS detector is built from ViT by borrowing the [DET] token from detection transformer (DETR). A rather interesting result is that such a pure seq-to-seq architecture, which only takes non-overlapping image patches as input, can do object detection with a reasonable accuracy.