ViDT: An Efficient and Effective Fully Transformer-based Object Detector
Authors: Hwanjun Song
Deqing Sun
Sanghyuk Chun
Varun Jampani
Dongyoon Han
Byeongho Heo
Wonjae Kim
Ming-Hsuan Yang
Venue: ICLR 2022 Poster (2022)
Keywords: object detection
vision transformer
detection transformer

Abstract: Transformers are transforming the landscape of computer vision, especially for recognition tasks. Detection transformers are the first fully end-to-end learning systems for object detection, while vision transformers are the first fully transformer-based architecture for image classification. In this paper, we integrate Vision and Detection Transformers (ViDT) to build an effective and efficient object detector. ViDT introduces a reconfigured attention module to extend the recent Swin Transformer to be a standalone object detector, followed by a computationally efficient transformer decoder that exploits multi-scale features and auxiliary techniques essential to boost the detection performance without much increase in computational load. Extensive evaluation results on the Microsoft COCO benchmark dataset demonstrate that ViDT obtains the best AP and latency trade-off among existing fully transformer-based object detectors, and achieves 49.2AP owing to its high scalability for large models. We release the code and trained models at https://github.com/naver-ai/vidt.

Published as a conference paper at ICLR 2022
VIDT: AN EFFICIENT AND EFFECTIVE
FULLY TRANSFORMER-BASER OBJECT DETECTOR
Hwanjun Song', Deqing Sun?, Sanghyuk Chun', Varun Jampani', Dongyoon Han',
Byeongho Heo',
Wonjae Kim', Ming-Hsuan Yang?.3.4
INAVER AT Lab
&Google Research University of California at Merced *Yonsei University
{hwanjun song, sanghyuk.em c, dongyoon. han, bh,hco, wonjae kin//navcrcoorp.om
fdogingsun, varunjampani }égoogle. com, mayangeucmercod..o edu
ABSTRACT
Transformers are transforming the landscape computer vision, especially for recognition tasks. Detection transformers are the first fully end-to-end learn- ing systems for object detection, while vision transformers are the first fully transformer-based architecture for image classification. In this paper, we integrate Vision and Detection Transformers (ViDT) to build an effective and efficient ob- ject detector. ViDT introduces reconfigured attention module extend the recent Swin Transformer to be a standalone object detector, followed by a computation- ally efficient transformer decoder that exploits multi-scale features and auxiliary techniques essential to boost the detection performance without much increase in computational load. Extensive evaluation results on the Microsoft COCO bench- mark dataset demonstrate that ViDT obtains the best AP and latency trade-off among existing fully transformer- based object detectors, and achieves 49.2AP ow- ing to its high scalability for large models. We release the code and trained models at https://github... ctwcuuu..ooooo wwwc//ooo/cceeeeeeeecoo htw///vvverraaae naver-ai/vie........
INTRODUCTION
Object detection the task of predicting both bounding boxes and object classes for each object of interest in an image. Modern deep object detectors heavily rely on meticulously designed compo- nents, such anchor generation and non- maximum suppression (Papageorgiou Poggio, 2000: Liu et al,, 2020). As a result, the performance of these object detectors depend on specific postprocessing steps, which involve complex pipelines and make fully end-to-end training difficult.
Motivated by the recent success of Transformers (Vaswani et al., 2017) in NLP, numerous stud- ies introduce Transformers into computer vision tasks. Carion al. (2020) proposed Detection Transformers (DETR) to eliminate the meticulously designed components by employing simple transformer encoder and decoder architecture, which serves as a neck component to bridge a CNN body for feature extraction and detector head for prediction. Thus, DETR enables end-to-end train- ing of deep object detectors. By contrast, Dosovitskiy et al. (2021) showed that a fully-transformer backbone without any convolutional layers, Vi- sion Transformer (VTT), achieves the state-of-the- results in image classification benchmarks. Ap- proaches like ViT have been shown to learn effective representation models without strong human induc- tive biases, e.g., meticulously designed components in object detection (DETR), locality-awars designs such as convolutional layers and pooling mecha- nisms. However, there is lack of effort to synergize DETR and ViT for better object detection archi- tecture. In this paper, we integrate both approaches to build fully transformer-based end-to-end object detector that achieves state-of ofte-of-thhear performance without increasing computational load.
DETR (DeiT)
DETR (Swin)
Deformable DETR (Deil)
Deformable DETR (Swin) ViDT
YOLOS
52
booo
boon
small
Smmll
46
small
base
timy
nacoo
40
base
bese
small
34
timy
small
"esse
tiny
small
28
tiy
20
80
140
200
260
Latency (MS per Image)
Figure AP and latency (milliseconds) summa- rized Table 2. The text in the plot indicates the backbone model size.
A straightforward integration of DETR and VIT can be achieved by replacing the ResNet backbone

---

Published as a conference paper at ICLR 2022
Hody
Neck
Body
Body
Neck
Head
Head
Heed
Patch Tokens
Path
Det.
Class.
Cless.
Class
VIT Backbone
Tranes Enc.
Trans.
Trans. Dec.
Dec.
Patch Tokens
Det.
Patch
Det.
Box Reys
Box Rag
Box
Patch Tokers
Det.
Patch Tokems
Det.
Patrh Tokess
Det.
Pairh Tokens
Det
Det.
(a) DETR (VIT).
(b) YOLOS.
(c) VIDT (Ours).
Figure 2. Pipelines of fully transformer-bssed object detectors. DETR (ViT) means Detection Transformer that uses VIT as body. The proposed ViDT synergizes DETR (ViT) and YOLOS and achieves best AP and latency trade-off among fully transformer-base object detectors.
(body) of DETR with ViT Figure 2(a). This naive integration, DETR (ViT)'. has two limitations. First, the canonical ViT suffers from the quadratic increase in complexity w.r.t. image size, resulting in the lack of scalability. Furthermore, the attention operation at the transformer encoder and decoder (i.e., the "neck" component) adds significant computational overhead to the detector. Therefore, the naive integration of DETR and ViT show very high latency the blue lines of Figure 1.
Recently, Fang et al. (2021) propose an extension of VIT to object detection, named YOLOS, by ap- pending the detection tokens [DET] to the patch tokens [PATCH] (Figure 2(b)), where [DET] tokens learnall embeddings to specify different objects to detect. YOLOS is a neck-free architecture and removes the additional computational costs from the neck encoder. However, YOLOS shows limited performance because it cannot use additional optimization techniques on the neck architecture, e.g., multi-scale features and auxiliary loss. addition, YOLOS can only accommodate the canonical transformer due to its architectural limitation, resulting in quadratic complexity w.r.t. the input size.
In this paper, we propose a novel integration of Vision and Detection Transformers (ViDT) (Figure 2(c)). Our contributions are three-folds. First, ViDT introduces a modified attention mechanism, named Reconfigured Attention Module (RAM), that facilitates any VIT variant to handle the ap- pended [DET] and [PATCH] tokens for object detection. Thus, former (Liu et al., 2021) backbone with RAM to be an object detector and obtain high scalabil- ity using its local attention mechanism with linear complexity. Second, VIDT adopts a lightweight encoder-free neck architecture to reduce the computational overhead while still enabling the ad- ditional optimization techniques the neck module. Note that the neck encoder is unnecessary because RAM directly extracts fine-grained representation for object detection, i.e., [DET] tokens. As a result, ViDT obtains better performance than neck-free counterparts. Finally, we introduce a new concept of token matching for knowledge distillation, which brings additional performance gains from a large model to small model without compromising detection efficiency.
modify the latest Swin Trans-
ViDT has two architectural advantages over existing approaches. First, similar to YOLOS, ViDT takes [ЕT)] tokens as the additional input, maintaining fixed scale for object detection, but con- structs hierarchical representations starting with small-sized image patches for [PATCH] tokens. Sec- ond, ViDT can use the hierarchical (multi-scale) features and additional techniques without a signifi- cant computation overhead. Therefore, as a fully transformer-based object detector, ViDT facilitates better integration of vision and detection transformers. Extensive experiments on Microsoft COCO benchmark (Lin et al., 2014) show that ViDT is highly scalable even for large ViT models, such as Swin-base with 0.1 billion parameters, and achieves the best AP and latency trade-off.
2 PRELIMINARIES
Vision transformers process an image a sequence of small-sized image patches, thereby allow- ing all the positions in the image to interact in attention operations (i.c., global attention). However, the canonical ViT (Dosovitskiy et al., 2021) is not compatible with broad range of vision tasks due to its high computational complexity. which increases quadratically with respect to image size. The Swin Transformer (Liu et al., 2021) resolves the complexity issue by introducing the notion of shifted windows that support local attention and patch reduction operations, thereby improving compatibility for dense prediction task such as object detection. few approaches use vision trans- formers detector backbones but achieve limited success (Heo et al., 2021; Fang et al.. 2021).
We refer to each model based the combinations of its body and neck. For example, DETR (DeiT) indicates that DeiT (vision transformers) is integrated with DETR (detection transformers).
2

---

Published as a conference paper at ICLR 2022
Detection transformers eliminate the meticulously designed components (e.g., anchor generation and non-maximum suppression) by combining convolutional network backbones and Transformer encoder-decoders. While the canonical DETR (Carion et al., 2020) achieves high detection perfor- mance, suffers from very slow convergence compared to previous detectors. For example, DETR requires 500 epochs while the conventional Faster R-CNN (Ren al., 2015) training needs only 37 epochs (Wu et al., 2019). To mitigate the issue, Zhu et al. (2021) propose Deformable DETR which introduces deformable attention for utilizing multi-scale features as well as expediting the slow training convergence of DETR. In this paper, we use the Deformable DETR as our base detection transformer framework and integrate with the recent vision transformers.
DETR (ViT) is a straightforward integration of DETR and ViT, which uses ViT as feature ex- tractor, followed by the transformer encoder-decoder in DETR. As illustrated in Figure 2(a), it is a body-neck-hen
structure; the representation of input [PATCH] tokens are extracted by the ViT backbone and then directly fed to the cransformer-based encoding and decoding pipeline. To predict multiple objects, a fixed number of Iearnable [DET] tokens are provided as additional input to the decoder. Subsequently, output embeddings by the decoder produce final predictions through the de- tection heads for classification and box regression. Since DETR (ViT) does not modify the backbone at all, it can be flexibly changed to any latest ViT model, e.g., Swin Transformer. Additionally, its neck decoder facilitates the aggregation of multi-scale features and the use of additional techniques, which help detect objects of different sizes and speed up training (Zhu et al.. 2021). However, the attention operation at the neck encoder adds significant computational overhead to the detector. In contrast, ViDT resolves this issue by directly extracting fine-grained [DET] features from Swin Transformer with RAM without maintaining the transformer encoder in the neck architecture.
YOLOS (Fang et al., 2021) is a canonical VIT architecture for object detection with minimal modifications. As illustrated in Figure 2(b), YOLOS achieves neck-free structure by appending randomly initialized learnable DET) tokens to the sequence of input [PATCH] tokens. Since all the embeddings for [PATCH] and [DET] tokens interact via global attention, the final [DET] tokens are gen- erated by the fine-tuned ViT backbone and then directly generate predictions through the detection heads without requiring any neck layer. While the naive DETR (ViT) suffers from the computational overhead from the neck layer, YOLOS enjoys efficient computations by treating the [DET] tokens as additional input for ViT. YOLOS shows that 2D object detection can be accomplished in a pure gequence-to-squeene manner, but this solution entails two inherent limitations:
1) YOLOS inherits the drawback of the canonical VIT; the high computational complexity attributed to the global attention operation. As illustrated in Figure 1, YOLOS shows very poor latency compared with other fully transformer-based detectors, especially when its model size becomes larger, i.e., small + base. Thus, YOLOS is not scalable for the large model.
2) YOLOS cannot benefit from using additional techniques essential for better performance, c.g., multi-scale features, due to the absence of the neck layer. Although YOLOS used the same DeiT backbone with Deformable DETR (DeiT), its AP was lower than the straightforward integration.
In contrast, the encoder- free neck architecture of ViDT enjoys the additional optimization techniques from Zhu et al. (2021), resulting in the faster convergence and the better performance. Further, our RAM enables to combine Swin Transformer and the sequence- seneence-to-sqqueee sequence paradigm for detection.
VIDT: VISION AND DETECTION TRANSFORMERS
ViDT first reconfigures the attention model of Swin Transformer to support standalone object detec- tion while fully reusing the parameters of Swin Transformer. Next, it incorporates encoder-fri free neck layer to exploit multi-scale features and two essential techniques: auxiliary decoding loss and iterative box refinement. We further introduce knowledge distillation with token matching to benefit from large ViDT models.
3.1 RECONFIGURED ATTENTION MODULE
Applying patch reduction and local attention scheme of Swin Transformer to the sequence-to- sequence paradigm is challenging because (1) the number of [EET] tokens must be maintained at a fixed-scale and (2) the lack of locality between [DET] tokens. To address this challenge, we introduce a reconfigured attention module (RAM)² that decomposes a single global attention associated with [PАTC]] and [DET] tokens into the three different attention, namely [ATCC] x [PATCH], [DET) x [DET],
Thii reconfiguration scheme be easily applied other VIT variants with simple modification.
3

---

Published as a conference paper at ICLR 2022
(AICHI-(PACCO with Window Partitions
PATCH] Tokens
Festure Map
Shared Project Layer
Relatve Pos
PATCHI Tokens
Reylinate
Feature
Resssemble
Map
DETT IDET & PATCII
Spataa Postional
Enooding
Deleciin Token
DET Tokens
Encoding
[ET]] Tolens
Figure 3. Reconfigured Attention Module (Q: query, K: key, V: value). The skip connection and feedforward networks following the attention operation omitted just for ease of exposition.
and [EET] x [PATCH] attention. Based on the decomposition, the efficient schemes of Swin Trans- former are applied only to [PATCH] x [PATCH] attention, which is the heaviest part in computational complexity, without breaking the two constraints on [DET] tokens. As illustrated in Figure 3, these modifications fully reuse all the parameters of Swin Transformer by sharing projection layers for [DET] and [РАTC]| tokens, and perform the three different attention operations:
[PATCH] x [РAТCH] Attention: The initial [РATCH] tokens are progressively calibrated across the attention layers such that they aggregate the key contents in the global feature map (i.c., spatial form of [PATCH] tokens) according to the attention weights, which are computed by (query, key) pairs. For [РATCH] x [PATCH] attention, Swin Transformer performs local attention on each window partition, but its shifted window partitioning in successive blocks bridges the windows of the preceding layer, providing connections among partitions to capture global information. Without modifying this concept,
the same policy to generate hierarchical [PATCH] tokens. Thus, the number of [PATCH] tokens is reduced by factor of 4 at each stage; the resolution of feature maps decreases from H/4 W/ 4 to H/32> W/32 over total of four stages, where H and W denote the width and height of the input image, respectively.
[ET) x [DET] Attention: Like YOLOS, we append one hundred learnable (DET) tokens as the ad- ditional input to the [PATCH| tokens. As the number of [DE]] tokens specifies the number of objects to detect, their number must be maintained with a fixed-scale over the transformer layers. In ad- dition, [DET] tokens do not have any locality unlike the [PATCH] tokens. Hence, for (DET) [DET) attention, we perform global self-attention while maintaining the number of them; this attention helps each [DET] token to localize different object by capturing the relationship between them.
[ET) [АТCH Attention: This is cross- antennattnnion between [DET] and [PATCH] tokens, which pro- duces an object embedding per [DET] token. For each [DET] token, the key contents in [PATCH] tokens are aggregated to represent the target object. Since the [DET] tokens specify different ob- jects, it produces different object embeddings for diverse objects in the image. Without the cross- attention, it is infeasible to realize the standalone object detector. As shown in Figure 3, ViDT binds [DET] x [DET] and [DET] X [PATCH] attention to process them at once to increase efficiency.
We replace all the attention modules in Swin Transformer with the proposed RAM, which receives [PATCH] and [DET) tokens (as shown in "Body" of Figure 2(c)) and then outputs their calibrated new tokens by performing the three different attention operations in parallel.
Positional Encoding. ViDT adopts different positional encodings for diffferent types of attention. For [PATCH| x РРТCC] attention, we use the relative position bias (Hu et al., 2019) originally used in Swin Transformer. In contrast, the learnable positional encoding added for [DET] tokens for [DET] x [DET] attention because there is no particular order between [DET] tokens. However, for [DET] x [PATCH attention, it is crucial to inject spatial bias to the [PATCH] tokens due to the cermutaaiionequivaarti in transformers, ignoring spatial information of the feature map. Thus, ViDT adds the sinusoidal- based spatial positional encoding to the feature map, which reconstructed from the [ААCC] tokens for [DET] x [РAТCH] attention, can be seen from the left side of Figure 3. We present thorough analysis of various spatial positional encodings in Section 4.2.
Use of [DET] X [PATCH] Attention. Applying cross-attention between [DET] and [PATCH] tokens adds additional computational overhead to Swin Transformer, especially when it is activated at the bottom layer due to the large number of [РАTCH] tokens. To minimize such computational overhead, ViDT only activates the aross-attention at the last stage (the top level of the pyramid) of Swin Trans- former, which consists of two transformer layers that receives [PATCH] tokens of size H H/322 W/32.
4

---

Published as a conference paper at ICLR 2022
Thus, only self- for [DET] and [PATCH] tokens are performed for the remaining stages except the last one. In Section 4.2.2 we show that this design choice helps achieve the highest FPS, while achieving similar detection performance when cross-attentie attention is enabled at every stage. We pro- vide more details on RAM including its complexity analysis and algorithmic design in Appendix A.
3.2 ENCODER-FREE FREE NECK STRUCTURE
To exploit multi-scale feature maps, ViDT incorporates a decoder of multi-layer deformable trans- formers (Zhu et al., 2021). In the DETR family (Figure 2(a)), a transformer encoder is required at the neck to transform features extracted from the backbone for image classification into the ones suitable for object detection; the encoder is generally computationally expensive since it involves [PATCH] x [PATCH] attention. However, ViDT maintains only a transformer decoder as its neck, in that Swin Transformer with RAM directly extracts fine-grained features suitable for object detection as a standalone object detector. Thus, the neck structure of ViDT is computationally efficient.
The decoder receives two inputs from Swin Transformer with RAM: (1) [PATCH] tokens generated from each stage (i.e., four multi-scale feature maps, {> ]/= where
4) and (2) [DET] tokens gen- erated from the last stage. The overview is illustrated in "Neck" of Figure 2(c). In each deformable transformer layer, [DET] [DET] attention is performed first. For each [DET] token, multi-scale de- formable attention is applied to produce a new [DET] token, aggregating small set of key contents sampled from the multi-scale feature maps (ar )(=1"
M
MSDeeformAttt (a' (a')
Σ
W.
Amax Wi. (d((p) + Apma)).
(I)
k=1
where indices the attention head and K is the total number of sampled keys for content aggrega- tion. In addition, (p) is the reference point of the [DET] token re-scaled for the I-th level feature map, while Apmik is the sampling offset for deformable attention; and Amik is the attention weights of the K sampled contents. Wm and W m are the projection matrices for multi-head attention.
Auxiliary Techniques for Additional Improvements. The decoder of VIDT follows the standard structure of multi- layer transformers, generating refined [DET] tokens each layer. Hence, ViDT leverages the two auxiliary techniques used in (Deformable) DETR for additional improvements:
Auxiliary Decoding Loss: Detection heads consisting of two feedforward networks (FNNs) for box regression and classification are attached to every decoding layer. All the training losses from detection heads at different scales added to train the model. This helps the model output the correct number of objects without non-maximum suppression (Carion et al., 2020).
Iterative Box Refinement: Each decoding layer refines the bounding boxes based predictions from the detection head in the previous layer. Therefore, the box regression process progressively improves through the decoding layers (Zhu et al., 2021).
object detectors because they significantly enhance detection performance without compromising detection efficiency. We provide an ablation study of their effectiveness for object detection in Section 4.3.1.
These two techniques are essential for transformer-
3.3 KNOWLEDGE DISTILLATION WITH TOKEN MATCHING FOR OBJECT DETECTION
While a large model has a high capacity to achieve high performance, it can be computationally ex- pensive for practical use. As such, we additionally present simple knowledge distillation approach that can transfer knowledge from the large ViDT model by token matching. Based on the fact that all ViDT models has exactly the same number of [PATCH] and [DET] tokens regardless of their scale, a small ViDT model (a student model) can easily benefit from pre-trained large ViDT (a teacher model) by matching its tokens with those of the large one, thereby bringing out higher detection performance at lower computational cost.
Matching all the tokens at every layer very inefficient in training. Thus, we only match the to- kens contributing the most to prediction. The two sets of tokens are directly related: (I) P: the set of [РATCH] tokens used as multi-scale feature maps, which are generated from each stage in the body, and (2) D: the set of [DET] tokens, which generated from each decoding layer in the neck. Accordingly, the distillation loss based on token matching is formulated by
IPs
Pel
Laia Pr, D.) Aac
Σ
P.
P.
D. 1.
(2)
P
D.
5

---

Published as a conference paper at ICLR 2022
Backbone
Type (Size)
Train Data
Epochs
Resolution
Params
ImageNet Acc.
DeiT-tiny (*)
ImageNet- K
300
224
6M
74.5
DeiT
Deil-small (*)
ImageNet- IK
300
224
22M
81.2
DesT-base (*)
ImageNet- IK
300
384
87M
85.2
Swin-nano Swin-tiny
ImageNet- IK
300
224
6M
74,9
Swin Transformer
ImageNet- 1K
300
224
28M
81.2
Swin-small
ImageNet-IK
300
224
50M
83.2
Swin-base
ImageNet-22K
90
224
88M
86.3
Table 1. Summary the ViT backbone. "*" is the distillation strategy for classification (Touvron et al., 2021).
where the subscripts and refer to the student and teacher model. Pli] and D[i] return the i-th [PATCH] and [DET] tokens, n-dimensional vectors, belonging to P and D, respectively. Adie is the coefficient to determine the strength of Edis, which is added to the detection loss if activated.
EVALUATION
In this section, we show that ViDT achieves the best trade-off between accuracy and speed (Section 4.1). Then, we conduct detailed ablation study of the reconfigured attention module (Section 4.2) and additional techniques to boost detection performance (Section 4.3). Finally, we provide a complete analysis of all components available for ViDT (Section 4.4).
Dataset. We carry out object detection experiments on the Microsoft COCO 2017 benchmark dataset (Lin al., 2014). All the fully transformer-base based object detectors are trained on 118K train- ing images and tested on 5K validation images following the literature (Carion et al., 2020).
Algorithms. We compare ViDT with two existing fully transformer-basee object detection pipelines, namely DETR (ViT) and YOLOS. Since DETR (ViT) follows the general pipeline of (Deformable) DETR by replacing its ResNet backbone with other ViT variants; hence, we use one canonical ViT and one latest VIT variant, DeiT' and Swin Transformer, its backbone without any modification. In contrast, YOLOS is the canonical ViT architecture, thus only DeiT is available. Table summarizes all the VIT models pre-trained on ImageNet used for evaluation. Note that publicly available pre- trained models are used except for Swin-nano. We newly configure Swin-nano" comparable to DeiT- tiny, which trained on ImageNet with the identical setting. Overall, with respect to the number of parameters, Deit-tiny, -small, and -base are comparable to Swin-nano, -tiny, and -base, respectively. Please see Appendix B.2 for the detailed pipeline of compared detectors.
Implementation Details. All the algorithms are implemented using PyTorch and executed using eight NVIDIA Tesla V100 GPUs. We train ViDT using AdamW (Loshchilov & Hutter, 2019) with the same initial learning rate of 10-4 for its body, neck and head. contrast, following the (De- formable) DETR setting, DETR (VIT) is trained with the initial learning rate of 10-5 for its pre- trained body (ViT backbone) and 10 for its neck and head. YOLOS and ViDT (w.o. Neck) are trained with the same initial learning rate of 10-5 which is the original setting of YOLOS for the neck-free detector. We do not change any hyperparameters used in transformer encoder and decoder for (Deformable) DETR; thus, the neck decoder of ViDT also consists of six deformable transformer layers using exactly the same hyperparameters. The only new hyperparameter introduced, the dis- tillation coefficient Aat in Eq. (2), is set to 4. For fair comparison, knowledge distillation is not applied for ViDT in the main experiment in Section 4.1. The efficacy of knowledge distillation with token matching is verified independently in Section 4.3.2. Auxiliary decoding loss and iterative box refinement are applied to the compared methods if applicable.
Regarding the resolution of input images, use scale augmentation that resizes them such that the shortest side least 480 and at most 800 pixels while the longest at most 1333 (Wu et al., 2019). More details of the experiment configuration can be found in Appendix B.3-B.5. All the source code and trained models will be made available to the public at
ai/vidt.
4.1 MAIN EXPERIMENTS WITH MICROSOFT COCO BENCHMARK
Table 2 compares ViDT with DETR (ViT) and YOLOS w.n. their AP, FPS, # parameters, where the two variants of DETR (VIT) simply named DETR and Deformable DETR. We report the result of ViDT without using knowledge distillation for fair comparison. A summary plot is provided in Figure 1. The experimental comparisons with CNN backbones are provided in Appendix C.
"wnnn-nano is designed such that its channel dimension half that of Swin-tiny, Please Appendix B.

---

Published as a conference paper at ICLR 2022
Method
Backbone
Epochs
AP50
APTs
APs
APM
APL
Param.
FPS
DeiT--iny
50
30.0
49.2
30.5
9.9
30.8
50.6
24M
10.9 (13.1)
DeiT-small
50
32.4
52.5
33.2
11.3
33.5
53.7
39M
7.8 (8.8)
DeiT-base
50
37.1
59.2
38.4
14.7
39.4
52.9
0.1B
4.3 (4.9)
DETR
Swin-nano
27.8
47.5
27.4
9.0
29.2
44.9
24M
24.7 (46.1)
Swin-tiny
50
34.1
55.1
35.3
12.7
35.9
54.2
45M
19.3 (28.1)
Swin-smal
50
37.6
59.0
39.0
15.9
40.1
58.9
66M
13.5 (17.7)
Swin-base
50
40.7
62.9
42.7
18.3
44.1
62.4
O.1B
9.7 (12.6)
DeiT-tiny
50
40.8
60.1
43.6
21.4
43,4
58.2
18M
12.4 (16.3)
DeiT-small
50
43.6
63.7
46.5
23.3
47.1
62.1
35M
8.5 (10.2)
Deformable DETR
DesT-base
46.4
67.3
49.4
26.7
50.1
65.4
0.IB
4.4 (5.3)
Swin-nano
50
43.1
61.4
46.3
25.9
45.2
59.4
18M
7.0 (7.8)
Swin-tiny
50
47.0
66.8
50.8
28.1
49.8
63.9
39M
6.3 (7.0)
Swin-small
50
49.0
68.9
52.9
30.3
52.8
66.6
60M
5.5 (6.1)
Swin-base
50
51.4
71.7
56.2
34.5
55.1
67.5
O.IB
4.8 (5.4)
DeiT-tiny
150
30.4
48.6
31.1
12.4
31.8
48.2
6M
28.1 (31.3)
YOLOS
DeiT-small
150
36.1
55.7
37.6
15.6
38.4
55.3
30M
(11.8)
DesT-base
42.0
62.2
44.5
19.5
45.3
62.1
0.IB
3.9 (5.4)
Swin-nano
150
28.7
48.6
28.5
12.3
30.7
44.1
7M
36.5 (64.4)
VIDT (w.o. Neck)
Swin-tiny
150
36,3
56.3
37.8
16.4
39.0
54.3
29M
28.6 (32.1)
Swin-small
150
41.6
62.7
43.9
20.1
45.4
59.8
52M
16.8 (18.8)
Swin-hase
150
43.2
64.2
45.9
21.9
46.9
63.2
91M
11.5 (12.5)
Swin-nano
50
40.4
59.6
43.3
23.2
42.5
55.8
16M
20.0 (45.8)
Swin-tiny
50
44.8
64.5
48.7
25.9
47.6
62.1
38M
17.2 (26.5)
VIDT
Swin-small
50
47.5
67.7
51,4
29.2
50.7
64.8
61M
12.1 (16.5)
Swin-base
49.2
69.4
53.1
30.6
52.6
66.9
O.IB
9.0 (11.6)
Table 2. Comparison of ViDT with other compared detectors COCO2017 val set. Two neck-free detectors, YOLOS and ViDT (w.o. Neck) are trained for 150 epochs due to slow convergence. FPS measured with batch size of 800 x 1333 resolution on a single Tesla V100 GPU, where the value inside the parentheses is measured with batch size 4 of the same resolution to maximize GPU utilization.
Highlights. ViDT achieves the best trade-off between AP and FPS. With its high scalability, it per- forms well even for Swin-base of 0.1 billion parameters, which is 2x faster than Deformable DETR with similar AP. Besides, ViDT shows 40. 40.4AP only with 16M parameters; it is 6.3-12.6AP higher than those of DETR (swin-nano) and DETR (swin-tiny), which exhibit similar FPS of 19.3-24.7.
VIDT vs. Deformable DETR. Thanks to the use of multi-scale features, Deformable DETR exhibits high detection performance in general. Nevertheless, encoder and decoder structure in the neck becomes critical bottleneck in computation. In particular, the encoder with multi-layer deformable transformers adds considerable overhead to transform multi-scale features by attention. Thus, shows very low FPS although it achieves higher AP with relatively small number of parameters. In contrast, ViDT removes the need for a transformer encoder in the neck by using Swin Transformer with RAM its body, directly extracting multi-scale features suitable for object detection.
VIDT (w.o. Neck) vs. YOLOS. For the comparison with YOLOS, we train ViDT without using its neck component. These two neck-free detectors show relatively low AP compared with other detectors in general. In terms of speed, YOLOS exhibits much lower FPS than ViDT (w.o. Neck) because of its quadratic computational complexity for attention. However, ViDT (w.o. Neck) extends Swin Transformers with RAM. thus requiring linear complexity for attention. Hence, shows AP comparable to YOLOS for various backbone size, but its FPS much higher.
One might argue that better integration could be also achieved by (1) Deformable DETR without its neck encoder because its neck decoder also has [ETT] x [PATCH cross- coossatttetiion or (2) YOLOS with VIDT's neck decoder because the use of multiple auxiliary techniques. Such integration is actually not effective; the former significantly drops AP, while the latter has much greater drop in FPS than an increase in AP. The detailed analysis can be found Appendix C.2.
4.2 ABLATION STUDY ON RECONFIGURED ATTENTION MODULE (RAM)
We extend Swin Transformer with RAM to extract fine- tne-grained features for object detection without maintaining an additional transformer encoder in the neck. We provide ablation study the two main considerations for RAM, which leads to high accuracy and speed. To reduce the influence of secondary factors, mainly neck-free version, ViDT (w.o. Neck), for the ablation study.
7

---

Published as a conference paper at ICLR 2022
4.2.1 SPATIAL POSITIONAL ENCODING
Spatial positional encoding essential for [DET] x [PATCH] attention in RAM. Typically. the spatial encoding can be added to the [PATCH] tokens be- fore or after the projection layer in Figure 3. We call the former "pre-addition" and the latter "post- addition". For each one, can design the encoding in a sinusoidal or learnable manner (Carion et al., 2020). Table 3 contrasts the results with different spatial positional encodings with ViDT (w.o. Neck). Overall, pre-addition results in performance improvement higher than post-addition, and specifically, the sinusoidal encoding is better than the learnable one; thus, the 2D inductive bias of the sinusoidal spatial encoding is more helpful in object detection. In particular, pre-addition with the sinusoidal encoding increases AP by 5.0 compared to not using any encoding.
Method
None
Pre-addition
Post-addition
Type
None
Sin.
Learn.
Sin.
Leamn
AP
23.7
28.7
27.4
28.0
24.1
Table
Results for differrnt spatial encodings for [DET) x [PATCH] cross-attention.
4.2.2 SELECTIVE [ET]] x [PATCH] CROSS-ATTENTION
The addition of cross-attention to Swin Transformer inevitably entails computational overhead, par- ticularly when the number of [PATCH] is large. To alleviate such overhead, we selectively enable cross-attention in RAM at the last stage of Swin Transformer; this is shown to greatly improve FPS, but barely drop AP. Table 4 summarizes AP and FPS when used different selective strategies for the cross-attention, where Swin Transformer consists of four stages in total. is interesting that all the strategies exhibit similar AP as long as cross-attention is activated at the last stage. Since features are extracted in bottom-up manner as they go through the stages, seems difficult directly obtain useful information about the target object at the low level of stages. Thus, only using the last stage is the best design choice in terms of high AP and FPS due to the smallest number of [РATCH] tokens.
Meanwhile, the detection fails completely or the performance significantly drops if all the stages are involved due to the lack of interaction between [ETT] and [PATCH] tokens that spatial positional encoding is associated with. A more detailed analysis of the [DET] x [PATCH] cross-attention and [DET] x [DET] self-attention is provided in appendices C.3 and C.4.
Stage Ids
(1, 3, 4)
[2, 3, 4]
(3, 4)
[4]
Metric
AP
FPS
AP
FPS
AP
FPS
AP
FPS
AP
FPS
w.o. Neck
29.0
21.8
28.8
29.1
28.5
34.3
28.7
36.5
FAIL
37.7
w.
Neck
40.3
14.6
40.1
18.0
40.3
19.5
40.4
20.0
37.1
20.5
Table 4. AP and FPS comparison with different selective cross-attention strategies.
4.3 ABLATION STUDY ON ADDITIONAL TECHNIQQES
We analyze the performance improvement of two additional techniques, namely auxiliary decoding loss and iterative box refinemett, and the proposed distillation approach in Section 3.3. Further- more, we introduce simple technique that can expedite the inference speed of ViDT by dropping unnecessary decoding layers at inference time.
4.3.1 AUXILIARY DECODING LOss AND ITERATIVE Box REFINEMENT
To thoroughly verify the efficacy of auxiliary decod- ing loss and iterative box refinement, extend them even for the neck-free detector like YOLOS; the
Aux. E Box Ref. Neck
AP
A
VIDT YOLOS
30.4
29,2
- 1.2
principle of them is applied to the encoding layers in the body, as opposed to the conventional way of using the decoding layers in the neck. Table shows the
20.1
10.3
28.7
performance of the two neck-free detectors, YOLOS and ViDT (w.o. Neck), decreases considerably with
27.2
- -16
22.9
559
36.2
+7.4
the two techniques. The use of them in the encoding layers is likely to negatively affect feature extraction of the transformer encoder. In contrast, an opposite trend observed with the neck component. Since the neck decoder is decoupled with the feature extraction in the body, the two techniques make synergistic effect and thus show significant improvement in AP. These results justify the use of the neck decoder in ViDT to boost object detection performance.
40.4
+11.6
Table 5. Effect of extra techniques with YOLOS (DeiT-tiny) and ViDT (Swin-nano).
8

---

Published as a conference paper at ICLR 2022
4.3.2 KNOWLEDGE DISTILLATION WITH TOKEN MATCHING
We show that a small ViDT model can benefit from large ViDT model via knowledge distil- lation. The proposed token matching is new concept of knowledge distillation for object de- tection, especially for fully transformer-basee object detector. Compared to very complex dis- tillation methods that rely heuristic rules with multiple hyperparameters (Chen et al., 2017; Dai al., 2021) 1). simply matches some tokens with a single hyperparameter, the distillation coefficient Adis- Table summarizes the AP improvement via knowledge distillation with token matching with varying distillation coefficients. Overall, the larger the size of the teacher model, the greater gain to the student model. Regarding coefficients, in general, larger values achieve better performance. Distillation increases AP by 1.0-1.7 without affecting the inference speed of the student model.
Student
(Swin-nnn)an))
VIDI (Swin-tiny)
ViDT (small)
ViDT (base)
ViDT (small)
ViDT (base)
Teacher
Adis 0
40.4
44.8
Adis 2
41.4
41.4
45.6
46.1
Adaa
41.5
41.9
45.8
46.5
Table 6. AP comparison of student models associated with different teacher models.
4.3.3 DECODING LAYER DROP
ViDT has six layers of transformers as its neck decoder. We emphasize that not all layers of the decoder are required at inference time for high performance. Table 7 show the performance of ViDT when dropping its decoding layer one by one from the top in the inference step. Although there is a trade-off relationship between accuracy and speed the layers are detached from the model. there is no significant AP drop even when the two layers are removed. This technique is not designed for performance evaluation in Table 2 with other methods, but we can accelerate the inference speed of trained ViDT model to over 10% by dropping its two decoding layers without a much decrease in AP.
Model Metric
ViDT (Swin-nano)
ViDT (Swin-tiny)
AP
Param.
FPS
AP
Param.
FPS
0 Drop
40.4
16M
20.0
44.8
38M
17.2
Drop
40.2
14M
20.9
44.8
37M
18.5
2 Drop 40.0
13M
22.3
44.5
35M
19.6
3 Drop 38.6
12M
24.7
43.6
34M
21.0
4 Drop 36.8
11M
26.0
41.9
33M
22.4
5 Drop
32.5
10M
28.7
38.0
32M
24.4
Table 7. Performance trade-off by decoding layer drop regarding AP, Param, and FPS.
4.4 COMPLETE COMPONENT ANALYSIS
In this section, we combine all the proposed components (even with distillation and decoding layer drop) to achieve high accuracy and speed for object detection. As summarized in Table 8, there are four components: (1) RAM to extend Swin Transformer as a standalone object detector, (2) the neck decoder to exploit multi-scale features with two auxiliary techniques, (3) knowledge distillation to benefit from a large model, and (4) decoding layer drop to further accelerate inference speed. The performance of the final version is very outstanding; it achieves 41.7AP with reasonable FPS by only using 13M parameters when used Swin-nano as its backbone. Further, it only loses 2.7 FPS while exhibiting 46.4AP when used Swin-tiny This indicates that fully transformer-basee object detector has the potential to be used as generic object detector when further developed in the future.
Component
S wiin-nano
Swin-liny AP75
#
RAM Neck Distil Drop
AP
APso
AP7s
Param.
FPS
AP
AP50
Param.
FPS
28.7
48.6
28.5
7M
36.5
36.3
56.3
37.8
29M
28.6
(2)
40.4
59.6
43.3
16M
20.0
44.8
64.5
48.7
38M
17.2
(3)
V
41.9
61.1
45.0
16M
20.0
46.5
66.3
50.2
38M
17.2
(4)
41.7
61.0
44.8
13M
22.3
46.4
66.3
50.2
35M
19.6
Table Detailed component analysis with Swin-nano and Swin-tiny.
CONCLUSION
We have explored the integration of vision and detection transformers to build an effective and efficient object detector. The proposed ViDT significantly improves the scalability and flexibility of transformer models to achieve high accuracy and inference speed. The computational complexity of its attention modules is linear w.r.t. image size, and ViDT synergizes several essential techniques to boost the detection performance. On the Microsoft COCO benchmark, ViDT achieves 49.2AP with a large Swin-base backbone, and AAP with the smallest Swin-nano backbone and only 13M parameters, suggesting the benefits of using transformers for complex computer vision tasks.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
This paper deals with the topic of general object detection in computer vision. We propose a novel integration of vision and detection transformers for a fully transformer-based object detector. There- fore, we do not expect any potential negative social impact of our work.
REPRODUCIBILITY STATEMENT
For reproducibility, we provide a detailed description of our experiment and hyperparameter settings in Appendix B. It includes the Swin-nano architecture (Appendix B.1), the pipelines of all compared object detectors (Appendix B.2), hyperparameters of neck transformers (Appendix B.3), detailed im- plementation (Appendix B.4), and training configuration (Appendix B.5). We will release the code and trained models upon acceptance.
ACKNOWLEDGMENTS
We thank NAVER Al Lab members for valuable discussion and advice. NAVER Smart Machine Learning (NSML) (Kim et al., 2018) has been used for experiment. M.-H. Yang is supported in part by the NSF CAREER grant 149783.
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pp. 213-229, 2020. 1, 3, 5, 6, 8, 14, 15
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. CrossViT: Cross-attention multi-scale vision transformer for image classification. In ICCV, 2021. 13
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In NeurIPS, 2017.
Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Liu, and Erjin Zhou. General instance distillation for object detection. In CVPR, pp. 7842-7851, 2021. 9
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021. 1,2
Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. arXiv preprint arXiv:210600066 2021. 2, 3, 14, 15
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In ICCV, 2021. 2, 13
Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In CVPR, pp. 3464-3473, 2019. 4
Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. NSML: Meet the mlaas platform with a real-world case study. arXiv preprint 170.991810.09957. 2018. 10
Tsung-Y Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects context. In ECCV, pp- 740-755, 2014. 2.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In CVPR, pp. 2980-2988, 2017. 14
10

---

Published as a conference paper at ICLR 2022
Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep learning for generic object detection: A survey. International Journal of Computer Vision, 128(2)261-311, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 2, 12
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 6, 15
Constantine Papageorgiou and Tomaso Poggio. A trainable system for object detection. Interna- tional Journal of Computer Vision, 38(1):15-33, 2000. 1
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. NeurlPS, 28:91-99, 2015. 3
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR, pp. 638-666, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, pp. 10347-10357, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 5998-6008, 2017. 1, 12, 17
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021. 13
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019. 3,6
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In ICLR, 2021, 3, 14, 15

---

Published as a conference paper at ICLR 2022
An Efficient and Effective Fully Transformer-based Ob- ject Detector (Supplementary Material)
A RECONFIGURED ATTENTION MODULE
The proposed RAM in Figure 3 performs three attention operations, namely [PATCH] x [PATCH]. [DET] [PATCH], and [DET] x [DET] attention. This section provides (1) computational complexity analysis and (2) further algorithmic design for [DET] tokens.
A.1 COMPUTATIONAL COMPLEXITY ANALYSIS
We analyze the computational complexity of the proposed RAM compared with the attention used in YOLOS. The analysis is based on the computational complexity of basic building blocks for
Canonical and Swin Transformer, which is summarized in Table 9 where Ti and T2 is the number of tokens for self- and cross-attention, and d is the embedding dimension.
Transformer
Canonical Transformer
Swin Transformer
Attention
Global Self-attention Global Cross-attention
Local Self-attention
Complexity
OddTT++ +TT)
(T T2) + dT)
Old + di²TT)
Table 9. Computational complexity of attention modules: In the canonical transformer, the complexity of global self-attention Old T2 + d) where (d) the cost computing the query, key, and value embed- dings and ( is the cost of computing the attention weights. The complexity of global cross cross-attention is Odd'(T, (T + T2) + T2), which is the interaction between the two different tokens T₁ and Tz. In contrast, Swin Transformer achieves much lower attention complexity of Odd 1 dk d with window partitioning, where le the width and height of the window (k- << T2).
Let P and D be the number of [PATC]] and [DET] tokens (D << P in practice, c.g., P 66, 650 and D 100 at the first stage of ViDT). Then, the computational complexity of the attention module for YOLOS and ViDT (RAM) is derived as below, also summarized in Table 10:
YOLOS Attention: DET tokens are simply appended [PATCH| tokens to perform global self- attention on [PATCH, DET tokens (i.e., T, = P + D). Thus, the computational complexity is (P + D) + d(P + D) which quadratic to the number of [PATCH] tokens. If breaking down the total complexity, we obtain ((dPp dP2) + (dFD dD²) + dPD). where the first and second terms are for the global self-attention for [PATCH] and [EET] tokens, respectively, and the last term is for the global cross-attention between them.
ViDT (RAM) Attention: RAM performs the three different attention operations: (1) [PATCH] x [PATCH] local self-attention with window partition, O + dd:²p); (2) DETT) x (DET) global self- attention, Odd + dD²); (3) [ET) x [PATCH] global cross-attention, ( (D + P) + dDP). In total, the computational complexity of RAM (d2 (D + P) + dl.2p + dD² + dDP), which linear the number of [PATCH] tokens.
Consequently, the complexity of RAM is much lower than the attention module used in YOLOS since D << P. Note that only RAM achieves the linear complexity to the patch tokens. In addition, one might argue that YOLOS can be efficient if the cross-attention is selectively removed similar to
RAM. Even if remove the complexity (ddDD for the global cross-attention, the computational complexity is (P + D) + dP2 + dD²), which is still quadratic to the number of [PATCH] tokens.
Attention Type
YOLOS
VIDT
[PACC]] [PATCH|
OdP + dP2 Odd + dD?) (ddDD
Od + dk-p)
DET x DET
O(dDD+
DET x [ААСH]]
Odd + dDP)
Total Complexity
d(P+
(dd d =D= dDP)
Table Summary computational complexity for different attention operations used YOLOS and ViDT (RAM), where P and D are the number of [РАТCн| and [DET] tokens, respectively (D << P).
We used the computational complexity reported in the original paper (Vaswani al., 2017; Liu et el., 2021)
12

---

Published as a conference paper at ICLR 2022
A.2 ALGORITHMIC DESIGN FOR [DET] TOKENS
A.2.1 BINDING [ЕT x DET AND [DET] x [PATCH] ATTENTION
Binding the two attention modules is very simple in implementation. [DET] x [DET] and [DET] x [PATCH| attention is generating a new [DET] token, which aggregates relevant contents in [DET] and [PATC] tokens, respectively. Since the two attention share exactly the same [DET] query embed- ding obtained after the projection as shown in Figure 3, they can be processed at once by perform- ing the scaled-dot product between [DET]q and [DET] K; [PATC]] embeddings, where Q, K are the key and query, and is the concatenation. Then, the obtained attention map is applied to the [DET v v]] embeddings, where V is the value and d is the embedding dimension,
D [ETT] K. [PATCH] k]
[DET] New = Softmax
[[EET], [
(3)
vd
This approach commonly used in the recent Transformer- based architectures, such as YOLOS.
A.2.2 EMBEDDING DIMENSION OF [DET] TOKENS
[DT]] [ET] attention is performed across all the stages, and the embedding dimension of [DET] tokens increases gradually like [PATCH] tokens. For the [PATCH] token, its embedding dimension increased by concatenating nearby [ATCH]] tokens in grid. However, this mechanism is not applicable for [DET] tokens since we maintain the same number of [DET) tokens for detecting a fixed number of objects in a scene. Hence, we simply repeat [DET] token multiple times along the embedding dimension to increase its size. This allows [DT tokens to reuse all the projection and normalization layers in Swin Transformer without any modification.
B EXPERIMENTAL DETAILS
B.1 SWIN-NANO ARCHITECTURE
Due to the absence of Swin models comparable to Deit-tiny, we configure Swin-nano, which a 0.25x model of Swin-tiny such that it has 6M training pa- rameters comparable to Deit-tiny. Table summa- rizes the configuration of Swin Transformer mod- els available, including the newly introduced Swin- nano; $1-S4 indicates the four stages in Swin Trans- former. The performance of all the pre-trained Swin Transformer models summarized in Table in the manuscript.
Model
Channel
Layer Numbers
Name
Dim.
S1
S2
S3
S4
Swin-nano
48
2
6
2
Swin-tiny
96
2
6
2
Swin-small
128
2
18
Swin-base
192
2
18
2
Table 11. Swin Transformer Architecture.
B.2 DETECTION PIPELINES OF ALL COMPARED DETECTORS
All the compared fully transformer-based detectors are composed of either (1) bodyy-neck-heal or (2) body-head structure, as summarized in Table 12. The main difference of ViDT is the use of reconfigured attention modules (RAM) for Swin Transformer, allowing the extraction of fine- ane-grained detection features directly from the input image. Thus, Swin Transformer is extended to a standalone object detector called ViDT (w.o. Neck). Further, its extension to ViDT allows features and multiple essential techniques for better detection, such as auxiliary decoding loss and iterative box refinement, by only maintaining transformer decoder at the neck. Except for the two neck-free detector, YOLOS and ViDT (w.o. Neck), all the pipelines maintain multiple FFNs; that is, single FFNs for each decoding layer the neck for box regression and classification.
multi-scale
We believe that our proposed RAM can be combined with even other latest efficient vision trans- former architectures, such PiT (Heo al., 2021), PVT (Wang al., 2021) and Cross- ViT (Chen et al., 1021). We leave this as future work.
B.3 HYPERPARAMETERS OF NECK TRANSFORMERS
The transformer decoder at the neck in ViDT introduces multiple hyperparameters. We follow ex- actly the same setting used in Deformable DETR. Specifically, six layers of deformable
13

---

Published as a conference paper at ICLR 2022
Pipeline
Body
Neck
Hcad
Method Name
Feature Extractor
Tran. Encoder
Tran. Decoder
Prediction
DETR (DeiT)
DeiT Transformer
8
Multiple FFNs
DETR (Swin)
Swin Transformer
Multiple FFNs
Deformable DETR (DeiT)
DeiT Transformer
O
Of
Multiple FFNs
Deformable DETR (Swin)
Swin Transformer
O
O!
Multiple FFNs
YOLOS
DeiT Transformer
Single FFNs
VIDT (w.o. Neck)
Swin Transformer
X
X
Single FFNs
VIDT
Swin Transformer RAMM
X
O
Multiple FFNs
Table 12. Comparison of detection pipelines for all available fully transformer based object detectors, where indicates that multi-scale deformable attention is used for neck transformers.
transformers with width 256; thus, the channel dimension of the [PATCH] and [DET] tokens extracted from Swin Transformer reduced to 256 to be utilized as compact inputs the decoder trans- former. For each transformer layer, multi-head attention with eight heads is applied, followed by the point-wise FFNs of 1024 hidden units. Furthermore, the layer normalization. All the weights in the decoder are initialized with Xavier initialization. For (Deformable) DETR, the tranformer decoder receives fixed number of learnable detection tokens. We set the number of detection tokens to 100, which the same number used for YOLOS and ViDT.
additive dropout of 0.1 is applied before
B.4 IMPLEMENTATION
B.4.1 DETECTION HEAD FOR PREDICTION
The last [DET] tokens produced by the body or neck are fed to a 3-layer FFNs for bounding box regression and linear projection for classification,
B FFNa-tayer ((DETT)) and P LiEear((DET)..
For box regression, the FFNs produce the bounding box coordinates for objects, B E [0, 1]dx4, that encodes the normalized box center coordinates along with its width and height. For classification, the linear projection uses a softmax function to produce the classification probabilities for all possible classes including the background class, P E [o, 1]dx(c+1), where is the number of object classes. When deformable attention is used the neck in Table 12, only classes are considered without the background class for classification. This the original setting used in DETR, YOLOS (Carion et al., 2020; Fang et al., 2021) and Deformable DETR (Zhu al., 2021).
B.4.2 LOSS FUNCTION FOR TRAINING
All the methods adopts the loss function of (Deformable) DETR. Since the detection head return fixed-size set of bounding boxes, where d is usually larger than the number of actual objects in an image, Hungarian matching is used to find bipartite matching between the predicted box B and the ground-truth box B. In total, there are three types of training loss: classification loss la², a box distance f,, and GloU loss Liou (Rezatofighi al., 2019),
(t) llg P Po(S.ot
le, (1) = ||B B Bo(G)11, and [B,B, B.))))))BBBBBBB Bacol |B(B,, Bo(i))
Lio (1) = 1- B, Ba(s)l
(5)
B, BUBB)
where and (1) are the target class label and bipartite assignment of the i-th ground-truth box, and B returns the largest box containing two given boxes. Thus, the final loss of object detection linear combination of the three types of training loss,
= Aala + Xe, li + >iounlion
(6)
"Cross-nnroppy loss is used with standard transformer architectures, while focal loss (Lin et al., 2017) is used with deformable transformer architecture.
14

---

Published as a conference paper at ICLR 2022
Method
Backbone
Epochs
AP
AP50
APrs
APs
APM
APL
Param.
FPS
DETR
ResNet-50
500
42.0
62,4
44.2
20.5
45.8
61.1
41M
22.8 (38.6)
DETR-DC5
ResNet-50
500
43.3
63.1
45.9
22.5
47.3
61.1
41M
12.8 (14.2)
DETR-DC5
ResNet-50
50
35.3
55.7
36,8
15.2
37.5
53.6
41M
12.8 (14.2)
Deform. DETR
ResNet-50
50
45.4
64.7
49.0
26.8
48.3
61.7
40M
13.7 (19.4)
VIDT
Swin-tiny 50
44.8
64.5
48.7
25.9
47.6
62.1
38M
17.2 (26.5)
ViDT
Swin-tiny 150
47.2
66.7
51.4
28.4
50.2
64.7
38M
17.2 (26.5)
Table 13. Evaluations of ViDT with other detectors using CNN backbones on COC02017 val set. FPS is
measured with batch size 1 of 800 x 1333 resolution on single Tesla V100 GPU, where the value inside the parentheses is measured with batch size 4 of the same resolution to maximize GPU utilization.
Method
Backbone
AP
A
AP7s
APs
APM
APL
Param.
FPS
Deformable DETR neck encoder
43.1
61.4
46.3
25.9
45.2
59.4
17M
7.0
Swin-nano
34.0
52.8
35.6
18.0
36.3
48.4
14M
22.4
YOLOS
30.4
48.6
31.1
12.4
31.8
48.2
6M
28.1
DeiT-tiny
+ neck decoder
38.1
57.1
40.2
20.1
40.2
56.0
14M
17.1
ViDT
40.4
59.6
43.3
23.2
42.5
55.8
16M
20.0
Swin-nano
+ neck encoder
46.1
64.1
49.7
28.5
48.7
61.7
19M
6.3
Table 14. Variations of Deformable DETR, YOLOS, and ViDT with respect to their neck structure. They are trained for 50 epochs with the same configuration used in main experimental results.
The coefficient for each training loss set to be Art 1, Ar = 5, and Xion = If we leverage auxiliary decoding loss, the final loss is computed for every detection head separately and merged with equal importance. Additionally, ViDT adds the distillation loss in Eq. (2) to the final loss if the distillation approach in Section 3.3 is enabled for training.
B.5 TRAINING CONFIGURATION
We train VIDT for 50 epochs using AdamW (Loshchilov & Hutter, 2019) with the same initial learming rate of 10 (4-4 for its body, neck and head. The learning rate is decayed by cosine annealing with batch size of 16, weight decay of 1 10 and gradient clipping of 0.1. In contrast, ViDT (w.o. Neck) is trained for 150 epochs using AdamW W with the initial learning rate of 5 x 10 10-5 by cosine annealing. The remaining configuration is the same as for ViDT.
Regarding DETR (ViT), we follow the setting of Deformable DETR, Thus, all the variants of this pipeline are trained for 50 epochs with the initial learning rate 10 for its pre-trained body (ViT backbone) and 10-4 for its neck and head. Their learning rates are decayed at the 40-th epoch by factor of 0. I. Meanwhile, the results of YOLOS are borrowed from the original paper (Fang et al., 2021) except YOLOS (DeiT-tiny); since the result of YOLOS (DeiT-tiny) for 800 x 1333 is not reported in the paper, we train it by following the training configuration suggested by authors.
C SUPPLEMENTARY EVALUATION
C.1 COMPARISON WITH OBJECT DETECTOR USING CNN BACKBONE
We compare VIDT with (Deformable) DETR using the ResNet-50 backbone, summarized in Table 13, where all the results except ViDT are borrowed from (Carion et al., 2020; Zhu et al., 2021), and DETR-DC5 is modification of DETR in ResNet. For a fair comparison, we compare ViDT (Swin-tiny) with similar parameter numbers. In general, ViDT shows better trade-off between AP and FPS even compared with (Deformable) DETR with the ResNet-50. Specifically, ViDT achieves FPS much higher than DETR-DCS and Deformable DETR with competitive AP. Particularly when training ViDT for 150 epochs. ViDT outperforms other compared methods using the 50 backbone in terms of both AP and FPS.
dilated convolution the last stage
C.2 VARIATIONS OF EXISTING PIPELINES
We study more variations of existing detection methods by modifying their original pipelines in Table 12. Thus, we remove the neck encoder of Deformable DETR to increase its efficiency, while adding neck decoder YOLOS to leverage multi-scale features along with auxiliary decoding
15

---

Published as a conference paper at ICLR 2022
Ingut Image
Query Id: 82 (Gat)
Query Id: 94 (Cat)
Query Id: 7 (Remote)
Queey Id: 53 (Remote)
Suage
Stage 2
Stage 3
Stage 4
(a) Cross-attention at all stages [1, 2, -3.
Ingut Image
Query Id: 44 (Cat)
Query Id: 49 (Cat)
Query Id: 25 (Remote)
Query Id: (Remote)
Stage
(b) Cross-attention at the last stage {1).
Figure 4. Visualization the attention map for cross-attention with VIDT (Swin-nano).
loss and iterative box refinement. Note that these modified versions follow exactly the same de- tection pipeline with ViDT, maintaining encoder-free neck between their body and head. Table 14 summarizes the performance of all the variations in terms of AP, FPS, and the number of parameters.
Deformable DETR shows significant improvement in FPS (-14.4) but its AP drops sharply (-9.1) when its neck encoder is removed. Thus, it is difficult to obtain fine- ane-grained object detection rep- resentation directly from the raw ViT backbone without using an additional neck encoder. How- ever, ViDT compensates for the effect of the neck encoder by adding [DET] tokens into the body (backbone), thus successfully removing the computational bottleneck without compromising AP; maintains 6.4 higher AP compared with the neck encodrrfrre Deformable DETR (the second row) while achieving similar FPS. This can be attributed to that RAM has great contribution to the performance w.r.t AP and FPS, especially for the trade-off between them.
YOLOS shows significant gain AP (++777 while losing FPS (-11.0) when the neck decoder added. Unlike Deformable DETR, its AP significantly increases even without the neck encoder due to the use of a standalone object detector as its backbone (i.c., the modified DeiT Figure 2(b)). However, its AP is lower than ViDT by 2.3AP. Even worse, it is scalable for large models because of its quadratic computational cost for attention. Therefore, in the aspects of accuracy and speed, ViDT maintains its dominance compared with the two carefully tuned baselines.
For complete analysis, we additionally add a neck encoder to ViDT. The inference speed of ViDT degrades drastically by 13.7 because of the self-attention for multi-scale features the neck encoder. However, it is interesting to see the improvement of AP by 5.7 while adding only 3M parameters; it is 3.0 higher even than Deformable DETR. This indicates that lowering the computational complexity of the encoder and thus increasing its utilization could be another possible direction for fully transformer- based object detector.
16

---

Published as a conference paper at ICLR 2022
C.3
[DET] [ATTH ATTENTION IN RAM
In Section 4.2.2, it turns out that the cross-attention in RAM is only necessary at the last stage of Swin Transformer; all the different selective strategies show similar AP long cross-attention is activated at the last stage. Hence, we analyze the attention map obtained by the cross-attention in RAM. Figure 4 shows attention maps for the stages of Swin Transformer where cross-attention is utilized; it contrasts (a) ViDT with cross-attention at all stages and (b) ViDT with cross-attention at the last stage. Regardless of the use of cross-attention at the lower stage, it is noteworthy that the finally obtained attention map at the last stage is almost the same. In particular, the attention map at Stage 1-3 does not properly focus the features on the target object, which is framed by the bounding box. In addition, the attention weights (color intensity) at Stage 1-3 are much lower than those at Stage Since features are extracted from a low level to a high level in a bottom-up manner as they go through the stages, it seems difficult to directly get information about the target object with such low-level features at the lower level stages. Therefore, this analysis provides strong empirical evidence for the use of selective [DET] x [PATH] cross-attention.
C.4 [DE x DET] ATTENTION IN RAM
Another possible consideration for VIDT is the use of [DET] x [DET] self-attention in RAM. We con-
Stage ld
Swin- nano
2
3
4
AP
FPS
duct an ablation study by removing the [DET] x [DET] attention one by one from the bottom stage, and
/
/
v
40.4
20.0
(2)
/
v
40.3
20.1
summarize the results in Table 15. When all the
/
40.4
20.2
[DET] x [DET] self-attention are removed, (5) the AP drops by 0.7, which is meaningful performance
(4)
40.1
20.3
(5)
39.7
20.4
degradation. On the other hand, as long as the self- attention is activated at the last two stages, (1) (3)
Table 15. AP and FPS comparison with different
DET] x [DET] self-attention strategies with ViDT.
all the strategies exhibit similar AP. Therefore, only keeping [DET] [DET] self-attention the last two stages further increase FPS (+-0.2) without degradation in AP. This observation could be used as another design choice for the AP and FPS trade-off. Therefore, we believe that [DET] [DET] self-attention is meaningful use in RAM.
D PRELIMINARIES: TRANSFORMERS
A transformer is deep model that entirely relies on the self-attention mechanism for machine trans- lation (Vaswani et al., 2017). this section, we briefly revisit the standard form of the transformer.
Single-head Attention. The basic building block of the transformer is self att--attentoon module, which generates weighted sum of the values (contents), where the weight assigned to each value is the attention score computed by the scaled dot-product between its query and key. Let Wo. Wr. and Wy be the learned projection matrices of the attention module, and then the output is generated by
(ZW)) T
Attention(2) 7) softmax
(ZWv) Roxd
Vd
where Wº. WK. Wy Rdddd.
Multi-head Attention. It is beneficial to maintain multiple heads such that they repeat the linear projection process times with different learned projection matrices. Let Wo., WK,, and Wy; be the learned projection matrices of the i-th attention head. Then, the output is generated by the concatenation of the results from all heads.
Multi-Head(2 Z) [Attention, Attention) iiienionn()....,AAeettooo
hwwdd
where V, Wo.: WK, Wv; E (d//A)
Typically, the dimension of each head is divided by the total number of heads.
Feed-Forward Networks (FFNs). The output of the multi-head attention is fed the point wise FFNS, which performs the linear transformation for each position separately and identically to allow the model focusing the contents of different representation subspaces. Here, the residual connec-
17

---

Published as a conference paper at ICLR 2022
tion and layer normalization applied before and after the FFNs, The final output is generated by = LayerNorm((Doopout (H') + H"),
(9)
where H' = FFN(H)) and H" = LayerNorm
(Z)) + Z).
Multi-Layer Transformers. The output previous layer is fed directly to the input of the next layer. Regarding the positional encoding, the same value is added to the input of each attention module for all layers.
18


Reviews:
[{"review_id": "kIt5I7ZB2Yr", "reviewer": "Reviewer_vxqH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Compared with other transformer-based detectors, the proposed detector is effective and efficient, which has a high performance with a fast speed.\n- The comparison with other transformer-based detectors is written well and readable.\n\nWeaknesses\n- Though the proposed detector is effective and efficient, the novelty is limited. To my knowledge, Reconfigured Attention Module (RAM) and token-based knowledge distillation is not novel. Also, other techniques, such neck structure and auxiliary decoding loss, are not original, which are borrowed from DETR and Deformable DETR. \n- There are some concerns about  Reconfigured Attention Module (RAM). First, there is little about the cross-attention between [DET] $\\times$ [PATCH]. Could you provide more details about how to process [DET] $\\times$ [DET] and [DET] $\\times$ [PATCH] at once? Second, the proposed RAM is not novel and may not have great contribution to the performance. RAM only has cross-attention between [DET] $\\times$ [PATCH] in the last stage. The function of this module is the same as the transformer decoder. For the rest stages,  there are two operations: [PATCH] $\\times$ [PATCH] and [DET] $\\times$ [DET]. [PATCH] $\\times$ [PATCH] is a default operation in Swin Transformer. To my knowledge, [DET] $\\times$ [DET] is not meaningful and may have no effect on performance. Could you provide an experiment that removing [DET] $\\times$ [DET] from the RAM module except the last stage?\n- In Table 4 (w. Neck), the detection performance drop largely if all the stages are not involved. In this setting (w. Neck), the transformer decoder also have interaction between [DET] $\\times$ [PATCH]. Thus, the performance drop is not only caused by the lack of interaction between [DET] $\\times$ [PATCH] in RAM. Could you provide an experiment that only interacting with {4} [PATCH] in the transformer decoder?\n- In Table 2, there is not an experiment about Deformable DETR + Swin-base. Could you provide this experiment? ", "summary_of_the_paper": "In this paper, the authors propose an efficient transformer-based detector. Specifically,  the proposed detector introduce Reconfigured Attention Module (RAM) , encoder-free neck and token-based knowledge distillation to boost performance with low computational overhead. ", "summary_of_the_review": "I appreciate the effective and efficient of the proposed detector, but I prefer to reject this paper for the above weaknesses. If the authors solve my concerns, I would like to raise my rate.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_iW2hkqxOBU", "reviewer": "Reviewer_JKxV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths\n\n- This paper is well motivated and clearly written. It proposes a high-performance Detection Transformer coined as ViDT, which synergizes the best of Swin Transformer, YOLOS, as well as DETR (Deformable DETR) architectural design and for the first time demonstrates that the Detection Transformer family can achieve a very competitive accuracy-speed trade-off in the challenging COCO object detection benchmark. A common pitfall is that some people believe Deformable DETR is \"fast\". Deformable DETR is indeed fast in terms of the convergence, *i.e.*, the total training epochs required by Deformable DETR are much smaller compared with the original DETR. However, Deformable DETR is not fast in terms of inference FPS, which is even slower than DETR. ViDT addresses the latency issues while maintaining a very high detection AP. It shows that Detection Transformer family also has the potential to be as strong as the highly-optimized detectors, *e.g.*, the YOLO series.\n\n- This paper proposes an efficient Detection Transformer design principle that can generalize well and inspire future Detection Transformer design, *i.e.*, the encoder part in current DETR & Deformable DETR design results in high latency. Inspired by YOLOS, this paper proposes to incorporate [DET] tokens in the earlier stage of the model, which can compensate for the effects of the encoder while maintaining high inference FPS. Therefore the Transformer encoder part can be ablated by adding [DET] tokens to stage-4 of the backbone. Meanwhile, this paper finds that the Transformer decoder part is crucial for fast convergence for several Detection Transformer instances, which cannot be easily removed.\n\n- This paper conducts a lot of experiments studying different configurations of DETR & Deformable DETR with ViT backbone, YOLOS, and the proposed ViDT in the same testbed. These results are quite meaningful and can be served as valuable references & baselines to the community as well as future work.\n\n  \n\n#### Weaknesses\n\n- I believe there is one missing detail about the model design. If the [DET] x [PATCH] attention starts at stage-4, I wonder what happens to the [DET] x [DET] attention? Is [DET] x [DET] attention performed across all previous stages? And what are the feature dimensions of [DET] tokens? Are the feature dims always the same as the stage-4 feature, or they are gradually raising during different stages. In a word, I think the details and configurations of [DET] x [DET] attention need to be clarified.\n\n\n--- Post rebuttal update ---\nAfter reading the rebuttal, other reviewers' comments, and the discussion, overall I think the detector presented in the paper is strong in terms of both precision and efficiency, and the advantages I mentioned above are valuable, especially the first one. I keep my original rating.", "summary_of_the_paper": "This paper proposes ViDT, a high-performance Detection Transformer with an impressive accuracy-speed trade-off. A lot of experiments as well as ablation studies are conducted to prove the effectiveness of the proposed detector. The design principle of ViDT can also generalize and inspire future detector design. Moreover, this paper also includes an in-depth analysis of several current Detection Transformer architectures.\n", "summary_of_the_review": "Overall, I believe this is a meaningful paper from the perspective of both object detection performance and research. It will also inspire the design of [DET] token based framework of other object- and region- level downstream tasks, *e.g.*, instance segmentation, video instance segmentation as well as panoptic segmentation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Sv7-nZmO6e3", "reviewer": "Reviewer_cEv4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The description of the cross-attention is a bit informal and hard to understand. There is a lot going on in Fig 3 which could be formally described for the readers benefit.\n\nYOLOS is not scalable because of global attention between [PATCH] tokens and [DET] tokens, however here the same issue arises with cross-attention (p4 last para) and is resolved by only considering global attention at the last stage. Could something similar be done with YOLOS? e.g. removing the global cross attention between patch and det in earlier stages. And if such global attention is removed in earlier stages, Swin type layers can also be used if I am not mistaken? In that case does the network correspond to YOLOS largely (without the neck?). This should be discussed as well.\n\nDETR (vanilla and deformable) results are not given with the base models. I understand that the FPS would be low, but the results should be reported. In addition the DeiT-tiny and DeiT-small should also be the distillation based training versions. Since they are +2 points better than the non distillation ones, one could suspect that the (a) Deformable DETR + DeiT-tiny will improve from 39.2 to ~41, which would be then higher than (b) ViDT+Swin-nano (which is comparable at 16M params vs. 18M for (a)). This would also be fair wrt to backbone, from Tab1 and Sec4(Algothims) discussion, it is mentioned that the DEIT tiny, small are eqv. to Swin nano and tiny.\n\nI am also wondering why are DETR(vanilla or Deformable) results with any base model (DeiT-base or Swin-base) not given in Tab2. While they might not be comparable wrt FPS, they should be given.\n\nSimilarly, the results with Conv backbones should also be given, since there are higher reported results in the literature. I wouldn't penalize the paper for not having \"state-of-the-art\" results. This request is so that the reader can be aware of the holistic landscape.\n\n\n--- Post rebuttal update ---\nI am not entirely convinced that a simpler baseline improvement to YOLOS can not be devised. But the proposed method is surely one way to go forward. The results and discussions could also be valuable to the readers. I am not upgrading my rating but I wouldn't argue strongly for rejections, specially when there is some support from the other reviewers.", "summary_of_the_paper": "The paper builds upon the recent advances in transformer based image classification methods (ViT variants) and detection methods (DETR variants). It argues that naively replacing the conv feature backbone in DETR with a ViT based one, is problematic due to (i) the quadratic complexity of the self attention module in ViT, and (ii) then again the attention module in the transformer encoder decoder part (which they call \"neck\"). To remedy, it proposes to build on top of Swin Transformer backbone, by proposing a novel reconfigured attention module (RAM), and further removing the encoder-decoder in the neck, replacing it with only a lightweight decoder.", "summary_of_the_review": "The paper is generally well written, however one of the main contribution, the reconfigurable attention module, should be more precisely and formally described, at least an appendix. Otherwise it is a little hard to get with Fig3 largely.\n\nThe contextualization of the paper wrt YOLOS is also something that could be improved. It seems that the model is more closely related to YOLOS than seems at first read.\n\nThe experiments should be made more complete with fair comparison for baseline backbones etc. And the results with conv backbones should also be given for completeness and benefit of the reader.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an object detection method that integrates vision and detection transformers through a novel Reconfigured Attention Module (RAM). Among other questions, the reviewers raised concerns about fair comparison with baselines, limited novelty of the RAM module, completeness of experiments, and missing details. The rebuttal adequately addressed these concerns with clarifications and additional experiments. R1 remained unconvinced that a simple modification to YOLOS could not be devised to improve the speed similar to the proposed method, but stated he/she wouldn’t argue strongly for rejection. While this is a legitimate concern, the AC agrees with R2 and R3 that the paper has enough merits to be accepted at ICLR, as the results are strong and are likely to have significant practical value.
Decision: Accept (Poster)