A Transformer-Based Object Detector with Coarse-Fine Crossing Representations
Authors: Zhishan Li
Ying Nie
Kai Han
Jianyuan Guo
Lei Xie
Yunhe Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Transformer-based object detectors have shown competitive performance recently.  Compared with convolutional neural networks limited by the relatively small receptive fields, the advantage of transformer for visual tasks is the capacity to perceive long-range dependencies among all image patches, while the deficiency is that the local fine-grained information is not fully excavated. In this paper, we introduce the Coarse-grained and Fine-grained crossing representations to build an efficient Detection Transformer (CFDT). Specifically, we propose a local-global cross fusion module to establish the connection between local fine-grained features and global coarse-grained features. Besides, we propose a coarse-fine aware neck which enables detection tokens to interact with both coarse-grained and fine-grained features. Furthermore, an efficient feature integration module is presented for fusing multi-scale representations from different stages. Experimental results on the COCO dataset demonstrate the effectiveness of the proposed method. For instance, our CFDT achieves 48.1 AP with 173G FLOPs, which possesses higher accuracy and less computation compared with the state-of-the-art transformer-based detector ViDT. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/CFDT.

A Transformer-Based Object Detector with Coarse-Fine Crossing Representations
Zhishan Lile Ying Nie² Kai Han Jianyuan Guo' Lei Xie Yunhe Wang"
'College of Control Science and Engineering, Zhejiang University
"Huawei Noah's Ark Lab
(zhishanli, 1xie) edu. en, fying.nie, kai.han, yunhe. wwgg:aanaueei.nnccoooo com
Abstract
Transformer-based object detectors have shown competitive performance recently. Compared with convolutional neural networks limited by the relatively small ceptive fields, the advantage of transformer for visual tasks is the capacity to perceive long-range dependencies among all image patches, while the deficiency is that the local fine-grained information is not fully excavated. In this paper, we introduce the Coarse- graiined and Vine-graines crossing representations build an efficient Detection Transformer (CFDT). Specifically, we propose local- global cross fusion module to establish the connection between local fine-grained features and global coarse-grained features. Besides, we propose coarse-fine aware neck which enables detection tokens to interact with both marse--raaied and fine-grained features. Furthermore, efficient feature integration module is presented for fusing multi-scale representations from different stages. Exper- imental results the COCO dataset demonstrate the effecttivnness of the pro- posed method. For instance, our CFDT achieves 48.1 AP with 173G FLOPs, which possesses higher accuracy and less computation compared with the state- of-the-art transformer-based detector ViDT. Code will be available at https: http://www.aapppoooo/e//////eeea
1 Introduction
Object detection is a fundamental task in the field of computer vision[] The former mainstream architectures for object detection are mostly based on convolutional neural networks (CNNs)
With the pioneering work of transformer 8 from natural language processing 9 I0 into
object detection by DETR [I]], its variants [22
show competitive detection performance
15 which can be attributed to the strong long-range dependency capturing ability.
Modern CNN-based object detectors, such FaSter-RCNN [16). YoloV3 [17] FCOS 18). and Effi- cientDet 19]. can be divided into three components: backbone, neck and head. With the development common manners deploy transformer for object detec-
of transformer in vision tasks, there tion. One is to replace the CNN-based backbones with transformer variants in object detectors. For example, some recently proposed transformer architectures like Swin Transformer
(0)). PVT 21 22
and CMT 23 are utilized as backbone in the Mask-RCNN 24] or RetinaNet [25] detection frame- works. However, this manner heavily relies on the original detection frameworks, while anchor gener- ation and post posteprocessing with non non-maximum suppression (26] are still indispensable. In this way, the role of transformer is just the backbone for feature extraction. The other manner is to replace the neck part with transformer [1I, which discards the post-processing and anchor setting in conventional detection frameworks. Such a method still requires CNN extract semantic information from images.
*Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The typical representatives of this ap- proach are DETR [T]] and its variants, such as Deformable-DFr DETT [2), Effi cient DETR [3], and DAB-DETR [27]. Carion et al. propose DETR [I] to firstly combine CNN and transformer to build end-to-end detector. In DETR,
47
COCO AP (%)
46
45
ResNet 44] is used as the backbone for extracting features, and transformer is proposed to integrate the relations be- tween learnable object queries and inter- mediate image features. However, there two limitations for DETR. The first is the redundant computation brought by the encoder-decoder architecture of neck. The other is the slow convergence speed, which requires 500 epochs for training. Inspired by deformable convo-
44
43-
42
FasterRCNN
+
DN DETR
41
DETA
VIDI
Conditional
40
50
100
150
200
250
300
FLOPs (G)
Figure 1: Performance comparison with other represen- tative detectors on COCO 2017 val set. The FLOPs is calculated with a S00 1333 input image.
lution networks 28 Zhu al. propose Deformable DETR [12], which replaces the original multi-heat attention with de-
formable attention module. Besides, it aggregates multi-scale features in different stages of backbone, which is effective for object detection. With the multi-scale deformable attention module, Deformable DETR greatly exceeds DETR in both accuracy and training speed. In order to simplify the complexity of the transformer- based detection framework, Fang el al. proposes YoloS which realizes fast ob- ject detection with simple structure. To fully dig the potential of transformer in object detection task, Song et al. construct an efficient and effective fully transformer-basee transformer-basee architectures) object detector called ViDT [4] It adopts Swin-Transformer [20) as the backbone and reconfigures the attention module to support standalone object detection. In addi- tion, it incorporates encoder-free neck structure to further boost the detection performance without introducing too much computational burden. ViDT obtains the best AP and latency trade-off among existing wansformer-based object detectors. Furthermore, some recently proposed transformer-base show better performance than original CNN-based detectors.
(both backbone and neck are
object detectors 29 30
31
32
33
For transformer-based detection models like DETR [1] and VIDT(14, the typical strategy is to perform longerange attention on the divided feature patches. Compared with convolutional neural networks limited by relatively small receptive field scale, the main advantage of transformer-based models is the capacity to perceive long-range dependencies among all image patches. However, the rich spatial information inside these divided patches is rarely considered by previous models. Take the general object detection benchmark as an example[[] [[] there are objects with various sizes. Fine-grained representations help to recognize multi-scale and irregular objects. For the previous transformer- based detectors, excessive pursuit of global feature representations yet paying less attention to local representations limits them for multi-scale perception. Therefore, it is crucial for object detectors to capture and fuse both fine-grained features inside the image patches and the global coarse- coarse-grained features to better detect objects with different scales.
In this paper, we propose to fully leverage both global Coarse-grained and local Fine-grained features to build an efficient Detection Transformer (CFDT) with transformer backbone and transformer neck. In the backbone, we maintain both coarse-grained and fine-graine features and introduce lightweight Local-Global Cross Fusion (LGCF) module. In this way, fully bidircctional cross fusion between local ane-grained and global coarse-graine grainee information is carried out in each stage. In terms of neck, we propose Coarse- Fine Aware Neck (CFAN) which allows detection tokens to make attention- based interaction with fine-grained representations firstly, and then perform further interaction with coarse gaare-grained representations. Finally, lightweight bottom-up feature integration algorithm called Efficient Multiscale Feature Intergration (EMFI) is designed for enriching high resolution feature maps in the early stages. The extensive experiments demonstrate the effectiveness of the proposed method. As shown in Figure our CFDT detectors obtain the best AP and FLOPs trade-off among existing transformer-bace object detectors.

---

2
Preliminaries
In this section, we briefly revisit the fine-grained representations in vision transformers and the transformer-based detection frameworks.
2.1 Fine-grained Representations in Vision Transformers
Transformer-based models are recently applied in visual tasks. general, they divide the original images into N patches for capturing long-range dependencies between these N patches [34] 35 36 However, such a framework destroys the internal relationship and ignores the une-grained representations inside each patch. Han et al. [39] propose Transformer iN Transformer (TNT) architecture that constructs not only the global connection among outer patches, but also the inner communication inside each patch. The outer patches describe global coarse- coarse-grained features while the inner patches represent local fine-gra mne-grained information. Without loss of generality, denote Fo-1 and Fリー1 as the outer patches and inner patches input the I-th stage, respectively. Correspondingly, denote F% and Fi as the outer patches and inner patches output by the (-th stage, respectively. Then, the basic TNT block can be formulated as follows:
37 38].
== + MLP(LN(F)) MSA(IN(F) 1)));
FO + MLP(LL(FE' + MSA(LN(F) + FC(F)))))
(2)
where MLP, LN and MSA represent Multi-Layer Perceptron, Layer Normalization 440 and Multi-head Self-Attention, respectively. FC represents the linear projection layer. In a word, Eq. represents the inner transformer and Eq. 2 represents the outer transformer with inner attention. With TNT block, each outer patch can not only obtain the long-range dependency with other outer patches, but also integrate its corresponding finer-grained inner representations.
PyramidTNT 41 is the improved version of TNT, which introduces pyramid architecture and convolutional stem. With the relatively small amount of computation, PyramidTNT achieves a higher accuracy ImageNet dataset 442. Besides, the pyramid architecture is more suitable the backbone of dense prediction tasks, such as object detection and instance segmentation. That is, with Py as the backbone, the multi-scale inner fine-grained features and outer coarse-grained features can be easily obtained for the downstream tasks.
2.2 Detection Transformers
DETR. DETR utilizes ResNet as the backbone to extract features. In the neck part, it first adopts transformer blocks to perform self-attention on features, and then adopts another 6x transformer blocks to perform cross attention between object queries and features. After the transformer based neck, the final classification and regression results are predicted directly through detection heads. For the selection of training samples, DETR constructs the matching cost matrix between object queries and ground truths, and uses the Hungarian algorithm to efficiently calculate the optimal assignment DETR is end-to-end framework, which does not need anchor boxes and non-maximum suppression.
43
Deformable DETR. There are still two deficiencies for DETR, including slow convergence and relatively poor detection performance for small objects. Deformable DETR proposes a deformable attention module, which attends to small set of key sampling points around reference. Besides, the deformable attention module can be naturally extended to aggregate multi-scale features, which is effective for object detection. Compared with DETR. Deformable DETR only needs 50 epochs to converge and greatly improve the detection performance of small objects.
ViDT. Compared with DETR and Deformable DETR, ViDT is fully transformer-basse detector. For the backbone, ViDT employs Swin-Transforma rather than ResNet. To fully utilize the transformer- based backbone, det tokens and patches share the same attention weights. In the last stage, ViDT constructs a reconfigured attention module to make cross attention between det tokens and patches. As for the neck, ViDT only retains the decoder part of Deformable DETR and its architecture is computationally efficient. Compared with other transformer-ban detectors, ViDT obtains the best AP and latency trade-off.
3

---

3 Approach
In this section, we describe the proposed modules in detail. Firstl, we present the local-global cross fusion module to improve the backbone for object detection. Then, illustrate the coarse-fine aware neck to further make det tokens interact with inner and outer patches. Finally, a lightweight bottom-up feature integration algorithm is introduced.
3.1 Local-Global Cross Fusion
Fiine Patches
Fine Patches
Fine Patches
Fine Patches
PynaiisTNT
LOG7
PyramiTNN
LOcT
PynanidTN
Image
Btocks
Blocks
Blocks
Blocks
Io/W33
Coarse Patches
Fine Patches NiGe
Crossed Fine Patches
Combao
Down Trans
Crossed Coarse Patches ATTEWWGAKG
Coarse Patches NISAWMMTENC
Up-Tram
Figure 2: Illustration of LGCF embedded the backbone PyramidTNT in CFDT. Patch Aggr. represents patches aggregation, which is used to merge and reduce patches. Down-Trans. and Up-Trans. indicate spatial town-transform and up- to-ttranffom of patches, respectively.
In TNT blocks or PyramidTNT blocks, from Eq, 2 although the outer patches can acquire the information of fine-grained inner representations, the restricted receptive field inner patches and the unidirectional "Inner to Outer" strategy limit the diversity of inner features. Besides, the inner patches corresponding to each outer patch can only make self-attention in a fixed x 4 region. As result, they have connection with inner patches that belong other outer patches. Therefore, Local-Global Cross Fusion (LGCF) module is proposed here, as shown in Figure 2 LGCFF can be divided into two sub-modules, including Local Cross Fusion (LCF) and Global Cross Fusion (GCF).
Formally, given a 2D image I E R. xxXXx x3, the outer coarse-grainee patches and inner fine-grained
patches output by the 1-th stage are denoted as Fi R244 2f12 xC₁ and Fl R ft respectively. For the inner patches, LCF is proposed to fuse them with long-range dependency to expand the receptive field. Since the relationship between outer patches global, LCF brings the perception of global information to the original inner patches which only attach fixed x 4 internal self-attention. The process of LCF can formulated as follows:
Cross/ = F + Upsample(Commn x x (Ft
(3)
(Crosss)))
(4)
Eq. 3 describes the cross operation from outer inner. In the equation, Convix1 and U psampll represent convolution with kernel size 1 x 1 and upsampling with bilinear interpolation respectively. The point convolution is adopted keep the features' channel of outer patches consistent with inner patches, and the upsampling operation is adopted to expand the spatial scale of outer patches to 16 times. The transformed outer patches are line with inner patches both in spatial dimension and channel dimension. Cross" is the summation of inner patches and transformed outer patches. Then, we use the combination of features, as described in Eq. Conezx3 represents the convolution operation with 3 x 3 kernel size. LN and F.U.U represent Layer Normalization 40 and Gaussian Error Linear Unit activation 44 respectively.
further fuse the crossed

---

Although the outer patches acquire fine- grained inner representations with original PyramidTNT blocks, the fusion method simple flattening and addition ignores spatial information. Similar to ICF, we propose GCF to integrate inner features into outer patches. This process can be formulated as follows:
Cross Fo + ConvAxa(F)) GELU (ww.sssCrrosssl)l
(5)
(6)
Eq. re.rreeeeentt the cross operation from inner patches to outer patches. Different from the combi- nation of point convolution and upsample in Eq. 3 we directly use Comu4x4 to get transformed
Fi consistent with F% in both spatial shape and channel. In Eq. we also use the combination of
"Convolution-Nee ceemalizaaion-Activane
to further fuse the crossed features.
With this module, bidirectional cross fusion is carried out between local features and global features after each stage. The local representations integrate more global information and the original global coarse-graines representations fuse the fine-grained information. Experimental results show that the proposed LGCF can greatly improve the detection performance.
3.2 Coarse-Fine Aware Neck
MSDA
MSDA Coarse coarse-griinedeee Patches
Fine-rained Patches
Det Token
Aggrsgated Sampled Usius
Aggregitel
Figure Illustration of the coarse-fine aware neck CFDT. MSDA the abbreviation of Multi-Scale Deformable Attention. The small brown star indicates sampling point.
No matter DETR or ViDT, the neck part always performs the cross attention operation between det tokens and global long-range dependency features. In object detection task, there are usually targets with different scales in different positions. Therefore, it is necessary to pay attention to multi-scale and multi-source useful features. Here, we propose a Coarse- Aware Neck (CFAN) module, which allows det tokens to interact with not only coarse-grained outer patches but also with fine-grained inner patches. In practice, we perform Multi-Scale Deformable Attention (MSDA) [12] between det tokens and local iineggraied features firstly, then between det tokens and global coarse-graine paarre-grainedd features, as illustrated in Figure The cross attention between det tokens and inner patches can be formulated follows:
Mr Σ
K2
MSDA(Ques (AIL)))
Wm
Σ
Amile W ((r(p) Apmit)
(7)
where
=det-Qua + MSSA((Qott (T
where Qdet represents the det tokens. Eq. represents the MSDA process between det tokens and inner patches. M, indicates the attention head and K2 is the total number of sampled keys in inner patches. Besides, (p) represents the reference point the I-th stage features, while Apmar represents the corresponding sampling offset for the next deformable attention operation. Amak is the attention weights of the K-th sampling contents. W.. and W'n are the projection matrices in multi-head attention operation.
After the deformable cross attention of det tokens and inner patches, we first combine the MSDA(Quee, { Qdet and then interact with Fo. This process can be formulated as follows:
Mo
Ko Σ
MSDA(Qdeer (F)11
Σ
Wm
Amil W (o( APmit)
(9)
5

---

where Mo indices the attention head and Ko is the total number of sampled keys in outer patches. Consistent with the original MSDA in Deformable DETR [22, we set the default values of M, and Mo to 8 in the following experiments.
3.3 Efficient Multi-scale Feature Intergration
x3 the shapes of output features in four stages in our backbone can be set as W in four stages in Swin- Transformer or
For an input image I E _xxx>x
Up-Trans.
H x 16 16 32 32 64 x 64 respectively. In contrast, the shapes of output features
W. H. x
Up-Trans.
x 8 8 16 T6 32 32 respectively. In ViDT or Deformable DETR, when utilizing Swin- Transformer or ResNet as backhone, the output features of the first stage is not used by the neck due to the insufficient useful information. The downsampled features of the last stage are usually taken additional features in their methods. In object detection task, the shapes of features have great impact on the final performance. In general, it is more effective to use large-scale features for small object detection. Therefore, although the feature extraction capability of the first stage is limited, still cannot ignore the output features of this stage.
ResNet should be H x W H. W H
Up-Trans.
Figure 4: Illustration of the efficient multi-scale feature intergration module in CFDT. F' represents the output patches (inner or outer) in (-th stage.
To effectively utilize the output of the first stage, we design an Efficient Multiscale Feature Integration (EMFI) module, as shown Figure We insert EMFI into the fine-grained and coarse grainedd features respectively, which can enhance the feature representation of the early stages. This module can be formulated as follows:
Fo Fo + U (psmmple(Conuix
(10)
F = F} U D'psampleeConnn..ll (C
(11)
The output of 1-th stage is the summation of original features and transformed output of (1 + 1)-th stage. The transformer operation is consist of bilinear interpolation and point convolution, which can bring ignorable computational cost. Such a bottom-up structure enables the high-resolution features of the early stages to integrate the later features with low resolution but richer semantic information.
4 Experiments
In this section, conduct extensive experiments. Thorough ablation studies also provided in this section.
4.1 Dataset and Implementation Details
We conduct experiments Microsoft COCO 2017 benchmark M. Following the usual practice, 1 18K images are used for training. 5K images for testing. In addition, we follow the training strategy provided ViDT [14], including Adam W (45) with the initial Iearning rate of 1 10 10-4, training with multi-scale input sizes and the total training epoch is set as 50. We use PyramidTNT series models pretrained on ImageNet- as the initial backbones of detectors. The results are reported over three backbones: PyramidTNT-Ting (P- (P-Tiny), PyramidTNT-Smmnn (P-Medium). Considering the existence of Batch Normalization [46] in the stem of backbone and the batch size of training object detection network is much smaller than that in image classification, we freeze all parameters of Batch Normalization layers in the training process. Besides, the training batch size per card of P-Tiny and P-Small is set as 2, while that of P-Medium is 1. For evaluation, Average Precision (AP) is calculated on COCO, and the floating point operations (FLOPs) is calculated under 800 1333 input image.
(P-Small), and PyramidTNT-MMdiin

---

Table 1: Comparisions of CFDT with other transformer-base detectors on COCO 2017 val set.
Model
Backbone
AP
AP
APrs
APs
APM
APt
FLOPs (G)
FLOPs (G) Range: 10~50
Y0LOS477 ViDT 14
DeiT-Tiny
30.4
48.6
31.1
12,4
31,8
48,2
21
Swin-Nano
40.4
59.6
43.3
23.2
42.5
55.8
37
CFDT
P-Tiny
43.0
62.5
45.8
23.9
45.7
59.7
33
FLOPs (G) Range: 50~150
DETR II
ResNct-50
42.0
62.4
44.2
20.5
45.8
61.1
86
Conditional DETR 48
ResNct-50
40.9
61.8
43.3
20.8
44.6
59.2
90
DAB DETR
ResNet-50
42.2
63.1
44.7
21.5
45.7
60.3
UP DETR 49
ResNet-50
42.8
63.0
45.3
20.8
47.1
61.7
86
DN DETR SQ
ResNet-50
44.1
64.4
46.7
22.9
48.0
63.4
94
SAM DETR
ResNet-50
41.8
63.2
43.9
22.1
45.9
60.9
100
ViDT
Swin-Tiny
44.8
64.5
48.7
25.9
47.6
62.1
114
CFDT
P-Small
45.8
65.3
49.2
25.9
48.5
63.6
77
FLOPx(G) Range: 150~300
DE
ResNet- 101
63.8
46.4
21.9
48.0
61.8
152
DETR
DC5-ResNet-50
43.3
63.1
45.9
22.5
47.3
61.1
187
DETR
DC5-ResNet-10
44.9
64.7
47.7
23.7
49.5
62.3
253
Efficient DETR 13
ResNct-50
45.1
63.1
49.1
28.3
48.4
59.0
210
Efficient DETR 13
ResNet-101
45.7
64.1
49.5
28.2
49.1
60.2
289
Conditional DETR 48
ResNet-101
63.7
46.0
21.7
46.6
60.9
156
Conditional DETR 48
DC5-ResNet-50
43.8
64.4
46.7
24.0
47.6
60.7
195
Conditional DETR 48
DC5-ResNet-100
45.0
65.5
48.4
26.1
48.9
62.8
262
SMCA 52
ResNet-50
45.6
63.5
49.1
25.9
49.3
62.6
152
SMCA
ResNet-101
46.3
66.6
50.2
27.2
50.5
63.2
218
DAB DETR 27
ResNet-101
43.5
63.9
46.6
23.6
47.3
61.5
174
DAB DETR 27
CC5-ResNet-50
44.5
65.1
47.7
25.3
48.2
62.3
202
DAB DETR 27
DC5-ResNet-101
45.8
65.9
49.3
27.0
49.8
63.8
282
DN DETR
ResNet-101
45.2
65.5
48.3
24.1
49.1
65.1
174
DN DETR
DC5-ResNet-50
46.3
66.4
49.7
26.7
50.0
64.3
DN DETR (5)
DC5-ResNet-100 101
47.3
67.5
50.8
28.6
51.5
65.0
282
Deformable DETR 12
ResNet-50
45.4
64.7
49.0
26.8
48.3
61.7
173
SAM 51
DCS- DC5-ResNet-50
45.0
65.4
47.9
26.2
49.0
63.3
210
47
DeiT-Small
36.1
-5.77
37.6
15.6
38.4
55.3
194
V4
Swin-Small
67.7
51.2
30.4
50.7
208
CFDT
P-Medium
48.1
67.8
51.8
28.1
50.9
66.4
173
denotes that backbone and neck both transformer-based architecture.
4.2 Main Results
We compare method with latest transformer-basee detectors, including DETR II], 52
UP DETR [49], Efficient DETR[3] Conditional DETR 48 DAB DETR(27), DN DETR 50), SAM 51] YOLOS and VIDT as shown in Table For fair comparison, all the results do not utilize the strategy of multi-scale test.
Compare with tiny detectors. YOLOS 47 is canonical VIT architecture for object detection. Although it has a small computational cost, the neck-free design withholds the YOLOS from obtaining CFDT achieves +12.7 AP compared to the Deit-tiny based YOLOS. When
high performance, compared to the recently proposed lightweight ViDT[14), our CFDT outperforms it by +2.6 AP with fewer FLOPs. More specifically, the backbones of ViDT and CFDT attain similar results ImageNet (74.9 of Swin-Nano v.s. 75.2 of P- P-Tiny), and the superiority in COCO further demonstrates the improvements brought by proposed LGCF, CFAN, and EMFI
Compare with small detectors. We further compare our P-small based CFDT with Swin-tiny based ViDT and several variants of ResNer-50 based DETR. For example, DN DETR 50) accelerates DETR training by introducing query denoising. Our CFDT outperforms by +1.7 AP with far less FLOPss (-17G), and we still exceed the ViDT by +1.0 AP, and the FLOPs is significantly reduced by 37G.

---

Compare with medium detectors, For the backbone with P-Medium, our CFDT achieves 48.1 AP with 173G FLOPs. In terms of AP, the detectors close our method are DN DETR with DCS-ResNet-101 and ViDT with Swin-Small. Compared with them, the FL.OPs of our method is lower than these detectors by 109G and 35G, respectively. Besides, our method still reaches better detection performance.
When compared those detectors with ResNet the backbone, transformer-base models like ViDT and CFDT show the better trade-off between accuracy and computational cost (higher AP and fewer FLOPs). This also reveals that the detectors which consist of transformer-base backbone and munsformer-basee neck possess great potential for efficient object detection.
4.3 Ablations
Comparison between PyramidTNT and Swin-Transformer. In order il- lustrate that our detection performance not due the backbone replace- ment of PyramidTNT,, we firstly show the comparison of PyramidTNT and Swin-Transformer in Table [] Al-
Table 2: Comparisons between Swin-Transformer and PyramidTNT.
Models
ImageNet (Top-1)
COCO (AP)*
Swin-Nano PyramidTNT-Tinn
40.4
75.2
40.8
though the Top-1 accuracy of Pyra- midTNT series on ImageNet is slightly higher than that of the corresponding sized Swin- Transformer models, AP of directly taking PyramidTNT- Small or PyramidTNT-Mediue
Swin-Tiny
81.3
44.8
PyramidTNT-SSaal
82.0
43.4
Swin-Small
83.0
47.4
PyramidTNT-
83.5
44.3
backbone of ViDT is significantly lower than that Swin-Transformer. For further analy- sis, compared with PyramidTNTT, Swin- Transformer utilizes the "Shift Window" obtain multi-scale features, which is effective for object detection. So for transformer-based object detection, it is not good enough to directly deploy PyramidTNT backbone.
AP is obtained by taking ViDT detection framework and corresponding models set as hackhone.
Local-Global Cross Fusion. We analyze the impact of different elements in LGCF, as shown in Table The baseline of our method with backbone of P- P-Tiny is 40.8 AP. After introducing the LCF to fuse the outer coarse- coarse-grained patches into fine- ane-grained inner patches, AP is improved to 41.3. Correspondingly, it more effective to embed GCF in the detector. After the introduction of GCF, we achieve an AP improvement of GCF is higher than LCF, but it is also accompanied by higher amount of calculation. The reason that the channels of outer patches are 16 times that of inner. So the convolution operation in GCF brings more computation. Finally, combine the two cross fusion strategies together and achieve an improvement of 1,4 AP compared with baseline.
1.2 compared with baseline. For further analysis, the improvement
Table 3: Analysis of Local-Glan Globll Cross Fusion module with backbone of P- P-Tiny.
Backbone
LCF
GCF
AP
AAP
FLOPs (G)
40.8
27.8
41.3
10.5
28.5
P-Tiny
42.0
11.
30.8
42.2
1.4
31.5
Sampling Points of CFAN. We conduct an ablation experiment the sampled points number K, of inner fine-grained patches. The baseline is the P-Tiny backbone with LGCF whose AP is 42.2. For outer patches, we set Ko as which consistent with Deformable DETR and ViDT. Considering that one outer patch corresponds to 16 inner patches, so K2 is set to an integer multiple of 16. The result shown in Table When K, set to there is no cross attention between tokens and fine-grained patches in the neck part. At this case, det tokens directly interact with outer global coarse-grained patches through Multi-Scale Deformable Cross Attention modules. When K2 equals to 32, AP increases to 42.3 with det tokens interacting with 32 sampled keys of inner patches. The

---

best performance is Ky set to 64, which obtains 42.6 AP and 0.4 higher than baseline. At this case, K equals 64 inner points that 4 sampled outer keys correspond to.
Table 4: Effect of sampled keys number K of inner fine-gr ane-ggrined patches in P-Tiny,
K
16
32
48
64
80
AP
42.2
42.2
42.3
42.5
42.6
42.4
Complete Component Analysis. We analyze all components in CFDT, and the detailed result is shown Table 5 For Tiny model backbone, we find that the LGCF, CFAN and EMFI improve the AP of 1.4, 0.4, 0.4 respectively. Combining all the proposed modules together, our method achieve 43.0 AP, which is 2.2 AP higher than baseline. In addition, the increase of computation brought by the introduction of these three modules is acceptable. Compared with ViDT based on Swin-Nano (40.4 AP, 37G FLOPs), our method has less computation cost, but possesses 2.6 AP higher than Swin-Nano. For other backbones of our method, although the proposed modules bring in little computation cost, FLOPs of CFDT is still less than that of other models. Compared with other wansformer-based detectors, our method achieve higher AP.
Table 5: Analysis of all components in CFDT.
Backbone
LGCF
CFAN
EMFI
AP
Aar
FLOPs (G)
40.8
27.8
42.2
11.4
31.5
P-Tiny
42.6
11.8
33.0
43.0
12.2
33.2
43.4
65.1
45.0
f1.6
74.6
P-Small
45.3
11.9
76.1
45,8
12.4
76.5
44.3
150.0
46.5
12.2
170.8
P-Medium
47.2
12.9
172.3
48.1
13.8
173.2
4.4 Visualization of sampling points in CFAN
To better understand the Coarse-Fine Aware Neck, we randomly select three images and visualize the sampling points in both coarse-grained patches and fine-grained patches, shown in Figure From the visualization, it is apparently that the sampling points on coarse-grained patches are more concentrated in a certain area on foreground, while the sampling points on fine-gre fine-grained patches are more widely distributed. However, many sampling points on fine-grained patches are assigned to the background area. Therefore, it is necessary to combine them together to capture the effective features for better detection.
5 Conclusion
In this paper, we propose an efficient object detector called CFDT with coarse-grained and fine- grained cross representations.
order further improve the performance of detector, we propose local-global cross fusion module, coarse-fine fine aware neck and efficient multi-scale feature intergration strategy. Compared with the state-of-the-art transformer-bas approach achieves better detection performance with less computation cost. Among other transformer. based detectors, our method obtains a better trade-off between AP and FLOPS. Experimental results demonstrate the effectiveness of the proposed method. For future research, we hope to transfer this idea to more transformer-base models, so as improve the performance of various visual tasks with transformer.
detector ViDT, the combination of our

---

TRIEK
Figure 5: Illustration of the sampling points in CFAN. We transform the sampling points location of corresponding prediction boxes to original images. The bigger red points indicate the sampling points on coarse- coarse-grained grained patches and the smaller blue points represent the sampling points on fine-grained patches.
Acknowledgement
We gratefully acknowledge the support of MindSpore [53] CANN(Compute Architecture for Neural Networks) and Ascend Processor used for this research. This research supported by National Natural Science Foundation of P.R. China (NSFC: 62073286).
References
[1] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[2] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
[3] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- arXiv preprint arXiv: 1556. 2014.
[4] Kaiming He, Xiangyu 7hang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778. 2016.
[5] Xiaoling Xia, Cui Xu, and Bing Nan. Inception-v3 v3 for flower classification. In 2017 2nd International Conference Image, Vision and Computing (ICIVC), pages 783- 787. IEEE, 2017.
[6] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-6114 14. PMLR, 2019.
[7] Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1580-1589, 2020.
[8] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems. 30, 2017.
10

---

[9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learerr.. Advances neural information processing systems, 33:1877-1901, 2020.
[11] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213 229. Springer, 2020.
[12] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end- coddoo-nd object detection. arXiv preprint arXiv:2010.04152 2020,
[13] Zhuyu Yao, Jiangbo Ai, Boxun Li. and Chi Zhang. Efficient detr: improving end-to-end object detector with dense prior. arviv preprint arXiv:2 1.X.v:2104.01316 2021.
[14] Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varum Jampani, Dongyoon Han, Bycongho Heo, Wonjac Kim, and Ming-Hsuan Yang. Vidt: An efficient and effective fully transformer-based object detector, arXiv preprint Xrx:: 10.00022,,
[15] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 2022.
[16] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[17] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint 18X.v7:804.00777, 2018.
[18] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Foos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9627-9636. 2019.
[19] Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10781-10790, 2020.
[20] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint 2ix::21111009066 2021.
[21] Wenhai Wang, Enze Xie, Xiang Li, Deng- Ling-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbonc for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568-578, 2021.
[22] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong La,, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, pages 1-10, 2022.
[23] Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolu- tional neural networks meet vision transformers. arXiv preprint arXiv:2107.06260 2021.
[24] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[25] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-29000 2988, 2017.
[26] Azriel Rosenfeld and Mark Thurston. Edge and curve detection for visual scene analysis. IEEE Transactions on computers, (00(5):562-50)) 1971.
[27] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi. Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201. 220.1229,,, 2022.
[28] Jifeng Dai, Haozhi Qi. Yuwen Xiong, Yi Li. Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764-773, 2017.
11

---

[29] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Modtr-moduulted detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[30] Tao Wang. Yuan, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. Pnp-detr: towards efficient visual analysis with transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 4661-667777 2021.
[31] Duy-Kin Kien Nguyen, Jihong Ju, Olaf Booji, Martin R Oswald, and Cees GM Snock. Boxer: Box-attention for 2d and 3d transformers, arXiv preprini arX1v:2111.13087, 2021.
[32] Pei Wang, Zhaowei Cai, Hao Yang, Gurumurthy Swaminathan, Nuno Vasconcelos, Bernt Schiele, and Stefano Soatto. Omni-detr: Omni-supervised object detection with transformers. arXiv preprint arXiv:2203. 16089, 2022.
[33] Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang. Multimodal token fusion for vision transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22186-12195, 2022.
[34] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1999, 2020.
[35] Hugo Touvron. Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[36] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao. Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang. Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint arXiv:2111.0988. 2021.
[37] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9981-99999 2021.
[38] Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. Soft: Softmax-free transformer with linear complexity. Advances Neural Information Processing Systems, 34, 2021.
[39] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 34, 2021.
[40] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 10X7v06607.06500 2016.
[41] Kai Han, Jianyuan Guo, Yehui Tang, and Yunhe Wang. Pyramidtnt: Improved trannfomer-inttrannsomeeer basclines with pyramid architecture. arXiv preprint 2rXiv:2201.00977, 2022.
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, ct al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 15225,222222 2015.
[43] Russell Stewart, Mykhaylo Andriluka, and Andrew Y Ng. End-to-end people detection in crowded scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2325-2333 2016.
[44] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint urXiv:1606.0844 2016.
[45] Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1.0 101, 2017.
[46] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456.. PMLR, 2015.
[47] Yuxin Fang. Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi. Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. Advances in Neural Information Processing Systems, 34, 2021.
12

---

[48] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng. Houqiang Li. Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3651-3660, 2021.
[49] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. In Proceddiggs of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 1601-1610, 2021.
[50] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. arXiv preprint arXiv:2203.00 01 305,, 2022.
[51] Gongjie Zhang, Zhipeng Luo, Yingchen Yu, Kaiwen Cui, and Shijian Lu. Accelerating detr convergence via semantic-aligned matching. arXiv preprint arXiv:2203. 06888, 2022.
[52] Peng Gao, Minghang Zheng, Xiaogang Wang. Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3621-3630, 2021.
[53] Huawei. Mindspore, https: /www///w........ http///wwwwidsorre.c.m cn/ 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See the Abstract Part and Introduction Part.
(b) Did you describe the limitations of your work? [Yes] We have discussed the limitations in the section of limitations and societal impacts in the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We have discussed the limitations in the section of limitations and societal impacts in the supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [No] We conduct experiments to show the results.
(b) Did you include complete proofs of all theoretical results? [No] We conduct experi- ments to show the results.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see the implemental details in experiments section.
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the ereators? [Yes] We cite the TNT and PyramidTNT papers.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "hLxWk4NBmnQ", "reviewer": "Reviewer_8pQM", "summary": "This paper introduces the Fine-grained and Coarse-grained crossing representations for building a Detection Transformer by using a local-global cross fusion module and Fine-Coarse Aware Neck.", "questions": "- Why are features of outer and features of inner different sizes in Line148?\n- What happens when det tokens interacts with coarse-grained outer patches first and then with fine-grained inner patches in section3.2?\n- Based on Eq. 8, my understanding is that Qdet and MSDA(Qdet, Fi) have a shortcut, but why Qdet and MSDA(Qdet, Fo) do not have shortcut?\n", "limitations": "The authors have discussed the limitations and potential negative societal impact of their work in A.4.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper is well organized.\n+ The performance boost seems good with a slight increase in Flops.\n\nWeaknesses:\n- The whole pipeline seems to be fine, but lacks deep insight. Most of the operations are feature aggregation between different levels, which has been deeply explored in object detection.\n- Some of the statements in this paper are confusing. For example, how to understand “combine the MSDA(...) to Qdet and interact with Fo” in Line192.\n- Writing errors. For example, “the lightweight bottom-up feature integration algorithm Efficient Multi-scale Feature Integration” should be “the lightweight bottom-up feature integration algorithm, i.e., Efficient Multi-scale Feature Integration” in Line138. The authors are advised to double-check for similar errors.\n\nOverall, the simple and lightweight structure brings a huge performance boost is the reason I agree to accept this article.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8h4wlFcerMG", "reviewer": "Reviewer_6R9f", "summary": "This paper studies the topic of 'Fully Transformer-based Object Detector' (except the stem layer, do not use CNN layers for operations). Existing Transformers mainly rely on global interaction between outer patches, but this work proposes a way of integrating attention of inner patches, capturing local & fine-grained information; (1) LGCF: mixing fine & coarse patches with interacting them, (2) FCAN: alternating cross-attention btw <det x fine-grained patch> and <det x coarse-grained patches>, and (3) EMFI: using the first block patches efficiently. The proposed architecture shows a better AP and Speed trade-off compared with other compared detectors. To support their justification, the authors provide some ablation studies on the proposed three components.", "questions": "1) Could you explain why $AP_s$ drops compared with ViDT (Swin-small)?\n2) I feel hard to know what components make the architecture faster than others. I understand your design is simple and effective but there is no explanation why it is faster than another efficient mechanism (e.g., local attention in Swin - ViDT).\n3) Your architecture seems to be coupled with PyramidTNT. This could be a limitation (low extendability). Do you think your three components can be combined with other backbones as well? If possible, doing and comparing them would be good to prove that the performance improvement is not from the backbone.\n\n", "limitations": "Refer to the strengths and weaknesses above.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "There are both strengths and weaknesses I found while reading the paper.\n\n### Strength\n1) Integrating the idea of 'local + global information' into Transformer-based object detection is reasonable because improving AP on varying-size objects (especially, small-size) is a major challenge in this domain. \n2) The proposed three components (blocks) look simple and effective when combined. Therefore, they did not harm the inference speed of the detector.\n3) Experimental results are not promising but reasonable improvements.\n\n### Weakness\n1) ***Performance drop for small-size objects***. According to the experiments, $AP_s$ drops from 30.4 to 28.1 compared to ViDT, although more fine-grained information was integrated. I can't understand why this happens. Intuitively, the proposed ideas of fusing more fine-grained information should improve $AP_s$.\n2) ***Incomplete Analysis on Backbones***. In Section 4.3, there is a comparison between PyramidTNT and Swin-Transformer. I think this analysis is not complete to show that the improvement did not come from the replacement. First, ViDT is not designed to be with PyramidTNT. Second, only accuracy and AP measures are reported without 'FLOPS or FPS'. To sum up these, I think a more suitable comparison is combining Swin-Transformer with your three components. The performance trends with ViDT could not tell us their impact on your architecture.\n3) ***Weak Novelty***. The proposed approach mostly relies on existing works. For local- fine-grained information, PyramidTNT is adopted as the backbone, and the proposed three components are very simple and straightforward; this could be an advantage in practice, but academically looks like an incremental paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4UeUdflDamq", "reviewer": "Reviewer_oBDA", "summary": "A fully transformer-based object detector (transformer-based backbone and neck) is an interesting topic. This paper proposes the method of the Fine-grained and Coarse-grained crossing representations for building efficient transformer-based object detectors. In the backbone, this paper maintains both the fine-grained and coarse-grained features and introduce a lightweight local-global cross fusion module. In the neck, the proposed module allows the detection tokens to make attention-based interaction with fine-grained representations firstly, and then perform further interaction with coarse-grained representations. The results on the COCO dataset demonstrate the effectiveness of the proposed method.", "questions": "1. Can the authors further explain the calculation process of Eq. (9)?\n\n2. Where does the proposed efficient multi-scale feature integration demonstrate efficiency compared to other FPNs?\n\n3. In which stage is the proposed multiscale feature integration used?\n", "limitations": "Yes, the authors have discussed the limitations and potential negative social impact of their work in the Appendix.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. This paper is technically sound and easy to understand.\n\n2. The proposed method is interesting and has a clear motivation. Other transformer-based detectors do not make good use of the fine-grained information, resulting in unsatisfactory performance. The proposed local-global cross fusion module, fine-coarse aware neck, and efficient multi-scale feature integration are novel.\n\n3. The experiments are extensive and ablation studies are comprehensive to understand the proposed method.\n\nWeaknesses\n\n1. The writing needs to be improved slightly.\n\n2. The results in Tab.1 are impressive, yet the authors are advised to make more discussions on the results to make the paper stronger.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept