DECO: Unleashing the Potential of ConvNets for Query-based Detection and Segmentation
Authors: Xinghao Chen
Siwei Li
Yijing Yang
Yunhe Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Detection
Segmentation
ConvNet
Segment Anything
Primary area: foundation or frontier models, including LLMs

Abstract: Transformer and its variants have shown great potential for various vision tasks in recent years, including image classification, object detection and segmentation. 
Meanwhile, recent studies also reveal that with proper architecture design, convolutional networks (ConvNets) also achieve competitive performance with transformers. 
However, no prior methods have explored to utilize pure convolution to build a Transformer-style Decoder module, which is essential for Encoder-Decoder architecture like Detection Transformer (DETR).
To this end, in this paper we explore whether we could build query-based  detection and segmentation framework with ConvNets instead of sophisticated transformer architecture.
We propose a novel mechanism dubbed InterConv to perform interaction between object queries and image features via convolutional layers. 
Equipped with the proposed InterConv, we build Detection ConvNet (DECO), which is composed of a backbone and convolutional encoder-decoder architecture. We compare the proposed DECO against prior detectors on the challenging COCO benchmark.
Despite its simplicity, our DECO achieves competitive performance in terms of detection accuracy and running speed. Specifically, 
with the ResNet-18 and ResNet-50 backbone, our DECO achieves $40.5\%$ and $47.8\%$ AP with $66$ and $34$ FPS, respectively. The proposed method is also evaluated on the segment anything task, demonstrating similar performance and higher efficiency.
We hope the proposed method brings another perspective for designing architectures for vision tasks.
Codes are available at \url{https://github.com/xinghaochen/DECO} and \url{https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/DECO}.

Published as a conference paper at ICLR 2025
DECO: UNLEASHING THE POTENTIAL OF CONVNETS FOR QUERY-BASED DETECTION AND SEGMENTATION
Xinghao Chen'; Siwei Li Yijing Yang', Yunhe Wang'
Huawei Noah's Ark Lab 2 Tsinghua University (xi (xinghac. chen, yunhe wang) @huawei.com
ABSTRACT
Transformer and its variants have shown great potential for various vision tasks in recent years, including image classification, object detection and segmenta- tion. Meanwhile, recent studies also reveal that with proper architecture design, convolutional networks (ConvNets) also achieve competitive performance with transformers. However, no prior methods have explored to utilize pure convolution build Transformer-style Decoder module, which essential for Encoder- Decoder architecture like Detection Transformer (DETR). To this end, in this paper we explore whether we could build query-based detection and segmentation framework with ConvNets instead of sophisticated transformer architecture. We propose novel mechanism dubbed InterConv to perform interaction between object queries and image features via convolutional layers. Equipped with the proposed InterConv, we build Detection ConvNet (DECO), which composed of backbone and convolutional enntiionnennnceee-eeeee encoder-decoder architecture. We compare the proposed DECO against prior detectors on the challenging COCO benchmark. Despite its simplicity, our DECO achieves competitive performance in terms detection accuracy and running speed. Specifically, with the ResNet-18 and ResNer-50 backbone, DECO achieves 40.5% and 47.8% AP with 66 and 34 FPS, respectively. The proposed method also evaluated on the segment any- thing task, demonstrating similar performance and higher efficiency. We hope the proposed method brings another perspective for designing architectures for vision tasks. Codes available (tttps:://ittuut
masterr/eeaacchhhnneeennna//ooo
INTRODUCTION
Object detection and segmentation are among the most foundational computer vision tasks and essential for many real-world applications (Ren al. 2015 Redmon & Farhadi 2017 2018
Bochkovskiy et al., 2020). The object detection pipeline has been developed rapidly, especially in the era of deep learning. Faster R-CNN (Ren et al. 2015) is one the most typical two-stage object detectors, which utilizes coarse-to-fine framework for bounding box prediction. Meanwhile, one-stage detectors like SSD (Liu et al., (016), YOLO series (Redmon Farhadi 2017 2018) Bochkovskiy et al., 2020) or FCOS (Tian et al. 2019) simplify the detection pipeline by directly predicting the objects of interest from the image features. Most the above object detectors are built upon convolutional neural networks Nets) and typically the Non-maximum Suppression (NMS) strategy utilized for post-processing to remove duplicated detection results.
The advancement of deep neural architectures have been benefiting the task of object detection. For example, more powerful architectures usually bring considerably significant improvement for the detection performance (Li et al. 2018: Liu et al.) 20200) Gao et al. 2019)] Guo al.) 2020). Recently the emergence of vision transformer and its variants (Dosovitskiy et al.] 2021 Liu et al. 202T
Touvron et al. 202 Wang al. (021b) have shown prominent performances on image classification tasks and have built solid foundation for the object detection field. Carion et al. (Carion et al.
Equal contribution. Correspondence Chen and Y. Wang.

---

Published as a conference paper at ICLR 2025
ConvNeXt-B
50
ResVet-550
O
45
(P(%)
40
RosNat-18 18
Faster R-CNN
Deformable DETR
DDQ-DETR-Sscale
Sparse
Cond-DETR
Lite-DETR
RetinaNet
DN-DETR
Co-DETR
FCDS
VIDT
H-DETR
DcFCN
DINO-4ccale
DETR
Stable-DINo -sccle
Rank-DET
30
YOLOS-TI
RT-DETR
DATE
Anchor-DETR
SpeedDETR
DECO (Ours)
80
100
Latency (ms)
Figure 1: Comparisons of our proposed Detection ConvNets (DECO) and recent detectors on COCO val set. The latency measured on a NVIDIA V100 GPU.
2020) proposes the Detection Transformer (DETR) that refactors the object detection pipeline set prediction problem and directly obtains a fixed set of objects via a transformer encoder-decoder architecture. This design enables DETR to get rid of the complicated NMS post-processing module
and results in a
end-to-end object detection pipeline. There are quite a lot of variants to
improve DETR via different aspects, e-g., training convergence (Meng et al.] 2021] Gao et al.) [2021)
multi-scale features and deformable attention (Zhu et al. 2021) or better query strategy (Li et al.
2022
Liu et al.
2022a Wang al. 2022b Zhang al.
2022).
Despite the strong performance of transformer, does introduce more challenges for Al chips (Tu
al., 2022). More specifically, attention layers introduce dynamic memory access whose weights and inputs are both generated during runtime, while convolutional layers have static memory access. Therefore, it is still common that certain operators like attention module is not well supported in some Al chips, which is a big challenge in industry.
Meanwhile, some recent work rethinks the strong performance and reveal that the pure CenvNets could also achieve competitive performance via proper architecture design (Liu et al. 2022b Yu al. 2022). For example, ConvNeXt (Liu et al. 2022b) competes favorably with vision transformers like Swin Transformer (Liu et al.) 2021) in terms of accuracy and computational cost. However, these methods mainly focus on encoder part of transformer, in which self-attention is utilized and could be replaced by convolution with careful design. These motivate us to explore one important question in this paper: could we obtain an architecture via pure ComNNet but still enjoys the excellent properties similar attention?
In this paper, we propose a novel mechanism dubbed InterCony to perform interaction between object queries and image features via pure convolutional layers. We abstract the general architecture of a decoder and divide into two components, i.e., Self-Interaction Module (SIM) and Cross- Interaction Module (CIM). In transformer-based models, the SIM and CIM are implemented with multi-head self-attention and cross-attention, while they are built upon our proposed InterConv in our method. The Self-InterCony is stacked with simple depthwise and InterConv mechanism is further carefully designed to perform interaction between object queries and image features via convolutional layers as well as simple upsampling and pooling operations.
convolutions. A novel Cross-
Equipped with the proposed InterConv, we develop Detection ConvNet (DECO), which is simple yet effective query-based end-to-end object detection framework. Our DECO model enjoys the similar favorable attributes as DETR. For example, using the mechanism of object query, our DECO directly obtains a fixed set object predictions and also discards the NMS procedure. Moreover, is stacked with only standard convolutional layers and does not rely on any sophisticated attention modules. To achieve this goal, we first carefully revisit the design of DETR and propose the DECO encoder and decoder architectures shown in Fig. [3] The DECO encoder is built upon ConNeXt blocks and no positional encodings are necessary since ConvNets are variant to input permutation, while the DECO decoder is built up with InterConv.
2

---

Published as a conference paper at ICLR 2025
We evaluate the proposed DECO on the challenging object detection benchmark, i.e., COCO Lin iet al. 2014). Experimental results demonstrate that our DECO achieves competitive performance in terms of detection accuracy and rumning speed, as shown in Fig. | Specifically, with the ResNet-18 and ResNet-50 backbone (He et al. 2016), our DECO achieves 40.5% and 47.8% AP with 66 and 34 FPS, respectively and outperforms the DETR model. Extensive ablation studies are also conducted to provide more discussions and insights about the design choices. We also apply the proposed method into the popular segment anything task. Our DECO-TinySAM obtains quite similar performance and higher efficiency on mobile phone with the TinySAM baseline, demonstrating the effectiveness of our proposed method.
2 RELATED WORK
Object detection is one of the most foundational computer vision task and has attracted large amount of research interest from the computer vision community. The object detection pipeline has been developed rapidly, especially in the era of deep learning. Faster R-CNN (Ren et al. 2015) is one of the most typical two-stage object detectors, which first generates region proposal and extracts regional features for final bounding box prediction. Two-stage detection pipeline has been improve from various aspects (Pang et al. 2019; Cai & Vasconcelos 2018). Mcanwhile, one-stage detectors like SSD (Liu et al.) (0166). YOLO series (Redmon et al. 2016
Object Detection.
2016]]Remmon Redmon & Farhadi (01 7. 2018, (Bochkosskiy et al.) (020), CenterNet (Zhou et 2019 Duan et al. (2019) or FCOS Tian ct al. 2019) simplify the detection pipeline by directly predicting the objects of interest from the
image features (Li et al. 2019 Lu et al. 2019 Zhu et al. 2019 Kong al. 2019 Zhu et al. 2020
Law et al. 2020 Zhang et al. 2020).
Transformer-based End-to-End Detectors. The pioneering work DETR (Carion et al. 2020
utilizes a transformer encoder-decoder architecture and models the object detection as a set prediction problem. It directly predicts fixed number of objects and get rid of the need for hand-designed
non-maximum suppression (NMS) (Neubeck & Van Gool 2006). More follow-up studies (Meng
et al. 2021 Gao et al., [021 Dai et al. [2021]] Wang et al. 2022b have made various optimizations and extensions based on the original DETR and achieve strong detection performance. For example, Deformable DETR (Zhu et al. 202I) only attends to small set of key sampling points by introducing multi-scale deformable
to improve the detection accuracy as well as the training convergence. DAB-DETH (Liu et al. (222a) improves DETR by using box coordinates as queries in decoder. DN-DETR and DINO (Li et al.] 2022 Zhang et 2022) introduce several novel techniques, including query denoising, mixed query selection etc., to achieve strong detection performance. RT- DETR (L.v et al., (023) designs the first real-time end-to-end detector, in which an efficient multi-scale hybrid encoder and an loU-aware query selection are proposed. One of the most important properties for DETR-based detectors is the query-based scheme for producing the final predictions, which streamlines the detection pipeline and make it an end-to-end detector.
atffeross-attenteo
ConyNet-based End-to-End Detectors. Inspired by the success of munsformer-basee detector like DETR variants, several studies also attemppt to remove the post- prot-procsssing NMS by introducing one-to-one assignment strategy (Sun et al., 2021a; a Wang et al. (021a) and set prediction loss (Sun) et al. 2021b). OneNet (Sun et al. a) systemically explores the importance of classification cost in one-to-one matching and applies on typical ConvNet- based detectors like RetinaNet (Lin et al.) 2017) and FCOS (Tian et al. 2019). DeFCN (Wang et al.) (2221a) introduces a new strategy of label assignment enhance the matching cost. Sparse R-CNN (Sun et al., 202Tb) integrates the fixed number of learnable anchor to two-stage detection pipeline. However, it interacts query and Rol feature by the dynamic head which a kind of learnable matrix multiplication.
ConvNets have been demonstrated to have competitive performance on various tasks and are ppployment-friendly in most hardware platforms (L.iu et al. 2022b; Yu et al. (022). Cheng er al.proposed SparseInst (Cheng et al.) 2022). an efficient and fully convolutional framework for real-time instance segmentation. SparseInst utilizes a sparse set of instance activation maps to predict objects in end-to-end style, which could be viewed as alternative for the query-based mechanism of DETR. In this paper we would like to design a DETR-like detection pipeline but built with standard convolutions, which could inherit both the advantages of ConvNets and the favorable properties of the DETR framework. Compared with SparseInst, our method follows the similar query-base mechanism of DETR, and also demonstrates good generalization capability like segment anything models.
3

---

Published as a conference paper at ICLR 2025
d' X
Pooling
FFN
Cross-IntarConu
Cross-lnarscton
Multitiead Cross-Attention
Module (C
9e8 ewcomv
Encoder E Embeddings
Encoder mbeddiings d X
Fision
d H W
o' NiP
Upsample
Self-InterConv
position encoding
Multi-Head Self- Alt--ttaatiin
Sehimeraccoo Module (SIN)
1xl conv
1xl conv
9x9 ewconv
K.A,
objectqueries
objectquenis
(e)
(b) InterCoov
The attention- based decoder and our proposed InterConv. We abstract the general ar- chitecture of decoder and divide it into two components, i.e., Self-Interaction Module (SIM) and Module (CIM). In DETR, the SIM and CIM are implemented with multi-head self-attention and cross-attention mechanism, while in our proposed DECO, the SIM is stacked with simple depthwise and convolutions. We further propose novel CIM mechanism for our DECO to perform interaction between object queries and image features via convolutional layers as well as simple upsampling and pooling operations.
Figure 2:
3 APPROACH
In this section, we first introduce the design of our proposed InterConv, which works similarly with attention mechanism but is simply built with pure convolution. We then provide details about utilizing InterConv develop efficient models for detection and segmentation.
3.1 INTERCONV: BUILDING ATTENTION-LIK MECHANISM WITH CONVOLUTION
Given small set of object queries, the decoder in transformer-based models like DETR or SAM aims to reason the relations of the objects and the global image feature. As shown in Fig. 2 (a), each layer in transformer decoder is mainly composed of a self-interaction module (SIM) and a cross-interaction module (CIM). The SIM in original DETR is a multi-heate self-attention layer and responsible for interacting information between the object queries. The CIM is the essential part for DETR decoder, which consists of cross-attention layers to perform interaction between the image embeddings from the output of encoder and the object queries. In this way, the object queries attend the global image feature and capture the essential information for each predicted objects. In this section, we aim explore how build an attention-lie mechanism with ConvNets while maintaining the capability similar to attention.
Self-InterCom for SIM. Take DETR an example, given N object queries Nxd first reshape the queries to R xh, xd and feed them into convolutional layers. For example, if we have N 100 object queries, the query embeddings are reshaped into 1 10xd, More design choices of reshaping will be discussed in ablation studies. As shown in Fig. (b). the SIM part for DECO decoder quite similar to the design scheme of DECO encoder, where stacking the depthwise convolution and 1 x convolution could lead to strong capability similar to the self-attention mechanism. We utilize a large kernel convolution up to 9 to perform long-range perceptual feature extraction.
Intes-Intrroov for CIM. The CIM mainly takes two features input, e.g., the image feature embeddings from the output of encoder (2g E RdXHxW), and the object query embeddings produced from the SIM part (o Re, xh,, xd). The cross-attention mechanism in DETR decoder allows each object query to interact with the image features to capture necessary information for object prediction. However, using ConvNets to perform such kind of interaction is not intuitive. As shown in
4

---

Published as a conference paper at ICLR 2025
image
predictonn
CNN
Transformrr Ercoder
Transformer
Selt Altention
Crose Alteniion
backbone
objectqueries
[b) Detection C Convitet (DECO)
setof image featuros
setoff
G aethos
CNN
ChNN Encoder
CNV
X gatin
Q
backbone
object quaries
Figure 3: The overall architecture of DETR (Carion et al. 2020 and our proposed Detection ConvNet (DECO). Our DECO is a simple yet effective query-based end-to-end object detection framework and enjoys the similar favorable attributes as DETR. Moreover, it is stacked with only standard convolutional layers and does not rely on any sophisticated attention modules.
Fig. (b), we first upsample the object queries o to obtain â E RdxHx so that has the same size with imgge feature Ze, i.e.,
Ô = Upsample(o).
(1)
There are also other design choices for the upsampling function, i.e., resizing both the object queries and encoder embeddings to a fixed size before fusion, or directly upsampling the object queries to dynamic size of encoder embeddings, which is related to the resolution of input image. We will provide more analysis in experimental section. Then the upsampled object queries and the image feature embeddings are fused together using (usion(.) function, followed by large kernel depthwise convolution (Liu et al. 2023b; Ding et al. 2022 2024) to allow object queries to capture the spatial information from the image feature. An intuitive implementation for Fusion(.) is element-wis. add operation. There are also some alternatives like element- wise multiply operation, or first concatenating two features then using convolution to reduce the number of channels. More discussions will be presented in the ablation studies. The skip connection is all utilized, as shown in the following equation:
o, ô + dwconv (Frsionn( ze)).
The output features further go through another FFN with skip connection. Finally, an adaptive maxpooling is utilized downsample the object queries back to the size of R xh, xd and will be further processed by the following decoder layers.
op = Pooling(â, FFN(),)).
The final output embeddings of the decoder will be fed into the detection head to obtain the class and bounding box prediction, which similar to the original DETR.
3.2 DECO: DETECTION CONVNETS EQUIPPED WITH INTERCONV
Carion al. (Carion al. 2020) proposes the Detection Transformer (DETR) that models object detection as a set prediction problem and directly produces a fixed set of objects. As shown in Fig. (a), DETR first utilizes a backbone extract image features, and feeds them into transformer encoder and decoder architecture. A fixed small set of learned object queries interact with the global image context directly output the final set of object predictions. DETR streamlines the end-to-end object detection pipeline and has attracted great research interest due to the good accuracy and run-time performance (Zhu al. 202T Zhang et al.) 2022, Dai et al. 202Tb a) for object detection. Although transformers have shown great power in computer vision tasks like image classification, object detection, segmentation etc., there are also some recent work that reveal the potential of ConvNet-based architecture the common backbone, e.g., ConvNext (Liu et al.) (022bb) and FonyFormer (Yu et al. 2022). In this work we re-examine the DETR design and explore whether ConyNet-based object detector could inherit the good properties of DETR.
One of the most important properties for DETR-based detectors is the auery-basel scheme for producing the final predictions. In this way, the object detector could directly obtain fixed number of objects and gets rid of any hand-designed NMS post processso....ng. We follow this paradigm to design Detection Netv (DECO), as shown in Fig. (b). DECO also utilizes CNN backbone
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparisons our proposed DECO+ with other detectors on COCO 2017 val set. The FPS is measured on V100 GPU.
Model
Backbone
GFLOPs
FPS
AP
APs0
AP75
AP3
AP
APr.
Faster R-CNN Ren ct al.
2015
R50-FPN
180
26
40.2
61.0
43.8
24.2
43.5
52.0
Faster R-CNN [Ren et al
2015
R101-FPN
246
20
42.0
62.5
45.9
25.2
45.6
54.6
FCOS Tian al.
2019)
R50-FPN
201
23
38.7
57.4
41.8
22.9
42.5
50.1
FCOS Tian et al.
2019)
R101-FPN
277
19
39.1
58.3
42.1
22.7
43.3
50.3
RetinaNct Lin cf al.
2017
R50-FPN
239
21
37.4
56.7
39.6
20.0
40.7
49.7
RetinaNet [Lmn cf al.
2017
R101-FPN
315
17
38.5
57.6
41.0
21.7
42.8
50.4
Sparse R-CNN (Sun
202
R50-FPN
150
23
42.8
61.2
45.7
26.7
44.6
57.6
OneNet-RetinaNeet (Sun
202
R50-FPN
239
21
37.5
55.4
10.7
21.5
10.5
17.4
OneNet-FCOS (Sun |
R50-FPN
206
26
38.9
57.2
42.2
23.9
11.8
19.4
DeFCN Wang et al.
202
R50-FPN
19
41.4
59.5
45.6
26.1
41.9
52.0
YOLOS (Fang
2021
Deiy-Tiny
21
52
28.7
47.2
28.9
9.7
29.2
46.0
YOLOS-S Fang
DeiT-Small
194
36.1
55.7
37.6
15.6
38.3
55.3
YOLOS-B |aang ct al.
2021
DeiT-Base
538
42.0
62.2
44.4
19.5
45.3
62.1
DETR (Carion et al. (2020)
R50
97
28
39.5
60.3
41.4
17.5
43.0
59.1
Deformabie-DDiilk (Zhu ct
2021
R50
173
16
43.8
62.6
47.7
26.4
47.1
58.0
Anchor-DETR (Wang
20226
R50
103
42.1
63.1
44.9
22.3
46.2
60.0
DAB-DETR (Liu ct al. (202.a)
R50
25
42.2
63.1
44.7
21.5
45.7
60.3
DN-DAB-DETR. (L. el al.
2022
R50
94
25
44.1
64.4
46.7
22.9
48.0
63.4
Cond-DETR [Meng et al.
2021
R50
90
28
43.0
64.0
45.7
22.7
46.7
61.5
ViDT (Sogg
2021
Swin-Nano
35
31
40.4
59.6
43.3
23.2
42.5
55.8
DINO-4scalC Zhang
2022
R50
279
12
50.9
69.0
55.3
34.6
54.1
64.6
Stable-DINO (Lu al.
2023a
R50
12
51.5
68.5
56.3
35.2
54.7
66.5
SPeedDETR Dong ct all 2023
R50
21
46.8
66.2
50.4
28.5
50.6
63.2
DDQ-DETR Zhhang et al. 2023
R50
9
52.8
69.9
58.1
37.4
55.7
66.0
Lite-DINO [Li
2023
R50
151
17
50.2
54.6
33.5
53.6
65.5
Co-DETR (Zong et al. 2022)
R50
12
49.5
67.6
54.3
32.4
52.7
63.7
H-DETR (Jia al. 2022)
R50
280
19
50.0
68.3
54.4
32.9
52.7
65.3
Focus-DETR (Zheng al.
2023
R50
154
20
50.4
68.5
55.0
34.0
53.5
64.4
Rank-DETR (Pu 2023
R50
280
19
51.2
68.9
56.2
34.5
54.9
64.9
RT-DETR (Lv al. 2023
R18
40
60
41.7
61.3
45.3
25.0
44.0
56.8
DECO+ (Ours)
R18
66
40.5
58.7
44.0
23.3
43.8
55.7
DECO+ (Ours)
R50
69
34
47.8
67.1
52.4
30.6
51.9
64.2
DECO+ (Ours)
ConvNeXt-B
159
20
50.6
70.7
55.0
34.4
55.3
67.8
to extract features from the input image. Specifically, given a RGB image Timg R3x the backbone generates feature map f E H and usually H, He. 32 32 The feature
Wa.
map f are then go through CNN encoder, i.e. DECO encoder, to obtain the output embeddings fenc E RC. HxW The CNN decoder takes fene as well as a fixed number learned object queries E Nxd input to make final detection prediction via feed forward network (FFN), where d is the size of encoder output embeddings. The detailed architecture of the CNN decoder is elaborated in Sec. 3.1 We utilize the same prediction loss as in DETR, which uses bipartite matching to find paired predicted and ground truth objects.
DECO Encoder. Similar DETR, 1 X convolution is first utilized reduce the channel dimension of from C' to d and obtain a new feature map zo E x In DETR, 20 is fed into stacked transformer encoder layers, which mainly consists of multi-head self-attention (MHSA) and feed-forward network (FFN) to perform spatial and channel information mixing respectively. Recent work such as ConvNeXt (Liu al., 2022b) has demonstrated that using stacked depthwise and pointwise convolutions could achieve comparable performance with Transformers. Therefore, we use the ConvNeXt blocks to build our DECO encoder. Specifically, each DECO encoder layer is stacked with 7 7 depthwise convolution, LayerNorm layer, x convolution, GELU acitvation and another 1 convolution. Besides, DETR, positional encodings are necessary to added to the input of each transformer encoder layer, since the transformer architecture is permutation- invaarinn.... However, the ConvNet architecture permutation-vant variiattiionn so that our DECO encoder layers could get rid of any positional encodings.
DECO+ Equiped with Multi-scale Feature. One limitation original DETR as well as our DECO the lack of multi-scale feature, which is demonstrated to be important for accurate object detection.

---

Published as a conference paper at ICLR 2025
Table 2: Comparisons of DECO with DETR on COCO 2017.
The FPS is measured on a V100 GPU.
Model
Backbone
GFLOPs
FPS
||
AP
APso
APrs
APs
AРм
APr.
DETR
Carionn al.
2020
R34
88
34
31.6
47.6
33.3
13.3
34.1
49.1
DETR Carion
2020
R50
97
28
39.5
60.3
41.4
17.5
43.0
59.1
DETR CCrronn et al.
2020
ConvNeXt-T
104
25
42.1
63.6
44.3
18.8
45.5
62.8
DECO (Ours)
R50
103
35
38.6
58.8
41.1
19.5
43.3
55.0
DECO (Ours)
ConvNeXt-T
110
28
40.8
61.5
43.5
20.5
45.7
58.4
Deformable DETR [huu et al. 2021) utilizes the multi-scale deformable attention module aggregate multi-scale features and obtains an improved performance compared with DETR. To equip DECO with multi-scale feature capability, we utilize the cross-scale feature-fusion module in RT-DETR (Lv et al. 2023) after obtaining the global feature from our DECO encoder. Different from RT-DETR, the features are then mapped the same scale and concatenated along the channels followed by linear projection layer to get the final encoder embedding. We also explore whether the performance can be further improved by utilizing deformable convolution, which detailed in the Appendix. More modern techniques for DETRs would also be compatible with DECO, which we leave for future exploration.
3.3 DECO- TINYSAM: INTERCONV FOR SEGMENT ANYTHING MODEL
The mechanism of InterConv could have great potential for applying to other architectures and tasks, especially for those involving interaction across domains. As a proof of concept, replace the mask decoder in Segment Anything Model (SAM) with our DECO decoder. The motivation is that as a prompt-based segmentation model, SAM interacts between prompt tokens and image embeddings through the mask decoder which consists of self- and cross-attentions sharing similar spirits of our proposed SIM and CIM. More specifically, we repeat the one-dimensional queries into two dimensional features, instead of reshaping queries as in object detection task discussed above. It mainly due the fact that the query tokens SAM contains learnable output tokens and a few prompt tokens, which are not reasonable to simply be reshaped into two dimensions. We utilize TinySAM (Shu et al., 2023) as the baseline model and replace the decoder to our proposed DECO architecture.
EXPERIMENTS
In this section, we first evaluate our proposed model on object detection benchmark and compare it against tate-of-thh-arr methods. Extensive ablation studies are also conducted to provide more discussions and insights about the design choices.
4.1 EXPERIMENTAL SETTING
For the vanilla DECO, we follow similar training settings as DETR (Carion et al. 2020). We train the proposed DECO models for 150 epochs using AdamW optimizer, with weight decay of 10 and initial learning rates as 10-4 and 10-5 for the encoder-decoder and backbone, respectively. The learning rate is dropped by factor of 10 after 100 epochs. The augmentation scheme is the same as DETR, which includes random horizontal flipping, random crop augmentation, and scale augmentation. The input image shorter side is resized to random size between 480 and 800 pixels in the scale augmentation while restricting the longer size to at most 1333, As to DECO+ that equipped with multi-scale feature fusion, the training image size is selected between 480 and 800 with 32 stride following the RT-DETR baseline. The inference size is set to 640 x 640.
4.2 COMPARISONS WITH STATE-OF-TIIB-ARIE
We evaluate the proposed DECO and DECO+ on COCO benchmark and compare with recent competitive object detectors, including DETR (Carion et al., 2020). YOLOS (Fang et al. (021]. FCOS (Tian et al. 2019), and DETR variants with strong performance, e.g., Anchor- (Sun et al. 2021b). Conditional-DEr DETR (Sun et al. 2021a) and ViDT. (Wang et al. 2021a) etc.. Experimental results in terms of detection AP and FLOOPs/PPS shown in Table
The FPS report is the
7

---

Published as a conference paper at ICLR 2025
Fusion Method
GFLOPS
AP
#layers
GFLOPS
FPS
AP
wise Mult.
103
37.8
5
101
35
38.3
Concat-Con
106
38.6
6
103
35
38.6 6
Element-wise Add
103
38.6
7
105
34
38.9
Table 4: Effect of number of layers in decoder. Table 3: Effect of different fusion methods.
Table 5: Ablation studies for different kernel sizes in decoder.
kernel size
5x5
7x7
9x9
11x11
13x 13
15x 15
AP (%)
37.8
37.9
38.6
38.6 103.70
38.4
38.6 104.27
GFLOPs
103.35
103.47
103.58
104.11
average number of the first 100 images in the COCO 2017 val set on a NVIDIA V100 GPU. The FLOPs are computed with the input size of (640, 640) for RT-DETR and the proposed DECO+, while (1280, 800) for others. more intuitive comparison the trade-off between AP and latency is also shown in Fig.
Comparisons with DETR variants. We consider different ConvNet- based backbones for DECO+ in the benchmarking. As shown in Table our DECO+ with ResNet-50 Backbone achieves 47.8% AP with 34 FPS on V100 GPU, which is better than most previous DETR variants considering the higher AP at 50.6% which is 0.2% higher than Cocus-DETK (Zheng et al. 2023) at the same FPS. Moreover, ResNet-18 based DECO+ obtains 40.5% AP with 66 FPS, achieving quite similar performance with variant of RT-DETR (Lv et al.) (023) that we modified to not use deformable attention and denoising training for fair comparison. Note that deformable attention and denoising training specifically designed for attention- based architecture, and similar improved strategies for DECO still remains for future exploration.
accuracy-latency trade-off. The Xo (Liu et al. 2022h) based DECO+ achieves
Comparisons with Other End-to-End Detectors. YOLOS is an encoder-only Transformer archi- tecture for object detection based on the vanilla pre-trained vision transformers. Our DECO+ models show clear advantage over YOLOS and have similar detection performance while running much faster. We also compare our DECO+ with recent end-to-end detectors with ConvNets, e.g., Sparse R-CNN (Sun et al. 20216), OneNet (Sun et al. 2021a) and DeFCN. (Wang al.) 2021a). As shown in Table I our DECO+ outperforms Sparse R-CNN (Sun et al. 2021b) and OneNet-Retinale
(Sun)
eet al. 202 with better accuracy and rumning speed. Similarly. DECO+ obtains 47.8% AP and 34 FPS while DeFCN (Wang et al. 2021a) only has 19 FPS with 41.4% AP.
Comparisons with DETR. We compare the performance of our vanilla DECO and DETR (Car ion et al. 2020) equipped with ResNet and ConvNeXt-Tiin in Table 2 The proposed DECO encoder/decoder are adopted to substitute the transformer encoder/decoder DETR for comparison. To align the FLOPs with DETR, we modify the DECO encoder to be three stages with the number of blocks of (2, 6, 2) and the channel dimension of (120, 240, 480). respectively. As shown in Table 2 for both ResNet-50 and ConvNeXt- Tiny (Liu et al. 2022b) backbones, despite higher FLOPs. our DECO obtain faster inference speed (FPS) than DETR. demonstrates that our pure ConyNet-based architecture is more deployment-friendly than the transformer-
DETR in GPU platform. Specif- ically, our DECO obtains 38.6% AP at 35 FPS and is 7.0 7.0% AP better than DETR with ResNet-34 backbone for similar running speed.
4.3 ABLATION STUDIES
We conduct ablation studies based on the R50-based DECO in Table and insights about different design choices and justify the effectiveness of our proposed method.
2 to provide more discussions
Upsampling Size in CTM. In CIM, the object queries are first upsampled and then fused with the encoder embeddings to deal with different dimensions. Here we have different design choices, i.e., resizing both the object queries and encoder embeddings to fixed size before fusion, directly upsampling the object queries to dynamic size of encoder embeddings, which is related to the resolution of input image. As shown in Table utilizing dynamic size achieves the best performance, since it is more flexible for different input resolution and has no information discarding. Noted that (25 38) is the average size of COCO training set and it leads to 0.5 AP drop than dynamic way.
8

---

Published as a conference paper at ICLR 2025
Size
GFLOPs
AP
#queries
Query Shape (w, x lt))
Ratio
AP
(20 x 20)
97
34.8
10 X 10
38.6
(25 x 38)
103
38.1
100
20 x
4:1
38.3
(40 x 40)
110
37.2
100
5 20
1:4
36.9
Dynamic
103
300
30 x 10
3:1
38,9
300
20 x 15
4:3
38.8
Table 6: Effect of different upsampling
size of object queries in CIM.
Table 7: Effect of different shape of queries.
Figure 4: Visualizations for box prompted segment anything for Our DECO-TinySAM (1 row) and TinySAM (2nd row). Our method obtains quite similar performance with TinySAM.
more number of layers decoder tends to have better performance. However, utilizing 6 decoder layers is good choice balance performance and computational cost and we keep this choice following DETR (Carion et al. (020).
Number of Decoder Layers. As shown in Table 4
Kernel size in decoder. The motivation for using 9 x dwconv is to enable sufficient receptive field. We conduct ablation experiments to explore the effect of different kernel sizes. As shown Table 5 using 5 x has unsatisfied performance due limited receptive field, and enlarging kernel size to 11 X 11 or even 15 x 15 brings negligible improvement.
Different design choices of fusion method in CIM. As discussed in Section 3 the upsampled object queries and the image feature embeddings are fused together using add operations. Here we conduct ablation studies for other design choices of the fusion method, e.g., using concatenation and convolution, or simple conducting element- wwie multiplication for fusion. As shown in Table 3 utilizing element-wise multiplication to fuse the object queries and the image feature embeddings does not obtain better performance. Moreover, using add operations achieves similar detection performance with using concatenation and convolution, but has slightly smaller FLOPSs
Different shapes of object queries. In our proposed method, the object queries should be in 2D shape of Wi x h, and there are several choices of query shape for N object queries. For example, W, x hú could be 10 x 10, 20 5 or 5 x 20 for N 100 queries. As shown in Table 7 using 10 10 obtains better detection performance. When N = 300, using query shape of 30 x 10 achieves slightly better performance than 20 x 15. A typical ratio of image size for COCO could be considered as 1333 800 ze 1.67 and we could conclude from Table 7 that better performance is obtained when the query shape is approximately the ratio of input image.
4.4 EXTENSION TO SEGMENT ANYTHING TASK
Our proposed DECO is original designed for object detection. However, the mech- anism DECO decoder could have great potential for applying to other architectures and tasks, especially for those involving interaction across domains. As proof of concept, we replace the mask decoder Segment Anything Model (SAM) with DECO decoder. The motivation is that prompt-based segmentation model, SAM interacts between prompt tokens and image embeddings through the mask decoder which consists of self- and cross-attentions, sharing similar spirits of proposed self-InterCony and cross- Interrovnr More specifically, we repeat the one-dimensional queries into two dimensional features, instead of reshaping queries in object detection task discussed above.
Method
COCO AP (obil Lat. (ms)
TinySAM (Shu et al. DECO-TinySAM
2023
41.9
34
41.4
29
Table Zero-shot instance segmentation results for TinySAM baseline and DECO-TinySAM.

---

Published as a conference paper at ICLR 2025
[al Visualization of box predictions DETR
[bl Visualzaaion of all box predictions in DECO
Figure 5: Visualizations of query slots for DETR and our DECO. We can find that both DECO and DETR tend to make different object queries focus different patterns in terms of spatial areas and box sizes. Interestingly, we could also observe that the slots of DETR for small objects are spatially unordered while the distributions of each slot for our DECO are spatially ordered for small boxes.
mainly due to the fact that the query tokens in SAM contains learnable output tokens and a few prompt tokens, which are not reasonable to be simply reshaped into two dimensions.
We utilize TAM (Shu et al. 2023) as the baseline model and replace the decoder to our proposed InterConv architecture. The results for zero-shot instance segmentation prompted by detected boxes on COCO dataset are shown Table g Despite without carefully tuning, our DECO-TinySAA obtains quite similar zero-shot instance segmentation performance with that of TinySAM baseline, with lower latency on mobile devices. We also provide some visualizations for box prompted segment anything for our DECO-TI DECO-TinySAAL TimySAM and the TinySAM bascline, demonstrating strong performance and generalizability to other tasks of our method.
4.5 VISUALIZATTION
Visualization of Query Slots. Following the same method in DETR, we visualize the boxes predicted by 20 out of total 100 query slots of our DECO. Each point represents one bounding box prediction and the coordinates are normalized by each image size. Different colors indicate objects with different scales, e.g., green, red and blue refer to small boxes, large horizontal boxes and large vertical boxes, respectively. As shown in Fig. 5 we can find that both DECO and DETR tend to make different object queries to focus on different patterns in terms of spatial areas and box sizes. Interestingly, we could also observe that the slots of DETR for small objects are spatially umordered, which indicates that the prediction of each slot is random in spatial dimension. However, things are bit different for our DECO, whose distributions of each slot are spatially ordered for small boxcs. This observation is most likely be related the intt--inteaaction mechanism of object queries and image features, where cross-attention module tends to capture global information and our proposed module tends to focus on local interaction through large kernel convolutions.
CONCLUSION AND DISCUSSION
In this paper, we aim explore whether we could build query-based detection and segmentation framework with ConvNets instead of sophisticated transformer architecture. We propose a novel mechanism dubbed InterConv to perform interaction between object queries and image features via convolutional layers. Equipped with the proposed InterConv, we build Detection ConvNet (DECO), which is composed of a backbone and convolutional encoder-decoder architecture. We compare the proposed DECO against prior detectors on the challenging COCO benchmark. Despite its simplicity, our DECO achieves competitive performance Specifically, with the ResNet-18 18 and ResNet-50 backbone, our DECO achieves 40.5% and 47.8% AP with 66 and 34 FPS, respectively. The proposed method is also evaluated on the segment anything task, demonstrating similar performance and higher efficiency. We hope the proposed method brings another perspective for designing architectures for vision tasks.
terms of detection accuracy and running speed.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alexey Bochkovskiy, Chien-Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. CoRR, abs/2004. 2020.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-enm: Delving into high quality object detection. In CVPR, pp. 6154-6162, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, pp. 213-229. Springer, 2020.
Tianheng Cheng, Xinggang Wang, Shaoyu Chen, Wengiang Zhang, Qian Zhang, Chang Huang, Zhaoxiang Zhang, and Wenyu Liu. Sparse instance activation for real- instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
4433 4442, 2022.
Xiyang Dai, Yinpeng Chen, Jianwei Yang. Pengchuan Zhang, Lu Yuan, and Lei Zhang. Dynamic detr: End-to-end object detection with dynamic attention. In ICCV, pp. 2988-2997. 2021a.
Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. In CVPR, pp. 1601-1610, 2021b.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in enns. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1196--11755. 2022.
Xiaohan Ding, Yiyuan Zhang, Yixiao Ge, Sijie Zhao. Lin Song. Xiangyu Yue, and Ying Shan. Unireplknet: A universal perception large-kernel convnet for audio video point cloud time-series and image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5513-5524, 2024.
Peiyan Dong, Zhenglun Kong, Xin Meng, Peng Zhang, Hao Tang, Yanzhi Wang, and Chih- Chou. Speeddetr: Speed-aware transformers for end-to-end object detection. In International Conference on Machine Learning, pp. 8227-8243. PMLR, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In ICCV, 2019.
Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. NeurIPS,
2021.
Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention. In ICCV, pp. 3621-3630, 2021.
Shang-Hur Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin- Yu Zhang, Ming- Hsuan Yang, and Philip Torr. Res2net: A new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence, 43(2):652-662, 2019.
Jianyuan Guo, Kai Han, Yunhe Wang, Chao Zhang, Zhaohui Yang, Han Wu, Xinghao Chen, and Chang Xu. Hit-detector: Hierarchical trinity architecture search for object detection. In CVPR, pp. 11404-11414, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. arXiv preprint 2.t.:2277..38880 2022.
Tao Kong, Fuchun Sun, Huaping Liu. Yuning Jiang, and Jianbo Shi. Foveabox: Beyond inchor-basce object detector. CoRR, abs/1 904.03797, 2019.
11

---

Published as a conference paper at ICLR 2025
Hei Law, Yun Teng, Olga Russakovsky, and Jia Deng. Comemet-lite: Efficient keypoint based object detection. In BMVC. BMVA Press, 2020. httpppaewww..bbbe222200oooooeeee
hatm//assett/pppee//// pdf
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. arXiv preprini arXiv:2203.01 1tt.:22030033,,,, 2022.
Feng Li, Ailing Zeng, Shilong Liu, Hao Zhang, Hongyang Li, Lei Zhang, and Lionel M Ni. Lite detr:
An interleaved multi-scale encoder for efficient detr. arXiv preprint arXiv:2303.07335, 2023,
Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Scale-aware trident networks for
object detection. In ICCV, pp. 6054-6063, 2019.
Zeming Li, Chao Peng. Gang Yu, Xiangyu Zhang, Yangdong Deng, and Jian Sun. Detnet: Design backbone for object detection. In ECCV, pp. 334-350, 2018.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014.
Tsung-Y. Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In ICCV, pp. 2980-2988, 2017.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. In ICLR, 2022a.
Shilong Liu, Tianhe Ren, Jiay Chen, Zhaoyang Zeng, Hao Zhang, Feng Li, Hongyang Li, Jun Huang, Hang Su, Jun Zhu, et al. Detection transformer with stable matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6491-6500, 2023a.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Tommi Kärkkainen, Mykola Pechenizkiy, Decebal C Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. In International Conference on Learning Repre- sentations, ICLR 2023, 2023b.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: single shot multibox detector. In ECCV, 2016.
Yudong Liu, Yongtao Wang, Siwei Wang, TingTing Liang, Qijie Zhao, Zhi Tang, and Haibin Ling. Cbnet: A novel composite backbone network architecture for object detection. In AAAI, volume 34, pp. 11653-11660, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp- 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, pp. 11976-11986, 2022b.
Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan. Grid R-CNN. In CVPR, 2019.
Wenyu Lx, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingqing Dang, and Yi Liu. Detrs beat yolos on real-time object detection. arXiv preprint arXiv:2304.08069, 2023.
Depu Meng. Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In /CCV, 2021.
Alexander Neubeck and Luc Van Gool. Efficient non- am-mmaimum suppression. In ICPR, volume 3, pp. 850-855, 2006.
Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin. Libra R-CNN: towards balanced learning for object detection. In CVPR, 2019.
12

---

Published as a conference paper at ICLR 2025
Yifan Pu, Weicong Liang, Yiduo Hao, Yuhui Yuan, Yukang Yang, Chao Zhang, Han Hu, and Gao
Huang. Rank-detr for high quality object detection. In NeurlPS, 2023.
Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. In CVPR, 2017.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. CoRR, abs/ 804.02767, 2018.
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. NeurIPS, 28, 2015.
Han Shu, Wenshuo Li, Yehui Tang, Yiman Zhang, Yihao Chen, Houqiang Li, Yunhe Wang, and Xinghao Chen. Tinysam: Pushing the envelope for efficient segment anything model. arXiv 2023.
preprint arXiv:2312.
Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varun Jampani, Dongyoon Han, Byeongho Heo, Wonjae Kim, and Ming-Hsuan Yang. Vidt: An efficient and effective fully wansformer-based object detector. In ICLR, 2021.
Peize Sun, Yi Jiang, Enze Xie, Wenqi Shao, Zehuan Yuan, Changhu Wang, and Ping Luo. What makes for end-to-end object detection? ICML, pp. 9934-9944. PMLR, 2021a.
Peize Sun, Rufeng Zhang. Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka. Lei Li, Zehuan Yuan, Changhu Wang, al. Sparse r-cnn: End-to-end object detection with learnable proposals. In CVPR, pp. 14454-1444 14463, 2021b.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In ICCV, pp. 9627-9636, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. In ICMI., pp- 10347-10357. PMLR, 2021.
Fengbin Tu, Zihan Wu, Yiqi Wang, Ling Liang. Liu Liu, Yufei Ding. Leibo Liu, Shaojun Wei. Yuan Xie, and Shouyi Yin. Trancim: Full-digital bitline-transpose cim-based sparse transformer accelerator with pipeline/paralle reconfigurable modes. IEEE Journal of Solid-State Circuits, 2022.
Jianfeng Wang, Lin Song, Zeming Li, Hongbin Sun, Jian Sun, and Nanning Zheng. End-to-end object detection with fully convolutional network. In CVPR, pp. 15849-15858, 2021a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, pp. 568-578, 2021b.
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowci Hu, Tong Lu, Lewei Lu, Hongsheng Li, al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. arXiv preprint arXiv:2211.081 105778, 2022a.
Yingming Wang. Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based detector. In AAAI, volume 36, pp. 2567-2575, 2022b.
Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng. Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. arXiv preprint arXiv:2210. 13452, 2022.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Harry Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In ICLR, 2022.
Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li. Bridging the gap between anchor-basee and anchor-free detection via adaptive training sample selection. In CVPR, 2020.
13

---

Published as a conference paper at ICLR 2025
Shilong Zhang, Xinjiang Wang, Jiaqi Wang, Jiangmiao Pang, Chengqi Lyu, Wenwei Zhang, Ping Luo. and Kai Chen. Dense distinct query for end-to-end object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7329-7338, 2023.
Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, and Yunhe Wang. Less more: Focus attention for efficient detr, 2023.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. CoRR, abs/ 1904.08000. 2019. http::///aaxxi wttp////////a.......//// 07850
Chenchen Zhu, Yihui He, and Marios Savvides. Feature selective anchor-frec module for single-shot object detection. In CVPR, 2019.
Chenchen Zhu, Fangyi Chen, Zhiqiang Shen, and Marios Savvides. Soft anchor-point dettcttii In ECCV. 2020. doi: 10.1007/978-33 3-00----000000---- 6. URL UBL/atpps.//00...0//01000.01 978 3- 030 58545-7
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training, 2022.
14


Reviews:
[{"review_id": "Qm87p9906F", "reviewer": "Reviewer_8onY", "summary": "The paper proposes an InterConv mechanism to facilitate interaction between object queries and image features via a convolutional layer, resulting in the Detection ConvNet (DECO). DECO is composed of a backbone and a convolutional encoder-decoder architecture. Results on COCO detection demonstrate that DECO achieves a decent accuracy-speed trade-off.", "questions": "Please help to address the clarity issues.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The DECO method requires only standard convolutions, making it computationally efficient and more compatible with low-end AI chips compared to DETR-based methods.\n\n- The designs of self-interconv and cross-interconv are extremely simple and effective, indicating that the paper captures the essence of DETR and presents an elegant pure-conv replacement.", "weaknesses": "- Clarity: In the method section, the Fusion operation is crucial; however, the paper fails to discuss it adequately. In the experiment section, the paper presents different fusion methods and query map upsampling strategies, but they remain unclear. These important details should be clearly stated and explained in the method section.\n\n- The paper is missing some important references, such as the large kernel CNN paper and the SparseInst paper, which is also a pure-conv query-based object detection/segmentation method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CyBKNQt4bO", "reviewer": "Reviewer_yhFU", "summary": "This paper proposes a query-based end-to-end Encoder-Decoder architecture, i.e., DECO with pure CNN modules for object detection, built upon a mechanism dubbed InterConv for the interaction between object queries and image features. Experimental results show that the proposed method without complicated components achieves competitive performance and efficiency over the DETR series.", "questions": "1. Can the denoising training technique in DN-DETR and DINO be introduced into convolutional-based DECO?\n2. To further improve the performance, it may be practical to introduce the deformable convolution (v1-v4) into the encoder and decoder. What about the efficiency?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The proposed InterConv consisting of pure convolutional layers can help the encoder and decoder capture long-range details without attention mechanisms and also help the decoder interact with object queries.\n2. Without many complicated designs, DECO achieves comparative performance and efficiency over DETR.\n3. DECO can be further improved to DECO+ by introducing the multi-scale design from Deformable-DETR.", "weaknesses": "1. On Lines 074-075, within the scope of object detection, the statement ``whose weights and inputs are both generated during runtime'' is unclear, as the only ones dynamically generated are intermediate variables (including attention weights).\n2. The performance can be further improved using more convolutional-based techniques like deformable convolutions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G3YTi4oe4Y", "reviewer": "Reviewer_wv8n", "summary": "The paper presents a new approach to object detection and segmentation using Convolutional Neural Networks (ConvNets), challenging the dominance of transformer-based architectures in these fields. This paper introduces InterConv, a mechanism that mimics the interaction between object queries and image features, traditionally handled by attention mechanisms in transformers, but using only convolutional layers. This paper presents Detection ConvNet (DECO), an end-to-end object detection framework that replaces transformers with ConvNets. The proposed DECO obtains comparable results compared to transformer-based methods.", "questions": "1. I'm concerned about the initialization of the object queries. Could the object queries be random initialized variables for different images?\n2. In Sec. 4.4, does this paper combine DECO with TinySAM?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents InterConv which mimics attention mechanism with convolutional layers, and formulates Self-Interaction Module and Cross-Interaction Module (CIM) as self-attention and cross-attention.\n2. This paper explores a convolution-based DETR framework with the proposed InterConv.\n3. The proposed DECO obtains comparable performance compared to DETR.", "weaknesses": "1. Although this paper presents the convolution-version DETR which replaces attentions with convolutions, the overall performance is inferior to recent transformer-based methods, such as DINO, Co-DETR, and Stable-DINO[1], showing the limitations of using convolutional architectures.\n2. In Tab. 2, it shows that DETR achieves better performance than DECO on both R50 and ConvNeXt-T while the improvements in inference speeds are not significant. In addition, DETR can be further optimized for acceleration therefore obtaining better inference speed and accuracy. Hence, the effectiveness of the proposed method in this paper is rather limited.\n3. This paper lacks experimental comparisons with many recent works, such as Stable DINO[1], RT-DETRv2[2], DDQ[3], and SpeedDETR[4], in terms of both accuracy and speed.\n4. It's hard to transfer the well-established techniques of recent DETR variants, such as denoising queries and deformable attention, into DECO for better performance.\n5. The proposed object queries require prior knowledge to determine the best shape/layout, for example, 30x10 queries for the COCO dataset. I'm very concerned about whether the pre-trained detector DECO can perform well on the other datasets with different aspect ratios, such as datasets with 1:3 aspect ratios.\n6. The details of extending DECO to segmentation tasks are unclear.\n\n\n\nReferences\\\n[1] Liu et al. Stable-DINO: Detection Transformer with Stable Matching. ICCV 2023.\\\n[2] Lv et al. RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer. arXiv 2024.\\\n[3] Zhang et al. Dense Distinct Query for End-to-End Object Detection. CVPR 2023.\\\n[4] Dong et al. SpeedDETR: Speed-aware Transformers for End-to-end Object Detection. ICML 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XWsDuMM6AN", "reviewer": "Reviewer_3qSX", "summary": "The authors introduced a CNN-based end-to-end detection architecture, DECO. DECO contains an encoder and a decoder with pure convolutions nad is easy to implement. The experimental results demonstrate the effectiveness.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well-written and easy to follow.\n+ Experiments are strong, demonstrating the effectivenss of CNN-based model.", "weaknesses": "+  My main concern is the use of cross-scale feature-fusion module which is proposed in RT-DETR, which weakens the contribution. It would make the paper more strong if the authors could further discuss the difference.\n+ There are many recent studies speeding up transformers, it would be nice to discuss more on using convolutions. Claims in L073-L074 are not convincing to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)