GiraffeDet: A Heavy-Neck Paradigm for Object Detection
Authors: yiqi jiang
Zhiyu Tan
Junyan Wang
Xiuyu Sun
Ming Lin
Hao Li
Venue: ICLR 2022 Poster (2022)
Keywords: Object Detection
fpn
space-to-depth
representation

Abstract: In conventional object detection frameworks, a backbone body inherited from image recognition models extracts deep latent features and then a neck module fuses these latent features to capture information at different scales. As the resolution in object detection is much larger than in image recognition, the computational cost of the backbone often dominates the total inference cost. This heavy-backbone design paradigm is mostly due to the historical legacy when transferring image recognition models to object detection rather than an end-to-end optimized design for object detection. In this work, we show that such  paradigm indeed leads to sub-optimal object detection models. To this end, we propose a novel heavy-neck paradigm, GiraffeDet, a giraffe-like network for efficient object detection. The GiraffeDet uses an extremely lightweight backbone and a very deep and large neck module which encourages dense information exchange among different spatial scales as well as different levels of latent semantics simultaneously. This design paradigm allows detectors to process the high-level semantic information and low-level spatial information at the same priority even in the early stage of the network, making it more effective in detection tasks.  Numerical evaluations on multiple popular object detection benchmarks show that GiraffeDet consistently outperforms previous SOTA models across a wide spectrum of resource constraints. The source code is available at
https://github.com/jyqi/GiraffeDet.

Published as a conference paper at ICLR 2022
GIRAFFEDET: A HEAVY-NECK PARADIGM FOR OBJECT DETECTION
Yiqi Jiang, Zhiyu Tan, Junyan Wang, Xiuyu Sun', Ming Lin, Hao Li DAMO Academy, Alibaba Group
(yiqi. -yg,, zhiyu. ttzzy,, wang waapunyannwiyyaaaa..... 0a waabbbaaanneeeeeeea com (xiuyu. sxy, ming. 1, lihao. whwphhaaaaaaaaiihaaaaaaaaaaaaaaa
ABSTRACT
In conventional object detection frameworks, backbone body inherited from im- age recognition models extracts deep latent features and then neck module fuses these latent features to capture information at different scales. As the resolution in object detection much larger than in image recognition, the computational cost of the backbone often dominates the total inference cost. This heavy-backbone backbone design paradigm is mostly due to the historical legacy when transferring image recognition models to object detection rather than an end-to-end optimized design for object detection. In this work, we show that such paradigm indeed leads to sub-optimal object detection models. To this end, we propose novel heavy-neck paradigm, GiraffeDet, a giraffe-like network for efficient object detection. The Gi- raffeDet uses an extremely lightweight backbone and a very deep and large neck module which encourages dense information exchange among different spatial scales as well as different levels of latent semantics simultaneously. This design paradigm allows detectors to process the high-level semantic information and low- level spatial information at the same priority even in the early stage of the network, making it more effective in detection tasks. Numerical evaluations multiple popular object detection benchmarks show that GiraffeDet consistently outper- forms previous SOTA models across wide spectrum of resource constraints. The source code is available at wtthtts::// http::::////tubbb//p hww.gtttt.oooooo...ooooooooe.t.t wttp:h/w...oo/oeeeaaaaaaaaaaaaea
INTRODUCTION
In the past few years, remarkable progress in deep learning based object detection methods has been witnessed. Despite object detection networks getting more powerful by different designing on architecture, training strategy and so on, the meta-goal that detecting all objects with large-scale variation has not been changed. For example, the scale of the smallest and largest 10% of object instances in COCO dataset is 0.024 and 0.472 respectively (Singh & Davis 2018). which scaling in almost times. This presents an extreme challenge to handle such a large-scale variation by using recent approaches. To this end, we aim to tackle this problem by designing scale robust approach.
To alleviate the problem arising from large-scale variations, an intuitive way is use multi-scale pyramid strategy for both training and testing. The work of (Singh & Davis 2018) trains and tests detectors on the same scales of an image pyramid, and selectively back- procpppppssatess the gradients of object instances of different sizes as function of the image scale. Although this approach improves the detection performance of most existing CNN-based methods, it is not very practical, as the image pyramid methods process every scale image, which could be computationally expensive. Moreover, the scale of objects between classification and detection datasets remains another challenge in do- main shift when using pre-trained classification backbones.
Alternatively, the feature pyramid network is proposed to approximate image pyramids with lower computational costs. Recent methods still rely on superior backbone designing, but insufficient information exchange between high-level features and low-level features. For example, some work enhances the entire feature hierarchy with accurate localization signals in lower layers by bottom-up path augmentation, however this bottom-up up path design might lack exchange between high-level semantic information and low-level spatial information. According to the above challenges, two questions in this task are raised as follows:
"Corresponding author

---

Published as a conference paper at ICLR 2022
Is the backbone of the image classification task indispensable in a detection model?
What types of multi-scale representations are effective for detection tasks?
These two questions motivate to design new framework with two sub-tasks two i.e., efficient feature down-sampling and sufficient multi-scale fusion. First, conventional backbones for scale- sensitive features generation are computationally expensive and exist domain-shift problem. An al- ternative lightweight backbone can solve these problems. Second, it is crucial for a detector to learn sufficient fused information between high-level semantic and low-level spatial features. According to the above motivations, we design a giraffe-like network, named as GiraffeDet, with the follow- ing insights: (1) An alternative lightweight backbone can extract multi-scale feature transformation without any additional computation costs. (2) A sufficient cross-scale connection, Queen- Fusion, like the Queen Piece pathway in chess, to deal with different levels and layers of feature fusion. (3) According to the designed lightweight backbone and flexible FPN, we propose a GiraffeDet family for each FLOOPs level. Notably, the experimental results suggest that our GiraffeDet family achieves higher accuracy and better efficiency in each FLOPs level.
In summary, the key contributions of our work as follows:
To the best of our knowledge, we present the first lightweight alternative backbone and flexible FPN combined as a detector. The proposed GiraffeDet family consists of Lightweight S2D-chain and Generalized-FPMy which demonstrates the state-of teftthe-att performance.
We design the lightweight space-to-depth chain (S2D-chain) instead of the conventional CNN- based backbone, and controlled experiments demonstrate that FPN is more crucial than conventional backbones in the object detection mode.
our proposed Generalized-FPN (GFPN), novel queen-fusion is proposed our cross-scale connection style that fuses both level features in previous and current layers, and log,n skip-layer link provides more effective information transmission that can scale into deeper networks.
Based on the light backbone and heavy neck paradigm, the GiraffeDet family models perform well in a wide range of FLOPs -OPs-performance trade-offs. In particular, with the multi-scale testing technique, GiraffeDet- D29 achieves 54.1% 1% mAP on the COCO dataset and outperforms other SOTA methods.
2 RELATED WORK
It is crucial for the object detector to recogmize and localize objects by learning scale-sensitie fea- tures. Traditional solutions for the large-scale variation problem mainly based on improved convo- lutional neural networks. CNN-based object detectors are mainly categorized by two-stage detectors
and one-stage detectors. Two-stage detectors (Ren et af.) 2015 5 Dai al. 2016 He et al. 2017 Cai
& Vasconcelos 2018 Pang et al. (019) predict region proposals and then refine them by a sub-
network, and one-stage detectors (Liu et al. 2016 Lin et al. 2017bb Redmon et al. 2016 Redmon & Farbadi 2017: Tan et al. 2019 Tian et al. 2019 Zhu et al.) [2019] Zhang et al. [020 2019; Ge
jtt al. (021) directly detecting bounding-bood boxes without the proposal generation step. In this work, mainly conduct experiments based on one-stage detector methods.
Recently, the main research line is utilizing pyramid strategy, including image pyramid and feature pyramid. The image pyramid strategy is used for detecting instances by scaling images. For ample, SNIPER (Singh et al. 2018) propose a fast multi-scale training method, which samples the foreground regions around ground-truth object and background regions for different scale training. Unlike image pyramid methods, feature pyramid methods Lin et al. (2017a); Liu et al. (201 8); Chen al. (019a); Tan et al. 2020))(Sun et al. (202T) fuse pyramidal representations that cross different scales and different semantic information layers. For instance, PANet (Liu et al., 2018) enhances the feature hierarchies on top of feature pyramid network by additional bottom-up path augmenta- tion. Our work focuses on feature pyramid strategy and proposes sufficient high-level semantic and low-level spatial information fusion method.
Some researchers start working on designing new architectures to solve the large-scale variation problem instead of "backboneeneeck-hedd"" architecture in detection tasks. The work of Sun et al. (019b) proposed the FishNet as an encoder-decoder architecture with skip connections to fuse multi-scale features. SpineNet (Du et al. 2020) designed as a backbone with scale- mereuteer inter- mediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search. Our work is inspired by these methods and proposes a lightweight space-to- depth backbone instead of CNN-based backbone. However, GiraffeDet still designed as the
2

---

Published as a conference paper at ICLR 2022
S2D-chiain
Generalized-IPN
Py
Box Predict Network
S20 Block
Pa
520 Blckk
P5
$20 Block
1x1
Predict Network
520 Block
Orea
P
Care
P
Glaffe Body
Giaffe Neck
Giraffe Head--att
Figure I: Overview of the GiraffeDet which has three parts: 1) Body contains image preprocessing and lightweight S2D-chain: 2) Heavy neck refines and fuses high-level semantic and low-level spatial features; 3) Head predicts the bounding box and class label of exist objects.
Beackbone-neck-heae head" architecture. Because this typical architecture is widely used and proved effective in detection tasks.
THE GIRAFFEDET
Although extensive research has been carried out to investigate efficient object detection, large-scale variation still remains challenge. To achieve the goal of sufficient multi-scale information exchange efficiently, we proposed the GiraffeDet for efficient object detection, and the "giraffe" consists of lightweight space-to-depth chain, generalized-FPN and prediction networks. The overall framework shown in Figure [ which largly follows the one-stage detectors paradigm.
3.1
LIGHTWEIGHT SPACE-TO-DEPTH CHAIN
Most feature pyramid networks apply conventional CNN- networks the backbone to extract multi-scale feature maps and even learn information exchange. However, recent backbones became much heavier with the development of CNN, it is computationally expensive to utilize them. More- over, most recent applied backbones are mainly pre-trained on classification dataset, e.g., ResNe150 pre-trained on ImageNet, and we argue these pre-trained backbones are inappropriate in detection task and remains the domain-shiff issue. Alternatively, FPN more emphasis on high-level semantic and low-level spatial information exchange. Therefore, we assume that FPN is more crucial than conventional backbones in the object detection model.
Inspired by (Shi et al.) 2016 Sajjadi al. 2018 we propose Space-to-Deple Chain (S2D Chain)
lightweight backbone, which includes 3x3 convolution networks and stacked S2D blocks. Coneretely, 3x3 convolutions are used for ini- tial down-sampling and introduce more non-linear transformations. Each S2D block consists of S2D layer and 1x1 convolution. S2D layer moves spa- tial dimension information depth dimension by uniformly sampling and reorganizing features with fixed gap, so as down-sample features with- out additional parameters. Then 1xl convolutions are used offer a channel-wise pooling to gener- ate fixed-dimension feature maps. More details are shown in Appendix A.1.
Space-to-Deptie (S20)
Figure 2: Illustration of the space-to-depth transformation. The S2D operation moves the activation from the spatial dimension to the channel dimension
To verify assumption, conduct controlled experiments on different backbone and neck com- putation ratios in multiple object detection of the same FLOPs in Section 4. The results show that neck is crucial than conventional backbones in object detection task.

---

Published as a conference paper at ICLR 2022
(a) FPN
FPN
(b) PANet
(c) BiFPN
(d) GFPN
Figure 3: Feature pyramid network evolution design from level level (P3 P7). (a) FPN (Ln
et al. 2017a) introduces top-down pathway to fuse multi-scale features; (b) PANet (Liu et al. 2018) adds an additional bottom-up pathway on top of FPN; (c) BiFPN (Tan et al., (020) introduces bidirectional cross-scale pathway; (d) GFPN contains both queen-fusion style pathway and skip-layer connection. The dashed box refers to the layer in each FPN design.
3.2 GENERALIZED-FPN
In feature pyramid network, multi-scale feature fusion aims to aggregate different resolution features that are extracted from the backbone. Figure shows the evolution feature pyramid network design. Conventional FPN (Lin al. (2177a) introduces a top-down pathway to fuse multi-scale features from level to Considering the limitation one-way information flow, PANet(Liu et al. 2018) adds extra bottom-up path aggregation network, but with more computational cost. Besides, BiFPN (Tan et al. 2020) removes nodes that only have one input edge, and add extra edge from the original imput the same level. However, observe that previous methods focus only on feature fusion, but lack the inner block connection. Therefore, we design a novel pathway fusion including skip-layer and cross-scale connections, shown in Figure B)..
Skip-layer Connection. Compared to other connection methods, skip connections have short dis- tances among feature layers during back-propagation. In order reduce gradient vanish in such a heavy "giraffe" neck, we propose two feature link methods: dense-link and log2n-link in our proposed GFPN, as shown in Figure
(a) Dense-link
(b) logyn-ink
Figure 4: Two link mode of skip-layer connection: (a) dense-link: the concatenation of all preceding
layers (b) logzn-link: the concatenation of at most logzl + layers.
dense-link: Inspired by DenseNet (Huang al. 2017). for each scale feature Pk in level k Consequently, the (th layer receives the feature-maps of all preceding layers:
P Conu Pk-
where Comcal() refers to the concatenation of the feature-maps produced all preceding layers, and Conv() represents 3x3 convolution.
Specifically, in each level k the 1th layer receives the feature-maps from at most logal + 1 number of preceding layers, and these input layers are exponentially apart from depth with base 2, as denoted:
n-link
P Cone (CoocaaCnat
pl P 2!-20 )),
where 1- 2d 2 0, Concat() and Conv() also represent concatenation and 3x3 convolution respec- tively. Compare to dense-link depth the time complexity of logın-link only cost O logzl), instead of O(P)). Moreover, loggn-link only increase the short distances among layers during back- propagation from to Hloggl Hence, logan-link scale deeper networks.

---

Published as a conference paper at ICLR 2022
P6
P5
Ps
55
P4
P4
1x1
Com
Mix pooling
Fusion in PANet
Queen-fusion
Figure 5: Illustration of cross-scale connection between PANet and our Queen-fusion in GFPN. and C represent summation and concatenation fusion style, and Pa denotes node in next layer.
Cross-scale Connection. Based on our assumption, our designed sufficient information exchange should contains not only skip-layer connection, but also cross-scale connection, overcome large- scale variation. Previous works in connecting features between adjacent layers only consider same level feature (Liu et al, 2018) or previous level feature (Tan et al., 2020p. Therefore, we propose new cross-scale fusion named as Queen-fusion, that considering both same level and neighbor level features as shown in Figure d), like playing the queen piece in chess. As an example shown in Figure b)), the concatenation of Queen-fusion in P% consists previous layer Pa down- sampling, previous layer Pa up-sampling, previous layer P3 and current layer P4. In this work, we apply bilin- ear interpolation and max-pooling as up-sampling and dawn-sampling functions respectively.
Therefore, the extreme large-scale variation scenario, it requires that the model has sufficient high-level and low-level information exchange. Based the mechanism of skip-layer and cross-scale connections, the proposed generalized-FPN can be expanded as long as possible, just like the "giraffe neck". With such "heavy neck" and lightweight backbone, our GiraffeDet can balance higher accuracy and better efficiency trade-off.
3.3 GIRAFFEDET FAMILY
According to our proposed S2D-chain and Generalized we can develop a family of different GiraffeDet scaling models that can overcome a wide range of resource constraints. Previous work scale up its detector the inefficient way, as changing bigger backbone networks like ResNeXt (Xie) (et al. 2017), stacking FPN blocks e.g. NAS-FPN (Ghiasi et al. (019). Specially, EfficientDet (Tan et al. 2020) start using compound coefficient & to jointly scale up all dimensions of backbone. Different from EfficientDet, we only focus the scaling of GFPN layers instead of the whole
framework including lightweight backbone. Specifically, we apply two coefficients da and &w to flexibly scale GFPN depth and width.
Based on our GFPN and eS2D chain, we have developed GiraffeDet family. Most previous work scale up base- line detector by changing bigger backbone networks, since their model mainly focus on a single limited scaling di- mensions. As we assume backbone is not critical for object detection task, the GiffeDet family only focus on the scal- ing of generalized-Pie -PNN. Two multipliers are proposed to control the depth (# of layers) and width (# of channels) for GFPN:
Table 1:
The scaling config for GiraffeDet family Φd the hyper-parameter that denotes the depth (# of layers) width (# of channels) of GFPN can be calculated based on &w by Equation
GFPN. The
Dyfpm= dd. W sfpn
256 dw:
(3)
Generalized
Following above setting and equation. 3 we have devel- oped six architectures GiraffeDet, as shown in Table T. CiraffeDee-DD..I11.DI..
Φa
Giraffe-D7
7
0.7
have the same level FLOPs with ResNet-series based model and we compare perfor- mance of GiraffeDet family with SOTA models in the next section. Note that the layer GFPN is different with other FPN design as shown in Figure 3 In our proposed GFPN, each layer represents one depth, while the layer of PANet and BiFPN contains depth.
Giraffe-D11
11
0.85
Giraffe-DIe
14
0.95
Giraffe-DIt -i6f
16
1.0
Giraffe-D25
25
1.15
Giraffe-D29
29
1.2
5

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
In this section, we first introduce the implementation details and present our experimental result on the COCO dataset (Lin et al., 2014). Then compare our proposed GiraffeDet family with other state-of-the-art methods, and an in-depth analysis is provided to better understand our framework.
4.1 DATASET AND IMPLEMENTATION DETAILS
COCO dataset. We evaluate GiraffeDet on COCO 2017 detection dataset with 80 object categories. It includes 115k images for training (train), 5k images for validation (val) and 20k images with no public ground-truth truth for testing (test dev). The training all methods is conducted on the 115k training images. We report results the validation dataset for ablation study and results of the test- dev dataset from the evaluation server for state-of-the-an comparison and DCN related comparison.
For fair comparison, all results produced under mmdetection (Chen et al. 2019b) and the stan- dard COCO-style evaluation protocol. GFocalV2 (Li et al. 2021) and ATSS (Zhang et al. 2020) applied as head and anchor assigner, respectively. Following the the work of (He et al.) 2019),
all models are trained from scratch to reduce the influence of pre-train backbones on ImageNet. The shorter side of input images is resized 800 and the maximum size is restricted within 1333, To enhance the stability of scratch training, we adopt multi-scale training for all models, including: 2x
(p-2x) learning schedule (24 epochs, decays 16 and 22 epochs) only in R2-101 12-N00 backhone experiments, and 3x scratch (s-3x) learning schedule (36 epochs, decays at 28 and 33 epochs) ablation study, and 6x scratch (s-6x) learning schedule (72 epochs, decays at
65 and 71 epochs) in state-of-the-an comparison. More implementation details in Appendix B.
4.2 QUANTITATIVE EVALUATION ON COCO DATASET
We compare GiraffeDet with state-of-the-aar approaches in Table. Unless otherwise stated, single- model and single-scale setting with test-time augmentation is applied. We report accuracy for both test-dev (20k images with no public ground-truit and val with 5k validation images. We group models together if they have similar FLOPs and compare their accuracy in each group. Notably, model performance depends on both network architecture and training settings. We refer most models from their paper. But for a fair comparison, we also reproduce some of RetinaNet (Lin) al. 2017b), FCOS (Tian al., 2019), HRNet (Sun et al. 2019a), GFLV2 (Li et al. 2021) with 6x training from scratch, which denoted as f.
Large-scale Variance. According to the per- formance of Figure [ we can observe that our proposed GiraffeDet achieves the best perfor- mance in each pixel scale range, which indi- cates that the light backbone and heavy-neck paradigm, as well as our proposed GFPN, can effectively solve large-scale variance prob- lem. Also, under the skip-layer and cross- scale connections, high-level semantic infor- mation and low-level spatial information can be sufficiently exchanged. Many object instances are smaller than 1% of the image area in the COCO dataset, making detectors difficult to de- tect. Even though extremely small instance are difficult to detect, our method still outperforms 5.7% mAP than RetinaNet in the pixel range 0-32, which outperforms the same mAP in the middle pixel range 80-144. Notably, in the scale of pixel range 192-256, the proposed Gi- raffeDet outperforms the most than other meth- ods, which proves that our design can learn scale-sensitive features effectively.
0.8
HRNet
0.7
FCOS
RetinaNet
0.6-
GFocalV2
GiraffeDet
0.5
mAP
0.4
0-32
32-80
80-144
144-192
192-256
Instance Scale
Figure 6: mAP on all scale of object instances (pix- els) in five different models under R50 FLOPs level and 6x scratch training, including HRNet (Sun al. 2019a), GFocalV2 (Li et al. (02T). RetinaNer (Lin et al. 76). FCOS (TTaa et al. 2019) and our pro- posed Ger

---

Published as conference paper at ICLR 2022
Table 2: GiraffeDet performance COCO Results for single-model single-scale. test-dev is the COCO test set val the validation set. f means that results are reproduced by 6x scratch training, others are referred from their paper. We group models together if they have similar FLOPs, and compare their accuracy in each group. MStest: multiscale testing, R: ResNet, X: ResNext, and W: low-level fcatures map width in HRNet (# of channels). The head and anchor assigner of GiraffeDet family are GFocal and ATSS.
#FLOPs Backhone
#FLOPs Neck
#FLOPs total
test-dev
val-2017
Model
AP Pest
APs
APM
APt
AP
Yolov3-darkne153
139.82
32.54
187.73
33.0
18.3
35.4
41.9
33.8
RetinaNct-R500f
76.15
16.6
229.59
40.4
23.1
43.3
52.2
40.2
FCOS-R50
76.15
16.6
192.39
42.9
26.6
46.5
53.8
42.7
GFLV2-R50
76.15
16.6
199.96
44.3
26.8
47.7
54.1
43.9
GHL.2-R50f
76.15
16.6
199.96
44.8
27.2
48.1
54.4
44.5
HRNerV2p-W18
65.13
16.34
182.06
38.3
23.1
41.5
49.2
HRNerV2p-W.
65.13
16.34
182.06
40.6
25.7
43.3
50.6
40.2
GiraffeDet-D7
3.89
76.13
186.71
45.6
28.8
48.7
55.6
44.9
RetinaNet-R101
153.74
16.6
302.62
39.1
21.8
42.7
50.2
38.9
FCOS-R101
153.74
16.6
265.38
41.5
24.4
44.8
51.6
40.8
ATSS-R101
153.74
16.6
269.94
43.6
26.1
47.0
53.6
41.5
PAA-RI01
153.74
16.6
269.94
44.8
26.5
48.8
56.3
43.5
GFLV2-R101
153.74
16.6
272.99
46.2
27.8
49.9
57.0
45,9
HRNerV2p-W332
155.8
19.64
276.03
40.5
23.4
42.6
51.0
40.3
HRN2t
155.8
19.64
276.03
44.6
27.9
48.0
56.8
44.1
GiraffeDet-D11
3.89
166.73
275.39
46.9
29.9
51.1
58.4
46.6
RetinaNct-R1152
226.84
16.6
375.72
45.1
28.4
48.8
58.2
HRNe
229.57
21.52
351.69
42.8
27.0
46.4
54.5
42.7
GiraffeDet-D14
3.89
251.9
361.98
47.7
30.9
51.6
60.3
#SAF-X101-64x44
304.68
16.6
421.86
42,9
26.6
46.2
52.7
42.4
i
304.68
16.6
424.32
43.0
25.3
45.6
54.6
42.7
#teeAmchhr-XX001-6kkkd
304.68
16.6
458.07
44.9
26.5
48.0
56.5
FCOS-X101-64x4d
304.68
16.6
420.87
43.2
26.5
46.2
53.3
42.6
ATSS-X101-64x4d
304.68
16.6
425.43
45.6
28.5
48.9
55.6
OTA-X101-64x44
304.68
16.6
453.55
47.0
29.2
50.4
57.9
GiraffeDet-D16
3.89
315.69
438.59
48.7
31.7
52.4
61.3
48.3
GiraffeDet-D25
3.89
681.02
785.4
50.5
32.2
54.2
63.5
49.9
GiraffeDet-D29
3.89
865.44
972.97
51.3
33.1
54.9
64.9
51.0
GiraffeDet-D2+-MSScce
3.89
865.44
972.97
54.1
35.9
56.8
67.2
53,9
Comparison with State-of-the-art Methods. Table ssoos shows better performance than previous detectors in each same level of FLOPs, which indicates that our method can detect objects effectively and efficiently. 1) Compared to ResNet-based methods on the low-level FLOPs scale, we found that, even if the overall performance obviously not increased too much, our method has significant performance in detecting small and large object cases. It indicates our method performs better in large-scale variation dataset. 2) Compared to ResNext- based methods in high-level FLOPs scale, we find that GiraffeDet has higher performance than in low-level FLOPs slot, which indicates that a good design of FPN can be more crucial than a heavy backbone. 3) Compared to other methods, the proposed GiraffeDet family also has the SOTA performance that proves our design achieves higher accuracy and better efficiency in each FLOPs level. Besides, the NAS-based method consumes ton of computational resources cover the search space in the training process, and therefore we do not consider comparing our method with them. Finally, with the multi-scale protocol, our GiraffeDet achieve 54.1 14. mAP, especially APs increases 2,8% and AP, increases 2.3% much more than 1.9% in APM-
that our GiraffeDet family achieves
4.3 ABLATION STUDY
The success our GiraffeDet can be attributed to both framework design and technical improve- ments in each component. To analyze the effect of each component in GiraffeDet,
construct ablation studies including: 1) Connection analysis generalized-FPe 2) Depth & Width in GFPN; 3) Backbone discussion; 4) Girrafeeet with DCN. More ablation study can be in Appendix C.

---

Published as a conference paper at ICLR 2022
Table Ablation study on the Connection analysis. The model of "GFPN w/o skip" neck designed without any skip-layer connection, "GFPN-dense" neck model utilizes dense-link and "GFPN- logen" neck model utilizes logen-link.
Backhone
Neck
training
FL.OPs(G)
AP vo!
AP
50
AP
APM
APr
S2D chain
stacked FPN
s-3x
276.11
38.8
54.3
42,2
23.1
41.9
50.6
S2D chain
stacked PANet
s-3x
275.32
40.5
56.3
43.9
24.3
43.8
52.1
$2D chain
stacked BiFPN
s-3x
273.1
41.0
57.1
44.3
24.0
43.6
51.9
S2D chain
GFPN w/o skip
s-3x
273.51
41.2
57.0
44.3
25.7
43.5
51.9
S2D chain
GFPN-dense
s-3x
273.43
41.3
57.1
44.4
26.0
43.8
52.2
S2D chain
GFPN-log.n
S-3x
275.39
41.8
58.1
45.7
26.4
44.9
52.7
Connection Analysis. There are multiple options for constructing pathways between nodes, which are mainly based on graph theory design and human empirical design. Different connections repre- feature maps. We construct ablation study models and conduct experiments to investigate the effects of our proposed connections. In addition, we stacked basic FPN, PANet and BiFPN several times for fair comparison the same FLOPs level and used the same backbone and prediction head. Results are given in Table
sent different exchanges of information
Skip-layer Connection. According to the results of GFPN-dense and GFPN-loggn neck of Gi- reffeDet, we observe that log2n connection has achieved the best performance, and dense connec- tion only performs slightly better than without any skip-layer connection. It indicates that the log2n connection provides more effective information transmission from early nodes to later, while dense connection might provides redundant information transmission. Meanwhile, log2n connection can provides deeper generalized- -PN on the same level of FLOPs. Notably, both generalized- con- nections obtain higher performance than stacked BiFPN, which can prove that our proposed Gi- raffeDet can be more efficient.
Cross-scale Connection. From Table 3 we can see that stacked PANet and stacked BiFPN can achieve higher accuracy than their basic structure with bidirectional information flow, which indi- cates the importance of information exchange FPN structure. Overall, our GiraffeDet model can achieve better performance, which proves that our Queen-fusion can obtain sufficient high-level and low-level information exchange from previous nodes. Especially, even without skip-layer connec- tion, our generalized-FPa can still outperform other methods.
Table 4: Ablation study on Depth & Width analysis. All models apply S2D-chain as their backbone. "GFPN-logan" denotes the GFPN neck utilizes logan-link.
Backbone
Neck
depth
width
training
FLOPs(G)
AP
AP30
AP75
APs
AP
AP,
S2D chain
stacked FPN
11
307
s-3x
274.67
38.3
55.0
41.2
22.0
42.1
51.5
S2D chain
stacked PANet
308
s-3x
274.41
40.6
56.9
44.3
24.2
44.1
S2D chain
stacked BiFPN
11
400
274.51
40.5
56.8
43.9
24.1
43.6
52.0
$2D chain
stacked FPN
19
221
s-3x
276.11
38.8
54.3
42.2
23.1
41.9
50.6
$2D chain
stacked PANet 19
221
s:3x
275.32
40.5
56.3
43.9
24.3
43.8
52.1
S2D chain
stacked BiFPN
29
221
s-3x
273.1
41.0
57.1
44.3
24.0
43.6
51.9
S2D chain GFPN-log.n
11
221
s-3x
275.39
41.8
58.1
45.7
26.4
44.9
52.7
Effect of Depth & Width. To further fairly comparison with different "Neck", we conduct two groups of experiments comparison with stacked basic FPN, PANet and BiFPN the same FLOPs level, in order to analysis the effectiveness of depth and width (number of channel) in our proposed generalized-FPN. Note that shown in Figure 3 each layer of our GFPN and FPN contains one depth, while the layer of PANet and BiFPN contains two depth. As shown in Table 4 we observe that our proposed GFPN outperforms both level of depth and width all kinds of FPN, which also indicates that the log2n connection can provide information transmission effectively and the de- signed Queen-fusion can provide information exchange sufficiently. Moreover, our proposed GFPN can achieve higher performance in a smaller design, as "11" "11" depth and "221" width, which indicates that our design can achieve multi-scale detection efficiently.
8

---

Published as a conference paper at ICLR 2022
Backbone Effects. Figure 7 shows the performance of different neck depth and different backbones in the same FLOPs level. The results show that the com- bination of S2D-chain and GFPN out- performs other backbone models, which can verify our assumption that FPN is more crucial and conventional back- bone would not improve performance as depth increasing. In particular, we can observe that performance even de- creases with the growth of the back- bone model. We consider this might be- cause the domain-shie problem remains higher in a large backbone, and it also proves our assumption.
41.8
41.6
40.9
(0)
BFFNN
GFPN
(9)
180
Figure 7: Ablation study on different backbones: a) S2D-chain with GFPN-D1 b) ResNet-18 with GFPN-D10: c) ResNet-34 with GFPN-D8: d) ResNet
stacked BiFPN; e) ResNet- with GFPN-D7; f) DarkNet with GFPN-D4; g) ResNet-101 with GFPN-D2.
Table 5: val-2017 results of the deformable convolution network applied in SiraffeDet-DIl denotes the GFPN with synchronized batch normalization (Zhang al. 2018) for multi-GPU training.
Backbone
Neck
training
DCN
AP
AP50
AP75
APs
APM
AP,
S2D chain
GFPN-D11 GFPN-D11 $ GFPN-DIL
s-3x
41.8
58.1
45.7
26.4
44.9
52.7
S2D chain
s-3x
42.9
59.6
46.9
27.1
46.5
54.1
S2D chain
s-3x
V
45.3
62.4
49.6
28.5
49.3
56.9
S2D chain
GFPN-D11
s-6x
46.6
64.0
51.1
29.6
50.8
57.9
S2D chain GFPN-DI1 $
s-6x
49.3
66.9
53.8
31.6
53.2
61.7
Table 6: val-2017 results Res2Net- 101- DDN (R2-101-D 101--0N1)))) backbone with multiple GFPN necks. GFPN- tiny refers to GFPN of depth 8 and width as 122 (same FLOPs level as FPN).
101ket-101-a
Backbone
Neck
Head
training
AP eul
AP50
AP75
APs
APM M
AP,
FPS
R2-101-DCN
FPN
GFLV2
p-2x
49,9
68.2
54.6
31.3
$4.0
65.5
11.7
R2-101-DCN
GFPN-ting
GFLV2
p-2x
50.2
68.0
54.8
32,4
54.7
65.5
11.2
R2-101-DCN
GFPN-DI1
GFLV2
p-2x
51.1
69.3
55.5
32.6
$6.0
65.7
10.1
R2-101-DCN
GFPN-D11
GFLV2
s-6x
70.2
56.7
33.9
56.8
66.9
10.1
Results with DCN
We then conduct experiments analyse deformable convolution network (DCN)(Dai et al.) 2017) in our GiraffeDet, which has been widely used for improving detection performance recently. As shown in Table55 we observe that DCN can significantly improve the performance of our Det Especially, according to Table GiraffeDet-D11 with DCN can achieve a better performance than GiraffeDet-Dr D16. Also under acceptable inference time, observe that such shallow GFPN (tiny) with a strong DCN backbone can improve the performance, and the performance has been largely increased with the growth GFPN depth, as shown in Table 6 Note that as the design of GFPN, Our GiraffeDet is more suitable for scratch training and has significant improvement.
5 CONCLUSION
In this paper, we propose novel heavy-neck paradigm framework, GiraffeDet, giraffe-like net- work, to address the problem of large-scale variation. particular, GiraffeDet uses a lightweight spatial-to-depth chain backbone, and the proposed generalized-FPNN as heavy neck. The spatial-to-deptt chain applied to extract multi-scale image features in a lightweight way, and the generalized-FPP is proposed to learn sufficient high-level semantic information and low-level spatial information exchange. Extensive results manifested that the proposed GiraffeDet family achieves higher accuracy and better efficiency, especially detecting small and large object instances.

---

Published as a conference paper at ICLR 2022
REFERENCES
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of IEEE conference on computer vision and pattern recognition, pp. 6154-162, 2018.
Hesen Chen, Ming Lin, Xiuyu Sun, Qian Qi, Hao Li, and Rong Jin. Muffnet: Multi-layer feature federation for mobile deep learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019a.
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:/906.000 0015000155,,.. 155, 2019b.
Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fen: Object detection via region-based fully convo- lutional networks. In Advances in neural information processing systems, pp. 379-387, 2016.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
Xianzhi Du, Tsung- Yi Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc V Le, and Xiaodan Song. Spinenet: Learning scale-permuted backbone for recognition and localization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp-
11592-11601. 2020.
Zheng Ge, Songtao Liu, Zeming Li, Osamu Yoshie, and Jian Sun. Ota: Optimal transport assignment for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 303-312, 2021.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc Le. Nas-fpn: Learning scalable feature pyramid architec- ture for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7036-7045, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4918 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Xiang Li, Wenhai Wang, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss v2: Learning reliable localization quality estimation for dense object detection. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11632-11641 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision. pp- 2980-2988, 2017b.
Shu Liu, Lu Qi. Haifang Qin, Jianping Shi, and Jiay Jia. Path aggregation network for instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8759-8768, 2018
10

---

Published as a conference paper at ICLR 2022
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin. Libra r-cmm: Towards balanced learning for object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Paltern Recognition, pp. 821-830, 2019.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. Frame-recurrent video super- resolution, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6626-6634, 2018.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874-1883, 2016.
Bharat Singh and Larry S Davis. An analysis of scale invariance in object detection snip. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 3578-3587, 2018.
Bharat Singh, Mahyar Najibi, and Larry S Davis. Sniper: Efficient multi-scale training. arXiv preprint 1XXiv:8805009000 2018.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693-5703, 2019a.
Shuyang Sun, Jiangmiao Pang, Jianping Shi, Shuai Yi, and Wanli Ouyang. Fishnet: A versatile backbone for image, region, and pixel level prediction. arXiv preprint arXiv: 03495, 2019b.
Zhenhong Sun, Ming Lin, Xiuyu Sun, Zhiyu Tan, and Rong Jin. Revisiting efficient object detection backbones from zero-shot neural architecture search. arXiv preprint arXiv:2111.13333 2021.
Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 10781-10790, 2020.
Zhiyu Tan, Xuecheng Nie, Qi Qian, Nan Li, and Hao Li. Learning to rank proposals for object detection. In ICCV, 2019.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEFF/CVF international conference computer vision, pp- 9637-9636, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In Proceedings of the IEEE confer- on Computer Vision and Pattern Recognition, pp. 7151-7160, 2018.

---

Published as a conference paper at ICLR 2022
Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li. Bridging the gap between muchor-based and anchor-free detection via adaptive training sample selection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9759-9768 2020.
Xiaosong Zhang, Fang Wan, Chang Liu, Rongrong Ji, and Qixiang Ye. Freeanchor: Learning to match anchors for visual object detection. arXiv preprint arXiv: 290999249960 2019.
Chenchen Zhu, Yihui He, and Marios Savvides. Feature selective anchor-free module for single-shot object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 840-849, 2019.
12

---

Published as a conference paper at ICLR 2022
A ARCHITECTURE DETAILS
A.1 S2D CHAIN DESIGN
SEULDS
10
S20 Block
10x6.
Siu Bd
20. 1006
Ps
Block
94 1xd
Spaseto-daphh
20x x0
40x24x:512
Ps
520 Block
Space to-dephh
40x 20x
S20 Block
Corw let
30x
Space depth
80x40x552
180x16x228
S20 Block
Cony
160x166x366
SEU
100x 100
Input Image
Figure 8: Architecture of Space-to-depth chain. "Conv": convolutional neural networks. "SiLU"; sigmoid Linear Units activation function, "Space-to-depth"" S2D layer, and "Bx" represents the number of S2D block.
Table 7: Structure Space-To-Deple chain used our experiments
Operation Layer
Number of Filters
Size of Each Filter
Stride Value
Padding Value
Size of Output Feature
FLOPs
Input Image
1280 768
Convolution Layer
32
3 3
2 x 2
1
640 x 384 x 32
0.21G
SiLU Layer
640 x 384 x 32
0.21G
Convolution Layer
64
3 3 x 32
2
1
320 x 192 x 64
1.34G
SiLU Layer
320 192 64
L.34G
Space-to- to-ttpth
x 256
1.34G
Convolution Layer
128 1 x 256
0x0
x 128
1.85G
SiLU Layer
160 x 96 128
L885G
Space-to-Depee
80 48 x 512
1.85G
Convolution Layer
256
1x1x
1
0x
80 x 48 256
2.35G
SiLU Layer
80 48 256
2.35G
Space-to-Depth
x 1024
2.35G
Convolution Layer
512
1024
1x
0x
24 512 x 24 x
2.85G
SiLU Layer
2.85G
Space-to-Depth
20 x 12 x 2048
2,85G
Convolution Layer
1024 x x 2048
1x1
0x
20 12 x 1024
3,36G
SiLU Layer
20 x 12 1024
3,36G
Space-to to-Depth
10 x 4096
3.36G
Convolution Layer
2048
4096
1x1
0x
6 2048
3.86G
SiLU Layer
x 2048
3.86G
13

---

Published as a conference paper at ICLR 2022
A.2
GENERALIZED FPN DESIGN
Stacked BIFPN
GFPN-DI1
166.73
SID chain 3,89 GFLOPS
chain 3,89 GFLOPs
chuin-BiFPP BiFFN-D29 AP
S2D chain- GFPN-DI1
AP
Figure Architccture comparison of stacked BiFPN and our proposed GFPN-D11.
B MORE IMPLEMENTATION DETAILS
Table 8: List of hyperparameters used.
Hyperparameter
Value
Batch Size per GPU
2
Optimizer
SGD
Learning Rate
0.02
Step Decrease Ratio
0.1
Momentum
0.9
Weight Decay
1.0 10
Input Image Size
[1333, 800]
Multi-Scale Range (Ablation Study)
[0.8, 1.0]
Multi-Scale Range (SOTA)
[0.6, 1.2]
GFPN Input Channels
[128, 256, 512, 1024, 2048]
GFPN Output Channels
[256, 256, 256, 256, 256]
Training Epochs (Ablation Study)
36 epochs from seratch (decays 28 and 33 epochs)
Training Epochs (SOTA)
72 epochs from scratch (decays at 65 and 71 epochs)
14

---

Published as a conference paper at ICLR 2022
C MORE ABLATION STUDIES
C.I FEATURE FUSION METHODS
45
44
43
mAp
40
39
Concatenation
38
Summetion Summattin simPaa-FLOPs
37
200
400
1000
GFLOPs
Figure 10: Ablation study on Fusion-style analysis consists three models: 1) "Concatenation" model:
raffeDet utilizes concatenation fusion style; 2) "Summation" model: GiraffeDet utilizes summation fusion style; "Summation smilar-FLOPs" model: same FLOPs level with "Concatenation" model.
Figure shows the performance of using sunmation-based feature fusion and concatenation-basee feature fusion style. We can observe that the concatenation-base fusion style of features can achieve better performance in the same FLOPs level. Although summation- -basdd feature fusion has fewer FLOPs than concatenation-base based style, performance is significantly lower. We think it is not worth sacrificing mAP to have fewer FLOPs. Notably, the performance the "Summation" model is growing slightly after GFLOPs over 300, which indicates the concatenation-base feature fusion style can be more accurate and efficient again.
C.2 INFERENCE TIME
47.0
S2D-chain GFPN
ResNet +PPN
46.5
mAP
45.5
45.0
44.5
44.0
12
14
16
18
20
22
24
FPS
Figure 11: Inference time comparison between "ResNet FPN" model and "S2D-chain + GFPN" model on the same FLOPs level. Orange line denotes "S2D-chain chain GFPN" and purple line denotes "ResNet + FPN".
Table 9: Comparison on inference time between "ResNet + FPN" model and "S2D-chain + GFPN" model on the same FLOPs level.
Backbone
Neck
Head
training
FLOPs(G)
APrest
AP50
APTs
APş
APи
AP;
ResNct-50
FPN
GFLV2
p-2x
199.96
44.3
62.3
48.5
26.8
47.7
54.1
20.5
S2D chain
GFPN-d7
GFLV2
s-6x
183.67
45.6
62.7
49,8
28.8
48.7
$7.6
19.9
ResNet-100
FPN
GFLV2
p-2x
272.99
46.2
64.3
50.5
27.8
49.9
57.0
15.1
S2D chain
GFPN-111
GFLV2
s-6x
275.39
46.9
64.3
$1.5
29,9
51.1
58.4
14.0
15

---

Published as a conference paper at ICLR 2022
We conduct inference time experiments to compare GiraffeDet with the basic detection model at the same FLOPs level. From Table 9 we can observe that our Gi- raffeDet achieves significant improvements with acceptable inference time. We think the reason might be that most popular GPUs are friendly for ResNet-hased backbone inferences and memory 1/ is sensitive to the concatenate-basee fusion GFPN that will affect the inference speed. No- tably, according to Figure II the performance of our GiraffeDet decreases slower than the standard model with FPS growth.
C.3 STANDARD BACKBONE
Table 10: val-2017 results of standard backbone with stacked BiFPN and proposed GFPN.
Backbone
Neck
Head
training
FLOPs(G)
ocl
AP50
AP75
APs
APM M
APp
resmet-18
stacked BiFPN
GFLV2
s-3x
275.71
40.8
57.1
44.3
43.6
51.9
resmct-18
GFPN-d9
GFLV2
8-3x
277.05
41.3
57.7
45.0
25.0
44.2
52.8
resmct-18
GFPN-d11
GFLV2
8-3x
308.64
42.1
59.0
45.8
25.3
45.3
53.5
resnet-18
GFPN-d14
GFLV2
8-3x
366.20
42.9
59.6
46.9
25.8
46.3
54.7
We also conduct experiments on the ResNet-18 18 backbone. According to Table TO our proposed GFPN with a standard backbone can increase with the depth of GFPN growth. Our designed GFPN also outperforms BiFPN under the same FLOPs level.
16

---

Published as a conference paper at ICLR 2022
D ADDITIONAL QUALITATIVE RESULTS
RetinaNet
FCOS
HRNet
GFocalV2
GiraffeDet
Figure 12: Qualitative Evaluation of different approaches for object detection COCO dataset.
To better illustrate the performance different approaches, we provide qualitative result in Figure 12 Overall, observe that all methods can detect object instances from each image. Fur- thermore, GiraffeDet can detect more instances than other SOTA methods, especially small object instances, which proves that designed FPN can be effective in the large-scale variation dataset.
17


Reviews:
[{"review_id": "zTDE2rGXn8c", "reviewer": "Reviewer_FXm4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strength\n1. The argument in this paper was interesting and the approach (GiraffeDet) to addressing this argument seems to be effective for object detection.\n2. Designing the proposed architecture was technically sound.\n3. Experiments supports the effectiveness of the proposed method properly.\n\n* Weakness\n1. I do not have a big concern on this paper. I recommend to add an experiment comparing the proposed method to baselines when the neck networks are of equal depth.", "summary_of_the_paper": "This paper raises the question of whether it is effective to conventionally use deep (or heavy) networks trained on ImageNet dataset as backbone of a feature encoder for object detection. Instead, it introduces GiraffeDet, which has a deep neck network that processes multi-scale representations with a shallow backbone. Experiments demonstrate that, given the same memory budget, the proposed method outperforms the conventional method, which rely on the heavy backbone, on MS COCO dataset.", "summary_of_the_review": "Object detecton accuracy also highly depends on network depth. Although the present experiments well support the effectiveness of the proposed method in terms of efficiency, further experiments comparing methods with various Neck networks sharing depth will provide another perspective demonstrating the advantages of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hEQh7NvgOjz", "reviewer": "Reviewer_MDN5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths: The paper had a good exploitation on the connections in skip-layer and cross-scale. The findings are very helpful for the community. \nweaknesses: 1. The paper in general does not read well, and more careful proofreading is needed.\n                      2. In S2D structure, it is not clear why the number of parameters does not change. If the kernel height/width stay the same, then its depth will increase, resulting in more parameters. I agree the efficiency could be improved since the FLOP is quadratic on activation side length. But in terms of parameters, more details are expected.\n                     ", "summary_of_the_paper": "This paper proposes a new network architecture called GiraffeDet. GiraffeDet consists of a light backbone, a heavy neck and a prediction head. In the backbone, the paper proposes the S2D (spatial to depth) structure, claiming its better efficiency. In the neck, the paper proposes denser connections based on FPN, taking cross-scale information exchange into account. The experimental results show the state-of-the-art accuracy compared with other single stage object detection framework on the same FLOP level.  ", "summary_of_the_review": "This paper gives another insight into the balance of backbone and neck in the particular problem of object detection. It proposes a framework called giraffeDet, which proves very effective from the experimental results. The main contribution, in my opinion, is about the exploitation on the connections in skip-layer and cross-scale. But this contribution is minor. Also FLOPs is an important metric in efficiency, but the more straightforward inference time will add onto the claim about the light backbone + heavy neck.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WycVHOz3Nz7", "reviewer": "Reviewer_ygpA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n - The overall message, an object detector can have a light-weighted bottom-up backbone and a heavy top-down neck is new. \n - The technical contribution of designing the neck in a log-scale connection is neat. It is good to know the log-scale connection performs better than the dense connection.\n - The paper is well written and easy to follow.\n\n### Weaknesses\n\n - The comparisons in Table. 2 are not clear. What is the detection head of the GiraffeDet? I didn't find the details in the paper. Are they RetinaNet-style anchor-based or FCOS/ ATSS-style single anchor? Ideally, Table 2 should only compare the same detector with different backbones, as the proposed contribution only changes the backbone. It is unclear what to conclude when comparing FCOS with Res50 to a RetinaNet with a different FPN architecture. If the authors are not using the best detection head (or are using a better head than what is listed in Table. 2), then it is necessary to show the proposed backbone can improve the best detection head.\n\n - At a high level, the paper only shows \"using a light backbone and heavy neck\" is feasible for object detection, but does not show this is better than the conventional architecture. From Table. 2, the proposed GiraffeDet outperforms the better counterpart in the same block by an ~1mAP. However, given the well-established toolboxes of conventional architecture, it is very hard to convince a user to switch to the proposed architecture. For example, conventional backbones can easily get a performance boost with deformable convolutional layers, while it is unclear how deformable convolutional layers can help the proposed backbones. Ideally, the authors should also compare to DCN-based backbones as they are used in most detection papers. Overall, a 54.1 mAP with test-time augmentation is a victory in Nov 2021.\n\n - While Table. 2 reports the FLOPs, it is more important to compare the actual inference time on the same machine. I am not sure if the log-scale connection in the proposed framework will make the runtime slower under the same FLOPs.\n\n - It would be interesting to see how the proposed neck works under a standard backbone. It is fine if the overall FLOPs increase if they significantly outperform BiFPN. Ideally, add the BiFPN entry to Fig 7 (left), with both S2D chain backbone or standard heavy backbone.\n\n - Fig. 8 is too small to note the difference.\n\n", "summary_of_the_paper": "This paper works on designing network architecture for object detection. While most existing object detectors adopt image classification backbones with a light-weighted \"neck\" (FPN), this paper proposes the inverse way: using a lightweight backbone and a heavy, dense neck. The proposed network performs ~1 mAP higher than (listed) existing works with similar computations on COCO.", "summary_of_the_review": "Overall the paper provides an interesting alternative to object detection architectures. However the current experiments do not directly support this claim (mixed change of network architecture and the detection head), and it is unclear if the proposed architecture is compatible with existing techniques (e.g., DCN). The proposed method is also at risk of low run time. \n\nMy current rating is a weak reject. In the rebuttal, if the authors show solely replacing the backbone (E.g., replacing ResNeXt-101-DCN-FPN (ideally, the largest EfficientDet or Swin-L) with the proposed network) can improve a state-of-the-art detector (e.g., CascadeRCNN, GFLV2 etc.) with acceptable time cost, I will increase my rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three experts reviewed the paper. Two reviewers recommended acceptance as they liked that the work identified a legacy design in object detection networks and resolved it by a new module. All reviewers found the empirical results strong. Reviewer MDN5 recommended rejection main for the concern that this newly designed module is a standard exploitation of network architectures. AC sided with the positive reviewers because of the paper's identification of a legacy design in object detection and the strong experimental results. Hence, the decision is to recommend the paper for acceptance. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (Poster)