Year: 2021

Abstract:
In conventional object detection frameworks, a backbone body inherited from image recognition models extracts deep latent features and then a neck module fuses these latent features to capture information at different scales. As the resolution in object detection is much larger than in image recognition, the computational cost of the backbone often dominates the total inference cost. This heavy-backbone design paradigm is mostly due to the historical legacy when transferring image recognition models to object detection rather than an end-to-end optimized design for object detection. In this work, we show that such paradigm indeed leads to sub-optimal object detection models. To this end, we propose a novel heavy-neck paradigm, GiraffeDet, a giraffe-like network for efficient object detection. The GiraffeDet uses an extremely lightweight backbone and a very deep and large neck module which encourages dense information exchange among different spatial scales as well as different levels of latent semantics simultaneously. This design paradigm allows detectors to process the high-level semantic information and lowlevel spatial information at the same priority even in the early stage of the network, making it more effective in detection tasks. Numerical evaluations on multiple popular object detection benchmarks show that GiraffeDet consistently outperforms previous SOTA models across a wide spectrum of resource constraints.

heading: 1 INTRODUCTION
text: In the past few years, remarkable progress in deep learning based object detection methods has been witnessed. Despite object detection networks getting more powerful by different designing on architecture, training strategy and so on, the meta-goal that detecting all objects with large-scale variation has not been changed. For example, the scale of the smallest and largest 10% of object instances in COCO dataset is 0.024 and 0.472 respectively (Singh & Davis, 2018), which scaling in almost 20 times. This presents an extreme challenge to handle such a large-scale variation by using recent approaches. To this end, we aim to tackle this problem by designing a scale robust approach.
To alleviate the problem arising from large-scale variations, an intuitive way is to use multi-scale pyramid strategy for both training and testing. The work of (Singh & Davis, 2018) trains and tests detectors on the same scales of an image pyramid, and selectively back-propagates the gradients of object instances of different sizes as a function of the image scale. Although this approach improves the detection performance of most existing CNN-based methods, it is not very practical, as the image pyramid methods process every scale image, which could be computationally expensive. Moreover, the scale of objects between classification and detection datasets remains another challenge in domain shift when using pre-trained classification backbones.
Alternatively, the feature pyramid network is proposed to approximate image pyramids with lower computational costs. Recent methods still rely on superior backbone designing, but insufficient information exchange between high-level features and low-level features. For example, some work enhances the entire feature hierarchy with accurate localization signals in lower layers by bottom-up path augmentation, however this bottom-up path design might lack exchange between high-level semantic information and low-level spatial information. According to the above challenges, two questions in this task are raised as follows:
• Is the backbone of the image classification task indispensable in a detection model? • What types of multi-scale representations are effective for detection tasks?
These two questions motivate us to design a new framework with two sub-tasks two i.e., efficient feature down-sampling and sufficient multi-scale fusion. First, conventional backbones for scalesensitive features generation are computationally expensive and exist domain-shift problem. An alternative lightweight backbone can solve these problems. Second, it is crucial for a detector to learn sufficient fused information between high-level semantic and low-level spatial features. According to the above motivations, we design a giraffe-like network, named as GiraffeDet, with the following insights: (1) An alternative lightweight backbone can extract multi-scale feature transformation without any additional computation costs. (2) A sufficient cross-scale connection, Queen-Fusion, like the Queen Piece pathway in chess, to deal with different levels and layers of feature fusion. (3) According to the designed lightweight backbone and flexible FPN, we propose a GiraffeDet family for each FLOPs level. Notably, the experimental results suggest that our GiraffeDet family achieves higher accuracy and better efficiency in each FLOPs level.
In summary, the key contributions of our work as follows: • To the best of our knowledge, we present the first lightweight alternative backbone and flexible FPN combined as a detector. The proposed GiraffeDet family consists of Lightweight S2D-chain and Generalized-FPN, which demonstrates the state-of-the-art performance. • We design the lightweight space-to-depth chain (S2D-chain) instead of the conventional CNNbased backbone, and controlled experiments demonstrate that FPN is more crucial than conventional backbones in the object detection mode. • In our proposed Generalized-FPN (GFPN), a novel queen-fusion is proposed as our cross-scale connection style that fuses both level features in previous and current layers, and log2n skip-layer link provides more effective information transmission that can scale into deeper networks.
Based on the light backbone and heavy neck paradigm, the GiraffeDet family models perform well in a wide range of FLOPs-performance trade-offs. In particular, with the multi-scale testing technique, GiraffeDet-D29 achieves 54.1% mAP on the COCO dataset and outperforms other SOTA methods.

heading: 2 RELATED WORK
text: It is crucial for the object detector to recognize and localize objects by learning scale-sensitive features. Traditional solutions for the large-scale variation problem mainly based on improved convolutional neural networks. CNN-based object detectors are mainly categorized by two-stage detectors and one-stage detectors. Two-stage detectors (Ren et al., 2015; Dai et al., 2016; He et al., 2017; Cai & Vasconcelos, 2018; Pang et al., 2019) predict region proposals and then refine them by a subnetwork, and one-stage detectors (Liu et al., 2016; Lin et al., 2017b; Redmon et al., 2016; Redmon & Farhadi, 2017; Tian et al., 2019; Zhu et al., 2019; Zhang et al., 2020; 2019; Ge et al., 2021) directly detecting bounding-boxes without the proposal generation step. In this work, we mainly conduct experiments based on one-stage detector methods.
Recently, the main research line is utilizing pyramid strategy, including image pyramid and feature pyramid. The image pyramid strategy is used for detecting instances by scaling images. For example, SNIPER (Singh et al., 2018) propose a fast multi-scale training method, which samples the foreground regions around ground-truth object and background regions for different scale training. Unlike image pyramid methods, feature pyramid methods fuse pyramidal representations that cross different scales and different semantic information layers. For instance, PANet (Liu et al., 2018) enhances the feature hierarchies on top of feature pyramid network by additional bottom-up path augmentation. In addition, NAS-FPN (Ghiasi et al., 2019) utilize neural architecture search to automatically search feature pyramid network topology Our work focuses on feature pyramid strategy and proposes a sufficient high-level semantic and low-level spatial information fusion method.
Some researchers start working on designing new architectures to solve the large-scale variation problem instead of “backbone-neck-head” architecture in detection tasks. The work of Sun et al. (2019b) proposed the FishNet as an encoder-decoder architecture with skip connections to fuse multi-scale features. SpineNet (Du et al., 2020) designed as a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search. Our work is inspired by these methods and proposes a lightweight space-todepth backbone instead of a CNN-based backbone. However, our GiraffeDet still designed as the “backbone-neck-head” architecture. Because this typical architecture is widely used and proved effective in detection tasks.

heading: 3 THE GIRAFFEDET
text: Although extensive research has been carried out to investigate efficient object detection, large-scale variation still remains a challenge. To achieve the goal of sufficient multi-scale information exchange efficiently, we proposed the GiraffeDet for efficient object detection, and the “giraffe” consists of lightweight space-to-depth chain, generalized-FPN and prediction networks. The overall framework is shown in Figure 1, which largly follows the one-stage detectors paradigm.

heading: 3.1 LIGHTWEIGHT SPACE-TO-DEPTH CHAIN
text: Most feature pyramid networks apply conventional CNN-based networks as the backbone to extract multi-scale feature maps and even learn information exchange. However, recent backbones became much heavier with the development of CNN, it is computationally expensive to utilize them. Moreover, most recent applied backbones are mainly pre-trained on classification dataset, e.g., ResNet50 pre-trained on ImageNet, and we argue these pre-trained backbones are inappropriate in detection task and remains the domain-shift issue. Alternatively, FPN more emphasis on high-level semantic and low-level spatial information exchange. Therefore, we assume that FPN is more crucial than conventional backbones in the object detection model.
Inspired by (Shi et al., 2016; Sajjadi et al., 2018), we propose Space-to-Depth Chain (S2D Chain) as our lightweight backbone, which includes two 3x3 convolution networks and stacked S2D blocks. Concretely, 3x3 convolutions are used for initial down-sampling and introduce more non-linear transformations. Each S2D block consists of a S2D layer and a 1x1 convolution. S2D layer moves spatial dimension information to depth dimension by uniformly sampling and reorganizing features with a fixed gap, so as to down-sample features without additional parameters. Then 1x1 convolutions are used to offer a channel-wise pooling to generate fixed-dimension feature maps. More details are shown in Appendix A.1.
To verify our assumption, we conduct controlled experiments on different backbone and neck computation ratios in multiple object detection of the same FLOPs in Section 4. The results show that neck is more crucial than conventional backbones in object detection task.

heading: 3.2 GENERALIZED-FPN
text: In feature pyramid network, multi-scale feature fusion aims to aggregate different resolution features that are extracted from the backbone. Figure 3 shows the evolution of feature pyramid network design. Conventional FPN (Lin et al., 2017a) introduces a top-down pathway to fuse multi-scale features from level 3 to 7. Considering the limitation of one-way information flow, PANet(Liu et al., 2018) adds an extra bottom-up path aggregation network, but with more computational cost. Besides, BiFPN (Tan et al., 2020) removes nodes that only have one input edge, and add extra edge from the original input on the same level. However, we observe that previous methods focus only on feature fusion, but lack the inner block connection. Therefore, we design a novel pathway fusion including skip-layer and cross-scale connections, as shown in Figure 3(d).
Skip-layer Connection. Compared to other connection methods, skip connections have short distances among feature layers during back-propagation. In order to reduce gradient vanish in such a heavy “giraffe” neck, we propose two feature link methods: dense-link and log2n-link in our proposed GFPN, as shown in Figure 4.
• dense-link: Inspired by DenseNet (Huang et al., 2017), for each scale feature P lk in level k, Consequently, the lth layer receives the feature-maps of all preceding layers:
P lk = Conv(Concat(P 0 k , . . . , P l−1 k )), (1)
where Concat() refers to the concatenation of the feature-maps produced in all preceding layers, and Conv() represents a 3x3 convolution.
• log2n-link: Specifically, in each level k, the lth layer receives the feature-maps from at most log2l + 1 number of preceding layers, and these input layers are exponentially apart from depth i with base 2, as denoted:
P lk = Conv(Concat(P l−2n k , . . . , P l−21 k , P l−20 k )), (2)
where l− 2n ≥ 0, Concat() and Conv() also represent concatenation and 3x3 convolution respectively. Compare to dense-link at depth l, the time complexity of log2n-link only cost O(l · log2l), instead of O(l2). Moreover, log2n-link only increase the short distances among layers during backpropagation from 1 to 1+log2l. Hence, log2n-link can scale to deeper networks.
Cross-scale Connection. Based on our assumption, our designed sufficient information exchange should contains not only skip-layer connection, but also cross-scale connection, to overcome largescale variation. Previous works in connecting features between adjacent layers only consider same level feature (Liu et al., 2018) or previous level feature (Tan et al., 2020). Therefore, we propose a new cross-scale fusion named as Queen-fusion, that considering both same level and neighbor level features as shown in Figure 3(d), like playing the queen piece in chess. As an example shown in Figure 5(b), the concatenation of Queen-fusion in P5 consists previous layer P4 down-sampling, previous layer P6 up-sampling, previous layer P5 and current layer P4. In this work, we apply bilinear interpolation and max-pooling as our up-sampling and down-sampling functions respectively.
Therefore, in the extreme large-scale variation scenario, it requires that the model has sufficient high-level and low-level information exchange. Based on the mechanism of our skip-layer and cross-scale connections, the proposed generalized-FPN can be expanded as long as possible, just like the “giraffe neck”. With such a “heavy neck” and a lightweight backbone, our GiraffeDet can balance higher accuracy and better efficiency trade-off.

heading: 3.3 GIRAFFEDET FAMILY
text: According to our proposed S2D-chain and Generalized-FPN, we can develop a family of different GiraffeDet scaling models that can overcome a wide range of resource constraints. Previous work scale up its detector in the inefficient way, as changing bigger backbone networks like ResNeXt (Xie et al., 2017), or stacking FPN blocks e.g., NAS-FPN (Ghiasi et al., 2019). Specially, EfficientDet (Tan et al., 2020) start using compound coefficient φ to jointly scale up all dimensions of backbone. Inspired by EfficientDet, we also apply compound coefficient φ for scaling, but we only focus on the scaling of GFPN layers instead of the whole framework including lightweight backbone.
Based on our GFPN and eS2D chain, we have developed a GiraffeDet family. Most previous work scale up a baseline detector by changing bigger backbone networks, since their model mainly focus on a single or limited scaling dimensions. As we assume backbone is not critical for object detection task, the GiffeDet family only focus on the scaling of generalized-FPN. Two multipliers are proposed to control the depth (# of layers) and width (# of channels) for GFPN:
Dgfpn = φ, Wgfpn = 64 ∗ (log2φ), (3)
Following above setting and equation. 3, we have developed six architectures of GiraffeDet, as shown in Table 1. GiraffeDet-D7,11,14,16 have the same level FLOPs with ResNet-series based model and we compare performance of GiraffeDet family with SOTA models in the next section. Note that the layer of GFPN is different with other FPN design as shown in Figure 3. In our proposed GFPN, each layer represents one depth, while the layer of PANet and BiFPN contains two depth.

heading: 4 EXPERIMENTS
text: In this section, we first introduce the implementation details and present our experimental result on the COCO dataset (Lin et al., 2014). Then compare our proposed GiraffeDet family with other state-of-the-art methods, and an in-depth analysis is provided to better understand our framework.

heading: 4.1 DATASET AND IMPLEMENTATION DETAILS
text: COCO dataset. We evaluate GiraffeDet on COCO 2017 detection dataset with 80 object categories. It includes 115k images for training (train), 5k images for validation (val) and 20k images with no public ground-truth for testing (test − dev). The training of all methods is conducted on the 115k training images. We report results on the validation dataset for ablation study and results of the testdev dataset from the evaluation server for state-of-the-art comparison and DCN related comparison.
For fair comparison, all results are produced under mmdetection (Chen et al., 2019) and the standard COCO-style evaluation protocol. GFocalV2 (Li et al., 2021) and ATSS (Zhang et al., 2020) are applied as head and anchor assigner, respectively. Following the the work of (He et al., 2019), all models are trained from scratch to reduce the influence of pre-train backbones on ImageNet. The shorter side of input images is resized to 800 and the maximum size is restricted within 1333. To enhance the stability of scratch training, we adopt multi-scale training for all models, including: 2x imagenet-pretrained (p-2x) learning schedule (24 epochs, decays at 16 and 22 epochs) only in R2-101-DCN backbone experiments, and 3x scratch (s-3x) learning schedule (36 epochs, decays at 28 and 33 epochs) in ablation study, and 6x scratch (s-6x) learning schedule (72 epochs, decays at 65 and 71 epochs) in state-of-the-art comparison. More implementation details in Appendix B.

heading: 4.2 QUANTITATIVE EVALUATION ON COCO DATASET
text: We compare GiraffeDet with state-of-the-art approaches in Table. 2. Unless otherwise stated, singlemodel and single-scale setting with no test-time augmentation is applied. We report accuracy for both test-dev (20k images with no public ground-truth) and val with 5k validation images. We group models together if they have similar FLOPs and compare their accuracy in each group. Notably, model performance depends on both network architecture and training settings. We refer most models from their paper. But for a fair comparison, we also reproduce some of RetinaNet (Lin et al., 2017b), FCOS (Tian et al., 2019), HRNet (Sun et al., 2019a), GFLV2 (Li et al., 2021) with 6x training from scratch, which denoted as †.
Large-scale Variance. According to the performance of Figure 6, we can observe that our proposed GiraffeDet achieves the best performance in each pixel scale range, which indicates that the light backbone and heavy-neck paradigm, as well as our proposed GFPN, can effectively solve a large-scale variance problem. Also, under the skip-layer and crossscale connections, high-level semantic information and low-level spatial information can be sufficiently exchanged. Many object instances are smaller than 1% of the image area in the COCO dataset, making detectors difficult to detect. Even though extremely small instance are difficult to detect, our method still outperforms 5.7% mAP than RetinaNet in the pixel range 0-32, which outperforms the same mAP in the middle pixel range 80-144. Notably, in the scale of pixel range 192-256, the proposed GiraffeDet outperforms the most than other methods, which proves that our design can learn scale-sensitive features effectively.
Comparison with State-of-the-art Methods. Table 2 shows that our GiraffeDet family achieves better performance than previous detectors in each same level of FLOPs, which indicates that our method can detect objects effectively and efficiently. 1) Compared to ResNet-based methods on the low-level FLOPs scale, we found that, even if the overall performance is obviously not increased too much, our method has a significant performance in detecting small and large object cases. It indicates our method performs better in large-scale variation dataset. 2) Compared to ResNextbased methods in high-level FLOPs scale, we find that GiraffeDet has a higher performance than in low-level FLOPs slot, which indicates that a good design of FPN can be more crucial than a heavy backbone. 3) Compared to other methods, the proposed GiraffeDet family also has the SOTA performance that proves our design achieves higher accuracy and better efficiency in each FLOPs level. Besides, the NAS-based method consumes a ton of computational resources to cover the search space in the training process, and therefore we do not consider comparing our method with them. Finally, with the multi-scale test protocol, our GiraffeDet achieve 54.1% mAP, especially APS increases 2.8% and APL increases 2.3% much more than 1.9% in APM .

heading: 4.3 ABLATION STUDY
text: The success of our GiraffeDet can be attributed to both framework design and technical improvements in each component. To analyze the effect of each component in GiraffeDet, we construct ablation studies including: 1) Connection analysis in generalized-FPN; 2) Depth & Width in GFPN; 3) Backbone discussion; 4) GirrafeDet with DCN. More ablation study can be seen in Appendix C.
Connection Analysis. There are multiple options for constructing pathways between nodes, which are mainly based on graph theory design and human empirical design. Different connections represent different exchanges of information on feature maps. We construct ablation study models and conduct experiments to investigate the effects of our proposed connections. In addition, we stacked basic FPN, PANet and BiFPN several times for fair comparison on the same FLOPs level and used the same backbone and prediction head. Results are given in Table 3.
• Skip-layer Connection. According to the results of GFPN-dense and GFPN-log2n neck of GiraffeDet, we observe that log2n connection has achieved the best performance, and dense connection only performs slightly better than without any skip-layer connection. It indicates that the log2n connection provides more effective information transmission from early nodes to later, while dense connection might provides redundant information transmission. Meanwhile, log2n connection can provides deeper generalized-FPN on the same level of FLOPs. Notably, both generalized-FPN connections obtain higher performance than stacked BiFPN, which can prove that our proposed GiraffeDet can be more efficient.
• Cross-scale Connection. From Table 3, we can see that stacked PANet and stacked BiFPN can achieve higher accuracy than their basic structure with bidirectional information flow, which indicates the importance of information exchange in FPN structure. Overall, our GiraffeDet model can achieve better performance, which proves that our Queen-fusion can obtain sufficient high-level and low-level information exchange from previous nodes. Especially, even without skip-layer connection, our generalized-FPN can still outperform other methods.
Effect of Depth & Width. To further fairly comparison with different “Neck”, we conduct two groups of experiments comparison with stacked basic FPN, PANet and BiFPN on the same FLOPs level, in order to analysis the effectiveness of depth and width (number of channel) in our proposed generalized-FPN. Note that as shown in Figure 3, each layer of our GFPN and FPN contains one depth, while the layer of PANet and BiFPN contains two depth. As shown in Table 4, we observe that our proposed GFPN outperforms both level of depth and width in all kinds of FPN, which also indicates that the log2n connection and Queen-fusion can effectively provides information transmission and exchange. Moreover, our proposed GFPN can achieve higher performance in a smaller design.
Backbone Effects. Figure 7 shows the performance of different neck depth and different backbones in the same FLOPs level. The results show that the combination of S2D-chain and GFPN outperforms other backbone models, which can verify our assumption that FPN is more crucial and conventional backbone would not improve performance as depth increasing. In particular, we can observe that performance even decreases with the growth of the backbone model. We consider this might because the domain-shift problem remains higher in a large backbone, and it also proves our assumption.
Results with DCN.
We then conduct experiments to analyse deformable convolution network (DCN)(Dai et al., 2017) in our GiraffeDet, which has been widely used for improving detection performance recently. As shown in Table 5, we observe that DCN can significantly improve the performance of our GiraffeDet. Especially, according to Table 2, GiraffeDet-D11 with DCN can achieve a better performance than GiraffeDet-D16. Also under acceptable inference time, we observe that such a shallow GFPN (tiny) with a strong DCN backbone can improve the performance, and the performance has been largely increased with the growth of GFPN depth, as shown in Table 6. Note that as the design of GFPN, Our GiraffeDet is more suitable for scratch training and has significant improvement.

heading: 5 CONCLUSION
text: In this paper, we propose a novel heavy-neck paradigm framework, GiraffeDet, a giraffe-like network, to address the problem of large-scale variation. In particular, GiraffeDet uses a lightweight spatial-to-depth chain as a backbone, and the proposed generalized-FPN as a heavy neck. The spatial-to-depth chain is applied to extract multi-scale image features in a lightweight way, and the generalized-FPN is proposed to learn sufficient high-level semantic information and low-level spatial information exchange. Extensive results manifested that the proposed GiraffeDet family achieves higher accuracy and better efficiency, especially detecting small and large object instances.

heading: A ARCHITECTURE DETAILS
text: A.1 S2D CHAIN DESIGN
A.2 GENERALIZED FPN DESIGN

heading: B MORE IMPLEMENTATION DETAILS

heading: C MORE ABLATION STUDIES
text: C.1 FEATURE FUSION METHODS
Figure 10 shows the performance of using summation-based feature fusion and concatenation-based feature fusion style. We can observe that the concatenation-based fusion style of features can achieve better performance in the same FLOPs level. Although summation-based feature fusion has fewer FLOPs than concatenation-based style, performance is significantly lower. We think it is not worth sacrificing mAP to have fewer FLOPs. Notably, the performance of the “Summation” model is growing slightly after GFLOPs over 300, which indicates the concatenation-based feature fusion style can be more accurate and efficient again.
C.2 INFERENCE TIME
We conduct inference time experiments to compare our GiraffeDet with the basic detection model (ResNet-FPN-GFocalV2) at the same FLOPs level. From Table 9, we can observe that our GiraffeDet achieves significant improvements with acceptable inference time. We think the reason might be that most popular GPUs are friendly for ResNet-based backbone inferences and memory I/O is sensitive to the concatenate-based fusion on GFPN that will affect the inference speed. Notably, according to Figure 11, the performance of our GiraffeDet decreases slower than the standard model with FPS growth.
C.3 STANDARD BACKBONE
We also conduct experiments on the ResNet-18 backbone. According to Table 10, our proposed GFPN with a standard backbone can increase with the depth of GFPN growth. Our designed GFPN also outperforms BiFPN under the same FLOPs level.

heading: D ADDITIONAL QUALITATIVE RESULTS
text: RetinaNet FCOS HRNet GFocalV2 GiraffeDet
Figure 12: Qualitative Evaluation of different approaches for object detection on COCO dataset.
To better illustrate the performance of different approaches, we provide qualitative result in Figure 12. Overall, we can observe that all methods can detect object instances from each image. Furthermore, GiraffeDet can detect more instances than other SOTA methods, especially small object instances, which proves that our designed FPN can be effective in the large-scale variation dataset.

Review summaries:
This paper works on designing network architecture for object detection. While most existing object detectors adopt image classification backbones with a light-weighted "neck" (FPN), this paper proposes the inverse way: using a lightweight backbone and a heavy, dense neck. The proposed network performs ~1 mAP higher than (listed) existing works with similar computations on COCO.

This paper raises the question of whether it is effective to conventionally use deep (or heavy) networks trained on ImageNet dataset as backbone of a feature encoder for object detection. Instead, it introduces GiraffeDet, which has a deep neck network that processes multi-scale representations with a shallow backbone. Experiments demonstrate that, given the same memory budget, the proposed method outperforms the conventional method, which rely on the heavy backbone, on MS COCO dataset.

This paper proposes a new network architecture called GiraffeDet. GiraffeDet consists of a light backbone, a heavy neck and a prediction head. In the backbone, the paper proposes the S2D (spatial to depth) structure, claiming its better efficiency. In the neck, the paper proposes denser connections based on FPN, taking cross-scale information exchange into account. The experimental results show the state-of-the-art accuracy compared with other single stage object detection framework on the same FLOP level.