GiraffeDet: A Heavy-Neck Paradigm for Object Detection

1 INTRODUCTION . In the past few years , remarkable progress in deep learning based object detection methods has been witnessed . Despite object detection networks getting more powerful by different designing on architecture , training strategy and so on , the meta-goal that detecting all objects with large-scale variation has not been changed . For example , the scale of the smallest and largest 10 % of object instances in COCO dataset is 0.024 and 0.472 respectively ( Singh & Davis , 2018 ) , which scaling in almost 20 times . This presents an extreme challenge to handle such a large-scale variation by using recent approaches . To this end , we aim to tackle this problem by designing a scale robust approach . To alleviate the problem arising from large-scale variations , an intuitive way is to use multi-scale pyramid strategy for both training and testing . The work of ( Singh & Davis , 2018 ) trains and tests detectors on the same scales of an image pyramid , and selectively back-propagates the gradients of object instances of different sizes as a function of the image scale . Although this approach improves the detection performance of most existing CNN-based methods , it is not very practical , as the image pyramid methods process every scale image , which could be computationally expensive . Moreover , the scale of objects between classification and detection datasets remains another challenge in domain shift when using pre-trained classification backbones . Alternatively , the feature pyramid network is proposed to approximate image pyramids with lower computational costs . Recent methods still rely on superior backbone designing , but insufficient information exchange between high-level features and low-level features . For example , some work enhances the entire feature hierarchy with accurate localization signals in lower layers by bottom-up path augmentation , however this bottom-up path design might lack exchange between high-level semantic information and low-level spatial information . According to the above challenges , two questions in this task are raised as follows : • Is the backbone of the image classification task indispensable in a detection model ? • What types of multi-scale representations are effective for detection tasks ? These two questions motivate us to design a new framework with two sub-tasks two i.e. , efficient feature down-sampling and sufficient multi-scale fusion . First , conventional backbones for scalesensitive features generation are computationally expensive and exist domain-shift problem . An alternative lightweight backbone can solve these problems . Second , it is crucial for a detector to learn sufficient fused information between high-level semantic and low-level spatial features . According to the above motivations , we design a giraffe-like network , named as GiraffeDet , with the following insights : ( 1 ) An alternative lightweight backbone can extract multi-scale feature transformation without any additional computation costs . ( 2 ) A sufficient cross-scale connection , Queen-Fusion , like the Queen Piece pathway in chess , to deal with different levels and layers of feature fusion . ( 3 ) According to the designed lightweight backbone and flexible FPN , we propose a GiraffeDet family for each FLOPs level . Notably , the experimental results suggest that our GiraffeDet family achieves higher accuracy and better efficiency in each FLOPs level . In summary , the key contributions of our work as follows : • To the best of our knowledge , we present the first lightweight alternative backbone and flexible FPN combined as a detector . The proposed GiraffeDet family consists of Lightweight S2D-chain and Generalized-FPN , which demonstrates the state-of-the-art performance . • We design the lightweight space-to-depth chain ( S2D-chain ) instead of the conventional CNNbased backbone , and controlled experiments demonstrate that FPN is more crucial than conventional backbones in the object detection mode . • In our proposed Generalized-FPN ( GFPN ) , a novel queen-fusion is proposed as our cross-scale connection style that fuses both level features in previous and current layers , and log2n skip-layer link provides more effective information transmission that can scale into deeper networks . Based on the light backbone and heavy neck paradigm , the GiraffeDet family models perform well in a wide range of FLOPs-performance trade-offs . In particular , with the multi-scale testing technique , GiraffeDet-D29 achieves 54.1 % mAP on the COCO dataset and outperforms other SOTA methods . 2 RELATED WORK . It is crucial for the object detector to recognize and localize objects by learning scale-sensitive features . Traditional solutions for the large-scale variation problem mainly based on improved convolutional neural networks . CNN-based object detectors are mainly categorized by two-stage detectors and one-stage detectors . Two-stage detectors ( Ren et al. , 2015 ; Dai et al. , 2016 ; He et al. , 2017 ; Cai & Vasconcelos , 2018 ; Pang et al. , 2019 ) predict region proposals and then refine them by a subnetwork , and one-stage detectors ( Liu et al. , 2016 ; Lin et al. , 2017b ; Redmon et al. , 2016 ; Redmon & Farhadi , 2017 ; Tian et al. , 2019 ; Zhu et al. , 2019 ; Zhang et al. , 2020 ; 2019 ; Ge et al. , 2021 ) directly detecting bounding-boxes without the proposal generation step . In this work , we mainly conduct experiments based on one-stage detector methods . Recently , the main research line is utilizing pyramid strategy , including image pyramid and feature pyramid . The image pyramid strategy is used for detecting instances by scaling images . For example , SNIPER ( Singh et al. , 2018 ) propose a fast multi-scale training method , which samples the foreground regions around ground-truth object and background regions for different scale training . Unlike image pyramid methods , feature pyramid methods fuse pyramidal representations that cross different scales and different semantic information layers . For instance , PANet ( Liu et al. , 2018 ) enhances the feature hierarchies on top of feature pyramid network by additional bottom-up path augmentation . In addition , NAS-FPN ( Ghiasi et al. , 2019 ) utilize neural architecture search to automatically search feature pyramid network topology Our work focuses on feature pyramid strategy and proposes a sufficient high-level semantic and low-level spatial information fusion method . Some researchers start working on designing new architectures to solve the large-scale variation problem instead of “ backbone-neck-head ” architecture in detection tasks . The work of Sun et al . ( 2019b ) proposed the FishNet as an encoder-decoder architecture with skip connections to fuse multi-scale features . SpineNet ( Du et al. , 2020 ) designed as a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search . Our work is inspired by these methods and proposes a lightweight space-todepth backbone instead of a CNN-based backbone . However , our GiraffeDet still designed as the “ backbone-neck-head ” architecture . Because this typical architecture is widely used and proved effective in detection tasks . 3 THE GIRAFFEDET . Although extensive research has been carried out to investigate efficient object detection , large-scale variation still remains a challenge . To achieve the goal of sufficient multi-scale information exchange efficiently , we proposed the GiraffeDet for efficient object detection , and the “ giraffe ” consists of lightweight space-to-depth chain , generalized-FPN and prediction networks . The overall framework is shown in Figure 1 , which largly follows the one-stage detectors paradigm . 3.1 LIGHTWEIGHT SPACE-TO-DEPTH CHAIN . Most feature pyramid networks apply conventional CNN-based networks as the backbone to extract multi-scale feature maps and even learn information exchange . However , recent backbones became much heavier with the development of CNN , it is computationally expensive to utilize them . Moreover , most recent applied backbones are mainly pre-trained on classification dataset , e.g. , ResNet50 pre-trained on ImageNet , and we argue these pre-trained backbones are inappropriate in detection task and remains the domain-shift issue . Alternatively , FPN more emphasis on high-level semantic and low-level spatial information exchange . Therefore , we assume that FPN is more crucial than conventional backbones in the object detection model . Inspired by ( Shi et al. , 2016 ; Sajjadi et al. , 2018 ) , we propose Space-to-Depth Chain ( S2D Chain ) as our lightweight backbone , which includes two 3x3 convolution networks and stacked S2D blocks . Concretely , 3x3 convolutions are used for initial down-sampling and introduce more non-linear transformations . Each S2D block consists of a S2D layer and a 1x1 convolution . S2D layer moves spatial dimension information to depth dimension by uniformly sampling and reorganizing features with a fixed gap , so as to down-sample features without additional parameters . Then 1x1 convolutions are used to offer a channel-wise pooling to generate fixed-dimension feature maps . More details are shown in Appendix A.1 . To verify our assumption , we conduct controlled experiments on different backbone and neck computation ratios in multiple object detection of the same FLOPs in Section 4 . The results show that neck is more crucial than conventional backbones in object detection task . 3.2 GENERALIZED-FPN . In feature pyramid network , multi-scale feature fusion aims to aggregate different resolution features that are extracted from the backbone . Figure 3 shows the evolution of feature pyramid network design . Conventional FPN ( Lin et al. , 2017a ) introduces a top-down pathway to fuse multi-scale features from level 3 to 7 . Considering the limitation of one-way information flow , PANet ( Liu et al. , 2018 ) adds an extra bottom-up path aggregation network , but with more computational cost . Besides , BiFPN ( Tan et al. , 2020 ) removes nodes that only have one input edge , and add extra edge from the original input on the same level . However , we observe that previous methods focus only on feature fusion , but lack the inner block connection . Therefore , we design a novel pathway fusion including skip-layer and cross-scale connections , as shown in Figure 3 ( d ) . Skip-layer Connection . Compared to other connection methods , skip connections have short distances among feature layers during back-propagation . In order to reduce gradient vanish in such a heavy “ giraffe ” neck , we propose two feature link methods : dense-link and log2n-link in our proposed GFPN , as shown in Figure 4 . • dense-link : Inspired by DenseNet ( Huang et al. , 2017 ) , for each scale feature P lk in level k , Consequently , the lth layer receives the feature-maps of all preceding layers : P lk = Conv ( Concat ( P 0 k , . . . , P l−1 k ) ) , ( 1 ) where Concat ( ) refers to the concatenation of the feature-maps produced in all preceding layers , and Conv ( ) represents a 3x3 convolution . • log2n-link : Specifically , in each level k , the lth layer receives the feature-maps from at most log2l + 1 number of preceding layers , and these input layers are exponentially apart from depth i with base 2 , as denoted : P lk = Conv ( Concat ( P l−2n k , . . . , P l−21 k , P l−20 k ) ) , ( 2 ) where l− 2n ≥ 0 , Concat ( ) and Conv ( ) also represent concatenation and 3x3 convolution respectively . Compare to dense-link at depth l , the time complexity of log2n-link only cost O ( l · log2l ) , instead of O ( l2 ) . Moreover , log2n-link only increase the short distances among layers during backpropagation from 1 to 1+log2l . Hence , log2n-link can scale to deeper networks . Cross-scale Connection . Based on our assumption , our designed sufficient information exchange should contains not only skip-layer connection , but also cross-scale connection , to overcome largescale variation . Previous works in connecting features between adjacent layers only consider same level feature ( Liu et al. , 2018 ) or previous level feature ( Tan et al. , 2020 ) . Therefore , we propose a new cross-scale fusion named as Queen-fusion , that considering both same level and neighbor level features as shown in Figure 3 ( d ) , like playing the queen piece in chess . As an example shown in Figure 5 ( b ) , the concatenation of Queen-fusion in P5 consists previous layer P4 down-sampling , previous layer P6 up-sampling , previous layer P5 and current layer P4 . In this work , we apply bilinear interpolation and max-pooling as our up-sampling and down-sampling functions respectively . Therefore , in the extreme large-scale variation scenario , it requires that the model has sufficient high-level and low-level information exchange . Based on the mechanism of our skip-layer and cross-scale connections , the proposed generalized-FPN can be expanded as long as possible , just like the “ giraffe neck ” . With such a “ heavy neck ” and a lightweight backbone , our GiraffeDet can balance higher accuracy and better efficiency trade-off .

Summary: This paper raises the question of whether it is effective to conventionally use deep (or heavy) networks trained on ImageNet dataset as backbone of a feature encoder for object detection. Instead, it introduces GiraffeDet, which has a deep neck network that processes multi-scale representations with a shallow backbone. Experiments demonstrate that, given the same memory budget, the proposed method outperforms the conventional method, which rely on the heavy backbone, on MS COCO dataset.