GiraffeDet: A Heavy-Neck Paradigm for Object Detection

1 INTRODUCTION . In the past few years , remarkable progress in deep learning based object detection methods has been witnessed . Despite object detection networks getting more powerful by different designing on architecture , training strategy and so on , the meta-goal that detecting all objects with large-scale variation has not been changed . For example , the scale of the smallest and largest 10 % of object instances in COCO dataset is 0.024 and 0.472 respectively ( Singh & Davis , 2018 ) , which scaling in almost 20 times . This presents an extreme challenge to handle such a large-scale variation by using recent approaches . To this end , we aim to tackle this problem by designing a scale robust approach . To alleviate the problem arising from large-scale variations , an intuitive way is to use multi-scale pyramid strategy for both training and testing . The work of ( Singh & Davis , 2018 ) trains and tests detectors on the same scales of an image pyramid , and selectively back-propagates the gradients of object instances of different sizes as a function of the image scale . Although this approach improves the detection performance of most existing CNN-based methods , it is not very practical , as the image pyramid methods process every scale image , which could be computationally expensive . Moreover , the scale of objects between classification and detection datasets remains another challenge in domain shift when using pre-trained classification backbones . Alternatively , the feature pyramid network is proposed to approximate image pyramids with lower computational costs . Recent methods still rely on superior backbone designing , but insufficient information exchange between high-level features and low-level features . For example , some work enhances the entire feature hierarchy with accurate localization signals in lower layers by bottom-up path augmentation , however this bottom-up path design might lack exchange between high-level semantic information and low-level spatial information . According to the above challenges , two questions in this task are raised as follows : • Is the backbone of the image classification task indispensable in a detection model ? • What types of multi-scale representations are effective for detection tasks ? These two questions motivate us to design a new framework with two sub-tasks two i.e. , efficient feature down-sampling and sufficient multi-scale fusion . First , conventional backbones for scalesensitive features generation are computationally expensive and exist domain-shift problem . An alternative lightweight backbone can solve these problems . Second , it is crucial for a detector to learn sufficient fused information between high-level semantic and low-level spatial features . According to the above motivations , we design a giraffe-like network , named as GiraffeDet , with the following insights : ( 1 ) An alternative lightweight backbone can extract multi-scale feature transformation without any additional computation costs . ( 2 ) A sufficient cross-scale connection , Queen-Fusion , like the Queen Piece pathway in chess , to deal with different levels and layers of feature fusion . ( 3 ) According to the designed lightweight backbone and flexible FPN , we propose a GiraffeDet family for each FLOPs level . Notably , the experimental results suggest that our GiraffeDet family achieves higher accuracy and better efficiency in each FLOPs level . In summary , the key contributions of our work as follows : • To the best of our knowledge , we present the first lightweight alternative backbone and flexible FPN combined as a detector . The proposed GiraffeDet family consists of Lightweight S2D-chain and Generalized-FPN , which demonstrates the state-of-the-art performance . • We design the lightweight space-to-depth chain ( S2D-chain ) instead of the conventional CNNbased backbone , and controlled experiments demonstrate that FPN is more crucial than conventional backbones in the object detection mode . • In our proposed Generalized-FPN ( GFPN ) , a novel queen-fusion is proposed as our cross-scale connection style that fuses both level features in previous and current layers , and log2n skip-layer link provides more effective information transmission that can scale into deeper networks . Based on the light backbone and heavy neck paradigm , the GiraffeDet family models perform well in a wide range of FLOPs-performance trade-offs . In particular , with the multi-scale testing technique , GiraffeDet-D29 achieves 54.1 % mAP on the COCO dataset and outperforms other SOTA methods . 2 RELATED WORK . It is crucial for the object detector to recognize and localize objects by learning scale-sensitive features . Traditional solutions for the large-scale variation problem mainly based on improved convolutional neural networks . CNN-based object detectors are mainly categorized by two-stage detectors and one-stage detectors . Two-stage detectors ( Ren et al. , 2015 ; Dai et al. , 2016 ; He et al. , 2017 ; Cai & Vasconcelos , 2018 ; Pang et al. , 2019 ) predict region proposals and then refine them by a subnetwork , and one-stage detectors ( Liu et al. , 2016 ; Lin et al. , 2017b ; Redmon et al. , 2016 ; Redmon & Farhadi , 2017 ; Tian et al. , 2019 ; Zhu et al. , 2019 ; Zhang et al. , 2020 ; 2019 ; Ge et al. , 2021 ) directly detecting bounding-boxes without the proposal generation step . In this work , we mainly conduct experiments based on one-stage detector methods . Recently , the main research line is utilizing pyramid strategy , including image pyramid and feature pyramid . The image pyramid strategy is used for detecting instances by scaling images . For example , SNIPER ( Singh et al. , 2018 ) propose a fast multi-scale training method , which samples the foreground regions around ground-truth object and background regions for different scale training . Unlike image pyramid methods , feature pyramid methods fuse pyramidal representations that cross different scales and different semantic information layers . For instance , PANet ( Liu et al. , 2018 ) enhances the feature hierarchies on top of feature pyramid network by additional bottom-up path augmentation . In addition , NAS-FPN ( Ghiasi et al. , 2019 ) utilize neural architecture search to automatically search feature pyramid network topology Our work focuses on feature pyramid strategy and proposes a sufficient high-level semantic and low-level spatial information fusion method . Some researchers start working on designing new architectures to solve the large-scale variation problem instead of “ backbone-neck-head ” architecture in detection tasks . The work of Sun et al . ( 2019b ) proposed the FishNet as an encoder-decoder architecture with skip connections to fuse multi-scale features . SpineNet ( Du et al. , 2020 ) designed as a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search . Our work is inspired by these methods and proposes a lightweight space-todepth backbone instead of a CNN-based backbone . However , our GiraffeDet still designed as the “ backbone-neck-head ” architecture . Because this typical architecture is widely used and proved effective in detection tasks . 3 THE GIRAFFEDET . Although extensive research has been carried out to investigate efficient object detection , large-scale variation still remains a challenge . To achieve the goal of sufficient multi-scale information exchange efficiently , we proposed the GiraffeDet for efficient object detection , and the “ giraffe ” consists of lightweight space-to-depth chain , generalized-FPN and prediction networks . The overall framework is shown in Figure 1 , which largly follows the one-stage detectors paradigm . 3.1 LIGHTWEIGHT SPACE-TO-DEPTH CHAIN . Most feature pyramid networks apply conventional CNN-based networks as the backbone to extract multi-scale feature maps and even learn information exchange . However , recent backbones became much heavier with the development of CNN , it is computationally expensive to utilize them . Moreover , most recent applied backbones are mainly pre-trained on classification dataset , e.g. , ResNet50 pre-trained on ImageNet , and we argue these pre-trained backbones are inappropriate in detection task and remains the domain-shift issue . Alternatively , FPN more emphasis on high-level semantic and low-level spatial information exchange . Therefore , we assume that FPN is more crucial than conventional backbones in the object detection model . Inspired by ( Shi et al. , 2016 ; Sajjadi et al. , 2018 ) , we propose Space-to-Depth Chain ( S2D Chain ) as our lightweight backbone , which includes two 3x3 convolution networks and stacked S2D blocks . Concretely , 3x3 convolutions are used for initial down-sampling and introduce more non-linear transformations . Each S2D block consists of a S2D layer and a 1x1 convolution . S2D layer moves spatial dimension information to depth dimension by uniformly sampling and reorganizing features with a fixed gap , so as to down-sample features without additional parameters . Then 1x1 convolutions are used to offer a channel-wise pooling to generate fixed-dimension feature maps . More details are shown in Appendix A.1 . To verify our assumption , we conduct controlled experiments on different backbone and neck computation ratios in multiple object detection of the same FLOPs in Section 4 . The results show that neck is more crucial than conventional backbones in object detection task . 3.2 GENERALIZED-FPN . In feature pyramid network , multi-scale feature fusion aims to aggregate different resolution features that are extracted from the backbone . Figure 3 shows the evolution of feature pyramid network design . Conventional FPN ( Lin et al. , 2017a ) introduces a top-down pathway to fuse multi-scale features from level 3 to 7 . Considering the limitation of one-way information flow , PANet ( Liu et al. , 2018 ) adds an extra bottom-up path aggregation network , but with more computational cost . Besides , BiFPN ( Tan et al. , 2020 ) removes nodes that only have one input edge , and add extra edge from the original input on the same level . However , we observe that previous methods focus only on feature fusion , but lack the inner block connection . Therefore , we design a novel pathway fusion including skip-layer and cross-scale connections , as shown in Figure 3 ( d ) . Skip-layer Connection . Compared to other connection methods , skip connections have short distances among feature layers during back-propagation . In order to reduce gradient vanish in such a heavy “ giraffe ” neck , we propose two feature link methods : dense-link and log2n-link in our proposed GFPN , as shown in Figure 4 . • dense-link : Inspired by DenseNet ( Huang et al. , 2017 ) , for each scale feature P lk in level k , Consequently , the lth layer receives the feature-maps of all preceding layers : P lk = Conv ( Concat ( P 0 k , . . . , P l−1 k ) ) , ( 1 ) where Concat ( ) refers to the concatenation of the feature-maps produced in all preceding layers , and Conv ( ) represents a 3x3 convolution . • log2n-link : Specifically , in each level k , the lth layer receives the feature-maps from at most log2l + 1 number of preceding layers , and these input layers are exponentially apart from depth i with base 2 , as denoted : P lk = Conv ( Concat ( P l−2n k , . . . , P l−21 k , P l−20 k ) ) , ( 2 ) where l− 2n ≥ 0 , Concat ( ) and Conv ( ) also represent concatenation and 3x3 convolution respectively . Compare to dense-link at depth l , the time complexity of log2n-link only cost O ( l · log2l ) , instead of O ( l2 ) . Moreover , log2n-link only increase the short distances among layers during backpropagation from 1 to 1+log2l . Hence , log2n-link can scale to deeper networks . Cross-scale Connection . Based on our assumption , our designed sufficient information exchange should contains not only skip-layer connection , but also cross-scale connection , to overcome largescale variation . Previous works in connecting features between adjacent layers only consider same level feature ( Liu et al. , 2018 ) or previous level feature ( Tan et al. , 2020 ) . Therefore , we propose a new cross-scale fusion named as Queen-fusion , that considering both same level and neighbor level features as shown in Figure 3 ( d ) , like playing the queen piece in chess . As an example shown in Figure 5 ( b ) , the concatenation of Queen-fusion in P5 consists previous layer P4 down-sampling , previous layer P6 up-sampling , previous layer P5 and current layer P4 . In this work , we apply bilinear interpolation and max-pooling as our up-sampling and down-sampling functions respectively . Therefore , in the extreme large-scale variation scenario , it requires that the model has sufficient high-level and low-level information exchange . Based on the mechanism of our skip-layer and cross-scale connections , the proposed generalized-FPN can be expanded as long as possible , just like the “ giraffe neck ” . With such a “ heavy neck ” and a lightweight backbone , our GiraffeDet can balance higher accuracy and better efficiency trade-off .

Summary: This paper proposes a new network architecture called GiraffeDet. GiraffeDet consists of a light backbone, a heavy neck and a prediction head. In the backbone, the paper proposes the S2D (spatial to depth) structure, claiming its better efficiency. In the neck, the paper proposes denser connections based on FPN, taking cross-scale information exchange into account. The experimental results show the state-of-the-art accuracy compared with other single stage object detection framework on the same FLOP level.