MSFM: Multi-Scale Fusion Module for Object Detection

Feature fusion is beneficial to object detection tasks in two folds . On one hand , detail and position information can be combined with semantic information when high and low-resolution features from shallow and deep layers are fused . On the other hand , objects can be detected in different scales , which improves the robustness of the framework . In this work , we present a Multi-Scale Fusion Module ( MSFM ) that extracts both detail and semantical information from a single input but at different scales within the same layer . Specifically , the input of the module will be resized into different scales on which position and semantic information will be processed , and then they will be rescaled back and combined with the module input . The MSFM is lightweight and can be used as a drop-in layer to many existing object detection frameworks . Experiments show that MSFM can bring +2.5 % mAP improvement with only 2.4M extra parameters on Faster R-CNN with ResNet-50 FPN backbone on COCO Object Detection minival set , outperforming that with ResNet-101 FPN backbone without the module which obtains +2.0 % mAP with 19.0M extra parameters . The best resulting model achieves a 45.7 % mAP on test-dev set . Code will be available . 1 INTRODUCTION . Object detection is one of the fundamental tasks in computer vision . It requires the detector to localize the objects in the image using bounding boxes and assign the correct category to each of them . In recent years , deep convolutional neural networks ( CNNs ) have seen great success in object detection , which can be divided into two categories : two-stage detectors , e.g. , Faster R-CNN ( Ren et al. , 2015 ) , and one-stage detectors , e.g. , SSD ( Liu et al. , 2016 ) . Two-stage detectors have high localization and recognition accuracy , while one-stage detectors achieve high inference speed ( Jiao et al. , 2019 ) . A typical two-stage detector consists of a backbone , a neck , a Region Proposal Network ( RPN ) , and a Region of Interest ( ROI ) head ( Chen et al. , 2019 ) . A backbone is a feature extractor usually pre-trained on ImageNet dataset ( Deng et al. , 2009 ) . A neck could be a Feature Pyramid Network ( FPN ) ( Lin et al. , 2017a ) that fuses the features from multiple layers . A RPN proposes candidate object bounding boxes , and a ROI head is for box regression and classification ( Ren et al. , 2015 ) . Compared to two-stage detectors , one-stage detectors propose predicted bounding boxes directly from the input image without the region proposal step , thus being more efficient ( Jiao et al. , 2019 ) . One of the key challenges in object detection is to solve the two subtasks , namely localization and classification , coordinately . Localization requires the network to capture the object position accurately , while classification expects the network to extract the semantic information of the objects . Due to the layered structure of the CNNs , detail and position-accurate information resides in shallow but high-resolution layers ; however , high-level and semantically strong information exists in deep but low-resolution layers ( Long et al. , 2014 ) . Another key challenge is scale invariance that the detector is expected to be capable of handling different object scales ( Liu et al. , 2016 ) . Feature Fusion is beneficial to object detectors in solving the two challenges . On one hand , through multi-layer fusion ( Chen et al. , 2020 ) , detail and position information can be combined with semantic information when high and low-resolution features from shallow and deep layers are fused . On the other hand , by fusing the results from different receptive fields ( Yu & Koltun , 2016 ) or scales ( Li et al. , 2019 ) via dilated convolutions or different kernel sizes ( Szegedy et al. , 2014 ) , objects can be detected in different scales , which improves the robustness of the model . In this paper , we present a Multi-Scale Fusion Module ( MSFM ) that extracts both detail and semantical information from a single input but at different scales within the same layer . Specifically , the input of the module will be resized into different scales on which position and semantic information will be processed , and then they will be rescaled back and combined with the module input . The MSFM is lightweight and can be used as a drop-in layer to many existing object detection frameworks , complementing shallow and deep layers with semantic and position information . Experiments show that MSFM can bring +2.5 % mAP improvement with only 2.4M extra parameters on Faster R-CNN with ResNet-50 FPN backbone on COCO Object Detection ( Lin et al. , 2014 ) minival set , outperforming that with ResNet-101 FPN backbone without the module which obtains +2.0 % mAP with 19.0M extra parameters . When applied on other frameworks , it also shows about +2.0 % mAP improvement , which show its generalizability . The best resulting model achieves a 45.7 % mAP on test-dev set . 2 RELATED WORK . 2.1 MULTI-LAYER FEATURE FUSION . FPN ( Lin et al. , 2017a ) is the de facto multi-layer feature fusion module in modern CNNs to compensate for the position information loss in the deep layer and lack of semantic information in shallow layers . By upsampling the deep features and fusing them with shallow features through a top-down path , it enables the model to coordinate the heterogenous information and enhances the robustness . NAS-FPN ( Ghiasi et al. , 2019 ) designs a NAS ( Zoph & Le , 2017 ) search space that covers all possible cross-layer connections , the result of which is a laterally repeatable FPN structure sharing the same dimensions between its input and output . FPG ( Chen et al. , 2020 ) proposes a multi-pathway feature pyramid , representing the feature scale-space as a regular grid of parallel bottom-up pathways fused by multi-directional lateral connections . EfficientDet ( Tan et al. , 2020 ) adopts a weighted bi-directional feature pyramid network for multi-layer feature fusion . M2Det ( Zhao et al. , 2018 ) presents a multi-level feature pyramid network , fusing the features with the same depth and dimension from multiple sequentially connected hourglass-like modules to generate multi-scale feature groups for prediction . Similar structures can also be seen in DSSD ( Fu et al. , 2017 ) , TDM ( Shrivastava et al. , 2016 ) , YOLOv3 ( Redmon & Farhadi , 2018 ) , and RefineDet ( Zhang et al. , 2017 ) . 2.2 MULTI-BRANCH FEATURE FUSION . In Inception ( Szegedy et al. , 2014 ) , kernels on Inception Module branches have different sizes , which makes the output of the module contain different receptive fields . However , a large kernel contains a large number of parameters . Instead , dilated convolution allows a kernel to have an enlarged receptive field while keeping the parameter size unchanged . MCA ( Yu & Koltun , 2016 ) utilizes dilated convolutions to systematically aggregate multi-scale contextual information . Going even further , TridentNet ( Li et al. , 2019 ) lets multiple convolutions share the same weight but with different dilation rates to explore a uniform representational capability . 3 MULTI-SCALE FUSION MODULE . In this section , we present our Multi-Scale Fusion Module ( MSFM ) and the possible configurations when inserting it into existing frameworks . 3.1 MODULE DEFINITION . An instantiation of MSFM is shown in Figure 1a . It can be formulated as follows : M ( x ) = x+ U { C [ F1 ( S ( x ) ) , F2 ( S ( x ) ) , ... , Fn ( S ( x ) ) ] } where x is the module input , M ( x ) is the module output , S ( ) is the squeeze module that makes the input x thinner , Fn ( ) is the operation on n-th branch , C ( ) is the combination function , and U ( ) is the unsqueeze module which will restore the depth of the branch output to make it the same as x . The branch operation Fn ( ) can be represented as below : Fn ( a ) = R −1 n ( CGNn , i ( CGNn , i−1 ( ... ( CGNn,1 ( Rn ( a ) ) ) ) ) ) where a = S ( x ) is the result of squeeze module , Rn ( ) is the resize function on n-th branch , CGNn , i is the i-th { Conv2D ⇒ GroupNormalization ⇒ NonLinearity } operation on n-th branch , R−1n is the resize function to restore the feature dimension ( height and width ) . To make the module lightweight , we utilize a bottleneck-like ( He et al. , 2015 ) structure where the module input will first be thinned channel-wise , then fed into the branches . Branch input is resized using bilinear interpolation , and the same method is used when resizing the feature back to its original size . All the 3x3 convolutions on the branches have the padding=1 to keep the spatial dimension unchanged , and the number of the output channel is the same as that of the input channel as well . We choose ReLU as the nonlinearity activation in the MSFM . By default , MSFM is inserted in stages 2 , 3 , and 4 for ResNet backbones ( He et al. , 2015 ) . 3.2 CONFIGURATIONS . MSFM acts as a drop-in layer to existing frameworks . To show several possible configurations when inserting it into an object detector , we take as an example inserting it into a ResNet backbone . A Residual Bottleneck ( He et al. , 2015 ) in ResNet ( He et al. , 2016 ) is shown in Figure 1b . Some tunable hyperparameters we can configure are listed in Table 1 . 4 EXPERIMENTS . To evaluate the proposed module , we carry out experiments on object detection and instance segmentation tasks on COCO ( Lin et al. , 2014 ) . Experimental results demonstrate that the MSFM can enhance the performance of common two-stage object detection frameworks with very light computational overhead . 4.1 EXPERIMENTS SETUP . We perform hyperparameter tuning on Faster R-CNN with ResNet-50 FPN backbone ( Ren et al. , 2015 ) . Unless otherwise stated , the backbone of the framework being mentioned is ResNet-50 FPN . To test the generalizability of MSFM , experiments are also conducted on Faster R-CNN with ResNet-101 FPN backbone ( Ren et al. , 2015 ) , Mask R-CNN ( He et al. , 2017 ) , Cascade R-CNN ( Cai & Vasconcelos , 2017 ) , Grid R-CNN ( Lu et al. , 2018 ) , Dynamic R-CNN ( Zhang et al. , 2020 ) , RetinaNet ( Lin et al. , 2017b ) , Reppoints ( Yang et al. , 2019 ) , and Faster R-CNN with ResNet-50 FPN and Deformable Convolution on c3-c5 ( Dai et al. , 2017 ) . We carry out our experiments on object detection and instance segmentation tasks on COCO ( Lin et al. , 2014 ) , whose train set contains 118k images , minival set 5k images , and test-dev set 20k images . Mean average-precision ( mAP ) scores at different boxes and mask IoUs are adopted as the metrics when evaluating object detection and instance segmentation tasks . Our experiments are implemented with PyTorch ( Paszke et al. , 2019 ) and MMDetection ( Chen et al. , 2019 ) . The input images are resized such that the shorter side is no longer than 800 pixels . and the longer side is no longer than 1333 pixels . All the models are trained on 8 GPUs with 2 images per GPU . The backbone of all models are pretrained on ImageNet classification dataset ( Deng et al. , 2009 ) . Unless otherwise stated , all models are trained for 12 epochs using SGD with a weight decay of 0.0001 , and a momentum of 0.9 . The learning rate is set to 0.02 initially and decays by a factor of 10 at the 8th and 11th epochs . Learning rate linear warmup is adopted for first 500 steps with a warmup ratio of 0.001 .

Summary: In this paper, the authors study the problem of scale-friendly feature fusion for object detection. Specifically, the authors propose to process features at each layer of a feature pyramid network at multiple scales and fuse them back into a single scale. To be specific, they resize features at a layer into multiple scales, process these rescaled features independently, rescale them back into the original scale and combine them with the original features.