MSFM: Multi-Scale Fusion Module for Object Detection
Authors: Xuesong Wang
Caisheng Wang
Venue:  (2021)
Keywords: Feature Fusion
Object Detection
Multi-Scale

Abstract: Feature fusion is beneficial to object detection tasks in two folds. On one hand, detail and position information can be combined with semantic information when high and low-resolution features from shallow and deep layers are fused. On the other hand, objects can be detected in different scales, which improves the robustness of the framework. In this work, we present a Multi-Scale Fusion Module (MSFM) that extracts both detail and semantical information from a single input but at different scales within the same layer. Specifically, the input of the module will be resized into different scales on which position and semantic information will be processed, and then they will be rescaled back and combined with the module input. The MSFM is lightweight and can be used as a drop-in layer to many existing object detection frameworks. Experiments show that MSFM can bring +2.5% mAP improvement with only 2.4M extra parameters on Faster R-CNN with ResNet-50 FPN backbone on COCO Object Detection minival set, outperforming that with ResNet-101 FPN backbone without the module which obtains +2.0% mAP with 19.0M extra parameters. The best resulting model achieves a 45.7% mAP on test-dev set. Code will be available.

Under review as a conference paper at ICLR 2021
MSFM: MULTI-SCALE FUSION MODULE FOR OBJECT DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Feature fusion is beneficial to object detection tasks in two folds. On one hand, detail and position information can be combined with semantic information when high and low-resolution features from shallow and deep layers are fused. On the other hand, objects can be detected in different scales, which improves the robust- ness of the framework. In this work, we present Multi-Scale Fusion Module (MSFM) that extracts both detail and semantical information from a single input but at different scales within the same layer. Specifically, the input of the module will resized into different scales on which position and semantic information will be processed, and then they will be rescaled back and combined with the mod- ule input. The MSFM is lightweight and can be used as a drop-in layer to many existing object detection frameworks. Experiments show that MSFM can bring +2.5% mAP improvement with only 2.4M extra parameters on Faster R-CNN with Re FPN backbone COCO Object Detection mi minival set, out- performing that with ResNet- 101 FPN backbone without the module which ob- tains +2.0% mAp with 19.0M extra parameters. The best resulting model achieves a 45.7% mAp on cest-dev set. Code will be available.
INTRODUCTION
Object detection is one of the fundamental tasks in computer vision. requires the detector to localize the objects in the image using bounding boxes and assign the correct category to each of them. In recent years, deep convolutional neural networks (CNNs) have seen great success object detection, which can be divided into two categories: two-stage detectors, e.g., Faster R-CNN (Ren) (ct al. 2015), and one-stage detectors, e.g., SSD (Liu et al. 2016). Two-stage detectors have high localization and recognition accuracy, while one-stage detectors achieve high inference speed (Jiao 2019). A typical two-stage detector consists of backbone, neck, a Region Proposal Network ct al. (RPN), and Region of Interest (ROI) head (Chen al. 2019). A backbone is feature extractor usually pre-trained on ImageNet dataset (Deng et al. 2009). A neck could be a Feature Pyramid Network (FPN) (Lin al.) 2017a) that fuses the features from multiple layers. A RPN proposes candidate object bounding boxes, and a ROI head is for box regression and classification (Ren et al.) 2015). Compared to two-stage detectors, one-stage detectors propose predicted bounding boxes directly from the input image without the region proposal step, thus being more efficient (Jiao et al.) 2019
One of the key challenges in object detection is to solve the two subtasks, namely localization and classification, coordinately. Localization requires the network capture the object position accu- rately, while classification expects the network to extract the semantic information of the objects. Due to the layered structure of the CNNs, detail and position-accuran information resides in shal- low but high-resolution layers; however, high-level and semantically strong information exists in deep but low-resolution layers (Long et al. 2014). Another key challenge is scale invariance that the detector is expected to be capable of handling different object scales (Liu al. 2016).
Feature Fusion beneficial to object detectors in solving the two challenges. On one hand, through multi-layer fusion (Chen et al. 2020), detail and position information be combined with seman- tie information when high and low-resolution features from shallow and deep layers are fused. On the other hand, by fusing the results from different receptive fields Yuu & Koluun 2016) scales

---

Under review as a conference paper at ICLR 2021
Li al. 2019) via dilated convolutions different kernel sizes (Szegeyy et al. 2014), objects can be detected in different scales, which improves the robustness of the model.
In this paper, present Multi-Scale Fusion Module (MSFM) that extracts both detail and seman- tical information from a single imput but at different scales within the same layer. Specifically, the input of the module will be resized into different scales on which position and semantic informa- tion will be processed, and then they will be rescaled back and combined with the module input.
The MSFM is lightweight and can be used a drop-in layer many existing object detection frameworks, complementing shallow and deep layers with semantic and position information.
Experiments show that MSFM bring +2.5% mAP improvement with only 2.4M extra parameters on Faster R-CNN with ResNet-50 FPN backbone on COCO Object Detection (Lin et al., 2014) minival set, outperforming that with ResNet-101 FPN backbone without the module which obtains +2.0% mAP with 19.0M extra parameters. When applied on other frameworks, it also shows about +2.0% mAP improvement, which show its generalizability. The best resulting model achieves 45.7% mAP on
set.
2 RELATED WORK
2.1 MULTI-LAYER FEATURE FUSION
FPN (Lin et al., 2017a) is the de facto multi-layer feature fusion module in modern CNNs to com- pensate for the position information loss in the deep layer and lack of semantic information in shallow layers. By upsampling the deep features and fusing them with shallow features through top-down path, it enables the model to coordinate the heterogenous information and enhances the robustness. NAS-FPN (Ghiasi et al. 2019) designs NAS (Zoph & Le 2017) search space that covers all possible cross-layer connections, the result of which is a laterally repeatable FPN struc- ture sharing the same dimensions between its input and output. FPG (Chen al., 2020) proposes a multi multi-pathway feature pyramid, representing the feature scale-space as a regular grid of parallel bottom-up pathways fused by multi-directional lateral connections. EfficientDet (Tan et al. 2020 adopts a weighted bi-directional feature pyramid network for multi-layer feature fusion. M2Det (Zhao al. 2018) presents multi-level feature pyramid network, fusing the features with the same depth and dimension from multiple sequentially connected hourglass-lik like modules to generate multi-scale feature groups for prediction. Similar structures can also be seen in DSSD (Fu et al.)
2017
TDM (Shrivastava et al. 2016), YOLOv3 (Redmon & Farhadi] 2018), and RefineDet (Zhang
et al. 2017).
2.2 MULTI-BRANCH FEATURE FUSION
In Inception (Szegedy et al. 2014), kernlls Inception Module branches have different sizes, which makes the output of the module contain different receptive fields. However, a large kernel contains a large number of parameters. Instead, dilated convolution allows kernel have an enlarged receptive field while keeping the parameter size unchanged. MCA (Yu & Koltun (016) utilizes dilated convolutions to systematically aggregate multi-scale contextual information. Going even further, TridentNet (Li et al.] (219)) lets multiple convolutions share the same weight but with different dilation rates to explore a uniform representational capability.
MULTI-SCALE FUSION MODULE
In this section, we present our Multi-Scale Fusion Module (MSFM) and the possible configurations when inserting it into existing frameworks.
3.1 MODULE DEFINITION
An instantiation of MSFM shown in Figure la It can be formulated as follows:
M(x) = x (SC))
2

---

Under review as a conference paper at ICLR 2021
where z is the module input, (x) is the module output, S() is the squeeze module that makes the input x thinner, () is the operation on n-th branch, C( is the combination function, and U() is the unsqueeze module which will restore the depth of the branch output to make it the same x. The branch operation F 7)) can be represented as below:
F (a) =
where = S(x) is the result of squeeze module, R, () is the resize function n-th branch, CGNN., the i-th {Con2D = GroupN ormalization = N on Linearity) operation on n-th branch, R, is the resize function to restore the feature dimension (height and width).
To make the module lightweight, we utilize a bottleneck-like (He et al., 2015) structure where the module input will first be thinned channel-wise, then fed into the branches. Branch input is resized using bilinear interpolation, and the same method used when resizing the feature back to its original size. All the 3x3 convolutions on the branches have the padding=1 to keep the spatial dimension unchanged, and the number of the output channel is the same as that of the input channel well. We choose ReLU the nonlinearity activation in the MSFM. By default, MSFM inserted in stages 2, 3, and 4 for ResNet backbones (He et al. 2015)
Radiee
3x3, GN,N
3x3, GN,
Reaiier
255-4
1xl. 04, HALL
Resliee
3x3, GN, N
3xd, GN.
Resize
aftr convt
1x1, BN
1x1,
aterr 0317
1:1,, 256, BN
Resize
3x0, GN,
Resize
(a) MSFM.
(b) Residual Bottleneck.
Figure 1: MSFM and Residual Bottleneck. BN=Batch Normalization (Toffe & Szegedy 2015 N=NooL.inearity GN=Group Normalization Convolutional with padding=1.
(Wu & 2018). 1x1=1xi Convolution, 3x3=3x3
3.2 CONFIGURATIONN
MSFM acts drop-in layer to existing frameworks. To show several possible configurations when inserting it into an object detector, we take as an example inserting it into a ResNet backbone. A Residual Bottleneck (He et al. 2015) ResNet (He et al. (016) is shown in Figure Tbb Some
tunable hyperparameters we can configure are listed in Table
Table 1: Tunable hyperparameters
Name
Description
Position
Insertion position of the MSFM, fiter_convv 1, conterrcoov2ee or after_conv3
Scales
Scales used to resize the module input all the branches
Ratios
Squeeze ratios used by the first Ixl Conv to make the module input thinner Number of groups separate the channels into for Group Normalization Number of (Conv2D, Group Normalization, Nonlincarity] } on each branch
Norm Norm-group group
Conv number
Fusion type
Combination method used to fuse the branch results, add or concatenation
4 EXPERIMENTS
To evaluate the proposed module, we carry out experiments on object detection and instance seg- mentation tasks COCO (L et al. 2014). Experimental results demonstrate that the MSFM can
3

---

Under review as a conference paper at ICLR 2021
enhance the performance of common two-stage object detection frameworks with very light com- putational overhead.
4.1
EXPERIMENTS SETUP
We perform hyperparameter tuning on Faster R-CNN with ResNet-50 FPN backbone (Ren al.
2013).
Unless otherwise stated, the backbone of the framework being mentioned is ResNet-50 MSFM, experiments are also conducted on Faster R-CNN with ResNet-101 FPN backbone (Ren et al. 2015), Mask R-CNN (He et al., 2017). Cascade R-CNN
FPN. To test the generalizability
(Cai & Vasconcelos
2017). Grid R-CNN (Lu et al. 2018), Dynamic R-CNN (Zhang et al.) 2020).
RetinaNct (Lin al. 2017b), Reppoints Yang et al. (019), and Faster R-CNN with ResNet-50 FPN
and Deformable Convolution on c3-c5 (Dai et al. (017). We carry out our experiments on object detection and instance segmentation tasks on COCO at.. 2014), whose train set contains 118k images, mi nival set 5k images, and test dev set 20k images. Mean average-precision (mAP) scores different boxes and mask loUs adopted the metries when evaluating object detection and instance segmentation tasks.
Our experiments are implemented with PyTorch (Paszke et al., (211)) and MMDetection (Chen al., 2019). The input images are resized such that the shorter side is no longer than 800 pixels. and the longer side no longer than 1333 pixels. All the models are trained 8 GPUs with images per GPU. The backbone of all models are pretrained on ImageNet classification dataset (Deng et al., 2009). Unless otherwise stated, all models are trained for 12 epochs using SGD with a weight decay of 0.0001, and a momentum of 0.9, The learning rate is set to 0.02 initially and decays by factor of 10 at 8th and Ith epochs. Learning rate linear warmup adopted for first 500 steps with a warmup ratio of 0.001.
4.2 ABLATION STUDIES
The ablation studies are performed on COCO 2017 (Lin et al. 2014) minival set.
Unless otherwise stated, the MSFM in the following experiments has the default configuration: the insertion position is after conv3, the resize scales of three branches are 0.5, 0.7, and 1, respectively, the squeeze ratios are 16, 32, and 64 for stage 2, 3, and 4 of ResNet-50 (He al., 2015). respectively, the number of groups in Group Normalization (Wu & He 2018) is 16, only one [Conv2D, Group Normalization, Nonlinearity] operation adopted on branches, and the method combine the branch results is add.
4.2.1 SCALES
As can be seen from Table 2 Scales part. small scales (3S=[0.5, 0.7, 11, 5S=[0.5, 0.6, 0.7, 0.85, 1J) are helpful for detecting large objects, while large scales (31=(1, 1.4, 2]) can enhance the detection of small objects. Compared to only using small or large scales, using compound scales (4=(0.5, 0.7, 1.4, 2). 5=[0.5, 0.7, 1, 1.4, 2D) turn out to be the optimal option, which can achieve better overall performance. This indicates that simultancously generating and inserting detail and semantic information to the same layer is beneficial.
4.2.2 RATIOS
We compare the effect of different squeeze ratios for different insertion positions, shown in Table P Ratios part. For position=after.comm/ information loss but less computational overhead; therefore, the ratios of 16, 32, and 64 for stages 2, 3 and 4, respectively, can be good trade-off between information loss and computational overhead. For postion=after cconvl (norm_gre group=8), MSFM not sensitive to the change of ratios. We guess that it might be because the channel number is already so low after convl that changing its channel number will have no further effect.
increase the ratios, the model will experience more
4.2.3 NORM_GROUP
We explore the optimal group number for Group Normalization (Wu & He 2018) when inserting into different positions. As from the Norm group part Table 2 the best group number
4

---

Under review as a conference paper at ICLR 2021
Table 2: Ablation Studies
Name
AP
AP 58.4
AP75 40.4
AP
AP 41.0
AP; 47.9
#Param 41.5M
50
Faster-RCNN R50 FPN
37.4
21.4
Name
Scales
AP
APso
AP73
AP.
AP m
APt
#Param
3S
38.8
60.0
42.3
22.6
42.6
49.9
42.9M
5S
38.9
60.1
42.3
23.0
42.4
50.3
43.1M
Scales
3L
38.6
59.8
41.8
23.0
42.6
49.5
42.8M
4
39
60.3
42.3
22.2
42.8
49.7
43.0M
5
38.9
59.9
42.4
22.7
42.7
50.3
43.1M
Name
Ratios
Pos
AP
AP30
30
AP
AP
AP
AP;
#Param
8,16,32
3
39.1
60.1
42.6
22.8
42.8
50.4
44.9M
16,32,64
3
38.8
60.0
42.3
22.6
42.6
49.9
42.9M
32,64,128
3
38.7
59.8
42.0
22.5
42.5
50.1
42.1M
Ratios
4,8,16
38.9
60.0
42.3
22.7
42.5
50.3
42.1M
8,16,32
1
38.9
59.9
42.5
22.1
42.6
50.5
41.8M
16,32.64
38.9
59.8
42.4
22.9
42.7
50.1
41.6M
Name
#Group
Pos
AP
AP
AP
AP
AP
AP,
#Param
3
38.7
59.8
42.4
22.5
42.4
49.8
42.9M
8
3
38.9
59.8
42.1
22.2
42.5
50.4
42.9M
16
3
38.8
60.0
42.3
22.6
42.6
49.9
42.9M
32
3
39.1
60.2
42.5
23.0
42.8
50.3
42.9M
Norm
38.6
59.5
42.0
22.4
42.1
50.2
41.6M
group
4
2
38.8
59.8
42.3
22.9
42.1
50.4
41.6M
2
38.7
59.8
42.0
22.5
42.6
50.0
41.6M
1
38.5
59.4
42.4
22.1
42.3
49.8
41.6M
1
38.8
59.8
42.5
22.8
42.4
50.0
41.6M
1
38.9
59.8
42.4
22.9
42.7
50.1
41.6M
Name #Cony
Scales
AP
AP
AP75
AP.
AP
AP,
#Param
50
38
39.1
60.2
42.5
23.0
42.8
50.3
42.9M
2
3S
39.2
60.3
42.5
22.6
42.8
50.7
43.2M
2*
5
38.9
60.2
42.2
23.3
42.7
50.3
43.3M
Conv
2
5
39.6
60.6
43.4
23.7
43.1
51.3
43.7M
num
2*
3L.
38.9
60.2
42.4
23.1
42.8
50.2
43.1M
2
3L
39.0
60.2
42.6
23.3
42.6
49.9
43.2M
2
4
39.2
60.3
42.6
23.2
43.0
50.6
43.2M
2
4
39.2
60.1
42.7
22.7
42.9
50.6
43.5M
Name Type
Pos
AP
APso
AP
AP
AP
AP:
#Param
m
add
3
38.8
60.0
42.3
22.6
42.6
49.9
42.9M
Fusion type
cat
3
39.0
60.2
42.3
22.3
42.8
50.5
43.8M
add
38.9
59.8
42.4
22.9
42.7
50.1
41.6M
cat
39.1
60.1
42.7
23.2
42.7
50.7
41.7M
for convv,, after_conv2 and after are 32, 4, and 8, respectively. Because the channel number is much larger for conv3 compared after_conv' and after,coov2, the group number for Group Normalization (Wu & He 2018) is much larger for after.conv3.
4.2.4 CONV_NUM
All the experiments of Conv num in Table 2 are conducted with Norm group=32. ** indicates that only the branches with scales larger than have 2 {Cony2D, Group Normalization, Nonlinearity)

---

Under review as a conference paper at ICLR 2021
operations. As see, the model with scale=[0.5, 0.7, 1, 1.4, 2] and conv -nmm=2mm2222m achieves the best performance. What's more, all the models of conv conv_num= achieves better or at least comparable performance with that of conv num=2*, which indicates that coordinate representational power among all the branches is important, even though they do not have the same receptive field size.
4.2.5 FUSION TYPE
As two typical feature fusion operations, add and concatenation are alternatives. We compare their effects in the models of position=after convl and the ones of position=after conv3. The results in Table show that concatenation slightly better than add.
4.2.6 MULTI-POSITION INSERTION
According to the experiment results and analysis above, we carry out a multi-position insertion ablation study, in order periments in this part have the following configurations for all the models: the resize scales of all the branches 0.5, 0.7, 1.4, and the squeeze ratios for stage 2, and are 16, 32, and 64, spectively, the number of [Conv2D, Group Normalization, Nonlinearity) operations on all branches and the combination method add. The number of groups used in Group Normalization (Wul & He 2018) is 8, 4, and 32 for after convl, after conv2, and after conv3, respectively. As can be seen from the results in Table 4 the combination of after conv2 and after conv3 turns out the best configuration, which we will use as the default configuration when applying the MSFM to other frameworks.
the effect of MSFM being inserted in multiple positions. All the
Table 3: Mutli-position insertion
Position
AP
APg
AP7s
AP
AP
AP;
#Param
50
1,2
39.3
60.2
42.8
23.1
43.0
50.9
41.8M
1,3
39.3
60.2
42.8
22.9
43.2
50.7
43.9M
2,
39.9
61.0
43.5
23.5
43.7
51.6
43.9M
1,
39.3
60.4
42.6
22.8
42.9
50.6
44.0M
Table 4: Mutli-position insertion for object detection.
"indicates with MSFM.
Framework
AP
AP
AP
AP
AP m
AP;
#Param
Faster R-CNN
37.4
58.4
40.4
21.4
41.0
47.9
41.5M
Faster R-CNN#
39.9
61.0
43.5
23.5
43.7
51.6
43.9M
Cascade R-CNN
40.4
58.7
44.2
22.7
43.8
53.0
69.2M
Cascade R-CNN#
42.6
61.5
46.6
24.9
46.3
56.3
71.5M
Grid R-CNN
39.1
57.3
42.3
22.5
43.0
50.0
64.3M
Grid R-CNN*
41.2
60.0
44.4
24.0
45.0
52.2
66.7M
Dynamic R-CNN
38.9
57.5
42.5
21.4
42.5
51.4
41.5M
Dynamic R-CNN#
40.6
59.3
44.3
23.8
43.7
53.8
43.9M
Faster R101 FPN
39.3
60.0
42.8
22.2
43.5
51.3
60.5M
Faster R101 FPN*
41.2
62.1
44.9
23.6
45.5
53.9
65.8M
RetinaNet
36.4
55.3
38.8
20.7
40.0
47.1
37.7M
RetinaNet
38.7
58.1
41.4
22.7
42.5
50.9
40.1M
Faster R-CNN Dcony
41.2
62.5
45.1
24.1
44.8
54.8
42.1M
Faster R-CNN Dconv*
42.1
63.6
46.0
25.3
45.7
54.8
44.4M
Reppoints
36.8
56.5
39.6
20.8
41.0
48.6
36.6M
Reppoints
38.5
58.7
41.5
22.4
43.2
50.4
38.9M
Mask R-CNN
38.1
58.6
41.6
21.7
41.5
49.3
44.2M
Mask R- R
40.3
61.1
43.9
23.1
44.0
52.4
46.5M

---

Under review as a conference paper at ICLR 2021
Table 5: Mutli-position insertion for instance segmentation. iindicates with MSFM.
Framework
A APm
A
A P5
A A
A APM
AP
#Param
Mask R-CNN
34.5
55.5
37.0
18.0
37.6
46.9
44.2M
Mask R-CNN#
36.3
57.9
38.8
19.2
39.8
49.2
46.5M
4.3 RESULTING MODELS
To test the generalizability of the proposed MSFM, we apply it multiple frameworks. The results are shown in Table and Table [ For a fair comparison, all baseline models are re-trained. As we can see, there is consistent improvement in the following models when the MSFM is applied, which demonstrates that the MSFM can used as drop-in layer for many existing object detection frameworks. Notice that when MSFM is applied to Faster R-CNN with ResNet FPN backbone (Ren et al. 2013), the performance the model even surpasses the one with ResNet- 101 FPN backbone. indicates that adding the MSFM to existing frameworks is more efficient than just adding more convolutional layers.
We also train Cascade R-CNN with ResNet-101 FPN backbone for 24 epochs using multi-scale training and submit the results to the evaluation server. The result in Table 6 shows it achieves a 45.7% mAP on the test-dev set.
Table 6: Result of Cascade R-CNN with ResNet-101 FPN backbone trained for 24 epochs with multi-scale training.
Dataset
Apm
Apm
APm
APm
Apm
AP..
#Param
minival
45.4
64.3
49.6
27.8
49.3
58.9
93.4M
test-dev
45.7
65.0
49.8
27.4
48.7
57.1
CONCLUSION
In this paper, we have presented Multi-Scale Fusion Module (MSFM) that extracts both detail and semantical information from a single input but at different scales within the same layer. Ab- lation studies have demonstrated that MSFM can bring +2.5% mAP improvement with only 2.4M extra parameters on Faster R-CNN with ResNet-50 FPN backbone on COCO Object Detection minival set, outperforming that with ResNet-101 FPN backbone without the module which ob- tains +2.0% mAP with 19.0M extra parameters. The best resulting model on Cascade R-CNN with ResNet- 101 FPN backbone achieved 45.7% mAP on COCO Object Detection test-dev set.
REFERENCES
Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: delving into high quality object detection. CoRR, abs/1712.00726, 2017. URL htttp::/awx...org////
.07266
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng. Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and 2019.
benchmark. arXiv preprint 1906v
Kai Chen, Yuhang Cao, Chen Change Loy, Dahua Lin, and Christoph Feichtenhofer. Feature pyra- mid grids, 2020.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. CoRR, 703.0113,,,,,,,, 2017. URL http: / /ariivivvvv...//vv/./avaaaavvv.v org/ abs/ 1703. 06211

---

Under review as a conference paper at ICLR 2021
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009). 20-25 June 2009, Miami, Florida, USA. pp. 248-255. IEEE Computer Society, 2009. doi: 10.1 POO/////...000..222222
Theng-Yang, Fu, W. Liu, Ananth Ranga, Ambrish Tyagi, and A. Berg. Dssd Deconvolutional single shot detector. ArXiv, abs/1701 .066590,66 2017.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoe V. Le. NAS-FPN: learning scalable feature pyramid archi- tecture for object detection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 7036-7045. Computer Vision Foun- dation / IEEE, 2019. doi: 10.1 2011009/CVR.221..0000
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, 151./335,, 2015. htt.://ppp/aaaaa...orr/////2//22 03385
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. CoRR, abs/1603.050027 2016.
05027
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. CoRR, abs/1703.0687 2017. URL httt::ttt//aaatt
org/ abs abs/ 06870
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015, URL http:://axxii.orrg
abs/1502. 03167
Licheng Jiao, Fan Zhang, Fang Liu, Shuyuan Yang, Lingling Li, Zhixi Feng, and Rong Qu A survey of deep learning- barning-based object detection. CoRR, 1is/1997.09408 2019.
Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Scale-aware trident networks for object detection. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2019.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/l Abss/145.0031,, 2014. URL 1405.0312
Tsung-Yi Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), July 2017a.
Tsung-Y Yigg--iiiiii Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. CoRR, 0os/1708.002 708.02002 2017b. URL 02002]
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng Yang Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.). Computer Vision ECCV 2016, pp. 21-37, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46448-0)
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. CORR, abs/1411.4038, 2014. URL
.11..003
Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan. Grid R-CNN. CoRR, 2018. URL httpttt::///wii.rrrttee
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bi Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
8

---

Under review as a conference paper at ICLR 2021
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. CoRR, abs//a00..00 1804.02767 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 91-99. Curran Associates, Inc., 2015.
Abhinav Shrivastava, Rahul Sukthankar, Jitendra Malik, and Abhinav Gupta. Beyond skip con- nections: Top-down modulation for object detection. CoRR, abs/1612.06851, 2016. URL
http://arxiva
06855
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR, Abs/1449.4842 2014. http::::/aaaii..r/////aa/aa
4842
Mingxing Tan, Ruoming Pang, and Quoc V. Le. Efficientdet: Scalable and efficient object detection, 2020.
Yuxin Wu and Kaiming He. Group normalization. CoRR, abs/1 01118803..4444 2018. URL http:
08494
Ze Yang. Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin. Reppoints: Point set representation for object detection. CoRR, abs/1904. 11490, 2019. URL http:tttp://arrii.org//ab//////s
11490
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In Interna- tional Conference on Learning Representations (ICLR), May 2016.
Hongkai Zhang, Hong Chang, Bingpeng Ma, Naiyan Wang, and Xilin Chen. Dynamic r-cnn: To- wards high quality object detection via dynamic training, 2020.
Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, and Stan Z. Li. Single-shot refinement neural network for object detection. CoRR, abs/1711.00699 2017. URL.pttpp///arxiv.orrr/aa.
1711.06897
Qijie Zhao, Tao Sheng, Yongtao Wang, Zhi Tang, Ying Chen, Ling Cai, and Haibin Ling. M2det: A single-shot object detector based on leeei- feature pyramid network. CoRR, abs/18 8.1.1.6 11.04353, 2018. U
1811.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. 2017. URL
httpss:/aaae org/ 01578


Reviews:
[{"review_id": "yHGFg6cjMt8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to obtain multi-scale features by `resize -> convolution -> resize (inverse)’. Extensive experimental results on COCO validate the effectiveness of the proposed approach. \n\nPros:\n\nExperimental results on the widely used benchmarking dataset.\nComparison with recent approaches.\nPaper is easy to understand, simply because the proposed method is simple.\n\nCons:\nLack of novelty. The use of multi-scale features is not new [a, b]. The formulation in this paper is not very different from that in [b]. It has been used for lots of applications. In object detection, dilated conv in TridentNet is not the only approach using multiple branches. Approaches like [c, d] also used multi-scale-multi-layer features by resizing. \n\n[a] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. \"Spatial pyramid pooling in deep convolutional networks for visual recognition.\" IEEE transactions on pattern analysis and machine intelligence 37, no. 9 (2015): 1904-1916.\n\n[b] Yang, Wei, Shuang Li, Wanli Ouyang, Hongsheng Li, and Xiaogang Wang. \"Learning feature pyramids for human pose estimation.\" In proceedings of the IEEE international conference on computer vision, pp. 1281-1290. 2017.\n\n[c] Gidaris, Spyros, and Nikos Komodakis. \"Object detection via a multi-region and semantic segmentation-aware cnn model.\" In Proceedings of the IEEE international conference on computer vision, pp. 1134-1142. 2015.\n\n[d] Zeng, Xingyu, Wanli Ouyang, Junjie Yan, Hongsheng Li, Tong Xiao, Kun Wang, Yu Liu et al. \"Crafting gbd-net for object detection.\" IEEE transactions on pattern analysis and machine intelligence 40, no. 9 (2017): 2109-2123.\n\n\nThe ablation study in the experimental results did not compare with existing works, like TridentNet, and [c, d] to justify why another multi-scale approach is needed.\n\nSymbols are not illustrated well (Authors need not answer this in the rebuttal but need to revise in the revised version).\n`S() is the squeeze module: What is the meaning of squeeze module?\nThere are no common definitions on `makes the input x thinner’, `combination function’, `unsqueeze module’.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "n2YSiKuodII", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new general feature fusion operation, Multi-Scale Fusion Module (MSFM). By adding MSFM layers between feature extraction layers, it is observed that the detection result is improved with minor added parameters.\n\nPros:\nGood to see new work to explore various of ways to perform feature fusion.\n\nCons:\nMajor comments:\n(1) The quality and clarify of the paper needs to be improved, for example, Table 3 has shifted horizontal line. \n(2) Ablation studies clarifies on the effects of the change of configurations, but does provide much evidence on why MSFM module helps with the detection task.\n\nSome minor comments:\n(1) Figure1a. could be further clarified by adding the notations mentioned in the equations to to the figure. \n(2) Good to report the variances/confidence-intervals of the metrics as well.", "extra_scores": {}, "extra_text": {}}, {"review_id": "csoADX-1jKc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a multi-scale feature fusion block and inserts the block into ResNet backbones for object detection. It is very similar to the inception block in IneceptionNets. The only difference is the proposed feature fusion contains feature map upsampling and downsampling (resize and resize^{-1}) for different branches. The paper has some merits as follows.\n\n1. The method has evaluated on different object detection frameworks, such as Faster R-CNN, Cascade R-CNN, Grid R-CNN, Dynamic R-CNN, RepPoints, etc.\n2. The method obvious performance gains on different frameworks with a few additional parameters.\n\nHowever, the flaws are obvious as follows.\n1. The novelty is very limited. Inception block is widely used in deep learning. The paper is only a small modification over Inception. The novelty is far below the bar of ICLR.\n2. The proposed feature fusion block is very computation expensive. The upsampling procedure makes the computation cost of the 3$\\times$3 convolutions very high. The paper only reports additional parameters but does not report the additional computation cost (e.g., additional FLOPs and running time).\n3. Related methods are not compared. At least, inserting inception blocks into ResNets can be compared.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bYfKpqYXuRD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors study the problem of scale-friendly feature fusion for object detection. Specifically, the authors propose to process features at each layer of a feature pyramid network at multiple scales and fuse them back into a single scale. To be specific, they resize features at a layer into multiple scales, process these rescaled features independently, rescale them back into the original scale and combine them with the original features.\n\nStrengths:\n- Scale is an important problem in object detection and the paper addresses an important issue.\n- Strong results showing significant improvements, around ~2AP, over baselines, including strong detectors like RepPoints.\n- Overall, the paper is very well written. I didn't find any typos or grammatical errors, which is very rare for a thorough reviewer like me. \n\nWeaknesses:\n- The novelty is limited. Multi-scale processing at a layer has been extensively studied with ResNext-type and inception-type architectures. The paper just takes such ideas and use them with FPN without any technical or theoretical insights or contributions.\n\nIn other words, this appears to be just FPN with inception module. It would have been nicer for the authors to motivate how/whether the novelty goes beyond this.\n\n- The paper does not make a comparison with multi-scale backbone networks such as ResNext. It has been shown that such multi-scale architectures improve the detection performance compared to ResNet type architectures, and this comparison is very crucial for the reader to grasp the significance of the contribution, if any.\n\n- The paper does not make a comparison with respect to methods trying to address limitations of FPN. There are many papers that extend FPN to address scaling issues. It is very crucial for the reader to see how the proposed solution performs in comparison with those methods.\n\n- It would have been nicer to see in Table 4 the details on the type of the backbone used. This might be a very crucial factor in analysing the differences in the gap among the different models.\n\n***AFTER AUTHOR RESPONSE***\n\nI have read the comments of the other reviewers, which revealed that all reviewers identified the same major issues with the paper (novelty and evaluation). The authors did not provide a rebuttal but kindly thanked the reviewers and stated their intention for improving the paper with the reviewer comments and submitting it for a future venue. Therefore, I changed my overall rating to rejection.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes an approach for fusing representations at multiple scales to improve object detection systems. Reviewers thought the paper was well-written and showed positive results on COCO, a common object detection benchmark. However, reviewers agreed that there was not sufficient methodological novelty or empirical improvement over existing approaches to warrant acceptance at ICLR: several prior works have addressed multiscale fusion and reviewers did not find the evaluation/ablations sufficient to demonstrate the approach yielded substantial improvements over these existing approaches. I hope the authors will consider resubmitting the paper after refining it based on the reviewers' feedback.
Decision: Reject