# Object Detection Algorithm Based on Improved Feature Pyramid

Authors: Bai Yu; Xuhua Pan; Xuefeng Li; Gaohua Liu; Yunpeng Ma
Journal: Scientific Programming
 (2022)

---

## Abstract

Feature pyramid network is widely used in advanced object detection. By simply changing the network connection, the performance of small object detection can be greatly improved without increasing the amount of calculation of the original model. However, the algorithm still has some shortcomings. Therefore, a new attention feature pyramid network (attention PFN) is proposed. Firstly, an improved receptive field module is added, which can make full use of global and local context information. Secondly, the connection mode of the pyramid is further optimized. Deconvolution is used to replace the nearest neighbor interpolation in top-down up-sampling and a channel attention module is added to the horizontal connection to highlight important context information. Finally, adaptive fusion and spatial feature balance are used for each feature pyramid, so that the network can learn the weights of different feature layers. Each pyramid layer contains more discrimination information. Attention PFN is tested on Pascal VOC and MS COCO datasets, respectively. The experiment results revealed that the proposed method has better performance than the original algorithm. Therefore, the attention PFN is an effective algorithm.


---

## Body

## 1. Introduction

In recent years, object detection algorithms based on deep learning have been widely used in various fields, such as face detection, vehicle-pedestrian detection, dangerous goods detection [1], transmission line defect identification [2], and so on. These object detection algorithms can be divided into two main categories. The first type is a two-stage algorithm based on a region proposal network, including Faster R-CNN [3], R-FCN [4], and others. The other type is a regression-based one-stage algorithm, such as YOLO [5], SSD [6], and others. However, no matter which type of algorithm it is, it faces the difficult problem of poor detection for small targets. In order to solve the problem of small target detection, the Feature Pyramid Network (FPN) was proposed in literature [7], which has been widely used in the two-stage detection algorithm (such as Mask-RCNN [8]) and one-stage detection algorithm (such as RetinaNet [9]).Based on the structure of ConvNet, the FPN adopts a top-down and horizontal connection method to transfer the top layer semantic information to the low layer. However, the top layer information will be lost. Features of different scales contain information from different abstract levels, so there are large semantic differences in direct addition, and the formed pyramids at all layers are not further fused. Therefore, various variants of the Feature Pyramid Network have been proposed.In literature [10], all information of a multilevel structure was used to generate a multilevel contextual features pyramid with multiple scales. In literature [11], authors proposed a global information extractor and a local information extractor as a framework for single-stage object detection. Yang et al. [12] used a multilayer feature map stacking method to fuse semantic information and detailed features. PANet [13] added an additional bottom-up path on the basis of FPN to transfer low layer location information to high layer. Libra-RCNN [14] designed a balanced feature pyramid; each layer of the pyramid has the information of all layers. NAS-FPN [15] used neural structure to find irregular characteristic network topology and then repeatedly applied the same block, achieving outstanding results. EfficientDet [16] used EfficientNet as the feature extraction network to design the BiFPN, which has an efficient two-way cross-scale connection and weighted feature fusion with a better precision and efficiency trade-off.This paper proposes an improved feature pyramid network based on the attention mechanism and receptive field. The improved receptive field module (ARFB) is added to the top of the feature extraction layer to obtain global and local context information. In the next place, we improved the connection method of feature layers with different resolutions. For the top-down up-sampling method, the deconvolution method is used to replace the nearest neighbor-interpolation for reducing information loss. In the horizontal connection, a channel attention mechanism is added to the feature layer of the backbone network before the 1 × 1 convolution, which can reduce the secondary channel to emphasize important channel information. Finally, the spatial adaptive fusion and balance mechanism are adopted for each layer of the pyramid. The weights of the fused features at all layers can be learned as well as information on each layer of the feature pyramid is fully enhanced.The principal contributions of the paper are summarized as follows:(1)
The top of the feature extraction layer adopts the ARFB module to obtain ample global and local context information.(2)
Deconvolution is used for up-sampling to reduce the loss of feature information. SE module is added at the horizontal connection of the feature layer of the backbone network to extract effective feature information and reduce the interference of noise information.(3)
The spatial adaptive fusion and balance mechanism is applied to further refine the feature information of each feature layer. It greatly improves the detection ability of the network.


## 2. Fundamental Knowledge

FPN has two paths: the bottom-up path and the top-down path. The bottom-up path is the feed-forward calculation of networks, such as ResNet [17]. The output of each stage is C1,C2,C3,C4,C5, the width and height are halved in turn, and the number of channels is {64, 256, 512, 1024, 2048}.In the top-down path, the output C5 reduces the number of channels to 256 by a1×1 convolution, formingP5. Subsequently, up-sampling is performed by the nearest neighbor interpolation method, and C4 with the same resolution is added to it after a 1 × 1 convolution becomes P4. Repeating the above operations, a top-down path composed of {P5, P4, P3, and P2} is formed, and they all use a further 3 × 3 convolution to avoid the aliasing effect caused by up-sampling. In addition, P5 performs max pooling operation to obtain P6. When FPN is combined with Faster R-CNN, a multiscale detection method is adopted. The RPN network searches for the prospect target proposed area in {P2, P3, P4, P5, P6}, and according to the size of the ROI, select the feature map in {P2, P3, P4, P5} to perform the Fast R-CNN operation to obtain the specific target category and more precise location.However, the FPN has several drawbacks: (1) C5 is located at the top layer. With the deepening of neural networks, the feature extracted by deep convolution has low spatial resolution and position information that can be easily recognized. P5 is formed by channel compression, and then P5 is pooled to form P6, which will cause further loss of characteristic information. (2) The features of different layers contain complex feature information. In the horizontal connection, the direct addition of features at different layers will contain a lot of background noises. It will result in great semantic ambiguity. (3) The formed pyramids at all layers are not further fused effectively.A novel neural network is proposed in this section called attention FPN, the structure of which is shown in Figure1. There are three main improvements in the proposed method: (1) An improved receptive field module is added between C5 and P5 to obtain more context information and supplement P5 and P6. (2) The channel attention mechanism is added to the horizontal connection to enhance the effective position information in the feature map in the path and suppress irrelevant background information. At the same time, we also optimize the up-sampling method. (3) The pyramids at all layers have added the adaptive spatial fusion and balance mechanism. The weight of each layer is learned by spatial attention, and the information of all layers of the pyramid makes a more reasonable balance and enhancement.Figure 1
Attention FPN network structure.


### 2.1. Improved Receptive Field Enhancement Module

Dilated convolution comes from DeepLab [18], which can increase the receptive field without reducing the scale of the feature map and perform well in semantic segmentation. Literature [19] designed a receptive field block RFB in the SSD network, and this structure adopted the multibranch idea of Inception-ResNet [20]. There are different convolution kernels and expansion coefficients on different branches, corresponding to various sizes of targets and receptive fields; their combinations can make full use of context information. As shown in Figure 2, the RFB module can only obtain local context information, so the global context information module GCM is introduced into it as a branch so that global context information and local context information can be well fused; the improved module is named ARFB (augment receptive field block). The specific structure of GCM is shown in Figure 3. First of all, the feature map is globally averaged and pooled to obtain the global feature vector. Then, up-sampling is performed to restore the original size and reduce the number of channels to splice with other branches. In order to prevent information loss, we added residual connections in the input and output feature maps . C5 is connected to P5 through the ARFB module, which can give P5 and P6 rich context information to overcome the deficiency of FPN in large target recognition.Figure 2
augment receptive field block.Figure 3
Global context module.


### 2.2. Improved Top-Down Path and Horizontal Connection

As shown in Figure4, for the top-down path, the original interpolation method lacks flexibility and learning ability, resulting in poor performance. Therefore, the deconvolution method is used to replace nearest neighbor interpolation for improving resolution. Moreover, the deconvolution can adjust the training hyperparameter, which can better express the relationship between different layers of features.Figure 4
Improved pyramid connection structure.In the horizontal connection, the spatial attention SE [21](Squeeze-and-Excitation Networks) module is introduced, which is shown in Figure 5. On the bottom-up path, the SE module is added to enhance the important information in the channel, reducing each feature layer channel to 256. The SE module mainly contains two parts, the input feature map is U=u1,u2,…,uc, and each channel is uc∈RH×W. First of all, the global average pooling is used to obtain the global vector Z, and the calculation formula is shown as follows:(1)Zc=1H×W∑iH∑jWuci,j.Figure 5
Squeeze-and-Excitation Networks.Then,Z is encoded by using the fully connected network to obtain the dependency between each channel and complete the space compression. The calculation formula is as follows:(2)Z^δW1W0Z,where W0∈Rc×c/2,W1∈Rc/2×c, δ is the ReLU activation function, Z^c is obtained through the sigmoid activation function, and the value range is adjusted to [0, 1], which is multiplied by the input feature map to obtain the feature after channel excitation:(3)U^SE=σZ^1u1,σZ^2u2,…,σZ^kuk,σZc^ can be regarded as the importance of each channel. When the network is learning, this module is adaptively tuned well to enhance important channels and suppress irrelevant channels. This module can merge with the feature layer of the top-down path well. Due to the small amount of calculation of the channel attention module, the increased time cost is negligible.


### 2.3. Adaptive Spatial Feature Fusion and Balance Mechanism

Different feature pyramids have different contributions to various target recognition. Inspired by AugFPN [22] and Libra R-CNN, we propose a spatial adaptive fusion and balance mechanism. The mechanism is shown in Figure 6. First, {P2, P3, P4, P5, P6} is integrated into the same scale by adopting up-sampling and down-sampling. They were merged into a branch through concatenation operation subsequently. The other branch remains unchanged. The combined features reduce the number of channels by 1 × 1 convolution. It is performed by 3 × 3 convolutions for feature extraction. Then, sigmoid is used to obtain the weight on each channel. The channels perform split operations. The weight of each feature layer is multiplied by the corresponding feature layer to extract effective features. Finally, obtained feature layer performs concatenation operation and reduces the dimension through 1 × 1 convolution. The subsequent steps are the same as Libra R-CNN. The fused features are, respectively, scaled to the original size, and the corresponding original features are added to avoid the loss of some detailed information. The enhanced feature pyramid {P2′, P3′, P4′, P5′, P6’} is formed. The location information and semantic information of each feature layer have been fully enhanced. It is conducive to the detection of multisize targets.Figure 6
Spatial adaptive feature fusion mechanism.


## 2.1. Improved Receptive Field Enhancement Module

Dilated convolution comes from DeepLab [18], which can increase the receptive field without reducing the scale of the feature map and perform well in semantic segmentation. Literature [19] designed a receptive field block RFB in the SSD network, and this structure adopted the multibranch idea of Inception-ResNet [20]. There are different convolution kernels and expansion coefficients on different branches, corresponding to various sizes of targets and receptive fields; their combinations can make full use of context information. As shown in Figure 2, the RFB module can only obtain local context information, so the global context information module GCM is introduced into it as a branch so that global context information and local context information can be well fused; the improved module is named ARFB (augment receptive field block). The specific structure of GCM is shown in Figure 3. First of all, the feature map is globally averaged and pooled to obtain the global feature vector. Then, up-sampling is performed to restore the original size and reduce the number of channels to splice with other branches. In order to prevent information loss, we added residual connections in the input and output feature maps . C5 is connected to P5 through the ARFB module, which can give P5 and P6 rich context information to overcome the deficiency of FPN in large target recognition.Figure 2
augment receptive field block.Figure 3
Global context module.


## 2.2. Improved Top-Down Path and Horizontal Connection

As shown in Figure4, for the top-down path, the original interpolation method lacks flexibility and learning ability, resulting in poor performance. Therefore, the deconvolution method is used to replace nearest neighbor interpolation for improving resolution. Moreover, the deconvolution can adjust the training hyperparameter, which can better express the relationship between different layers of features.Figure 4
Improved pyramid connection structure.In the horizontal connection, the spatial attention SE [21](Squeeze-and-Excitation Networks) module is introduced, which is shown in Figure 5. On the bottom-up path, the SE module is added to enhance the important information in the channel, reducing each feature layer channel to 256. The SE module mainly contains two parts, the input feature map is U=u1,u2,…,uc, and each channel is uc∈RH×W. First of all, the global average pooling is used to obtain the global vector Z, and the calculation formula is shown as follows:(1)Zc=1H×W∑iH∑jWuci,j.Figure 5
Squeeze-and-Excitation Networks.Then,Z is encoded by using the fully connected network to obtain the dependency between each channel and complete the space compression. The calculation formula is as follows:(2)Z^δW1W0Z,where W0∈Rc×c/2,W1∈Rc/2×c, δ is the ReLU activation function, Z^c is obtained through the sigmoid activation function, and the value range is adjusted to [0, 1], which is multiplied by the input feature map to obtain the feature after channel excitation:(3)U^SE=σZ^1u1,σZ^2u2,…,σZ^kuk,σZc^ can be regarded as the importance of each channel. When the network is learning, this module is adaptively tuned well to enhance important channels and suppress irrelevant channels. This module can merge with the feature layer of the top-down path well. Due to the small amount of calculation of the channel attention module, the increased time cost is negligible.


## 2.3. Adaptive Spatial Feature Fusion and Balance Mechanism

Different feature pyramids have different contributions to various target recognition. Inspired by AugFPN [22] and Libra R-CNN, we propose a spatial adaptive fusion and balance mechanism. The mechanism is shown in Figure 6. First, {P2, P3, P4, P5, P6} is integrated into the same scale by adopting up-sampling and down-sampling. They were merged into a branch through concatenation operation subsequently. The other branch remains unchanged. The combined features reduce the number of channels by 1 × 1 convolution. It is performed by 3 × 3 convolutions for feature extraction. Then, sigmoid is used to obtain the weight on each channel. The channels perform split operations. The weight of each feature layer is multiplied by the corresponding feature layer to extract effective features. Finally, obtained feature layer performs concatenation operation and reduces the dimension through 1 × 1 convolution. The subsequent steps are the same as Libra R-CNN. The fused features are, respectively, scaled to the original size, and the corresponding original features are added to avoid the loss of some detailed information. The enhanced feature pyramid {P2′, P3′, P4′, P5′, P6’} is formed. The location information and semantic information of each feature layer have been fully enhanced. It is conducive to the detection of multisize targets.Figure 6
Spatial adaptive feature fusion mechanism.


## 3. Experimental Results and Analysis

The environment configuration of these experiments is presented as follows: CPU model is Intel Sliver 4210; GPU model is NVIDIA GTX 2080Ti and a total of 4 blocks. The operating system is Ubuntu16.04; the acceleration library is CUDA 10.0 and cuDNN 7.6.5. Based on the deep learning framework of PyTorch 1.4 and the implementation network of python 3.7, experimenting them is carried out separately on the PASCAL VOC dataset and MS COCO dataset.


### 3.1. PASCAL VOC Experiment Results

The PASCAL VOC datasets have a total of 20 categories of targets. We use the training validation sets including 16551 pictures of PASCAL VOC 2007 and PASCAL VOC 2012 for training and the testing set of PASCAL VOC 2007 including 4952 pictures. Stochastic gradient descent (SGD) is used for training, momentum is set to 0.9, the input image is re-sized to 1000 × 600, the batch size is set to 16, the weight attenuation coefficient is set to 0.0005, the initial learning rate is set to 0.02, and the maximum number of iterations epoch is 5. At epoch 3, the learning rate is multiplied by 0.1.The results of our algorithm on the VOC2007 testing set are shown in Table1 (with ∗ indicating the reimplemented version of PyTorch). We use Faster R-CNN + FPN as the baseline. It can be seen from the result that when ResNet-50 and ResNet-101 are used as feature extraction networks, respectively, we use attention FPN (abbreviated as AFPN in the table) to replace FPN, which not only increases mAP by 2.8% and 1.9%, respectively, but also has a small speed loss. The accuracy of ResNet-50 exceeds that of most algorithms in the table, while the recognition accuracy of ResNet-100 is the highest among all algorithms. Table 2 shows the specific results of 20 categories. Our model obtains the best accuracy in multiple categories. Compared with FPN, attention-FPN has an obvious improvement effect on small targets (such as monitors, water cups, bottles, cows, sheep, and so on).Table 1
PASCAL VOC 2007 test detection results.
Two-stageOne-stageMethodBackbonemAP (%)FPSMethodBackbonemAP (%)FPSFaster R-CNNVGG-1673.27FA-SSD [24]ResNet5078.130Faster R-CNNResNet-10176.42.4EFIPNet512 [25]VGG-1681.828IONVGG-1676.51.25SSD512VGG-1679.519MR-CNNVGG-1678.20.03DSSD321ResNet-10178.69.5Faster R-CNN∗ResNet-50-FPN79.419.1DSSD512ResNet-10181.55.5Faster R-CNN∗ResNet-101-FPN81.616.0DFPR512 [26]VGG-1681.134R-FCNResNet-10180.59RefineDet320 [27]VGG-1680.040.3CoupleNet[23]ResNet-10182.78.2RefineDet512VGG-1681.824.1OursResNet-50-AFPN82.216.8RFB Net 300VGG-1680.583OursResNet-101-AFPN83.514.1RFB Net 512VGG-1682.238Table 2
PASCAL VOC 2007 test detection results by categories.
mAPAreoBikeBirdBoatBottleBusCarCatChairCowTableDogM-bikePersonPlantSheepSofaTrainTvFaster(Res101)76.479.880.776.268.355.985.185.389.856.787.869.488.380.978.441.778.679.885.372.0R-FCN80.579.987.281.57269.886.888.589.86788.174.589.879.981.253.781.881.585.979.9RefineDet32080.083.985.481.475.560.286.488.189.162.783.977.085.486.782.655.382.778.588.179.4RFBNet30080.583.787.678.974.859.888.887.587.965.085.077.186.186.681.758.181.581.288.480.2SSD30077.579.583.97669.650.58785.788.160.381.57786.183.979.452.377.979.587.676.8DSSD32178.681.984.980.568.453.985.686.288.961.183.578.786.786.779.751.77880.987.279.4FPN(Res50)∗79.482.386.984.568.069.785.687.587.962.985.671.987.982.985.154.081.477.184.676.2Ours(Res50)82.286.787.384.875.172.086.488.789.567.987.177.788.785.686.359.084.278.687.482.3Ours(Res101)83.587.788.585.376.972.788.289.390.071.088.478.189.886.887.160.687.281.487.383.8In order to further study the influence of the added module on the detection effect of the algorithm, we do an ablation experiment. As shown in Table3, all experiments were carried out on FPN (ResNet-50). The mAP of the original FPN could achieve 79.4%. The RFB module was embedded in the benchmark network for training and testing. The mAP of 80.7% was obtained. The RFB is replaced by the proposed ARFB. The map can reach 80.9%. It proves that ARFB further expands the receptive field and improves detection abilities. The SE module is added to the network based on the embedded RFB. The mAP is improved by 0.6% compared to the embedded RFB only. The SE module can extract effective features and reduce the impact of background information. Next, the spatial adaptive fusion and balance mechanism continue to be added to the network. The mAP can reach 81.8%. The experimental results demonstrate that the spatial adaptive fusion and balance mechanism can significantly intensify the connection between different feature layers and enhance the detection ability. Then, the deconvolution operation is used to replace the nearest neighbor for up-sampling. The improvement effect is not obvious. The possible reason for this phenomenon is that deconvolution will blur the feature of the input. It leads to the gain brought by deconvolution being limited. Lastly, the enhanced receptive field module ARFB is used to replace RFB. The mAP is further increased by 0.3% to 82.2%, which also proves the effectiveness of global context information.Table 3
Effects on mAP using different modules.
RFBARFBSEDeconvFusionmAPFPN (ResNet-50)79.4√80.7√80.9√√81.3√√√81.8√√√√81.9Ours√√√√82.2Figure7 shows the qualitative results of the comparative experiment. The left side of each picture is the detection result of FPN (ResNet50), and the right side is the detection result of attention FPN (ResNet50). It can be seen that the improved network has fewer redundant frames in terms of large targets because the expanded receptive field makes the ARFB module supplement effective global information for P5 and P6, so the detection of large targets is more accurate. In terms of small targets, ours has a higher recall rate, thanks to the fact that the attention mechanism filters out irrelevant information, giving the feature fusion mechanism more detailed information and semantic information at the bottom.Figure 7
Test result comparison on PASCAL VOC 2007 datasets.In order to directly reflect the advantages of the proposed method, the visualization results of some feature layers are displayed in Figure8. The a is the original picture. The b is the visualization result of the C5 layer of FPN. The c is the visualization result of the C5 layer of FPN which introduces the RFB module. The d shows the visualization result of the C5 layer which introduces the SE module and RFB module. The e is the visualization result of the P5 layer after spatial adaptive fusion and balance mechanism. The f is the visualization result of our proposed methods. We can observe that the proposed methods can lead the networks to pay more attention to object features while ignoring the influence of background noise.Figure 8
The visualization results in different feature layers. (a) Original picture. (b) FPN. (c) FPN + RFB. (d) FPN + RFB + SE. (e) FPN + RFB + SE + Fusion four.


### 3.2. MS COCO Experiment Results

MS COCO 2017 dataset is the authoritative dataset currently used to evaluate the performance of target detection algorithms. The data include a total of 80 target objects, of which the training set contains 118287 pictures, the verification set contains 5000 pictures, and the testing set contains 20288 pictures. In COCO, there are more small objects than large objects, of which about 41% are small objects (area <322), 34% are medium objects (322 < area <962), and 24% are large objects (area >962). The measured area is the number of pixels in the segmentation mask. The corresponding evaluation indicators are divided into APS, APM, and APL. AP0.5 and AP0.75, respectively, represent the average detection accuracy of all categories when the intersection ratio IoU is 0.5 and 0.75. AP represents all 10 IoU thresholds (0.5 to 0.95) and the average of all 80 categories, and it is considered the most important indicator. We train the model on the COCO 2017 training, testing the experimental results on val2017 and test2017-dev, respectively.The stochastic gradient descent (SGD) method is used for training, the momentum of which is set to 0.9, the weight attenuation coefficient is set to 0.0005, the input image is resized to 1333 × 800, each GPU is assigned two images, and the batch size is set to 8. The maximum number of iterations epoch is 12, and the initial learning rate is set to 0.01, which is multiplied by 0.1 when the epoch is 8 and 11.As shown in Table4, on val2017, we choose three backbones: ResNet-50, ResNet-101, and the more powerful ResNeXt-101 [28](32 × 4d), which are compared with the baseline. As we all know the same backbone, Faster R-CNN combined with FPN only has advantages in small target recognition compared to the original Faster R-CNN. The detection ability of medium and large targets will be reduced a lot. However, the detection accuracy of attention FPN on objects of different sizes is all improved. As seen from Table 5, on test2017-dev, the test results of the proposed algorithm are also better than the baseline. The AP values of attention FPN of the three feature extraction networks increased by 1.5%, 0.7%, and 0.2% respectively, reaching 37.7%, 39.5%, and 40.6%. It is superior to algorithms such as Mask R-CNN and CoupleNet which also use ResNet-101.Table 4
COCO val 2017 detection results.
BackboneAPAP0.5AP0.75APSAPMAPLFaster R-CNN∗ResNet-5036.658.539.220.740.547.9Faster R-CNN∗ResNet-10138.860.542.323.343.150.3Faster R-CNN∗ResNet-50-FPN36.458.439.121.540.046.6Faster R-CNN∗ResNet-101-FPN38.560.341.622.343.049.8Faster R-CNN∗ResNeXt-101(32 × 4d)-FPN40.162.043.823.444.651.7OursResNet-50-AFPN37.559.940.522.341.548.1OursResNet-101-AFPN39.461.642.823.443.750.9OursResNeXt-101(32 × 4d)-AFPN40.363.443.724.144.852.6Table 5
COCO test 2017-dev detection results.
BackboneAPAP0.5AP0.75APSAPMAPLFaster R-CNNResNet-10134.955.737.415.638.750.9Faster R-CNNResNet-101-FPN36.259.139.018.239.048.2Faster R-CNN∗ResNet-50-FPN36.258.538.921.038.945.3Faster R-CNN∗ResNet-101-FPN38.860.942.122.642.448.5Faster R-CNN∗ResNeXt-101(32 × 4d)-FPN40.462.243.624.043.850.3Mask R-CNNResNet-101-FPN29.960.341.720.141.150.2R-FCNResNet-10133.151.9—10.832.845.0CoupleNetResNet-10137.753.535.411.636.350.1YOLO v2Darknet-1921.644.019.05.222.435.5YOLO v3Darknet-5333.057.934.418.335.441.9SSD512VGG-1631.250.433.310.234.549.8DSSD513ResNet-10133.253.335.213.035.451.1RetinaNetResNet-101-FPN39.159.142.321.842.750.2RFB Net512VGG-1633.854.235.916.237.147.4M2Det [29]VGG-1633.553.435.614.437.647.6IBi-FPN [11]VGG-163250.933.7123747.1M2Det [29]ResNet-10138.859.441.720.543.953.4NETNet [30]ResNet-10138.558.641.319.042.353.9MREFP-Net [10]ResNet-10139.360.143.120.643.952.0MREFP-Net [10]VGG-1638.559.141.118.642.949.4OursResNet-50-AFPN39.560.340.522.240.846.5OursResNet-101-AFPN40.662.142.923.143.049.5OursResNeXt-101(32 × 4d)-AFPN41.864.144.124.144.251.4


## 3.1. PASCAL VOC Experiment Results

The PASCAL VOC datasets have a total of 20 categories of targets. We use the training validation sets including 16551 pictures of PASCAL VOC 2007 and PASCAL VOC 2012 for training and the testing set of PASCAL VOC 2007 including 4952 pictures. Stochastic gradient descent (SGD) is used for training, momentum is set to 0.9, the input image is re-sized to 1000 × 600, the batch size is set to 16, the weight attenuation coefficient is set to 0.0005, the initial learning rate is set to 0.02, and the maximum number of iterations epoch is 5. At epoch 3, the learning rate is multiplied by 0.1.The results of our algorithm on the VOC2007 testing set are shown in Table1 (with ∗ indicating the reimplemented version of PyTorch). We use Faster R-CNN + FPN as the baseline. It can be seen from the result that when ResNet-50 and ResNet-101 are used as feature extraction networks, respectively, we use attention FPN (abbreviated as AFPN in the table) to replace FPN, which not only increases mAP by 2.8% and 1.9%, respectively, but also has a small speed loss. The accuracy of ResNet-50 exceeds that of most algorithms in the table, while the recognition accuracy of ResNet-100 is the highest among all algorithms. Table 2 shows the specific results of 20 categories. Our model obtains the best accuracy in multiple categories. Compared with FPN, attention-FPN has an obvious improvement effect on small targets (such as monitors, water cups, bottles, cows, sheep, and so on).Table 1
PASCAL VOC 2007 test detection results.
Two-stageOne-stageMethodBackbonemAP (%)FPSMethodBackbonemAP (%)FPSFaster R-CNNVGG-1673.27FA-SSD [24]ResNet5078.130Faster R-CNNResNet-10176.42.4EFIPNet512 [25]VGG-1681.828IONVGG-1676.51.25SSD512VGG-1679.519MR-CNNVGG-1678.20.03DSSD321ResNet-10178.69.5Faster R-CNN∗ResNet-50-FPN79.419.1DSSD512ResNet-10181.55.5Faster R-CNN∗ResNet-101-FPN81.616.0DFPR512 [26]VGG-1681.134R-FCNResNet-10180.59RefineDet320 [27]VGG-1680.040.3CoupleNet[23]ResNet-10182.78.2RefineDet512VGG-1681.824.1OursResNet-50-AFPN82.216.8RFB Net 300VGG-1680.583OursResNet-101-AFPN83.514.1RFB Net 512VGG-1682.238Table 2
PASCAL VOC 2007 test detection results by categories.
mAPAreoBikeBirdBoatBottleBusCarCatChairCowTableDogM-bikePersonPlantSheepSofaTrainTvFaster(Res101)76.479.880.776.268.355.985.185.389.856.787.869.488.380.978.441.778.679.885.372.0R-FCN80.579.987.281.57269.886.888.589.86788.174.589.879.981.253.781.881.585.979.9RefineDet32080.083.985.481.475.560.286.488.189.162.783.977.085.486.782.655.382.778.588.179.4RFBNet30080.583.787.678.974.859.888.887.587.965.085.077.186.186.681.758.181.581.288.480.2SSD30077.579.583.97669.650.58785.788.160.381.57786.183.979.452.377.979.587.676.8DSSD32178.681.984.980.568.453.985.686.288.961.183.578.786.786.779.751.77880.987.279.4FPN(Res50)∗79.482.386.984.568.069.785.687.587.962.985.671.987.982.985.154.081.477.184.676.2Ours(Res50)82.286.787.384.875.172.086.488.789.567.987.177.788.785.686.359.084.278.687.482.3Ours(Res101)83.587.788.585.376.972.788.289.390.071.088.478.189.886.887.160.687.281.487.383.8In order to further study the influence of the added module on the detection effect of the algorithm, we do an ablation experiment. As shown in Table3, all experiments were carried out on FPN (ResNet-50). The mAP of the original FPN could achieve 79.4%. The RFB module was embedded in the benchmark network for training and testing. The mAP of 80.7% was obtained. The RFB is replaced by the proposed ARFB. The map can reach 80.9%. It proves that ARFB further expands the receptive field and improves detection abilities. The SE module is added to the network based on the embedded RFB. The mAP is improved by 0.6% compared to the embedded RFB only. The SE module can extract effective features and reduce the impact of background information. Next, the spatial adaptive fusion and balance mechanism continue to be added to the network. The mAP can reach 81.8%. The experimental results demonstrate that the spatial adaptive fusion and balance mechanism can significantly intensify the connection between different feature layers and enhance the detection ability. Then, the deconvolution operation is used to replace the nearest neighbor for up-sampling. The improvement effect is not obvious. The possible reason for this phenomenon is that deconvolution will blur the feature of the input. It leads to the gain brought by deconvolution being limited. Lastly, the enhanced receptive field module ARFB is used to replace RFB. The mAP is further increased by 0.3% to 82.2%, which also proves the effectiveness of global context information.Table 3
Effects on mAP using different modules.
RFBARFBSEDeconvFusionmAPFPN (ResNet-50)79.4√80.7√80.9√√81.3√√√81.8√√√√81.9Ours√√√√82.2Figure7 shows the qualitative results of the comparative experiment. The left side of each picture is the detection result of FPN (ResNet50), and the right side is the detection result of attention FPN (ResNet50). It can be seen that the improved network has fewer redundant frames in terms of large targets because the expanded receptive field makes the ARFB module supplement effective global information for P5 and P6, so the detection of large targets is more accurate. In terms of small targets, ours has a higher recall rate, thanks to the fact that the attention mechanism filters out irrelevant information, giving the feature fusion mechanism more detailed information and semantic information at the bottom.Figure 7
Test result comparison on PASCAL VOC 2007 datasets.In order to directly reflect the advantages of the proposed method, the visualization results of some feature layers are displayed in Figure8. The a is the original picture. The b is the visualization result of the C5 layer of FPN. The c is the visualization result of the C5 layer of FPN which introduces the RFB module. The d shows the visualization result of the C5 layer which introduces the SE module and RFB module. The e is the visualization result of the P5 layer after spatial adaptive fusion and balance mechanism. The f is the visualization result of our proposed methods. We can observe that the proposed methods can lead the networks to pay more attention to object features while ignoring the influence of background noise.Figure 8
The visualization results in different feature layers. (a) Original picture. (b) FPN. (c) FPN + RFB. (d) FPN + RFB + SE. (e) FPN + RFB + SE + Fusion four.


## 3.2. MS COCO Experiment Results

MS COCO 2017 dataset is the authoritative dataset currently used to evaluate the performance of target detection algorithms. The data include a total of 80 target objects, of which the training set contains 118287 pictures, the verification set contains 5000 pictures, and the testing set contains 20288 pictures. In COCO, there are more small objects than large objects, of which about 41% are small objects (area <322), 34% are medium objects (322 < area <962), and 24% are large objects (area >962). The measured area is the number of pixels in the segmentation mask. The corresponding evaluation indicators are divided into APS, APM, and APL. AP0.5 and AP0.75, respectively, represent the average detection accuracy of all categories when the intersection ratio IoU is 0.5 and 0.75. AP represents all 10 IoU thresholds (0.5 to 0.95) and the average of all 80 categories, and it is considered the most important indicator. We train the model on the COCO 2017 training, testing the experimental results on val2017 and test2017-dev, respectively.The stochastic gradient descent (SGD) method is used for training, the momentum of which is set to 0.9, the weight attenuation coefficient is set to 0.0005, the input image is resized to 1333 × 800, each GPU is assigned two images, and the batch size is set to 8. The maximum number of iterations epoch is 12, and the initial learning rate is set to 0.01, which is multiplied by 0.1 when the epoch is 8 and 11.As shown in Table4, on val2017, we choose three backbones: ResNet-50, ResNet-101, and the more powerful ResNeXt-101 [28](32 × 4d), which are compared with the baseline. As we all know the same backbone, Faster R-CNN combined with FPN only has advantages in small target recognition compared to the original Faster R-CNN. The detection ability of medium and large targets will be reduced a lot. However, the detection accuracy of attention FPN on objects of different sizes is all improved. As seen from Table 5, on test2017-dev, the test results of the proposed algorithm are also better than the baseline. The AP values of attention FPN of the three feature extraction networks increased by 1.5%, 0.7%, and 0.2% respectively, reaching 37.7%, 39.5%, and 40.6%. It is superior to algorithms such as Mask R-CNN and CoupleNet which also use ResNet-101.Table 4
COCO val 2017 detection results.
BackboneAPAP0.5AP0.75APSAPMAPLFaster R-CNN∗ResNet-5036.658.539.220.740.547.9Faster R-CNN∗ResNet-10138.860.542.323.343.150.3Faster R-CNN∗ResNet-50-FPN36.458.439.121.540.046.6Faster R-CNN∗ResNet-101-FPN38.560.341.622.343.049.8Faster R-CNN∗ResNeXt-101(32 × 4d)-FPN40.162.043.823.444.651.7OursResNet-50-AFPN37.559.940.522.341.548.1OursResNet-101-AFPN39.461.642.823.443.750.9OursResNeXt-101(32 × 4d)-AFPN40.363.443.724.144.852.6Table 5
COCO test 2017-dev detection results.
BackboneAPAP0.5AP0.75APSAPMAPLFaster R-CNNResNet-10134.955.737.415.638.750.9Faster R-CNNResNet-101-FPN36.259.139.018.239.048.2Faster R-CNN∗ResNet-50-FPN36.258.538.921.038.945.3Faster R-CNN∗ResNet-101-FPN38.860.942.122.642.448.5Faster R-CNN∗ResNeXt-101(32 × 4d)-FPN40.462.243.624.043.850.3Mask R-CNNResNet-101-FPN29.960.341.720.141.150.2R-FCNResNet-10133.151.9—10.832.845.0CoupleNetResNet-10137.753.535.411.636.350.1YOLO v2Darknet-1921.644.019.05.222.435.5YOLO v3Darknet-5333.057.934.418.335.441.9SSD512VGG-1631.250.433.310.234.549.8DSSD513ResNet-10133.253.335.213.035.451.1RetinaNetResNet-101-FPN39.159.142.321.842.750.2RFB Net512VGG-1633.854.235.916.237.147.4M2Det [29]VGG-1633.553.435.614.437.647.6IBi-FPN [11]VGG-163250.933.7123747.1M2Det [29]ResNet-10138.859.441.720.543.953.4NETNet [30]ResNet-10138.558.641.319.042.353.9MREFP-Net [10]ResNet-10139.360.143.120.643.952.0MREFP-Net [10]VGG-1638.559.141.118.642.949.4OursResNet-50-AFPN39.560.340.522.240.846.5OursResNet-101-AFPN40.662.142.923.143.049.5OursResNeXt-101(32 × 4d)-AFPN41.864.144.124.144.251.4


## 4. Conclusion

This paper proposes a target detection algorithm based on the improved feature pyramid network, called attention FPN. In the attention FPN, the improved receptive field module is used to obtain the global and local context information; the channel attention module is added to the transverse connection to enhance the characteristics that contribute greatly to the key information. We also used deconvolution instead of nearest neighbor interpolation to reduce information loss in the up-down up-sampling method. Finally, the spatial attention style is used to weigh the integration of the various characteristic layers. The mAP of our improved algorithm on the PASCAL VOC 2007 test set reached 83.5% and the AP on COCO test 2017-dev reached 40.6%; the results show that our algorithm is better than the original algorithm and some mainstream algorithms with a considerable speed.Nevertheless, there are certain limitations in our proposed algorithm. Compared to the original algorithm, the number of parameters of our proposed algorithm has increased by 10%. In addition, a small amount of background noise still exists, which affects the detection efficiency. In the future, we will reduce the number of parameters of the network and study a more effective mechanism to identify target features. Furthermore, we will combine the proposed algorithm with practical application.



---

*Source: 3583399-2022-09-13.xml*