A method of single‐shot target detection with multi‐scale feature fusion and feature enhancement

IET Image ProcessingVolume 16, Issue 6 p. 1752-1763 ORIGINAL RESEARCH PAPEROpen Access A method of single-shot target detection with multi-scale feature fusion and feature enhancement Zhong Qu, Corresponding Author Zhong Qu quzhong@cqupt.edu.cn orcid.org/0000-0001-7013-4854 The College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, No.2, Chongwen Road, Nan'an District, Chongqing, People's Republic of China Correspondence Zhong Qu, The College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, No.2, Chongwen Road, Nan'an District, Chongqing, People's Republic of China. Email: quzhong@cqupt.edu.cnSearch for more papers by this authorXue Shang, Xue Shang The College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, No.2, Chongwen Road, Nan'an District, Chongqing, People's Republic of ChinaSearch for more papers by this authorShu-Fang Xia, Shu-Fang Xia The College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, No.2, Chongwen Road, Nan'an District, Chongqing, People's Republic of ChinaSearch for more papers by this authorTu-Ming Yi, Tu-Ming Yi Institute of Information Technology, Southwest Computer Company Ltd., Chongqing, People's Republic of ChinaSearch for more papers by this authorDong-Yang Zhou, Dong-Yang Zhou The Chongqing College of Mobile Communication, No.1 Vacation Road, Hechuan District, Chongqing, People's Republic of ChinaSearch for more papers by this author Zhong Qu, Corresponding Author Zhong Qu quzhong@cqupt.edu.cn orcid.org/0000-0001-7013-4854 The College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, No.2, Chongwen Road, Nan'an District, Chongqing, People's Republic of China Correspondence Zhong Qu, The College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, No.2, Chongwen Road, Nan'an District, Chongqing, People's Republic of China. Email: quzhong@cqupt.edu.cnSearch for more papers by this authorXue Shang, Xue Shang The College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, No.2, Chongwen Road, Nan'an District, Chongqing, People's Republic of ChinaSearch for more papers by this authorShu-Fang Xia, Shu-Fang Xia The College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, No.2, Chongwen Road, Nan'an District, Chongqing, People's Republic of ChinaSearch for more papers by this authorTu-Ming Yi, Tu-Ming Yi Institute of Information Technology, Southwest Computer Company Ltd., Chongqing, People's Republic of ChinaSearch for more papers by this authorDong-Yang Zhou, Dong-Yang Zhou The Chongqing College of Mobile Communication, No.1 Vacation Road, Hechuan District, Chongqing, People's Republic of ChinaSearch for more papers by this author First published: 13 February 2022 https://doi.org/10.1049/ipr2.12445AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract The Single Shot MultiBox Detector (SSD) is one of the fastest detection algorithms. Although it has achieved good results in detection, it also has the problem of poor detection effect for small targets and occlusion between objects. Here, the authors propose a new target detection method called single-shot target detection with multi-scale feature fusion and feature enhancement. Here, the authors introduce multi-scale feature fusion module, feature enhancement module and efficient channel attention module, and integrate them into the detection module of the original SSD target detection algorithm to improve the ability of network feature extraction. Experimental results on pascal VOC 2007 datasets show that the proposed algorithm works well when the input size is 300 × 300, the detection speed reaches 41.7 frames per second (FPS) and the detection accuracy reaches 79.6%, which is 2.4% higher than the original SSD target detection algorithm. When the input size is 512 × 512, the detection accuracy is 81.9%, and the detection speed reaches 36.5 FPS, which is 3.2% higher than the original SSD target detection algorithm. According to the experimental results, our algorithm has a better performance when there are many objects in the image and there is occlusion. 1 INTRODUCTION Object detection is one of the research hotspots in the field of computer vision. With the rise of artificial intelligence, deep learning technology has developed rapidly, and the ability of computers to analyse and process images has become stronger. In recent years, with the breakthrough of object detection algorithm and its wide application, more and more people pay attention to it. Object detection is one of the basic tasks of computer vision, and it is a prerequisite for object tracking, action recognition, and behaviour expression. Nowadays, object detection is mainly divided into two categories: (1) Object detection algorithms based on traditional image processing, (2) Machine learning and object detection algorithms based on deep learning. Object detection algorithms based on traditional image processing and machine learning are generally divided into three stages: (1) Select some candidate regions on a given image, (2) perform feature extraction on these regions, (3) use a pre-trained classifier for classification. Object detection algorithms based on deep learning can be divided into two categories: two-stage methods and single-stage methods. The two-stage method divides the detection problem into two stages. The first stage is generated candidate regions, and then the candidate regions are classified. The representative algorithms include Faster R-CNN [1], Mask R-CNN [2] etc. The single-stage method directly generates the category probability and position coordinates of the object on the input image, and the representative algorithms include YOLO [3], SSD [4] etc. In general, the two-stage method is usually more accurate but slower than the single-stage method, while the single-stage method loses a certain accuracy but is faster than the two-stage method. In recent years, object detection algorithms based on deep learning have been continuously proposed and improved by scholars. This paper proposes improvements to the problems of low detection accuracy and inaccurate positioning of small objects in the object detection algorithm SSD. In the convolution neural network, the receiving field of each layer is fixed. The SSD algorithm directly uses the feature maps of different layers in the convolution neural network to predict objects of different scales. Because the shallow feature maps have small receptive fields, they are often used to detect small objects. The deep feature map has a large receptive field and is used to detect large objects. However, shallow features have less semantic information, which may affect the performance of small object detection. FSSD [5] fuses feature of different levels and different scales on the basis of the SSD network structure to generate a new feature pyramid, and finally sends the generated new feature pyramid to the detector for detection. The YOLO algorithm is proposed to divide the image into S × S grids. Only when the centre point of the object falls within a certain grid, the grid is responsible for detecting the object. The CornerNet [6] uses a pair of key points (upper left corner and lower right corner) to detect the border of an object. The RFBNet [7] network proposes the Receptive Field Block to expand the receptive field of the prediction layer and improve the detection accuracy. In this paper, we propose a method references the structure of the characteristics pyramid, and fuses multi-scale feature maps to enrich the semantic information of shallow feature maps and the detail information of deep feature maps. As it shows in Figure 1, our entire model is consisted of four parts: the feature extraction, the multi-scale feature fusion, the feature enhancement and the detection layers. The contributions of this paper can be summarized as follow: We modify the network structure of the SSD algorithm to improve the target detection accuracy. We use the feature pyramid structure [8] for reference and fuse some deep feature maps with shallow feature maps to enrich the semantic information of shallow feature maps and the detail information of deep feature maps. We propose a new initial model. In order to ensure the stability of the network, SSD algorithm uses VGG-16 [9] as the basic network. Because the structure of VGG16 network model is too simple to fully extract the features needed to detect the target, resulting in low target detection accuracy. Therefore, inspired by the emergence structure, we use the feature enhancement module in the network to increase the width and depth of the network and improve the ability of feature extraction. Experimental results show that the proposed network is superior to the existing target detection algorithms in detection accuracy, and has obvious advantages when more targets are in the image and mutual occlusion exists between targets. FIGURE 1Open in figure viewerPowerPoint The network structure of this paper when input size is 300 × 300. Overall architecture of our single-stage detection approach using VGG backbone. It consists of three main components. (1) The VGG16 network is used for feature extraction. (2) Multi-scale feature fusion is the fusion of shallow feature map with rich detail information and deep feature map with rich semantic information. (3) The feature enhancement part is mainly composed of attention mechanism module and feature enhancement module. (4) The detection is carried out in different output feature layers, and then the final result is produced by non-maximum suppression The rest of the paper is structured as follow. Section 2 introduces related works. Section 3 introduces the target detection method in detail. Section 4 presents the details of the experimental implementation and gives the corresponding experimental results. Section 5 concludes this paper. 2 RELATED WORK The task of object detection is to correctly classify the categories of objects in the image and accurately locate the position and border information of the objects. The AlexNet [10] is proposed by the Alex et al. in 2012. Since then, deep learning networks began to be studied by a large number of scholars. In 2014, Girshick et al. proposed the R-CNN [11] model that enabled the object detection based on deep learning. Faster R-CNN introduces a region proposal network (RPN) on the basis of R-CNN to directly generate candidate regions, which improves the speed and accuracy of training. Mask R-CNN uses bilinear interpolation to replace the RoI Align layer in the Faster R-CNN algorithm with the RoI Pool layer, which solves the problem of misalignment between the feature map and the RoI on the original image, and combines detection with segmentation. The DSSD [12] algorithm replaces the basic network VGG16 with the Residual101 [13] network structure with better feature extraction capabilities on the basis of SSD, and proposed that the deconvolution module introduces context information to improve the resolution of feature maps and the detection accuracy of small objects. YOLO9000 [14] can detect more than 9000 different target categories. The algorithm combines different data sets using the hierarchical view of target classification, and proposes a joint training method that can train target detectors on detection and classification data. YOLOv3 [15] uses up-sample and fusion methods similar to FPN to detect on feature maps of multiple scales. The STDN [16] network improves accuracy while maintaining speed by proposing the scale transfer module that can be directly embedded into the network to obtain high-resolution feature maps for small objects, in addition to using the pooling layer to generate feature maps with large receptive fields for large objects. The EFGRNet [17] introduces a simple and effective feature enrichment scheme to generate multi-scale contextual features, and proposed a cascade refinement scheme to refine the anchor point position and enrich the features, and solve the problem of class imbalance. Focal loss [18] improves the cross-entropy loss function in the single-order detector. By controlling the weight of positive and negative samples and the weight of difficult and easy samples, it increases the difficulty of learning small classes of samples in the loss function. The M2Det [19] network proposes a multi-level feature pyramid network (MLFPN) to solve the multi-scale problem in object detection. The RSSD [20] algorithm establishes a relationship between each scale of the feature pyramid, and effectively increases the number of feature maps in each layer of the feature pyramid, and finally matches the number of feature maps at different levels, so that a single classifier can be applied and Multiple scales improve generalization ability. Zhu et al. [21] uses key point estimation to find the centre point of the object, and then returned to other attributes, including size, three-dimensional position, orientation, and posture. CenterNet extends CornerNet to improve accuracy and recall rate through three points instead of two. FSAF [22] adds an anchor-free branch and online feature selection mechanism on top of RetinaNet. This branch defines the central area of the object as a positive sample, and uses the distance from the four sides of the object to locate it. FCOS [23] defines all positions within the object frame as positive samples, and then detects the object through 4 distance values and 1 centerness score. CSP [24] only defines the centre point of the object as a positive sample, and detects pedestrians through a fixed aspect ratio. FoveaBox [25] defines the position of the middle part of the object as a positive sample, and each position has 4 distances for detection. 3 PROPOSED METHODS In this part, we review the network structure of the SSD algorithm, and then introduce the structure of the algorithm in this paper, finally describe the various components of the network in this paper: Multi-scale feature fusion module, feature enhancement module, attention module and loss function. 3.1 SSD algorithm network structure The object detection algorithm SSD based on convolution neural network uses VGG16 as the basic network to discretise the output space of the bounding box into a series of default boxes with different aspect ratios and sizes, which are distributed in each of the feature maps of different resolutions during the down-sampling process. The SSD algorithm detection model directly uses Conv4_3, Conv7 (fc7), conv8, conv9, conv2, conv10 and conv11 in VGG16 network to predict both locations and confidences. By combining the prediction of multiple feature graphs, the algorithm can deal with objects of various sizes naturally. The network structure when input size is 300 × 300 is shown in Figure 2. FIGURE 2Open in figure viewerPowerPoint SSD network structure when input size is 300 × 300 3.2 The network structure of this paper The original SSD algorithm does not consider the relationship between layers, only predicts the location and reliability of each layer. However, the semantic information of shallow feature map is less, which is not conducive to small target detection. This paper uses VGG16 as the backbone network. The algorithm is based on SSD framework and integrates the idea of FPN into the structure. Inspired by the feature pyramid structure, our network structure combines the high-resolution information of low-level features with the high-level semantic information of high-level features through the hybrid structure of bottom-up and top-down, which fully integrates the convolution feature information of each scale, and improves the semantic information of shallow feature mapping. At the same time, the VGG16 is too simple to extract features as the backbone network. Therefore, we propose a feature enhancement module based on the idea of inception, which increases the width and depth of the network on the basis of VGG16 network, so as to improve the ability of network feature extraction. Our detection model is consisted of four parts: (1) Feature extractor, (2) feature fusion module, (3) feature enhancement, (4) detection layers. When input size is 300 × 300, its structure is shown in Figure 1. 3.3 The feature fusion module This paper uses conv4_3, conv7 (fc7), conv8, conv9, conv2, conv10 and conv11 for feature extraction, and the feature maps extracted from conv5_3 layers are retained. Then the extracted features are fused by using the idea of feature pyramid. Small targets are mainly detected in the shallow feature map, that is, conv4_3 and conv7 in the detection module of SSD algorithm. Due to the lack of semantic information in shallow feature map, our method concatenates the feature maps of different scales. For conv4_3, we up-sample conv7 and conv8_2 to the size of conv4 feature map, and then fuse the three layers to generate a new feature map. For conv7, we use the feature map extracted by conv5_3 for down-sampling, and use the feature map extracted by conv8_2 for up-sampling, and then fuse to get a new feature map. For deep features, we down-sample the feature map from the upper layer, and then fuse it with the feature map from this layer. In order to connect the features of different scales simply and effectively, we use concatenate method to splice all the feature images of the same size. Its structure is shown in Figures 1a,b and 3a. FIGURE 3Open in figure viewerPowerPoint The feature fusion module 3 is F3 in Figure 1 and the convolution output module in Figure 1 Figure 3a is the feature fusion module 3 in the multi-scale feature fusion, and Figure 3b is the convolution output module in the feature enhancement. Finally, the two convolutional layers of 3 × 3 filters are used to fuse different features individually. 3.4 Feature enhancement module The original SSD uses VGG16 network as the backbone network. In this paper, VGG16 is still used as the backbone network. At the same time, in order to enrich the extracted feature information, we fuse the extracted features. In this part, we further process the obtained features. In order to extract higher dimensional features while ensuring the stability of the model, we propose a feature enhancement module. Its structure is shown in Figure 4. Our feature enhancement module is consisted of four branches. In the left branch, we first use 1 × 1 convolution for dimension reduction, and then use a 3 × 3 convolution. In the middle two branches, we first use a 1 × 1 convolution, and then connect 1 × n and n × 1 convolutions respectively, and then followed by a 3 × 3 and 1 × 1 convolutions respectively, which can learn more non-linear relations while reducing the model parameters. In the last branch, we directly use a 1 × 1 convolution, and finally we concatenate all the features. FIGURE 4Open in figure viewerPowerPoint Feature enhancement module 3.5 Efficient channel attention module In this paper, feature fusion method is used to combine the detail information of low-level feature graph with the semantic information of high-level feature graph. However, when the image background is complex, small-scale target detection is still prone to errors. In order to solve this problem, the ECA module is used to weight the features of the target region in the conv4_3 and conv7 of the prediction layers. The feature extraction network can focus on the target region containing important information selectively, suppress other irrelevant information, and reduce the impact of the background on the detection results, so as to improve the detection performance of the model for small targets. The ECA module is shown in Figure 5. FIGURE 5Open in figure viewerPowerPoint Schematic diagram of ECA module In the ECA module, after the feature map is pooled by the global average of the channel, the channel weight is learned through a one-dimensional convolution with the size of k weight sharing, and then the weight coefficient is obtained through an activation function, finally the weight coefficient and the original feature map are used by element-wise product for the new feature map. The size of the convolution kernel k is adaptively determined by the nonlinear mapping of the channel dimension, and it represents the coverage of the local cross-channel interaction. Because the coverage of cross-channel information interaction is directly proportional to the channel dimension C. Therefore, there may be a mapping φ between k and C: C = φ k \begin{equation}C = \varphi \left( k \right)\end{equation} (1) The simplest mapping method is linear mapping, but due to the limitations of linear functions for certain related features, and the channel dimension is usually an exponential multiple of 2. Here, an exponential function with base 2 is used to represent the nonlinear mapping relationship: C = φ k = 2 r ∗ k − b \begin{equation}C = \varphi \left( k \right) = {2^{\left( {r * k - b} \right)}}\end{equation} (2) Therefore, given the channel dimension C, the convolution kernel size (k) can be calculated according to the following formula: k = ψ C = log 2 C r + b r o d d \begin{equation}k = \psi \left( C \right) = {\left| {\frac{{{{\log }_2}\left( C \right)}}{r} + \frac{b}{r}} \right|_{odd}}\end{equation} (3)among them, this article sets r to 2 and b to 1, and | t | o d d ${| t |_{odd}}$ represents the odd number closest to t. 3.6 Loss function Our target detection model mainly includes two types of loss functions, one is category loss, and the other is the location loss. In the last part of the detection model, these two types of loss functions calculate the category loss and location loss according to the model output and the actual label box. The loss function of this paper is shown as follow: L = 1 N L c o n f x , c + α L l o c x , l , g \begin{equation}L = \frac{1}{N}\left( {{L_{conf}}\left( {x,c} \right) + \alpha {L_{loc}}\left( {x,l,g} \right)} \right)\end{equation} (4)among them, L c o n f ${L_{conf}}$ and L l o c ${L_{loc}}$ are the objective functions corresponding to classification and positioning, respectively. N is the number of prediction boxes (positive examples) whose prediction box and real box cross ratio is greater than the threshold, and α is a weight parameter used to adjust the loss weight of the multitask loss function. 4 EXPERIMENTAL RESULTS AND ANALYSIS In this section, we first introduce the datasets. Then, this section describes the parameter setting, evaluation index and comparison method in the network. Finally, the experimental results are introduced and analysed. 4.1 Datasets In order to test the performance of our proposed the model, we train and evaluate it on the PASCAL VOC [26] dataset. During the experiment, we use the union of VOC 2007 trainval and VOC 2012 trainval to train the model, where the VOC 2007 trainval contains 5k images, and the VOC 2012 trainval contains 11k images. We evaluate on the VOC 2007 testset containing 5k images. 4.2 Implementation details All the experiments are performed on the GPU server, which has a Tesla-V100-SXM2-32GB GPU and a 4-core Inter(R) Xeno(R) Sliver 4214 CPU. In addition, our codes are based on PyTorch. Our framework uses VGG-16 pre-trained on ImageNet [27] as the backbone. In VOC 2007 and VOC 2012, a predicted bounding box is positive if the jaccard overlap is higher than 0.5. During the training process, the input image size is set to 300 × 300, the weight decay is set to 0.0005, the momentum is set to 0.9, and the batch size is set to 32, the initial learning rate is set to 0.001, the number of iterations is set to 120,000, and the attenuation increases with the increase of the number of iterations. From the 80,000th to 100,000th iterations, we set the learning rate to 0.0001. For 100,000th to 120,000th iterations, we set the learning rate to 0.00001. 4.3 Evaluation metrics This paper uses the mean average precision (mAP) and the frames per second (FPS) as the evaluation indicators to compare different target detection methods. 4.3.1 Mean average precision (mAP) The mAP is defined as the average value of average precision (AP) for all the object categories, and AP is the area under the precision and recall (P–R) curve. The calculation method of AP is expressed as: A P = ∫ 0 1 p r d r \begin{equation}AP = \int_0^1 {p\left( r \right)dr} \end{equation} (5)where p represents precision and r represents recall. The calculation method of the precision and recall is as follow: P r e c i s i o n = T P T P + F P \begin{equation}Precision = \frac{{TP}}{{TP + FP}}\end{equation} (6) R e c a l l = T P T P + F N \begin{equation}Recall = \frac{{TP}}{{TP + FN}}\end{equation} (7)where TP is the number of true-positive, FP is the number of false-positive and FN is the number of false-negative. 4.3.2 Frames per second Frames per second (FPS) represents the number of pictures that can be processed per second, it is generally accepted that when the frame rate reaches 24, it can be considered as smooth. 4.4 Comparison methods In this section, we use the SSD algorithm as a benchmark. DSSD, STDN, RefineDet [28], ASSD [29], and TDFSSD [30] are all upgraded versions of SSD. Faster R-CNN, SPPnet [31] and R-FCN are two-stage detectors, YOLO, YOLOv2, YOLOv3 and YOLOv5 are single-shot detectors. All of the comparison algorithms in this section train on the VOC 2007 and VOC 2012 training sets, and evaluate on the VOC 2007 test set. Faster R-CNN: Faster R-CNN introduces a region proposal network (RPN) on the basis of R-CNN to directly generate candidate regions, which improves the speed and accuracy of training. SPPnet: The SPP-net network is based on Fast R-CNN, and proposes to remove operations such as crop/warp on the original image and replaces it with a spatial pyramid pooling layer on convolutional features, which speeds up target detection. R-FCN: The R-FCN proposes the contradiction between position sensitive score map translation invariance and translation variance based on the Faster R-CNN. The full convolution network is used to realize the complete calculation sharing and improve the detection speed and accuracy of the network. YOLO: Based on a single end-to-end network, YOLO completes the input of the original image to the output of the object position and category, proposes the object detection as a regression problem. Although the detection speed of the YOLO algorithm is fast, its detection accuracy is poor. YOLOv2: On the basis of YOLO, YOLOv2 is improved by using a series of methods and proposed a joint training method of target classification and detection, which not only ensured the calculation speed, but also improved the detection accuracy. YOLOv3: YOLOv3 is already based on YOLO and YLOv2. By adjusting the network structure and using multi-scale features for target detection, the detection accuracy is improved while maintaining the speed advantage. YOLOv5: On the basis of YOLOv4, YOLOv5 has made some improvements to the input end, reference network, neck network, and head output layer, so that its speed and accuracy have been greatly improved. DSSD: The DSSD algorithm uses ResNet-101 as the backbone network, proposes that the deconvolution module introduces context information to improve the resolution of feature maps and the detection accuracy of small objects. STDN: The STDN algorithm uses DenseNet-169 as the basic network for feature extraction, and proposes scale-transfer layer to generate large-scale feature maps, which solves the problem of multi-scale detection. RefineDet: The RefineDet network is based on the improvement of the SSD algorithm. Its entire network consists of two modules, ARM and ODM. This method maintains high efficiency while ensuring. ASSD: The ASSD algorithm merges prime-level feature relationships into the first-level detector on the basis of the SSD algorithm, highlighting useful areas on the feature map while suppressing irrelevant information, thereby improving the detection accuracy of the network. TDFSSD: The TDFSSD network is based on the SSD target detection algorithm, and proposes a new feature fusion module to enrich shallow detail information and deep semantic information, thereby improving the detection accuracy of the entire network. 4.5 Experimental results Compared with other typical target detectors of above-mentioned, the network structure of SSD is improved in this paper. We use the joint training set of VOC 2007 and VOC 2012 to train these algorithms, and then use the VOC–2007 test datasets to test. The test results are shown in Table 1. We mainly consider mAP and FPS. TABLE 1. Comparison of detection methods of VOC test data set in 2007. All algorithms are trained on the joint training set of VOC2007 and VOC2012, and tested on the VOC2007. Finally, the detection accuracy and speed of these detectors are compared Method Network Train Test Size mAP FPS Faster R-CNN ResNet101 VOC2007+2012 V