Airport small object detection based on feature enhancement

IET Image ProcessingEarly View ORIGINAL RESEARCH PAPEROpen Access Airport small object detection based on feature enhancement Xuan Zhu, Xuan Zhu orcid.org/0000-0002-2745-2934 School of Aeronautics and Astronautics, Sichuan University, Chengdu, ChinaSearch for more papers by this authorBinbin Liang, Binbin Liang School of Aeronautics and Astronautics, Sichuan University, Chengdu, ChinaSearch for more papers by this authorDaoyong Fu, Daoyong Fu School of Aeronautics and Astronautics, Sichuan University, Chengdu, ChinaSearch for more papers by this authorGuoxin Huang, Guoxin Huang orcid.org/0000-0003-4189-1746 College of Computer Science, Sichuan University, Chengdu, ChinaSearch for more papers by this authorFan Yang, Fan Yang School of Aeronautics and Astronautics, Sichuan University, Chengdu, ChinaSearch for more papers by this authorWei Li, Corresponding Author Wei Li li.wei@scu.edu.cn orcid.org/0000-0002-3786-4959 School of Aeronautics and Astronautics, Sichuan University, Chengdu, China Correspondence Wei Li, School of Aeronautics and Astronautics, Sichuan University, Chengdu 610065, China. Email: li.wei@scu.edu.cnSearch for more papers by this author Xuan Zhu, Xuan Zhu orcid.org/0000-0002-2745-2934 School of Aeronautics and Astronautics, Sichuan University, Chengdu, ChinaSearch for more papers by this authorBinbin Liang, Binbin Liang School of Aeronautics and Astronautics, Sichuan University, Chengdu, ChinaSearch for more papers by this authorDaoyong Fu, Daoyong Fu School of Aeronautics and Astronautics, Sichuan University, Chengdu, ChinaSearch for more papers by this authorGuoxin Huang, Guoxin Huang orcid.org/0000-0003-4189-1746 College of Computer Science, Sichuan University, Chengdu, ChinaSearch for more papers by this authorFan Yang, Fan Yang School of Aeronautics and Astronautics, Sichuan University, Chengdu, ChinaSearch for more papers by this authorWei Li, Corresponding Author Wei Li li.wei@scu.edu.cn orcid.org/0000-0002-3786-4959 School of Aeronautics and Astronautics, Sichuan University, Chengdu, China Correspondence Wei Li, School of Aeronautics and Astronautics, Sichuan University, Chengdu 610065, China. Email: li.wei@scu.edu.cnSearch for more papers by this author First published: 29 November 2021 https://doi.org/10.1049/ipr2.12387AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Video object detection is essential for airport surface surveillance, but the objects on the scene are mostly small objects with low resolution, they have no obvious feature information. Due to the scale differences of the objects and the fixed receptive field on the feature maps, detectors cannot model multi-scale context information and cover all objects. In addition, although the video detection algorithm can be used as a method to solve the problem of small object detection, the temporal feature fusion method of current video detection is very dependent on the quality of a single feature map. Therefore, this paper aims to enhance the features of small objects of a single image. First, an attentional multi-scale feature fusion enhancement (A-MSFFE) network is built on the memory-enhanced global-local aggregation (MEGA) to supplement semantic and spatial information of small objects. Then, a context feature enhancement (CFE) module is designed for obtaining different receptive fields through different dilated convolutions. Meanwhile, a video detection dataset about the airport is established. Finally, the experimental results show that the proposed method can improve the detection accuracies of small objects and outperform other state-of-the-art video object detection algorithms in self-built airport dataset. 1 INTRODUCTION Among air transportation systems, airport surface surveillance is vitally important for operational security. Since cameras can provide fine-grained information about the airport surface, the applications of cameras in airport surveillance are becoming more and more extensive. With the development of communication and computation technology, camera networks can provide wide field of views and automatically capture any movement in the region of interest with 24x7 service [1], which leads them to be very competent in the surveillance for airport surface. Besides, as deep learning went on, the object detection methods [2-6] based on deep learning have achieved great success. This method can not only obtain accurate object region, but also obtain fine-grained features such as texture and colour. Therefore, the detection technology based on deep learning has gradually become an effective tool for scene monitoring. Since there are mostly small objects on the airport surface, which lack adequate feature, it is difficult to detect small objects. Although common object detection algorithms apply feature fusion methods [7, 8] to increase semantic information of small objects, a high-level semantic has not been used mostly [9]. Recent work [10] obtains more contextual semantic information to improve small object detection by using multi-branch parallel feature pyramid networks, and there is research [11] which uses recurrent neural networks to generate fused semantics feature. Since it is difficult to extract silhouette features of small objects, the research of silhouette-based enhancement feature [12] can improve silhouette features of small objects to increase the detection precision. Besides, context information plays a vital role in visual recognition [13-15], so some existed methods [16, 17] acquire more feature information through modelling context to improve accuracies of small objects. Although more and more research proposed different and novel methods to tackle problems of small object detection, the existing research [18-20] on small objects is still limited to the field of images. By analysing how people distinguish small objects in real scenes, we find that small objects can be identified by using the image information of the pervious and next frames. Therefore, video detection algorithms are introduced as a solution to solve small object detection. Meanwhile, the particularities of airport surface surveillance bring new challenges for effective object detection. For the sake of safety, video cameras always have to be installed on the land-side facilities and buildings [21], such as terminal buildings, air traffic control towers, boarding bridges, and so forth, so the objects including people, distant aircrafts and vehicles all have low pixels in the video. For example, the vehicles or person may contain just a few pixels in images, which will dramatically increase the difficulties of detection and may further affect the safe operation of the airport surface. Therefore, accurately detecting these objects and providing early warning will avoid major aviation accidents and economic losses [22]. In airport surface surveillance, the cameras always have wide field of views to capture information about the objects in the airport surface as much as possible [23], so there are scale gaps between the same objects in the far or near scenes. Besides, different objects have size gaps in the real world, such as an aircraft and a person. Therefore, the detector needs to be able to detect objects of different scales at the same time. At present, the application of temporal information in the detection field is based on the feature map of a single image, and current video detection algorithms focus on occlusion, motion blur and rare poses of objects [24]. Thus, these algorithms do not consider how to handle the small objects in the video. When there are several small objects in the airport image, and they are convolved several times, the small object features in the image are very small or even disappear. Therefore, for a series of feature maps with insufficient features of small objects, using temporal information fusion method is not enough to improve the detection results of small objects. In this paper, we propose an airport small object video detection algorithm, which integrates two components in memory-enhanced global-local aggregation (MEGA) [25] to deal with the above problems. First, to address the problem of insufficient feature for airport small objects, an attentional multi-scale feature fusion enhancement (A-MSFFE) network is built. It introduces high-level features into low-level features through deconvolution attentional feature fusion (D-AFF) module. At the same time low-level features with spatial information are embedded into high-level features with semantic information. Thus, the fusion network can achieve the complementary of high-level and low-level features. Among them, we also use the non-local module [26] to make up for the lack of global information. Second, we propose a context feature enhancement (CFE) module to increase context information and solve the problem of multi scale objects. CFE module with only a few computational burdens can generate different receptive fields through three different dilated convolutions to cover objects of different scales and obtain multi scale context information. Finally, to improve the effect of the temporal fusion, we select a single image as the processing object, and every image passes an enhanced feature extraction network to obtain high-quality feature map. In sum, we summarize our contributions as follows:(1) The video detection algorithm is applied to the field of small object detection as a new idea to handle the small object detection. (2) We propose A-MSFFE network which can supplement spatial and semantic information of small objects, and we also introduce deconvolution into attentional feature fusion module to merge feature maps of different levels. (3) We design simple and effective CFE modules to address fixed receptive fields on the feature maps. The rest of the structure in this paper is as follows. Section 2 states the related work about video detection, multi scale feature fusion and context information. Section 3 introduces A-MSFFE that can compensate for lack of information and the CFE module that can adapt multi scale objects. Section 4 describes the self-built airport dataset and evaluation metrics. Section 5 shows comparative experimental results of different algorithms and the corresponding ablation analysis. Section 6 gives the relevant conclusions and future research directions. 2 RELATED WORK 2.1 Video object detection At present, there are many video object detection algorithms, most of which are based on single-frame image object detection algorithms [2, 3, 27, 28]. The mainstream video detection algorithms are mainly divided into three categories, tracking technology, optical flow technology, and attention mechanism. T-CNN [29] used the temporal information to propagate the detection results between adjacent frames to reduce the mutation of the detection results, then it used the tubelets generated by the tracking algorithm to correct the detection confidence, and finally, it used the context information to suppress the detection scores with low confidence to reduce the false detection rate. Detect to Track and Track to Detect [30] proposed ROI tracking module to associate two frames of object to achieve tracking. Zhu et al. proposed DFF [31] which used the flow field to propagate the key frame features to adjacent frames to reduce the calculation of feature extraction. Then, the team proposed FGFA [32], which used optical flow [33] and adaptive weight to guide the aggregation of adjacent frame features to the current frame features in order to improve the object detection accuracy. MANet [34] proposed an instance level calibration method to accurately aggregate features. Wu et al. proposed SELSA [35] using the similarity of the same object proposals of adjacent frames to aggregate features in the whole video sequence. RDN [36] enhanced features by fully mining the relationship between objects in adjacent frames with the relationship module. STMN [37] proposed a spatiotemporal memory module which modelled temporary information and aligned frame-by-frame to solve the problems of modelling the long-time information. Lu et al. [38] proposed the association LSTM method to solve the problem where the current LSTM [39] could not model the object association between consecutive frames. Chin et al. [40] proposed adaptive-scale method which can adjust input images to the best proportion to improve both speed and accuracy for video object detection. RN-VID [41] took use of the fusion module to merge consecutive frames to deal with the problem that detectors do not reuse every frame. Chen et al. [25] proposed the global and local information aggregation of MEGA, which made up the shortage of only using global or local information in current video detection. It aggregated the global frames and local frames through the relationship module, and then it also aggregated the previous frame features stored in the memory module to the current frame. Gong et al. [42] proposed a temporal RoI align module to extract RoI features from the support frame feature maps by feature similarity. SGE-NET [43] utilized squeezed GRU to address the problem of inference speed in video object detection. True et al. and Cui et al. [44] proposed TF-Blender, which consisted of a temporal relation module and feature adjustment to tackle modelling lower-level temporal relations in current video object methods. STFT [45] used proposal-guided deformable convolutions to deal with inter-frame variations and proposed a channel-aware attention module to fuse features of adjacent frames better. 2.2 Multi-scale feature fusion The original images pass through the convolutional neural network to generate feature maps of different sizes containing different types of information. The multi-scale feature fusion method can supplement the information lacking in the feature maps at different stages. FPN [7] constructed a top-down hierarchical structure with lateral connection to construct high-level semantic features to deal with multi-scale change in object detection. PANet [46] added bottom-up path augmentation to FPN to retain more shallow features information and improve positioning accuracy. Libra R-CNN [8] proposed a balance frame pyramid, which effectively alleviated the problems of feature imbalance in the process of feature fusion. Aug-FPN [47] introduced consistent supervision and residual feature augmentation based on FPN to deal with the semantic differences between different levels and the information loss of the top features of the pyramid structure, respectively. FSSD [48] reconstructed the feature pyramid on the fusion feature to supplement the semantic information of small objects and improve their detection accuracy. DSSD [49] proposed a universal top-down structure to increase the semantic features of small objects in shallow feature maps. Hong et al. [50] proposed a SSPNet which contains CAM, SEM and SSM to tackle inconsistency in gradient, which can fully utilize shallow layers to detect tiny objects. Gong et al. [51] proposed effective fusion factor which controls information delivery from deep layers to shallow layers to make FPN adapt tiny object detection. Our method used attention mechanism to fuse and enhance input features. When feature fusion is carried out, more attention is paid to the object areas to obtain more detail information of the objects. 2.3 Context information Adding context information in a convolutional neural network can help the network to learn the surrounding background information of the objects, which is conducive to the accurate classification of the objects in the scene and improvement of the accuracy of detection. ION [16] used spatial recurrent neural network to integrate the context information outside the region of interest to increase information available to the detector. Liu et al. [52] simulated a human receptive field and proposed an RFB module with the inception structure [53] to obtain context information and enhance the expression of lightweight feature. SSH [54] obtained a larger receptive field by stacking some convolutions with smaller convolution kernels to model context information. Therefore, the method can reduce detection time and memory requirements. EFGRNet [55] designed a multi-scale context information module to deal with the fixed context information in the prediction layer of SSD [3]. MS-CNN [56] introduced a context area larger than the object areas to obtain context information to solve the inconsistency between the object size and the fixed receptive field. CoupleNet [18] doubled the feature corresponding to ROI to obtain context information to reduce the chance of misclassification. Chu et al. [57] proposed an object detection system integrating local appearance, context information of objects and global scene to alleviate semantic inconsistency of input images. Lim et al. [58] concatenated the object features which were detect objects and were featured from higher layers to provide context information. 3 METHODS In this paper, we propose a multi feature fusion network and a context module to deal with small object detection and multi scale object detection. The pipeline of the whole algorithm is shown in Figure 1. First, for small object detection, although the high-resolution low-level features are suitable, small objects lack semantic information in the low-level features. After many operations such as convolution and pooling, the small objects lose the detail and location information, which will lead to inaccurate position regression and difficult to locate correctly. Therefore, as shown in Figure 1 of the black-dotted box, we integrate low-resolution high-level features and high-resolution low-level features together, which supplements the semantic information and location information to better classify and locate. Second, since the convolution kernel with a fixed size is used in the network, only a fixed receptive field can be generated. However, the receptive field with a fixed size cannot completely cover all the objects in the scene, and it also cannot obtain different context information. Therefore, as shown in Figure 1 of the red dotted box, we introduce a variable receptive field by dilated convolutions to deal with multi scale objects, and multi scale context information is modelled for small objects. RPN network (in blue box) generates a set of object proposals with objectness scores. ROI align (in green box) can obtain output features of fixed size, and it uses bilinear interpolation to compute the exact values of the input features at four regularly sampled locations in each RoI bin. Global-local aggregation method (in yellow box) uses location-free module to merge global information from global features into local features. Long-range memory module increases the number of features which are used by key frames through taking full use of previous features. Enhanced local aggregation method (in purple box) uses location-based relation module which can mine spatial-temporal information to enhance the local features. FIGURE 1Open in figure viewerPowerPoint The overall framework of the proposed algorithm 3.1 An attentional multi-scale feature fusion enhancement network Multi-scale feature fusion is one of the common methods for small object detection. It integrates high-level features into low-level features to supplement the semantic information of small objects order to improve the detection effect of small objects. However, the method has an expensive calculation, which will bring difficulties to small object detection in the video. Therefore, we propose a more efficient feature fusion method, which not only integrates high-level features into low-level features to supplement semantic features of small objects, but also integrates low-level features into high-level features to supplement spatial location information. The proposed feature fusion network is shown in Figure 2. Inspired by the idea of feature extraction by MEGA, we select ResNet-50 [59] as feature extractor, and RPN head is placed at the top of c o n v _ 4 stage. Then, we define the feature maps of different stages generated by the standard convolutional neural network as { C 2 、 C 3 、 C 4 }, and we also define the size of the C 2 as H × W × C . The feature maps of the three stages pass A-MSFF to form an enhanced feature map P 3 with semantic and spatial information. To make the object features of the enhanced images more obvious, the non-local attention module [26] is chosen to make the detector pay more attention to the object features, and P 3 can capture the long-distance dependence between any pixel points through this module to model the global information and generate P E 3 . In addition, the module belongs to self-attention, which can make the object areas get more attention to obtain more detail information. Thereby, the object feature and background information are more distinguishable to improve the detection effect of small objects which are similar to the background features. Finally, to alleviate the loss of semantic information, P E 3 and C 4 are fused by 3 × 3 convolution, so output features are obtained with relatively complete spatial and semantic information. FIGURE 2Open in figure viewerPowerPoint The overall pipeline of A-MSFFE Among them, the specific fusion process of the A-MSFF method is shown in Figure 3. Because C 2 undergoes less convolution, it lacks semantic features and contains more noise. C 4 passes more convolutional layers so that its size is small, so the pixels of small objects in this layer may be only a few pixels or may even disappear, and the loss of spatial position information is serious. Therefore, the middle feature map C 3 is selected as the standard size of fusion. First, the deconvolution attentional feature fusion module is used to integrate the high-level features into the low-level features, and the module combines deconvolution with attentional feature fusion (AFF) module [60]. While the spatial resolution of C 4 and C 3 are matched, the losses of features in the process of upsampling can be reduced as much as possible. Then, the input feature maps are enhanced by calculating the attention of the fusion feature maps, and feature maps of the fusion with multi-scale feature information are acquired by further fuse on this basis. Next, C 2 through the 3 × 3 convolution is fused with F 3 to obtain P 3 , in which the low-level features are integrated into high-level features to supplement detail information. FIGURE 3Open in figure viewerPowerPoint The structure of the A-MSFF 3.2 Context feature enhancement module In order to alleviate the negative impacts of large-scale changes between the objects, we introduce a multi branch structure with dilated convolutions [61] into the feature extraction network to get different receptive fields to cover different scale objects. At the same time, we can obtain different scale context information to help small object detection. The module is not only lightweight, but it also can be plugged into any basic convolutional neural network. This module is inspired by MSCF [55] to construct three parallel convolution branches. The identity structure is added to further strengthen the input features. Meanwhile, compared with Trident Block [62], our module combines features with different scales into a feature map, so that a feature map has the ability to express multiple features. The specific structure is shown in Figure 4. It includes a multi branch structure which can enhance the ability of network feature extraction and dilated convolutions, which can increase the range of receptive field. Our context feature enhanced module divides the input initial feature map into three low dimensional feature maps through a 1 × 1 convolution. Then, the three branches use three dilated convolutions with ratios of 1, 2 and 4 to obtain receptive fields of different sizes. The small receptive fields cover the small objects, while the large receptive fields cover the large objects, and the small objects can obtain context information to assist their detection. Through a concatenation and a 1 × 1 convolution, the feature maps with different feature representations are fused and the channels are expanded. The input feature map and the feature maps with context information are fused by element-wise sum to further enhance the expressive ability of feature map. Finally, the output feature is used to RPN network and ROI feature extraction, respectively. FIGURE 4Open in figure viewerPowerPoint The overall structure of the CFE module 4 DATASET AND EVALUATION METRICS 4.1 Dataset Due to the particularity of airports, there is a lack of public dataset in the field of video-based airport surface object detection. Therefore, the paper constructs an airport detection dataset composed of 26 videos, which includes three types of objects: aircrafts, vehicles and people. In this paper, we set up our own surveillance cameras to obtain the airport scene surveillance video, and we cut out 26 videos related to the airport scene, including sunny days, haze days and nights. Part of the dataset images are shown in Figure 5. The training set includes 21 videos, and the test set includes five videos. The 26 videos are processed by frame extraction. Each video is extracted every three frames. A total of 6330 training set pictures and 1288 test set pictures are obtained. The resolution of each image is 1920 × 1080. Finally, according to the format of PASCAL VOC2007 [63], a total of 25,594 aircrafts, 14,760 vehicles, and 14,803 people, are labelled. In addition, the paper also adopts the definition standard of small objects by Society of Photo-Optical Instrumentation Engineers (SPIE) [64], and we count the number of small objects, the maximum and the minimum pixels of the objects in the original images. The statistical results are shown in Table 1. Meanwhile, the paper shows the scale difference between different objects in the same scene and the scale difference between the same objects in different scenes, as shown in Figure 6. From the scale difference pictures and data statistics table, we can see that the small objects in the airport dataset are mainly people and vehicles, and their pixels are generally too small to be detected correctly. In addition, when the images are trained and detected, the size of images are reduced so that the pixels of objects in the resized images become lower, which increases the difficulty of detection. There are obvious scale differences in the airport dataset, especially for aircrafts, their scale gaps can be more than thousands of times, and for different types of objects, the range of scale change is wide, from several times to thousands of times. FIGURE 5Open in figure viewerPowerPoint Some sample images of the airport dataset TABLE 1. Statistics of object number and pixel in an airport surface dataset Object Samples Small objects Max pixel Min pixel Aircraft 25,594 358 2,258,880 836 Vehicle 14,760 6698 146,584 552 Person 14,803 13,433 17,446 216 FIGURE 6Open in figure viewerPowerPoint (A) Scale difference of the same objects in the near and far scene. (B) Scale difference of different objects in far/near scenes 4.2 Evaluation metrics To reasonably evaluate the detection performance of a detector, it is necessary to select the appropriate evaluation metrics to evaluate. The location and category of the object are needed to detect the objects. The correct position of object prediction is judged by I o U , and its calculation method is shown in (1): I o U = P r e ∩ G T P r e ∪ G T (1) Among them, P r e represents the prediction box area and G T represents the real box area. R e c a l l and P r e c i s i o n are used to evaluate the performance of classification. The calculation formulas are shown in (2) and (3): R e c a l l = T P T P + F N (2) P r e c i s i o n = T P T P + F P (3) Among them, T P means correct prediction, F N means missed detection and F P means false prediction. R e c a l l measures whether all positive samples in the images are detected. P r e c i s i o n measures whether the detected positive samples are the actual positive samples. However, in the actual detection task, the two indicators are negatively correlated. When the threshold is adjusted to make the R e c a l l higher, the P r e c i s i o n decreases, otherwise, when the R e c a l l gets lower, the P r e c i s i o n gets higher. The AP of a certain category is obtained by averaging the accuracy on 11 R e c a l l scales. mAP is obtained by summation and averaging of AP of multiple categories. 5 RESULTS AND DISCUSSION 5.1 Implementation details All experiments are performed on Pytorch. The running environment is Ubuntu 18.04, and the input images are resized to the shortest edge of 600 pixels. In order to ensure the fairness of the experiments, we use ResNet-50 as the feature extractor and use SGD to train on a RTX 2080ti GPU. A total of 80K iterations are carried out, and the initial learning rate is 10−3. After 60K iterations, the learning rate is reduced to 10−4. At t