Aircraft tracking in aerial videos based on fused RetinaNet and low‐score detection classification

IET Image ProcessingVolume 17, Issue 3 p. 687-708 ORIGINAL RESEARCHOpen Access Aircraft tracking in aerial videos based on fused RetinaNet and low-score detection classification M. Ahmed, M. Ahmed orcid.org/0000-0002-9077-3028 Image Processing Center, Beihang University, Beijing, China Contribution: Methodology, Software, Writing - original draftSearch for more papers by this authorAli Maher, Ali Maher orcid.org/0000-0003-1124-2243 Military Technical College, Cairo, Egypt Contribution: Formal analysis, Resources, ValidationSearch for more papers by this authorX. Bai, Corresponding Author X. Bai [email protected] Image Processing Center, Beihang University, Beijing, China State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, China Correspondence X. Bai, Image Processing Center, Beihang University, Beijing, China. Email: [email protected] Contribution: Funding acquisition, Methodology, Project administration, Supervision, Writing - review & editingSearch for more papers by this author M. Ahmed, M. Ahmed orcid.org/0000-0002-9077-3028 Image Processing Center, Beihang University, Beijing, China Contribution: Methodology, Software, Writing - original draftSearch for more papers by this authorAli Maher, Ali Maher orcid.org/0000-0003-1124-2243 Military Technical College, Cairo, Egypt Contribution: Formal analysis, Resources, ValidationSearch for more papers by this authorX. Bai, Corresponding Author X. Bai [email protected] Image Processing Center, Beihang University, Beijing, China State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, China Correspondence X. Bai, Image Processing Center, Beihang University, Beijing, China. Email: [email protected] Contribution: Funding acquisition, Methodology, Project administration, Supervision, Writing - review & editingSearch for more papers by this author First published: 18 October 2022 https://doi.org/10.1049/ipr2.12665AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Aircraft tracking in aerial videos is a challenge due to the small size and appearance affinity with other targets. Recently, the tracking-by-detection trackers, such as simple online and realtime tracking (SORT), attracted the attention of many researchers due to the great evolution in deep learning-based object detectors. SORT depends on motion cues in solving the data association problem by matching high-score detection bounding boxes to tracklets. However, motion cues are insufficient to track occluded or similar aircraft efficiently. In addition, most occluded aircrafts have a low-confidence percentage. The main objective of this paper is to propose an F-SORT tracker, an enhanced SORT tracker with the following improvements. Firstly, fused RetinaNet for small aerial targets is used as a detection framework to boost the detectability of SORT. Then, aircraft appearance features are extracted from the shallowest prediction layer and utilized as additional cues to improve the data association algorithm. Finally, the extracted features distinguish aircraft online from backgrounds via a K-nearest neighbour classifier. Experiments on an airport dataset show that F-SORT can significantly improve aircraft tracking. F-SORT outperforms other state-of-the-art trackers, achieving 72.75%, 59.63%, and 82.89% on MOTA, HOTA, and IDF1. In addition, F-SORT classifying algorithm boosts other trackers when applied. 1 INTRODUCTION Multi-object tracking (MOT), also known as multi-target tracking (MTT), is a crucial sub-branch of computer vision. It aims to identify and track targets over time in successive video frames. MOT is used in many civilian and military applications. In civilians, MOT is used in intelligent video surveillance (IVS), intelligent traffic systems (ITS), virtual reality (VR), and advanced driver assistant systems [1-3]. Also, MOT is used in missiles and unmanned air vehicle guidance systems in military. Due to the evolution in remote sensing field, MOT has recently gained more attention. Using MOT, many videos could be analysed faster than humans. For example, analysing satellite videos allows early knowledge of intentions and directions for enemy ground forces. MOT trackers aim to track multiple objects in different frames. These objects can be pedestrians in street, vehicles on road, a ball in a court, boats in sea, aeroplanes in airports etc. Most existing studies are concerned with tracking pedestrians that could be distinguished by different features such as colour and shape. On the other hand, a few studies are carried out on tracking airplanes due to their importance in civilian and military fields. Multi-aircraft tracking is challenging due to the high intra-class similarity, which imposes difficulties in establishing a distinctive appearance model for each aircraft in the tracking scenario. In addition, aircraft camouflaging using smoke or structural materials may exist in the case of military aircraft tracking. Although many existing studies aim to improve MOT trackers, there are still some challenges facing current trackers, such as changing a target ID during tracking (ID switch), losing a target for some time (fragmentation), and missing a target completely. These challenges occur for many reasons, including partially or fully target occlusion, background clutter, object illumination, scale variation, position change, low resolution of an input image, realtime processing requirements etc. Researchers have resorted to using the "tracking-by-detection" approach to improve the performance of trackers and meet these challenges [4-8]. Especially, after the recent rapid development in detection performance, which positively affects the performance of trackers [9-13]. A detector is first used to detect objects and pass the detected bounding box to a tracker. Then, a tracker matches between detections and targets utilizing any algorithm of distance affinity such as Euclidean distance, IOU (intersection-over-union) value, cosine function etc. Figure 1 shows various scenarios for a failure of a tracker to track aircraft correctly. Figure 1a shows a detector missing an occluded plane (red circle). Consequently, a tracker will miss that plane. As occluded targets are small, many detectors may fail to detect these targets. That reflects the importance of a detector in the "tracking-by-detection" paradigm. Figure 1b shows the detection score confidence above each plane. Most existing trackers only track planes with high-score detection boxes (detection threshold greater than or equal to 0.5) (white labels) due to false positives (red labels) in the low-score detection boxes. However, many planes have low-score confidence (green labels). Missing targets in low-score detection boxes will cause fragmentation and identity switch problems. Thus, the overall performance of a tracker will be affected, and more ghost trajectories will be generated. FIGURE 1Open in figure viewerPowerPoint Different scenarios for a failure of a tracker to track aircraft correctly Figure 1c is the same sequence in Figure 1b but after many frames. Figure 1c shows two cases of changing a target identity during tracking. The first case is illustrated in a dotted red rectangle where two planes change their identities. This switch happens due to the sameness of planes, and the presented features are insufficient to discriminate between them. The second case is for a plane surrounded by a dark purple bounding box. The identity of this plane changed over time as a tracker could not find the affinity between a detection box and a tracklet. Thus, a tracker considers the object as a new target, and a new ID is assigned. So, Figure 1c emphasizes the importance of multiple cues to represent a target. Many studies use the "tracking-by-detection" paradigm to overcome different challenges facing MOT. The simple online and realtime tracker (SORT) [14] is chosen for good performance on MOT benchmarks [15, 16]. SORT consists of a Kalman filter [17] to predict a position of a target in the next frame and a Hungarian algorithm [18] to associate each predicted target to detection produced by a detector. Despite the simplicity and high execution frame rate, the affinity between detections and tracklets depends on insufficient motion cues, especially when the target motion is non-linear, small, partially occluded etc. As a result, Deep_SORT [19] and MF_SORT [20] use another convolutional neural network (CNN) to exploit target appearance features. Thus, the affinity between detections and tracklets relies on motion and appearance cues. Deep_SORT and MF_SORT could boost SORT performance by reducing identity switches and fragments. On the other hand, execution time is significantly affected, and storage size increases due to the additional CNN. DON_SORT [21] improves Kalman filter prediction bounding boxes by embedding orientation cues in target detection. The tracker speed is slightly downgraded. Also, fragment trajectory and identity switches are reduced. Furthermore, DON_SORT possesses high false negatives. Besides, heightened uncertainty for the Kalman filter will significantly affect the performance. Fusion of histogram of oriented gradients (HOG) and colour features tracker (HOG_SORT) [22] is proposed to enhance the affinity distance between detections and tracklets. HOG_SORT depends on adding context and colour features with motion cues. False negatives (FN) and identity switches are reduced. However, HOG_SORT depends on a handcrafted feature extraction method which is less accurate than deep neural networks. In addition, all the mentioned SORT tracker improvements use detections with confidence probability equal to or more than 50%. Most state-of-the-art (SOTA) trackers ignore low-score detection bounding boxes containing many false positives (FP). However, these boxes have a lot of occluded and small targets. ByteTrack [23] is concerned with these low-score boxes. ByteTrack significantly improves SORT tracker performance and increases accuracy and tracking ability. However, a new target will not be initialized without a high-score confidence bounding box. Thus, a target with a low confidence score across the trajectory will not be tracked. This paper aims to improve aircraft tracking in aerial videos by boosting the SORT tracker performance via exploiting tracked aircraft appearance features. Fused RetinaNet [24], a one-stage detector based on deep learning for small target detection in aerial images, is used to extract appearance features. Each detection is associated with a 256-dimensional feature vector that represents a target. These appearance features will be fused with SORT tracker motion cues to possess robust representative features. Thus, the affinity between detections and tracklets relies on motion and appearance cues. In addition, appearance features are used to classify every low-score detection box and decide whether a detection box includes a background or an existing target using a simple neighbour classifier. The proposed featured SORT (F-SORT) tracker could boost SORT tracking performance concerning detection and data association by extracting deep learning features from an existing pre-trained detection model without additional parameters. F-SORT is validated and tested on an airport dataset [21]. Fragments, identity switches, and false negatives (FN) are reduced. In addition, the tracking performance of SORT is significantly improved. On the other hand, the execution time is slightly affected. F-SORT works in realtime and is very simple to be utilized with any SORT tracking system. The contribution of this paper can be summarized as follows: 1. Fused RetinaNet is chosen as a detection framework to improve the detectability of the SORT tracker. In addition, appearance features are exploited from the shallowest prediction layer of the fused RetinaNet. 2. The performance of SORT is improved by combining appearance and motion cues to enhance the affinity between detections and tracked targets. 3. Data association is enhanced by proposing a novel algorithm for classifying low-score detection boxes to decide whether a detected box includes a background or a target using a lightweight lazy classifier. 4. A qualitative content analysis examines the proposed tracker (F-SORT) on an airport video dataset. Besides, a comparison between F-SORT and SOTA trackers is performed. Also, the introduced low-score detection boxes classifying algorithm is applied to improve the tracking accuracy of many SOTA trackers. 2 RELATED WORKS Object detection Recently object detection based on deep neural networks, especially CNN, attracted the attention of many researchers over the traditional methods due to the high accuracy and ability to extract more robust features. Object detectors can be categorized based on the top-most and multi-scale features. Detectors based on the top-most features such as faster RCNN [25] and others [26-29] only detect objects from the deepest layer. These detectors are not performing well for small instances because these objects mainly exist at shallow layers. On the other hand, detectors based on multi-scale features such as SSD [30] and others [31, 32] can predict different scales of objects from various layers of a CNN. However, shallow layers miss semantic information. Thus, detection from shallow layers is affected by this missed information. Many works such as RetinaNet [33] and others [34-39] aggregate different network layers to enrich shallow layers with semantic information and improve the spatial resolution of deep layers. Hence, the overall detection performance is improved, particularly for small objects. Fused RetinaNet is a recent small object detector for targets in aerial images [24]. Fused RetinaNet could boost the detectability of a tracker as occluded targets are small in size. Thus, the proposed tracker adopts fused RetinaNet for the detection step. Target tracking Tracking can be categorized into single object tracking (SOT) or multiple object tracking (MOT) [40]. In SOT, only one object or target is initialized in the first frame by a bounding box, and then a tracker is responsible for tracking the same target in every video frame. While in MOT, multiple targets or objects are initialized in the first frame. Then, a tracker assigns a unique identification number to each target and monitors all targets in a video without switching their identities. MOT can be classified into tracking-by-detection and joint tracking-detection approaches. Detection and tracking phases are carried out individually in the tracking-by-detection approach [14, 19, 21, 22, 41]. Detection bounding boxes are generated from a deep learning network in the detection phase. Then, a tracker finds the affinity between detection boxes and tracklets in the tracking phase. On the other hand, the detection and tracking phases are carried out in one step in the joint tracking-detection approach [4, 5, 42]. Such that the deep neural network could detect and track objects simultaneously. The advantage of the tracking-by-detection approach is the high accuracy, while the joint tracking-detection approach has a high tracking speed. The proposed tracker uses a tracking-by-detection paradigm. Most tracking-by-detection trackers use different cues such as motion and appearance cues in tracking step to match detection bounding boxes to tracklets. More cues about targets increase the performance of a tracker in a complex scene. 2.2.1 Motion estimation model The motion estimation model aims to estimate the location of a target across multiple frames utilizing detection bounding boxes provided by a detector. Kalman Filter is one of the most used approaches to predict a target location in the next frame by knowing a target location in the current frame. SORT uses IOU similarity to compare a predicted target position by Kalman filter in the next frame and a target position in the current frame. The Kalman filter is replaced by a learned correlation filter based on HOG features to improve tracker robustness against target scale variation and reduce fragments [43]. Also, IOU_tracker [44] uses a correlation filter to improve speed and accuracy. A work presented in ref. [45] uses PCA (principle component analysis) to reduce features extracted from a neural network based on a correlation filter to decrease identity switches. Extracting spatial and temporal information from different sequences is another approach many researchers use to estimate the target location based on the variation of target state across multiple frames. TRAT (tracking by attention) aggregates spatial and temporal features extracted from different CNNs using local channel correlation to enhance the prediction of tracked targets [46]. An existing work learned spatial correlation filters from temporal information of previous frames to improve target motion prediction [47]. Another work proposed an end-to-end spatial–temporal network that could provide a learnable location representation of targets [48]. Many researchers have used recurrent neural network (RNN) [49-52] to predict a target location in the next frame. However, RNN requires massive data for training. Moreover, the tracking accuracy of RNN would be unsatisfactory. Recently, other presented works depending on single object tracking (SOT) were used to improve multi-target tracking [53-56]. Despite existing SOTA (state-of-the-art) trackers based on a motion model only achieving an acceptable tracking accuracy, tracking performance based on a motion model is unsatisfactory, especially in a crowded scene. 2.2.2 Appearance model The appearance model represents each target by distinctive features in all frames. Such target features across different frames should be more similar than features of other targets. Hand-crafted features such as HOG, colour histogram, shape, optical flow etc., are used to represent a target [22, 57, 58]. Nevertheless, the tracking performance is not good in complex scenes. By the arising of deep neural networks, extracted features representing targets become more robust. So, many existing works rely on deep neural networks to exploit more features about a target. A work presented in ref. [59] uses an offline pre-trained CNN to enrich the similarity function with additional features. Also, the work presented in ref. [60] uses a CNN learned online to enhance the discriminability between different tracked targets. Many works use Siamese networks to measure the similarity between two feature vectors [61]. Triplet loss significantly helps lost target re-identification via different feature comparisons [62]. Thus, the appearance features could boost the tracking performance. However, extracting appearance features affects the speed of a tracker, mainly if an additional neural network is used. 2.2.3 Data association Data association is responsible for matching detection bounding boxes to tracklets according to their similarity. Similarity measurement uses cues about a target, such as motion and appearance. Also, data association assigns identities to matched targets. SORT uses the Hungarian algorithm [18] to link detection boxes with tracked targets via IOU similarity. CenterTrack [63] utilizes the greedy matching algorithm [64] to associate bounding box results from a detector to a tracklet. MOTDT [65] uses a tracklet confidence to deal with the performance degradation in the Kalman filter for many missed detections across a track. Besides, MOTDT uses Euclidean distance similarity based on appearance cues to match detection boxes with tracklets and then uses IOU similarity based on motion cues to match the other tracklets. JDE (joint detection and embedding) [4] and others [66-68] are embedding an association head beside classification and regression heads to reduce tracker execution time. The result of an association head branch is passed to the Hungarian algorithm for data association. GMTracker (learnable graph matching tracker) [69] solves the data association problem by a graph matching approach based on appearance cues extracted from a deep neural network for each target. Even though many studies enhance the data association by using different matching algorithms, detection boxes still play a vital role in improving the data association. Most studies use detection boxes with high-score confidence and ignore those with low-score confidence. Nevertheless, most occluded targets have low-score detection boxes, and missing these boxes will affect the performance of a tracker. ByteTrack handles all detection boxes in two association steps. The first association links high-score confidence detection boxes with tracklets by measuring the similarity based on different cues. Then, a second association assigns unmatched tracklets from the first association with low-score detection boxes by measuring the IOU similarity based on motion cues. Inspired by these works, a proposed tracker utilizes motion and appearance cues to enhance the data association of the SORT tracker. In addition, it uses low-score detection boxes for more improvements in the data association and to handle occluded targets in a complex scene. The proposed tracker extracts the appearance features from an object detector in the detection phase. Thus, the proposed tracker has a low computational cost as a SORT tracker. In addition, different SOTA trackers could use the algorithm for further improvements. 3 METHODOLOGY F-SORT MOT tracker follows the "tracking-by-detection" paradigm. This section briefly illustrates the components of the fused RetinaNet detection framework. In addition, a detailed explanation of F-SORT will be presented. Fused RetinaNet RetinaNet is a one-stage detector adopted by Facebook AI research (FAIR) in 2017 [33]. This detector comes with an idea out of the box using a weighted classification loss function named focal loss. The additional hyperparameters used in classification loss solve an imbalanced class problem. This improvement let RetinaNet become accurate as a two-stage detector. It outperforms many SOTA detectors, mainly when a scene includes small and dense-area instances [70]. Moreover, RetinaNet inference speed is fast as one-stage detectors. Fused RetinaNet [24] is an enhanced RetinaNet detector that can detect more small and dense objects by fusing multi-level layers to enrich the lower layers with semantic information. Thus, the small object detection of lower layers is improved. In tracking, targets are small when entering or leaving a scene, partially or fully occluded, moving far away from a camera etc. So, the Fused RetinaNet detector is used to detect targets accurately, especially when the targets are small, according to the abovementioned reasons. RetinaNet and Fused RetinaNet models produce only a bounding box, a label, and a confidence score for each detection. Thus, no representative features are produced by these detectors. In this paper, the Fused RetinaNet detector is modified by adding a proposed feature extraction module to associate each detection with a bounding box, a label, a confidence score, and a feature vector without additional training on the Fused RetinaNet model. In this paper, an improved Fused RetinaNet is a fused RetinaNet with the addition of a feature extraction module to allow the Fused RetinaNet detection model to extract features of targets besides bounding boxes, labels, and confidence scores. Thus, an improved Fused RetinaNet is picked up as a detector part for the proposed tracker. The improved Fused RetinaNet includes Fused RetinaNet with the addition of a feature extraction module to include an extra vector representing the features of a detected object. Figure 2 shows the Fused RetinaNet, consisting of a ResNet-50 backbone, fusion and context network, and classification and regression subnets. In order to solve the multi-scale variation problem, target detection is produced from five layers (P3, P4, P5, P6, and P7). The shallowest prediction layer (P3) is suitable for predicting small objects. Thus, the feature extraction module utilizes the P3 layer to extract the feature vector for each target. The feature module takes a bounding box for each target as input and exploits a feature vector representing each object. So, Fused RetinaNet output is bounding boxes, scores, classifications, and feature vectors. The output detections of Fused RetinaNet are passed to the proposed tracker to localize the existing targets. FIGURE 2Open in figure viewerPowerPoint The structure of improved Fused RetinaNet includes Fused RetinaNet with a feature extraction module. The feature vector of each detected object is extracted from the shallowest prediction layer (P3) by utilizing the detection bounding box to crop the feature maps by the appearance feature module. Then, the feature extraction module output a fixed feature vector equal to 256 × 1 for each detected object. Thus, the model output of Fused RetinaNet is bounding boxes, scores, classifications, and feature vectors The proposed F-SORT tracker F-SORT tracker is a featured SORT tracker with three main aspects: F-SORT uses Fused RetinaNet as a detection framework instead of Faster RCNN. F-SORT uses additional cues (appearance features) besides motion cues to measure the affinity between detection boxes and tracklets. F-SORT filters low-score detection bounding boxes before target initialization. F-SORT includes a motion estimation model based on the Kalman filter and data association, as shown in Figure 3. The motion estimation model predicts a target orientation in the next frame based on motion cues. In contrast, data association aims to assign detections of the current frame to targets in the previous frame based on appearance and motion cues. FIGURE 3Open in figure viewerPowerPoint The structure of the proposed tracker (F-SORT) 3.2.1 Kalman filter motion estimation model The Kalman filter motion estimation model illustrates the ability of a tracker to predict instance orientation in the next frame, where each object has a state vector, as shown in Equation (1). The state vector (x) represents centre (u,v), area (s), and aspect ratio (r), respectively. The aspect ratio is assumed to be constant. The inter-frame displacement is calculated via a Kalman filter with a constant velocity. x = [ u , v , s , r , u ̇ , v ̇ , s ̇ ] T $$\begin{equation} x = [u,v,s,r,\dot{u},\dot{v},\dot{s}]^T \end{equation}$$ (1) Assigning a track for each target is done by activating a counter initialized by the birth of tracked target until the last successful detected time. Then, the counter is zeroed when a track is set to a target. All tracks that have exceeded the specified maximum age will be removed. Also, any track set to a new target and not successfully associated with a target for three successive frames will be automatically deleted. 3.2.2 Feature extraction module (appearance model) Appearance features are exploited by utilizing the feature extraction module shown in Figure 2. The structure of feature extraction module is simple and applied to Fused RetinaNet for extracting a feature vector representing each detected object, as shown in Figure 4. In addition, the feature extraction module could be used with any detector without any additional training. A crop and resize function takes the feature maps (256) of the prediction layer (P3) and crops each feature map by the given detection bounding boxes to extract a feature vector for each detected object. The output of the crop and resize function is fixed to a size of 14 × 14 for all feature maps. Thus, the extracted features are robust to various scales of tracked targets. Finally, feature maps of each object are encoded by a global average pooling layer to produce a feature vector with a dimension of 256 × 1. Each feature vector that represents a target appearance cue is given to an affinity matrix to measure the similarity between detected object features and tracklets. FIGURE 4Open in figure viewerPowerPoint The structure of the proposed feature extraction module. Firstly, a crop and resize function takes the prediction layer (P3) with a dimension of w × h × 256 $w\times h\times 256$ and detection bounding boxes as input. Then, the function crops each detected object by a fixed size of 14 × 14 × 256 $14\times 14\times 256$ . Finally, an average pooling layer is applied to reduce each extracted object to a feature vector with a dimension of 256 × 1 3.2.3 Data association Data association aims to find the affinity between detection bounding boxes and tracklets using a cost matrix, a probability matrix representing each detected instance probability to all targets. The input of the cost matrix is appearance and motion cues. The cost matrix uses IOU distance matrix to measure affinity between detected and estimated target locations. In addition, the cost matrix measures the similarity of features between detected targets and tracklets by utilizing the cosine distance matrix based on extracted feature vector. The cost matrix fuses the tw