Multiple object tracking using a dual‐attention network for autonomous driving

IET Intelligent Transport SystemsVolume 14, Issue 8 p. 842-848 Special Section: Sustainably Intelligent Mobility (SIM)Free Access Multiple object tracking using a dual-attention network for autonomous driving Ming Gao, Ming Gao College of Transportation, Jilin University, Changchun, People's Republic of ChinaSearch for more papers by this authorLisheng Jin, Corresponding Author Lisheng Jin jinls@ysu.edu.cn College of Transportation, Jilin University, Changchun, People's Republic of China School of Vehicle and Energy, Yanshan University, Qinhuangdao, People's Republic of ChinaSearch for more papers by this authorYuying Jiang, Yuying Jiang Department of Ophthalmology, China-Japan Union Hospital of Jilin University, Changchun, People's Republic of ChinaSearch for more papers by this authorJing Bie, Jing Bie Department of Civil Engineering, University of Nottingham Ningbo China, Ningbo, People's Republic of ChinaSearch for more papers by this author Ming Gao, Ming Gao College of Transportation, Jilin University, Changchun, People's Republic of ChinaSearch for more papers by this authorLisheng Jin, Corresponding Author Lisheng Jin jinls@ysu.edu.cn College of Transportation, Jilin University, Changchun, People's Republic of China School of Vehicle and Energy, Yanshan University, Qinhuangdao, People's Republic of ChinaSearch for more papers by this authorYuying Jiang, Yuying Jiang Department of Ophthalmology, China-Japan Union Hospital of Jilin University, Changchun, People's Republic of ChinaSearch for more papers by this authorJing Bie, Jing Bie Department of Civil Engineering, University of Nottingham Ningbo China, Ningbo, People's Republic of ChinaSearch for more papers by this author First published: 07 April 2020 https://doi.org/10.1049/iet-its.2019.0536Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Multiple object tracking (MOT) remains an open and challenging problem for autonomous vehicles. Existing methods mainly ignore prior information from real traffic scenes. Here, the authors propose a novel MOT algorithm that considers traffic safety for vulnerable road users. The proposed method integrates two attention modules with a novel detection refinement strategy. Since skilled drivers pay more attention to pedestrians and cyclists, the authors employ a saliency detection method to extract scene attention region. Then, a detection refinement strategy achieved a good trade-off between parallel single object trackers and detection results. Channel attention can mine the most useful feature channel for traffic road users. In the end, the authors operate their method on the popular MOT 17 benchmark in comparison with other high-level MOT algorithms. The tracking results show that the proposed dual-attention network achieves the state-of-the-art performance. 1 Introduction Multiple object tracking (MOT) has appeared as one of the most fundamental tasks in the field of computer vision. For a given input video, the goal of MOT is to locate multiple objects, keep their identities, and generate individual accurate trajectories [1]. The importance of MOT is reflected by the wide variety of applications including intelligent surveillance systems [2], autonomous driving [3], intelligent transport systems [4], medical image processing [5], and sports analysis [6]. In general, most recent multiple object trackers employ a popular tracking-by-detection paradigm, which consists of detection generation and data association. For traffic scenes, the MOT actually focuses on pedestrians and vehicles, which aim to generate stable motion trajectories. However, the MOT problem is still challenging due to faulty and noisy detections, frequent and heavy occlusions, similar appearance, appearance variations, and the initialisation and termination of objects. When dealing with the main challenges of MOT, researchers tend to divide two rough categories from the processing mode [7]. Online tracking only depends on the current image frame to count the number of multiple objects and locate object positions, which is highly suitable for real-time applications. Conversely, offline tracking utilises both the current and future frames to generate more accurate final trajectories. However, the average performance of batch tracking is better than that of online tracking. These methods require substantial computational resources and storage memory, making them poorly suited for real-world situations. From the above analysis, online multiple tracking is more capable of autonomous driving. Nevertheless, in real autonomous driving scenes, MOT faces more challenging environments, such as changeable weather, high-density objects, and spatial and temporal variances of background, which increase the complexity of occlusions. As a solution to occlusions, researchers have proposed graph-based methods. One popular graph theory for online MOT is the min-cost network flow [1], which models each detection object as a graph node and the link between nodes as a trajectory. The conditional random field also plays an important role in graph-based multi-target tracking [8]. The above graph theory-based online multiple tracking algorithms have achieved great success, yet a gap remains for autonomous vehicle applications. Recently, the end-to-end deep learning framework has provided a good choice for solving the challenging factors of MOT in an overall level, which benefits from the boom of deep neural networks. A recurrent neural network (RNN) can dig the long-term dependencies, which is appropriate for modelling the motion and interaction patterns of moving objects. Milan et al. [9] proposed a state estimation module RNN combined with a data association module LSTM (long short-term memory) to train an end-to-end multi-target tracker. Similarly, Sadeghian et al. [10] presented an online multiple RNN structure that can learn appearance, motion, and interaction. Based on the classical multiple hypothesis tracking (MHT) framework, Kim et al. [11, 12] incorporated the bilinear LSTM to achieve near-online tracking. Although this network can solve the motion model problem, it is insufficient for representing the image feature. Therefore, researchers have found a promising solution: fusing the RNNs and convolutional neural networks (CNNs). Cong et al. [13] proved that Siamese bi-gated recurrent unit can cleave and re-connect tracklets to overcome various occlusions. Babaee et al. [14] are inspired by person re-identification task that proposed a CNN module to extract appearance features, an RNN module to encode spatiotemporal features, and a novel hierarchical clustering mechanism to achieve tracklet association. Different from general discriminative methods, a recurrent autoregressive network [15] was proposed to obtain a generative modelling framework that couples an external memory and an internal memory. Despite the merits of the CNN and RNN combination, single CNN-based trackers have also attracted attention. Many CNN-based trackers have been proposed to integrate lightweight single object trackers (SOTs) into multiple object trackers. Successful integration lies in the advanced SOT obtaining competitive results compared to the detection algorithm. Qi et al. [16] used a CNN-based SOT and popular spatial–temporal attention to achieve the state-of-the-art performance. In a similar fashion, a cost-sensitive tracking loss combined with an efficient convolution operators [17] tracker and dual-attention maps was proposed to compress noisy detections [18]. To better utilise the SOT tracker to refine the accurate detection results, a switcher-aware classification was proposed that includes an SOT submodule to get short-term cues and a reidentification submodule to capture long-term cues [19]. Recently, an instance-aware tracker with a dynamic model refreshing strategy was proposed to improve the tracking robustness [20]. In summary, with the help of the SOT tracker component, MOT trackers have made more progress. Aiming to operate on an autonomous driving perception system and leveraging the advantages of deep learning, we proposed a dual-attention network to track multiple objects. The core motivation of this paper is to assign hierarchical attention at various levels for objects of different importance levels. To utilise the end-to-end deep learning pipeline, deepMOT [21], which can directly optimise MOT metrics from a differentiable framework, was employed as our baseline tracker to integrate attention modules. The base tracker makes good use of the representative power of the convolutional feature and the motion model ability of the RNN. Subsequently, a novel scene attention module was developed to protect vulnerable road users (VRUs). This module derives from saliency detection which is inspired by the human visual system to generate the most important area. Finally, a channel attention module was incorporated with a deep convolutional architecture to reflect and reweight the channel feature. The contributions of this paper are summarised as follows: A novel dual-attention MOT algorithm was proposed for autonomous driving scenes. We investigated a novel scene attention module to classify different importance levels for input images to protect VRUs. A novel channel attention module was presented to better extract the image feature for bad detections. The proposed MOT tracker has achieved competitive performance on the MOT 17 benchmark. The rest of the paper is organised as follows. In Section 2, a detailed description of the developed MOT algorithm is given. In Section 3, experimental quantitative and qualitative results from the popular benchmark are shown and finally conclusions and future work are drawn in Section 4. 2 Proposed dual-attention network In this section, we first introduce the deepMOT pipeline and then introduce the two attention based modules that capture the most important information in the image area and channel feature. Finally, we describe our novel detection refinement strategy. 2.1 Baseline deepMOT 2.1.1 Problem formulation We first formulate the mathematical expression of SOT in MOT. The tracking process can be modelled as an optimisation problem, the overall target location set in the current image is denoted as , and the candidates in the detection set are . Then, the tracking process maximises a score: (1) (2) where is the association parameter between the ith targets in at frame and the jth candidate location in a frame t. When , is associated with ; otherwise the association does not exist if . Note that each candidate can only be assigned to no more than one target. is the set of learnable parameter pools, which can learn from the appearance, motion, and interaction of targets. The objective function is the metric of global quality of these tracking results for all at frame t, which is defined as follows: (3) where is the single object tracking objective function on behalf of the assigned score jth candidate location on in accordance with the ith model parameter . 2.1.2 DeepMOT pipeline For the tracking-by-detection paradigm, the coarse detection results possibly import noisy background information and mismatches. A potential solution is to amalgamate the SOT tracker to refine the original detections. As discussed in the introduction, SOT in MOT could produce promising results. The highlights of deepMOT incorporate the SOT module into MOT and directly optimise the index MOTA (MOT accuracy) and MOTP (MOT precision). For every single object, deepMOT considers the original confidence , the reference single tracking result , and the weight of SOT . Formally, the final detection result can be written as follows: (4) In order to achieve a differentiable MOTA and MOTP networks, the network computes the difference between estimated bounding box and the ground truth bounding box during the training process. Inspired by the semantic segmentation task, the network is based on an intersection over union (IoU) index to calculate similarity. The proposed similarity matrix is as follows: (5) where the term denotes the normalised Euclidean distance which is suitable for image pixel distance calculation. Then, deepMOT proposed the deep Hungarian network (DHN) to attach and the loss module. The DHN inspired by the original Hungarian algorithm [22], which can calculate the optimal soft-assignment matrix from the similarity matrix . The DHN is composed of two stages of hidden representation and two bidirectional RNNs (BiRNNs). The differentiable operation DHN can be as follows: (6) where are the weights of the BiRNN. Note that the DHN only updates during the training process. To better understand the differentiable approximation of MOT metrics, we first introduce the definition of MOTA and MOTP. The above indexes are part of the CLEAR MOT [23] metrics, which can evaluate the overall tracking results of MOT trackers, and are defined as follows: (7) (8) where MOTP denotes the total normalised estimated position error over all frames, which can show the ability of the tracker to identify precise object positions. Then in (2), , , and present the number of misses, of false positive (FP), and of mismatches (IDS), respectively, for the period t. The MOTA is a comprehensive index that can account for all target errors resulting from the tracker and the above failure cases. The differentiable and can be calculated from each differentiable component in (4) and (5). The final deepMOT loss considers both MOTA and MOTP metrics at every time step: (9) where is a calibration factor. Via minimising the loss, the network can find the best parameters and achieve end-to-end tracking. The simplified deepMOT pipeline can be shown in Fig. 1. Fig. 1Open in figure viewerPowerPoint Simplified deepMOT pipeline 2.2 Scene attention module For traffic scenes, there are likely dozens of objects in the vehicle perspective simultaneously. Yet not all of the objects share the same importance, some objects even may induce serious driving secondary tasks [24]. Considering traffic safety, the closer VRUs should be of greater concern, which is the essential characteristic of a skilled driver mindset. From the perspective of bionics, we define this characteristic as scene attention. Thus, we combine a saliency detection module to help find significant users and area. The concept of salient detection comes from the attention research of neuroscience. Thanks to Illi's seminal work [25], salient detection has become a popular topic of cognitive psychology, neuroscience, and computer vision [26]. Generally, salient detection mainly focuses on detecting the most salient object and segmenting the accurate region of that object. It is a fundamental process of many high-level computer vision tasks. In this paper, we utilise salient detection to find the most important area in a traffic scene image. A novel salient detection method [27], which proposed a dynamic spatiotemporal saliency network, can be a good choice for our attention module. The input of this network consists of a static saliency subnetwork and image pairs, and the output of this network is a spatiotemporal map, as shown in Fig. 2. The static saliency network feeds a single RGB image and generates a non-scale change saliency map . This static network models the process as a fully convolutional network (FCN). The deconvolution subnetworks are on the right of the convolution layers: (10) where I is the input image, presents the final feature maps calculated by the convolutional layers with stride value S, and is the deconvolution layers that achieve the same size between input I and output Y by upsampling with parameter S. 's are learnable parameters. During the training process, we have many labelled samples , where the input is an RGB image, and the ground truth saliency binary map ; then, the training loss on network prediction P can be given by (11) where and , and denotes the ratio between salient pixels and the whole pixels in the ground truth image. Fig. 2Open in figure viewerPowerPoint Scene attention module The dynamic network is similar to the static network which is also derived from the FCN and appends multiple convolution and deconvolution layers. In contrast, the dynamic network jointly learns the dynamic saliency information and static saliency results, which can generate spatiotemporal saliency estimation. Note that the input of this module is frame pair and last step static saliency map , thus the output is a dimension tensor I : (12) where W 's denote different convolution kernels and b is the bias parameter. Compared with previous salient detection methods, there are two obvious advantages that make the algorithm gain more computation efficiency. One is explicitly fusing the dynamic and static saliency to generate a final saliency map instead of a two-stream network structure. Secondly, the method we used reasoning the temporal information from adjacent frames rather than utilising optical flow. 2.3 Channel attention module As discussed in the last section, the scene attention module could find the most important area in an image to help the algorithm focus on VRUs. We also observed that different channels of CNN's feature map can express unequal meaningful information. Thus, a channel attention model aimed at focusing on the most relevant features to get more useful information was used. The operating principle is to choose semantic attributes for different tracking contexts [28]. Plenty of works have proven the efficiency of this model in the field of computer vision. For channel attention, there are two main types of pooling methods: global average pooling and global max pooling. From early experiment analyses, we take global average pooling to generate channel descriptors. For a given d channel feature with , denotes the input channels. By shrinking the input feature Z, we can capture channel-wise statistic value . The cth component of u can be calculated by (13) where denotes the global pooling function and denotes the value at position of the cth feature . To fully use the channel-wise dependencies and following the two criteria, we utilise an easy gating mechanism with a popular sigmoid function to activate: (14) where and are the sigmoid function and ReLU function, respectively, and and are the weights of the convolution layer and ReLU layer, respectively. Then the final feature map can be calculated as follows: (15) where and are a learnable scaling factor and feature map for channel attention, respectively. The whole process is shown in Fig. 3. The channel attention module consists of five layers, including an average pooling layer to reduce dimension, an ReLU layer with a fully connected layer to increase the network's non-linear characteristic, and a fully connected layer with a sigmoid activation layer to activate the network. Fig. 3Open in figure viewerPowerPoint Simplified deepMOT pipeline 2.4 Dual-attention network In the original deepMOT, the tracker operates the SOT tracker in parallel on all given detection bounding boxes. However, the given bounding boxes possibly could not provide perfect information for single object tracking. For some bad detections, such as low-overlap detections, we need to refine the detection results to improve the tracking performance. Thus, we propose a novel SOT detection refinement strategy. The motivation of this strategy is to assign different detection components according to significance confidence. First, we input the image to the scene attention module Y to extract the scene attention area. After postprocessing, the connected scene attention region includes no more than three objects. For scene attention region, we utilise the SOT tracker as the original tracker. The SOT tracker can achieve better performance on distinguishing the specified object from the background and accurately locating the object. For the external attention region, we check the detection confidence . If the confidence is greater than the predefined threshold , we directly transmit the image to the appearance extractor branch . Otherwise, we use the channel attention module to refine the appearance of the extra branch. The proposed detection strategy ensures that sensitive road users have received more attention. The flowchart is shown in Fig. 4. Fig. 4Open in figure viewerPowerPoint Proposed dual-attention network architecture 3 Experimental evaluation In this section, we perform our proposed MOT algorithm in comparison with other competitive trackers on the popular MOT 17 dataset. The MOT17 dataset is a representative benchmark, where the filmed sequences consist of pedestrians and vehicles in a free and crowded environment. 3.1 Experiment implementation details The presented approach is implemented in an Ubuntu 16.04 desktop computer with an Intel i7-7700 CPU, 16G RAM, and an Nvidia GTX1080Ti GPU. The development environment is Pytorch1.0. For the salient detection module, we first offline training the model and then fine tune the model by manually annotating MOT 16 images. During the testing process, we fixed the weight parameter. The SOT tracker in this paper is a modified SiameseRPN [29]. The threshold is empirically set to 0.6. For appearance extractor branch , we utilise VGG-16 [30] with retraining first the two pooling layers. In addition to MOTA and MOTP, the CLEAR MOT [23] metrics have other evaluation indexes. We also give the results of percentage of mostly tracked objects (MT), FP, ID switches (IDS), false negatives (FN), the percentage of right identified detections (IDF1), fragmentation (FM), and the percentage of mostly lost objects (ML). To note that the thresholds of MT and ML are 80 and 20%, respectively. 3.2 Results on MOT 17 3.2.1 Quantitative analysis The MOT 17 benchmark consists of crowded real-world video sequences for both dynamic and static scenes. Due to the difficulty of MOT annotation, the dataset provides seven sequences for training, seven sequences for testing, and a totally 564,228 manually annotated ground-truth bound boxes. The sequences are difficult to track mainly due to the high density of average targets, various occlusions, and appearance changes. For the purpose of comparing the association method on different detection algorithms, the benchmark provides three different detection results: deformable part model (DPM) [31], Faster R-CNN [32] and scale dependent pooling (SDP) [33]. The tracking results are shown in Table 1. Both online and offline methods are concerned. The best and second scores are in red and green, respectively. Observed from the results, our tracker achieves the best performance on the MOTA, MOTP, and FN indexes for the online methods, and operates competitively compared with more complex offline trackers. Due to the utilisation of multiple frames of information, the average overall performance of offline trackers is better than that of online methods. However, with the help of deep learning, the gap between online and batch tends to be smaller. Gaussian mixture probability hypothesis density (GM_PHD) [34] and simple online and realtime tracking (SORT) [35] are 2 early non-deep learning methods with slightly weaker results on MOTA. Benefiting from CNN, feature pyramid siamese network (FPSN) [36] and dual matching attention network(DMAN) [18] show a clear improvement over MOTA and MOTP. While DMAN is also based on the attention network to get rather good performance, our proposed tracker outperforms DMAN on MOTA and MOTP. The success of our tracker might be due to the combination of a CNN and an attention module. Table 1. Comparisons with state-of-the-art online methods on the MOT17 benchmark Mode Method MOTA(↑) MOTP(↑) IDF1(↑) MT(↑), % ML(↓), % FP(↓) FN(↓) IDSw(↓) offline IOU 45.5 76.9 39.4 15.7 40.5 19,993 281,643 5988 — MHT_bLSTM 47.5 77.5 51.9 18.2 41.7 25,981 268,042 2069 — EDMT 50.0 77.3 51.3 21.6 36.3 32,279 247,297 2264 — MHT_DAM 50.7 77.5 47.2 20.8 36.9 22,875 252,889 2314 online GM_PHD 36.4 76.2 33.9 4.1 57.3 23,723 330,767 4607 — SORT 43.1 — 39.8 12.5 42.3 28,398 287,582 4852 — FPSN 44.9 76.6 48.4 16.5 35.8 33,757 269,952 7136 — DMAN 48.2 75.7 55.7 19.3 38.3 26,218 263,608 2194 — Ours 48.4 76.9 43.7 17.6 39.0 24,915 262,654 3423 To explore the internal relationship between detectors, we compare the performances on MOT17 with different detectors in Table 2. As shown in the table, the overall performance with the SDP detector is better than the DPM and Faster R-CNN (FRCNN) detectors. SDP as well as FRCNN are both deep learning detection methods that are more capable of detecting objects than traditional DPM. Note that the difficulty trend between the detectors is the same. It seems that testing sequence 14 is the most challenging, while testing sequence 3 is fairly easy. From comparisons, we can infer that finding a better detector and refining worse detection are still two fundamental strategies for MOT. Table 2. Comparisons of proposed method with three individual detectors on the MOT 17 benchmark Sequence MOTA(↑) MOTP(↑) IDF1(↑) MT(↑) ML(↓) FP(↓) FN(↓) IDSw(↓) MOT17-01-DPM 31.1 73.6 22.2 3 11 333 4071 43 MOT17-03-DPM 49.8 76.4 40.3 31 39 1319 51,095 157 MOT17-06-DPM 34.3 71.0 30.8 20 127 942 6643 155 MOT17-07-DPM 29.8 71.7 28.0 4 34 491 11,309 61 MOT17-08-DPM 22.7 74.8 21.4 6 42 402 15,848 75 MOT17-12-DPM 36.7 77.9 42.0 8 51 247 5216 24 MOT17-14-DPM 19.1 74.2 21.5 2 106 1076 13,753 117 average 31.9 74.2 29.5 11 59 687 15,419 90 MOT17-01-FRCNN 20.2 75.8 29.9 9 6 2187 2922 41 MOT17-03-FRCNN 60.6 77.6 53.6 42 26 2308 38,694 222 MOT17-06-FRCNN 49.9 78.1 44.5 53 59 1007 4666 226 MOT17-07-FRCNN 33.0 75.3 36.2 5 19 1081 10,019 211 MOT17-08-FRCNN 24.1 78.8 28.4 9 40 1147 14,812 82 MOT17-12-FRCNN 34.9 77.9 46.0 11 44 777 4830 34 MOT17-14-FRCNN 16.1 71.5 28.4 5 80 2366 12,551 587 average 37.1 77.3 39.8 22 32 1418 12,657 136 MOT17-01-SDP 46.7 74.7 36.7 9 4 1169 2219 49 MOT17-03- SDP 75.7 78.4 58.2 81 13 2675 22,421 304 MOT17-06- SDP 50.7 76.5 45.2 66 67 1194 4403 215 MOT17-07- SDP 46.8 76.0 40.4 14 11 1372 7485 133 MOT17-08- SDP 30.2 78.6 27.3 13 32 649 13,952 142 MOT17-12- SDP 40.5 78.8 47.0 19 43 696 4414 46 MOT17-14- SDP 28.0 72.7 35.4 4 65 1477 11,331 499 average 41.8 75.9 39.8 26 39 1450 9847 247 3.2.2 Qualitative analysis Fig. 5 denotes the proposed tracker result fragments on MOT 17 testing sequences. In sequence 1, the camera is static on the ground, we only focus on the woman in black. In the tracking process, the woman interacts with the walking woman from the opposite way, while our tracker can track the woman accurately. There are tens of pedestrians on the street at night shoot from the elevated viewpoint camera in sequence 3. We pay attention to the man looking at the mobile phone and a cluster of four pedestrians. Since the motion pattern of the man is different from others, the speed is slightly lower than a normal pedestrian. The proposed tracker with good distinguishing ability can overcome the noisy background to achieve good tracking results. Continuous occlusion is one of the most troublesome hindrances for MOT. During the tracking process in sequence 6, the presented tracker undergoes two heavy occlusions and can still locate the man precisely under a moving platform. Bad detection is also very challenging for MOT. The initial image of sequence 7 contains a partially occluded man, while the tracker solves the occlusion problem. In real traffic situations, pedestrians are more likely to go together, which may induce more random and complex interactions. For this purpose, we observe the tracking performance on the pedestrian group in sequence 8. The two women are accompanied by each other on a crowded street, which is filmed by a stationary camera. They are completely overlapped by another pedestrian group, yet the proposed algorithm does not lose the objects. In addition to outdoor scenes, MOT 17 also consists of an indoor scene. In testing sequence 12, there are many people in a busy shopping mall. The presented tracker succeeded in tracking the fast-moving and temporary occlusion man. In testing sequence 14, the bus station is so crowded that even if we utilise a detection refinement strategy, the tracking algorithm still loses many objects. However, our method performs well on the scene attention region (blue ellipse), where many pedestrians are walking on the pavement. Fig. 5Open in figure viewerPowerPoint Qualitative tracking analysis on MOT 17 benchmark (from top to bottom, the image sequence are 1, 3, 6, 7, 8, 12, 14) 4 Conclusion Real-world practical MOT for autonomous driving remains challenging because of the contradiction between high computing cost and low compute power on automobile-embedded processors. To address this problem, we propose a dual-attention network to achieve end-to-end tracking. In particular, a saliency detection-based scene attention module handles the attention region, which can select the area where the algorithm should pay more attention. Then, a novel detection strategy with channel attention can efficiently handle detection refinement. The tracking performance on the MOT 17 benchmark shows that our tracker is competitive in comparison with other state-of-the-art multiple object trackers. In future work, we aim to extend this work by adding another SOT tracker or lighter salient detection module to achieve faster tracking speed. 5 Acknowledgments This work was supported in part by the National Natural Science Foundation of China under grant no. 51575229, in part by the National Key Research and Development Program of China under grant no. 2018YFB1600500, in part by the National Key Research and Development Program of China under grant no. 2017YFB0102600, and in part by the Electric Intelligent Vehicle Innovation Team of the Science and Technology Department of Jilin Province. 6 References 1Zhang L. Li Y. Nevatia R.: ' Global data association for multi-object tracking using network flows'. IEEE Conf. on Computer Vision & Pattern Recognition, Anchorage, AK, USA., 2008 2Benfold B. Reid I.: ' Stable multi-target tracking in real-time surveillance video'. Computer Vision & Pattern Recognition, Providence, RI, USA., 2011 3Liu J. Cao X. Yan L. et al.: 'Online multi-object tracking using hierarchical constraints for complex scenarios', IEEE Trans. Intell. Transp. Syst., 2017, PP, (99), pp. 1– 11 4Sun Y.D. Yang J. Liu H.B. et al.: 'Real-time multi-class moving target tracking and recognition', IET Intell. Transp. Syst., 2016, 10, (5), pp. 308– 317 5Turetken E. Wang X. Becker C.J. et al.: 'Network flow integer programming to track elliptical cells in time-lapse sequences', IEEE Trans. Med. Imaging, 2017, 36, (4), pp. 942– 951 6Romeas T. Guldner A. Faubert J.: '3d-multiple object tracking training task improves passing decision-making accuracy in soccer players', Psychol. Sport Exercise, 2016, 22, pp. 1– 9 7Luo W. Xing J. Milan A. et al.: ' Multiple object tracking: a literature review', 2014 8Nevatia R.: ' An online learned crf model for multi-target tracking'. Computer Vision & Pattern Recognition, Providence, RI, USA., 2012 9Milan A. Rezatofighi S.H. Dick A. et al.: ' Online multi-target tracking using recurrent neural networks', 2016 10Sadeghian A. Alahi A. Savarese S.: ' Tracking the untrackable: learning to track multiple cues with long-term dependencies'. IEEE Int. Conf. on Computer Vision, Venice, Italy, 2017 11Kim C. Li F. Rehg J.M.: ' Multi-object tracking with neural gating using bilinear Lstm' ( Springer International Publishing, Germany, 2018) 12Kim C. Li F. Ciptadi A. et al.: ' Multiple hypothesis tracking revisited'. IEEE Int. Conf. on Computer Vision, Santiago, Chile, 2015 13Cong M. Yang C. Fan Y. et al.: ' Trajectory factory: tracklet cleaving and re-connection by deep Siamese bi-gru for multiple object tracking', 2018 14Babaee M. Athar A. Rigoll G.: ' Multiple people tracking using hierarchical deep tracklet re-identification', 2018 15Fang K. Yu X. Savarese S.: ' Recurrent autoregressive networks for online multi-object tracking', 2018 16Qi C. Ouyang W. Li H. et al.: ' Online multi-object tracking using cnn-based single object tracker with spatial-temporal attention mechanism', 2017 17Danelljan M. Bhat G. Khan F.S. et al.: ' Eco: efficient convolution operators for tracking', 2016 18Zhu J. Yang H. Liu N. et al.: ' Online multi-object tracking with dual matching attention networks', 2019 19Feng W. Hu Z. Wu W. et al.: ' Multi-object tracking with multiple cues and switcher-aware classification', 2019 20Chu P. Ling H.: ' Famnet: joint learning of feature, affinity and multi-dimensional assignment for online multiple object tracking', 2019 21Xu Y. Ban Y. Alameda-Pineda X. et al.: ' Deepmot: a differentiable framework for training multiple object trackers', arXiv, 2019 22Kuhn H.W.: 'The Hungarian method for the assignment problem', Nav. Res. Logist., 2010, 52, (1), pp. 7– 21 23Bernardin K. Stiefelhagen R.: 'Evaluating multiple object tracking performance: the clear mot metrics', Eurasip J. Image Video Process., 2008, 2008, (1), pp. 1– 10 24Guo B. Jin L. Sun D. et al.: 'Establishment of the characteristic evaluation index system of secondary task driving and analyzing its importance', Transp. Res. F, Traffic Psychol. Behav., 2019, 64, pp. 308– 317 25Itti L. Koch C. Niebur E.: 'A model of saliency-based visual attention for rapid scene analysis', IEEE Trans. Pattern Anal. Mach. Intell., 1998, 20, (11), pp. 1254– 1259 26Borji A. Sihite D.N. Itti L.: 'Salient object detection: a benchmark', IEEE Trans. Image Process., 2012, 24, (12), pp. 5706– 5722 27Wang W. Shen J. Ling S.: 'Video salient object detection via fully convolutional networks', IEEE Trans. Image Process., 2017, 27, (1), pp. 38– 49 28Jie H. Li S. Gang S. et al.: 'Squeeze-and-excitation networks', IEEE Trans. Pattern Anal. Mach. Intell., 2017, PP, (99), pp. 1– 1 29Li B. Yan J. Wu W. et al.: ' High performance visual tracking with Siamese region proposal network'. The IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 2018 30Simonyan K. Zisserman A.: ' Very deep convolutional networks for large scale image recognition', arXiv:1409.1556, 2014 31Felzenszwalb P.F. Girshick R.B. McAllester D. et al.: 'Object detection with discriminatively trained part-based models', IEEE Trans. Pattern Anal. Mach. Intell., 2010, 32, (9), pp. 1627– 1645 32Ren S. He K. Girshick R. et al.: 'Faster R-CNN: towards real-time object detection with region proposal networks', IEEE Trans. Pattern Anal. Mach. Intell., 2017, 39, (6), pp. 1137– 1149 33Fan Y. Choi W. Lin Y.: ' Exploit all the layers: fast and accurate CNN object detector with scale dependent pooling and cascaded rejection classifiers'. Computer Vision & Pattern Recognition, Las Vegas, NV, USA., 2016 34Eiselein V. Arp D. Patzold M. et al.: ' Real-time multi-human tracking using a probability hypothesis density filter and multiple detectors'. IEEE Ninth Int. Conf. on Advanced Video & Signal-based Surveillance, Beijing, People's Republic of China, 2012 35Bewley A. Ge Z. Ott L. et al.: ' Simple online and realtime tracking', 2016 36Lee S. Kim E.: 'Multiple object tracking via feature pyramid Siamese networks', IEEE Access, 2019, 7, pp. 8181– 8194 Citing Literature Volume14, Issue8August 2020Pages 842-848 FiguresReferencesRelatedInformation