Boundary‐aware vehicle tracking upon UAV

Electronics LettersVolume 56, Issue 17 p. 873-876 Image and vision processing and display technologyFree Access Boundary-aware vehicle tracking upon UAV Yuqi Han, Yuqi Han orcid.org/0000-0001-7905-0163 School of Information and Electronics, Beijing Institute of Technology, Beijing, 100081 People's Republic of ChinaSearch for more papers by this authorHongshuo Wang, Hongshuo Wang School of Information and Electronics, Beijing Institute of Technology, Beijing, 100081 People's Republic of ChinaSearch for more papers by this authorZengshuo Zhang, Zengshuo Zhang School of Information and Electronics, Beijing Institute of Technology, Beijing, 100081 People's Republic of ChinaSearch for more papers by this authorWenzheng Wang, Corresponding Author Wenzheng Wang wwz@bit.edu.cn School of Information and Electronics, Beijing Institute of Technology, Beijing, 100081 People's Republic of ChinaSearch for more papers by this author Yuqi Han, Yuqi Han orcid.org/0000-0001-7905-0163 School of Information and Electronics, Beijing Institute of Technology, Beijing, 100081 People's Republic of ChinaSearch for more papers by this authorHongshuo Wang, Hongshuo Wang School of Information and Electronics, Beijing Institute of Technology, Beijing, 100081 People's Republic of ChinaSearch for more papers by this authorZengshuo Zhang, Zengshuo Zhang School of Information and Electronics, Beijing Institute of Technology, Beijing, 100081 People's Republic of ChinaSearch for more papers by this authorWenzheng Wang, Corresponding Author Wenzheng Wang wwz@bit.edu.cn School of Information and Electronics, Beijing Institute of Technology, Beijing, 100081 People's Republic of ChinaSearch for more papers by this author First published: 01 August 2020 https://doi.org/10.1049/el.2020.1170Citations: 4AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Discriminative correlation filters (DCFs) have recently achieved competitive performance in visual tracking benchmarks. However, such trackers perform poorly when the target undergoes occlusion, viewpoint variation or other challenging attributes. To tackle these issues, in this Letter, the authors combine the fast DCF trackers with the precise deep learning methods to eliminate the accumulating drift for the vehicle tracking based on unmanned aerial vehicle platform. Specifically, the authors employ the tracking result of the DCF tracker as the input of the boundary regressing network. After judging the existence of the target in the input patch, the proposed network would estimate the boundary of the target vehicle. Furthermore, the output would be updated to the tracking template, aiming at eliminating the accumulation errors and achieving a long-term tracking. The effectiveness of the proposed algorithm is validated through experimental comparison on widely used tracking benchmark data sets. Introduction Visual tracking is one of the fundamental task in computer vision, especially in unmanned aerial vehicle (UAV) applications. Recently, tracking approaches based on discriminative correlation filters (DCF) ([1-4]) have received considerable attentions due to the high computational efficiency with the use of fast Fourier transforms and their outstanding performance in public evaluation data set and benchmark. Despite of their huge success and development, the existing DCF trackers still suffer from model drift and tracking failure due to the limitation of the tracking application itself. Unlike other tasks with clearly-defined target category, the goal of visual tracking is to estimate all kinds of targets' state and trajectory with only reliable information given at the initial frame. Thus, the traditional DCF trackers is easy to overfit and drift gradually when the target appearance varies a lot from the first frame, especially in long-term videos based upon UAV platform. To tackle the above issues, the main motivation of our work is to combine the 'fast' DCF trackers with the 'precise' deep learning methods to eliminate the accumulating drift during tracking and further guide a high-confidence updating for the target template. The framework of the proposed algorithm is shown in Fig. 1. The reminder of the Letter is organised as follows. We first describe the backbone module of the proposed correcting network. Afterwards, we design the loss function to regress the vehicle boundary. The training procedure of the network parameter and the preparation of the training data are reported orderly. Experiments on publicly available UAV data set [5] are conducted to demonstrate the effectiveness of our tracking algorithm. Fig 1Open in figure viewerPowerPoint Overall framework of the proposed algorithm. The blue flowchart denotes the standard DCF tracking procedure, while the black flow is the boundary regressing process for template updating and error correcting Network structure We directly utilise a milestone DCF tracker (DSST) [6] as the base tracker to generate the coarse target position for each frame. We refer readers [6] for more details. In this section, we mainly introduce the structure of the correcting network. As shown in convolutional network in Fig. 2, a Stem block and a dense module from the PeleeNet [7] is utilised to regress the boundary of the input patch. Specifically, we first employ a conv kernel to extract the shallow features for the input patch, obtaining a 32-channel feature layer with the size of . Afterwards, two convolution channels in a Stem Block are used to enhance the model receptive ability for multiscale targets. Furthermore, with the internal dense connections, the dense layer could combine multiple features and decrease the risk of overfitting. Lastly, another conv kernel is incorporated to sequentially downsample the network and obtain the final output. For a better understanding, we detailed describe the function of the core module in our boundary regressing network (i.e. Stem block and Dense module) as follows. Fig 2Open in figure viewerPowerPoint Schematic diagram of the boundary regressing network Stem block module connects target's shallow feature with the following dense layer. With the convolution and pooling operation in each channel, the Stem block could improve the feature representation ability for multiscale targets. Furthermore, such designment could achieve a spatial downsampling for the input image without increasing the computational burden, ensuring a robust appearance modelling ability. As for the dense module, we report its structure in Fig. 3. One can see that it consists of several identical dense blocks. We perform average pooling operation for the output of each dense block as spatial downsampling, which reduces the network parameters but increases the efficiency for feature extracting. As shown in Fig. 3, the feature map in each dense block has the same size and is densely connected with each other, which could ensure the maximum information transmission and avoid transmission loss across different layers. Furthermore, the very shallow feature could be shared to the later layers with the directly connected path in its internal structure as shown in Fig. 3. Fig 3Open in figure viewerPowerPoint Schematic diagram for the internal structure of dense module For a better understanding of our network, we give essential description for the network parameters and the output size for each feature layer in Table 1. One can see that each dense module consists of four dense blocks and each dense block is made up of three dense layers. It should be noted that the convolution operator is smaller than to compress the network parameters and accelerate the overall network calculation. Table 1. Description for network parameters and output for each feature layer Operation Implementation and description Feature map input layer original input patch stem block shallow feature extraction dense block-1 3*DenseLayer + 1*1 conv + 2*2 pooling dense block-2 3*DenseLayer + 1*1 conv + 2*2 pooling dense block-3 3*DenseLayer + 1*1 conv + 2*2 pooling dense block-0 3*DenseLayer + 1*1 conv + 2*2 pooling Conv-4 3*3 conv, stride=2, padding=1 Conv-5 3*3 conv, stride=2, padding=1 output layer Softmax and regression loss probability and boundary Loss function Since the category of the target is constrained to vehicle in this Letter, it is necessary to determine whether a vehicle target is located by the DCF tracker in the input patch to avoid outputting a wrong regression boundary for non-vehicle targets. To this end, we argue that the output of the network should not only contain vehicle boundary information but also provide the classification probability. In this way, the loss function is designed as follows: (1) Here, N is the number of training samples, while p and t denote the output of probability and the estimated target position, respectively. is a binary vector indicating whether the input image contains a vehicle sample. Similarly, represents the real position of the vehicle target in the image. If no vehicle exists in the image, returns with the boundary of the image patch. t has the same format with , which has been normalised on the basis of the width and height of the input patch. It consists of four elements, indicating the upper, bottom, left, right boundary for the target. indicates the Euclidean distance between two vectors, which is defined as below: (2) Furthermore, we employ a entropy function to determine whether the input image contains vehicle targets. Seeing from the equation below, the loss is zero if the network prediction matches the actual existence of the vehicle : (3) Network training As mentioned, the proposed network should be equipped with the ability to predict the existence of the vehicle to prevent the mismatching and template contamination. Specifically, we select the Standford Car data set [8], as shown in Fig. 4, which contains 16,185 car images from multiple viewpoints and multiple vehicles as the positive samples. As for the negative samples, we randomly select the non-vehicle objects from the ImageNet. The ratio between the positive and negative samples are set as 1:1, which indicates that 16,185 vehicle patches and 16,185 non-vehicle patches are utilised into the training process. Fig 4Open in figure viewerPowerPoint Illustration for the training data selected from the Standford Car data set Back-propagation and gradient descent are employed to accomplish the training. We randomly select a mini-batch instead of taking all the training samples when calculating the gradient. Such scheme could not only relieve the computational burden and improve the optimisation efficiency, but also make the convergence state more stable. (4) Here, is the network parameter, s indicates the training samples. is the learning rate for the network. n denotes the randomly picked samples to optimise the gradient descent. Furthermore, inspired by the concept of 'momentum' [9] in physics, we determine the current direction of gradient optimisation by calculating an exponential weighted averaging of the historical gradients, which could avoid repeated oscillations during the canyon-like optimisation procedure. The proposed gradient descent iteration is shown as below. (5) is the weight for the historical gradient, while and denote the gradient direction for the current frame and previous frame, respectively. According to the experimental validation, we find that the employment of 'momentum' could indeed accelerate model convergence compared with the traditional gradient descent technique. In the practical training, we set as 0.9 and the size of the mini-batch samples as 32. The learning rate varies according to the training iterations. The initial learning rate is set as . It decreases at a rate of 10 times every 20,000 iterations. The training would be cut off after 100,000 iterations. Fig. 5 shows the relationship between the convergence and the iteration number. One can see that the network loss converges rapidly in the initial training stage and remains stable around 0.25 after 60,000 iterations. Fig 5Open in figure viewerPowerPoint Convergence for the model training. The training is chocked off after 100,000 iterations We employ 3737 positive samples and 3737 negative samples excluded from the training data to test the network's classification performance. The classification threshold is set as 0.8 to detect the existence of the vehicle. A total of 3586 positive instances and 3655 negative instances are predicted accurately among all the testing samples. To sum up, our classification network achieves 97 classification accuracy and 2.5 false alarm rate, which demonstrates a favuorable performance. Experimental results In this section, we validate our proposed tracker on vehicle videos selected from the UAV-123 [5] data sets. Furthermore, both the deep-learning trackers (MDNet [10], DCFNet [11], ECO [12], HCF [13]) and outstanding DCF-based trackers reported in the top conferences (STAPLE [14], SAMF [15], DSST [6]) are compared with our tracking algorithm to verify its tracking behaviour. For all the compared trackers, we implement all the experiments in C++ on an Intel i5 core 2.90 GHz CPU with 8GB RAM. Our tracking algorithm is built upon DSST tracker. Here, we report the implementation details for some essenstial parameters. Same as the baseline tracker, HOG is selected for target feature representation. We set the cell-size as and the orientation as 9. The learning rate for the tracker is set to 0.01, while the regularisation parameter equals to 0.01. When addressing the scale variation attribute, we empirically set the number of target pyramid layers as 3, with the scale factor equals 1.02. The classification threshold is set as 0.8 to determine the existence of the vehicle to be tracked. We quantitatively evaluate the trackers with success plot and precision plot as suggested in [16], in which the success plot is based on the bounding box overlap, and the area under curve (AUC) of each success plot is utilised to rank the evaluated methods. Meanwhile, the precision plot demonstrates the percentage of frames whose locations are within a threshold distance. Fig. 6 shows the quantitative results using one pass evaluation. In the success plot, the proposed algorithm achieves the AUC of 0.585, which outperforms the baseline DSST tracker by 16%. Meanwhile, for the precision plot, the score of our tracker is 0.785, which outperforms MDNet by 5.9%. The top performance of our tracker could be attributed to the employment of the prior information for the vehicle targets. By taking such advantage, our tracker could correct the accumulation errors and adapt to the variations in target's scale and viewpoint in time. Fig 6Open in figure viewerPowerPoint Overall performance of precision and success rate with the scores for each trackers reporting in the legend Since the target undergoes different challenging attributes during tracking process, we note that it's crucial to investigate the tracking performance on these factors. Considering that the success plot is more informative in demonstrating the strengths and weaknesses of the tracker when comparing with the precision plot which mainly considers the centre location error and ignore the overlap ratio. Therefore, we report the success rate for all the comparing trackers under six common challenging interferences in UAV tracking, namely aspect ratio variation, scale variation, viewpoint change, out-of-view, camera shake and fast motion. It can be seen from Fig. 7, our tracker yields better performance on every attribute compared with the other trackers. This could be owed to the stored vehicle features in the CNN network, which could catch up to the target appearance and reduce the risk of model drift caused by the interferences during tracking. Fig 7Open in figure viewerPowerPoint Success score on six challenging attributes (aspect ratio variation, scale variation, viewpoint change, out-of-view, camera shake and fast motion) of the proposed tracker and other state-of-the-art algorithms Lastly, we illustrate the qualitative comparison between our tracker and other cutting-edge trackers in Fig. 8. One can see that the target vehicles undergo several challenging factors (i.e. viewpoint change, scale variation, rotation, out-of-view, partial occlusion) in the sequences. It can be seen that the other trackers either drift from the target centre or couldn't adapt to the significant target appearance change. While our tracker could locate the target precisely and robustly (target centre and its boundary), demonstrating its effectiveness against the challenging interference in UAV tracking application. Fig 8Open in figure viewerPowerPoint Qualitative performance evaluation of our tracker and other state-of-the-art trackers from UAV data set Conclusion In this Letter, we propose a long-term boundary-aware vehicle tracking algorithm based upon UAV platform. Specifically, we employ the tracking result of the DCF tracker as the input of the boundary regressing network. After judging the existence of the target, the proposed network would refit the boundary of the target vehicle. Finally, the output would be updated to the tracking template, aiming at eliminating the accumulation errors and achieving a long-term tracking. Experimental results reflect the superiority and robustness of our algorithm. Acknowledgments This work was supported by 111 Project of China (grant no. B14010). The authors also thank Chen Wu and Ruofei Liang for their instructions and unconditional backup. References 1Bolme, D.S., Ross Beveridge, J., Draper, B.A. et. al.,: ' Visual object tracking using adaptive correlation filters'. Conf. on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 2010 2Henriques, J.F., Caseiro, R., Martins, P. et. al.,: 'High-speed tracking with kernelized correlation filters', IEEE Trans. Pattern Anal. Mach. Intell., 2015, 37, (3), pp. 583– 596 (https://doi/org/10.1109/TPAMI.2014.2345390) 3Han, Y., Deng, C., Zhao, B. et. al.,: 'State-aware anti-drift object tracking', IEEE Trans. Image Process., 2019, 28, (8), pp. 4075– 4086 (https://doi/org/10.1109/TIP.2019.2905984) 4Han, Y., Deng, C., Zhao, B. et. al.,: 'Spatial-temporal context-aware tracking', IEEE Signal Process. Lett., 2019, 26, (3), pp. 500– 504 (https://doi/org/10.1109/LSP.2019.2895962) 5Mueller, M., Smith, N., Ghanem, B.: ' A benchmark and simulator for UAV tracking'. European Conf. Computer Vision, Amsterdam, Netherlands, 2016, pp. 445– 461 6Danelljan, M., Häger, G., Khan, F. et. al.,: ' Accurate scale estimation for robust visual tracking'. British Machine Vision Conf., 2014 7Wang, R.J., Li, X., Ling, C.X. et. al.,: ' Pelee: a real-time object detection system on mobile devices'. Conf. Neural Information Processing Systems, Montreal, Canada, 2018, pp. 1963– 1972 8Krause, J., Stark, M., Deng, J. et. al.,: ' 3D object representations for fine-grained categorization'. Int. Conf. Computational Vision, Sydney, Australia, 2013, pp. 554– 561 9Sutskever, I., Martens, J., Dahl, G. et. al.,: ' On the importance of initialization and momentum in deep learning'. Int. Conf. Machine Learning, Atlanta, GA, USA, 2013, pp. 1139– 1147 10Nam, H., Han, B.: ' Learning multi-domain convolutional neural networks for visual tracking'. Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 4293– 4302 11Wang, Q., Gao, J., Xing, J. et. al.,: 'Discrimitive correlation filters network for visual tracking', arXiv:1704.04057 12Danelljan, M., Bhat, G., Khan, F.S. et. al.,: ' Efficient convolution operators for tracking'. Conf. Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 6931– 6939 13Ma, C., Huang, J.-B., Yang, X. et. al.,: ' Hierarchical convolutional features for visual tracking'. Int. Conf. on Computer Vision, Santiago, Chile, 2015, pp. 3074– 3082 14Bertinetto, L., Valmadre, J., Golodetz, S. et. al.,: ' Staple: complementary learners for real-time tracking'. Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 1401– 1409 15Li, Y., Zhu, J.: ' A scale adaptive kernel correlation filter tracker with feature integration'. European Conf. Computer Vision, Zurich, Switzerland, 2014 16Wu, Y., Lim, J., Yang, M.: 'Object tracking benchmark', IEEE Trans. Pattern Anal. Mach. Intell., 2015, 37, (9), pp. 1834– 1848 (https://doi/org/10.1109/TPAMI.2014.2388226) Citing Literature Volume56, Issue17August 2020Pages 873-876 FiguresReferencesRelatedInformation