Target tracker with masked discriminative correlation filter

IET Image ProcessingVolume 14, Issue 10 p. 2227-2234 Research ArticleFree Access Target tracker with masked discriminative correlation filter Hang Liu, Hang Liu School of Computer Science, Xiangtan University, Xiangtan, 411105 People's Republic of China ICIP MOE Key Lab, Xiangtan University, Xiangtan, 411105 People's Republic of ChinaSearch for more papers by this authorBodong Li, Corresponding Author Bodong Li bodong.li@gmail.com orcid.org/0000-0001-7427-9057 School of Computer Science, Xiangtan University, Xiangtan, 411105 People's Republic of China ICIP MOE Key Lab, Xiangtan University, Xiangtan, 411105 People's Republic of ChinaSearch for more papers by this author Hang Liu, Hang Liu School of Computer Science, Xiangtan University, Xiangtan, 411105 People's Republic of China ICIP MOE Key Lab, Xiangtan University, Xiangtan, 411105 People's Republic of ChinaSearch for more papers by this authorBodong Li, Corresponding Author Bodong Li bodong.li@gmail.com orcid.org/0000-0001-7427-9057 School of Computer Science, Xiangtan University, Xiangtan, 411105 People's Republic of China ICIP MOE Key Lab, Xiangtan University, Xiangtan, 411105 People's Republic of ChinaSearch for more papers by this author First published: 09 June 2020 https://doi.org/10.1049/iet-ipr.2019.0881Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The discriminative correlation filter (DCF) method is widely used in target tracking due to its real-time performance. However, the computational efficiency of DCF results in boundary effect, which reduces the tracking accuracy in fast motion scene. Besides, background noise is always required to be carefully handled for they will cause trouble in scenes such as background clutter, occlusion, deformation etc. To address the two issues, this study proposes masked discriminative correlation filter, which uses mask to process DCF filter as well as target samples so as to suppress boundary effect and background noise. Experimental results on benchmark datasets show that the proposed tracker performs better than a series of benchmark trackers, and is superior to them in almost various scenes. 1 Introduction Target tracking is useful in military, transportation, media, and many other applications. One of the effective tracking technologies is based on the discriminative correlation filter (DCF) [1–7], which trains a filter template with target samples so that the maximum response value corresponds to the target position. DCF method works in real time, for the involved correlation filter is trained with samples obtained using cyclic sliding window operation. Such operation means periodic expansions of the samples, and therefore the training and detecting of targets can be implemented using fast Fourier transform. However, the periodic assumption leads to periodic repetition in training samples and inaccurate representation of image content and consequently produces harmful boundary effect, especially in fast motion scene [8]. The resulting boundary effect will reduce the discriminative ability of the learned model due to incorrect training samples used. Besides, it will pollute the detection scores away from the regional centre, owing to the periodic repetition of detection samples. Apart from the boundary effect, the DCF method will be degraded by background noise. Such trouble can also be seen in other types of tracking methods (rather than DCF type) and reduces tracking performance in multiple scenes such as background clutter, occlusion, deformation etc. To produce a target tracker that can reduce boundary effect and background noise, and perform well in various scenes, masked DCF (MDCF) will be proposed in this study. Masks will be used on DCF filter to decrease boundary effect and applied on samples to suppress background noise. The operation of masking samples aims to remove noise, but will also cut off signal. To offer a better trade-off between noise removing and signal retaining, two MDCFs, one obtained with masked samples and the other obtained with un-masked samples, will be combined to produce the final tracker. Experimental results show that our tracker works well for various scenes, and outperforms a multitude of state-of-the-art trackers. The remainder of this paper is organised as follows. Related work will be described in Section 2. Standard correlation tracking will be revisited in Section 3. Our method will be described in Section 4, and the associated experimental results will be shown in Section 5. Finally, we conclude the paper in Section 6. 2 Related work Up till now, a lot of DCF-based tracking methods have been published. Bolme et al. [9] proposed to train a minimum output sum of squared error (MOSSE) filter, which uses only grey as feature and therefore depicts target characteristics unsatisfactorily. Danelljan et al. [10, 11] improved MOSSE filter and presented discriminative scale-space tracker (DSST) filter, which applied histogram of gradient (HOG) [12] feature that is far better than a grey level feature. Bertinetto et al. [13] proposed to fuse DSST filter and distractor-aware tracking (DAT) [14] and produced their Staple tracker. Henriques et al. [15] proposed high-speed tracking with kernelised correlation filter (KCF) to improve tracking performance by extending the correlation filter to multi-channel input and kernel-based training. Li and Zhu [16] improved the KCF tracking method by proposing a scale adaptive kernel correlation filter tracker with feature integration (SAMF), which first extended a single feature into multiple features, and then used the scale pool method for adaptive tracking. Though DCF trackers become better and better, they commonly suffer from boundary effects [8]. Such effects, as mentioned before, produce shifted samples plagued by circular boundary, and consequently reduce the discriminative power of the learning model. Several methods have been proposed to deal with the boundary effect. Galoogahi et al. [8] pre-multiplied the samples with a masked matrix containing the target regions, and the method was restricted for single-channel DCF. Danelljan et al. [17] used a spatial regularisation (SR) term to penalise DCF coefficients according to their spatial locations relative to the sample centre. The method, called Spatially Regularised Correlation Filters (SRDCF) in [17], runs well for fast motion scene due to the ability to reduce boundary effect. Lukezic et al. [18] applied the mask matrix generated by colour probability model to limit the boundary problem of the filter, and introduced the concepts of channel and spatial reliability (CSR) to obtain the final tracker, i.e. Discriminative Correlation Filter with Channel and Spatial Reliability (CSR_DCF). Though boundary effect has been considered in the above methods, other troubles such as background noise should also be carefully processed. 3 Correlation tracking For the DCF tracking method such as the one in [10], the key step is to find an optimal correlation filter f by a set of samples. The filter can be obtained following the optimisation below: (1)where is the dth channel of the feature of the ith cyclic shift sample relative to the original sample X of size , is the dth channel of the desired filter, is the desired output represented by Gaussian function tag of size , is the number of the channel of the feature, is the regularisation weight parameter, and represents circular convolution in the space domain. In the field of signal processing, it is well known that the circular convolution in the space domain can be expressed as the Hadamard product in the frequency domain. Therefore, one can express the optimisation in (1) equivalently as follows: (2)where denotes Fourier transform. As in [10], the convolution filter can be analytically obtained as follows: (3)where represents a conjugate operator, and means point-wise multiplication. 4 Our approach The proposed tracking method will be described as follows. First, the way to reduce the boundary effect will be shown. Then, we depict how to suppress background noise. Finally, the final tracker will be detailed. In the following discussion, a DCF will be denoted by its Fourier transformed version for simplicity. 4.1 Reduction of boundary effect The correlation filter is trained with circular sliding window processed samples. However, such samples obtained using periodic expansion produce boundary effects. As a result, the DCF filter yielded with the samples will have an unreasonable high value of energy not near the regional centre. Hence, one can reduce the boundary effect by discarding the energy of the DCF filter away from the regional centre. This can be implemented by masking the DCF filter with a rectangle, and we obtain the MDCF as shown in Fig. 1. Fig. 1Open in figure viewerPowerPoint MDCF for reducing boundary effect As to the DCF denoted by in Fig. 1, each feature channel will be trained separately and the filter can be obtained following the minimisation below (4)where . Notice that different from the DCF revisited in Section 3, each channel in is trained separately rather than simultaneously, according to trial. Similar to the standard DCF, can be obtained as (5)Since the boundary effect occurs in the boundary region of samples, a mask not including the boundary will be used to process and yield (6)where the name of the obtained filter implies that it is once masked, MASK is chosen to be the labelled rectangle of the target position in the initial frame, and has the same size as the searching area. Lastly, will be updated with a learning rate as below (7)where superscripts t and correspond to the current frame and previous frame, respectively. 4.2 Suppression of background noise Background noise is produced by the objects which are not the target. The noise can be suppressed by removing the objects away from the target, and one can implement the idea by masking the original samples using a rectangle. The corresponding MDCF is depicted in Fig. 2, where the reduction of boundary effect is also considered. Fig. 2Open in figure viewerPowerPoint MDCF for suppressing background noise Regarding the DCF denoted as , the training samples will be processed by a mask to decrease noise. Here, the mask is also chosen to be the labelled target position rectangle in the initial frame. For more details see Fig. 3. Fig. 3Open in figure viewerPowerPoint Training samples used in in Fig. 2 (a) Mask, (b) Samples processed by the mask In common with , can be obtained as (8)where is the dth channel of feature related to the masked samples. Similarly as in Section 4.1, will be masked to relieve boundary effect and we obtain (9)where the obtained filter is so named for it is twice masked. Finally, we update with a learning rate as the following: (10) 4.3 Tracking method based on MDCF The MDCF in Fig. 2, i.e. twice-MDCF, is capable of noise removing as well as boundary effect reducing. Nevertheless, some signals useful to track the target will also be discarded. Therefore, one has to make a better compromise between noise removing and signal retaining. This can be realised by combining MDCF obtained with masked samples and the one acquired with un-masked samples. The two MDCFs can be chosen as twice-MDCF and once-MDCF, and the final tracker is depicted in Fig. 4. Fig. 4Open in figure viewerPowerPoint Proposed tracking method based on MDCF The tracker is applied in a sliding-window-like manner by evaluating the scores on all cyclic shifts of searching area. Let z denote the feature map acquired in the next frame, and the corresponding searching area has a centre identical to the target position in the current frame. Hence, the response map y can be obtained by (11), where d is the number of filter channels, represents the weight parameter, denotes inverse Fourier transform. Here, the position of the maximum response value corresponds to the new target centre. (11) 5 Experiments and results 5.1 Experimental setup Our MDCF tracker will be compared with nine state-of-the-art trackers mostly published recently, including fast DSST Fast Discriminative Scale Space Tracking (fDSST) [11], Staple [13], DAT [14], KCF [15], SAMF [16], SRDCF [17], CSR_DCF [18], Compressive Tracking (CT) [19], and siamfc3s [20]. The nine trackers cover DCF type as well as other types (i.e. DAT, CT, and siamfc3s). Experiment results about the nine trackers are cited from published papers (i.e. SRDCF and siamfc3s), or produced by the related authors' code (i.e. other trackers). Standard HOG [21] and Colornames [22] features will be used in filter training and testing. 5.1.1 Dataset The benchmark datasets we used include OTB2013 [23] and OTB2015 [24]. The former consists of 50 annotated image sequences with 11 different properties such as fast motion, object deformation, occlusion, and blur motion. The latter (i.e. OTB-2015) extends the former from 50 to 100 sequences. 5.1.2 Evaluation metrics Two metrics, precision and success rate, will be used to evaluate trackers. Other involved metrics are obtained by extending the success rate. The precision is represented as a ratio. The ratio's denominator is the total number of frames. The ratio's numerator is the number of frames with tracking error smaller than a given threshold a (called location error threshold). The tracking error is defined as Euler distance between the tracked target and the manually labelled target. Therefore, the precision metric can be expressed as (see (12)), (12)where #frames means the number of frames, N is the total number of frames, is the centre of the tracked target in the tth frame, and is the centre corresponds to the manually labelled target. Besides, the location error threshold is generally set to pixels [25]. The success rate is a ratio of two numbers. One is the number of frames whose overlap score is greater than a given threshold b (named overlap threshold), the other is the total number of frames (i.e. N). In another word, the success rate can be represented as (13)where overlap score is defined as (14)wherein denotes the tracked target's rectangle region in the tth frame and is related to the manually labelled target. Success rate is commonly extended to be a more useful metric as the area under the curve (AUC) of , i.e. (15)which is generally considered to be the most important metric [23]. The other metric, called mean overlap precision (mean OP), is the success rate with threshold , i.e. (16) 5.1.3 Experimental parameters The experimental parameters include the learning rate in (7) and (10), regularisation parameter in (5) and (8), as well as weight parameter in (11). There are dozens of optional values for each parameter, so the number of optional values for all the three parameters is too large. It is difficult to try all of the parameters' values during the experiment, and we determine them by greedy strategy. The value of is first obtained, then , and lastly . Following the way, we have , , and . 5.2 Results 5.2.1 Comparison among trackers With regard to OTB-2013, the success rate, precision, and mean OP of trackers are shown in Figs. 5a and b, and Table 1, where the legend values of Fig. 5a depict AUCs and those of Fig. 5b shows the precisions with the location error threshold . As can be seen, though our tracker ranked second in terms of precision, it performs better than all of the benchmark trackers from the aspect of AUC and mean OP. Fig. 5Open in figure viewerPowerPoint Performance metrics of trackers on OTB-2013 (a) Success rate, (b) Precision Table 1. Mean OP (in %) of trackers on OTB2013 MDCF SAMF fDSST CT DAT CSR_DCF SRDCF siamfc3s Staple KCF mean OP 80.1 72.8 74.7 28.8 35.1 75 78.1 77.8 73.7 62.3 Regarding OTB-2015, the performance metrics of trackers are depicted in Figs. 6a and b, and Table 2. As can be seen, our tracker outperforms all of the benchmark trackers in terms of precision, AUC, and mean OP. Besides, our tracker improves the second-ranked one by 4.3, 3.7, and 4.0%, respectively, in precision, AUC, and mean OP. Fig. 6Open in figure viewerPowerPoint Performance metrics of trackers on OTB-2015 (a) Success rate, (b) Precision Table 2. Mean OP (in %) of trackers on OTB2015 MDCF SAMF fDSST CT DAT CSR_DCF SRDCF siamfc3s Staple KCF mean OP 75.9 66.9 67.2 25.9 35.8 70.2 72.9 73 69.9 55.2 5.2.2 Attribute-based comparison As mentioned earlier, OTB-2015 includes more image sequences than OTB-2013, and the subsequent attributed-based comparison of trackers will be performed on the former. All of the 100 image sequences in OTB-2015 dataset are annotated with 11 different properties, i.e. fast motion, background clutter, motion blur, deformation, illumination variation, in-plane rotation, low resolution, occlusion, out-of-plane rotation, out of view and scale variation, and can be considered as 11 scenes. The success rate curves about the 11 scenes are shown in Fig. 7. As can be seen, our tracker achieves the best results in fast motion, object deformation, occlusion, motion blur, out-of-plane rotation, background clutter, out of view and scale variation scenes, and ranked third in low resolution and illumination variation scenes. Fig. 7Open in figure viewerPowerPoint Success rate of trackers about 11 video attributes on OTB-2015 The success of our tracker comes from the reduction of boundary effect and background noise. Boundary effect is serious in fast motion scene as mentioned before, and also severe in scenes such as motion blur. Apart from background clutter, background noise is also common in scenes such as scale variation, out of view, object deformation, and out-of-plane rotation. The best four trackers among the others are SRDCF, siamfc3s, CSR_DCF, and Staple, according to Fig. 6a. From Fig. 7, the failed scene of SRDCF is the out of view, the one of siamfc3s is the background clutter, those of CSR_DCF include the illumination variation and the out-of-plane rotation, and those related with Staple are the fast motion, the motion blur, and the low resolution. Here, a method is marked as a failure when it is not in the top five. One, therefore, can see that the tracker Staple is not good at suppressing boundary effect, and the other three (i.e. SRDCF, siamfc3s, and CSR_DCF) runs not well for removing background noise. Nine sequences in OTB-2015, processed by trackers are shown in Fig. 8, where the targets located by our tracker, as well as four typical trackers (i.e. SRDCF, siamfc3s, CSR_DCF, and Staple) are, respectively, marked by red, blue, light blue, yellow, and green rectangles. The nine sequences include Panda, Skating2, Trans, CarScale, BlurCar1, Shaking, FleetFace, Jogging, and Bolt2, which correspond to challenging tracking scenes, such as fast motion, background clutter, target deformation, and occlusion. Fig. 8Open in figure viewerPowerPoint Some frames (i.e. Panda, Skating2, Trans, CarScale, BlurCar1, Shaking, FleetFace, Jogging, Bolt2) produced by trackers The sequences about fast motion scenes are shown in rows 4, 5, 7 of Fig. 8. As shown in the rightmost frame, CSR_DCF and SRDCF almost fail in row 4, Staple cannot track the target in row 5, and siamfc3s in row 7. In contrast, the corresponding targets are successfully located by our tracker, thanks to better ability than the other four in reducing boundary effect. The sequences regarding the deformation scene are displayed in rows 1, 2, and 3 of Fig. 8. As can be seen from the rightmost frame, Staple lose the target in row 1, CSR_DCF in row 2, and SRDCF in row 3, while our tracker performs well in the three rows and more accurately than siamfc3s in row 2. The sequences concerned background clutter scenes are shown in rows 6 and 9 of Fig. 8. As can be seen, CSR_DCF, SRDCF, and Staple cannot track the target in several frames in row 6, siamfc3s loses the target in several frames in row 9. Our tracker, by contrast, can track the targets correctly. The sequence related to the occlusion scene is shown in row 8 of Fig. 8. As displayed, Staple fails to track the target in most of the frames, siamfc3s cannot accurately track the target in the rightmost frame, whereas our tracker can keep track the target on this sequence. The tracking illustrations in deformation, background clutter, and occlusion scenes show that our tracker works better than the others, mainly due to its noise removing capacity in these scenes. 6 Conclusion In this study, the MDCF tracker is proposed to deal with two troubles, i.e. boundary effect and background noise. The former is significant in the fast motion scene, and the latter is remarkable in many other scenes. The tracker reduces the boundary effect by masking the DCF and suppresses background noise by masking the target samples. Experimental data show that our tracker outperforms benchmark ones, and displayed the best result in almost various scenes. It should be noticed that our tracker only ranked third or fourth in illumination variation, low resolution, and in-plane rotation scenes. The result is still good, but unsatisfactory. We will try to improve it in the future. 7 Acknowledgments This work was supported by the National Natural Science Foundation of China under grant no. 61771415. The authors would like to thank the associate editor and peer reviewers for their constructive suggestions which significantly improved the quality of this paper. 8 References 1Hao Z., Liu G., Zhang H.: 'Correlation filter-based visual tracking via adaptive weighted CNN features fusion', IET Image Process., 2018, 12, (8), pp. 1423– 1431 2Henriques J.F., Caseiro R., Martins P. et al.: ' Exploiting the circulant structure of tracking-by-detection with kernels'. Proc. European Conf. on Computer Vision, Florence, Italy, November 2012, pp. 702– 715 3Jiang M., Shen J., Kong J. et al.: 'Regularisation learning of correlation filters for robust visual tracking', IET Image Process., 2018, 12, (9), pp. 1586– 1594 4Galoogahi K.H., Sim T., Lucey S. et al.: ' Multi-channel correlation filters'. Proc. IEEE Int. Conf. on Computer Vision, Sydney, Australia, 2014 5Ma C., Huang B.J., Yang X. et al.: ' Hierarchical convolutional features for visual tracking'. Proc. IEEE Int. Conf. on Computer Vision, Santiago, Chile, 2016 6Valmadre J., Bertinetto L., Henriques J. et al.: ' End-to-end representation learning for correlation filter based tracking'. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Hawaii, USA, July 2017, pp. 5000– 5008 7Ma C., Yang X., Zhang C. et al.: ' Long-term correlation tracking'. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Boston, MA, USA, June 2015, pp. 5388– 5396 8Galoogahi K.H., Sim T., Lucey S. et al.: ' Correlation filters with limited boundaries'. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015, pp. 4630– 4638 9Bolme S.D., Beveridge R.J., Draper A.B.: ' Visual object tracking using adaptive correlation filters'. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 2010, pp. 2544– 2550 10Danelljan M., Häger G., Khan F. et al.: ' Accurate scale estimation for robust visual tracking'. Proc. British Machine Vision Conf., Nottingham, UK, September 2014 11Danelljan M., Hager G., Khan S.F. et al.: 'Discriminative scale space tracking', IEEE Trans. Pattern Anal. Mach. Intell., 2016, 39, (8), pp. 1561– 1575 12Dalal N., Triggs B.: ' Histograms of oriented gradients for human detection'. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, San Diego, CA, USA, 2005, pp. 886– 893 13Bertinetto L., Valmadre J., Golodetz S. et al.: ' Staple: complementary learners for real-time tracking'. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 1401– 1409 14Possegger H., Mauthner T., Bischof H. et al.: ' In defense of color-based model-free tracking'. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015, pp. 2113– 2120 15Henriques J. F., Caseiro R., Martins P. et al.: 'High-speed tracking with kernelized correlation filters', IEEE Trans. Pattern Anal. Mach. Intell., 2015, 37, (3), pp. 583– 596 16Li Y., Zhu J.: ' A scale adaptive kernel correlation filter tracker with feature integration'. Proc. European Conf. on Computer Vision, Zurich, Switzerland, 2014 17Danelljan M., Häger G., Khan S.F. et al.: ' Learning spatially regularized correlation filters for visual tracking'. Proc. IEEE Int. Conf. on Computer Vision, Santiago, Chile, 2016, pp. 4310– 4318 18Lukezic A., Vojir T., Zajc C.L. et al.: 'Discriminative correlation filter with channel and spatial reliability'. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 2017, pp. 4847– 4856 19Zhang K., Zhang L., Yang H.M. et al.: ' Real-time compressive tracking'. Proc. European Conf. on Computer Vision, Florence, Italy, 2012 20Bertinetto L., Valmadre J., Henriques F.J. et al.: ' Fully-convolutional Siamese networks for object tracking'. Proc. European Conf. on Computer Vision, Amdsterdam, The Netherlands, 2016 21Felzenszwalb P.F., Girshick R.B., McAllester D. et al.: 'Object detection with discriminatively trained part-based models', IEEE Trans. Softw. Eng., 2010, 32, (9), pp. 1627– 1645 22Van de J.W., Schmid C., Verbeek J. et al.: 'Learning color names for real-world applications', IEEE Trans. Image Process., 2009, 18, (7), pp. 1512– 1523 23Wu Y., Lim J., Yang H.M. et al.: ' Online object tracking: a benchmark'. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Portland, OR, USA, 2013, pp. 2411– 2418 24Wu Y., Lim J., Yang H.M. et al.: 'Object tracking benchmark', IEEE Trans. Pattern Anal. Mach. Intell., 2015, 37, (9), pp. 1834– 1848 25Babenko B., Yang M.H., Belongie S. et al.: 'Robust object tracking with online multiple instance learning', IEEE Trans. Pattern Anal. Mach. Intell., 2011, 33, (8), pp. 1619– 1632 Citing Literature Volume14, Issue10August 2020Pages 2227-2234 FiguresReferencesRelatedInformation