Real‐time long‐term tracking with reliability assessment and object recovery

IET Image ProcessingVolume 15, Issue 4 p. 918-935 ORIGINAL RESEARCH PAPEROpen Access Real-time long-term tracking with reliability assessment and object recovery Jun Liu, Jun Liu School of Aeronautics and Astronautics, Shanghai Jiao Tong University, China School of Automation and Information Engineering, Sichuan University of Science and Engineering, ChinaSearch for more papers by this authorPing Ye, Ping Ye School of Aeronautics and Astronautics, Shanghai Jiao Tong University, ChinaSearch for more papers by this authorXingchen Zhang, Corresponding Author Xingchen Zhang xingchenzhang1990@gmail.com orcid.org/0000-0002-8012-0103 School of Aeronautics and Astronautics, Shanghai Jiao Tong University, China Correspondence Xingchen Zhang, School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai 200240, China. Email: xingchenzhang1990@gmail.comSearch for more papers by this authorGang Xiao, Gang Xiao School of Aeronautics and Astronautics, Shanghai Jiao Tong University, ChinaSearch for more papers by this author Jun Liu, Jun Liu School of Aeronautics and Astronautics, Shanghai Jiao Tong University, China School of Automation and Information Engineering, Sichuan University of Science and Engineering, ChinaSearch for more papers by this authorPing Ye, Ping Ye School of Aeronautics and Astronautics, Shanghai Jiao Tong University, ChinaSearch for more papers by this authorXingchen Zhang, Corresponding Author Xingchen Zhang xingchenzhang1990@gmail.com orcid.org/0000-0002-8012-0103 School of Aeronautics and Astronautics, Shanghai Jiao Tong University, China Correspondence Xingchen Zhang, School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai 200240, China. Email: xingchenzhang1990@gmail.comSearch for more papers by this authorGang Xiao, Gang Xiao School of Aeronautics and Astronautics, Shanghai Jiao Tong University, ChinaSearch for more papers by this author First published: 05 January 2021 https://doi.org/10.1049/ipr2.12072AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract In recent years, many visual tracking algorithms based on discriminative correlation filters have been proposed and proved to be successful in short-term tracking. However, most algorithms do not handle long-term tracking well due to factors such as occlusion and deformation. The aim of this paper is to propose a long-term tracking method with reliability assessment and object recovery. First, the relationship between the overlap rate and the response value is extensively studied, and then from the perspective of the time axis, the tracking process is evaluated by the fluctuation trend of the continuous response value. In the object recovery mechanism, we propose to alternately use local search and global search to improve the efficiency of detection. To this end, a sliding window is designed for cyclic shifting in the local region to achieve dense sampling within the region of interest, and EdgeBox is used in the global search to achieve target detection. Further, flexible switching between local search and global search is achieved by the difference in displacement of the object. Extensive experimental results on several benchmark datasets demonstrate that the proposed long-term tracker can achieve state-of-the-art accuracy with real-time speed of about 27 frames per second. 1 INTRODUCTION Visual object tracking has been applied to many fields, such as human-computer interaction, security and surveillance, and auto-control systems. Existing object tracking algorithms have made great achievements, but due to factors such as illumination variation, occlusion, deformation, scale variation and background clutter, there are still many problems to be further investigated. According to the appearance model, visual object tracking approaches can be broadly classified into two categories, namely generative and discriminative methods. The generative methods [1-9] transform the tracking problem as a nearest-neighbor searching task for the target model. The insufficiency of the generative method is that the background information of the target is not considered, so the image information cannot be applied well. The discriminative methods [10-19] formulate the target tracking as a binary classification problem. The extracted target and background information are used to train the classifier and then separate the target from the background. Since the background information is additionally utilized, the discriminative methods will be more advantageous than the generative methods in practical applications. Recently, correlation filter based trackers have sparked a lot of interests due to their real-time speed and state-of-the-art accuracy [11, 17, 18, 20, 21]. In general, these trackers learn a correlation filter online to localize the object in consecutive frames. However, the correlation filter based tracker still faces two serious problems. One major issue is boundary effects due to the circulant assumption. The other is that the target search region only contains a small local neighbourhood to limit drift. Therefore, the trackers easily drift in cases of fast motion, occlusion or background clutter. It is important to note that occlusion can have a serious adverse effect on the performance of trackers. When occlusion occurs, the similarity of desired target region degrades and cluttered background distracts the template to match with other mistaken regions. To make it worse, the template is updated with the false information, which further brings unsatisfactory templates to the subsequent frames. Therefore, how to overcome the performance degradation caused by occlusion is a key concern when designing a long-term tracker. The aim of this paper is to develop a long-term tracker based on correlation filters to alleviate the problem of tracking failure caused by factors such as deformation and occlusion. The main difficulties in the overall framework of the tracker are reflected in the process evaluation and object recovery. To this end, we designed a criterion to achieve an accurate assessment of the reliability of the tracking process, and also established a complete object recovery mechanism. The main contributions of this paper are as follows: • Different from the methods using peak to sidelobe ratio (PSR) [20], median flow (MF) [59] and average peak-to-correlation energy (APCE) [49], we regard the reliability of the tracking results as correlated back and forth on the time axis, and design a new tracking result evaluation criterion based on continuous response values. • We propose a new object recovery mechanism in which the sliding window performs a local search and the Edgebox [61] performs a global search. The recovery mechanism can adaptively select an appropriate search mode to find the missing object based on the average displacement of the object. • The proposed algorithm is validated on several widely used large-scale benchmark datasets [22, 24-26] and the experimental results show that our long-term tracker is able to achieve state-of-the-art accuracy and real-time speed. The remaining of this paper is organized as follows. Previous related works are reviewed briefly in Section 2. Section 3 presents our approach for robust object tracking which includes short-term trackers based on correlation filtering, reliability assessment of tracking process, object recovery mechanism and model updates strategy. Section 4 summarizes the overall workflow of the algorithm and illustrates the performance of the proposed tracker. Finally, the conclusions and future research directions are discussed in Section 5. 2 RELATED WORK 2.1 Tracking-by-detection methods Detection-based tracking framework has attracted widespread attention due to the fact that classifiers are suitable for online learning, such as support vector machines (SVM) [27], Random Forest classifiers [28] or boosting variants [29, 30], and can be used to achieve real-time tracking. For each frame, a set of positive and negative training samples are collected for incrementally learning a discriminative classifier to separate a target from its background. However, extracting samples to learn online classifiers can cause sample ambiguity problems, while slight inaccuracies in labelling samples can affect the classifier and gradually lead to tracker drift. In order to alleviate the problem of model updating caused by sample ambiguity, considerable efforts have been made in previous studies, such as multiple instance learning (MIL) [29], Struck [31], semi-supervised learning [30, 32], and P-N learning [33]. In addition, a robust appearance model containing both holistic templates and local representations was proposed in [34] to handle drastic appearance changes. Similarly, a learning framework in [35] with 1-class SVM and structured output SVM was designed for collaborating a descriptive component and a discriminative component for tracking. Instead of learning only one single classifier, multiple classifiers with different learning rates were constructed in [36] to address the model drift problem. By alleviating the problem of sample ambiguity, above methods have achieved considerable performance on the recent benchmark dataset [22]. Wang et al. [23] proposed a three-step procedure to convert a general object detector into a tracker. They considered the tracking problem as a special type of object detection problem, which called instance detection. With proper initialization, a detector can be quickly converted into a tracker by learning the new instance from a single image. 2.2 Correlation filter-based methods High computational efficiency and state-of-the-art accuracy are two significant advantages of the correlation filter based tracker. By transferring the model into the Fourier domain, the matrix algebra can be solved by element-wise operation. In [20], Bolme et al. proposed the minimum output sum of squared error (MOSSE) method and learned a minimum output sum of squared error filter on gray-scale images, which used intensity features to represent the object. Heriques et al. [17] proposed the circulant structure of tracking by detection with kernel (CSK) algorithm which generated correlation filters by exploiting the circulant structure of training samples and transferred filters into the Fourier domain. In their work, the CSK achieved a highest tracking speed with considerable performance on benchmark dataset. As an extension to CSK, the histogram of oriented gradients (HOG) features, Gaussian kernels, and ridge regression were used in the kernelized correlation filter (KCF) [18]. The discriminative scale space correlation filter tracker (DSST) [13] learned discriminative correlation filters with a scale pyramid representation to handle the scale change of target objects. They learned two kinds of filters, one for translation estimation and the other for scale estimation. Besides, the scale adaptive with multiple features tracker (SAMF) [14] estimated the scale variation by training the KCF to search the target around the latest estimate position on various sizes of the image patch. Huang et al. [37] proposed the aberrance repressed correlation filter. By introducing a regularization term to restrict the response map, their method was capable of suppressing aberrances that was caused by both background noise information and appearance changes of the tracked objects. Hand-crafted features have been widely used in correlated filter based tracking, and deep features have recently begun to be combined with correlated filtering to exploit their enormous potential. Following the end-to-end philosophy, the recent work [38] integrated DCF into the Siamese framework [39], and [40] employed DCF as a one-layer convolutional neural network (CNN) for end-to-end training. Some other DCF methods [41-43] focused on integrating convolutional features from a fixed pre-trained deep network. Ma et al. [41] proposed a hierarchical ensemble method of independent DCF trackers to combine multiple convolutional layers. Danelljan et al. [42] proposed the continuous convolution operator tracker (C-COT) to efficiently integrate multi-resolution shallow and deep feature maps. The efficient convolution operators (ECO) tracker [43] further optimized the speed and accuracy of the C-COT tracker at three levels: model size, training set size, and model update. Due to the high complexity of CNN features, trackers with deep-features are not superior in speed to traditional hand-crafted feature trackers. Sun et al. [44] proposed the region-of-interest (ROI) pooled correlation filters for visual tracking. They showed that the ROI-based pooling can be equivalently achieved by enforcing additional constraints on the learned filter weights, which made the ROI-based pooling feasible on the virtual circular samples. 2.3 Long-term tracking methods Correlation filters with short-term memory have significant limitations in dealing with object occlusion and disappearance. Therefore, many methods aim to make correlation filters with long-term memory to achieve more advanced tracking. The combination of a short-term tracker and detector is a commonly used long-term tracker architecture which was first used in tracking-learning-detection (TLD) [33]. The seminal work of [33] proposes a memory-less flock of flows as a short-term tracker and a template-based detector. Another paradigm was pioneered in [45], where the authors casted localization as local keypoint descriptors matching with a weak geometrical model. Ma et al. [46, 47] proposed using KCF as a short-term tracker and random fern classifier as a detector to construct a long-term tracker. Similarly, Hong et al. [48] combined a KCF tracker with a SIFT-based detector which was also used to detect occlusions. Dai et al. [49] established a simplified long-term tracker by employing assistant re-detection, which combined the short-term components with the SVM classifier in constructing a long-term component. In [4], the target model was decomposed into a grid of cells and an occlusion classifier was learned for each cell. In [50], a Bayes filter was developed for target loss detection and re-detection for multi-target tracking. Alan et al. [51] utilized the novel DCF constrained filter learning method to design a detector that was able to re-detect the target in the whole image efficiently. Moreover, they proposed to use the correlation response value and PSR to evaluate tracking failure. Sun et al. [52] treated the filter as the element-wise product of a base filter and a reliability term to mitigate model degradation. Dai et al. [53] proposed an adaptive spatial constraint correlation filtering algorithm to optimize both filter weights and spatial constraint matrices. In addition, Goutam et al. [54] leveraged the complementary properties of deep and shallow features to improve both robustness and accuracy. Their work has played a significant role in unlocking the potential of deep features for tracking. Recently, some methods have tried to combine deep learning-based proposal networks with correlation filter tracking. For example, Yang et al. [55] proposed a novel long-term correlation filtering tracking method that applied a DCF-based tracking model with a novel target-aware detector in a collaborative way. In their method, they evaluated the reliability of the tracking results based on the enhanced PSR, and used region proposal network (RPN) based detectors to perform object detection under tracking failure. Yan et al. [56] proposed a novel robust and real-time long-term tracking framework based on the proposed skimming and perusal modules. The perusal module aims to precisely locate the tracked object in a local search region using the offline-trained regression and verification networks. The skimming module focuses on efficiently selecting the most possible regions from densely sampled sliding windows. 3 OUR APPROACH 3.1 The basic correlation filter tracking component Let x be the base sample of size M × N , and the training sample x i can be obtained after cyclic shift. The decomposition of circulant matrix can be employed to simplify the solution of linear regression. The objective function of linear ridge regression can be formulated as follows: min w ∑ i n ( f ( x i ) − y i ) 2 + λ | | w | | 2 (1)where f ( x ) = w T x is used to minimize the squared error over samples x i and their regression targets y i . In the case of no-linear regression, kernel trick, f ( z ) = w T z = ∑ i = 1 n α i κ ( z , x i ) , is applied to allow more powerful classifier. For the most commonly used kernel functions, the circulant matrix trick can also be used [18]. The dual space coefficients α can be learnt as below: α ̂ ∗ = y ̂ k ̂ xx + λ (2)where k xx is defined as kernel correlation in [18]. In this paper, we adopt the Gaussian kernel which can be applied with the circulant matrix trick as below: k xx ′ = exp − 1 δ 2 | | x | | 2 + | | x ′ | | 2 − 2 F − 1 ( x ̂ ⊙ x ̂ ′ ∗ ) (3)where F − 1 denotes the inverse FFT transform. The patch z at the same location in the next frame is treated as the base sample to compute the response in Fourier domain, f ̂ ( z ) = ( k ̂ x ∼ z ) ∗ ⊙ α ̂ (4)where x ∼ denotes the data to be learnt in the model. When we transform f ̂ ( z ) back into the spatial domain, the translation with respect to the maximum response is considered as the movement of the tracked target. For the scale variation of the objects, we apply the method of the scale pool [14]. The template size is fixed as s T = ( s x , s y ) , and the scaling pool is defined as S = { t 1 , t 2 , … , t k } . Suppose that the target window size is s t in the original image space. The samples can be resized into the fixed template size s T by using bilinear-interpolation, and the final response is calculated by arg max F − 1 f ̂ ( z t i ) (5)where z t i is the sample patch with the size of t i s t , which is resized to s T . To adapt to changes in the appearance of objects, we linearly combine the new filter with the old one as below: T ¯ = θ T n e w + ( 1 − θ ) T ¯ (6)where T = [ α T , x ∼ T ] T is the template to be updated. It is worth noting that T is very useful for object recovery because it contains the most recent appearance information of the object. 3.2 Reliability assessment of tracking process Accurate assessment of the tracking process is an effective guarantee for the intervention of object recovery mechanism. Since the correlation filtering tracking locates the target according to the response map, the intensity of the response can be used to characterize the accuracy of target positioning. Figure 1 shows the fluctuation trend of the response value during object tracking. To explore a good tracking process evaluation method, we use the groundtruth given by the dataset to investigate the distribution of response values at different overlap rates. The overlapping is defined as O ( B 1 , B 0 ) = | B 1 ∩ B 0 | | B 1 ∪ B 0 | , where B 1 and B 0 indicates the predicted bounding box and groundtruth, respectively. Since the tracker predicts the new position based on the response map, we can obtain not only the predicted bounding box B 1 but also the response value corresponding to B 1 . For this reason, we can get the distribution of response values at different overlap rates. In the one-pass evaluation [22, 24], for the tracking result of each frame, when O ( B 1 , B 0 ) ⩾ 0.5 , it is considered that the tracker works well. According to the visual sense, when O ( B 1 , B 0 ) = 0.5 , the bounding box still falls on the tracked object. However, as the overlap rate decreases, the bounding box will gradually deviates from the object. When the overlap rate is about 0.4, the tracker almost drifts, and when the overlap rate drops to about 0.2, the tracker completely loses the object. Therefore, we can investigate the relationship between the response values and the reliability of the tracking results by examining the distribution of response values at different overlap rates. FIGURE 1Open in figure viewerPowerPoint The distribution of response values in two presentation cases Considering that correlation filtering based tracking is based on the result of the previous frame to predict the state of the object in the next frame, the reliability of the tracking process can be regarded as being correlated back and forth on the time axis. Therefore, we observe the fluctuations of the response values ϕ = { R K − 4 , R K − 3 , R K − 2 , R K − 1 , R K } of five consecutive frames to evaluate the reliability of the tracking results. To this end, we design two conditions as follows ∀ R ′ ∈ { R K − 4 , R K − 3 , R K − 2 , R K − 1 , R K } , R ′ < d · τ (7) sum ( ϕ < ξ · d · τ ) ⩾ 2 , ξ < 1 (8)where R K is the response value corresponding to the K t h frame and τ is the response value of the initial frame. We regard the second frame as the initial frame because the response value of the first frame is always 1. As for the second frame, since it reflects the state of the target in the original environment, the response value of the second frame has a high reference value. The operator sum is used to calculate the number of elements in the set ϕ whose response value is less than ξ · d · τ . d and ξ are constraint coefficients less than 1. Equation (7) is mainly used to find the time period when the response value drops significantly, and then Equation (8) is used to further determine whether there are two or more drastically decreased response values. When Equations (7) and (8) are reached, the object recovery mechanism will intervene immediately. 3.3 Object recovery mechanism 3.3.1 Template pool During the tracking process, the appearance of the object changes over time. An important issue in long-term tracking is how to make the tracker have the ability to recover the object successfully in the case of tracking failure caused by significant changes in the object appearance. However, using the target sample of the initial frame as a fixed template is not well suited for long-term tracking because the fixed template cannot be adapted to the appearance changes of the target during the tracking process [39]. In [60], the short-term classifier-pool uses the last M samples as templates and partially occluded objects are also used as samples, so the classifier-pool is to some extent contaminated by samples containing error information. Different from the methods in [39] and [60], the availability of each template will be examined by its corresponding response value when building a template pool in this work. Assuming that the object recovery mechanism intervenes at the K t h frame, the template pool Θ is constructed as a collection of the last 15 frame templates. Θ = { T K − 14 , T K − 13 , … , T K − 2 , T K − 1 , T K } (9)where T K is the template of the target at the K t h frame. According to Equations (4) and (5), the template T i with the largest response value in Θ can be determined. The template T i will be used to recover objects from tracking failures. 3.3.2 Local search strategy The boundary effect is a significant flaw in correlation filter tracking, and the search area is usually set to 1.5 times the size of the object. When the object part moves out of the search area, the positioning of the target will be inaccurate. In particular, fast-moving objects tend to move out of the search area easily, which eventually leads to tracking failure. Therefore, a search area of 1.5 times the object size is usually insufficient. For this reason, SRDCF [21] limits the boundary effect and expands the search area to 4 times the size of the object, but greatly sacrifices the running speed. Generally, when the object appears again after losing, it is usually located in the surrounding area of the losing position. Therefore, the local search method can detect the object in the surrounding area of interest directly with high efficiency. In order to improve the tracking speed and accuracy, we propose a local search strategy which is activated after Equations (7) and (8) are reached. When tracking fails, the tracker will record the position of the object at the current frame, and then the detector searches the target in the local region of the position. To this end, we designed a sliding window to perform a cyclic shift in the local region for dense sampling while extracting image features to generate a sample corresponding to each window. This is equivalent to dividing the expanded search area into many blocks and performing correlation measurement on each block separately, which can effectively reduce the influence of boundary effects. The specific search strategy is shown in Figure 2. The red dot is used to represent the center coordinate of the object's bounding box. When the criterion is triggered, the tracker will record the coordinates of the red dot. The red box indicates the search region centered on the red dot, where W search is the width and H search is the height. The green box with a green dot is a sliding window that will perform a cyclic shift within the search region. FIGURE 2Open in figure viewerPowerPoint The schematic diagram of the local search strategy Let W K and H K denote the width and height of the bounding box at K frame, respectively. In our algorithm, we use W K and H K to represent the search region, i.e. W search = φ · W K and H search = φ · H K . Therefore, the search region S search can be expressed as S search = W search · H search = φ 2 · ( W K · H K ) (10)where φ is a positive integer greater than 1. The larger the values of φ, the larger the search region S search . Then, the sliding step size of the sliding window in the direction of the x coordinate axis and the y coordinate axis can be written as follows Δ x = W search / γ (11) Δ y = H search / γ (12)where γ is the number of sample points along the x and y axes. This means that a total of γ 2 samples are generated throughout the local search region. 3.3.3 Global search strategy In Section 4, extensive experiments demonstrate the effectiveness of the proposed local search strategy. However, local search is obviously faced with a serious problem that the method will fail when the target moves out of the search region S search . One of the most straightforward ways to alleviate this flaw is to increase φ and γ, but this can cause serious degradation to the real-time performance of the tracker. This is mainly because the number of samples increases quadratically with an increase in γ, and such a computational burden is unaffordable for the tracker. Considering that some computationally efficient target detection algorithms have been applied to the object tracking field, we use the EdgeBox [61] to perform global search on images. In simple terms, the core idea of EdgeBox is to calculate a score for each sliding window that represents the likelihood that the area corresponding to the sliding window contains objects. If the score is large, the area is likely to contain objects. On the contrary, if the score is small, the possibility that the area contains objects is small. EdgeBox can quickly generate object bounding box proposals directly from the edges and it typically takes only about 0.35s to generate 1000 proposals in a 480 × 720 image. Given the advantages of EdgeBox, using this method to perform global search on images does not have a significant adverse effect on tracking speed. More details about EdgeBox can be found in the literature [61]. As for the switch between local search and global search, we will record the average displacement D ¯ of the target in the last 5 frames. Assuming that the tracking failure occurs at the Kth frame, in order to improve real-time performance and detection efficiency, we directly filter out the subsequent 10 frames of images before starting detection. In the actual process, most objects are moving at a constant speed and the average displacement is generally in the range of 4–8 pixels, while the fast-moving objects reach 20 pixels and above. On the basis of ensuring tracking accuracy and running speed, we set the local search area to 4 times the object area and the number of sampling points to 10, because too large settings of φ and γ will seriously affect the real-time performance of the tracker. Taking the general object size of 80 × 80 as a reference, the local search area is 320 × 320 , and the search radius is about 160 pixels. Therefore, in order to determine whether the lost object is still distributed in the range of 160 pixels after 10 frames, we set the average displacement boundary value to 15 pixels. If D ¯ is less than 15 pixels, local search is used, otherwise global