# A Real-Time Structured Output Tracker with Scale Adaption for Visual Target Tracking

Authors: Kaiyun Yang; Xuedong Wu; Jingxiang Xu
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

The structured output tracking algorithm is a visual target tracking algorithm with excellent comprehensive performance in recent years. However, the algorithm classifier will produce error information and result in target loss or tracking failure when the target is occluded or the scale changes in the process of tracking. In this work, a real-time structured output tracker with scale adaption is proposed: (1) the target position prediction is added in the process of target tracking to improve the real-time tracking performance; (2) the adaptive scheme of target scale discrimination is proposed in the structured support to improve the overall tracking accuracy; and (3) the Kalman filter is used to solve the occlusion problem of continuous tracking. Extensive evaluations on the OTB-2015 benchmark dataset with 100 sequences have shown that the proposed tracking algorithm can run at a highly efficient speed of 84 fps and perform favorably against other tracking algorithms.


---

## Body

## 1. Introduction

Visual target tracking is an important task in computer vision, which has been widely used in various fields, such as intelligent transportation systems, military, medical, and so on [1–3]. Many scholars have done a lot of research on target tracking and have achieved great progress. However, the tracking target faces many challenges such as target scale changes, occlusion, and illumination changes [4, 5]. Therefore, ensuring the accuracy of the tracking process and improving the real-time performance are of great theoretical significance [6, 7].The Struck tracker is an algorithm based on discriminant classifier, and it has excellent adaptability to various complex backgrounds in visual target tracking [8, 9]. The Struck tracker is to construct a structured output support vector machine classifier through an online learning method. In the tracking process, the algorithm first samples the local area of target position in the previous frame and then takes the maximum value of classifier discriminant function as the current tracking result [10]. In the process of updating classifier, the classifier abandons the sample labeling process of classifier. Therefore, in the case of occlusion and scale variation, the Struck tracker cannot track target correctly [11, 12]. This paper suggests a structured output SVM tracker with real-time and occlusion detection capabilities. This work includes the following. (1) The target position prediction process is introduced. The sparse sample is used to calculate the rough position of target from the discriminator, which reduces the amount of calculation to determine the target position and improves the real-time performance. (2) A multiscale sampling adaptive scale tracking strategy is proposed. The scale discriminator calculates the scale change of target, which can adjust the size of scale adaptively. (3) The discriminant function of SVM classifier is used to implement occlusion detection. The SVM classifier will stop updating when a certain percentage of occlusion is detected, and the Kalman filter is applied to predict target position when the target has certain motion information, so the target can be successfully tracked.


## 2. Related Works

With the development of target tracking, many visual target tracking algorithms have been investigated [13]. The mainstream target tracking algorithms include traditional tracking methods, correlation filtering-based tracking methods, and tracking methods based on deep learning [14, 15].


### 2.1. Traditional Tracking Methods

Caulfield and Dawson-Howe[16] proposed a FragTrack tracking method by comparing template histograms with sub-block histograms to obtain the possible locations of target. Wang and Li [17] suggested a mean shift algorithm which used color as feature to obtain the probability density map of overall image. Wu et al. [18] introduced a multi-instance learning method to solve target drift and improve tracking accuracy. Babenko et al. [19] provided a compressive tracking algorithm to deal with the problem of occlusions and image noise. Chunxiao et al. [20] developed a tracking-learning-detection method combining traditional tracking algorithms with detection algorithms to solve the target occlusions. Hare et al. [21] presented a structured output SVM tracker which was robust to occlusions of tracking target.


### 2.2. Correlation Filtering-Based Tracking Methods

Bolme et al. [22] used a minimum output sum of squared error filter, which could improve tracking accuracy compared with many traditional algorithms. Henriques et al. [23] developed a circulant structure kernel tracker by building a cyclic matrix to solve the ridge regression problem. Henriques et al. [24] proposed a kernel correlation filter which had excellent performance in tracking accuracy, but it could not solve illumination and scale changes. Zhang and Zheng [25] presented a spatiotemporal context tracking algorithm which had achieved excellent real-time tracking performance. Li and Zhu [26] suggested a scale adaptive multifeature tracking algorithm for solving the scale changes of tracking target.


### 2.3. Tracking Methods Based on Deep Learning

Held et al [27] presented a GOTURN tracking method and applied the end-to-end deep learning model to target tracking for the first time. Nam and Han [28] developed an MDNet tracking algorithm and updated the model online to adapt to the changing targets and scenarios. Danelljan [29] proposed a C-COT tracking algorithm, and it used the deep neural network to extract target features and solved target scale changes. Danelljan [30] proposed an effective convolution operator tracking algorithm, which solved the problem of too large C-COT model and improved the real-time performance.Although these research methods have some effect in dealing with complex background, there are two shortcomings which can be summarized as follows: (1) these algorithms cannot judge the current occlusion state and introduce error information into the classifier when the target is occluded; (2) the performance of the algorithm is significantly reduced when the target scale changes. This work uses the discriminant function value of SVM classifier to achieve occlusion detection by combining the framework of Struck tracker and the motion information of target. The classifier stops updating when partial occlusion is detected, and the Kalman filter is used to predict the target position when the target has certain motion information. Aiming at the change of target scale, a multiscale sampling strategy is proposed, and the optimal scale is calculated with scale discriminator.


## 2.1. Traditional Tracking Methods

Caulfield and Dawson-Howe[16] proposed a FragTrack tracking method by comparing template histograms with sub-block histograms to obtain the possible locations of target. Wang and Li [17] suggested a mean shift algorithm which used color as feature to obtain the probability density map of overall image. Wu et al. [18] introduced a multi-instance learning method to solve target drift and improve tracking accuracy. Babenko et al. [19] provided a compressive tracking algorithm to deal with the problem of occlusions and image noise. Chunxiao et al. [20] developed a tracking-learning-detection method combining traditional tracking algorithms with detection algorithms to solve the target occlusions. Hare et al. [21] presented a structured output SVM tracker which was robust to occlusions of tracking target.


## 2.2. Correlation Filtering-Based Tracking Methods

Bolme et al. [22] used a minimum output sum of squared error filter, which could improve tracking accuracy compared with many traditional algorithms. Henriques et al. [23] developed a circulant structure kernel tracker by building a cyclic matrix to solve the ridge regression problem. Henriques et al. [24] proposed a kernel correlation filter which had excellent performance in tracking accuracy, but it could not solve illumination and scale changes. Zhang and Zheng [25] presented a spatiotemporal context tracking algorithm which had achieved excellent real-time tracking performance. Li and Zhu [26] suggested a scale adaptive multifeature tracking algorithm for solving the scale changes of tracking target.


## 2.3. Tracking Methods Based on Deep Learning

Held et al [27] presented a GOTURN tracking method and applied the end-to-end deep learning model to target tracking for the first time. Nam and Han [28] developed an MDNet tracking algorithm and updated the model online to adapt to the changing targets and scenarios. Danelljan [29] proposed a C-COT tracking algorithm, and it used the deep neural network to extract target features and solved target scale changes. Danelljan [30] proposed an effective convolution operator tracking algorithm, which solved the problem of too large C-COT model and improved the real-time performance.Although these research methods have some effect in dealing with complex background, there are two shortcomings which can be summarized as follows: (1) these algorithms cannot judge the current occlusion state and introduce error information into the classifier when the target is occluded; (2) the performance of the algorithm is significantly reduced when the target scale changes. This work uses the discriminant function value of SVM classifier to achieve occlusion detection by combining the framework of Struck tracker and the motion information of target. The classifier stops updating when partial occlusion is detected, and the Kalman filter is used to predict the target position when the target has certain motion information. Aiming at the change of target scale, a multiscale sampling strategy is proposed, and the optimal scale is calculated with scale discriminator.


## 3. Proposed Method

First, the Struck tracker and a real-time structured output tracker with scaleadaption and then the whole details of the presented algorithm are given.


### 3.1. The Struck Tracker

During the tracking process, letPt−1 be the estimated bounding box at time t−1. The Struck algorithm estimates the target displacement yt∈Y, where Y is the search space which is given as Y=u,v|u2+v2<r2 (r is the search radius, and are two-dimensional space coordinates). The current frame target position Pt which is expressed as Pt=Pt−1∘yt can be obtained by shifting the previous frame position.The prediction functionf:X⟶Y is constructed to estimate target transformations between frames, and the output space is the space of all transformations Y instead of the binary labels. Now, we introduce a discriminant function to predict the target position between frames as follows:(1)yt=fxtPt−1=argmaxy∈YFxtPt−1,y,which evaluates the similarity between sample and target (w is the coefficient vector and Φ is the kernel function that maps the input space to the feature space). In the given sample set x1,y1,…,xn,ynn≥1, for finding the optimal hyperplane, the optimization objective function is(2)minw12w2+C∑i=1nξi,s.t.∀i,∀y≠yi:w,δΦiy≥Δyi,y−ξi,where Δyi,y is the loss function and C is the regularization parameter. This loss function should decrease towards 00 as y and y¯ become more similar and is Δyi,y¯=0 only on the condition that y=y¯. We now choose the loss function based on the overlap of target bounding boxes as(3)Δy,y¯=1−sPtoy,y¯,where sPtoy,y¯ is the overlap between target bounding boxes. Using the standard Lagrangian duality techniques, the solution of equation (2) can be converted into its equivalent dual form:(4)maxα∑i,y≠yiΔy,yiαiy−12∑i,y≠yij,y¯≠yiαiyαjy¯δΦiy,δΦjy¯,s.t.∀i,∀y≠yi:αiy≥0,∀i:∑y≠yiαiy≤C,replacing the parameter into equation (4) as follows:(5)βiy=−αiy,y≠yi,∑y¯≠yiαiy¯,otherwise.According to equation (5), equation (4) can be written as(6)maxβ−∑i,yΔy,yiβiy−12∑i,y,j,y¯βiyβjy¯Φxi,yΦxj,y¯,s.t.∀i,∀y:βiy≤δy,yiC,∀i:∑yβiy=0,where(7)δy,y¯=1,y=y¯,0,otherwise.Then, the discriminant function can be simplified as(8)Fx,y=∑i,y¯βjy¯Φxi,y,Φxj,y¯.We refer the samplexi,y with βiy≠0 as support vector. For a given support pattern xi, only the support vector xi,yi is βiy>0 and any other support vectors xi,y with y≠yi is βiy<0. We refer to these as positive and negative support vectors, respectively. The selection of support vector is controlled by the following gradient:(9)hiy=−Δy,yi−∑j,y¯βjy¯Φxi,y,Φxj,y¯=−Δy,yi−Fxi,y.It can be known fromΔy,yi that the gradient calculation includes the overlap between the sample and the target bounding box, and the algorithm updates βiy and gradient hi incrementally during the update process of each frame to implement the classifier learning and the update of the algorithm.


### 3.2. The Real-Time Structured Output Tracker with Scale Adaption

The output of the Struck tracker is only the target position, and the accuracy of target position decreases when the scale changes, so we suggest a real-time structured output model with scale adaption. By taking the target tracking framex of the previous frame, the candidate possible target position set Y in the current frame, and the candidate target scale set S in the current frame as input and taking the exact position y and the target scale in the current frame as output, the model can be represented using the following decision function:(10)gy,s=argmaxy,s∈Y,SGx,y,s,where G is the discriminant function, and the discriminant function G in the model is(11)Gx,y,s=w,Φx,y,s,where w is the coefficient vector, Φx,y,s is the structured feature function, and , is the inner product operation. By combining equation (12), the adaptive scale tracking model is(12)minw12w2+C∑i=1Nεi,s.t.∀i:εi>0,∀i,∀j,∀y≠yi,∀s≠sj:w,Φxi,yi,sj−Φxi,y,s≥Δyi,sj;y,s−εi,where is the sampling number, εi is the relaxation amount, and Δyi,sj;y,s is the loss function. The loss function can be written as(13)Δyi,sj;y,s=1−Tyi,sj∩Ty,sTyi,sj∪Ty,s,where Ty,s is the sample blocks when the target positions and scales are y,s and Tyi,sj is the sample blocks when the target positions and scales are yi,sj.We can transform the structured SVM problem of equation (12) into a dual problem(14)maxα∑i,y≠yi;j,s≠sjΔyi,sj;y,sαijy,s−12∑i,y≠yi,m,s≠smj,y¯≠yj,n,s¯≠snαimy,sαjny¯,s¯δΦimy,s,δjny¯,s¯,s.t.∀i,∀y≠yi,∀s≠yj:αijy,s≥0,∀i:∑y≠yi,m,s≠sjαijy,s≤C,where δΦimy,s=Φxi,yi,sm−Φxi,y,s. The discriminant function is(15)Fx,y,s=∑i,y≠yi;j,s≠sjαijy¯,s¯δΦijy¯,s¯,Φx,y,s.Wheny¯≠yi,s¯≠sj, set βijy,s=−αijy,s or βijy,s=∑y¯≠yi;j,s¯≠sjαijy¯,s¯, and equation (14) can be converted as(16)maxβ∑i,y,j,sΔy,s;yi,sjβijy,s−12∑i,y,s,j,y¯,s¯βimy,sβjny¯,s¯Φxi,y,s,Φxj,y¯,s¯,s.t.∀i,∀y,∀j,∀s:βijy,s≤δy,s;yi,sjC,∀i,∀j:∑y,sβimy,s=0,where δy,s;yi,sj=1 when y = yi and s = sj, else δy,s;yi,sj=0. We can get the discriminant function as follows:(17)Gx,y,s=∑i,j,y¯,s¯βijy¯,s¯Φxi,y¯,s¯,Φx,y,s.By selecting the proper kernel functionKx,y,s;x¯,y¯,s¯=Φxi,y¯,s¯,Φx,y,s, the discriminant function is(18)Gx,y,s=∑i,j,y¯,s¯βijy¯,s¯Kx,y,s;x¯,y¯,s¯.Equation (18) is the final form of discriminant function in the adaptive scale tracking model.The algorithm of tracking process is divided into two parts: the target position prediction and the target position and scale determination.In the process of target position prediction, the target position is sampled equally at first, and then the previous frame of target position is evenly sampled according to the sparse sampling method:(19)Yrough,s=y1rough,s,y2rough,s…yjrough,s…ylrough,s.The structural feature vector of the corresponding position is obtained by extracting the feature of the samples:(20)Φx,y1rough,s,Φx,y2rough,s…Φx,yjrough,s…Φx,ylrough,s.The feature vectorΦx,yjrough,s (the corresponding sample yjrough,s is the target position prediction result and yjrough is the rough target position) that maximizes the decision function is selected according to the discriminant function equation (18) and the decision function equation (10).The process of determining the target position and scale includes four steps:(1)
We set up a set of scalesS=s1,s2,…sj…sk (k≥1), and the collected elements are satisfied with sj−sj−1=gap (where gap is the step size).(2)
By taking the rough estimated target locationyjrough as center, the search area is further reduced and we can sample the scale set S in the reduced search area and get k sampling results:(21)Yaccurate,S=y11accurate,s1,y12accurate,s1…y1qaccurate,s1…y1maccurate,s1;y21accurate,s2,y22accurate,s2⋯y2qaccurate,s2…y2maccurate,s2;⋮yp1accurate,sp,yp2accurate,sp…ypqaccurate,sp…ypmaccurate,sp;⋮yk1accurate,sk,yk2accurate,sk…ykqaccurate,sk…ykmaccurate,sk.(3)
Perform feature extraction for each sample and obtaink structured feature vector(22)Φx,y11accurate,s1,Φx,y12accurate,s1⋯Φx,y1qaccurate,s1⋯Φx,y1maccurate,s1;Φx,y21accurate,s2,Φx,y22accurate,s2⋯Φx,y2qaccurate,s2⋯Φx,y2maccurate,s2;⋮Φx,yp1accurate,sp,Φx,yp2accurate,sp⋯Φx,ypqaccurate,sp⋯Φx,ypmaccurate,sp;⋮Φx,yk1accurate,sk,Φx,yk2accurate,sk⋯Φx,ykqaccurate,sk⋯Φx,ykmaccurate,sk;.(4)
Select the feature vectorΦxpq,ypqaccurate,sp (the corresponding sample ypqaccurate,sp,in whichypqaccurate is the exact position and sp is the target scale, is the tracking result) that maximizes the decision function according to equations (18) and (22).The performance of the Struck tracker is decreased significantly when the target is partially occluded or completely occluded. We perform an active occlusion detection during the tracking process of each frame.LetItPt−1Οy be the value of classifier discriminant function at the search position P=Pt−1Oy in frame It (where t is the frame number of tracking target):(23)ItPt−1Oy=IxtPt−1,y.For the target position of the current frame, the classifier discriminant function value at the satisfying target positionP=Pt−1Oyt is ItPt−1Oyt, then(24)It=IxtPt−1,yt.The change rate ofIt can be defined as(25)VIt=It−I¯I¯.The queueQ=I1,…,Iv (v∈t) is constructed, and it stores the value of v frame history It, where I¯=1/v∑i=1i=vQi is the average value of elements in queue Q. After each frame target tracking, the current frame It value is added to the queue Q to update the occlusion detection.The threshold of occlusion detectionγ is introduced to distinguish whether the target is occluded or not. When VIt>γ, the algorithm continues to track the target. When VIt≤γ, the algorithm stops updating the element of queue Q, and the classifier to ensure that the classifier does not introduce error information.


### 3.3. Steps of the Suggested Method

The proposed algorithm steps are shown in Algorithm1. The steps of the developed method are as follows.Algorithm 1:A real-time structured output tracker with scale adaption for visual target tracking.
Input:It: the tth frame video t≥1Y: target position set of the previous frameS: target scale set of the previous frameγ: occlusion thresholdOutput:Yrough: target prediction position set of the current frameΦx,ylrough,s: the structured feature vector of rough positionyjrough: the target rough positionΦxpq,ypqaccurate,sp: the structured feature vector of accurate positionsp: target scale of the current frameypqaccurate: the accurate position of targetEndTarget rough position prediction process(1)
Obtain the sparse sampling result Yrough of target prediction position from equation (19)(2)
Get the structured feature vectorsΦx,ylaccurate,s according to equation (20) by performing feature extraction on the samples(3)
Take the structured feature vector with maximal decision function as the target rough positionyjrough according to equations (10) and (18)Determination of target accurate position and scale(4)
Set a set of scalesS and obtain k scale samples(5)
Densely sample each scale of the setS and get ksets sampling results from equation (21) taking estimated position yjrough as the center(6)
Perform feature extraction on the samples to obtaink groups of structured feature vectors according to equation (22)(7)
The target scalesp and the accurate position ypqaccurate can be obtained by calculating the feature vectors Φxpq,ypqaccurate,sp which maximize the decision functionThe target occlusion judgment process(8)
Set the size of occlusion thresholdγ(9)
Calculate the change rate valueVIt of the discriminant function according to equations (24) and (25)IfVIt≤γ(10)
The tracking target is occluded and the classifier and the queueQ element stop updatingelse(11)
The tracking target is not occluded and the tracker continues to track the targetUntil the end of all framesEnd


## 3.1. The Struck Tracker

During the tracking process, letPt−1 be the estimated bounding box at time t−1. The Struck algorithm estimates the target displacement yt∈Y, where Y is the search space which is given as Y=u,v|u2+v2<r2 (r is the search radius, and are two-dimensional space coordinates). The current frame target position Pt which is expressed as Pt=Pt−1∘yt can be obtained by shifting the previous frame position.The prediction functionf:X⟶Y is constructed to estimate target transformations between frames, and the output space is the space of all transformations Y instead of the binary labels. Now, we introduce a discriminant function to predict the target position between frames as follows:(1)yt=fxtPt−1=argmaxy∈YFxtPt−1,y,which evaluates the similarity between sample and target (w is the coefficient vector and Φ is the kernel function that maps the input space to the feature space). In the given sample set x1,y1,…,xn,ynn≥1, for finding the optimal hyperplane, the optimization objective function is(2)minw12w2+C∑i=1nξi,s.t.∀i,∀y≠yi:w,δΦiy≥Δyi,y−ξi,where Δyi,y is the loss function and C is the regularization parameter. This loss function should decrease towards 00 as y and y¯ become more similar and is Δyi,y¯=0 only on the condition that y=y¯. We now choose the loss function based on the overlap of target bounding boxes as(3)Δy,y¯=1−sPtoy,y¯,where sPtoy,y¯ is the overlap between target bounding boxes. Using the standard Lagrangian duality techniques, the solution of equation (2) can be converted into its equivalent dual form:(4)maxα∑i,y≠yiΔy,yiαiy−12∑i,y≠yij,y¯≠yiαiyαjy¯δΦiy,δΦjy¯,s.t.∀i,∀y≠yi:αiy≥0,∀i:∑y≠yiαiy≤C,replacing the parameter into equation (4) as follows:(5)βiy=−αiy,y≠yi,∑y¯≠yiαiy¯,otherwise.According to equation (5), equation (4) can be written as(6)maxβ−∑i,yΔy,yiβiy−12∑i,y,j,y¯βiyβjy¯Φxi,yΦxj,y¯,s.t.∀i,∀y:βiy≤δy,yiC,∀i:∑yβiy=0,where(7)δy,y¯=1,y=y¯,0,otherwise.Then, the discriminant function can be simplified as(8)Fx,y=∑i,y¯βjy¯Φxi,y,Φxj,y¯.We refer the samplexi,y with βiy≠0 as support vector. For a given support pattern xi, only the support vector xi,yi is βiy>0 and any other support vectors xi,y with y≠yi is βiy<0. We refer to these as positive and negative support vectors, respectively. The selection of support vector is controlled by the following gradient:(9)hiy=−Δy,yi−∑j,y¯βjy¯Φxi,y,Φxj,y¯=−Δy,yi−Fxi,y.It can be known fromΔy,yi that the gradient calculation includes the overlap between the sample and the target bounding box, and the algorithm updates βiy and gradient hi incrementally during the update process of each frame to implement the classifier learning and the update of the algorithm.


## 3.2. The Real-Time Structured Output Tracker with Scale Adaption

The output of the Struck tracker is only the target position, and the accuracy of target position decreases when the scale changes, so we suggest a real-time structured output model with scale adaption. By taking the target tracking framex of the previous frame, the candidate possible target position set Y in the current frame, and the candidate target scale set S in the current frame as input and taking the exact position y and the target scale in the current frame as output, the model can be represented using the following decision function:(10)gy,s=argmaxy,s∈Y,SGx,y,s,where G is the discriminant function, and the discriminant function G in the model is(11)Gx,y,s=w,Φx,y,s,where w is the coefficient vector, Φx,y,s is the structured feature function, and , is the inner product operation. By combining equation (12), the adaptive scale tracking model is(12)minw12w2+C∑i=1Nεi,s.t.∀i:εi>0,∀i,∀j,∀y≠yi,∀s≠sj:w,Φxi,yi,sj−Φxi,y,s≥Δyi,sj;y,s−εi,where is the sampling number, εi is the relaxation amount, and Δyi,sj;y,s is the loss function. The loss function can be written as(13)Δyi,sj;y,s=1−Tyi,sj∩Ty,sTyi,sj∪Ty,s,where Ty,s is the sample blocks when the target positions and scales are y,s and Tyi,sj is the sample blocks when the target positions and scales are yi,sj.We can transform the structured SVM problem of equation (12) into a dual problem(14)maxα∑i,y≠yi;j,s≠sjΔyi,sj;y,sαijy,s−12∑i,y≠yi,m,s≠smj,y¯≠yj,n,s¯≠snαimy,sαjny¯,s¯δΦimy,s,δjny¯,s¯,s.t.∀i,∀y≠yi,∀s≠yj:αijy,s≥0,∀i:∑y≠yi,m,s≠sjαijy,s≤C,where δΦimy,s=Φxi,yi,sm−Φxi,y,s. The discriminant function is(15)Fx,y,s=∑i,y≠yi;j,s≠sjαijy¯,s¯δΦijy¯,s¯,Φx,y,s.Wheny¯≠yi,s¯≠sj, set βijy,s=−αijy,s or βijy,s=∑y¯≠yi;j,s¯≠sjαijy¯,s¯, and equation (14) can be converted as(16)maxβ∑i,y,j,sΔy,s;yi,sjβijy,s−12∑i,y,s,j,y¯,s¯βimy,sβjny¯,s¯Φxi,y,s,Φxj,y¯,s¯,s.t.∀i,∀y,∀j,∀s:βijy,s≤δy,s;yi,sjC,∀i,∀j:∑y,sβimy,s=0,where δy,s;yi,sj=1 when y = yi and s = sj, else δy,s;yi,sj=0. We can get the discriminant function as follows:(17)Gx,y,s=∑i,j,y¯,s¯βijy¯,s¯Φxi,y¯,s¯,Φx,y,s.By selecting the proper kernel functionKx,y,s;x¯,y¯,s¯=Φxi,y¯,s¯,Φx,y,s, the discriminant function is(18)Gx,y,s=∑i,j,y¯,s¯βijy¯,s¯Kx,y,s;x¯,y¯,s¯.Equation (18) is the final form of discriminant function in the adaptive scale tracking model.The algorithm of tracking process is divided into two parts: the target position prediction and the target position and scale determination.In the process of target position prediction, the target position is sampled equally at first, and then the previous frame of target position is evenly sampled according to the sparse sampling method:(19)Yrough,s=y1rough,s,y2rough,s…yjrough,s…ylrough,s.The structural feature vector of the corresponding position is obtained by extracting the feature of the samples:(20)Φx,y1rough,s,Φx,y2rough,s…Φx,yjrough,s…Φx,ylrough,s.The feature vectorΦx,yjrough,s (the corresponding sample yjrough,s is the target position prediction result and yjrough is the rough target position) that maximizes the decision function is selected according to the discriminant function equation (18) and the decision function equation (10).The process of determining the target position and scale includes four steps:(1)
We set up a set of scalesS=s1,s2,…sj…sk (k≥1), and the collected elements are satisfied with sj−sj−1=gap (where gap is the step size).(2)
By taking the rough estimated target locationyjrough as center, the search area is further reduced and we can sample the scale set S in the reduced search area and get k sampling results:(21)Yaccurate,S=y11accurate,s1,y12accurate,s1…y1qaccurate,s1…y1maccurate,s1;y21accurate,s2,y22accurate,s2⋯y2qaccurate,s2…y2maccurate,s2;⋮yp1accurate,sp,yp2accurate,sp…ypqaccurate,sp…ypmaccurate,sp;⋮yk1accurate,sk,yk2accurate,sk…ykqaccurate,sk…ykmaccurate,sk.(3)
Perform feature extraction for each sample and obtaink structured feature vector(22)Φx,y11accurate,s1,Φx,y12accurate,s1⋯Φx,y1qaccurate,s1⋯Φx,y1maccurate,s1;Φx,y21accurate,s2,Φx,y22accurate,s2⋯Φx,y2qaccurate,s2⋯Φx,y2maccurate,s2;⋮Φx,yp1accurate,sp,Φx,yp2accurate,sp⋯Φx,ypqaccurate,sp⋯Φx,ypmaccurate,sp;⋮Φx,yk1accurate,sk,Φx,yk2accurate,sk⋯Φx,ykqaccurate,sk⋯Φx,ykmaccurate,sk;.(4)
Select the feature vectorΦxpq,ypqaccurate,sp (the corresponding sample ypqaccurate,sp,in whichypqaccurate is the exact position and sp is the target scale, is the tracking result) that maximizes the decision function according to equations (18) and (22).The performance of the Struck tracker is decreased significantly when the target is partially occluded or completely occluded. We perform an active occlusion detection during the tracking process of each frame.LetItPt−1Οy be the value of classifier discriminant function at the search position P=Pt−1Oy in frame It (where t is the frame number of tracking target):(23)ItPt−1Oy=IxtPt−1,y.For the target position of the current frame, the classifier discriminant function value at the satisfying target positionP=Pt−1Oyt is ItPt−1Oyt, then(24)It=IxtPt−1,yt.The change rate ofIt can be defined as(25)VIt=It−I¯I¯.The queueQ=I1,…,Iv (v∈t) is constructed, and it stores the value of v frame history It, where I¯=1/v∑i=1i=vQi is the average value of elements in queue Q. After each frame target tracking, the current frame It value is added to the queue Q to update the occlusion detection.The threshold of occlusion detectionγ is introduced to distinguish whether the target is occluded or not. When VIt>γ, the algorithm continues to track the target. When VIt≤γ, the algorithm stops updating the element of queue Q, and the classifier to ensure that the classifier does not introduce error information.


## 3.3. Steps of the Suggested Method

The proposed algorithm steps are shown in Algorithm1. The steps of the developed method are as follows.Algorithm 1:A real-time structured output tracker with scale adaption for visual target tracking.
Input:It: the tth frame video t≥1Y: target position set of the previous frameS: target scale set of the previous frameγ: occlusion thresholdOutput:Yrough: target prediction position set of the current frameΦx,ylrough,s: the structured feature vector of rough positionyjrough: the target rough positionΦxpq,ypqaccurate,sp: the structured feature vector of accurate positionsp: target scale of the current frameypqaccurate: the accurate position of targetEndTarget rough position prediction process(1)
Obtain the sparse sampling result Yrough of target prediction position from equation (19)(2)
Get the structured feature vectorsΦx,ylaccurate,s according to equation (20) by performing feature extraction on the samples(3)
Take the structured feature vector with maximal decision function as the target rough positionyjrough according to equations (10) and (18)Determination of target accurate position and scale(4)
Set a set of scalesS and obtain k scale samples(5)
Densely sample each scale of the setS and get ksets sampling results from equation (21) taking estimated position yjrough as the center(6)
Perform feature extraction on the samples to obtaink groups of structured feature vectors according to equation (22)(7)
The target scalesp and the accurate position ypqaccurate can be obtained by calculating the feature vectors Φxpq,ypqaccurate,sp which maximize the decision functionThe target occlusion judgment process(8)
Set the size of occlusion thresholdγ(9)
Calculate the change rate valueVIt of the discriminant function according to equations (24) and (25)IfVIt≤γ(10)
The tracking target is occluded and the classifier and the queueQ element stop updatingelse(11)
The tracking target is not occluded and the tracker continues to track the targetUntil the end of all framesEnd


## 4. Experiments

The tacking performance of six trackers, which include the structured output tracking with kernels (Struck), the kernel correlation filter (KCF), the background-aware correlation filters (BACF), the compressive tracking (CT), the tracking-learning-detection (TLD), and the OURS tracker, is tested on the OTB-2015 benchmark dataset. The OTB-2015 dataset comprises 11 different complex backgrounds, namely, scale variation (SV), deformation (DEF), illumination variation (IV), out of view (OV), background clutter (BC), occlusion (OCC), motion blur (MB), fast motion (FM), out of plane rotation (OPR), in-plane rotation (IPR), and low resolution (LR). The overlap success rate and the center position error are used as evaluation criteria for trackers [31, 32]. The overlap success rate reflects the degree of overlap between the tracked target frame and the actual target frame [33, 34]. The center position error indicates the offset between the output target center position and the labeled position [35, 36].


### 4.1. Parameter Setup

Experiments are implemented in Matlab on an Intel I7–8565U 1.8 GHz CPU with 8 GB RAM. The regularization parameterC is set to 100, the size of support vector threshold is set to 100, the original search range is three times the target size of the previous frame, the number of scales is 31, the historical frame Ft is set as G=I1,…,Ik, the size of k is 10, the threshold value of occlusion detection γ is −0.3, the target boundary box size change factor δ is 0.1, and the search radius r is 30.


### 4.2. Experiments on OTB-2015 Benchmark Dataset

The tracking performance of different trackers is tested on the OTB-2015 benchmark dataset. The tracking results under the OTB-2015 dataset are shown in Figure1, and we can get that the tracking results of OURS tracker are better than comparsion trackers. The OURS tracker and the Struck tracker are also tested under various complex background, and the selected sequence information is shown in Table 1, and their average success rate plot and average precision in various complex background are shown in Figure 2. Figure 2 shows that the performance of the OURS tracker is better than that of the Struck tracker on the OTB-2015 benchmark dataset.Figure 1
The tracking results under the OTB-2015 dataset.Table 1
The selected sequences information.
SequenceFPSBackgroundsFaceocc1659OCCCoke569OCC, BC, FMFaceocc2307IV, OCCDeer71FM, MBMan1741IVDog1500OPR, SV, IRPFigure 2
Average success rate plot and average precision in various complex background.The success rate of various trackers in selected sequences is demonstrated in Table2. From Table 2, the tracking performance of OURS tracker is better than those of other comparison algorithms.Table 2
The success rate of various trackers in selected sequences.
MethodFaceocc1CokeFaceocc2DeerDog1ManOURS99.110099.884.599.399.8Struck80.571.582.558.486.678.0KCF87.170.379.980.188.181.3BACF84.363.475.660.182.072.1TLD79.754.874.343.370.167.7CT85.932.060.547.253.355.3The frames per second of various trackers in selected sequences are shown in Table3. From Table 3, the running speeds of OURS tracker is higher than those of other comparison algorithms.Table 3
The frames per second of various trackers in selected sequences.
MethodFaceocc1CokeFaceocc2DeerDog1ManOURS9710811958112109Struck345376286275KCF87811054799107BACF695561327177TLD121423113545CT373946176066


### 4.3. The Success Rate on More Challenging Video Sequences

In addition, the tracking success rate plots under 11 challenging video sequences are shown in Figure3. We can see from Figure 3 that the OURS tracker shows excellent tracking performance under 11 challenging video sequences than other trackers. Therefore, OURS tracker processes challenging video sequence effects better than other comparison trackers.Figure 3
Tracking success rate of trackers in 11 challenging video sequences.


### 4.4. Experiments on Various Complex Video Sequences

In the experiment on various complex video sequences, the comparison algorithms including OURS, Struck, KCF, and TLD are selected to test various video sequences, and their corresponding performance is discussed. The video sequences are Faceocc1, Coke, Faceocc2, Deer, Man, and Dog. The sequences information is shown in Table1, and the test results of sequences are shown in Figures 4–9 (the original images of Figures 4–9 are from https://cvlab.hanyang.ac.kr/tracker_benchmark/datasets.html).Figure 4
Faceocc1 (frames: 18, 44, 69).Figure 5
Coke (frames: 127, 256, 276).Figure 6
Faceocc2 (frames: 233, 261, 294).Figure 7
Deer (frames: 6, 23, 60).Figure 8
Man (frames: 13, 35, 70).Figure 9
Dog (frames: 589, 911, 1034).In the video sequence Faceocc1, the challenge during tracking is partial occlusion. The TLD tracker and the Struck tracker occur to drift at the 44th frame. Due to the removal of the target from partial occlusion, the comparsion trackers lose the target at the 69th frame, but the OURS tracker can deal with the problem of partial occlusion.In the video sequence Coke, the target in video sequence is affected by complete occlusion. The target is completely occluded at the 256th frame. The Struck tracker, the KCF tracker, and TLD tracker tracking fails at the 276th frame, but the OURS tracker can deal with full occlusion problem.In the video sequence Faceocc2, the partial occlusion affects the tracking target during the tracking process. The target is partial occluded at the 261th frame. The comparsion trackers lose the target when target removes from partial occlusion at the 294th frame, but the OURS tracker can solve the partial occlusion problem.In the video sequence Deer, the tracking target appears fast motion. The Struck tracker and the TLD tracker lose the target at the 19th frame. The KCF tracker loses the target at the 60th frame, but OURS tracker can solve the problem of fast motion.In the video sequence Man, the illumination variation affects the tracking target during the tracking process. The TLD tracker and the Struck tracker occur to drift in video sequence at the 35th frame. All comparison trackers lose the target when the background illumination is changed gradually, but the OURS tracker can solve the illumination variation problem.In the video sequence Dog, the tracking target scale changes. The comparsion trackers occur to drift at the 911th frame. The comparsion trackers lose the target at the 1034th frame, but the OURS tracker can adapt to changes in target scale and track the target correctly.


## 4.1. Parameter Setup

Experiments are implemented in Matlab on an Intel I7–8565U 1.8 GHz CPU with 8 GB RAM. The regularization parameterC is set to 100, the size of support vector threshold is set to 100, the original search range is three times the target size of the previous frame, the number of scales is 31, the historical frame Ft is set as G=I1,…,Ik, the size of k is 10, the threshold value of occlusion detection γ is −0.3, the target boundary box size change factor δ is 0.1, and the search radius r is 30.


## 4.2. Experiments on OTB-2015 Benchmark Dataset

The tracking performance of different trackers is tested on the OTB-2015 benchmark dataset. The tracking results under the OTB-2015 dataset are shown in Figure1, and we can get that the tracking results of OURS tracker are better than comparsion trackers. The OURS tracker and the Struck tracker are also tested under various complex background, and the selected sequence information is shown in Table 1, and their average success rate plot and average precision in various complex background are shown in Figure 2. Figure 2 shows that the performance of the OURS tracker is better than that of the Struck tracker on the OTB-2015 benchmark dataset.Figure 1
The tracking results under the OTB-2015 dataset.Table 1
The selected sequences information.
SequenceFPSBackgroundsFaceocc1659OCCCoke569OCC, BC, FMFaceocc2307IV, OCCDeer71FM, MBMan1741IVDog1500OPR, SV, IRPFigure 2
Average success rate plot and average precision in various complex background.The success rate of various trackers in selected sequences is demonstrated in Table2. From Table 2, the tracking performance of OURS tracker is better than those of other comparison algorithms.Table 2
The success rate of various trackers in selected sequences.
MethodFaceocc1CokeFaceocc2DeerDog1ManOURS99.110099.884.599.399.8Struck80.571.582.558.486.678.0KCF87.170.379.980.188.181.3BACF84.363.475.660.182.072.1TLD79.754.874.343.370.167.7CT85.932.060.547.253.355.3The frames per second of various trackers in selected sequences are shown in Table3. From Table 3, the running speeds of OURS tracker is higher than those of other comparison algorithms.Table 3
The frames per second of various trackers in selected sequences.
MethodFaceocc1CokeFaceocc2DeerDog1ManOURS9710811958112109Struck345376286275KCF87811054799107BACF695561327177TLD121423113545CT373946176066


## 4.3. The Success Rate on More Challenging Video Sequences

In addition, the tracking success rate plots under 11 challenging video sequences are shown in Figure3. We can see from Figure 3 that the OURS tracker shows excellent tracking performance under 11 challenging video sequences than other trackers. Therefore, OURS tracker processes challenging video sequence effects better than other comparison trackers.Figure 3
Tracking success rate of trackers in 11 challenging video sequences.


## 4.4. Experiments on Various Complex Video Sequences

In the experiment on various complex video sequences, the comparison algorithms including OURS, Struck, KCF, and TLD are selected to test various video sequences, and their corresponding performance is discussed. The video sequences are Faceocc1, Coke, Faceocc2, Deer, Man, and Dog. The sequences information is shown in Table1, and the test results of sequences are shown in Figures 4–9 (the original images of Figures 4–9 are from https://cvlab.hanyang.ac.kr/tracker_benchmark/datasets.html).Figure 4
Faceocc1 (frames: 18, 44, 69).Figure 5
Coke (frames: 127, 256, 276).Figure 6
Faceocc2 (frames: 233, 261, 294).Figure 7
Deer (frames: 6, 23, 60).Figure 8
Man (frames: 13, 35, 70).Figure 9
Dog (frames: 589, 911, 1034).In the video sequence Faceocc1, the challenge during tracking is partial occlusion. The TLD tracker and the Struck tracker occur to drift at the 44th frame. Due to the removal of the target from partial occlusion, the comparsion trackers lose the target at the 69th frame, but the OURS tracker can deal with the problem of partial occlusion.In the video sequence Coke, the target in video sequence is affected by complete occlusion. The target is completely occluded at the 256th frame. The Struck tracker, the KCF tracker, and TLD tracker tracking fails at the 276th frame, but the OURS tracker can deal with full occlusion problem.In the video sequence Faceocc2, the partial occlusion affects the tracking target during the tracking process. The target is partial occluded at the 261th frame. The comparsion trackers lose the target when target removes from partial occlusion at the 294th frame, but the OURS tracker can solve the partial occlusion problem.In the video sequence Deer, the tracking target appears fast motion. The Struck tracker and the TLD tracker lose the target at the 19th frame. The KCF tracker loses the target at the 60th frame, but OURS tracker can solve the problem of fast motion.In the video sequence Man, the illumination variation affects the tracking target during the tracking process. The TLD tracker and the Struck tracker occur to drift in video sequence at the 35th frame. All comparison trackers lose the target when the background illumination is changed gradually, but the OURS tracker can solve the illumination variation problem.In the video sequence Dog, the tracking target scale changes. The comparsion trackers occur to drift at the 911th frame. The comparsion trackers lose the target at the 1034th frame, but the OURS tracker can adapt to changes in target scale and track the target correctly.


## 5. Conclusion

In this work, a real-time structured output tracker with scale adaption is proposed: (1) the process of position target prediction which can improve the tracking real-time performance is added during the tracking process; (2) multiscale sampling is used to obtain samples of different scales, and the best scale is obtained by using a discriminator to improve the accuracy of tracking; and (3) the occlusion judgment mechanism is suggested to determine whether to update the classifier or not, and the Kalman filtering is applied to solve the problem of continuous tracking with occlusion.The tracking performance of OURS tracker is better than those of other trackers in different research cases due to the following advantages. The OURS tracker uses a multiscale sampling strategy to estimate the scale of target during tracking. The OURS tracker uses Kalman filter to solve tracking problem with target occlusion. From the experimental results, the tracker proposed in this paper shows excellent performance when processing various complex backgrounds under the OTB-2015 dataset, and it also achieves excellent success rate and tracking accuracy in different challenging complex backgrounds.In the future, our research work will focus on applying the proposed algorithm to multitarget tracking due to the successful application of the proposed algorithm on the single target tracking.



---

*Source: 4649358-2021-02-10.xml*