# Object Tracking with Multi-Classifier Fusion Based on Compressive Sensing and Multiple Instance Learning

Authors: Si Chen; Xiaoshun Lu; Xiaosen Chen; Min Chen; Jianghu Chen; Dahan Wang; Shunzhi Zhu
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Object tracking is a critical research in computer vision and has attracted significant attention over the past few years. However, the traditional object tracking algorithms often suffer from the object drifting problem due to various challenging factors in complex environments such as object occlusion and background clutter. This paper proposes a robust and effective object tracking algorithm, called MCM, which combines compressive sensing and online multiple instance learning in a multi-classifier fusion framework. In this framework, we integrate the different discriminative classifiers by learning the varied and compressed feature vectors based on different random projection matrices. And then an improved online multiple instance learning mechanism SMILE is adopted, which introduces the relative similarity to select and weight the instances in the positive bag. The experiments show that the proposed algorithm can improve the performance of object tracking on the challenging video sequences.


---

## Body

## 1. Introduction

Object tracking [1] is an important research direction in the field of computer vision and has a wide range of applications. For example, video surveillance [2], mobile robots [3, 4], and defense reconnaissance [5]. The object tracking methods continuously predict the location and the other information of the object for the subsequent frames in a video sequence, given the initial state in the first frame. At the same time, there are many challenges in object tracking, such as object deformation, out-of-plane rotation, and illumination variation. In the above scenes, the traditional tracking algorithms are easy to lead to tracking failure [6]. In this paper, we mainly focus on the online object tracking problem when the object location in the first frame of a video sequence is only given without both the other training samples and prior knowledge.During the past several years, many scholars have proposed many object tracking algorithms, for example, CT (Compressive Tracking) [7], MIL (Multiple Instance Learning) tracker [6], MOSSE (Minimum Output Sum of Squared Error) filter [8], TLD (Tracking-Learning-Detection) tracker [9], Struck (Structured Output Tracking with Kernels) [10], CSK (Circulant Structure of Tracking-by-detection with Kernels) method [11], MD-Net (Multi-Domain Convolutional Neural Networks) tracker [12], and DLC (Discriminative Local Collaborative Representation) tracker [13]. These algorithms also have their own advantages and disadvantages. For example, the compressive sensing technology is introduced in the CT algorithm [7] for feature extraction. However, only a fixed random projection matrix is used to feature extraction, which easily loses discriminative features when the appearance of the object changes drastically. Although the MIL paradigm has been successfully used in the object tracking [6], there are still many problems to be solved. The traditional MIL-based object tracking algorithm [6] may be inaccurate in the complex scenes such as fast movements and occlusions because the model easily introduces the error information due to the unsuitable choice of the positive bag.In this paper, we propose a novel object tracking algorithm with multi-classifier fusion based on compressive sensing and multiple instance learning, termed as MCM. In the multi-classifier fusion framework, the MCM algorithm firstly generates the different feature vectors of instances with the different random projection matrices based on the compressive sensing technology. Then, an improved online multiple instance learning mechanism, i.e., SMILE (Relative Similarity Based Online Multiple Instance Learning), is adopted, which further selects and weights the instances in the positive bag by introducing the concept of relative similarity. Finally, by fusing the tracking results of multiple strong classifiers, the MCM algorithm can obtain the more discriminative classifier to effectively deal with the dramatic appearance changes in the complex environments.The main contributions of our paper are summarized as follows:(1)
We introduce the multi-classifier fusion framework to object tracking, by utilizing multiple single classifiers to ensemble learning, so as to obtain better tracking results, benefited from the diversity of the multiple discriminative classifiers(2)
We can extract the different low-dimensional feature vectors by using multiple different random projection matrices based on the compressive sensing technology, in order to obtain the more discriminative classifiers in the multi-classifier fusion framework(3)
The MCM algorithm employs an improved SMILE mechanism to filter and weight the instances in the positive bag, so as to avoid the mistake information to be introduced in the tracking processThe rest of this paper is organized as follows. We first review the related work about object tracking in Section2 and then describe the overview of the proposed algorithm in Section 3. Section 4 illustrates the experimental results tested on the challenging video sequences. The conclusion is made in Section 5.


## 2. Related Work

The object tracking methods [6–49] can be divided into offline methods [14, 15] and online methods [6–13, 16–49]. The offline object tracking methods [14, 15] use the object model built before tracking to track the object, and the learned model is not further updated during the tracking process. The shortcoming of the offline methods is that it is difficult to adapt to the drastic changes of the object appearance in the complex environment. On the other hand, the online tracking methods [16–21] can keep updating the object appearance model during the tracking process, so that the dynamic changes of the object appearance can be more robustly dealt with, and the object drifting problem can be avoided to some extent. Therefore, this paper mainly focuses on the online tracking methods. According to the different modelling patterns, the online tracking algorithms can be further divided into generative learning [8, 12, 17, 18, 22–25, 33–38, 41, 42, 44, 45] and discriminative learning [6, 7, 9–11, 16, 19–21, 26–32, 39, 40, 43, 46–49].


### 2.1. Generative Tracking

The online tracking algorithm based on generative learning usually establishes an object appearance model through the object representation technology and then searches for the region most similar to the object appearance model in a new image frame. For example, Havangi [18] performed the MCMC (Markov chain Monte Carlo) motion processing after resampling. However, the required number of probability transfers is large, and the convergence is difficult to achieve. In order to avoid the noise interference and the inaccurate observation value of the Jacobian matrix, Zhou et al. [44] proposed a robust Kalman filter (KF) algorithm with long short-term memory (LSTM) for an image-based visual servo control system. Mei and Ling [23] applied the sparse representation to the object tracking, which uses the reconstruction error of the sparse representation as the weight of the candidate object and selects the candidate object of the maximum weight as the tracking result. An adaptive image processing method [34] is suggested to determine the 3D position and the velocity of moving objects by using a distributed camera array. Marata et al. [35] proposed the Separate Monte Carlo Mean (SMC-MEAN), which is applied to an autonomous object tracking problem in both Gaussian and non-Gaussian scenarios. Nam et al. [32] proposed the convolutional neural networks (CNNs) algorithm to represent the appearance of the object, which constructs a tree structure based on multiple CNNs and maintains the reliability of the model by smoothly updating along the tree path. Lu et al. [33] presented a transform-aware attentive tracking framework, which uses a deep attentive network to directly predict the target states via spatial transform parameters. Pan et al. [37] proposed a robust object tracking framework, based on a canonical CF tracker. Specifically, an adaptive model update strategy is proposed to evaluate this model with a siamese network. Liu et al. [41] used the spatial feature and the temporal feature which are fused into the deep residual network model with the multi-scale feature vector, so that they can obtain the deep multi-scale spatiotemporal feature model. Zhou et al. [45] proposed a novel object tracking method with the fusion of the extended Kalman particle filter (EKPF) and the least squares support vector regression (LSSVR) to effectively improve the accuracy. Zhang et al. [42] put forward a multi-feature integration framework, including the gray features, histogram of gradient (HOG), color-naming (CN), and illumination invariant features (IIF), in order to overcome the problem of poor representation of single feature in a complex image sequence.Although many online generative tracking methods have been proposed, there are still several issues to be solved. Firstly, the online appearance model is built which is largely dependent on a number of training samples, for example, deep learning-based trackers [32, 33, 37, 41]. If there are only a few training samples at the beginning of tracking, the appearance model is hard to be robust when the appearance of the object changes significantly in the process of tracking. Secondly, when many samples around the current object location are used to build an object model, the error background information is potentially introduced. Thirdly, the generative algorithms usually only use the object information, without considering the helpful background.


### 2.2. Discriminative Tracking

The online tracking algorithm based on discriminative learning learns a classifier to distinguish the object sample from the background. Collins et al. [26] embedded an online multi-feature selection mechanism in the mean shift tracking method, and the features with high discrimination are selected. This method easily suffers from the object drifting, when the object is similar to the background. Huang et al. [43] proposed the MD-JITS (multiple detection joint integrated track splitting) tracker, which is used to the multiple-detection multi-target tracking in cluttered environments. Grabner et al. [28] proposed a well-known online boosting framework for real-time object tracking, which uses only one positive sample in the object region and several negative samples from the background to update the classifier. However, the tracker may not keep very accurately tracking the object, and thus, some possibly mispositioned samples are easily introduced to update the appearance model.To guard against selecting the incorrect samples to update the classifier, Grabner et al. [20] proposed an online semi-supervised boosting method, where the samples at the first frame are labelled and the samples from the other frames are unlabelled. Chen et al. [46] proposed a novel visual tracking algorithm via online semi-supervised co-boosting, and the algorithm has a good ability to recover from drifting. Babenko et al. [6] introduced the multiple instance learning (MIL) paradigm to online object tracking by using the positive and negative bags, where some samples approaching the object location are added to a positive bag. A large number of samples far from the object location are taken to form a negative bag. In order to solve the unselective treatment of instances in the positive bag during the MIL tracking process and the lack of prior knowledge of the object in instance modelling, Chen et al. [47] proposed an online multiple instance boosting algorithm with instance-level semi-supervised learning, termed SemiMILBoost, to achieve more robust object tracking. Zhang and Song [16] proposed an improved online weighted MIL tracker (WMIL), where the instances are weighted by the distance from the object location. Zhou et al. [48] improved the MIL tracking algorithm by optimizing a bag Fisher information function and integrating the co-training criterion. And, Zhou et al. [49] also proposed a multiple instance learning (MIL) tracking method based on a semi-supervised learning model with Fisher linear discriminant. However, in the above methods [6, 16, 47, 48, 49], the positive bag may contain some negative samples because the radius of the positive bag is difficult to be very accurately selected. Therefore, when all the instances in the positive bag are used to update the classifiers, the above algorithms might suffer from the drifting problem in the complex environments.


## 2.1. Generative Tracking

The online tracking algorithm based on generative learning usually establishes an object appearance model through the object representation technology and then searches for the region most similar to the object appearance model in a new image frame. For example, Havangi [18] performed the MCMC (Markov chain Monte Carlo) motion processing after resampling. However, the required number of probability transfers is large, and the convergence is difficult to achieve. In order to avoid the noise interference and the inaccurate observation value of the Jacobian matrix, Zhou et al. [44] proposed a robust Kalman filter (KF) algorithm with long short-term memory (LSTM) for an image-based visual servo control system. Mei and Ling [23] applied the sparse representation to the object tracking, which uses the reconstruction error of the sparse representation as the weight of the candidate object and selects the candidate object of the maximum weight as the tracking result. An adaptive image processing method [34] is suggested to determine the 3D position and the velocity of moving objects by using a distributed camera array. Marata et al. [35] proposed the Separate Monte Carlo Mean (SMC-MEAN), which is applied to an autonomous object tracking problem in both Gaussian and non-Gaussian scenarios. Nam et al. [32] proposed the convolutional neural networks (CNNs) algorithm to represent the appearance of the object, which constructs a tree structure based on multiple CNNs and maintains the reliability of the model by smoothly updating along the tree path. Lu et al. [33] presented a transform-aware attentive tracking framework, which uses a deep attentive network to directly predict the target states via spatial transform parameters. Pan et al. [37] proposed a robust object tracking framework, based on a canonical CF tracker. Specifically, an adaptive model update strategy is proposed to evaluate this model with a siamese network. Liu et al. [41] used the spatial feature and the temporal feature which are fused into the deep residual network model with the multi-scale feature vector, so that they can obtain the deep multi-scale spatiotemporal feature model. Zhou et al. [45] proposed a novel object tracking method with the fusion of the extended Kalman particle filter (EKPF) and the least squares support vector regression (LSSVR) to effectively improve the accuracy. Zhang et al. [42] put forward a multi-feature integration framework, including the gray features, histogram of gradient (HOG), color-naming (CN), and illumination invariant features (IIF), in order to overcome the problem of poor representation of single feature in a complex image sequence.Although many online generative tracking methods have been proposed, there are still several issues to be solved. Firstly, the online appearance model is built which is largely dependent on a number of training samples, for example, deep learning-based trackers [32, 33, 37, 41]. If there are only a few training samples at the beginning of tracking, the appearance model is hard to be robust when the appearance of the object changes significantly in the process of tracking. Secondly, when many samples around the current object location are used to build an object model, the error background information is potentially introduced. Thirdly, the generative algorithms usually only use the object information, without considering the helpful background.


## 2.2. Discriminative Tracking

The online tracking algorithm based on discriminative learning learns a classifier to distinguish the object sample from the background. Collins et al. [26] embedded an online multi-feature selection mechanism in the mean shift tracking method, and the features with high discrimination are selected. This method easily suffers from the object drifting, when the object is similar to the background. Huang et al. [43] proposed the MD-JITS (multiple detection joint integrated track splitting) tracker, which is used to the multiple-detection multi-target tracking in cluttered environments. Grabner et al. [28] proposed a well-known online boosting framework for real-time object tracking, which uses only one positive sample in the object region and several negative samples from the background to update the classifier. However, the tracker may not keep very accurately tracking the object, and thus, some possibly mispositioned samples are easily introduced to update the appearance model.To guard against selecting the incorrect samples to update the classifier, Grabner et al. [20] proposed an online semi-supervised boosting method, where the samples at the first frame are labelled and the samples from the other frames are unlabelled. Chen et al. [46] proposed a novel visual tracking algorithm via online semi-supervised co-boosting, and the algorithm has a good ability to recover from drifting. Babenko et al. [6] introduced the multiple instance learning (MIL) paradigm to online object tracking by using the positive and negative bags, where some samples approaching the object location are added to a positive bag. A large number of samples far from the object location are taken to form a negative bag. In order to solve the unselective treatment of instances in the positive bag during the MIL tracking process and the lack of prior knowledge of the object in instance modelling, Chen et al. [47] proposed an online multiple instance boosting algorithm with instance-level semi-supervised learning, termed SemiMILBoost, to achieve more robust object tracking. Zhang and Song [16] proposed an improved online weighted MIL tracker (WMIL), where the instances are weighted by the distance from the object location. Zhou et al. [48] improved the MIL tracking algorithm by optimizing a bag Fisher information function and integrating the co-training criterion. And, Zhou et al. [49] also proposed a multiple instance learning (MIL) tracking method based on a semi-supervised learning model with Fisher linear discriminant. However, in the above methods [6, 16, 47, 48, 49], the positive bag may contain some negative samples because the radius of the positive bag is difficult to be very accurately selected. Therefore, when all the instances in the positive bag are used to update the classifiers, the above algorithms might suffer from the drifting problem in the complex environments.


## 3. The MCM Algorithm

### 3.1. Overview

The MCM algorithm first uses the different random projection matrices to reduce the dimensions of instances in the positive and negative bags, respectively, thus obtaining the different compressed feature vectors. In the MCM algorithm, the SMILE mechanism is presented to improve the traditional online multiple instance learning by introducing the relative similarity metric. After the SMILE mechanism, the final strong classifier is obtained by the multi-classifier fusion framework. The whole flow of the MCM algorithm is shown in Figure1.Figure 1
The algorithm flow.During the tracking process, the original multi-scale image feature vectors (i.e., Haar-like features are used here) of the instances in the positive and negative bags are firstly obtained at thei-th frame. As shown in Figure 1, the feature vector in the green row represents an instance in the positive bag, and the feature vector in the red row represents the sample in the negative bag. To reduce the time complexity and improve the robustness of object tracking, we use the different random projection matrices to compress the high-dimensional multi-scale image feature vectors into the different low-dimensional feature vectors. Then, by exploiting the improved SMILE mechanism, the compressed Haar-like feature vectors are, respectively, used to update and select the best weak classifiers to obtain the G different strong classifiers H1,H2,…,HG. Then, the G different strong classifiers are combined into a final strong classifier by a multi-classifier fusion framework. Thereby, the object location with the largest response value is found by the combined final strong classifier in the search area of the (i + 1)-th frame.In the next sections, we will further elaborate compressive sensing, the SMILE mechanism, and the multi-classifier fusion framework.


### 3.2. Compressive Sensing

To avoid the high computational complexity and large storage consumption, compressive sensing is employed to object tracking [7, 17]. The compressive sensing technology [50–52] satisfies the condition of RIP (Restricted Isometry Property) [53] and uses the random projection matrix to reduce the high-dimensional multi-scale image features to the low-dimensional image domain in real time. The random projection matrix satisfying the Johnson–Lindenstrauss (J-L) lemma is proved to hold true for the RIP condition in compressive sensing [54].Given thatR∈ℝn×m (n≪m) denotes a random projection matrix, where the n and m, respectively, denote the row number and column number of R. Then, the high-dimensional spatial image feature vector x∈ℝm can be compressed into the low-dimensional image feature vector v∈ℝn, expressed as v=Rx. As long as the matrix R satisfies the J-L lemma, x can be reconstructed from v with a low error probability. Similar to topology reconstruction, the compressed feature v still retains most of the original information [7].At present, there are the four main types of random projection matrices suitable for the J-L lemma as follows, and the reduced low-dimensional image feature vector can preserve basically the information of the original image feature vector [7, 17, 50–52].(1)
Plus-Minus-One (PMO) or Bernoulli random projection [50]: R=1/mrij is represented by n×m matrices, where rij belongs to the set −1,1, with the probability Probrij=1=Probrij=−1=1/2 (rij is Bernoulli’s random variable).(2)
Achlioptas random projection [51]: R=1/mrij is represented by n×m matrices, where rij belongs to the set −3,0,3, and the probability is Probrij=0=2/3, and Probrij=3=Probrij=−3=1/6.(3)
Normal distribution random projection is also called Gaussian distribution random projection [17, 52]. According to J-L lemma, the random projection is represented by a n×m matrix R=1/mrij, where rij is the Gaussian distribution with N0,1.(4)
The CT algorithm [7] uses the following random projection matrix:(1)rij=F×1with probability12F,0with probability1−1F,−1with probability12F.The verification result in [55] shows that when F is 2 or 3 in equation (1), R satisfies J-L lemma.The traditional CT algorithm [7] only uses a fixed random projection matrix in the whole process of tracking, and thus, the helpful information might be omitted, especially in the complex environments. To avoid the above problem, we randomly select a sparse random projection matrix R from the above four main types of random projection matrices and thus obtain the different compressed feature vectors for each instance, which is inputted to the following online SMILE mechanism.


### 3.3. Online SMILE Mechanism

The workflow of the developed online SMILE mechanism is shown in Figure2. In order to avoid the object drifting problem caused by the accumulated errors, we can filter some samples more likely to be negative in the positive bag by calculating the relative similarity of all instances in the positive bag. Then, the filtered positive bag and the negative bag are used to update all the weak classifiers. At the same time, the relative similarity is also employed to weight the objective function and then the best weak classifiers are selected to integrate a strong classifier from the updated weak classifier pool.Figure 2
The SMILE mechanism.In the multiple instance learning paradigm, the training data areX0,y0,…,XN−1,yN−1, where the bag Xi=xi1,…,xij,…,xiNi; yi is the bag label; xij is the j-th instance in the i-th bag; Ni is the number of instances in the i-th bag; and N is the total number of bags. The bag label is defined as yi=maxyij, where yij is the label of the j-th instance in the i-th bag, but the instance label is unknown during MIL training. In the MIL, it is known that a positive bag contains at least one positive instance, and all the instances in a negative bag are negative. In this paper, we select a set of image patches X1=x:dx−dt∗<γ,γ<s as a positive bag, where ⋅ represents the Euclidean distance; γ is the radius from the center point (in pixels); x is an image patch; dx indicates the location of the image patch x; dt∗ represents the location of the object at time t; and s is the search radius of the tracker. For the negative bag, we select the image patches from the annular region X0=x:γ<dx−dt∗<β. Because a large set of potential samples are generated, we select a random subset of these samples as a negative bag as in [1].In most cases, there are some negative samples in the positive bag. However, the traditional online multiple instance learning algorithms [1] treat all the instances in the positive bag as positive in the process of updating all the weak classifiers, which will degrade the performance of the tracker. In order to improve the robustness and accuracy of object tracking, the SMILE mechanism introduces the concept of relative similarity [44] to further filter the instances in the positive bag, and then, the relative similarity is also used to weight the instance probability and then optimize the objective function.First, in the SMILE mechanism, the similarity between the two instancesxij and xiq is defined as follows:(2)Sxij,xiq=0.5NCCxij,xiq+1,where NCC is the normalized correlation coefficient.For any given instancexij in a bag, we construct an object model E=x11,…,x1N+,x01,…,x0N− to represent the object and the background information observed so far. In this paper, x11,…,x1N+ are the object samples from the first frame to the previous frame. And x01,…,x0N− are the instances in the negative bag in the previous frame. Given an arbitrary instance xij and the object model E. The similarity measures for the MIL problem are defined as follows:① Similarity with the positive nearest neighbour:(3)S+xij,E=maxx1n∈ESxij,x1n.② Similarity with the negative nearest neighbour:(4)S−xij,E=maxx0n∈ESxij,x0n.③ Relative similarity:(5)Srxij,E=S+xij,ES+xij,E+S−xij,E.The relative similarity varies from 0 to 1, and the largerSr indicates that the image patch is more likely to be the object. Given the positive relative similarity coefficient θ, and when Srxij,E<θ, the image patch xij is removed from the positive bag, and otherwise, it is retained in the positive bag.In addition, the SMILE mechanism also uses the relative similarity to weight the instance probability, in order to more accurately estimate the bag probabilitypyi|Xi. The probability of the positive bag to be positive is calculated as follows:(6)py1=1X1=∑j=1N1w1jpy1=1x1j.Different from the WMIL algorithm [10], our algorithm uses the relative similarity to weight the instances, and the weight of each instance in the positive bag is calculated as w1j=1/cSrx1j,E, where c is a normalization constant.The negative bag probability to be negative is calculated as follows:(7)py0=0X0=∑j=1N0w01−py0=1x0j.Since all negative instances are far from the object,w0 is set to be a constant [10].As shown in Figure1, we compress the original high-dimensional multi-scale Haar-like feature vector to the low-dimensional feature vector by the random projection matrix in Section 3.2, a very sparse measurement matrix, in order to extract a small number of effective Haar-like features [7]. In the SMILE mechanism, each weak classifier hk corresponding to a Haar-like feature fk in the compressed feature vectors is constructed based on the Bayesian theorem, as follows:(8)hkxij=lnpfkxijyi=1pfkxijyi=0,where the conditional distributions are modelled as a Gaussian function, i.e., pfkxij|yi=1∼Nμ1,σ1 and pfkxij|yi=0∼Nμ0,σ0. The four parameters μ1,σ1,μ0,σ0 are updated with a learning rate parameter by the same schemes as the WMIL algorithm [10].Then, the MCM algorithm selects the best weak classifiers by maximizing the log-likelihood function of bagsL=∑ilogpyi|Xi. After the K iterations, the K best weak classifiers are integrated into a strong classifier Hg. In the multi-classifier fusion framework, we obtain the different G strong classifiers {H1,H2,…,HG}.


### 3.4. Multi-classifier Fusion Framework

In order to fuse the differentG strong classifiers {H1,H2,…,HG}, the final strong classifier Hfinal=Hg∗ is calculated as follows:(9)g∗=argminHg,g∈G−1G∑g=1GHg.In equation (9), the g represents the g-th strong classifier in the total G strong classifiers. The 1/G∑g=1GHg represents the mean response value of all G strong classifiers, and g∗ is the subscript of the selected strong classifier. Thereby, the final strong classifier Hfinal is used to find the location of the sample with the largest response value in the search area as the tracking object. The pseudocode of the MCM algorithm is summarized in Algorithm 1.Algorithm 1: Outline of our algorithm.
Input: t-th video frame.
Initialization: strong classifier H=H0=0.
(1)
Select a set of image patchesχs=x:dx−dt∗<s and calculate their feature vectors;
(2)
Use the final strong classifier to estimatepy|x for x∈Xs;
(3)
Update the object locationdt∗=dargmaxx∈χspy|x of the current frame and the object model E;
(4)
Select a positive bagX1=x:dx−dt∗<γ and a negative bag X0=x:γ<dx−dt∗<β;
(5)
Initialize the iteration numberk = 1;
(6)
Randomly choose a random projection matrixR∈ℝn×m, and calculate the low-dimensional feature vector of each instance in the positive and negative bags;
(7)
Normalize each instance in the positive bag, and calculate the similarity of any two instancesx1j and x1q in equation (2);
(8)
CalculateS+x1j,E in equation (3) and S−x1j,E in equation (4) for each instance in the positive bag;
(9)
Calculate the relative similaritySrx1j,E in equation (5) for each instance in the positive bag;
(10)
IfSrx1j,E<θ, the image patch x1j is deleted from the original positive bag;
(11)
Update all the weak classifiers with the instances by using the filtered positive bag and the negative bag;
(12)
Calculate the weightw1j of each instance in the positive bag;
(13)
Calculate the instance probabilitypyi=1|xij=σHk−1xij=1/1+e−Hk−1xij;
(14)
Calculate the weighted bag probabilitypyi|Xi according to equations (6) and (7);
(15)
Calculate the log-likelihood functionL=∑ilogpyi|Xi, and select the best weak classifier hk∗=argmaxLHk−1+h;
(16)
Repeat Steps 13–15 forK iterations. The k best weak classifier is integrated into the current strong classifier Hk=Hk−1+hk∗;
(17)
Obtain theg‐th strong classifier Hg=HK;
(18)
Repeat Steps 5–17 forG iterations to generate the G strong classifier;
(19)
Obtain the final strong classifierHg∗ by equation (9).
Output: the final strong classifier Hfinal=Hg∗.


## 3.1. Overview

The MCM algorithm first uses the different random projection matrices to reduce the dimensions of instances in the positive and negative bags, respectively, thus obtaining the different compressed feature vectors. In the MCM algorithm, the SMILE mechanism is presented to improve the traditional online multiple instance learning by introducing the relative similarity metric. After the SMILE mechanism, the final strong classifier is obtained by the multi-classifier fusion framework. The whole flow of the MCM algorithm is shown in Figure1.Figure 1
The algorithm flow.During the tracking process, the original multi-scale image feature vectors (i.e., Haar-like features are used here) of the instances in the positive and negative bags are firstly obtained at thei-th frame. As shown in Figure 1, the feature vector in the green row represents an instance in the positive bag, and the feature vector in the red row represents the sample in the negative bag. To reduce the time complexity and improve the robustness of object tracking, we use the different random projection matrices to compress the high-dimensional multi-scale image feature vectors into the different low-dimensional feature vectors. Then, by exploiting the improved SMILE mechanism, the compressed Haar-like feature vectors are, respectively, used to update and select the best weak classifiers to obtain the G different strong classifiers H1,H2,…,HG. Then, the G different strong classifiers are combined into a final strong classifier by a multi-classifier fusion framework. Thereby, the object location with the largest response value is found by the combined final strong classifier in the search area of the (i + 1)-th frame.In the next sections, we will further elaborate compressive sensing, the SMILE mechanism, and the multi-classifier fusion framework.


## 3.2. Compressive Sensing

To avoid the high computational complexity and large storage consumption, compressive sensing is employed to object tracking [7, 17]. The compressive sensing technology [50–52] satisfies the condition of RIP (Restricted Isometry Property) [53] and uses the random projection matrix to reduce the high-dimensional multi-scale image features to the low-dimensional image domain in real time. The random projection matrix satisfying the Johnson–Lindenstrauss (J-L) lemma is proved to hold true for the RIP condition in compressive sensing [54].Given thatR∈ℝn×m (n≪m) denotes a random projection matrix, where the n and m, respectively, denote the row number and column number of R. Then, the high-dimensional spatial image feature vector x∈ℝm can be compressed into the low-dimensional image feature vector v∈ℝn, expressed as v=Rx. As long as the matrix R satisfies the J-L lemma, x can be reconstructed from v with a low error probability. Similar to topology reconstruction, the compressed feature v still retains most of the original information [7].At present, there are the four main types of random projection matrices suitable for the J-L lemma as follows, and the reduced low-dimensional image feature vector can preserve basically the information of the original image feature vector [7, 17, 50–52].(1)
Plus-Minus-One (PMO) or Bernoulli random projection [50]: R=1/mrij is represented by n×m matrices, where rij belongs to the set −1,1, with the probability Probrij=1=Probrij=−1=1/2 (rij is Bernoulli’s random variable).(2)
Achlioptas random projection [51]: R=1/mrij is represented by n×m matrices, where rij belongs to the set −3,0,3, and the probability is Probrij=0=2/3, and Probrij=3=Probrij=−3=1/6.(3)
Normal distribution random projection is also called Gaussian distribution random projection [17, 52]. According to J-L lemma, the random projection is represented by a n×m matrix R=1/mrij, where rij is the Gaussian distribution with N0,1.(4)
The CT algorithm [7] uses the following random projection matrix:(1)rij=F×1with probability12F,0with probability1−1F,−1with probability12F.The verification result in [55] shows that when F is 2 or 3 in equation (1), R satisfies J-L lemma.The traditional CT algorithm [7] only uses a fixed random projection matrix in the whole process of tracking, and thus, the helpful information might be omitted, especially in the complex environments. To avoid the above problem, we randomly select a sparse random projection matrix R from the above four main types of random projection matrices and thus obtain the different compressed feature vectors for each instance, which is inputted to the following online SMILE mechanism.


## 3.3. Online SMILE Mechanism

The workflow of the developed online SMILE mechanism is shown in Figure2. In order to avoid the object drifting problem caused by the accumulated errors, we can filter some samples more likely to be negative in the positive bag by calculating the relative similarity of all instances in the positive bag. Then, the filtered positive bag and the negative bag are used to update all the weak classifiers. At the same time, the relative similarity is also employed to weight the objective function and then the best weak classifiers are selected to integrate a strong classifier from the updated weak classifier pool.Figure 2
The SMILE mechanism.In the multiple instance learning paradigm, the training data areX0,y0,…,XN−1,yN−1, where the bag Xi=xi1,…,xij,…,xiNi; yi is the bag label; xij is the j-th instance in the i-th bag; Ni is the number of instances in the i-th bag; and N is the total number of bags. The bag label is defined as yi=maxyij, where yij is the label of the j-th instance in the i-th bag, but the instance label is unknown during MIL training. In the MIL, it is known that a positive bag contains at least one positive instance, and all the instances in a negative bag are negative. In this paper, we select a set of image patches X1=x:dx−dt∗<γ,γ<s as a positive bag, where ⋅ represents the Euclidean distance; γ is the radius from the center point (in pixels); x is an image patch; dx indicates the location of the image patch x; dt∗ represents the location of the object at time t; and s is the search radius of the tracker. For the negative bag, we select the image patches from the annular region X0=x:γ<dx−dt∗<β. Because a large set of potential samples are generated, we select a random subset of these samples as a negative bag as in [1].In most cases, there are some negative samples in the positive bag. However, the traditional online multiple instance learning algorithms [1] treat all the instances in the positive bag as positive in the process of updating all the weak classifiers, which will degrade the performance of the tracker. In order to improve the robustness and accuracy of object tracking, the SMILE mechanism introduces the concept of relative similarity [44] to further filter the instances in the positive bag, and then, the relative similarity is also used to weight the instance probability and then optimize the objective function.First, in the SMILE mechanism, the similarity between the two instancesxij and xiq is defined as follows:(2)Sxij,xiq=0.5NCCxij,xiq+1,where NCC is the normalized correlation coefficient.For any given instancexij in a bag, we construct an object model E=x11,…,x1N+,x01,…,x0N− to represent the object and the background information observed so far. In this paper, x11,…,x1N+ are the object samples from the first frame to the previous frame. And x01,…,x0N− are the instances in the negative bag in the previous frame. Given an arbitrary instance xij and the object model E. The similarity measures for the MIL problem are defined as follows:① Similarity with the positive nearest neighbour:(3)S+xij,E=maxx1n∈ESxij,x1n.② Similarity with the negative nearest neighbour:(4)S−xij,E=maxx0n∈ESxij,x0n.③ Relative similarity:(5)Srxij,E=S+xij,ES+xij,E+S−xij,E.The relative similarity varies from 0 to 1, and the largerSr indicates that the image patch is more likely to be the object. Given the positive relative similarity coefficient θ, and when Srxij,E<θ, the image patch xij is removed from the positive bag, and otherwise, it is retained in the positive bag.In addition, the SMILE mechanism also uses the relative similarity to weight the instance probability, in order to more accurately estimate the bag probabilitypyi|Xi. The probability of the positive bag to be positive is calculated as follows:(6)py1=1X1=∑j=1N1w1jpy1=1x1j.Different from the WMIL algorithm [10], our algorithm uses the relative similarity to weight the instances, and the weight of each instance in the positive bag is calculated as w1j=1/cSrx1j,E, where c is a normalization constant.The negative bag probability to be negative is calculated as follows:(7)py0=0X0=∑j=1N0w01−py0=1x0j.Since all negative instances are far from the object,w0 is set to be a constant [10].As shown in Figure1, we compress the original high-dimensional multi-scale Haar-like feature vector to the low-dimensional feature vector by the random projection matrix in Section 3.2, a very sparse measurement matrix, in order to extract a small number of effective Haar-like features [7]. In the SMILE mechanism, each weak classifier hk corresponding to a Haar-like feature fk in the compressed feature vectors is constructed based on the Bayesian theorem, as follows:(8)hkxij=lnpfkxijyi=1pfkxijyi=0,where the conditional distributions are modelled as a Gaussian function, i.e., pfkxij|yi=1∼Nμ1,σ1 and pfkxij|yi=0∼Nμ0,σ0. The four parameters μ1,σ1,μ0,σ0 are updated with a learning rate parameter by the same schemes as the WMIL algorithm [10].Then, the MCM algorithm selects the best weak classifiers by maximizing the log-likelihood function of bagsL=∑ilogpyi|Xi. After the K iterations, the K best weak classifiers are integrated into a strong classifier Hg. In the multi-classifier fusion framework, we obtain the different G strong classifiers {H1,H2,…,HG}.


## 3.4. Multi-classifier Fusion Framework

In order to fuse the differentG strong classifiers {H1,H2,…,HG}, the final strong classifier Hfinal=Hg∗ is calculated as follows:(9)g∗=argminHg,g∈G−1G∑g=1GHg.In equation (9), the g represents the g-th strong classifier in the total G strong classifiers. The 1/G∑g=1GHg represents the mean response value of all G strong classifiers, and g∗ is the subscript of the selected strong classifier. Thereby, the final strong classifier Hfinal is used to find the location of the sample with the largest response value in the search area as the tracking object. The pseudocode of the MCM algorithm is summarized in Algorithm 1.Algorithm 1: Outline of our algorithm.
Input: t-th video frame.
Initialization: strong classifier H=H0=0.
(1)
Select a set of image patchesχs=x:dx−dt∗<s and calculate their feature vectors;
(2)
Use the final strong classifier to estimatepy|x for x∈Xs;
(3)
Update the object locationdt∗=dargmaxx∈χspy|x of the current frame and the object model E;
(4)
Select a positive bagX1=x:dx−dt∗<γ and a negative bag X0=x:γ<dx−dt∗<β;
(5)
Initialize the iteration numberk = 1;
(6)
Randomly choose a random projection matrixR∈ℝn×m, and calculate the low-dimensional feature vector of each instance in the positive and negative bags;
(7)
Normalize each instance in the positive bag, and calculate the similarity of any two instancesx1j and x1q in equation (2);
(8)
CalculateS+x1j,E in equation (3) and S−x1j,E in equation (4) for each instance in the positive bag;
(9)
Calculate the relative similaritySrx1j,E in equation (5) for each instance in the positive bag;
(10)
IfSrx1j,E<θ, the image patch x1j is deleted from the original positive bag;
(11)
Update all the weak classifiers with the instances by using the filtered positive bag and the negative bag;
(12)
Calculate the weightw1j of each instance in the positive bag;
(13)
Calculate the instance probabilitypyi=1|xij=σHk−1xij=1/1+e−Hk−1xij;
(14)
Calculate the weighted bag probabilitypyi|Xi according to equations (6) and (7);
(15)
Calculate the log-likelihood functionL=∑ilogpyi|Xi, and select the best weak classifier hk∗=argmaxLHk−1+h;
(16)
Repeat Steps 13–15 forK iterations. The k best weak classifier is integrated into the current strong classifier Hk=Hk−1+hk∗;
(17)
Obtain theg‐th strong classifier Hg=HK;
(18)
Repeat Steps 5–17 forG iterations to generate the G strong classifier;
(19)
Obtain the final strong classifierHg∗ by equation (9).
Output: the final strong classifier Hfinal=Hg∗.


## 4. Experimental Results

In this section, we first introduce the implementation details of the experiments. Secondly, we discuss the suitable number of the strong classifier. Finally, we evaluate the proposed algorithm and several competing algorithms on the two baseline datasets, i.e., OTB-2013 [56] and OTB-2015 [57], from the perspective of quantitative evaluation, time complexity analysis, and qualitative evaluation.


### 4.1. Datasets and Implementation Details

We evaluate the proposed tracking method and the six completing methods on two challenging object tracking datasets, i.e., OTB-2013 and OTB-2015, which contains the 50 and 100 challenging video sequences, respectively. The two datasets face different challenges in complex environments, such as rotation, occlusion, background clutter, and illumination variation. The search radiusγ of the positive bag is 4.0 pixels, and the radii of the annular region of the negative bag are 4.0 pixels and 6.0 pixels, respectively, and the search radius s for searching the new object location is 25 pixels. The number of instances in the positive and negative bags is limited to be 45 and 50, respectively. The relative coefficient θ is set to be 0.7. The size of the sample box is a rectangle of 25 × 25 pixels. The weak classifier pool consists of 150 weak classifiers. The number of selected best weak classifiers is set to be 15. Our experiments are implemented on MATLAB 2016a, OpenCV 2.4.4, and visual studio 2010 with Intel Core i5-6300HQ, 2.30 GHz CPU and 8.0 GB RAM.


### 4.2. Parameter Analysis

We report the precision at an adaptive threshold [6, 13], i.e., 0.5×minw,h when the number of strong classifiers is from 30 to 100 on the OTB-2013 and OTB-2015 datasets. Here, w and h denote the width and height of the object, respectively. This adaptive threshold roughly corresponds to the percentage of frames with at least a 50% overlap between the bounding box and the ground truth.Figure3 shows the precision under the different numbers of strong classifiers on the four video sequences, i.e., Bird1, Box, Crowds, and Soccer on the two datasets. From Figure 3, we can see that the tracking precision increases gradually and then begins to drop with the increase of the number of strong classifiers. When the number of strong classifiers is in [55, 85], the tracking precision is the best for most of the video sequences.Figure 3
Precision at the different numbers of strong classifiers for our algorithm on the four video sequences. (a) Bird1, (b) Box, (c) Suv, and (d) Soccer.
(a)
(b)
(c)
(d)


### 4.3. Comparisons with Representative Trackers

We evaluated the proposed trackers with six competing trackers, including CSK [11], CT [6], KCF [58], Struck [10], WMIL [16], and TLD [9]. Quantitative evaluation, time complexity analysis, and qualitative evaluation are reported in this section.


#### 4.3.1. Quantitative Evaluation

The center location error and precision are employed to evaluate the experimental results of all the comparison algorithms. The center location error (CLE) is defined as the Euclidean distance between the object center location and the ground truth. The CLE is formally expressed as follows:(10)CLE=1Nf∑i=1Nfdistcenteri,gti.The functiondist.,. is used to calculate Euclidean distance. Nf is the total number of frames; centeri and gti represent the center location given by the tracking algorithm and the ground truth center location in each frame. The smaller the value of CLE is, the lower the error of the tracking algorithm is.In addition, we also report the precision at an adaptive threshold, i.e.,0.5×minw,h. The precision plots show the percentage of the frames whose center location errors are lower than a certain threshold, defined as follows:(11)precisionζ=1N∑i=1Nψdistcenteri,gti≤ζ,where ζ represents the threshold. When dist.,.≤ζ, ψ⋅=1, and otherwise, ψ⋅=0.In the experiments, we compare the center location error and the tracking precision of the proposed algorithm MCM with the competing trackers on the OTB-2013 and OTB-2015 datasets according to the 11 challenging factors, i.e., Illumination Variation (IV), Scale Variation (SV), Occlusion (OCC), Deformation (DEF), Motion Blur (MB), Fast Motion (FM), In-Plane Rotation (IPR), Out-of-Plane Rotation (OPR), Out-of-View (OV), Background Clutters (BC), and Low Resolution (LR).Tables1 and 2 show, respectively, the average center location error (in pixels) for each challenge factor on the OTB-2013 and OTB-2015 datasets. From the last row of Tables 1 and 2, we can see that the MCM algorithm obtains the lowest average center location error compared with the other competing algorithms. In the total of eleven challenge factors, the MCM has the lower center location errors than the other compared methods at the ten challenges on the OTB-2015 dataset and has the lower CLE values at the eight challenges on the OTB-2013 dataset. On the two datasets, the MCM works best in Out-of-View (OV) due to the employed relative similarity based on the object model, which can overcome the dramatic appearance changes to significantly improve the tracking performance for the most challenging environments on the two datasets. Overall, the center location errors of the MCM algorithm with multiple strong classifiers are lower than CT and MIL using the single strong classifier.Table 1
Average center location error (in pixels) for each challenge factor in OTB-2013.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
122.78
140.88
57.42
150.62
94.33
124.23
76.59
SV
450.79
92.21
69.57
118.74
78.92
130.93
57.72
OCC
523.59
103.39
63.44
122.27
79.26
150.62
51.08
DEF
721.04
147.85
55.74
161.01
99.33
186.27
69.20
MB
189.68
103.06
81.99
125.26
83.37
113.97
64.67
FM
204.85
106.38
77.04
119.60
78.23
128.46
54.45
IPR
146.50
101.37
73.84
124.34
79.45
125.53
53.99
OPR
473.08
111.67
69.33
131.17
80.29
155.43
56.17
OV
121.16
71.07
66.26
125.35
69.42
122.82
45.54
BC
661.00
142.34
86.08
161.67
84.86
113.14
61.49
LR
154.15
97.59
50.22
136.84
73.68
131.85
50.31
ALL
377.53
117.94
69.14
131.46
82.66
148.08
55.49
The last row represents the average center location error on all the 50 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Table 2
Average center location error (in pixels) for each challenge factor on the OTB-2015 dataset.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
484.84
89.58
64.37
197.16
205.23
134.66
48.05
SV
379.01
79.47
75.95
180.59
186.45
129.80
42.62
OCC
429.49
78.67
72.02
202.60
192.44
133.90
40.10
DEF
481.93
93.87
80.69
227.46
179.21
142.79
44.60
MB
897.57
86.94
106.29
178.91
179.25
131.01
43.89
FM
147.24
77.35
83.93
176.75
181.55
128.54
41.03
IPR
100.36
72.08
57.11
181.02
187.67
110.84
37.70
OPR
580.14
69.84
62.41
204.64
189.66
120.64
38.10
OV
162.25
87.60
98.38
152.75
208.33
153.18
36.30
BC
481.65
98.96
67.46
186.94
201.99
143.19
45.15
LR
130.46
49.92
55.30
114.25
157.96
104.11
56.71
ALL
388.63
80.39
74.90
182.10
188.16
130.24
43.11
The last row represents the average center location error on all the 100 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Tables3 and 4 list, respectively, the average tracking precision at the threshold of 0.5×minw,h of the competing algorithms under the different challenge factors on the OTB-2013 and OTB-2015 datasets. In the last row of Table 3, the average precision of the MCM algorithm is only lower than KCF by 0.043 on the OTB-2013 dataset. In Scale Variation (SV), Out-of-View (OV), and Low Resolution (LR), the precision of the MCM algorithm is higher than that of the KCF algorithm on the OTB-2013 dataset. In the last row of Table 4, the average precision of the MCM algorithm is 0.553, which is highest among all the comparison algorithms on the OTB-2015 dataset. In Scale Variation (SV), Occlusion (OCC), Deformation (DEF), Out-of-View (OV), and Background Clutters (BC), the precision of the MCM algorithm is higher than that of the other competing algorithms on the OTB-2015 dataset. Overall, by fusing multiple strong classifiers, the precision of the MCM algorithm is higher than that of CT and MIL using the single strong classifier on the two datasets. Therefore, the effectiveness of the multi-classifier fusion of the MCM method is verified.Table 3
Average precision at the threshold of0.5×minw,h for each challenge factor in OTB-2013.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
0.470
0.237
0.630
0.062
0.408
0.375
0.451
SV
0.278
0.303
0.446
0.076
0.344
0.394
0.459
OCC
0.303
0.284
0.550
0.081
0.294
0.376
0.468
DEF
0.300
0.304
0.580
0.069
0.386
0.330
0.450
MB
0.203
0.216
0.408
0.080
0.340
0.436
0.393
FM
0.200
0.237
0.407
0.077
0.339
0.429
0.421
IPR
0.228
0.282
0.502
0.064
0.379
0.432
0.451
OPR
0.309
0.269
0.539
0.080
0.365
0.403
0.461
OV
0.203
0.370
0.375
0.095
0.460
0.485
0.568
BC
0.422
0.264
0.575
0.048
0.418
0.403
0.546
LR
0.291
0.232
0.441
0.070
0.391
0.444
0.498
ALL
0.330
0.284
0.508
0.066
0.371
0.433
0.465
The last row represents the average precision on all the 50 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Table 4
Average precision at the threshold of0.5×minw,h for each challenge factor on the OTB-2015 dataset.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
0.472
0.334
0.685
0.051
0.033
0.532
0.609
SV
0.320
0.426
0.471
0.055
0.086
0.533
0.541
OCC
0.392
0.370
0.550
0.057
0.056
0.522
0.588
DEF
0.376
0.407
0.543
0.049
0.075
0.486
0.582
MB
0.200
0.337
0.365
0.085
0.107
0.529
0.519
FM
0.329
0.379
0.468
0.073
0.073
0.568
0.555
IPR
0.228
0.282
0.502
0.064
0.379
0.432
0.451
OPR
0.460
0.400
0.602
0.041
0.054
0.558
0.582
OV
0.248
0.380
0.313
0.069
0.073
0.459
0.629
BC
0.513
0.379
0.626
0.035
0.042
0.459
0.643
LR
0.387
0.305
0.525
0.076
0.070
0.465
0.386
ALL
0.357
0.364
0.513
0.060
0.095
0.505
0.553
The last row represents the average precision on all the 100 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Figures4 and 5, respectively, show the center location error plots on several representative video sequences on the OTB-2013 and OTB-2015 datasets. In addition, as shown in Figures 4 and 5, we can intuitively see that the MCM tracker yields much lower center location errors for most of the frames than the other six algorithms on the two benchmarks. Figures 6 and 7, respectively, present the precision plots over thresholds in [0, 50] on eight representative sequences on the two datasets. Our algorithm has the higher precision than the other comparison algorithms among the thresholds [0, 50] for most of video sequences. This is because compressive sensing and the improved online SMILE mechanism play an important role in the MCM algorithm. As a whole, the proposed algorithm is more effective to avoid the severe drifting problem than the other competing trackers in the complex environments.Figure 4
Center location error plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2013 dataset. In the upper right corner, the average center location errors of all the frames on each sequence are listed for all the comparison methods. (a) Bird1, (b) Box, (c) Couple, (d) Human6, (e) Matrix, (f) MotorRolling, (g) Walking, and (h) Walking2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)Figure 5
Center location error plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2015 dataset. In the upper right corner, the average center location errors of all the frames on each sequence are listed for all the comparison methods. (a) ClifBar, (b) Coupon, (c) Crossing, (d) DragonBaby, (e) Human5, (f) Singer1, (g) Tiger2, and (h) Trans.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)Figure 6
Precision plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2013 dataset. In the upper right corner, the precision at the threshold0.5×minw,h of all the comparison methods is listed. (a) Bird1, (b) Couple, (c) Box, (d) Human3, (e) Panda, (f) Matrix, (g) Walking, and (h) Walking2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)Figure 7
Precision plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2015 dataset. In the upper right corner, the precision at the threshold0.5×minw,h of all the comparison methods is listed. (a) ClifBar, (b) Coupon, (c) Crossing, (d) DragonBaby, (e) Human 5, (f) Singer1, (g) Tiger2, and (h) Trans.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)


#### 4.3.2. Time Complexity Analysis

We use the number of frames per second (FPS) to evaluate the time complexity of the algorithm. The larger the FPS is, the shorter the running time of the algorithm is, and the lower the time complexity of the algorithm is. Table5 lists the average FPS values of different algorithms on the 100 challenging video sequences of the OTB-2015 dataset. As shown in Table 5, the tracking speed of the MCM algorithm is around 12 frames per second, and the MCM has the faster tracking speed than the Struck, WMIL, and TLD methods. Although it has the lower tracking speed than the CSK, KCF, and CT algorithms, the MCM algorithm obtains the lower center location errors and the higher precision than the three compared algorithms, due to the multi-classifier fusion framework employed in this paper.Table 5
The average number of frames per second (FPS) for the state-of-the-art trackers and our algorithm on the 100 challenging video sequences of the OTB-2015 dataset.
Methods
CSK
CT
KCF
Struck
WMIL
TLD
MCM
FPS
342
64
192
8
10
4
12


#### 4.3.3. Qualitative Evaluation

Figure8 summarizes the qualitative evaluation of the proposed algorithm and the six competing trackers under seven representative challenge sequences. In the Basketball video sequence (see Figure 8(a)), there are three kinds of challenging factors, i.e., fast motion, scale variation, and occlusion. In the #17 frame, the Struck algorithm loses the object, and the KCF and WMIL methods are partially deviated from the object when the occlusion occurs. When the object moves fast and the scale variations happen, the KCF algorithm drifts heavily (see the #63 frame), and the CT and TLD algorithms also suffer from drifting (see the #247 frame). The CSK algorithm gradually deviates from the object and then the object is lost from the #489 frame. The CT algorithm also has a severe offset in the subsequent #629 frame. The proposed MCM algorithm performs well in most frames, where the object can be accurately tracked in the face of rapid movement and scale variation.Figure 8
Tracking results of CSK, CT, KCF, Struck, WMIL, TLD, and the proposed MCM algorithm on seven image sequences on the OTB-2013 and OTB-2015 datasets. From top to bottom, (a) Basketball, (b) Bird 1, (c) BlurBody, (d) Box, (e) Panda, (f) Singer2, and (g) Walking2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)There are three tracking difficulties in the Bird1 sequence (see Figure8(b)), i.e., fast motion of bird wings, long-term object occlusion, and rapid scale variation. In the initial fast motion, CSK loses the object firstly (see the #17 frame), and then the KCF and Struck algorithms drift severely (see the #35 frame). After long-term occlusion of cloud, only the MCM algorithm still tracks the object and the other algorithms are severely deviated from the ground truth (see the #187 frame). For the subsequent scale variations, the MCM algorithm still keeps tracking the object well (see the #266 and #403 frames).In the BlurBody sequence (see Figure8(c)), because the camera always shakes, some frames become blurred, which will have a great impact on the object tracking. During the tracking process, most algorithms have serious large-scale drifting, but the MCM algorithm is better to adapt to this situation.In the Box video sequence (see Figure8(d)), due to the occlusion problem in the moving process of the box, the CSK, TLD, and Struck algorithms drift successively, and then the object is lost (see the #73, #354, and #429 frames). The CT and WMIL and KCF algorithms miss the target in the subsequent frames with scale variations and rotations (see the #552 and #737 frames). However, the MCM algorithm has excellent performance in face of these challenge factors and can accurately track the target.In the Panda sequence (see Figure8(e)), a small Panda suffers from fast movements and occlusions. The Struck, CSK, and KCF algorithms have lost the object gradually, and the TLD algorithm fails due to scale variations during the tracking process (see the #160 and #609 frames), and the other algorithms are more accurate and stable to track the object.A singer in the Singer2 sequence (see Figure8(f)) has the large-scale variations and rotational deformations during the tracking process. In the scale variations of the object, the TLD, Struck, CSK, and CT algorithms work poor (see the #40, #107, and #269 frames), while the other algorithms can accurately track the target. In the movements of the singer, the KCF and WMIL algorithms are partially drifting from the object, and the MCM algorithm tracks the target more accurately during the tracking process.In the Walking2 dataset (see Figure8(g)), the main challenging factor is the occlusions of two persons. After the first occlusion, the Struck, TLD, CT, and WMIL algorithms drift to track the error object (see the #243 frame). After the second occlusion, the CSK method also begins to have a large offset (see the #389 frame). When the target becomes smaller and the scale variation occurs, the MCM algorithm is able to track the target accurately, and yet the other algorithms partially depart from the object (see the #412 frame).In summary, the MCM algorithm outperforms the other compared trackers and can be more accurate and robust to track objects in the challenging factors of rapid motions, scale variations and object occlusions, and so on.


## 4.1. Datasets and Implementation Details

We evaluate the proposed tracking method and the six completing methods on two challenging object tracking datasets, i.e., OTB-2013 and OTB-2015, which contains the 50 and 100 challenging video sequences, respectively. The two datasets face different challenges in complex environments, such as rotation, occlusion, background clutter, and illumination variation. The search radiusγ of the positive bag is 4.0 pixels, and the radii of the annular region of the negative bag are 4.0 pixels and 6.0 pixels, respectively, and the search radius s for searching the new object location is 25 pixels. The number of instances in the positive and negative bags is limited to be 45 and 50, respectively. The relative coefficient θ is set to be 0.7. The size of the sample box is a rectangle of 25 × 25 pixels. The weak classifier pool consists of 150 weak classifiers. The number of selected best weak classifiers is set to be 15. Our experiments are implemented on MATLAB 2016a, OpenCV 2.4.4, and visual studio 2010 with Intel Core i5-6300HQ, 2.30 GHz CPU and 8.0 GB RAM.


## 4.2. Parameter Analysis

We report the precision at an adaptive threshold [6, 13], i.e., 0.5×minw,h when the number of strong classifiers is from 30 to 100 on the OTB-2013 and OTB-2015 datasets. Here, w and h denote the width and height of the object, respectively. This adaptive threshold roughly corresponds to the percentage of frames with at least a 50% overlap between the bounding box and the ground truth.Figure3 shows the precision under the different numbers of strong classifiers on the four video sequences, i.e., Bird1, Box, Crowds, and Soccer on the two datasets. From Figure 3, we can see that the tracking precision increases gradually and then begins to drop with the increase of the number of strong classifiers. When the number of strong classifiers is in [55, 85], the tracking precision is the best for most of the video sequences.Figure 3
Precision at the different numbers of strong classifiers for our algorithm on the four video sequences. (a) Bird1, (b) Box, (c) Suv, and (d) Soccer.
(a)
(b)
(c)
(d)


## 4.3. Comparisons with Representative Trackers

We evaluated the proposed trackers with six competing trackers, including CSK [11], CT [6], KCF [58], Struck [10], WMIL [16], and TLD [9]. Quantitative evaluation, time complexity analysis, and qualitative evaluation are reported in this section.


### 4.3.1. Quantitative Evaluation

The center location error and precision are employed to evaluate the experimental results of all the comparison algorithms. The center location error (CLE) is defined as the Euclidean distance between the object center location and the ground truth. The CLE is formally expressed as follows:(10)CLE=1Nf∑i=1Nfdistcenteri,gti.The functiondist.,. is used to calculate Euclidean distance. Nf is the total number of frames; centeri and gti represent the center location given by the tracking algorithm and the ground truth center location in each frame. The smaller the value of CLE is, the lower the error of the tracking algorithm is.In addition, we also report the precision at an adaptive threshold, i.e.,0.5×minw,h. The precision plots show the percentage of the frames whose center location errors are lower than a certain threshold, defined as follows:(11)precisionζ=1N∑i=1Nψdistcenteri,gti≤ζ,where ζ represents the threshold. When dist.,.≤ζ, ψ⋅=1, and otherwise, ψ⋅=0.In the experiments, we compare the center location error and the tracking precision of the proposed algorithm MCM with the competing trackers on the OTB-2013 and OTB-2015 datasets according to the 11 challenging factors, i.e., Illumination Variation (IV), Scale Variation (SV), Occlusion (OCC), Deformation (DEF), Motion Blur (MB), Fast Motion (FM), In-Plane Rotation (IPR), Out-of-Plane Rotation (OPR), Out-of-View (OV), Background Clutters (BC), and Low Resolution (LR).Tables1 and 2 show, respectively, the average center location error (in pixels) for each challenge factor on the OTB-2013 and OTB-2015 datasets. From the last row of Tables 1 and 2, we can see that the MCM algorithm obtains the lowest average center location error compared with the other competing algorithms. In the total of eleven challenge factors, the MCM has the lower center location errors than the other compared methods at the ten challenges on the OTB-2015 dataset and has the lower CLE values at the eight challenges on the OTB-2013 dataset. On the two datasets, the MCM works best in Out-of-View (OV) due to the employed relative similarity based on the object model, which can overcome the dramatic appearance changes to significantly improve the tracking performance for the most challenging environments on the two datasets. Overall, the center location errors of the MCM algorithm with multiple strong classifiers are lower than CT and MIL using the single strong classifier.Table 1
Average center location error (in pixels) for each challenge factor in OTB-2013.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
122.78
140.88
57.42
150.62
94.33
124.23
76.59
SV
450.79
92.21
69.57
118.74
78.92
130.93
57.72
OCC
523.59
103.39
63.44
122.27
79.26
150.62
51.08
DEF
721.04
147.85
55.74
161.01
99.33
186.27
69.20
MB
189.68
103.06
81.99
125.26
83.37
113.97
64.67
FM
204.85
106.38
77.04
119.60
78.23
128.46
54.45
IPR
146.50
101.37
73.84
124.34
79.45
125.53
53.99
OPR
473.08
111.67
69.33
131.17
80.29
155.43
56.17
OV
121.16
71.07
66.26
125.35
69.42
122.82
45.54
BC
661.00
142.34
86.08
161.67
84.86
113.14
61.49
LR
154.15
97.59
50.22
136.84
73.68
131.85
50.31
ALL
377.53
117.94
69.14
131.46
82.66
148.08
55.49
The last row represents the average center location error on all the 50 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Table 2
Average center location error (in pixels) for each challenge factor on the OTB-2015 dataset.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
484.84
89.58
64.37
197.16
205.23
134.66
48.05
SV
379.01
79.47
75.95
180.59
186.45
129.80
42.62
OCC
429.49
78.67
72.02
202.60
192.44
133.90
40.10
DEF
481.93
93.87
80.69
227.46
179.21
142.79
44.60
MB
897.57
86.94
106.29
178.91
179.25
131.01
43.89
FM
147.24
77.35
83.93
176.75
181.55
128.54
41.03
IPR
100.36
72.08
57.11
181.02
187.67
110.84
37.70
OPR
580.14
69.84
62.41
204.64
189.66
120.64
38.10
OV
162.25
87.60
98.38
152.75
208.33
153.18
36.30
BC
481.65
98.96
67.46
186.94
201.99
143.19
45.15
LR
130.46
49.92
55.30
114.25
157.96
104.11
56.71
ALL
388.63
80.39
74.90
182.10
188.16
130.24
43.11
The last row represents the average center location error on all the 100 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Tables3 and 4 list, respectively, the average tracking precision at the threshold of 0.5×minw,h of the competing algorithms under the different challenge factors on the OTB-2013 and OTB-2015 datasets. In the last row of Table 3, the average precision of the MCM algorithm is only lower than KCF by 0.043 on the OTB-2013 dataset. In Scale Variation (SV), Out-of-View (OV), and Low Resolution (LR), the precision of the MCM algorithm is higher than that of the KCF algorithm on the OTB-2013 dataset. In the last row of Table 4, the average precision of the MCM algorithm is 0.553, which is highest among all the comparison algorithms on the OTB-2015 dataset. In Scale Variation (SV), Occlusion (OCC), Deformation (DEF), Out-of-View (OV), and Background Clutters (BC), the precision of the MCM algorithm is higher than that of the other competing algorithms on the OTB-2015 dataset. Overall, by fusing multiple strong classifiers, the precision of the MCM algorithm is higher than that of CT and MIL using the single strong classifier on the two datasets. Therefore, the effectiveness of the multi-classifier fusion of the MCM method is verified.Table 3
Average precision at the threshold of0.5×minw,h for each challenge factor in OTB-2013.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
0.470
0.237
0.630
0.062
0.408
0.375
0.451
SV
0.278
0.303
0.446
0.076
0.344
0.394
0.459
OCC
0.303
0.284
0.550
0.081
0.294
0.376
0.468
DEF
0.300
0.304
0.580
0.069
0.386
0.330
0.450
MB
0.203
0.216
0.408
0.080
0.340
0.436
0.393
FM
0.200
0.237
0.407
0.077
0.339
0.429
0.421
IPR
0.228
0.282
0.502
0.064
0.379
0.432
0.451
OPR
0.309
0.269
0.539
0.080
0.365
0.403
0.461
OV
0.203
0.370
0.375
0.095
0.460
0.485
0.568
BC
0.422
0.264
0.575
0.048
0.418
0.403
0.546
LR
0.291
0.232
0.441
0.070
0.391
0.444
0.498
ALL
0.330
0.284
0.508
0.066
0.371
0.433
0.465
The last row represents the average precision on all the 50 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Table 4
Average precision at the threshold of0.5×minw,h for each challenge factor on the OTB-2015 dataset.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
0.472
0.334
0.685
0.051
0.033
0.532
0.609
SV
0.320
0.426
0.471
0.055
0.086
0.533
0.541
OCC
0.392
0.370
0.550
0.057
0.056
0.522
0.588
DEF
0.376
0.407
0.543
0.049
0.075
0.486
0.582
MB
0.200
0.337
0.365
0.085
0.107
0.529
0.519
FM
0.329
0.379
0.468
0.073
0.073
0.568
0.555
IPR
0.228
0.282
0.502
0.064
0.379
0.432
0.451
OPR
0.460
0.400
0.602
0.041
0.054
0.558
0.582
OV
0.248
0.380
0.313
0.069
0.073
0.459
0.629
BC
0.513
0.379
0.626
0.035
0.042
0.459
0.643
LR
0.387
0.305
0.525
0.076
0.070
0.465
0.386
ALL
0.357
0.364
0.513
0.060
0.095
0.505
0.553
The last row represents the average precision on all the 100 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Figures4 and 5, respectively, show the center location error plots on several representative video sequences on the OTB-2013 and OTB-2015 datasets. In addition, as shown in Figures 4 and 5, we can intuitively see that the MCM tracker yields much lower center location errors for most of the frames than the other six algorithms on the two benchmarks. Figures 6 and 7, respectively, present the precision plots over thresholds in [0, 50] on eight representative sequences on the two datasets. Our algorithm has the higher precision than the other comparison algorithms among the thresholds [0, 50] for most of video sequences. This is because compressive sensing and the improved online SMILE mechanism play an important role in the MCM algorithm. As a whole, the proposed algorithm is more effective to avoid the severe drifting problem than the other competing trackers in the complex environments.Figure 4
Center location error plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2013 dataset. In the upper right corner, the average center location errors of all the frames on each sequence are listed for all the comparison methods. (a) Bird1, (b) Box, (c) Couple, (d) Human6, (e) Matrix, (f) MotorRolling, (g) Walking, and (h) Walking2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)Figure 5
Center location error plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2015 dataset. In the upper right corner, the average center location errors of all the frames on each sequence are listed for all the comparison methods. (a) ClifBar, (b) Coupon, (c) Crossing, (d) DragonBaby, (e) Human5, (f) Singer1, (g) Tiger2, and (h) Trans.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)Figure 6
Precision plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2013 dataset. In the upper right corner, the precision at the threshold0.5×minw,h of all the comparison methods is listed. (a) Bird1, (b) Couple, (c) Box, (d) Human3, (e) Panda, (f) Matrix, (g) Walking, and (h) Walking2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)Figure 7
Precision plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2015 dataset. In the upper right corner, the precision at the threshold0.5×minw,h of all the comparison methods is listed. (a) ClifBar, (b) Coupon, (c) Crossing, (d) DragonBaby, (e) Human 5, (f) Singer1, (g) Tiger2, and (h) Trans.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)


### 4.3.2. Time Complexity Analysis

We use the number of frames per second (FPS) to evaluate the time complexity of the algorithm. The larger the FPS is, the shorter the running time of the algorithm is, and the lower the time complexity of the algorithm is. Table5 lists the average FPS values of different algorithms on the 100 challenging video sequences of the OTB-2015 dataset. As shown in Table 5, the tracking speed of the MCM algorithm is around 12 frames per second, and the MCM has the faster tracking speed than the Struck, WMIL, and TLD methods. Although it has the lower tracking speed than the CSK, KCF, and CT algorithms, the MCM algorithm obtains the lower center location errors and the higher precision than the three compared algorithms, due to the multi-classifier fusion framework employed in this paper.Table 5
The average number of frames per second (FPS) for the state-of-the-art trackers and our algorithm on the 100 challenging video sequences of the OTB-2015 dataset.
Methods
CSK
CT
KCF
Struck
WMIL
TLD
MCM
FPS
342
64
192
8
10
4
12


### 4.3.3. Qualitative Evaluation

Figure8 summarizes the qualitative evaluation of the proposed algorithm and the six competing trackers under seven representative challenge sequences. In the Basketball video sequence (see Figure 8(a)), there are three kinds of challenging factors, i.e., fast motion, scale variation, and occlusion. In the #17 frame, the Struck algorithm loses the object, and the KCF and WMIL methods are partially deviated from the object when the occlusion occurs. When the object moves fast and the scale variations happen, the KCF algorithm drifts heavily (see the #63 frame), and the CT and TLD algorithms also suffer from drifting (see the #247 frame). The CSK algorithm gradually deviates from the object and then the object is lost from the #489 frame. The CT algorithm also has a severe offset in the subsequent #629 frame. The proposed MCM algorithm performs well in most frames, where the object can be accurately tracked in the face of rapid movement and scale variation.Figure 8
Tracking results of CSK, CT, KCF, Struck, WMIL, TLD, and the proposed MCM algorithm on seven image sequences on the OTB-2013 and OTB-2015 datasets. From top to bottom, (a) Basketball, (b) Bird 1, (c) BlurBody, (d) Box, (e) Panda, (f) Singer2, and (g) Walking2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)There are three tracking difficulties in the Bird1 sequence (see Figure8(b)), i.e., fast motion of bird wings, long-term object occlusion, and rapid scale variation. In the initial fast motion, CSK loses the object firstly (see the #17 frame), and then the KCF and Struck algorithms drift severely (see the #35 frame). After long-term occlusion of cloud, only the MCM algorithm still tracks the object and the other algorithms are severely deviated from the ground truth (see the #187 frame). For the subsequent scale variations, the MCM algorithm still keeps tracking the object well (see the #266 and #403 frames).In the BlurBody sequence (see Figure8(c)), because the camera always shakes, some frames become blurred, which will have a great impact on the object tracking. During the tracking process, most algorithms have serious large-scale drifting, but the MCM algorithm is better to adapt to this situation.In the Box video sequence (see Figure8(d)), due to the occlusion problem in the moving process of the box, the CSK, TLD, and Struck algorithms drift successively, and then the object is lost (see the #73, #354, and #429 frames). The CT and WMIL and KCF algorithms miss the target in the subsequent frames with scale variations and rotations (see the #552 and #737 frames). However, the MCM algorithm has excellent performance in face of these challenge factors and can accurately track the target.In the Panda sequence (see Figure8(e)), a small Panda suffers from fast movements and occlusions. The Struck, CSK, and KCF algorithms have lost the object gradually, and the TLD algorithm fails due to scale variations during the tracking process (see the #160 and #609 frames), and the other algorithms are more accurate and stable to track the object.A singer in the Singer2 sequence (see Figure8(f)) has the large-scale variations and rotational deformations during the tracking process. In the scale variations of the object, the TLD, Struck, CSK, and CT algorithms work poor (see the #40, #107, and #269 frames), while the other algorithms can accurately track the target. In the movements of the singer, the KCF and WMIL algorithms are partially drifting from the object, and the MCM algorithm tracks the target more accurately during the tracking process.In the Walking2 dataset (see Figure8(g)), the main challenging factor is the occlusions of two persons. After the first occlusion, the Struck, TLD, CT, and WMIL algorithms drift to track the error object (see the #243 frame). After the second occlusion, the CSK method also begins to have a large offset (see the #389 frame). When the target becomes smaller and the scale variation occurs, the MCM algorithm is able to track the target accurately, and yet the other algorithms partially depart from the object (see the #412 frame).In summary, the MCM algorithm outperforms the other compared trackers and can be more accurate and robust to track objects in the challenging factors of rapid motions, scale variations and object occlusions, and so on.


## 4.3.1. Quantitative Evaluation

The center location error and precision are employed to evaluate the experimental results of all the comparison algorithms. The center location error (CLE) is defined as the Euclidean distance between the object center location and the ground truth. The CLE is formally expressed as follows:(10)CLE=1Nf∑i=1Nfdistcenteri,gti.The functiondist.,. is used to calculate Euclidean distance. Nf is the total number of frames; centeri and gti represent the center location given by the tracking algorithm and the ground truth center location in each frame. The smaller the value of CLE is, the lower the error of the tracking algorithm is.In addition, we also report the precision at an adaptive threshold, i.e.,0.5×minw,h. The precision plots show the percentage of the frames whose center location errors are lower than a certain threshold, defined as follows:(11)precisionζ=1N∑i=1Nψdistcenteri,gti≤ζ,where ζ represents the threshold. When dist.,.≤ζ, ψ⋅=1, and otherwise, ψ⋅=0.In the experiments, we compare the center location error and the tracking precision of the proposed algorithm MCM with the competing trackers on the OTB-2013 and OTB-2015 datasets according to the 11 challenging factors, i.e., Illumination Variation (IV), Scale Variation (SV), Occlusion (OCC), Deformation (DEF), Motion Blur (MB), Fast Motion (FM), In-Plane Rotation (IPR), Out-of-Plane Rotation (OPR), Out-of-View (OV), Background Clutters (BC), and Low Resolution (LR).Tables1 and 2 show, respectively, the average center location error (in pixels) for each challenge factor on the OTB-2013 and OTB-2015 datasets. From the last row of Tables 1 and 2, we can see that the MCM algorithm obtains the lowest average center location error compared with the other competing algorithms. In the total of eleven challenge factors, the MCM has the lower center location errors than the other compared methods at the ten challenges on the OTB-2015 dataset and has the lower CLE values at the eight challenges on the OTB-2013 dataset. On the two datasets, the MCM works best in Out-of-View (OV) due to the employed relative similarity based on the object model, which can overcome the dramatic appearance changes to significantly improve the tracking performance for the most challenging environments on the two datasets. Overall, the center location errors of the MCM algorithm with multiple strong classifiers are lower than CT and MIL using the single strong classifier.Table 1
Average center location error (in pixels) for each challenge factor in OTB-2013.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
122.78
140.88
57.42
150.62
94.33
124.23
76.59
SV
450.79
92.21
69.57
118.74
78.92
130.93
57.72
OCC
523.59
103.39
63.44
122.27
79.26
150.62
51.08
DEF
721.04
147.85
55.74
161.01
99.33
186.27
69.20
MB
189.68
103.06
81.99
125.26
83.37
113.97
64.67
FM
204.85
106.38
77.04
119.60
78.23
128.46
54.45
IPR
146.50
101.37
73.84
124.34
79.45
125.53
53.99
OPR
473.08
111.67
69.33
131.17
80.29
155.43
56.17
OV
121.16
71.07
66.26
125.35
69.42
122.82
45.54
BC
661.00
142.34
86.08
161.67
84.86
113.14
61.49
LR
154.15
97.59
50.22
136.84
73.68
131.85
50.31
ALL
377.53
117.94
69.14
131.46
82.66
148.08
55.49
The last row represents the average center location error on all the 50 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Table 2
Average center location error (in pixels) for each challenge factor on the OTB-2015 dataset.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
484.84
89.58
64.37
197.16
205.23
134.66
48.05
SV
379.01
79.47
75.95
180.59
186.45
129.80
42.62
OCC
429.49
78.67
72.02
202.60
192.44
133.90
40.10
DEF
481.93
93.87
80.69
227.46
179.21
142.79
44.60
MB
897.57
86.94
106.29
178.91
179.25
131.01
43.89
FM
147.24
77.35
83.93
176.75
181.55
128.54
41.03
IPR
100.36
72.08
57.11
181.02
187.67
110.84
37.70
OPR
580.14
69.84
62.41
204.64
189.66
120.64
38.10
OV
162.25
87.60
98.38
152.75
208.33
153.18
36.30
BC
481.65
98.96
67.46
186.94
201.99
143.19
45.15
LR
130.46
49.92
55.30
114.25
157.96
104.11
56.71
ALL
388.63
80.39
74.90
182.10
188.16
130.24
43.11
The last row represents the average center location error on all the 100 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Tables3 and 4 list, respectively, the average tracking precision at the threshold of 0.5×minw,h of the competing algorithms under the different challenge factors on the OTB-2013 and OTB-2015 datasets. In the last row of Table 3, the average precision of the MCM algorithm is only lower than KCF by 0.043 on the OTB-2013 dataset. In Scale Variation (SV), Out-of-View (OV), and Low Resolution (LR), the precision of the MCM algorithm is higher than that of the KCF algorithm on the OTB-2013 dataset. In the last row of Table 4, the average precision of the MCM algorithm is 0.553, which is highest among all the comparison algorithms on the OTB-2015 dataset. In Scale Variation (SV), Occlusion (OCC), Deformation (DEF), Out-of-View (OV), and Background Clutters (BC), the precision of the MCM algorithm is higher than that of the other competing algorithms on the OTB-2015 dataset. Overall, by fusing multiple strong classifiers, the precision of the MCM algorithm is higher than that of CT and MIL using the single strong classifier on the two datasets. Therefore, the effectiveness of the multi-classifier fusion of the MCM method is verified.Table 3
Average precision at the threshold of0.5×minw,h for each challenge factor in OTB-2013.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
0.470
0.237
0.630
0.062
0.408
0.375
0.451
SV
0.278
0.303
0.446
0.076
0.344
0.394
0.459
OCC
0.303
0.284
0.550
0.081
0.294
0.376
0.468
DEF
0.300
0.304
0.580
0.069
0.386
0.330
0.450
MB
0.203
0.216
0.408
0.080
0.340
0.436
0.393
FM
0.200
0.237
0.407
0.077
0.339
0.429
0.421
IPR
0.228
0.282
0.502
0.064
0.379
0.432
0.451
OPR
0.309
0.269
0.539
0.080
0.365
0.403
0.461
OV
0.203
0.370
0.375
0.095
0.460
0.485
0.568
BC
0.422
0.264
0.575
0.048
0.418
0.403
0.546
LR
0.291
0.232
0.441
0.070
0.391
0.444
0.498
ALL
0.330
0.284
0.508
0.066
0.371
0.433
0.465
The last row represents the average precision on all the 50 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Table 4
Average precision at the threshold of0.5×minw,h for each challenge factor on the OTB-2015 dataset.
Challenge factor
CSK
CT
KCF
Struck
WMIL
TLD
MCM
IV
0.472
0.334
0.685
0.051
0.033
0.532
0.609
SV
0.320
0.426
0.471
0.055
0.086
0.533
0.541
OCC
0.392
0.370
0.550
0.057
0.056
0.522
0.588
DEF
0.376
0.407
0.543
0.049
0.075
0.486
0.582
MB
0.200
0.337
0.365
0.085
0.107
0.529
0.519
FM
0.329
0.379
0.468
0.073
0.073
0.568
0.555
IPR
0.228
0.282
0.502
0.064
0.379
0.432
0.451
OPR
0.460
0.400
0.602
0.041
0.054
0.558
0.582
OV
0.248
0.380
0.313
0.069
0.073
0.459
0.629
BC
0.513
0.379
0.626
0.035
0.042
0.459
0.643
LR
0.387
0.305
0.525
0.076
0.070
0.465
0.386
ALL
0.357
0.364
0.513
0.060
0.095
0.505
0.553
The last row represents the average precision on all the 100 datasets. Bold fonts indicate the best performance while italic fonts indicate the second best.Figures4 and 5, respectively, show the center location error plots on several representative video sequences on the OTB-2013 and OTB-2015 datasets. In addition, as shown in Figures 4 and 5, we can intuitively see that the MCM tracker yields much lower center location errors for most of the frames than the other six algorithms on the two benchmarks. Figures 6 and 7, respectively, present the precision plots over thresholds in [0, 50] on eight representative sequences on the two datasets. Our algorithm has the higher precision than the other comparison algorithms among the thresholds [0, 50] for most of video sequences. This is because compressive sensing and the improved online SMILE mechanism play an important role in the MCM algorithm. As a whole, the proposed algorithm is more effective to avoid the severe drifting problem than the other competing trackers in the complex environments.Figure 4
Center location error plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2013 dataset. In the upper right corner, the average center location errors of all the frames on each sequence are listed for all the comparison methods. (a) Bird1, (b) Box, (c) Couple, (d) Human6, (e) Matrix, (f) MotorRolling, (g) Walking, and (h) Walking2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)Figure 5
Center location error plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2015 dataset. In the upper right corner, the average center location errors of all the frames on each sequence are listed for all the comparison methods. (a) ClifBar, (b) Coupon, (c) Crossing, (d) DragonBaby, (e) Human5, (f) Singer1, (g) Tiger2, and (h) Trans.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)Figure 6
Precision plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2013 dataset. In the upper right corner, the precision at the threshold0.5×minw,h of all the comparison methods is listed. (a) Bird1, (b) Couple, (c) Box, (d) Human3, (e) Panda, (f) Matrix, (g) Walking, and (h) Walking2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)Figure 7
Precision plots for six state-of-the-art trackers and our algorithm on several representative video sequences on the OTB-2015 dataset. In the upper right corner, the precision at the threshold0.5×minw,h of all the comparison methods is listed. (a) ClifBar, (b) Coupon, (c) Crossing, (d) DragonBaby, (e) Human 5, (f) Singer1, (g) Tiger2, and (h) Trans.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)


## 4.3.2. Time Complexity Analysis

We use the number of frames per second (FPS) to evaluate the time complexity of the algorithm. The larger the FPS is, the shorter the running time of the algorithm is, and the lower the time complexity of the algorithm is. Table5 lists the average FPS values of different algorithms on the 100 challenging video sequences of the OTB-2015 dataset. As shown in Table 5, the tracking speed of the MCM algorithm is around 12 frames per second, and the MCM has the faster tracking speed than the Struck, WMIL, and TLD methods. Although it has the lower tracking speed than the CSK, KCF, and CT algorithms, the MCM algorithm obtains the lower center location errors and the higher precision than the three compared algorithms, due to the multi-classifier fusion framework employed in this paper.Table 5
The average number of frames per second (FPS) for the state-of-the-art trackers and our algorithm on the 100 challenging video sequences of the OTB-2015 dataset.
Methods
CSK
CT
KCF
Struck
WMIL
TLD
MCM
FPS
342
64
192
8
10
4
12


## 4.3.3. Qualitative Evaluation

Figure8 summarizes the qualitative evaluation of the proposed algorithm and the six competing trackers under seven representative challenge sequences. In the Basketball video sequence (see Figure 8(a)), there are three kinds of challenging factors, i.e., fast motion, scale variation, and occlusion. In the #17 frame, the Struck algorithm loses the object, and the KCF and WMIL methods are partially deviated from the object when the occlusion occurs. When the object moves fast and the scale variations happen, the KCF algorithm drifts heavily (see the #63 frame), and the CT and TLD algorithms also suffer from drifting (see the #247 frame). The CSK algorithm gradually deviates from the object and then the object is lost from the #489 frame. The CT algorithm also has a severe offset in the subsequent #629 frame. The proposed MCM algorithm performs well in most frames, where the object can be accurately tracked in the face of rapid movement and scale variation.Figure 8
Tracking results of CSK, CT, KCF, Struck, WMIL, TLD, and the proposed MCM algorithm on seven image sequences on the OTB-2013 and OTB-2015 datasets. From top to bottom, (a) Basketball, (b) Bird 1, (c) BlurBody, (d) Box, (e) Panda, (f) Singer2, and (g) Walking2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)There are three tracking difficulties in the Bird1 sequence (see Figure8(b)), i.e., fast motion of bird wings, long-term object occlusion, and rapid scale variation. In the initial fast motion, CSK loses the object firstly (see the #17 frame), and then the KCF and Struck algorithms drift severely (see the #35 frame). After long-term occlusion of cloud, only the MCM algorithm still tracks the object and the other algorithms are severely deviated from the ground truth (see the #187 frame). For the subsequent scale variations, the MCM algorithm still keeps tracking the object well (see the #266 and #403 frames).In the BlurBody sequence (see Figure8(c)), because the camera always shakes, some frames become blurred, which will have a great impact on the object tracking. During the tracking process, most algorithms have serious large-scale drifting, but the MCM algorithm is better to adapt to this situation.In the Box video sequence (see Figure8(d)), due to the occlusion problem in the moving process of the box, the CSK, TLD, and Struck algorithms drift successively, and then the object is lost (see the #73, #354, and #429 frames). The CT and WMIL and KCF algorithms miss the target in the subsequent frames with scale variations and rotations (see the #552 and #737 frames). However, the MCM algorithm has excellent performance in face of these challenge factors and can accurately track the target.In the Panda sequence (see Figure8(e)), a small Panda suffers from fast movements and occlusions. The Struck, CSK, and KCF algorithms have lost the object gradually, and the TLD algorithm fails due to scale variations during the tracking process (see the #160 and #609 frames), and the other algorithms are more accurate and stable to track the object.A singer in the Singer2 sequence (see Figure8(f)) has the large-scale variations and rotational deformations during the tracking process. In the scale variations of the object, the TLD, Struck, CSK, and CT algorithms work poor (see the #40, #107, and #269 frames), while the other algorithms can accurately track the target. In the movements of the singer, the KCF and WMIL algorithms are partially drifting from the object, and the MCM algorithm tracks the target more accurately during the tracking process.In the Walking2 dataset (see Figure8(g)), the main challenging factor is the occlusions of two persons. After the first occlusion, the Struck, TLD, CT, and WMIL algorithms drift to track the error object (see the #243 frame). After the second occlusion, the CSK method also begins to have a large offset (see the #389 frame). When the target becomes smaller and the scale variation occurs, the MCM algorithm is able to track the target accurately, and yet the other algorithms partially depart from the object (see the #412 frame).In summary, the MCM algorithm outperforms the other compared trackers and can be more accurate and robust to track objects in the challenging factors of rapid motions, scale variations and object occlusions, and so on.


## 5. Conclusions

In this paper, we present an effective and robust object tracking algorithm, termed as MCM. We employ the multi-classifier fusion framework to integrate the advantages of multiple different discriminative strong classifiers, where the different random projection matrices are used to extract the low-dimensional feature vectors based on the compressive sensing technology. Then, an improved SMILE mechanism is presented to further filter and weight the instances in the positive bag by introducing the concept of relative similarity. Experimental results show that the proposed MCM algorithm performs favourably against several representative tracking algorithms on the two popular tracking benchmark datasets.



---

*Source: 1574054-2020-03-10.xml*