# Multifeatures Based Compressive Sensing Tracking

Authors: Liang He; Yuming Bo; Gaopeng Zhao
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

To benefit from the development of compressive sensing, we cast tracking as a
sparse approximation problem in a particle filter framework based on multifeatures.
In this framework, the target template is composed of multiple
features extracted from visible and infrared frames; in addition, occlusion,
interruption, and noises are addressed through a set of trivial templates. With
this model, the sparsity is achieved via a compressive sensing approach without
nonnegative constraints; then the residual between sparsity representation
and the compressed sensing observation is used to measure the likelihood
which weights particles. After that, the target template is adaptively updated
according to the Bhattacharyya coefficients. Some experimental results
demonstrate that the proposed tracker appears to have better robustness compared with
four different algorithms.


---

## Body

## 1. Introduction

Visual tracking is an essential task in computer vision. It is applicable in many fields such as vehicle tracking, medical imaging, robotics, and surveillance. Many efforts have been paid in developing a robust visual tracking algorithm to overcome the challenges when occlusion, illumination changes, viewpoints variation, or noise interruption occurs [1–3].With the help of sparse representation technique, [4] proposed the L1 tracker to perform a robust visual tracking, casting tracking as a sparse approximation problem under a particle filter framework. This approach shows better performance in dealing with occlusions, pose changes, and illumination changes, when compared with mean shift tracker, covariance tracker, and appearance adaptive particle filter. But it is computationally demanding because it needs to solve an l1 norm related minimization problem for many times. Hereafter many efforts are paid to accelerate tracking progress and improve robustness based on it: [5] developed a real-time compressive sensing tracking (RTCST) algorithm to reduce the computational complexity; meanwhile [6] promoted an accelerated proximal gradient approach instead of the interior point method for acceleration; the result shows that both algorithms perform with higher accuracy and robustness than standard L1 tracker in many complex scenarios but are still insufficient in accommodating extreme illumination variations. A main reason causing this insufficiency may be that both algorithms directly use the tailored target image to generate the target templates, which is not sensitive to environmental changes; besides, it will be unstable when tracking target is similar to background. To improve robustness and accuracy, a multifeature based method is employed in our approach, which leads us to treat the tracking target as a sparse representation in the linear span of multifeature space.Actually, the multifeature methods are widely used in image fusion and face recognition; many attempts are taken to extend them to deal with tracking problems [7, 8]. A mixture of infrared and visible features is a typical method for modeling the tracking object. Reference [9] uses the mixed visible and infrared features and combines a mean shift method with the level set evolution algorithm to track visual objects; [10] employs the intensity cue and edge cue of infrared target as the feature template and applies a particle filter framework to track with pedestrian. All these algorithms could successfully cope with the complex environmental cases such as illumination change, shadow, and occlusion but should be improved to handle challenges under more severe conditions. To benefit from these previous works, we use some of the features mentioned above, together with a modified compressed sensing tracking method to receive a more robust tracking result.The reminder of the paper is arranged as follows. We briefly review some related works in Section2. Section 3 details the proposed tracker, including a sparse target template representation model, a compressive sensing tracking strategy, and an adaptive template update scheme. Experimental results and conclusions are given in Sections 4 and 5.


## 2. Related Work

### 2.1. Particle Filtering

The particle filter is a sequential Monte Carlo sampling method for Bayesian filtering; it provides a convenient framework for estimating the posterior distribution of state variables characterizing a dynamic system which is nonlinear and non-Gaussian. Assume thatst is the state variable at time t which can be defined as affine motion or any other parameters reflecting the properties of the system. A predicting distribution of st given all available observations y1:t-1 up to time t-1 can be computed as
(1)p(st∣y1:t-1)=∫‍p(st∣st-1)p(st-1∣y1:t-1)dst-1.
When observation yt is available at time t, the state vector is updated using the Bayes rule
(2)p(st∣y1:t)=p(yt∣st)p(st∣y1:t-1)p(yt∣y1:t-1)
in particle filter; the posterior p(st|y1:t) is approximated by a weighted sample set S={(s(n),ω(n))∣n=1,…,N}. Because it is generally impossible to sample from the state posterior p(st∣y1:t) directly, an importance distribution q(st∣s1:t-1,y1:t) is used for updating the weights of the samples
(3)ωti=ωt-1ip(yt∣sti)p(sti∣st-1i)q(st∣s1:t-1,y1:t)
and a typical value of importance function is q(st∣s1:t-1,y1:t)=p(st∣st-1); in this situation, the weights become the observation likelihood p(yt|st). Then the samples are resampled to generate a set of equally weighted particles according to their importance weights to avoid degeneracy.


### 2.2. Compressive Sensing

Assuming that a signalx∈RN can be sparsely expressed in a basis of N×1 vectors
(4)x=∑i=1N‍siψiorx=Ψs
the signal x is K-sparse if ∥s∥0=K, in which ∥s∥0 counts the nonzero elements in vector s. The compressive sensing theory demonstrates that x can be recovered with an overwhelming probability by using few measurements:
(5)y=Φx=ΦΨs=Θs,
where Φ is incoherent with Ψ and Θ satisfies the RIP (restricted isometry property) [11].Many studies indicate that the Gaussian measurement matrixΦ is stable to protect the salient information of any K-sparse or compressible signal in dimensionality reduction. With a small constant c, an M×N iid Gaussian matrix Φ can be shown to have the RIP with high probability if M≥cKlog⁡(N/K); besides, for Φ being universal, Θ=ΦΨ will be iid Gaussian, thus having the RIP regardless of the choice of Ψ [12].Since directly obtains (which equals to x in the Ψ-domain) via
(6)s^=argmin⁡∥s∥0s.tΘs=y
is an NP-complete problem, it is commonly relaxed to solve the optimization
(7)s^=argmin⁡∥s∥1s.t.Θs=y.
When considering noises, the modified optimization
(8)s^=argmin⁡∥s∥1s.t.∥Θs-y∥2≤ε,
where ε is a prespecified threshold.


### 2.3. L1 Tracker and Real-Time Compressive Sensing Tracking

Thel1 tracker considers the tracking target as a sparse representation in the linear span of target template set, along with noises. Given target template set T=[t1⋯tn]∈Rd×n(d≫n) and trivial templates I=[i1,i2,…,id]∈Rd×d, a tracking result y∈Rd can be modeled as
(9)y=[T,I,-I][ae+e-]=^Bc,s.t.c≥0,
where a=(a1,a2,…,an)T∈Rn is a target coefficient vector and e+∈Rd, e-∈Rd are called positive and negative trivial coefficient vectors; with this definition, at time t, the coefficient vector cti of each candidate target yti is gained by solving the l1 regularized least squares problem
(10)min⁡c⁡12∥yti-Bc∥22+λ∥c∥1,c⪰0,
where i=1,…,Ns is the number of particles; then the residuals of particles rti=∥yti-Ttati∥22 are imported into particle filter to estimate the tracking result [4].Based on the model above, to reduce computational load, [6] modifies (10) to the following minimization model:
(11)min⁡c12∥y-B′c∥22+λ∥c∥1+μt2∥e∥22,a⪰0,
where B′=[Tt,I], c=[a,e]T, e=[e+,e-]T, and μt is a parameter controlling energy in trivial templates; after that, an accelerated proximal gradient approach is proposed for solving problem (11). At the same time, RTCST uses compressive sensing to deal with the extremely high dimension of feature space in l1 tracker. Given the measurement matrix Φ, we get the measurement vector z=Φy=ΦBc; then a sparse coefficient vector c can be recovered from
(12)min⁡∥c∥1s.t.∥ΦBc-Φy∥≤ε.


## 2.1. Particle Filtering

The particle filter is a sequential Monte Carlo sampling method for Bayesian filtering; it provides a convenient framework for estimating the posterior distribution of state variables characterizing a dynamic system which is nonlinear and non-Gaussian. Assume thatst is the state variable at time t which can be defined as affine motion or any other parameters reflecting the properties of the system. A predicting distribution of st given all available observations y1:t-1 up to time t-1 can be computed as
(1)p(st∣y1:t-1)=∫‍p(st∣st-1)p(st-1∣y1:t-1)dst-1.
When observation yt is available at time t, the state vector is updated using the Bayes rule
(2)p(st∣y1:t)=p(yt∣st)p(st∣y1:t-1)p(yt∣y1:t-1)
in particle filter; the posterior p(st|y1:t) is approximated by a weighted sample set S={(s(n),ω(n))∣n=1,…,N}. Because it is generally impossible to sample from the state posterior p(st∣y1:t) directly, an importance distribution q(st∣s1:t-1,y1:t) is used for updating the weights of the samples
(3)ωti=ωt-1ip(yt∣sti)p(sti∣st-1i)q(st∣s1:t-1,y1:t)
and a typical value of importance function is q(st∣s1:t-1,y1:t)=p(st∣st-1); in this situation, the weights become the observation likelihood p(yt|st). Then the samples are resampled to generate a set of equally weighted particles according to their importance weights to avoid degeneracy.


## 2.2. Compressive Sensing

Assuming that a signalx∈RN can be sparsely expressed in a basis of N×1 vectors
(4)x=∑i=1N‍siψiorx=Ψs
the signal x is K-sparse if ∥s∥0=K, in which ∥s∥0 counts the nonzero elements in vector s. The compressive sensing theory demonstrates that x can be recovered with an overwhelming probability by using few measurements:
(5)y=Φx=ΦΨs=Θs,
where Φ is incoherent with Ψ and Θ satisfies the RIP (restricted isometry property) [11].Many studies indicate that the Gaussian measurement matrixΦ is stable to protect the salient information of any K-sparse or compressible signal in dimensionality reduction. With a small constant c, an M×N iid Gaussian matrix Φ can be shown to have the RIP with high probability if M≥cKlog⁡(N/K); besides, for Φ being universal, Θ=ΦΨ will be iid Gaussian, thus having the RIP regardless of the choice of Ψ [12].Since directly obtains (which equals to x in the Ψ-domain) via
(6)s^=argmin⁡∥s∥0s.tΘs=y
is an NP-complete problem, it is commonly relaxed to solve the optimization
(7)s^=argmin⁡∥s∥1s.t.Θs=y.
When considering noises, the modified optimization
(8)s^=argmin⁡∥s∥1s.t.∥Θs-y∥2≤ε,
where ε is a prespecified threshold.


## 2.3. L1 Tracker and Real-Time Compressive Sensing Tracking

Thel1 tracker considers the tracking target as a sparse representation in the linear span of target template set, along with noises. Given target template set T=[t1⋯tn]∈Rd×n(d≫n) and trivial templates I=[i1,i2,…,id]∈Rd×d, a tracking result y∈Rd can be modeled as
(9)y=[T,I,-I][ae+e-]=^Bc,s.t.c≥0,
where a=(a1,a2,…,an)T∈Rn is a target coefficient vector and e+∈Rd, e-∈Rd are called positive and negative trivial coefficient vectors; with this definition, at time t, the coefficient vector cti of each candidate target yti is gained by solving the l1 regularized least squares problem
(10)min⁡c⁡12∥yti-Bc∥22+λ∥c∥1,c⪰0,
where i=1,…,Ns is the number of particles; then the residuals of particles rti=∥yti-Ttati∥22 are imported into particle filter to estimate the tracking result [4].Based on the model above, to reduce computational load, [6] modifies (10) to the following minimization model:
(11)min⁡c12∥y-B′c∥22+λ∥c∥1+μt2∥e∥22,a⪰0,
where B′=[Tt,I], c=[a,e]T, e=[e+,e-]T, and μt is a parameter controlling energy in trivial templates; after that, an accelerated proximal gradient approach is proposed for solving problem (11). At the same time, RTCST uses compressive sensing to deal with the extremely high dimension of feature space in l1 tracker. Given the measurement matrix Φ, we get the measurement vector z=Φy=ΦBc; then a sparse coefficient vector c can be recovered from
(12)min⁡∥c∥1s.t.∥ΦBc-Φy∥≤ε.


## 3. Multifeature Based Compressive Sensing Tracking

### 3.1. Target Template Initialization

Previous studies indicate that multifeature methods perform better robustness than methods employing single feature [13, 14]; moreover, good feature descriptors are helpful for stable tracking. This motivates us to employ a multifeature method to form the target template set. Suppose pk(y),k=1,…,Nf to be the kth feature extracted from target image whose central coordinate is y; then an observation of the target can be characterized by a vector p(y)=[p1(y),p2(y),…,pk(y)]T. In the first frame, after manually selecting the target, the features of target image qk(y0)=pk(y0), k=1,…,Nf, are calculated to form a template t1=[q1(y0),q2(y0),…,qk(y0)]T; for initialization, we first perturb the center of target image y0 to generate candidate image center yi,i=1,…,Nt, crop candidates with the same size as target image according to yi, extract features of each candidate to form the rest templates ti=[q1(yi),q2(yi),…,qk(yi)]T,k=1,…,Nf, and finally initialize the template set as T=[t1,t2,…,ti],i=1,…,Nt. Here we use a combination of visible and infrared features to improve robustness of the tracking algorithm; the mainly used visible feature is the RGB histogram and local binary patterns (LBP); in addition, the main infrared feature used is the intensity histogram.


#### 3.1.1. RGB and Intensity Histogram

Histogram is an estimate of the probability distribution of a continuous variable; here we employ an alternative histogram extracting approach where pixels further away from the region center are assigned smaller weights. The weighting function (also can be called kernel function) is(13)k(r)={1-r2,r<10,otherwise,
where r is the distance from region center; then the histogram at location yk can be described as
(14)hi,y(u)=C∑i=1m‍k(∥yi-y0h∥2)δ(b(yi)-u);C=[∑i=1m‍k(∥(yi-y0)/h∥2)]-1 here is a normalization factor, {yi},i=1,…,m is the current coordinates of pixels in candidate target, and h=hx2+hy2 is the bandwidth of kernel function. δ is a Kronecker delta function, where u=1,…,BI is the histogram bin and b(yi) associates pixel yi with the histogram bin according to the corresponding RGB or gray intensity.


#### 3.1.2. LBP

LBP (local binary pattern) is a type of feature characterizing the texture spectrum of an image; it is proved to be a powerful feature for texture classification. The original LBP feature first divides the examined window into cells and then, for each pixelx0 in a cell, compares it to each of its 8 neighbors {xi},i=1,…,7; the LBP value can be calculated as
(15)LBP(x0)=∑i=17‍s(xi-x0)2i,s.t.s(x)={1,x>00,x<0.
After that, a histogram is computed over the cell to generate the feature vector for the window.


### 3.2. Compressive Sensing Tracking Based on Multifeatures

As done in [5, 6], we apply affine image warping to model the object motion of two consecutive frames; the state variable st is defined to be st=(α1,α2,α3,α4,tx,ty) which combine deformation parameters (α1,α2,α3,α4) with 2D translation parameters (tx,ty); the state transition distribution p(st∣st-1) is modeled as a Gaussian distribution.Inl1 tracker and RTCST, the dictionary B is composed of target template T, positive trivial template I, and negative trivial template -I; meanwhile the coefficient vector is divided into target coefficient vector a, positive trivial coefficient vector e+, and negative trivial coefficient e- to enforce nonnegativity constraints. The main reason of exploiting nonnegativity constraints into account is that they help filter out clutter which is similar to target templates but in reversed intensity patterns; furthermore, it is normal that the target image in current frame appears to be more like that in previous frame. Since multifeatures have the potential of filtering clutter and occlusion, the nonnegativity constraints can be relaxed; besides, an extra occlusion handling method becomes less necessary; for example, when tracking with pedestrian, the occlusion may occur in visible image but not in related infrared image; in other words, infrared features help filter out the occlusions. While the target template set T=[t1,t2,…,tNt] is available after initialization or update, we model the observation as
(16)z=[T,I][xtxe]=^Ax,
where I is the trivial template representing noises along with the trivial coefficient vector xe and xt is the target coefficient vector.Under a certain threshold more weak correlated features will increase robustness and stability but will take more computational cost. To deal with this problem, we adopt the compressed sensing theory. Choosing random Gaussian matrixΦ∈Rm×n, ϕi,j~N(0,1) as the measurement matrix, we take the measurements z′=Φz=ΦAx; then by using orthogonal matching pursuit (OMP), we recover the coefficient vector x from
(17)min⁡∥x∥1,s.t.∥ΦAx-Φz∥2≤ε.
The main flow of multifeature based compressive sensing tracking (MFCST) algorithm is presented in Algorithm 1.Algorithm 1:Multifeature based compressive sensing tracking.
Input:
(i) Current visible frameIυk∈Rh×w and infrared frame Iirk∈Rh×w
(ii) Particlessk-1i,i=1,2,…,Ns
(iii) Particles Measurement matrixΦ∈Rm×n
(iv) Template setA=[T,I]∈Rn×(n+Nt)
(v) Preset parameterλ>0
(1) Normalize each column ofΦA;
(2) Generate particlesski,i=1,2,…,Ns according to state transition distribution;
(3)for  i=1→Ns  do
(4)    Obtain target candidatesgi and relevant candidate center yi according to ski;
(5)    Calculate RGB histogrampRGB, LBP pLBP of candidate in Iυk and intensity histogram pgray of candidate in Iirk;
(6)    StackpLBP,pgray and pRGB into an observation zi=[pLBP,pgray,pRGB]T;
(7)    Get measurementsz′=Φzi;
(8)    Solving (17) to get coefficient vector x;
(9)    Calculate residualri=∥Φzi-ΦAxt∥2;
(10) Obtain observation likelihoodli=exp⁡(-λri);
(11)end
(12) Resample the particlesski according to li;
(13) Using Maximum A Posterior (MAP) or Mean Square Error (MSE) to
estimate the state sk of current frame, which characterizes the current tracking target g;
(14)  Get target observationz^ corresponding to current tracking result g;
(15) Recalculatex^ for z^ by solving (17);
(16) Update target templateT;
Output:
(i) Tracked targetg
(ii) Updated target dynamic statesk
(iii) Updated target template setT


### 3.3. Template Update

Intuitively, as time goes by, a static template will not be able to exactly capture the appearance variations of target due to illumination or pose changes; otherwise, frequently updating the template will accumulate errors and drift the tracker from target. We tackle this problem by adaptively updating the target template. Notice thatxt can be seen as a sparse representation of z in the linear span of T, which means that the bigger xi is, the more important ti will be. The Bhattacharyya coefficient is applied to measure the similarity between target observation z^ and the template tm which has maximum coefficient xm, which can be defined as
(18)ρ(z^,tm)=∑i=1n‍z^itmi;
a typical value of ρ(·) is between (0, 1); in addition, bigger ρ(z^,tm) means that z^ is more similar to tm. We adopt two thresholds τ2>τ1 to guide the updating progress, where ρ(z^,tm)<τ2 indicates that z^ is not so similar to tm, which reflects variations in tracking; moreover, when ρ(z^,tm) decreases under a certain threshold that is ρ(z^,tm)<τ1, we consider it as a strong interference occurring in the tracking progress. The template update scheme is summarized in Algorithm 2.Algorithm 2:Template update.
Input:
(i) Newly tracking resultg and the corresponding observation z^
(ii) Target coefficient vectorx^t
(iii) Template setA=[T,I]∈Rn×(n+Nt), where T=[t1,t2,…,tNt]∈Rn×Nt
(iv) Bhattacharyya thresholdsτ1, τ2, 1>τ2>τ1>0
(1)m←argmax1≤i≤Nt⁡xi;
(2)if  τ2>ρ(z^,tm)>τ1  then
(3)n←argmin1≤i≤Nt⁡xi;
(4)tn←z^;
(5)end
(6) Normalize every columns ofT;
Output: Updated template set A=[T,I]


## 3.1. Target Template Initialization

Previous studies indicate that multifeature methods perform better robustness than methods employing single feature [13, 14]; moreover, good feature descriptors are helpful for stable tracking. This motivates us to employ a multifeature method to form the target template set. Suppose pk(y),k=1,…,Nf to be the kth feature extracted from target image whose central coordinate is y; then an observation of the target can be characterized by a vector p(y)=[p1(y),p2(y),…,pk(y)]T. In the first frame, after manually selecting the target, the features of target image qk(y0)=pk(y0), k=1,…,Nf, are calculated to form a template t1=[q1(y0),q2(y0),…,qk(y0)]T; for initialization, we first perturb the center of target image y0 to generate candidate image center yi,i=1,…,Nt, crop candidates with the same size as target image according to yi, extract features of each candidate to form the rest templates ti=[q1(yi),q2(yi),…,qk(yi)]T,k=1,…,Nf, and finally initialize the template set as T=[t1,t2,…,ti],i=1,…,Nt. Here we use a combination of visible and infrared features to improve robustness of the tracking algorithm; the mainly used visible feature is the RGB histogram and local binary patterns (LBP); in addition, the main infrared feature used is the intensity histogram.


### 3.1.1. RGB and Intensity Histogram

Histogram is an estimate of the probability distribution of a continuous variable; here we employ an alternative histogram extracting approach where pixels further away from the region center are assigned smaller weights. The weighting function (also can be called kernel function) is(13)k(r)={1-r2,r<10,otherwise,
where r is the distance from region center; then the histogram at location yk can be described as
(14)hi,y(u)=C∑i=1m‍k(∥yi-y0h∥2)δ(b(yi)-u);C=[∑i=1m‍k(∥(yi-y0)/h∥2)]-1 here is a normalization factor, {yi},i=1,…,m is the current coordinates of pixels in candidate target, and h=hx2+hy2 is the bandwidth of kernel function. δ is a Kronecker delta function, where u=1,…,BI is the histogram bin and b(yi) associates pixel yi with the histogram bin according to the corresponding RGB or gray intensity.


### 3.1.2. LBP

LBP (local binary pattern) is a type of feature characterizing the texture spectrum of an image; it is proved to be a powerful feature for texture classification. The original LBP feature first divides the examined window into cells and then, for each pixelx0 in a cell, compares it to each of its 8 neighbors {xi},i=1,…,7; the LBP value can be calculated as
(15)LBP(x0)=∑i=17‍s(xi-x0)2i,s.t.s(x)={1,x>00,x<0.
After that, a histogram is computed over the cell to generate the feature vector for the window.


## 3.1.1. RGB and Intensity Histogram

Histogram is an estimate of the probability distribution of a continuous variable; here we employ an alternative histogram extracting approach where pixels further away from the region center are assigned smaller weights. The weighting function (also can be called kernel function) is(13)k(r)={1-r2,r<10,otherwise,
where r is the distance from region center; then the histogram at location yk can be described as
(14)hi,y(u)=C∑i=1m‍k(∥yi-y0h∥2)δ(b(yi)-u);C=[∑i=1m‍k(∥(yi-y0)/h∥2)]-1 here is a normalization factor, {yi},i=1,…,m is the current coordinates of pixels in candidate target, and h=hx2+hy2 is the bandwidth of kernel function. δ is a Kronecker delta function, where u=1,…,BI is the histogram bin and b(yi) associates pixel yi with the histogram bin according to the corresponding RGB or gray intensity.


## 3.1.2. LBP

LBP (local binary pattern) is a type of feature characterizing the texture spectrum of an image; it is proved to be a powerful feature for texture classification. The original LBP feature first divides the examined window into cells and then, for each pixelx0 in a cell, compares it to each of its 8 neighbors {xi},i=1,…,7; the LBP value can be calculated as
(15)LBP(x0)=∑i=17‍s(xi-x0)2i,s.t.s(x)={1,x>00,x<0.
After that, a histogram is computed over the cell to generate the feature vector for the window.


## 3.2. Compressive Sensing Tracking Based on Multifeatures

As done in [5, 6], we apply affine image warping to model the object motion of two consecutive frames; the state variable st is defined to be st=(α1,α2,α3,α4,tx,ty) which combine deformation parameters (α1,α2,α3,α4) with 2D translation parameters (tx,ty); the state transition distribution p(st∣st-1) is modeled as a Gaussian distribution.Inl1 tracker and RTCST, the dictionary B is composed of target template T, positive trivial template I, and negative trivial template -I; meanwhile the coefficient vector is divided into target coefficient vector a, positive trivial coefficient vector e+, and negative trivial coefficient e- to enforce nonnegativity constraints. The main reason of exploiting nonnegativity constraints into account is that they help filter out clutter which is similar to target templates but in reversed intensity patterns; furthermore, it is normal that the target image in current frame appears to be more like that in previous frame. Since multifeatures have the potential of filtering clutter and occlusion, the nonnegativity constraints can be relaxed; besides, an extra occlusion handling method becomes less necessary; for example, when tracking with pedestrian, the occlusion may occur in visible image but not in related infrared image; in other words, infrared features help filter out the occlusions. While the target template set T=[t1,t2,…,tNt] is available after initialization or update, we model the observation as
(16)z=[T,I][xtxe]=^Ax,
where I is the trivial template representing noises along with the trivial coefficient vector xe and xt is the target coefficient vector.Under a certain threshold more weak correlated features will increase robustness and stability but will take more computational cost. To deal with this problem, we adopt the compressed sensing theory. Choosing random Gaussian matrixΦ∈Rm×n, ϕi,j~N(0,1) as the measurement matrix, we take the measurements z′=Φz=ΦAx; then by using orthogonal matching pursuit (OMP), we recover the coefficient vector x from
(17)min⁡∥x∥1,s.t.∥ΦAx-Φz∥2≤ε.
The main flow of multifeature based compressive sensing tracking (MFCST) algorithm is presented in Algorithm 1.Algorithm 1:Multifeature based compressive sensing tracking.
Input:
(i) Current visible frameIυk∈Rh×w and infrared frame Iirk∈Rh×w
(ii) Particlessk-1i,i=1,2,…,Ns
(iii) Particles Measurement matrixΦ∈Rm×n
(iv) Template setA=[T,I]∈Rn×(n+Nt)
(v) Preset parameterλ>0
(1) Normalize each column ofΦA;
(2) Generate particlesski,i=1,2,…,Ns according to state transition distribution;
(3)for  i=1→Ns  do
(4)    Obtain target candidatesgi and relevant candidate center yi according to ski;
(5)    Calculate RGB histogrampRGB, LBP pLBP of candidate in Iυk and intensity histogram pgray of candidate in Iirk;
(6)    StackpLBP,pgray and pRGB into an observation zi=[pLBP,pgray,pRGB]T;
(7)    Get measurementsz′=Φzi;
(8)    Solving (17) to get coefficient vector x;
(9)    Calculate residualri=∥Φzi-ΦAxt∥2;
(10) Obtain observation likelihoodli=exp⁡(-λri);
(11)end
(12) Resample the particlesski according to li;
(13) Using Maximum A Posterior (MAP) or Mean Square Error (MSE) to
estimate the state sk of current frame, which characterizes the current tracking target g;
(14)  Get target observationz^ corresponding to current tracking result g;
(15) Recalculatex^ for z^ by solving (17);
(16) Update target templateT;
Output:
(i) Tracked targetg
(ii) Updated target dynamic statesk
(iii) Updated target template setT


## 3.3. Template Update

Intuitively, as time goes by, a static template will not be able to exactly capture the appearance variations of target due to illumination or pose changes; otherwise, frequently updating the template will accumulate errors and drift the tracker from target. We tackle this problem by adaptively updating the target template. Notice thatxt can be seen as a sparse representation of z in the linear span of T, which means that the bigger xi is, the more important ti will be. The Bhattacharyya coefficient is applied to measure the similarity between target observation z^ and the template tm which has maximum coefficient xm, which can be defined as
(18)ρ(z^,tm)=∑i=1n‍z^itmi;
a typical value of ρ(·) is between (0, 1); in addition, bigger ρ(z^,tm) means that z^ is more similar to tm. We adopt two thresholds τ2>τ1 to guide the updating progress, where ρ(z^,tm)<τ2 indicates that z^ is not so similar to tm, which reflects variations in tracking; moreover, when ρ(z^,tm) decreases under a certain threshold that is ρ(z^,tm)<τ1, we consider it as a strong interference occurring in the tracking progress. The template update scheme is summarized in Algorithm 2.Algorithm 2:Template update.
Input:
(i) Newly tracking resultg and the corresponding observation z^
(ii) Target coefficient vectorx^t
(iii) Template setA=[T,I]∈Rn×(n+Nt), where T=[t1,t2,…,tNt]∈Rn×Nt
(iv) Bhattacharyya thresholdsτ1, τ2, 1>τ2>τ1>0
(1)m←argmax1≤i≤Nt⁡xi;
(2)if  τ2>ρ(z^,tm)>τ1  then
(3)n←argmin1≤i≤Nt⁡xi;
(4)tn←z^;
(5)end
(6) Normalize every columns ofT;
Output: Updated template set A=[T,I]


## 4. Experiments

We test our algorithm on many real-world visible and infrared video sequences obtained from the data set 03 OSU Color-Thermal Database of the OTCBVS Benchmark Dataset [15]; three scenarios of two videos are involved in our experiments. To evaluate the performance of the proposed tracking framework, we compare the tracking results of our proposed multifeature based compressive sensing tracking (MFCST) with the adaptive multicue particle filter (AMC-PF) [10], the tracking method based on infrared and visible dual-channel video (IVDT) [9], the l1 tracker [4], and the real-time compressive sensing tracking (RTCST) [5]. All targets are manually marked in the first tracking frame without careful selection.The test sequences used in our first and second experiments S1 and S2 are based on location 2 in the data set. In S1, two pedestrians intersect with each other, which causes a partial occlusion. From the visible frames, we can see that the whole scene is under well illumination; two pedestrians are distinct from both each other and the background but in corresponding infrared frames; these two pedestrians are similar to each other. Some tracking results are given in Figure1, where the frame indexes are labeled in the upper left corner of the images. It can be observed that although all trackers can well follow the target pedestrian, AMC-PF, RTCST, and MFCST do more precise work than those of l1 tracker and IVDT.Tracking results of S1: AMC-PF (a), thel1 tracker (b), RTCST (c), IVDT (d), and our proposed MFCST (e).
(a)
(b)
(c)
(d)
(e)The second experiment S2 is based on the scene in which target pedestrian passes by a static car. The target pedestrian has similar appearance as the front part and tire of the car in both visible and corresponding infrared frames. Results in Figure2 indicate that AMC-PF drifts to the car when target occludes the tire. This is because AMC-PF uses the intensity cue and edge cue of the infrared image; however, the appearance of pedestrian and the front part of the car in infrared frame is easily confused.Tracking results of S2: AMC-PF (a), thel1 tracker (b), RTCST (c), IVDT (d), and our proposed MFCST (e).
(a)
(b)
(c)
(d)
(e)The third experiment S3 considers a more complex scene; we use a video sequence from location 1 in the OSU Color-Thermal Database and then test 5 trackers mentioned above from frame 220 to 429. Since it is hard to distinct target pedestrian from background, we manually labeled the tracking target in the first infrared frame (frame 220) at the beginning of our experiment S3, without careful selection. The whole video sequence can be described in 5 clips. In the first clip, which covers from starting frame 220 to frame 250, the target walking in the shadow is confused with background in visible frame but is distinct in the infrared frame. As target walks straightforward, in clip 2, from frame 250 to 322, the tracked pedestrian particle occludes the first street lamp, which we sign as lamp 1; then, as target walks out of shadow, the illumination conditions are drastically changed. A heavy occlusion appears in clip 3, from frame 322 to 374; when target passes by the second street lamp, street lamp 2 completely occludes the pedestrian both in visible and infrared image. After that, target is particle occluded by pedestrian 2, wearing a red coat, which is obviously different from target pedestrian in visible frame; after several frames, the target walks by pedestrian 2 and encounters the third street lamp; these sequences are described in clip 4, from frame 374 to 390. At last, in clip 5, from frame 390 to 427, the target walks into a square and finishes the whole tracking process. The performance of 5 methods is shown in Figure3, from which we can see that the l1 tracker and RTCST failed to track with target in clip 1; the main reason may be that these two algorithms only use pixel information of visible images; unfortunately, it is severely a hard work to distinguish target from background in these visible frames. AMC-PF drifts to street lamp 1 from clip 2, where the illumination is severely changed and lamp 1 occludes the target both in visible and infrared frames. The dual-channel method IVDT succeeded in clips 1 and 2 but began to lose its target from clip 3. Meanwhile our proposed MFCST accurately tracks the target pedestrian throughout the whole sequence; this indicates that our approach appears to have better robustness in dealing with heavy occlusions and drastic illumination changes. In addition, when comparing with multifeature tracking methods AMC-PF and IVDT, our approach takes less measurements; furthermore, comparing with l1 and RTCST, our algorithm receives better accuracy.Tracking results of S3: AMC-PF (a), thel1 tracker (b), RTCST (c), IVDT (d), and our proposed MFCST (e).
(a)
(b)
(c)
(d)
(e)


## 5. Conclusion

We model the tracking target as a sparse representation in the linear span of multifeature space. To generate the dictionary, we use RGB histogram and local binary patterns (LBP) of the visible frames with intensity histogram of the infrared frames to create the target templateT and then combine T with the trivial template I. A compressive sensing method is employed under particle filter framework to reduce the high dimension of the feature space and solve the coefficients for sparse representation. For further robustness, we introduce an adaptive template update scheme for this system. The proposed tracker receives robust and stable performance in dealing with occlusions and illumination variations.



---

*Source: 439614-2014-05-21.xml*