# Discriminative Fusion Correlation Learning for Visible and Infrared Tracking

Authors: Xiao Yun; Yanjing Sun; Xuanxuan Yang; Nannan Lu
Journal: Mathematical Problems in Engineering
 (2019)

---

## Abstract

Discriminative correlation filter- (DCF-) based trackers are computationally efficient and achieve excellent tracking in challenging applications. However, most of them suffer low accuracy and robustness due to the lack of diversity information extracted from a single type of spectral image (visible spectrum). Fusion of visible and infrared imaging sensors, one of the typical multisensor cooperation, provides complementarily useful features and consistently helps recognize the target from the background efficiently in visual tracking. Therefore, this paper proposes a discriminative fusion correlation learning model to improve DCF-based tracking performance by efficiently combining multiple features from visible and infrared images. Fusion learning filters are extracted via late fusion with early estimation, in which the performances of the filters are weighted to improve the flexibility of fusion. Moreover, the proposed discriminative filter selection model considers the surrounding background information in order to increase the discriminability of the template filters so as to improve model learning. Extensive experiments showed that the proposed method achieves superior performances in challenging visible and infrared tracking tasks.


---

## Body

## 1. Introduction

Visual tracking has received widespread attention for its extensive applications in video surveillance, autonomous driving and human-machine interaction, military attack, robot vision, etc. [1, 2]. Depending on the appearance model, existing tracking algorithms can be categorized into two categories: generative and discriminative tracking. Generative tracking algorithms build a target model and search for the candidate image patch with maximal similarity. For example, Wang et al. [3] proposed a novel regression-based object tracking framework which successfully incorporates Lucas and Kanade algorithm into an end-to-end deep learning paradigm. Chi et al. [4] trained a dual network with random patches measuring the similarities between the network activation and target appearance to leverage the robustness of visual tracking. On the contrary, the goal of discriminative algorithms is to learn a classifier to discriminate between its appearance and that of the environment given an initial image patch containing the target. Yang et al. [5] proposed a temporal restricted reverse-low-rank learning algorithm for visual tracking to jointly represent target and background templates via candidates, which exploits the low-rank structure among consecutive target observations and enforces the temporal consistency of target in a global level. A new peak strength metric [6] is proposed to measure the discriminative capability of the learned correlation filter that can effectively strengthen the peak of the correlation response, leading to more discriminative performance than previous methods.Besides these efforts, other researchers have worked on tracking methods that are both generative and discriminative. For instance, Zhang et al. [7] obtained an object likelihood map to adaptively regularize the correlation filter learning by suppressing the clutter background noises while making full use of the long-term stable target appearance information. Qi et al. [8] proposed a structure-aware local sparse coding algorithm, which encodes a target candidate using templates with both global and local sparsity constraints, and also obtains a more precise and discriminative sparse representation to account for appearance changes. In [9], an adaptive set of filtering templates is learned to alleviate drifting problem of tracking by carefully selecting object candidates in different situations to jointly capture the target appearance variations. Moreover, a variety of simple yet effective features are effectively integrated into the learning process of filters to further improve the discriminative power of the filters. In the salient-sparse-collaborative tracker [10], an object salient feature map is built to create a salient-sparse discriminative model and a salient-sparse generative model to both handle the appearance variation and reduce tracking drifts effectively. A multilayer convolutional network-based visual tracking algorithm based on important region selection [11] is proposed to build high entropy selection and background discrimination models and to obtain the feature maps by weighting the template filters with cluster weights, which enables the training samples to be informative in order to provide enough stable information and also be discriminative so as to resist distractors. Generally speaking, discriminative and generative methods have complementary advantages in appearance modeling, and the success of a visual tracking method depends not only on its representation ability against appearance variations but also on the discriminability between target and background, thus leading to the requirement of a more robust training model [12].Recently, discriminative correlation filter- (DCF-) based visual tracking methods [13–18] have shown excellent performances on real-time visual tracking for its advantage of robustness and computational efficiency. The DCF-based methods work by learning an optimal correlation filter used to locate the target in the next frame. The significant gain in speed is obtained by exploiting the fast Fourier transform (FFT) at both learning and detection stages [14]. Bolme et al. [13] presented an adaptive correlation filter, named Minimum Output Sum of Squared Error (MOSSE) filter, which produces stable correlation filters by optimizing the output sum of squared error. Based on MOSSE, Danelljan et al. [14, 15] proposed a novel scale adaptive tracking approach by learning separate discriminative correlation filters for translation and scale estimation, which achieves accurate and robust scale estimation in a tracking-by-detection framework. Galoogahi et al. [16] proposed a computationally efficient Background-aware correlation filter-based on hand-crafted features that can efficiently model how both the foreground and background of the object varies over time. The work in [17] reformulates DCFs as a one-layer convolutional neural network composed of integrates feature extraction, response map generation, and model update with residual learning. Johnander et al. [18] proposed a unified formulation for learning a deformable convolution filter in which the deformable filter is represented as a linear combination of subfilters, and both the subfilter coefficients and their relative locations are inferred jointly in our formulation. However, the above trackers fail when the target undergoes severe appearance changes due to limited data supplied by single features.Multiple feature fusion contains more useful information than single feature, thus providing higher precision, certainty, and reliability for visual tracking. Wu et al. [19] proposed a data fusion approach via sparse representation with applications to robust visual tracking. Uzkent et al. [20] proposed an adaptive fusion tracking method that combines likelihood maps from multiple bands of hyperspectral imagery into one single more distinctive representation, which increases the margin between mean value of foreground and background pixels in the fused map. Chan et al. [21] proposed a robust adaptive fusion tracking method, which incorporates a novel complex cell into the group of object representation to enhance the global distinctiveness. Feature fusion also achieves superior performances on correlation filter-based tracking. For example, Rapuru et al. [22] proposed a robust tracking algorithm by efficiently fusing tracking, learning, and detection with the systematic model update strategy of kernelized correlation filter tracker.Although much efforts have been made, single-sensor feature fusion-based tracking suffer low accuracy and robustness due to the lack of diversity information. Fusion of visible and infrared sensors, one of the typical multisensor cooperations, provides complementarily useful features, which is able to achieve a more robust and accurate tracking result [23]. Li et al. [24] designed a fusion scheme containing joint sparse representation and colearning update model to fuse color visual spectrum and thermal spectrum images for object tracking. Li et al. [25] proposed an adaptive fusion scheme based on collaborative sparse representation in Bayesian filtering framework for online tracking. Mangale and Khambete [26] developed reliable camouflaged target detection and tracking system using fusion of visible and infrared imaging. Yun et al. [23] proposed a compressive time-space Kalman fusion tracking with time-space adaptability for visible and infrared images and introduced extended Kalman filter to update fusion coefficients optimally. A visible and infrared fusion tracking algorithm based on multiview multikernel fusion model is presented in [27]. Zhang et al. [28] transferred visible tracking data to infrared data to obtain better tracking performances. Lan et al. [29] proposed joint feature learning and discriminative classifier framework for multimodality tracking, which jointly eliminate outlier samples caused by large variations and learn discriminability-consistent features from heterogeneous modalities. Li et al. [30] proposed a convolutional neural network architecture including a two-stream ConvNet and a FusionNet, which proves that tracking with visible and infrared fusion outperforms that with single sensor in terms of accuracy and robustness.DCF-based trackers have significant low computational load and are especially suitable for a variety of real-time challenging applications. However, most of the DCF-based trackers suffer low accuracy and robustness due to the lack of diversity information extracted from a single type of spectral image (visible spectrum). Therefore, this paper proposes a discriminative fusion correlation learning model to improve DCF-based tracking performance by combining multiple features from visible and infrared imaging sensors. The main contributions of our work are summarized as follows:(i)
A discriminative fusion correlation learning model is presented to fuse visible and infrared features such that valuable information from all sensors is preserved.(ii)
The proposed fusion learning filters are obtained via late fusion with early estimation, in which the performances of the filters are weighted to improve the flexibility of fusion.(iii)
The proposed discriminative filter selection model considers the surrounding background information in order to increase the discriminability of the template filters so as to improve model learning.The remainder of this paper is organized as follows. In Section2, the multichannel discriminative correlation filter is introduced. In Section 3, we describe our work in detail. The experimental results are presented in Section 4. Section 5 concludes with a general discussion.


## 2. Multichannel Discriminative Correlation Filter

Multichannel DCF provides superior robustness and efficiency in dealing with challenging tracking tasks [14]. In the multichannel DCF-based tracking algorithm, d channel Histogram of Oriented Gradient (HOG) features [14] from the target sample s are extracted to maintain diverse information. During training process, the goal is to learn correlation filter h, which is achieved by minimizing the error of the correlation response compared to the desired correlation output g as(1)ε=g-∑l=1dhl∗sl2+λ∑l=1dhl2,where ∗ denotes circular correlation and λ is the weight parameter [14]. sl and hl(l=1,⋯,d) are the l-th channel feature and the corresponding correlation filter, respectively. The correlation output g is supposed to be a Gaussian function with a parametrized standard deviation [14].The minimization of (1) can be solved by minimizing (2) in the Fourier domain as(2)Hl=G-Sl∑k=1dS-kSk+λ,l=1,⋯,d,where H, G, and S are the discrete Fourier transform (DFT) of h, g, and s, respectively. The symbol bar ·- denotes complex conjugation. The multiplications and divisions in (2) are performed pointwise. The numerator Atl and denominator Bt of the filter Htl in (2) are updated as(3)Atl=1-ηAt-1l+ηG-Stl,l=1,⋯,d,Bt=1-ηAt-1+η∑k=1dS-kSk,where η is a learning rate parameter.During tracking process, the DFT of the correlation scoreyt of the test sample zt is computed in the Fourier domain as(4)Yt=∑l=1dA-t-1lZtlBt-1+λ,where Yt and Ztl are the DFTs of yt and ztl, respectively. Atl and Bt are the numerator and denominator of the filter updated in the previous frame, respectively. Then the correlation scores yt is obtained by taking the inverse DFT yt=F-1Yt. The estimate of the current target state is obtained by finding the maximum correlation score among the test samples.


## 3. Proposed Discriminative Fusion Correlation Learning

In this section, we introduce the tracking framework of the proposed algorithm, the general scheme of which is described in Figure1. Firstly, the multichannel features are extracted, respectively, from the visible and infrared images according to [14]. Secondly, the proposed discriminative filter selection and the fusion filter learning are applied to get the fusion response map. Finally, the discriminative filters and fusion filters are updated via the tracking result obtained by the response map. We will discuss them specifically below.Figure 1
Discriminative fusion correlation learning for visible and infrared tracking. Blue and green boxes denote the target and background samples extracted from the tracking result, respectively.


### 3.1. Discriminative Filter Selection

According to DCF-based trackers, we obtain the correlation outputg by(5)g=∑l=1dhTl∗sTl,l=1,⋯,d,where sTl and hTl are the target sample and target correlation filter corresponding to the l-th channel feature among d channels, respectively. In this paper, g is selected as a 2D Gaussian function where σ=2.0 [13].Before tracking, we need to choose the optimal target correlation filters in the training step via minimizing (6) as(6)HTl=G-STl∑k=1dS-TkSTk+λ,l=1,⋯,d,where STl, HTl, and G are the DFTs of sTl, hTl, and g, respectively.Different from a single training sample of the target appearance, multiple background samples at different locations around target need to be considered to maintain a stable model. However, extracting multichannel features from each background sample increase computational complex significantly. Moreover, in practice, single channel features from multiple background samples are enough to present satisfied performances. Therefore, in this paper, we extractM background samples randomly in the range of an annulus around the target location [11] and obtain the correlation output g as(7)g=∑m=1MhB∗sBm,m=1,⋯,M,where sBm,m=1,⋯,M denotes the m-th background sample.Similarly, the optimal background correlation filters in the training step are selected via minimizing (8) as(8)HBm=G-SBm∑j=1MS-BjSBj+λ,m=1,⋯,M,where SBm and HBm are the DFTs of sBm and hBm, respectively.While tracking, DFTYt,es of the estimated discriminative correlation score yt,es of the test sample zt is defined as(9)Yt,es=Yt,es,TYt,es,B=∑l=1dA-t-1,TlZtl/Bt-1,T+λ∑m=1MA-t-1,BmZt/Bt-1,B+λ=∑l=1dA-t-1,TlZtlBt-1,BZt∑m=1MA-t-1,BmBt-1,T+λ,where Yt,es,T and Yt,es,B are the DFTs of the target and background correlation scores yt,es,T and yt,es,B, respectively, and Ztl is the DFTs of ztl. At-1,Tl and Bt-1,T denote the numerator and denominator of the filter HTl in (6). At-1,Bm and Bt-1,B denote the numerator and denominator of the filter HBm in (8). Then the discriminative correlation scores yt,es are obtained by taking the inverse DFT yt,es=F-1Yt,es. The estimate of the current target state zt0,es is obtained by finding the maximum correlation score among the test samples as zt0,es→t0:t0=argmaxt⁡yt,es.


### 3.2. Fusion Learning Filter

As proved by Wagner et al. [31], late fusion with early estimation provides better performance than early fusion with late estimation. Based on this conclusion, we use the discriminative correlation filters to obtain the estimate of target location in visible and infrared images, respectively, and then do fusion with the fusion correlation filters. Let zt,es,i,i∈vi,ir denote the estimate of target location of visible vi or infrared ir images. Then we define the region R∘·,• denoting the minimum bounding rectangle that contains the regions of samples · and • in image ∘. Thus, we extract the fusion test samples through zt,i∈Rizt,es,vi,zt,es,ir,i∈vi,ir and define the DFT Yt,fu of the fusion correlation score yt,fu of fusion sample zt,i that is defined as(10)Yt,fu=Yt,T,fuYt,B,fu=∑i∈vi,irαiYt,es,T,i∑i∈vi,irαiYt,es,B,i=∑i∈vi,irαi∑l=1dA-t-1,T,ilZt,il/Bt-1,T,i+λ∑i∈vi,irαi∑m=1MA-t-1,B,imZt,i/Bt-1,B,i+λ=∑i∈vi,irαi∑l=1dA-t-1,T,ilZt,ilBt-1,B,iZt,i∑m=1MA-t-1,B,imBt-1,T,i+λ,where Yt,fu,T and Yt,fu,B are the DFTs of the target and background fusion correlation scores yt,fu,T and yt,fu,B, respectively. zt,il∈Rilzt,es,vi,zt,es,ir,i∈vi,ir in which il is the l-th channel feature of image i. Zt,il and Zt,i are the DFTs of samples zt,il and zt,i, respectively. αi denotes the image weights that are denoted as (11)αi=maxi⁡yt,es,i∑i-∈vi,irmaxi⁡yt,es,i-,where yt,es,i is the correlation score of the i-th image computed by (9).After obtainingYt,fu, the fusion correlation score yt,fu is obtained by taking the inverse DFT yt,fu=F-1Yt,fu. The fusion location of the current target state is obtained by finding the maximum correlation score among the test samples.The whole tracking process of DFCL is summarized in Algorithm1.Algorithm 1:Discriminative fusion correlation learning for visible and infrared tracking.
Input: The t-th visible and infrared images For  t = 1 to number of frames do 1. Crop the samples and extract the l-th (l = 1, · · ·, d) channel features ztl for visible and infrared images, respectively. 2. Compute the discriminative correlation scores yt,es using Eq. (9). 3. Compute the fusion correlation scores yt,fu using Eq. (10). 4. Obtain the tracking result by maximizing yt,fu. 5. Extract the l-th (l = 1, · · ·, d) channel feature sTl of the target samples and the m-th (m = 1, · · ·, M) sample sBl. 6. Update the discriminative correlation filters HTl and HBm using Eq. (6) and Eq. (8), respectively. end for Output: Target result and the discriminative correlation filters HTl and HBm


## 3.1. Discriminative Filter Selection

According to DCF-based trackers, we obtain the correlation outputg by(5)g=∑l=1dhTl∗sTl,l=1,⋯,d,where sTl and hTl are the target sample and target correlation filter corresponding to the l-th channel feature among d channels, respectively. In this paper, g is selected as a 2D Gaussian function where σ=2.0 [13].Before tracking, we need to choose the optimal target correlation filters in the training step via minimizing (6) as(6)HTl=G-STl∑k=1dS-TkSTk+λ,l=1,⋯,d,where STl, HTl, and G are the DFTs of sTl, hTl, and g, respectively.Different from a single training sample of the target appearance, multiple background samples at different locations around target need to be considered to maintain a stable model. However, extracting multichannel features from each background sample increase computational complex significantly. Moreover, in practice, single channel features from multiple background samples are enough to present satisfied performances. Therefore, in this paper, we extractM background samples randomly in the range of an annulus around the target location [11] and obtain the correlation output g as(7)g=∑m=1MhB∗sBm,m=1,⋯,M,where sBm,m=1,⋯,M denotes the m-th background sample.Similarly, the optimal background correlation filters in the training step are selected via minimizing (8) as(8)HBm=G-SBm∑j=1MS-BjSBj+λ,m=1,⋯,M,where SBm and HBm are the DFTs of sBm and hBm, respectively.While tracking, DFTYt,es of the estimated discriminative correlation score yt,es of the test sample zt is defined as(9)Yt,es=Yt,es,TYt,es,B=∑l=1dA-t-1,TlZtl/Bt-1,T+λ∑m=1MA-t-1,BmZt/Bt-1,B+λ=∑l=1dA-t-1,TlZtlBt-1,BZt∑m=1MA-t-1,BmBt-1,T+λ,where Yt,es,T and Yt,es,B are the DFTs of the target and background correlation scores yt,es,T and yt,es,B, respectively, and Ztl is the DFTs of ztl. At-1,Tl and Bt-1,T denote the numerator and denominator of the filter HTl in (6). At-1,Bm and Bt-1,B denote the numerator and denominator of the filter HBm in (8). Then the discriminative correlation scores yt,es are obtained by taking the inverse DFT yt,es=F-1Yt,es. The estimate of the current target state zt0,es is obtained by finding the maximum correlation score among the test samples as zt0,es→t0:t0=argmaxt⁡yt,es.


## 3.2. Fusion Learning Filter

As proved by Wagner et al. [31], late fusion with early estimation provides better performance than early fusion with late estimation. Based on this conclusion, we use the discriminative correlation filters to obtain the estimate of target location in visible and infrared images, respectively, and then do fusion with the fusion correlation filters. Let zt,es,i,i∈vi,ir denote the estimate of target location of visible vi or infrared ir images. Then we define the region R∘·,• denoting the minimum bounding rectangle that contains the regions of samples · and • in image ∘. Thus, we extract the fusion test samples through zt,i∈Rizt,es,vi,zt,es,ir,i∈vi,ir and define the DFT Yt,fu of the fusion correlation score yt,fu of fusion sample zt,i that is defined as(10)Yt,fu=Yt,T,fuYt,B,fu=∑i∈vi,irαiYt,es,T,i∑i∈vi,irαiYt,es,B,i=∑i∈vi,irαi∑l=1dA-t-1,T,ilZt,il/Bt-1,T,i+λ∑i∈vi,irαi∑m=1MA-t-1,B,imZt,i/Bt-1,B,i+λ=∑i∈vi,irαi∑l=1dA-t-1,T,ilZt,ilBt-1,B,iZt,i∑m=1MA-t-1,B,imBt-1,T,i+λ,where Yt,fu,T and Yt,fu,B are the DFTs of the target and background fusion correlation scores yt,fu,T and yt,fu,B, respectively. zt,il∈Rilzt,es,vi,zt,es,ir,i∈vi,ir in which il is the l-th channel feature of image i. Zt,il and Zt,i are the DFTs of samples zt,il and zt,i, respectively. αi denotes the image weights that are denoted as (11)αi=maxi⁡yt,es,i∑i-∈vi,irmaxi⁡yt,es,i-,where yt,es,i is the correlation score of the i-th image computed by (9).After obtainingYt,fu, the fusion correlation score yt,fu is obtained by taking the inverse DFT yt,fu=F-1Yt,fu. The fusion location of the current target state is obtained by finding the maximum correlation score among the test samples.The whole tracking process of DFCL is summarized in Algorithm1.Algorithm 1:Discriminative fusion correlation learning for visible and infrared tracking.
Input: The t-th visible and infrared images For  t = 1 to number of frames do 1. Crop the samples and extract the l-th (l = 1, · · ·, d) channel features ztl for visible and infrared images, respectively. 2. Compute the discriminative correlation scores yt,es using Eq. (9). 3. Compute the fusion correlation scores yt,fu using Eq. (10). 4. Obtain the tracking result by maximizing yt,fu. 5. Extract the l-th (l = 1, · · ·, d) channel feature sTl of the target samples and the m-th (m = 1, · · ·, M) sample sBl. 6. Update the discriminative correlation filters HTl and HBm using Eq. (6) and Eq. (8), respectively. end for Output: Target result and the discriminative correlation filters HTl and HBm


## 4. Experiments

The proposed DFCL algorithm was tested on several challenging real-world sequences, and some qualitative and quantitative analyses were performed on the tracking results in this section.


### 4.1. Experimental Environment and Evaluation Criteria

DFCL was implemented with C++ programming language and.Net Framework 4.0 in Visual Studio 2010 on an Intel Dual-Core 1.70GHz CPU with 4 GB RAM. Two metrics, i.e., location error (pixel) and overlapping rate, are used to evaluate the tracking results quantitatively. The location error is computed aserror=xG-xT2+yG-yT2, where xG,yG and xT,yT are the ground truth (either downloaded from a standard database or located manually) and tracking bounding box centers, respectively. The tracking overlapping rate is defined as overlapping=areaROIG∩ROIT/areaROIG∪ROIT, where ROIG and ROIT denote the ground truth and tracking bounding box, respectively, and area· is the rectangular area function. A smaller location error and a larger overlapping rate indicate higher accuracy and robustness.


### 4.2. Experimental Results

The performance of DFCL was compared with state-of-the-art trackers Struck [32], ODFS [33], STC [34], KCF [35], ROT [36], DCF-based trackers MOSSE [13], DSST [14], fDSST [15], and visible-infrared fusion trackers TSKF [23], MVMKF [27], L1-PF [19], JSR [24], and CSR [25]. Figures 2–6 present the experimental results of the test trackers in challenging visible sequences namedBiker [37],Campus [37],Car [38],Crossroad [39],Hotkettle [39],Inglassandmobile [39],Labman [40],Pedestrian [41], andRunner [38], as well as their corresponding infrared sequencesBiker-ir,Campus-ir,Car-ir,Crossroad-ir,Labman-ir,Hotkettle-ir,Inglassandmobile-ir,Pedestrian-ir, andRunner-ir. Single-sensor trackers were separately tested on visible and the corresponding infrared sequences, while visible-infrared fusion trackers obtain the results with information from both visible and infrared sequences. For the convenience of presentation, some tracking curves are not shown entirely in the Figures. Next, the performance of the trackers in each sequence is described in detail.Figure 2
Tracking performances of the test sequences.Figure 3
Location error (pixel) of the test sequences.Figure 4
Overlapping rate of the test sequences.Figure 5
Average location error (pixel) of the test sequences.Figure 6
Success rate of the test sequences.(a) SequencesBiker andBiker-ir:Biker presents the example of complex background clutters. The target human in the visible sequence encounters similar background disturbance (i.e., bikes), which causes the ODFS, MOSSE, fDSST, TSKF, and MVMKF trackers to drift away from the target. The corresponding infrared sequenceBiker-ir provides temperature information that eliminates the background clutter inBiker. But when the target is approaching another person at around Frame #20, Struck, ODFS, STC, MOSSE, TSKF, and MVMKF do not perform well because they are not able to distinguish target from persons with similar temperature in infrared sequences. Only KCF, ROT, DSST, and our DFCL have achieved precise and robust performances in these sequences.(b) SequencesCampus andCampus-ir: the target inCampus andCampus-ir undergoes background clutters, occlusion, and scale variation. At the beginning ofCampus, ODFS, STC, KCF, and ROT lose the target due to background disturbance. Only TSKF and DFCL perform well, while Struck, fDSST, and MVMKF do not achieve accurate results. Because of the infrared information provided byCampus-ir, fewer test trackers lose tracking when background clutters happen as shown in Figure 2. But Struck, KCF, and ROT mistake another person for the target. As shown in Figure 2, most of the trackers result in tracking failures, whereas DFCL outperforms the others in most metrics (location accuracy and success rate).(c) SequencesCar andCar-ir:Car andCar-ir demonstrate the efficiency of DFCL on coping with heavy occlusions. The target driving car is occluded by lampposts and trees many times, which cause tracking failures of most trackers. Only TSKF, MVMKF, and DFCL are able to handle the occlusion throughout the tracking process in this sequence. As shown in Figure 2, most trackers perform better inCar-ir than inCar because the infrared features can overcome the difficulties of target detection among surrounding similar background. STC, TSKF, MVMKF, and DFCL are able to handle this problem, whereas the result of DFCL is the most accurate, as shown in Figure 2.(d) SequencesCrossroad andCrossroad-ir: the target inCrossroad andCrossroad-ir undergoes heavy background clutters when she crosses the road. While the target is passing by the road lamp, both ODFS and JSR lose the target. Then, when a car passes by the target, Struck, TSKF, and MVMKF drift away from the target. When the target goes toward the sidewalk, most of the trackers are not able to handle the problem of heavy background clutters, but our tracker performs satisfying tracking results as shown in Figures 2–4.(e) SequencesHotkettleandHotkettle-ir: in these sequences, tracking is hard because of the changes of the complex background clutters. Most trackers perform better inHotkettle-ir than inHotkettle for the reason that the temperature diverge makes the hot target more distinct in the cold background. Struck, KCF, DSST, fDSST, and DFCL can achieve robust and accurate tracking performances as shown in Figures 2–4.(f) SequencesInglassandmobile andInglassandmobile-ir: SequencesInglassandmobile andInglassandmobile-ir demonstrate the performances of the 14 trackers under the circumstances of background clutters, illumination changes, and occlusion. As shown in Figure 2, when the illumination changes at around Frames #300, ODFS and fDSST lose the target, and KCF, TSKF, and L1-PF drift a little away from the target. When the target is approaching a tree, the background clutters makes most of trackers cause tracking failures that can be seen from Figure 2. Our DFCL can overcome these challenges and perform well in these sequences.(g) SequencesLabman andLabman-ir: the experiments in SequencesLabman andLabman-ir aim to evaluate the performances on tracking under appearance variation, rotation, scale variation, and background clutter. InLabman, when the target man is walking into the laboratory, ODFS, STC, and MOSSE lose the target. When the man keeps shaking and turning around his head at around Frame #400, KCF, ROT, and DSST cause tracking failures. Also, most trackers achieve better tracking performances inLabman-ir as shown in Figure 2.(h) SequencesPedestrian andPedestrian-ir: the target inPedestrian andPedestrian-ir undergoes heavy background clutters and occlusion. As shown in Figure 2, other trackers result in tracking failures inPedestrian, whereas our tracker shows satisfying performances in terms of both accuracy and robustness. The efficient infrared features extracted fromPedestrian-irensure the tracking successes of Struck, STC, and DFCL, as can be seen from Figures 2–4.(i) SequencesRunner andRunner-ir:Runner andRunner-ir contain examples of heavy occlusion, abrupt movement, and scale variation. The target running man is occluded by lampposts, trees, stone tablet, and bushes many times, resulting in tracking failures of most trackers. Also, the abrupt movement and scale variation cause many trackers to drift away the target in bothRunner andRunner-ir as shown in Figure 2. Once again, our DFCL is able to overcome the above problems and achieve good performances.Figures5 and 6 are included here to demonstrate quantitatively the performances on average location error (pixel) and success rate. The success rate is defined as the number of times success is achieved in the whole tracking process by considering one frame as a success if the overlapping rate exceeds 0.5 [33]. A smaller average location error and a larger success rate indicate increased accuracy and robustness. Figures 5 and 6 show that DFCL performs satisfying most of the tracking sequences.To validate the effectiveness of the discriminative filter selection model of DFCL, we compare the tracker DCL (the proposed DFCL without the fusion learning model) with DFCL and the original DCF tracker MOSSE on visible sequences. The performances shown in Figure7 demonstrate the efficiency of the discriminative filter selection model especially in the sequences with background clutters, i.e., SequencesBiker,Hotkettle,Inglassandmobile, andPedestrian.Figure 7
Average location error (pixel) and success rate of DCL, DFCL, and MOSSE on the test sequences.


## 4.1. Experimental Environment and Evaluation Criteria

DFCL was implemented with C++ programming language and.Net Framework 4.0 in Visual Studio 2010 on an Intel Dual-Core 1.70GHz CPU with 4 GB RAM. Two metrics, i.e., location error (pixel) and overlapping rate, are used to evaluate the tracking results quantitatively. The location error is computed aserror=xG-xT2+yG-yT2, where xG,yG and xT,yT are the ground truth (either downloaded from a standard database or located manually) and tracking bounding box centers, respectively. The tracking overlapping rate is defined as overlapping=areaROIG∩ROIT/areaROIG∪ROIT, where ROIG and ROIT denote the ground truth and tracking bounding box, respectively, and area· is the rectangular area function. A smaller location error and a larger overlapping rate indicate higher accuracy and robustness.


## 4.2. Experimental Results

The performance of DFCL was compared with state-of-the-art trackers Struck [32], ODFS [33], STC [34], KCF [35], ROT [36], DCF-based trackers MOSSE [13], DSST [14], fDSST [15], and visible-infrared fusion trackers TSKF [23], MVMKF [27], L1-PF [19], JSR [24], and CSR [25]. Figures 2–6 present the experimental results of the test trackers in challenging visible sequences namedBiker [37],Campus [37],Car [38],Crossroad [39],Hotkettle [39],Inglassandmobile [39],Labman [40],Pedestrian [41], andRunner [38], as well as their corresponding infrared sequencesBiker-ir,Campus-ir,Car-ir,Crossroad-ir,Labman-ir,Hotkettle-ir,Inglassandmobile-ir,Pedestrian-ir, andRunner-ir. Single-sensor trackers were separately tested on visible and the corresponding infrared sequences, while visible-infrared fusion trackers obtain the results with information from both visible and infrared sequences. For the convenience of presentation, some tracking curves are not shown entirely in the Figures. Next, the performance of the trackers in each sequence is described in detail.Figure 2
Tracking performances of the test sequences.Figure 3
Location error (pixel) of the test sequences.Figure 4
Overlapping rate of the test sequences.Figure 5
Average location error (pixel) of the test sequences.Figure 6
Success rate of the test sequences.(a) SequencesBiker andBiker-ir:Biker presents the example of complex background clutters. The target human in the visible sequence encounters similar background disturbance (i.e., bikes), which causes the ODFS, MOSSE, fDSST, TSKF, and MVMKF trackers to drift away from the target. The corresponding infrared sequenceBiker-ir provides temperature information that eliminates the background clutter inBiker. But when the target is approaching another person at around Frame #20, Struck, ODFS, STC, MOSSE, TSKF, and MVMKF do not perform well because they are not able to distinguish target from persons with similar temperature in infrared sequences. Only KCF, ROT, DSST, and our DFCL have achieved precise and robust performances in these sequences.(b) SequencesCampus andCampus-ir: the target inCampus andCampus-ir undergoes background clutters, occlusion, and scale variation. At the beginning ofCampus, ODFS, STC, KCF, and ROT lose the target due to background disturbance. Only TSKF and DFCL perform well, while Struck, fDSST, and MVMKF do not achieve accurate results. Because of the infrared information provided byCampus-ir, fewer test trackers lose tracking when background clutters happen as shown in Figure 2. But Struck, KCF, and ROT mistake another person for the target. As shown in Figure 2, most of the trackers result in tracking failures, whereas DFCL outperforms the others in most metrics (location accuracy and success rate).(c) SequencesCar andCar-ir:Car andCar-ir demonstrate the efficiency of DFCL on coping with heavy occlusions. The target driving car is occluded by lampposts and trees many times, which cause tracking failures of most trackers. Only TSKF, MVMKF, and DFCL are able to handle the occlusion throughout the tracking process in this sequence. As shown in Figure 2, most trackers perform better inCar-ir than inCar because the infrared features can overcome the difficulties of target detection among surrounding similar background. STC, TSKF, MVMKF, and DFCL are able to handle this problem, whereas the result of DFCL is the most accurate, as shown in Figure 2.(d) SequencesCrossroad andCrossroad-ir: the target inCrossroad andCrossroad-ir undergoes heavy background clutters when she crosses the road. While the target is passing by the road lamp, both ODFS and JSR lose the target. Then, when a car passes by the target, Struck, TSKF, and MVMKF drift away from the target. When the target goes toward the sidewalk, most of the trackers are not able to handle the problem of heavy background clutters, but our tracker performs satisfying tracking results as shown in Figures 2–4.(e) SequencesHotkettleandHotkettle-ir: in these sequences, tracking is hard because of the changes of the complex background clutters. Most trackers perform better inHotkettle-ir than inHotkettle for the reason that the temperature diverge makes the hot target more distinct in the cold background. Struck, KCF, DSST, fDSST, and DFCL can achieve robust and accurate tracking performances as shown in Figures 2–4.(f) SequencesInglassandmobile andInglassandmobile-ir: SequencesInglassandmobile andInglassandmobile-ir demonstrate the performances of the 14 trackers under the circumstances of background clutters, illumination changes, and occlusion. As shown in Figure 2, when the illumination changes at around Frames #300, ODFS and fDSST lose the target, and KCF, TSKF, and L1-PF drift a little away from the target. When the target is approaching a tree, the background clutters makes most of trackers cause tracking failures that can be seen from Figure 2. Our DFCL can overcome these challenges and perform well in these sequences.(g) SequencesLabman andLabman-ir: the experiments in SequencesLabman andLabman-ir aim to evaluate the performances on tracking under appearance variation, rotation, scale variation, and background clutter. InLabman, when the target man is walking into the laboratory, ODFS, STC, and MOSSE lose the target. When the man keeps shaking and turning around his head at around Frame #400, KCF, ROT, and DSST cause tracking failures. Also, most trackers achieve better tracking performances inLabman-ir as shown in Figure 2.(h) SequencesPedestrian andPedestrian-ir: the target inPedestrian andPedestrian-ir undergoes heavy background clutters and occlusion. As shown in Figure 2, other trackers result in tracking failures inPedestrian, whereas our tracker shows satisfying performances in terms of both accuracy and robustness. The efficient infrared features extracted fromPedestrian-irensure the tracking successes of Struck, STC, and DFCL, as can be seen from Figures 2–4.(i) SequencesRunner andRunner-ir:Runner andRunner-ir contain examples of heavy occlusion, abrupt movement, and scale variation. The target running man is occluded by lampposts, trees, stone tablet, and bushes many times, resulting in tracking failures of most trackers. Also, the abrupt movement and scale variation cause many trackers to drift away the target in bothRunner andRunner-ir as shown in Figure 2. Once again, our DFCL is able to overcome the above problems and achieve good performances.Figures5 and 6 are included here to demonstrate quantitatively the performances on average location error (pixel) and success rate. The success rate is defined as the number of times success is achieved in the whole tracking process by considering one frame as a success if the overlapping rate exceeds 0.5 [33]. A smaller average location error and a larger success rate indicate increased accuracy and robustness. Figures 5 and 6 show that DFCL performs satisfying most of the tracking sequences.To validate the effectiveness of the discriminative filter selection model of DFCL, we compare the tracker DCL (the proposed DFCL without the fusion learning model) with DFCL and the original DCF tracker MOSSE on visible sequences. The performances shown in Figure7 demonstrate the efficiency of the discriminative filter selection model especially in the sequences with background clutters, i.e., SequencesBiker,Hotkettle,Inglassandmobile, andPedestrian.Figure 7
Average location error (pixel) and success rate of DCL, DFCL, and MOSSE on the test sequences.


## 5. Conclusion

Discriminative correlation filter- (DCF-) based trackers have the advantage of being computationally efficient and more robust than most of the other state-of-the-art trackers in challenging tracking tasks, thereby making them especially suitable for a variety of real-time challenging applications. However, most of the DCF-based trackers suffer low accuracy due to the lack of diversity information extracted from a single type of spectral image (visible spectrum). Fusion of visible and infrared sensors, one of the typical multisensor cooperation, provides complementarily useful features and consistently helps recognize the target from the background efficiently in visual tracking. For the above reasons, this paper proposes a discriminative fusion correlation learning model to improve DCF-based tracking performance by combining multiple features from visible and infrared imaging sensors. The proposed fusion learning filters are obtained via late fusion with early estimation, in which the performances of the filters are weighted to improve the flexibility of fusion. Moreover, the proposed discriminative filter selection model considers the surrounding background information in order to increase the discriminability of the template filters so as to improve model learning. Numerous real-world video sequences were used to test DFCL and other state-of-the-art algorithms, and here we only selected representative videos for presentation. Experimental results demonstrated that DFCL is highly accurate and robust.



---

*Source: 2437521-2019-05-15.xml*