# Learning Enhanced Feature Responses for Visual Object Tracking

Authors: Runqing Zhang; Chunxiao Fan; Yue Ming
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

Visual object tracking is an important topic in computer vision, which has successfully utilized pretrained convolutional neural networks, such as VGG and ResNet. However, the features extracted by these pretrained models are high dimensional, and the redundant feature channels reduce target localization and scale estimation precision, leading to tracking drifting. In this paper, a novel visual object tracking method, called learning enhanced feature responses tracking (LEFRT), is proposed, which adopts the target-specific features to enhance target localization and scale estimation responses. First, a channel attention module, called target-specific network (TSNet), is presented to reduce the redundant feature channels. Secondly, the scale estimation network (SCENet) is introduced to extract spatial structural features to generate a more precise response for the scale estimation. Extensive experiments on six tracking benchmarks, including LaSOT, GOT-10k, TrackingNet, OTB-2013, OTB-2015, and TC-128, demonstrate that the proposed algorithm can effectively improve the precision and speed of visual object tracking. LEFRT achieves 90.4% precision and a 71.2% success rate on the OTB-2015 dataset, improving the tracking methods based on the pretrained features.


---

## Body

## 1. Introduction

Visual object tracking is one of the fundamental tasks in computer vision, widely used in the civil and military fields, such as image segmentation [1], intelligent transportation [2], object detection [3], and human-computer interaction [4].Recently, pretrained deep features bring state-of-the-art performance to existing trackers, effectively separating foreground objects from the background. However, the tracking task aims to distinguish the object/foreground and the background, and the pretrained features with high dimensions are redundant. What is more, the pretrained VGG and ResNet are trained for the preset object categories, but the target in a tracking task can be an arbitrary object. As shown in Figure1, the response generated by the pretrained features may focus on the object in the pretrained set. At the same time, tracking methods based on the correlation filters utilized the convolutional features and the a priori scale coefficients to estimate the shape of the targets. The a priori scale coefficients are set as the discrete constant term parameters, which leads to the precision limit. Therefore, it is of great importance to exploit more compact features to represent specific targets.Figure 1
Comparison of pretrained (VGG-16) features responses and our target-specific features responses in LaSOT [5] test sets. For example, the left figure is the original frame in the first row, including the target (yoyo) and the human (pretrained distractor). The central figure is the responses generated from VGG-16, which focuses on the human head. The right figure is the response from our methods, which focuses on the target (yoyo). (a) Yoyo-19. (b) Responses from pretrained features. (c) Ours. (d) Coin-6. (e) Responses from pretrained features. (f) Ours. (g) Bus-2. (h) Responses from pretrained features. (i) Ours.In this paper, we propose a novel visual object tracking algorithm called learning enhanced feature responses tracking (LEFRT), as shown in Figure2, including the TSNet and the SCENet. We proposed the TSNet to reduce the redundant feature channels for the tracking methods based on the correlation filters and the pretrained features. What is more, we initialed the TSNet with the target’s appearance in the first frame to reverse the channels paying attention to the given targets. We proposed the SCENet to regress the target’s shape. The bounding box regression was introduced to accurately predict the target’s finer shape. What is more, the convolutional features lacked spatial topology information for shape estimation, and we introduced 2D-RNN layers to establish the spatial relation for the target’s structure.Figure 2
The framework of our work.Specifically, we extract the features using the pretrained model firstly. Secondly, the TSNet is proposed to reduce the redundant channels of the features, and then these features are then sent into the correlation filter for target center localization. We calculate the attention of each feature channel to the target and activate the channels that focus on the given target. Thirdly, we regress the proposal target region on the SCENet to estimate the target’s scale with the predicted target center. We construct the spatial structural backbone to establish the spatial relationship of the target image region. The main contributions of our proposed algorithm are summarized as follows:(1)
We propose the TSNet to generate channel attention and select the effective channels for arbitrary targets, significantly reducing the redundant feature channels and locating the specific target more precisely.(2)
2D RNN feature is utilized to represent the spatial structure of the target for scale estimation in the SCENet, which describes the spatial relationship and enhances the response of the target’s boundary.(3)
The proposed algorithm has achieved superior performance on different datasets with many challenges, including OTB-2013 [6], OTB-2015 [7], TC-128 [8], TrackingNet [9], LaSOT [5], and GOT-10k [10]. The experimental results show that our method effectively improves the precision of both the target’s center and scale estimation.The rest of the paper is organized as follows: we survey related works on visual object tracking in Section2. Our proposed algorithm is described in Section 3. The following Section 4 demonstrates the experimental evaluations and results. Finally, we conclude the paper in Section 5.


## 2. Related Work

In this section, we introduce the researches closely related to this work. First, we discuss the features utilized in visual object tracking. Second, we present the scale estimation methods used to estimate the target’s shape during tracking.


### 2.1. Features in Visual Object Tracking

Tracking algorithms utilize features to represent targets’ appearance, mainly including traditional manual features, features extracted from end-to-end networks, and features extracted from pretrained models.Traditional manual features are efficient for visual object tracking. The color attributes method was proposed as a fast color representation, which allowed the tracker to operate at more than 100 frames per second without significant loss in precision [11]. KCF [12] and DSST [13] utilized fHOG to extract features efficiently from different scales and achieved real-time speed. However, traditional manual features perform weakly in complex tracking scenes, such as targets with color change and blurred appearance.Features extracted from the end-to-end network are adaptive for given tasks and scenes. The end-to-end tracking networks include deep feature methods and matching methods. On the one hand, deep feature methods predict accurate location during tracking. Multi-Domain Convolutional Neural Networks (MDNet) [14] used a multiple-domain network to obtain the convolutional neural network (CNN) features, which achieved 94.8\% precision on the OTB50 dataset [7]. To enhance the backbone network without detriment to the tracking speed, DML proposed a mutual-learning-based methodology to use a heavyweight network [15]. On the other hand, matching methods, such as SiamFC [16], Reinforced Attentional Representation (RAR) [17], and SiamRPN [18], utilized Siamese convolution networks to achieve high-speed trackers. End-to-end learning methods use high-dimensional convolutional features to accurately separate foreground and nonsemantic background, but they are computationally expensive to train huge amounts of data on semantic negative pairs [19, 20].Features that distinguish objects from the pretrained model are efficient without online learning and update. Some correlation filters methods [21–23] combine pretrained features from VGG [24, 25] and ResNet [26, 27]. Joint Group Feature Selection and Discriminative Filter (GFSDCF) [26] combined features of ResNet with correlation filters to achieve group channel responses. ROI Pooled Correlation Filters (RPCF) [28] introduced deep network and traditional handcraft features for target localization. However, an arbitrary target may not be in the training dataset, making the pretrained models focus on background objects. The pretrained models utilize categories information but lack spatial information, which leads to an inaccurate scale estimation.


### 2.2. Scale Estimation

The relative motion changes the target’s state in the complex tracking scene, including spatial location and scale. The consequent spatial location can be predicted by temporal estimation, such as trajectory [29] and RNN [30], but scale estimation is also important for the target’s shape prediction. Tracking methods based on the correlation filter [13, 31–33] applied a multiscale correlation filter with prior parameters to select the scale of the maximum response. Tracking methods based on the regression network used a deep network to predict the bounding box of the target. Some tracking methods introduced the region proposal network (RPN) [18, 19] to estimate the scale variation based on the Siamese network. In addition, ATOM [34] used Intersection over Union (IoU) predictors to estimate IoU for each proposal bounding box. However, deep regression networks, IoU, and RPN networks rely on a huge amount of training data and cost much time. The prior scale coefficient is used in the IoU network and the one-dimensional scale filter, making the scale prediction inaccurate. The object recognition estimated the object’s shape accurately [35]. In our work, we learn a spatial response from the first frame and use fast regression to predict the scale variation accurately.


## 2.1. Features in Visual Object Tracking

Tracking algorithms utilize features to represent targets’ appearance, mainly including traditional manual features, features extracted from end-to-end networks, and features extracted from pretrained models.Traditional manual features are efficient for visual object tracking. The color attributes method was proposed as a fast color representation, which allowed the tracker to operate at more than 100 frames per second without significant loss in precision [11]. KCF [12] and DSST [13] utilized fHOG to extract features efficiently from different scales and achieved real-time speed. However, traditional manual features perform weakly in complex tracking scenes, such as targets with color change and blurred appearance.Features extracted from the end-to-end network are adaptive for given tasks and scenes. The end-to-end tracking networks include deep feature methods and matching methods. On the one hand, deep feature methods predict accurate location during tracking. Multi-Domain Convolutional Neural Networks (MDNet) [14] used a multiple-domain network to obtain the convolutional neural network (CNN) features, which achieved 94.8\% precision on the OTB50 dataset [7]. To enhance the backbone network without detriment to the tracking speed, DML proposed a mutual-learning-based methodology to use a heavyweight network [15]. On the other hand, matching methods, such as SiamFC [16], Reinforced Attentional Representation (RAR) [17], and SiamRPN [18], utilized Siamese convolution networks to achieve high-speed trackers. End-to-end learning methods use high-dimensional convolutional features to accurately separate foreground and nonsemantic background, but they are computationally expensive to train huge amounts of data on semantic negative pairs [19, 20].Features that distinguish objects from the pretrained model are efficient without online learning and update. Some correlation filters methods [21–23] combine pretrained features from VGG [24, 25] and ResNet [26, 27]. Joint Group Feature Selection and Discriminative Filter (GFSDCF) [26] combined features of ResNet with correlation filters to achieve group channel responses. ROI Pooled Correlation Filters (RPCF) [28] introduced deep network and traditional handcraft features for target localization. However, an arbitrary target may not be in the training dataset, making the pretrained models focus on background objects. The pretrained models utilize categories information but lack spatial information, which leads to an inaccurate scale estimation.


## 2.2. Scale Estimation

The relative motion changes the target’s state in the complex tracking scene, including spatial location and scale. The consequent spatial location can be predicted by temporal estimation, such as trajectory [29] and RNN [30], but scale estimation is also important for the target’s shape prediction. Tracking methods based on the correlation filter [13, 31–33] applied a multiscale correlation filter with prior parameters to select the scale of the maximum response. Tracking methods based on the regression network used a deep network to predict the bounding box of the target. Some tracking methods introduced the region proposal network (RPN) [18, 19] to estimate the scale variation based on the Siamese network. In addition, ATOM [34] used Intersection over Union (IoU) predictors to estimate IoU for each proposal bounding box. However, deep regression networks, IoU, and RPN networks rely on a huge amount of training data and cost much time. The prior scale coefficient is used in the IoU network and the one-dimensional scale filter, making the scale prediction inaccurate. The object recognition estimated the object’s shape accurately [35]. In our work, we learn a spatial response from the first frame and use fast regression to predict the scale variation accurately.


## 3. Method

This section introduces our novel visual object tracking algorithm for learning enhanced feature response tracking (LEFRT). First, we introduce the framework of our tracking method. Second, we propose the TSNet to select effective features. Finally, we propose a novel SCENet for the scale estimation.


### 3.1. Tracking Framework

Generally, the proposed LEFRT includes TSNet and SCENet for location prediction and scale estimation separately. As shown in Figure2, the TSNet reduces redundant feature channels during the training process and retains features focusing on the given target. The SCENet containing 2D RNN layers is trained to enhance the boundary response of the target. During the tracking process, the correlation filters (CF) compute the center coordinates of the target with the input of target-specific features. The scale estimation module uses the spatial structural information in 2D RNN features to estimate the scale variation of the target.For the TSNet, the pretrained feature extracted model based on VGG-16 is trained with large-scale image database ImageNet. The TSNet fine-tunes the pretrained model using the target appearance from the first frame that focuses on the specific target. The TSNet learns the channel weights of the pretrained features to pay more attention to the given target in each sequence. We fine-tune the TSNet in the first frame and apply the models in the subsequent frames. We extract the pretrained features and apply the features’ channel weights from the trained TSNet to select the most relative channels to the given target when a new frame comes. After feature selection, we use the correlation filter to localize the center location of the target.For the SCENet, we train the model utilizing a set of annotated video sequences. We combine the CNN layers trained by the Stochastic Gradient Decent (SGD) method and 2D RNN layers to model spatial-relationship between local object areas and obtain a confidence map to estimate object scale. We apply RNNs to modeling the object’s structure and use such structure information to enhance the response of objects in the confidence map. The RNNs include four directed acyclic graphs: southeast, southwest, northwest, and northeast. With these directed acyclic graphs, we can perform forward and backward propagation on each directed acyclic graph. The recurrent layer is trained by the method in Section3.3. We apply the bounding box regression technique [36] to estimate the target’s scale.We also apply a redetection model for our tracking method. We utilize a detection model every 500 frames to obtain ten candidates in the current frame. Then, we compare the similarity between the candidates and the target from the first frame with cosine distance, and the most similar candidate is initialized as the new target template.


### 3.2. TSNet

Given an arbitrary sequence, we train our TSNet in the first frame to obtain feature channel weights, and then we generate target-specific features with these weights to track the target in the subsequent frames.


#### 3.2.1. Target-Specific Features Extraction

The TSNet is proposed to learn the feature channel weights for the pretrained model. We utilize the VGG-16 network as the pretrained model to extract basic features and discriminate between different videos objects. To reduce the redundant channels that focus on the target, we use the target’s appearance in the first frame to train our TSNet. We crop the frame with the given ground truth to achieve the training samples during the training process in the first frame. After extracting the features from the pretrained model, we input the pretrained features into our TSNet.We regress the pretrained featuresϕi,j to the Gaussian labels yi,j=e−i2+j2/2σ2, where i,j represents the relative coordinate difference of samples against the target center, and σ is the width of the Gaussian kernel. The loss function for regression can be formulated as(1)Lf=||yi,j−Wn∗ϕi,j||2+λf||Wn||2,where ∗ denotes the convolution operation, Wn is the learnable parameters, and λf is a regularization parameter. The center of 2D Gaussian kernel is aligned with the specific target center. For each sequence, we train our TSNet with the feature in the first frame extracted from the pretrained model with given epochs. To alleviate target’s deformation, we add a dropblock layer after the convolution layer. The closer the image is to the target center, the higher response Wn∗ϕi,j the features produce.In order to estimate the importance of each pretrained feature channel in producing the regression response, the learnable parameterWn represents the contribution of feature ϕi,j to fit the Gaussian label in equation (1). We utilize the average value of the weight Wn to compute the importance of the n-th channels Δn:(2)Δn=avgWn.Here, we select theC channels with positive values Δn>0, which have a positive relationship with the target loss. Then, we obtain the target-specific features of the given target in the first frame. In the following video frames, we use the selected channel features to localize the target in the search area.


#### 3.2.2. Target Center Localization

During the tracking process in the subsequent frames, the target-specific features are extracted and input to the correlation filter for target center localization. The appearance model of the correlation filters trackersWCF is trained on a M×N pixels sample xi,j. The training samples are generated by circular shifts of xm,n, where m,n∈1,2,…,M×1,2,…,N with the Gaussian label ym,n. In our method, we adopt the target-specific features ϕxi,j instead of the image samples xi,j. The filter WCF can be solved by minimizing the loss function\cite{kcf}:(3)LCF=ϕxi,j⊙WCF+λCFWCF2=∑m,nWCFTϕxm,n−ym,n2+λCF∑m,nWCF2,where λCF is a nonnegative regularization parameter. The minimizer of equation (3) has a closed-form, which is acquired by the following formula:(4)WCF=Xm,nTXm,n+λCFI−1Xm,nTYm,n,where the matrix Xm,n has one sample xm,n per row, and each element of Ym,n is a label ym,n. Expressed by fast Fourier transform and inverse fast Fourier transform, the solution WCF can be acquired by the following formula:(5)WCF=−F−1Fxm,n⊙Fym,nFxm,n∗⊙Fxm,n+λCF,where F, F−1 denote the fast Fourier transform and inverse fast Fourier transform, respectively. xm,n∗ is a complex-conjugate of xm,n and ⊙ denotes the element-wise product. The training process is concluded in Algorithm 1.During tracking, the response mapfz is calculated by(6)fz=−F−1Fxm,n⊙Fym,n⊙Fzm,nFxm,n∗⊙Fxm,n+λCF.Here,zm,n is the patch cropped from the new frame. And the correlation filters place the peak of the response map as the tracking target. In our method, we adopt the target-specific features ϕxm,n instead of the image samples x in equation (3). The TSNet activates the feature channels that focus on the given target and reduces the feature channels that focus on the background.Algorithm 1: Training target center localization model (TSNet).
Input:The first frame of the sequence,I1;The initial target state,S1;The pretrained VGGNet (VGG-16),w1Output:Target-specific features channel weight,Δn;Weights of correlation filtersWCF;(1)
Crop the search region and extract the pretrained featuresϕi,j using S1 for I1;(2)
Generate the Gaussian label mapyi,j;(3)
ObtainWn by equation (1);(4)
Compute channel weightsΔn by equation (2);(5)
Obtain target-specific features using channel weightsΔn and pretrained features;(6)
Train correlation filters by equation (4);(7)
ReturnWCF


### 3.3. SCENet

After target localization, we propose SCENet to estimate the target’s scale and shape. We utilize 2D RNN to establish the spatial relationship between the target and its surrounding regions for scale estimation, which can effectively enhance the boundary responses of the target.


#### 3.3.1. Spatial Structure Feature

RNNs are developed for modeling dependencies in sequential data. Given an input sequencextt=1,2,…,T of length T, the hidden layer ht and output layer ft at each time step t are calculated with(7)ht=σUxt+Wht−1,ft=ψVht,where U, W and V represent weight matrices of the hidden layer, the current hidden layer, and the output layer separately; and σ and ψ are nonlinear activation functions. Since the inputs are progressively stored in hidden layers, RNNs can model long-range contextual dependencies among the sequence elements.Different from one-dimension sequential data, the self-structure of two-dimensional image data is encoded in an undirected cyclic graph. 2D RNN establishes spatial structural relationships from four directions. For each direction, the output of the convolution layerϕxi,j for the image region xi,j is input, and the spatial structural relationship can be computed in the 2D RNN unit as follows:(8)hi,j=σURϕxi,j+∑αWRhα,fi,j=ψVRhi,j,where hi,j is the state of the hidden layer, and fi,j is the state of the output layer. UR, WR, and VR are the learnable weight matrices of the hidden layer, the current hidden layer, and the output layer separately. σ and ψ are the nonlinear activation functions. α is the neighborhood region of xi,j. The parameters of the neighborhood region WR establish the relationship between the neighborhood state hα and the current state hi,j. The undirected cyclic graph is decomposed into four directions, including southeast, southwest, northwest, and northeast. With equation (8), we can perform forward and backward passes on one directed acyclic graph.The candidates are sent into the convolution layers during the tracking process to obtain the initial features. With equation (2), our method can generate a new hidden state from the sample to store the representation information, simultaneously containing the location and neighborhood region information. Considering the summation of all hidden layers for the four directions, the forward pass can be calculated as(9)hi,jm=σURmϕxi,j+∑αWRmhαmxi,j,fi,jm=ψ∑m4VRmhi,jm,where URm, WRm, and VRm are matrix parameters for direction m, and the hαmxi,j is the hidden state of the forward neighborhood of sample xi,j.


### 3.4. Scale Estimation

Using the target center location predicted from Section3.3, we can crop k anchor boxes similar with faster-RCNN [37]. For each anchor box pi,i∈1,2,…,k. We extract the spatial structure feature ϕpi and then input ϕpi into the bounding box regression model. The loss function of the bounding box regression model is(10)W∗=argminW∗L2t∗iw^∗Tϕpi+λbb||w^∗||2,where w^∗ is the learnable parameter. ∗∈x,y,w,h denotes the box’s center coordinates and its width and height. λbb is the constant parameter. Label ti can be calculated as follows:(11)tx=x^−xwty=y^−yhtw=logw^wth=logh^h,where variables x^ and x are for the predicted box and anchor box, respectively (likewise for y,w,h). The bounding box regression model predicts k bounding boxes, and we average them to obtain the final bounding box.We propose a fast vision of the SCENet, using the proposal bounding box predicted by correlation filter from Section3.3 instead of anchor boxes. In the fast vision fSCENet, the feature extraction and bounding box regression model only run once, but the final bounding box’s precision might decrease.


## 3.1. Tracking Framework

Generally, the proposed LEFRT includes TSNet and SCENet for location prediction and scale estimation separately. As shown in Figure2, the TSNet reduces redundant feature channels during the training process and retains features focusing on the given target. The SCENet containing 2D RNN layers is trained to enhance the boundary response of the target. During the tracking process, the correlation filters (CF) compute the center coordinates of the target with the input of target-specific features. The scale estimation module uses the spatial structural information in 2D RNN features to estimate the scale variation of the target.For the TSNet, the pretrained feature extracted model based on VGG-16 is trained with large-scale image database ImageNet. The TSNet fine-tunes the pretrained model using the target appearance from the first frame that focuses on the specific target. The TSNet learns the channel weights of the pretrained features to pay more attention to the given target in each sequence. We fine-tune the TSNet in the first frame and apply the models in the subsequent frames. We extract the pretrained features and apply the features’ channel weights from the trained TSNet to select the most relative channels to the given target when a new frame comes. After feature selection, we use the correlation filter to localize the center location of the target.For the SCENet, we train the model utilizing a set of annotated video sequences. We combine the CNN layers trained by the Stochastic Gradient Decent (SGD) method and 2D RNN layers to model spatial-relationship between local object areas and obtain a confidence map to estimate object scale. We apply RNNs to modeling the object’s structure and use such structure information to enhance the response of objects in the confidence map. The RNNs include four directed acyclic graphs: southeast, southwest, northwest, and northeast. With these directed acyclic graphs, we can perform forward and backward propagation on each directed acyclic graph. The recurrent layer is trained by the method in Section3.3. We apply the bounding box regression technique [36] to estimate the target’s scale.We also apply a redetection model for our tracking method. We utilize a detection model every 500 frames to obtain ten candidates in the current frame. Then, we compare the similarity between the candidates and the target from the first frame with cosine distance, and the most similar candidate is initialized as the new target template.


## 3.2. TSNet

Given an arbitrary sequence, we train our TSNet in the first frame to obtain feature channel weights, and then we generate target-specific features with these weights to track the target in the subsequent frames.


### 3.2.1. Target-Specific Features Extraction

The TSNet is proposed to learn the feature channel weights for the pretrained model. We utilize the VGG-16 network as the pretrained model to extract basic features and discriminate between different videos objects. To reduce the redundant channels that focus on the target, we use the target’s appearance in the first frame to train our TSNet. We crop the frame with the given ground truth to achieve the training samples during the training process in the first frame. After extracting the features from the pretrained model, we input the pretrained features into our TSNet.We regress the pretrained featuresϕi,j to the Gaussian labels yi,j=e−i2+j2/2σ2, where i,j represents the relative coordinate difference of samples against the target center, and σ is the width of the Gaussian kernel. The loss function for regression can be formulated as(1)Lf=||yi,j−Wn∗ϕi,j||2+λf||Wn||2,where ∗ denotes the convolution operation, Wn is the learnable parameters, and λf is a regularization parameter. The center of 2D Gaussian kernel is aligned with the specific target center. For each sequence, we train our TSNet with the feature in the first frame extracted from the pretrained model with given epochs. To alleviate target’s deformation, we add a dropblock layer after the convolution layer. The closer the image is to the target center, the higher response Wn∗ϕi,j the features produce.In order to estimate the importance of each pretrained feature channel in producing the regression response, the learnable parameterWn represents the contribution of feature ϕi,j to fit the Gaussian label in equation (1). We utilize the average value of the weight Wn to compute the importance of the n-th channels Δn:(2)Δn=avgWn.Here, we select theC channels with positive values Δn>0, which have a positive relationship with the target loss. Then, we obtain the target-specific features of the given target in the first frame. In the following video frames, we use the selected channel features to localize the target in the search area.


### 3.2.2. Target Center Localization

During the tracking process in the subsequent frames, the target-specific features are extracted and input to the correlation filter for target center localization. The appearance model of the correlation filters trackersWCF is trained on a M×N pixels sample xi,j. The training samples are generated by circular shifts of xm,n, where m,n∈1,2,…,M×1,2,…,N with the Gaussian label ym,n. In our method, we adopt the target-specific features ϕxi,j instead of the image samples xi,j. The filter WCF can be solved by minimizing the loss function\cite{kcf}:(3)LCF=ϕxi,j⊙WCF+λCFWCF2=∑m,nWCFTϕxm,n−ym,n2+λCF∑m,nWCF2,where λCF is a nonnegative regularization parameter. The minimizer of equation (3) has a closed-form, which is acquired by the following formula:(4)WCF=Xm,nTXm,n+λCFI−1Xm,nTYm,n,where the matrix Xm,n has one sample xm,n per row, and each element of Ym,n is a label ym,n. Expressed by fast Fourier transform and inverse fast Fourier transform, the solution WCF can be acquired by the following formula:(5)WCF=−F−1Fxm,n⊙Fym,nFxm,n∗⊙Fxm,n+λCF,where F, F−1 denote the fast Fourier transform and inverse fast Fourier transform, respectively. xm,n∗ is a complex-conjugate of xm,n and ⊙ denotes the element-wise product. The training process is concluded in Algorithm 1.During tracking, the response mapfz is calculated by(6)fz=−F−1Fxm,n⊙Fym,n⊙Fzm,nFxm,n∗⊙Fxm,n+λCF.Here,zm,n is the patch cropped from the new frame. And the correlation filters place the peak of the response map as the tracking target. In our method, we adopt the target-specific features ϕxm,n instead of the image samples x in equation (3). The TSNet activates the feature channels that focus on the given target and reduces the feature channels that focus on the background.Algorithm 1: Training target center localization model (TSNet).
Input:The first frame of the sequence,I1;The initial target state,S1;The pretrained VGGNet (VGG-16),w1Output:Target-specific features channel weight,Δn;Weights of correlation filtersWCF;(1)
Crop the search region and extract the pretrained featuresϕi,j using S1 for I1;(2)
Generate the Gaussian label mapyi,j;(3)
ObtainWn by equation (1);(4)
Compute channel weightsΔn by equation (2);(5)
Obtain target-specific features using channel weightsΔn and pretrained features;(6)
Train correlation filters by equation (4);(7)
ReturnWCF


## 3.2.1. Target-Specific Features Extraction

The TSNet is proposed to learn the feature channel weights for the pretrained model. We utilize the VGG-16 network as the pretrained model to extract basic features and discriminate between different videos objects. To reduce the redundant channels that focus on the target, we use the target’s appearance in the first frame to train our TSNet. We crop the frame with the given ground truth to achieve the training samples during the training process in the first frame. After extracting the features from the pretrained model, we input the pretrained features into our TSNet.We regress the pretrained featuresϕi,j to the Gaussian labels yi,j=e−i2+j2/2σ2, where i,j represents the relative coordinate difference of samples against the target center, and σ is the width of the Gaussian kernel. The loss function for regression can be formulated as(1)Lf=||yi,j−Wn∗ϕi,j||2+λf||Wn||2,where ∗ denotes the convolution operation, Wn is the learnable parameters, and λf is a regularization parameter. The center of 2D Gaussian kernel is aligned with the specific target center. For each sequence, we train our TSNet with the feature in the first frame extracted from the pretrained model with given epochs. To alleviate target’s deformation, we add a dropblock layer after the convolution layer. The closer the image is to the target center, the higher response Wn∗ϕi,j the features produce.In order to estimate the importance of each pretrained feature channel in producing the regression response, the learnable parameterWn represents the contribution of feature ϕi,j to fit the Gaussian label in equation (1). We utilize the average value of the weight Wn to compute the importance of the n-th channels Δn:(2)Δn=avgWn.Here, we select theC channels with positive values Δn>0, which have a positive relationship with the target loss. Then, we obtain the target-specific features of the given target in the first frame. In the following video frames, we use the selected channel features to localize the target in the search area.


## 3.2.2. Target Center Localization

During the tracking process in the subsequent frames, the target-specific features are extracted and input to the correlation filter for target center localization. The appearance model of the correlation filters trackersWCF is trained on a M×N pixels sample xi,j. The training samples are generated by circular shifts of xm,n, where m,n∈1,2,…,M×1,2,…,N with the Gaussian label ym,n. In our method, we adopt the target-specific features ϕxi,j instead of the image samples xi,j. The filter WCF can be solved by minimizing the loss function\cite{kcf}:(3)LCF=ϕxi,j⊙WCF+λCFWCF2=∑m,nWCFTϕxm,n−ym,n2+λCF∑m,nWCF2,where λCF is a nonnegative regularization parameter. The minimizer of equation (3) has a closed-form, which is acquired by the following formula:(4)WCF=Xm,nTXm,n+λCFI−1Xm,nTYm,n,where the matrix Xm,n has one sample xm,n per row, and each element of Ym,n is a label ym,n. Expressed by fast Fourier transform and inverse fast Fourier transform, the solution WCF can be acquired by the following formula:(5)WCF=−F−1Fxm,n⊙Fym,nFxm,n∗⊙Fxm,n+λCF,where F, F−1 denote the fast Fourier transform and inverse fast Fourier transform, respectively. xm,n∗ is a complex-conjugate of xm,n and ⊙ denotes the element-wise product. The training process is concluded in Algorithm 1.During tracking, the response mapfz is calculated by(6)fz=−F−1Fxm,n⊙Fym,n⊙Fzm,nFxm,n∗⊙Fxm,n+λCF.Here,zm,n is the patch cropped from the new frame. And the correlation filters place the peak of the response map as the tracking target. In our method, we adopt the target-specific features ϕxm,n instead of the image samples x in equation (3). The TSNet activates the feature channels that focus on the given target and reduces the feature channels that focus on the background.Algorithm 1: Training target center localization model (TSNet).
Input:The first frame of the sequence,I1;The initial target state,S1;The pretrained VGGNet (VGG-16),w1Output:Target-specific features channel weight,Δn;Weights of correlation filtersWCF;(1)
Crop the search region and extract the pretrained featuresϕi,j using S1 for I1;(2)
Generate the Gaussian label mapyi,j;(3)
ObtainWn by equation (1);(4)
Compute channel weightsΔn by equation (2);(5)
Obtain target-specific features using channel weightsΔn and pretrained features;(6)
Train correlation filters by equation (4);(7)
ReturnWCF


## 3.3. SCENet

After target localization, we propose SCENet to estimate the target’s scale and shape. We utilize 2D RNN to establish the spatial relationship between the target and its surrounding regions for scale estimation, which can effectively enhance the boundary responses of the target.


### 3.3.1. Spatial Structure Feature

RNNs are developed for modeling dependencies in sequential data. Given an input sequencextt=1,2,…,T of length T, the hidden layer ht and output layer ft at each time step t are calculated with(7)ht=σUxt+Wht−1,ft=ψVht,where U, W and V represent weight matrices of the hidden layer, the current hidden layer, and the output layer separately; and σ and ψ are nonlinear activation functions. Since the inputs are progressively stored in hidden layers, RNNs can model long-range contextual dependencies among the sequence elements.Different from one-dimension sequential data, the self-structure of two-dimensional image data is encoded in an undirected cyclic graph. 2D RNN establishes spatial structural relationships from four directions. For each direction, the output of the convolution layerϕxi,j for the image region xi,j is input, and the spatial structural relationship can be computed in the 2D RNN unit as follows:(8)hi,j=σURϕxi,j+∑αWRhα,fi,j=ψVRhi,j,where hi,j is the state of the hidden layer, and fi,j is the state of the output layer. UR, WR, and VR are the learnable weight matrices of the hidden layer, the current hidden layer, and the output layer separately. σ and ψ are the nonlinear activation functions. α is the neighborhood region of xi,j. The parameters of the neighborhood region WR establish the relationship between the neighborhood state hα and the current state hi,j. The undirected cyclic graph is decomposed into four directions, including southeast, southwest, northwest, and northeast. With equation (8), we can perform forward and backward passes on one directed acyclic graph.The candidates are sent into the convolution layers during the tracking process to obtain the initial features. With equation (2), our method can generate a new hidden state from the sample to store the representation information, simultaneously containing the location and neighborhood region information. Considering the summation of all hidden layers for the four directions, the forward pass can be calculated as(9)hi,jm=σURmϕxi,j+∑αWRmhαmxi,j,fi,jm=ψ∑m4VRmhi,jm,where URm, WRm, and VRm are matrix parameters for direction m, and the hαmxi,j is the hidden state of the forward neighborhood of sample xi,j.


## 3.3.1. Spatial Structure Feature

RNNs are developed for modeling dependencies in sequential data. Given an input sequencextt=1,2,…,T of length T, the hidden layer ht and output layer ft at each time step t are calculated with(7)ht=σUxt+Wht−1,ft=ψVht,where U, W and V represent weight matrices of the hidden layer, the current hidden layer, and the output layer separately; and σ and ψ are nonlinear activation functions. Since the inputs are progressively stored in hidden layers, RNNs can model long-range contextual dependencies among the sequence elements.Different from one-dimension sequential data, the self-structure of two-dimensional image data is encoded in an undirected cyclic graph. 2D RNN establishes spatial structural relationships from four directions. For each direction, the output of the convolution layerϕxi,j for the image region xi,j is input, and the spatial structural relationship can be computed in the 2D RNN unit as follows:(8)hi,j=σURϕxi,j+∑αWRhα,fi,j=ψVRhi,j,where hi,j is the state of the hidden layer, and fi,j is the state of the output layer. UR, WR, and VR are the learnable weight matrices of the hidden layer, the current hidden layer, and the output layer separately. σ and ψ are the nonlinear activation functions. α is the neighborhood region of xi,j. The parameters of the neighborhood region WR establish the relationship between the neighborhood state hα and the current state hi,j. The undirected cyclic graph is decomposed into four directions, including southeast, southwest, northwest, and northeast. With equation (8), we can perform forward and backward passes on one directed acyclic graph.The candidates are sent into the convolution layers during the tracking process to obtain the initial features. With equation (2), our method can generate a new hidden state from the sample to store the representation information, simultaneously containing the location and neighborhood region information. Considering the summation of all hidden layers for the four directions, the forward pass can be calculated as(9)hi,jm=σURmϕxi,j+∑αWRmhαmxi,j,fi,jm=ψ∑m4VRmhi,jm,where URm, WRm, and VRm are matrix parameters for direction m, and the hαmxi,j is the hidden state of the forward neighborhood of sample xi,j.


## 3.4. Scale Estimation

Using the target center location predicted from Section3.3, we can crop k anchor boxes similar with faster-RCNN [37]. For each anchor box pi,i∈1,2,…,k. We extract the spatial structure feature ϕpi and then input ϕpi into the bounding box regression model. The loss function of the bounding box regression model is(10)W∗=argminW∗L2t∗iw^∗Tϕpi+λbb||w^∗||2,where w^∗ is the learnable parameter. ∗∈x,y,w,h denotes the box’s center coordinates and its width and height. λbb is the constant parameter. Label ti can be calculated as follows:(11)tx=x^−xwty=y^−yhtw=logw^wth=logh^h,where variables x^ and x are for the predicted box and anchor box, respectively (likewise for y,w,h). The bounding box regression model predicts k bounding boxes, and we average them to obtain the final bounding box.We propose a fast vision of the SCENet, using the proposal bounding box predicted by correlation filter from Section3.3 instead of anchor boxes. In the fast vision fSCENet, the feature extraction and bounding box regression model only run once, but the final bounding box’s precision might decrease.


## 4. Experiments

To evaluate the performance of the proposed tracking method, we conduct extensive experiments on three public tracking datasets, including OTB-2013 [6], OTB-2015 [7] dataset, TC-128 [8] dataset, LaSOT [5] dataset, TrackingNet dataset [9], and GOT-10k dataset [10] as shown in Figure 3. Firstly, we detail the implementations and parameters in our experiments and introduce the datasets. Then, in Section 4.1, we evaluate the effectiveness of our tracking method by providing ablation experiments. In Sections 4.2–4.6, we evaluate the performance of the quantitative comparison with several state-of-the-art tracking methods on the OTB-2013/OTB-2015, TC-128, LaSOT, GOT-10k, and TrackingNet datasets, respectively.Figure 3
Some samples of the sequences in our evaluation dataset.Our tracking method is implemented in Matlab using MatconvNet, which runs at an average speed of 30.7 fps with a 2.6 GHz Intel Core i7 CPU with 16 GB RAM and a Titan Xp GPU. In the proposed target-specific features, we utilize the Con43 and Conv41 features from the pretrained model VGG-16. The value of regulation parameterλf in equation (1) is set as 0.01. The training epoch for the TSNet is 100, and the learning rate is 5e − 7. The activation functions σ and ψ in the SCENet are tanh and Sigmoid, respectively. The constant parameter λbb in equation (9) is set as 0.012.The following datasets are used in our experiments:OTB-2013/OTB-2015 [6, 7]: OTB-2013/OTB-2015 are popular datasets for visual object tracking. OTB-2013 and OTB-2015 contain 50 and 100 fully annotated videos, respectively, with substantial variations. We adopt the straightforward One-Pass Evaluation (OPE) [6] as the performance evaluation method. For the performance evaluation metrics, we use precision plots and success plots. Following the protocol in the OTB-2015 benchmark, the threshold of 20 pixels and success rate (Succ.) are presented to compare the representative precision plots and success plots of tracking methods. The main metrics for performance evaluation in the tracking tasks include the precision plot and the success plot. The value in the precision plot can be calculated as 1/M∑j=1M∑i=1Nj1cij−c^ij2<τp/Nj, while the value in the success rate can be calculated as 1/M∑j=1M∑i=1NjIoUcij,c^ij<τs/Nj. The ground-truth of the target’s state is cij, while the predicted value is c^ij. M is the total number of video sequences in the current dataset. Nj represents the total number of images in the j-th video sequence. IoU is the Intersection over Union function to calculate the overlap rate between the ground-truth and the predicted value. When τp=20, the value in the precision plot is chosen as the precision. When τs=0.5, the value in the success plot is chosen as the success rate.TC-128 [8]: the TC-128 dataset contains 128 fully annotated color video sequences with many challenging factors. The dataset includes sequences from two main sources: 50 previous studies and 78 new collections.LaSOT [5]: the LaSOT dataset has longer sequences with an average of 2,500 frames per sequence, including a public dataset and test set for tracking. All the videos in the LaSOT test set are annotated with eight challenges, including Aspect Ration Change (ARC), Low Resolution (LR), Out-of-view (OV), Fast Motion (FM), Full Occlusion (FO), Scale Variation (SV), Rotation (RO), and Deformation (DEF). We evaluate our approach on the test set consisting of 280 videos, and online model adaption is crucial for this dataset.GOT-10k [10]: GOT-10k is a large benchmarking dataset for visual object tracking. It has 180 test video sequences in 84 categories. The main metrics of GOT-10k include AO, SR0.5, SR0.75, and speed (Hz). SR0.5 and SR0.75 are the SR metric with a threshold of 0.5 and 0.75 separately.TrackingNet [9]: the TrackingNet Dataset contains 511 sequences for visual object tracking in the wild. The main metrics of TrackingNet Dataset include Precision, NormalizePrecision, and Success. The precision and success are measured separately as the centers’ distance and the Intersection over Union (IoU). The NormalizePrecision is measured as the weighted precision, and the weights are calculated by the size of the ground truth bounding box.


### 4.1. Ablation Studies

In this section, we conduct the ablation study on OTB-2013 and OTB-2015 to evaluate the effectiveness of each component. The main components contain the TSNet and the SECNet. To show the contribution of the components, four variants of LEFRT are designed:Baseline: the original correlation filter tracker (see Section 3.2.2) uses the pretrained VGG-16 features, and the scale estimation is based on the prior scale coefficients.Baseline + TSNet: a variant without the proposed scale estimation strategy, which applies the TSNet to localize the target center.Baseline + TSNet + SCENet-: a variant replaces the SCENet in scale estimation strategy with CNN layers SCENet-LEFRT-HS: an accelerated version of LEFRT, which applies fSCENet to estimate the scale.As shown in Table1, we evaluate three variants on the OTB-2015 dataset and compare their tracking performance with the proposed LEFRT. All the variants perform worse than LEFRT in terms of tracking accuracy. Pre-Train-T extracts the features from the Conv41 and Conv43 layers of pretrained VGG-16 models, and it estimates the target’s scale in the subsequent frames using the prior scale coefficient. The baseline method utilized the pretrained CNN feature from VGG-16. The high dimensional pretrained features are trained for the classification task, and not each channel is useful to distinguish the object and the background. Redundant features increase the parameters of the model and pay too much attention to the background image, resulting in model drift. Our TSNet selected the feature channels that focused on the target itself and inactivated the redundant channels, improving the tracking model’s speed and precision. The baseline method utilized the scale pyramid with the a priori scale coefficients, which limited the accuracy of prediction results. We utilized the bounding box regression to estimate a more accurate target’s shape, What is more, the 2D-RNN features contain rich self-structure information of the target, which improves the ability of features to describe the target shape. In the comparison, the accelerated version LEFRT-HS using the standard high-speed features for scale estimation obtains a higher speed than LEFRT. The tracking speed of LEFRT-HS is similar to TS, which indicates that our scale estimation strategy has a fast speed.Table 1
The precision scores (Prec20), the success rate (Succ.) scores and speed (fps) on the OTB-2015 dataset. The best results are displayed in bold fonts.
MethodsTSNetSCENetPrec20Succ.Speed (FPS)Baseline81.165.646.1Baseline + TSNet√86.569.749.0Baseline + TSNet + SCENet√SCENet-86.570.135.0LEFRT-HS√fSCENet90.470.642.3LEFRT√fSCENet90.471.230.7


### 4.2. Evaluation on OTB-2013/OTB-2015 Dataset

We conduct the quantitative, qualitative, and challenging attributes on the OTB-2015 dataset. What is more, we also conduct the ablation study experiment on the OTB-2013 dataset.


#### 4.2.1. Quantitative Comparison

We compare the precision score and success rate obtained by our LEFRT and several state-of-the-art tracking methods including SiamFC++ [38], ASRCF [24], ARCF [39], UDT [20], ECO [40], CNN-SVM [41], fDSST [13], ATTF [21], TADT [42], MLT [43], CFNet [44], SiamFC [16], Siam-tri [45], EACOFT [22], and Dinesh et al. [23]. Since our tracking method adopts target-specific features to select the channels of pretrained features, we choose ASRCF, ARCF, UDT, CNN-SVM, and ECO, because they use pretrained models to extract features. The pretrained features contain redundant channels, which lead to the model drifting for the tracking task. AdaCFNet, TADT, and MLT applied attention mechanism to optimize features. CFNet, SiamFC, Siamtri, fDSST, MLT, and RAR use prior scale coefficient to estimate the scale, limiting the accuracy for target’s shape estimation.Table2 demonstrates the comparisons of the precision scores, success rate, and speeds obtained by our LEFRT and other state-of-the-art tracking methods. Our LEFRT performs favorably against other state-of-the-art tracking methods in precision scores and success rate. Compared with high-speed trackers based on correlation filters such as fDSST, our LEFRT improves accuracy in precision scores and success rate noticeably. Compared with the trackers based on the correlation filters with deep learning features, such as ASRCF and ARCF, our LEFRT shows performance superiority. The proposed scale estimation strategy provides high accuracy to track the target with scale variation compared with the other methods.Table 2
The precision score and the success rate (Succ.) on the OTB-2013 and OTB-2015 datasets. The best results are displayed in bold fonts.
MethodsOTB-2013OTB-2015PrecisionSucc.PrecisionSucc.CNN-SVM76.058.281.455.4fDSST74.055.469.352.0STDCF63.9———MLT—62.174.559.3SiamFC70.852.268.449.8TADT80.266.578.465.0ASRCF86.071.484.168.8ARCF77.163.974.861.2UDT73.160.868.757.9SiamFC++———68.3EACOFT———67.6AFFT-50——89.969.3Dinesh et al.84.671.283.753.5LEFRT91.373.590.471.8


#### 4.2.2. Challenges Attributes

In this subsection, we compare the success plots between our LEFRT and other tracking methods with eleven challenging attributes, including background clutter (BC), deformation (DEF), fast motion (FM), in-plane rotation (IR), low resolution (LR), illumination variation, (IV), motion blur (MB), occlusion (OCC), out-of-plane rotation (OR), out of view (OV), and scale variation (SV).In Figure4, compared with the state-of-the-art tracking methods, our LEFRT obtains much better performance under the most challenging attributes, beneficial for target-specific features. Targets that did not get trained in the pretrain process can still be tracked robustly, because our TSNet dropped the redundant feature channels for the background objects. Our LEFRT also achieves significant performance improvements under SV. This proves that the proposed scale estimation effectively predicts the target’s size. For the deformation attribute, LEFRT performs worse than ECO and ASRCF since the model is sensitive to the deformation of the targets. Even so, LEFRT obtains a higher tracking accuracy than ECO and ASRCF on the whole dataset.Figure 4
The success plots on the OTB-2015 dataset for eleven challenging attributes, including background clutter, deformation, fast motion, in-plane rotation, low resolution, illumination variation, motion blur, occlusion, out-of-plane rotation, out of view, and scale variation. Only shown are the top ten performance tracking methods.


#### 4.2.3. Qualitative Comparison

In Figure5, we qualitatively compare the performance obtained by our LEFRT, ASRCF [24], ECO [40], ARCF [39], TADT [42], CFNet [44], and SiamFC [16] on five challenging sequences.Figure 5
Qualitative results of our LEFRT, ASRCF [24], ECO [40], ARCF [39], TADT [42], CFNet [44], and SiamFC [16] on OTB-2015 dataset (from top to down: CarScale, Ironman, Matrix, MotorRolling, and Skiing, respectively).Our LEFRT accurately tracks the object in terms of both positions and scale for the most challenging sequences, while most tracking methods fail to locate the target positions or incorrectly estimate the target scale. For the sequences of CarScale (row 1), the compared tracking methods locate the target position correctly, but they only discriminate a part of the object instead of the whole object when they undergo the large-scale variation. At the same time, our LEFRT correctly estimates both the position and scales of the object. For the sequences of Ironman and Matrix (row 2 and row 3), the most compared tracking methods drift away because of the significant illumination variation and occlusion. In contrast, our LEFRT successfully handles these challenges and accurately tracks the object despite the complex backgrounds. In the sequences of MotorRolling and Skiing (row 4 and row 5), the compared tracking methods are hard to realize the stable tracking when encountering fast motion and significant rotation, while our LEFRT keeps robust tracking of the object throughout the sequences.


### 4.3. Evaluation on TC-128 Dataset

We compare our LEFRT with several state-of-the-art tracking methods that have publicly available results on the TC-128, including ASRCF [24], ARCF [39], ECO [40], DSST [13], TADT [42], UDT [20], GFSDCF [26], and IGSSRTCF [46]. As shown in Table 3, we can observe that our LEFRT achieves the best performance in both precision plots and success plots among all the compared tracking methods. LEFRT outperforms the other tracking methods that use deep features,Table 3
Precision plots and success plots on TC-128 dataset.
MethodsPrecisionSuccess rateUDT [20]0.3100.103DSST [13]0.5430.477ARCF [39]0.7020.646TADT [42]0.7650.703ASRCF [24]0.7840.723ECO [40]0.7890.729GFSDCF [26]0.8310.605IGSSRTCF [46]0.7770.573Ours0.8410.755That is, there are ECO, ASRCF, and TADT, with a relative improvement of 0.6% and 0.8% compared with ECO, respectively. Compared with the tracking methods based on end-to-end networks, such as SiamFC and CFNet, our LEFRT achieves higher tracking accuracy, and the visual results are shown in Figure6.Figure 6
Qualitative results of our LEFRT, ASRCF [24], ECO [40], ARCF [39], TADT [42], DSST [13], and UDT [20] on TC-128 dataset (from top to down: Basketball, Iceskater, Jogging, Singer1, and Skyjumping, respectively).


### 4.4. Evaluation on LaSOT Dataset

We conduct the quantitative comparison and the challenge attributes on the LaSOT dataset.


#### 4.4.1. Quantitative Comparison

We compare our LEFRT with several state-of-the-art tracking methods that have publicly available results on the LaSOT, including ATOM [34], SiamFC++ [38], Ocean [47], ASRCF [24], ARCF [39], ECO [40], SiamFC [16], CFNet [44], MCCT [33], TADT [42], UDT [20], SiamDW [48], and SiamRPN++ [19]. Table 4 shows the precision plots and the success plots of the comparisons between our LEFRT and the state-of-the-art tracking methods. Our LEFRT ranks first with a precision of 0.522 and a success rate of 0.541 in this dataset, verifying the effectiveness of the proposed target-specific features and scale estimation strategy. Ocean trains a special model for the LaSOT dataset and achieves better performance than ours. However, the model of the proposed method is universal and does not need to be fine-tuned for each dataset.Table 4
The precision scores, the precision (threshold = 25), and success rate (threshold = 0.5) on the LaSOT test dataset.
MethodsPrecisionSuccess rateSiamFC++0.5120.537Ocean0.5660.560ATOM0.4790.515SiamRPN++0.4670.496SiamDW0.3780.385TADT0.3700.397ASRCF0.3280.344ARCF0.3020.324MCCT0.2360.259UDT0.2450.250CFNet0.2630.275ECO0.3020.324SiamFC0.3310.336Ours0.5220.541


#### 4.4.2. Challenge Attributes

We further analyze the performance of LEFRT under different challenges on the LaSOT test dataset.For the most attributes in LaSOT test dataset, our LEFRT performs the first rank against other compared tracking methods. Benefiting from localization based on the target-specific features, our LEFRT achieves high precision in most cases in Figure7. At the same time, our LEFRT can obtain the best performance in scale variation. Compared with other tracking methods that utilize the prior scale parameters, our SCENet predicts a more accurate target’s shape. Our LEFRT performs worse than SiamFC++ for deformation, which is likely that the feature focuses on the target’s appearance in the first frame, and the model is sensitive to the target deformation. Even so, LEFRT obtains a higher tracking accuracy than others on the whole dataset. The visual results are shown in Figure 8.Figure 7
The precision and success rate plots (top ten) on the LaSOT test dataset for challenging attributes.Figure 8
Qualitative results of our LEFRT, ASRCF [24], ECO [40], ARCF [39], TADT [42], SiamRPN++ [19], and SiamDW [16] on LaSOT test dataset (from top to down: Basketball-6, Bear-4, Bicycle-9, Bicycle-18, and Bird-15, respectively).


### 4.5. Evaluation on GOT10K Dataset

In Table5, we compare our LEFRT with several state-of-the-art tracking methods that have publicly available results on the GOT-10K, including SiamFC++ [38], Ocean [47], TADT [42], SiamDW [48], SiamRPN++ [19], ASRCF [24], and Autotrack [49]. Our LEFRT achieves 0.619 AO, 0.721 SR0.5, and 0.477 SR0.75 separately, which is better than most of the methods based on the correlation filter and Siamese network. The pretrained features utilized in the correlation filter lead to the model drifting, which decrease the precision and success rate. Compared with the end-to-end Siamese network, the proposed LEFRT utilized the incremental update to avoid model drifting by the deformation targets. Our method is not as good as Ocean, because Ocean utilizes anchor-free image segmentation for target shape estimation, and the metrics of GOT-10k pay more attention on the overlap rate. Same as the LaSOT dataset, Ocean also trains a special model for the GOT-10k dataset. Compared with Ocean, the model of the proposed method is more comprehensive.Table 5
The precision scores, the average overlap (AO), success rate (SR) SR0.5, SR0.75, and speed (Hz) on the GOT-10k dataset.
MethodsAOSR0.5SR0.75Speed (FPS)SiamFC++0.6100.7160.46340.65Ocean0.6490.7280.59722.79SiamRPN++0.6110.7170.49234.05SiamDW0.6070.7310.47627.37TADT0.3670.3890.13011.39ASRCF0.3130.3170.11311.11AutoTrack0.2860.2770.10915.37Ours0.5220.7210.47721.19


### 4.6. Evaluation on TrackingNet Dataset

In Table6, we compare our LEFRT with several state-of-the-art tracking methods on the TrackingNet, including SiamFC++ [38], Ocean [47], TADT [42], SiamDW [48], SiamRPN++ [19], ASRCF [24], and Autotrack [49]. The proposed method has the advantage in precision and achieves 71.23% Precision and 81.92% NormalizePrecision separately. Compared with the state-of-the-art tracking methods based on the correlation filers, our method improves the effectiveness of the pretrained features. However, the speed of LEFRT may decrease because the time complexity of 2D-RNN is higher than the convolutional layer. Different from the results on other datasets, Ocean achieves a lower overlap rate and obtains a lower Success score than ours. We check the results and find that Ocean trained independent models for each of the other datasets except TrackingNet dataset. Because the TrackingNet dataset does not provide the ground truth file, the Ocean cannot fine-tune the model specifically. And in this experiment, we have to test the Ocean tracker using its OTB2015 model. The result proves that the comprehensiveness of our method is higher. SiamDW utilized the ResNet as the feature extraction backbone, which has deeper network structure. SiamDW enlarged the receptive field for the convolutional layer, which enlarged the feature size and stride. Therefore, the estimation for the target’s shape could be carried out on a larger feature map, which achieves a higher success rate than ours.Table 6
The precision, normalize precision, and success on the TrackingNet dataset.
MethodsPrecisionNormalize precisionSuccess rateSiamFC++70.8881.0674.33Ocean68.2879.0169.22SiamRPN++69.3879.9873.30SiamDW63.7576.0679.92TADT53.9967.1159.28AutoTrack44.0855.5657.63Ours71.2381.9275.71


## 4.1. Ablation Studies

In this section, we conduct the ablation study on OTB-2013 and OTB-2015 to evaluate the effectiveness of each component. The main components contain the TSNet and the SECNet. To show the contribution of the components, four variants of LEFRT are designed:Baseline: the original correlation filter tracker (see Section 3.2.2) uses the pretrained VGG-16 features, and the scale estimation is based on the prior scale coefficients.Baseline + TSNet: a variant without the proposed scale estimation strategy, which applies the TSNet to localize the target center.Baseline + TSNet + SCENet-: a variant replaces the SCENet in scale estimation strategy with CNN layers SCENet-LEFRT-HS: an accelerated version of LEFRT, which applies fSCENet to estimate the scale.As shown in Table1, we evaluate three variants on the OTB-2015 dataset and compare their tracking performance with the proposed LEFRT. All the variants perform worse than LEFRT in terms of tracking accuracy. Pre-Train-T extracts the features from the Conv41 and Conv43 layers of pretrained VGG-16 models, and it estimates the target’s scale in the subsequent frames using the prior scale coefficient. The baseline method utilized the pretrained CNN feature from VGG-16. The high dimensional pretrained features are trained for the classification task, and not each channel is useful to distinguish the object and the background. Redundant features increase the parameters of the model and pay too much attention to the background image, resulting in model drift. Our TSNet selected the feature channels that focused on the target itself and inactivated the redundant channels, improving the tracking model’s speed and precision. The baseline method utilized the scale pyramid with the a priori scale coefficients, which limited the accuracy of prediction results. We utilized the bounding box regression to estimate a more accurate target’s shape, What is more, the 2D-RNN features contain rich self-structure information of the target, which improves the ability of features to describe the target shape. In the comparison, the accelerated version LEFRT-HS using the standard high-speed features for scale estimation obtains a higher speed than LEFRT. The tracking speed of LEFRT-HS is similar to TS, which indicates that our scale estimation strategy has a fast speed.Table 1
The precision scores (Prec20), the success rate (Succ.) scores and speed (fps) on the OTB-2015 dataset. The best results are displayed in bold fonts.
MethodsTSNetSCENetPrec20Succ.Speed (FPS)Baseline81.165.646.1Baseline + TSNet√86.569.749.0Baseline + TSNet + SCENet√SCENet-86.570.135.0LEFRT-HS√fSCENet90.470.642.3LEFRT√fSCENet90.471.230.7


## 4.2. Evaluation on OTB-2013/OTB-2015 Dataset

We conduct the quantitative, qualitative, and challenging attributes on the OTB-2015 dataset. What is more, we also conduct the ablation study experiment on the OTB-2013 dataset.


### 4.2.1. Quantitative Comparison

We compare the precision score and success rate obtained by our LEFRT and several state-of-the-art tracking methods including SiamFC++ [38], ASRCF [24], ARCF [39], UDT [20], ECO [40], CNN-SVM [41], fDSST [13], ATTF [21], TADT [42], MLT [43], CFNet [44], SiamFC [16], Siam-tri [45], EACOFT [22], and Dinesh et al. [23]. Since our tracking method adopts target-specific features to select the channels of pretrained features, we choose ASRCF, ARCF, UDT, CNN-SVM, and ECO, because they use pretrained models to extract features. The pretrained features contain redundant channels, which lead to the model drifting for the tracking task. AdaCFNet, TADT, and MLT applied attention mechanism to optimize features. CFNet, SiamFC, Siamtri, fDSST, MLT, and RAR use prior scale coefficient to estimate the scale, limiting the accuracy for target’s shape estimation.Table2 demonstrates the comparisons of the precision scores, success rate, and speeds obtained by our LEFRT and other state-of-the-art tracking methods. Our LEFRT performs favorably against other state-of-the-art tracking methods in precision scores and success rate. Compared with high-speed trackers based on correlation filters such as fDSST, our LEFRT improves accuracy in precision scores and success rate noticeably. Compared with the trackers based on the correlation filters with deep learning features, such as ASRCF and ARCF, our LEFRT shows performance superiority. The proposed scale estimation strategy provides high accuracy to track the target with scale variation compared with the other methods.Table 2
The precision score and the success rate (Succ.) on the OTB-2013 and OTB-2015 datasets. The best results are displayed in bold fonts.
MethodsOTB-2013OTB-2015PrecisionSucc.PrecisionSucc.CNN-SVM76.058.281.455.4fDSST74.055.469.352.0STDCF63.9———MLT—62.174.559.3SiamFC70.852.268.449.8TADT80.266.578.465.0ASRCF86.071.484.168.8ARCF77.163.974.861.2UDT73.160.868.757.9SiamFC++———68.3EACOFT———67.6AFFT-50——89.969.3Dinesh et al.84.671.283.753.5LEFRT91.373.590.471.8


### 4.2.2. Challenges Attributes

In this subsection, we compare the success plots between our LEFRT and other tracking methods with eleven challenging attributes, including background clutter (BC), deformation (DEF), fast motion (FM), in-plane rotation (IR), low resolution (LR), illumination variation, (IV), motion blur (MB), occlusion (OCC), out-of-plane rotation (OR), out of view (OV), and scale variation (SV).In Figure4, compared with the state-of-the-art tracking methods, our LEFRT obtains much better performance under the most challenging attributes, beneficial for target-specific features. Targets that did not get trained in the pretrain process can still be tracked robustly, because our TSNet dropped the redundant feature channels for the background objects. Our LEFRT also achieves significant performance improvements under SV. This proves that the proposed scale estimation effectively predicts the target’s size. For the deformation attribute, LEFRT performs worse than ECO and ASRCF since the model is sensitive to the deformation of the targets. Even so, LEFRT obtains a higher tracking accuracy than ECO and ASRCF on the whole dataset.Figure 4
The success plots on the OTB-2015 dataset for eleven challenging attributes, including background clutter, deformation, fast motion, in-plane rotation, low resolution, illumination variation, motion blur, occlusion, out-of-plane rotation, out of view, and scale variation. Only shown are the top ten performance tracking methods.


### 4.2.3. Qualitative Comparison

In Figure5, we qualitatively compare the performance obtained by our LEFRT, ASRCF [24], ECO [40], ARCF [39], TADT [42], CFNet [44], and SiamFC [16] on five challenging sequences.Figure 5
Qualitative results of our LEFRT, ASRCF [24], ECO [40], ARCF [39], TADT [42], CFNet [44], and SiamFC [16] on OTB-2015 dataset (from top to down: CarScale, Ironman, Matrix, MotorRolling, and Skiing, respectively).Our LEFRT accurately tracks the object in terms of both positions and scale for the most challenging sequences, while most tracking methods fail to locate the target positions or incorrectly estimate the target scale. For the sequences of CarScale (row 1), the compared tracking methods locate the target position correctly, but they only discriminate a part of the object instead of the whole object when they undergo the large-scale variation. At the same time, our LEFRT correctly estimates both the position and scales of the object. For the sequences of Ironman and Matrix (row 2 and row 3), the most compared tracking methods drift away because of the significant illumination variation and occlusion. In contrast, our LEFRT successfully handles these challenges and accurately tracks the object despite the complex backgrounds. In the sequences of MotorRolling and Skiing (row 4 and row 5), the compared tracking methods are hard to realize the stable tracking when encountering fast motion and significant rotation, while our LEFRT keeps robust tracking of the object throughout the sequences.


## 4.2.1. Quantitative Comparison

We compare the precision score and success rate obtained by our LEFRT and several state-of-the-art tracking methods including SiamFC++ [38], ASRCF [24], ARCF [39], UDT [20], ECO [40], CNN-SVM [41], fDSST [13], ATTF [21], TADT [42], MLT [43], CFNet [44], SiamFC [16], Siam-tri [45], EACOFT [22], and Dinesh et al. [23]. Since our tracking method adopts target-specific features to select the channels of pretrained features, we choose ASRCF, ARCF, UDT, CNN-SVM, and ECO, because they use pretrained models to extract features. The pretrained features contain redundant channels, which lead to the model drifting for the tracking task. AdaCFNet, TADT, and MLT applied attention mechanism to optimize features. CFNet, SiamFC, Siamtri, fDSST, MLT, and RAR use prior scale coefficient to estimate the scale, limiting the accuracy for target’s shape estimation.Table2 demonstrates the comparisons of the precision scores, success rate, and speeds obtained by our LEFRT and other state-of-the-art tracking methods. Our LEFRT performs favorably against other state-of-the-art tracking methods in precision scores and success rate. Compared with high-speed trackers based on correlation filters such as fDSST, our LEFRT improves accuracy in precision scores and success rate noticeably. Compared with the trackers based on the correlation filters with deep learning features, such as ASRCF and ARCF, our LEFRT shows performance superiority. The proposed scale estimation strategy provides high accuracy to track the target with scale variation compared with the other methods.Table 2
The precision score and the success rate (Succ.) on the OTB-2013 and OTB-2015 datasets. The best results are displayed in bold fonts.
MethodsOTB-2013OTB-2015PrecisionSucc.PrecisionSucc.CNN-SVM76.058.281.455.4fDSST74.055.469.352.0STDCF63.9———MLT—62.174.559.3SiamFC70.852.268.449.8TADT80.266.578.465.0ASRCF86.071.484.168.8ARCF77.163.974.861.2UDT73.160.868.757.9SiamFC++———68.3EACOFT———67.6AFFT-50——89.969.3Dinesh et al.84.671.283.753.5LEFRT91.373.590.471.8


## 4.2.2. Challenges Attributes

In this subsection, we compare the success plots between our LEFRT and other tracking methods with eleven challenging attributes, including background clutter (BC), deformation (DEF), fast motion (FM), in-plane rotation (IR), low resolution (LR), illumination variation, (IV), motion blur (MB), occlusion (OCC), out-of-plane rotation (OR), out of view (OV), and scale variation (SV).In Figure4, compared with the state-of-the-art tracking methods, our LEFRT obtains much better performance under the most challenging attributes, beneficial for target-specific features. Targets that did not get trained in the pretrain process can still be tracked robustly, because our TSNet dropped the redundant feature channels for the background objects. Our LEFRT also achieves significant performance improvements under SV. This proves that the proposed scale estimation effectively predicts the target’s size. For the deformation attribute, LEFRT performs worse than ECO and ASRCF since the model is sensitive to the deformation of the targets. Even so, LEFRT obtains a higher tracking accuracy than ECO and ASRCF on the whole dataset.Figure 4
The success plots on the OTB-2015 dataset for eleven challenging attributes, including background clutter, deformation, fast motion, in-plane rotation, low resolution, illumination variation, motion blur, occlusion, out-of-plane rotation, out of view, and scale variation. Only shown are the top ten performance tracking methods.


## 4.2.3. Qualitative Comparison

In Figure5, we qualitatively compare the performance obtained by our LEFRT, ASRCF [24], ECO [40], ARCF [39], TADT [42], CFNet [44], and SiamFC [16] on five challenging sequences.Figure 5
Qualitative results of our LEFRT, ASRCF [24], ECO [40], ARCF [39], TADT [42], CFNet [44], and SiamFC [16] on OTB-2015 dataset (from top to down: CarScale, Ironman, Matrix, MotorRolling, and Skiing, respectively).Our LEFRT accurately tracks the object in terms of both positions and scale for the most challenging sequences, while most tracking methods fail to locate the target positions or incorrectly estimate the target scale. For the sequences of CarScale (row 1), the compared tracking methods locate the target position correctly, but they only discriminate a part of the object instead of the whole object when they undergo the large-scale variation. At the same time, our LEFRT correctly estimates both the position and scales of the object. For the sequences of Ironman and Matrix (row 2 and row 3), the most compared tracking methods drift away because of the significant illumination variation and occlusion. In contrast, our LEFRT successfully handles these challenges and accurately tracks the object despite the complex backgrounds. In the sequences of MotorRolling and Skiing (row 4 and row 5), the compared tracking methods are hard to realize the stable tracking when encountering fast motion and significant rotation, while our LEFRT keeps robust tracking of the object throughout the sequences.


## 4.3. Evaluation on TC-128 Dataset

We compare our LEFRT with several state-of-the-art tracking methods that have publicly available results on the TC-128, including ASRCF [24], ARCF [39], ECO [40], DSST [13], TADT [42], UDT [20], GFSDCF [26], and IGSSRTCF [46]. As shown in Table 3, we can observe that our LEFRT achieves the best performance in both precision plots and success plots among all the compared tracking methods. LEFRT outperforms the other tracking methods that use deep features,Table 3
Precision plots and success plots on TC-128 dataset.
MethodsPrecisionSuccess rateUDT [20]0.3100.103DSST [13]0.5430.477ARCF [39]0.7020.646TADT [42]0.7650.703ASRCF [24]0.7840.723ECO [40]0.7890.729GFSDCF [26]0.8310.605IGSSRTCF [46]0.7770.573Ours0.8410.755That is, there are ECO, ASRCF, and TADT, with a relative improvement of 0.6% and 0.8% compared with ECO, respectively. Compared with the tracking methods based on end-to-end networks, such as SiamFC and CFNet, our LEFRT achieves higher tracking accuracy, and the visual results are shown in Figure6.Figure 6
Qualitative results of our LEFRT, ASRCF [24], ECO [40], ARCF [39], TADT [42], DSST [13], and UDT [20] on TC-128 dataset (from top to down: Basketball, Iceskater, Jogging, Singer1, and Skyjumping, respectively).


## 4.4. Evaluation on LaSOT Dataset

We conduct the quantitative comparison and the challenge attributes on the LaSOT dataset.


### 4.4.1. Quantitative Comparison

We compare our LEFRT with several state-of-the-art tracking methods that have publicly available results on the LaSOT, including ATOM [34], SiamFC++ [38], Ocean [47], ASRCF [24], ARCF [39], ECO [40], SiamFC [16], CFNet [44], MCCT [33], TADT [42], UDT [20], SiamDW [48], and SiamRPN++ [19]. Table 4 shows the precision plots and the success plots of the comparisons between our LEFRT and the state-of-the-art tracking methods. Our LEFRT ranks first with a precision of 0.522 and a success rate of 0.541 in this dataset, verifying the effectiveness of the proposed target-specific features and scale estimation strategy. Ocean trains a special model for the LaSOT dataset and achieves better performance than ours. However, the model of the proposed method is universal and does not need to be fine-tuned for each dataset.Table 4
The precision scores, the precision (threshold = 25), and success rate (threshold = 0.5) on the LaSOT test dataset.
MethodsPrecisionSuccess rateSiamFC++0.5120.537Ocean0.5660.560ATOM0.4790.515SiamRPN++0.4670.496SiamDW0.3780.385TADT0.3700.397ASRCF0.3280.344ARCF0.3020.324MCCT0.2360.259UDT0.2450.250CFNet0.2630.275ECO0.3020.324SiamFC0.3310.336Ours0.5220.541


### 4.4.2. Challenge Attributes

We further analyze the performance of LEFRT under different challenges on the LaSOT test dataset.For the most attributes in LaSOT test dataset, our LEFRT performs the first rank against other compared tracking methods. Benefiting from localization based on the target-specific features, our LEFRT achieves high precision in most cases in Figure7. At the same time, our LEFRT can obtain the best performance in scale variation. Compared with other tracking methods that utilize the prior scale parameters, our SCENet predicts a more accurate target’s shape. Our LEFRT performs worse than SiamFC++ for deformation, which is likely that the feature focuses on the target’s appearance in the first frame, and the model is sensitive to the target deformation. Even so, LEFRT obtains a higher tracking accuracy than others on the whole dataset. The visual results are shown in Figure 8.Figure 7
The precision and success rate plots (top ten) on the LaSOT test dataset for challenging attributes.Figure 8
Qualitative results of our LEFRT, ASRCF [24], ECO [40], ARCF [39], TADT [42], SiamRPN++ [19], and SiamDW [16] on LaSOT test dataset (from top to down: Basketball-6, Bear-4, Bicycle-9, Bicycle-18, and Bird-15, respectively).


## 4.4.1. Quantitative Comparison

We compare our LEFRT with several state-of-the-art tracking methods that have publicly available results on the LaSOT, including ATOM [34], SiamFC++ [38], Ocean [47], ASRCF [24], ARCF [39], ECO [40], SiamFC [16], CFNet [44], MCCT [33], TADT [42], UDT [20], SiamDW [48], and SiamRPN++ [19]. Table 4 shows the precision plots and the success plots of the comparisons between our LEFRT and the state-of-the-art tracking methods. Our LEFRT ranks first with a precision of 0.522 and a success rate of 0.541 in this dataset, verifying the effectiveness of the proposed target-specific features and scale estimation strategy. Ocean trains a special model for the LaSOT dataset and achieves better performance than ours. However, the model of the proposed method is universal and does not need to be fine-tuned for each dataset.Table 4
The precision scores, the precision (threshold = 25), and success rate (threshold = 0.5) on the LaSOT test dataset.
MethodsPrecisionSuccess rateSiamFC++0.5120.537Ocean0.5660.560ATOM0.4790.515SiamRPN++0.4670.496SiamDW0.3780.385TADT0.3700.397ASRCF0.3280.344ARCF0.3020.324MCCT0.2360.259UDT0.2450.250CFNet0.2630.275ECO0.3020.324SiamFC0.3310.336Ours0.5220.541


## 4.4.2. Challenge Attributes

We further analyze the performance of LEFRT under different challenges on the LaSOT test dataset.For the most attributes in LaSOT test dataset, our LEFRT performs the first rank against other compared tracking methods. Benefiting from localization based on the target-specific features, our LEFRT achieves high precision in most cases in Figure7. At the same time, our LEFRT can obtain the best performance in scale variation. Compared with other tracking methods that utilize the prior scale parameters, our SCENet predicts a more accurate target’s shape. Our LEFRT performs worse than SiamFC++ for deformation, which is likely that the feature focuses on the target’s appearance in the first frame, and the model is sensitive to the target deformation. Even so, LEFRT obtains a higher tracking accuracy than others on the whole dataset. The visual results are shown in Figure 8.Figure 7
The precision and success rate plots (top ten) on the LaSOT test dataset for challenging attributes.Figure 8
Qualitative results of our LEFRT, ASRCF [24], ECO [40], ARCF [39], TADT [42], SiamRPN++ [19], and SiamDW [16] on LaSOT test dataset (from top to down: Basketball-6, Bear-4, Bicycle-9, Bicycle-18, and Bird-15, respectively).


## 4.5. Evaluation on GOT10K Dataset

In Table5, we compare our LEFRT with several state-of-the-art tracking methods that have publicly available results on the GOT-10K, including SiamFC++ [38], Ocean [47], TADT [42], SiamDW [48], SiamRPN++ [19], ASRCF [24], and Autotrack [49]. Our LEFRT achieves 0.619 AO, 0.721 SR0.5, and 0.477 SR0.75 separately, which is better than most of the methods based on the correlation filter and Siamese network. The pretrained features utilized in the correlation filter lead to the model drifting, which decrease the precision and success rate. Compared with the end-to-end Siamese network, the proposed LEFRT utilized the incremental update to avoid model drifting by the deformation targets. Our method is not as good as Ocean, because Ocean utilizes anchor-free image segmentation for target shape estimation, and the metrics of GOT-10k pay more attention on the overlap rate. Same as the LaSOT dataset, Ocean also trains a special model for the GOT-10k dataset. Compared with Ocean, the model of the proposed method is more comprehensive.Table 5
The precision scores, the average overlap (AO), success rate (SR) SR0.5, SR0.75, and speed (Hz) on the GOT-10k dataset.
MethodsAOSR0.5SR0.75Speed (FPS)SiamFC++0.6100.7160.46340.65Ocean0.6490.7280.59722.79SiamRPN++0.6110.7170.49234.05SiamDW0.6070.7310.47627.37TADT0.3670.3890.13011.39ASRCF0.3130.3170.11311.11AutoTrack0.2860.2770.10915.37Ours0.5220.7210.47721.19


## 4.6. Evaluation on TrackingNet Dataset

In Table6, we compare our LEFRT with several state-of-the-art tracking methods on the TrackingNet, including SiamFC++ [38], Ocean [47], TADT [42], SiamDW [48], SiamRPN++ [19], ASRCF [24], and Autotrack [49]. The proposed method has the advantage in precision and achieves 71.23% Precision and 81.92% NormalizePrecision separately. Compared with the state-of-the-art tracking methods based on the correlation filers, our method improves the effectiveness of the pretrained features. However, the speed of LEFRT may decrease because the time complexity of 2D-RNN is higher than the convolutional layer. Different from the results on other datasets, Ocean achieves a lower overlap rate and obtains a lower Success score than ours. We check the results and find that Ocean trained independent models for each of the other datasets except TrackingNet dataset. Because the TrackingNet dataset does not provide the ground truth file, the Ocean cannot fine-tune the model specifically. And in this experiment, we have to test the Ocean tracker using its OTB2015 model. The result proves that the comprehensiveness of our method is higher. SiamDW utilized the ResNet as the feature extraction backbone, which has deeper network structure. SiamDW enlarged the receptive field for the convolutional layer, which enlarged the feature size and stride. Therefore, the estimation for the target’s shape could be carried out on a larger feature map, which achieves a higher success rate than ours.Table 6
The precision, normalize precision, and success on the TrackingNet dataset.
MethodsPrecisionNormalize precisionSuccess rateSiamFC++70.8881.0674.33Ocean68.2879.0169.22SiamRPN++69.3879.9873.30SiamDW63.7576.0679.92TADT53.9967.1159.28AutoTrack44.0855.5657.63Ours71.2381.9275.71


## 5. Conclusions

This paper proposes a novel visual object tracking method, LEFRT, which can effectively track an arbitrary object and estimate the scale variation. Specifically, the TSNet can generate effective responses for target localization even if the target is not in the pretrained dataset. Moreover, the 2D RNN structure in the SCENet enhances the boundary responses of the target, which produces a precise shape for the target’s scale estimation. Experiments on six challenging datasets demonstrate that our method effectively improves the target’s center precision and the average overlap rate. Although the proposed tracking method has achieved competitive tracking results, its performance can be improved for long-term tracking in complex scenes. In future work, our research will further focus on long-term tracking in complex scenes. To improve the tracking performance in the deformation and occlusion scenes, we will utilize the Transformer to extract the temporal information for the target localization.



---

*Source: 1241687-2022-02-08.xml*