Year: 2020

Abstract:
Most traditional Siamese trackers are used to regard the location of the max response map as the center of target. However, it is difficult for these traditional methods to calculate response value accurately when face the similar object, deformation, background clutters and other challenges. So how to get the reliable response map is the key to improve tracking performance. Accordingly, a simple yet effective short-term tracking framework (called SiamCAN),by which bridging the information flow between search branch and template branch, is proposed to solve the above problem in this paper. Moreover, in order to get more accurate target estimation, an anchor-free mechanism and specialized training strategy are applied to narrow the gap between the predicted bounding box and groundtruth. The proposed method achieves state-of-the-art performance on four visual tracking benchmarks including UAV123, OTB100, VOT2018 and VOT2019, outperforming the strong baseline, SiamBAN, by 0.327→ 0.331 on VOT2019 and 0.631 → 0.638 success score, 0.833→ 0.850 precision score on UAV123.

heading: 1 INTRODUCTION
text: Visual object tracking is the fundamental task of computer vision, aiming at tracking unknown object of which the information is given by the first frame. Although great progress has been achieved in recent years, a robust tracker is still in desperate demand due to tricky challenge such as scale variation, appearance deformation and similar object with complex background which can deteriorate tracking performance (Wu et al. (2013); Zhang et al. (2014)).
Recently, Siamese Network based trackers have taken a vital place in SOT field due to its accuracy and speed. Since (Tao et al. (2016)) and (Bertinetto et al. (2016)) introduced Siamese networks in visual tracking, Siamese structure has been adopted as baseline for researchers to design efficient trackers (Li et al. (2018); Zhu et al. (2018a); Zhang & Peng (2019); Li et al. (2019); Xu et al. (2020); Chen et al. (2020)). After siamRPN (Li et al. (2018)) being proposed to gain more accurate anchor boxes, region proposal network has become an essential part of tracker. However, the anchor scales are manual-set which go against the fact that the tracking target is unknown. Besides, the performance of the Siamese based trackers depends greatly on offline training by using massive frame pairs. Therefore, it highly increases the risk of tracking drift when facing significant deformation, similar object distractors, or complex background, due to the undiscriminating feature learned from the target when the category of the target is excluded from the training dataset.
In these years, the attention mechanism has become the spotlight in computer vision which inspires the relative works not only in detection task but also in visual tracking (He et al. (2018); Abdelpakey et al. (2018); Wang et al. (2018); Zhu et al. (2018b)). The attention mechanism includes channel attention and spatial attention, the former tends to generate a set of channel-weights for modeling interdependencies between channels while the latter focuses on finding the informative part by utilizing the inter-spatial relationship of features. Considering these benefits, Siamese based trackers try to introduce attention module to distinguish target from complex background. Nevertheless, the performance of these trackers is not satisfactory for exploiting the expressive power of the attention mechanism inappropriately.
Based on the limitations discussed above, we design a simple Cross-attention Guided Siamese network (SiamCAN) based tracker with anchor-free strategy which performs better than the state-ofthe-art trackers when facing the similar object challenge. SiamCAN takes template channel attention
to guide the feature extraction of search image by which can strengthen the ability of tracker to overcome distractors and complex backgrounds, performing better than most of Siamese-based trackers, as shown in Figure 1. The main contributions of this work are:
• We formulate a cross-attention guided Siamese framework (SiamCAN) including crosschannel attention and self-spatial attention. The cross-channel attention builds an interactive bridge between the target template and search frame to share the identical channel weights. The self-spatial attention focuses on the discriminative part of the correlated feature map, which is complementary to the cross-channel attention.
• The proposed tracker is adaptive box regression, without numerous hyper-parameters setting. In order to get more accurate bounding box, we adopt the proper strategy to utilize the merits of anchor-free at the stage of training.
• SiamCAN achieves state-of-the-art results on four large tracking benchmarks, including OTB100 (Wu et al. (2013)), UAV123 (Mueller et al. (2016)), VOT2018 (Kristan et al. (2018)) and VOT2019 (Kristan et al. (2019)). The speed of tracker can also achieve 35 FPS.

heading: 2 RELATED WORK
text: In this section, we briefly review the recent Siamese based trackers, the anchor-free approaches and attention mechanism in both tracking and detection filed.

heading: 2.1 SIAMESE NETWORK BASED TRACKER
text: The pioneering works, SINT (Tao et al. (2016)) and SiamFC (Bertinetto et al. (2016)), first introduce the siamese network in tracking filed. Due to its fast speed with light structure, Siamese network draws great attention from the visual tracking community. SiamFC tries to use siamese network to learn the feature of both target template and search frame, and compare the similarity of them to find the most confident candidates. Although tracks fast, it cannot handle the scale variation problem by applying several scales of feature map. Inspired by Faster-RCNN (Ren et al. (2015)) from object detection, SiamRPN (Li et al. (2018)) draws on the region proposal network to get more various scale ratio bounding boxes. Since then, the RPN module has become an essential part of the tracker (Zhu et al. (2018a); Zhang & Peng (2019); Li et al. (2019); Dong & Shen (2018); Fan & Ling (2019)). However, the complexity of anchor design makes the performance of trackers depend greatly on the effect of anchor training.

heading: 2.2 ANCHOR-FREE APPROACHES
text: In recent time, Anchor-free approaches have developed fast. The achor-free work can be divided into two categories. The first one (Kong et al. (2019); Law & Deng (2018)) aims to estimate the keypoints of the objects, while, the other (Redmon et al. (2016); Tian et al. (2019))tends to predict the bounding box for each pixel which can avoid presetting the scale ratio of anchors. Not only is anchor-free approach popular in detection field, but it is suitable for target estimation in tracking field due to its high efficiency. SiamFC++ takes example by FCOS (Tian et al. (2019)) to design regression subnetwork and add centerness branch to eliminate the low quality samples. SiamCAR (Guo et al. (2020)) changes the basic network structure additionally, merging the multi-layers features before correlation. Different from SiamCAR, SiamBAN (Chen et al. (2020)) puts emphasis on the label assignment which improves the tracking performance. Our method differs from the above trackers in details (Section4.3).

heading: 2.3 ATTENTION MECHANISM
text: Attention mechanism has been the focus of the detection filed, on account to its powerful ability of enhancing deep CNNs. SE-Net (Hu et al. (2018)) firstly puts forward the mechanism to generate channel weights in return to direct the learning of channel attention. After that, CBAM (Woo et al. (2018)) utilizes both max-pooling and average-pooling to generate the merged attention, includes channel and spatial attention. Recently, ECA-Net (Wang et al. (2020b)) finds that avoiding dimensionality is of great importance for channel attention learning, and propose a cross-channel interaction strategy which performs better than SE-Net. In the tracking field, the recent trackers began to equip with the attention mechanism to get better performance. SA Siam (He et al. (2018)) simply combines the SE-Net and SiamFC to get both discriminative and general features which boost the tracking performance. RASNet (Wang et al. (2018)) designs residual attention, general attention and channel attention to learn target feature better. SATIN (Gao et al. (2020)) uses hourglass network as backbone and designs a cross-attention module for exemplar branch to combine the channel and spatial attention from shallow and deep layers. However, these trackers only calculate alongside each branch and neglect the information flow between them, as a result, the ability of attention mechanism can not be fully utilized.
3 OUR APPROACH
As shown in Figure 2, the proposed framework mainly consists of three components, the feature extracted Siamese network, the cross-attention module and anchor-free bounding box regression subnetwork with foreground classification subnetwork.

heading: 3.1 FEATURE EXTRACTED SIAMESE NETWORK
text: Like the most Siamese based tracker, we adopt the fully convolution network without padding, which guarantees the accurate location calculation. Feature extracted network composes of two parts, template branch and search branch. Both of them share the same layer parameters of backbone, by this mean, CNNs can learn the relative feature for them to calculate similarity in the subsequent operation. The template branch intends to encode the exemplar feature in the first frame while the other branch aims to encode the candidates feature which may involve target in the follow-up frames. Set the input in template branch as It, the subsequent frames in search branch as Is. We feed the It and Is into backbone, and get the feature φl(It) and φl(Is) from different l-th backbone layers. Next, the given features are sent to the corresponding branch after having a convolution with a neck layer to reduce feature channel size to 256 and get the template feature ψt(It) with the search feature ψs(Is). At last, crop the 7×7 patch from the center of template feature.

heading: 3.2 CROSS-ATTENTION NETWORK
text: Attention mechanism is created for the purpose that enforce CNNs to focus on the special parts which is of great importance, i.e., channels information and spatial information. The channel attention is designed to explore the interdependencies between channels while the spatial attention tends to make CNNs pay more attention to the most critical areas of the feature. Different from (He et al. (2018); Wang et al. (2018)), the channel attention is used between two branches not being applied as self-attention. Moreover, SATIN (Gao et al. (2020)) designs a module also called cross-attention , but the ’cross’ means the combination of different layers which is different from our method. In this paper, the target branch feature ψt(It) is sent to global average pooling to get aggregated feature Yt, i.e.,
Yt = 1
WH
∑W,H i=0 ψt(It) (1)
Given the aggregated feature, the channel weight is obtained by performing a 1D convolution of size k, i.e., Vi = σ( ∑k j=1 ω jyji ), y j i ∈ Ω k i (2)
Where σ is a Sigmoid function, ω indicates the parameters of 1D convolution and Ωki indicates the set of k adjacent channels of yi. To let search branch learns the information from target template, we multiply the channel weights with the search feature, i.e.,
ψ̃s(Is) = ψs(Is) ∗ V (3)

heading: 3.3 CLASSIFICATION AND ANCHOR-FREE REGRESSION SUBNETWORK
text: As shown in Figure 2, the correlation feature map is calculated by the depth-wise correlation operation between ψ̃s(Is) and ψt(It), i.e.,
F clsw×h×c = ψ̃s(Is) ? ψt(It) (4)
F regw×h×c = ψ̃s(Is) ? ψt(It) (5) where ? denotes depth-wise convolution operation. Then, we apply self-spatial attention to the feature map in order to focus on discriminative part automatically, i.e.,
F̃ clsw×h×c = σ(f([AvgP (F cls w×h×c);MaxP (F cls w×h×c)])) (6)
F̃ regw×h×c = σ(f([AvgP (F reg w×h×c);MaxP (F reg w×h×c)])) (7)
After that, we use two convolution layers with kernel size 1×1 to reduce the number of channel from 256 to 2 and 4 respectively for each branch and concatenate the feature maps from different layers of backbone by the trainable weights α, i.e.,
P clsw×h×2 = ∑N l=1 αl ∗ F̃ clsl:w×h×2 (8)
P regw×h×4 = ∑N l=1 αl ∗ F̃ regl:w×h×4 (9)
where N denotes the total number of the backbone layers we use. The classification feature map has two channels, the one represents the foreground and the points (i, j) on Pclsw×h×2(0, i, j) refer to the probability scores of target, the other represents the background and the points (i, j) on Pclsw×h×2(1, i, j) refer to the probability scores of background. The regression feature map has four channels, each of them represents the four direction distances from the points location in search branch input to the four sides of the bounding box respectively, that is to say, each point (i, j) in Pregw×h×2(:, i, j) is a vector which can denoted as (l, r, t, b).
Classification label and regression label. For anchor based methods, the positive sample and the negative one are classified by the value of Intersection over Union between anchor and groundtruth. In this paper, We use ellipse and circle figure region to design label for points (i, j) in feature map, which is inspired by (Chen et al. (2020)). The ellipse E1 center and axes length are set by groundtruth center (gxc , gyc) of groundtruth size (
gw 2 , gh 2
), We also get the circle C2 with 0.5gw ∗ 0.5gh
(( gw2 ) 2 + ( gh2 ) 2) 1 2
as radius, i.e.,
(B(pi)− gxc)2
( gw2 ) 2
+ (B(pj)− gyc)2
( gh2 ) 2
= 1 (10)
B(pi) 2 +B(pj) 2 = r2 (11)
where B denotes the calculation for the location of points (i, j) in feature map P clsw×h×2 back to the search frame. If the point B(pi, pj) falls within the C2 region, it will be given a positive label, and if it falls outside the E1 area, it will be given a negative label, i.e.,
label =  1, ifC2(p(i, j)) < r 2
−1, ifE1(p(i, j)) > 1 0, otherwise
(12)
For regression branch, the regression targets can be defined by:
dl(i,j) = pi − gx0 , d t (i,j) = pj − gy0 (13)
dr(i,j) = gx1 − pi, d b (i,j) = gy1 − pj (14)
where (gx0 , gy0), (gx1 , gy1) denote the left-top and right-bottom coordinates location of the groundtruth.
Loss function. We employ cross entropy loss to train the classification network. To predict more accurate bounding box, we adopt the DIoU loss (Zheng et al. (2020)) to train the regression network, i.e.,
Lreg = 1− IoU + ρ2(p, pgt)
c (15)
where ρ(.) is the Euclidean distance, p and pgt denote the central points of predicted box and groundtruth and c is the diagonal length of the smallest enclosing box covering the two boxes. For regression branch training, DIoU loss can optimize the bounding faster than GIoU loss (Rezatofighi et al. (2019)). The overall loss function is:
L = λ1Lcls + λ2Lreg (16)
where constants λ1 and λ2 weight the classification loss and regression loss. During model training, we simply set λ1 = 1, λ2 = 1 without hyper-parameters searching.

heading: 3.4 TRAINING AND INFERENCE
text: Training. We train our model by using image pairs, a 127×127 pixels template patch and a 255×255 pixels search patch. The training datasets include ImageNet VID (Russakovsky et al. (2015)), COCO (Lin et al. (2014)), YouTube-Bounding Boxes (Real et al. (2017)), ImageNet Det (Real et al. (2017)) and GOT10k (Huang et al. (2019)). Due to the numbers of negative samples are more than the positive samples, we set at most 16 positive samples and 48 negative samples from the search image.
Besides, in order to get more accurate regression information, we adopt the DIoU loss to optimize the regression branch.
Inference. We feed the cropped first frame and the subsequent frames to the feature abstract network as template image and search image. Next, the features are sent to the cross-attention module and pass the classification branch and regression branch. After that, we get the classification map. The location of the highest score represents the most probable center of the tracking target. Then, we use the scale change penalty and the cosine window as that introduced in (Li et al. (2018)) to guarantee the smooth movements of target. According to the location p of the final score, we can get the predicted boxes B, i.e.,
bx1 = pi − dregl , by1 = pj − d reg t (17) bx2 = pi + d reg r , by2 = pj + d reg b (18)
where dregl,r,t,b denotes the predicted values of the regression targets on the regression map, (bx1, by1) and (bx2, by2) are the top-left corner and bottom-left corner of the predicted box.

heading: 4 EXPERIMENTS

heading: 4.1 IMPLEMENTATION DETAILS
text: Our approach is implemented in Python with Pytorch on 1 RTX2080Ti. The backbone is modified ResNet-50 as in (He et al. (2016)), and the weights are pre-trained on ImageNet (Russakovsky et al. (2015)). During the training phase, the model is optimized by the stochastic gradient descent (SGD), at the meantime, the total number of epochs is 20 and the batch size is set as 28. For the first 10 epochs, we frozen the parameters of the backbone and only train the heads structures, for the last 10 epochs, we unfrozen the last 3 blocks of backbone to be trained together. Besides, we warm up the training during the first 5 epoch with a warmup learning rate of 0.001 to 0.005, and in the last 15 epochs, the learning rate exponentially decayed from 0.005 to 0.00005.

heading: 4.2 RESULTS ON THREE BENCHMARKS
text: To affirm the effect of our method, we evaluate our tracker performance with the recent trackers MAML (Wang et al. (2020a)), PriDiMP (Danelljan et al. (2020)), SiamBAN (Chen et al. (2020)), SiamCAR (Guo et al. (2020)), ROAM (Yang et al. (2020)), SiamFC++ (Xu et al. (2020)), STN (Tripathi et al. (2019)), ARTCS (Kristan et al. (2019)), SiamRPN++ (Li et al. (2019)),SATIN (Gao et al. (2020)), ATOM (Danelljan et al. (2019)), DIMP-18 (Bhat et al. (2019)), DaSiamRPN (Zhu et al. (2018a)), SPM (Wang et al. (2019))ECO (Danelljan et al. (2017)), CFNet (Valmadre et al. (2017)), SiamRPN (Li et al. (2018)), DeepSRDCF (Danelljan et al. (2015a)), SRDCF (Danelljan et al. (2015b)) on four benchmarks UAV123 (Mueller et al. (2016)), VOT2018 (Kristan et al. (2018)), VOT2019 (Kristan et al. (2019)) and OTB100 (Wu et al. (2013))(details in Appendix A.1).

heading: 4.2.1 RESULTS ON UAV123
text: UAV123 contains 123 challenging video sequences, which can be divided into 11 categories according to their attributes. The performance of the tracker is evaluated by two evaluation metrics, the precision scores and the AUC scores. The AUC scores reflect the overlap between the predict bounding box and ground-truth box, while the precision scores are relative with the distance between the center of the predict bounding box and ground-truth box. In Figure 3, our method achieves the best performance in precision scores, i.e., 0.850, and the second best AUC scores 0.678. As for the 11 categories of the challenge, SiamCAN ranks 1st or 2nd in Scale Variation, Similar Object, Fast Motion and Low Resolution, in Appendix A.2. The results demonstrate that our tracker can handle the similar object and scale change challenge, due to the learning of cross-attention subnetwork and anchor-free mechanism.

heading: 4.2.2 RESULTS ON VOT2018
text: VOT2018 consists of 60 challenging videos. The evaluation metrics to rank the performance of the tracker based on EAO (Expected Average Overlap) which depends on the accuracy and the robustness. As shown in Table 1, our tracker outperforms SiamRPN++ by 2.8 points by introducing
the anchor-free regression network. Compared with SiamFC++, we achieve EAO improvements of 4.5 points. Although our method fails to achieve the best EAO, the robustness scores of listed Siamese-based trackers are higher than our method, which affect the EAO scores, in other words, SiamCAN is a robust Siamese-based tracker.

heading: 4.2.3 RESULTS ON VOT2019
text: VOT2019 video sequences are 20% different from VOT2018, adding more fast motion and similar object videos. Table 2 reflects the evaluation results on VOT2019 compared with the recent trackers. We can see that the recent proposed MAML gets the highest accuracy scores, while our SiamCAN surpassing MAML by 2.5 points in terms of EAO. Besides, our robustness scores also rank 2nd.

heading: 4.3 ANALYSIS OF THE PROPOSED METHOD
text: Discussion on effective sample selection. Anchor-free method has the weakness that network may produce low-quality predicted bounding boxes far away from the center of target, even though the predicted box is accurate. To address this issue, SiamFC++ and SiamCAR introduce centerness branch to get high quality sample, forcing the network focus on the target center. While, SiamBAN uses ellipse figure region to design label which has the same effect. Accordingly, we do several experiments to find which method performs better. As shown in Table 3, baseline tracker consists of cross-attention module and anchor-free network. Ellipse label does better than Circle label ( 2© vs 1©),while the centerness branch with ellipse label even have worse effects ( 2© vs 3©). Based on
the performance of 2©, we visualize the tracking of 2© and compare with 4©, details in Figure 4. At the training stage, 2© gives the positive label to the points fall within E2 region, while 4© gives the positive label to the points fall within the C2 region, the more central position. In this aspect, the comparision in Table 3 ( 2© vs 4©) can be explained.
Discussion on components of tracker. To verify the role of each components of our tracker, we use the results of components on VOT2018 to analyze the details. As shown in Table 4, the baseline model can get an EAO of 0.351, which consists of a regular backbone, i.e., ResNet50, a classification and an anchor-free regression network. Exchanging DIoU Loss for GIoU Loss during training can get higher scores, due to the more accurate predicted bounding box ( 2© vs 1©). Adding crossattention module can obtain a large improvement, i.e., 4.3 points on EAO ( 4© vs 2©). This demonstrates the information interaction between the template branch and the search branch is of great significance. Finally, the tracker utilizes the self-spatial attention can gain another improvement of 2.4 points on EAO ( 4© vs 5©). Feature visualization. We visualize the features extracted from tracker 1©, tracker 3© and tracker 5© in Figure 5. On the left side of Figure 5, tracker 1© vs tracker 3© shows the tracker not equipped with cross-attention module is easier to lose target and focus on the wrong object when appear similar object and the visualized feature demonstrates that cross-attention module can enable tracker to tell target from similar distactors. On the right side of Figure 5, tracker 5© shows the power of crossattention module combined with proper training strategy.

heading: 5 CONCLUSION
text: In this paper, we propose a simple Siamese-based tracker called SiamCAN, which combined with cross-attention and anchor-free mechanism. The cross-attention module utilizes the target template channel attention to guide the feature learning of search frame, bridging the information flow between each branch. The anchor-free regression discards the fussy design of the anchors, and adjusts the scale ratio of the bounding box automatically. In order to use them to their fullest potential, choosing appropriate label assignment strategy and suitable loss funtion to boost the tracking performance with limited laboratory equipments. Extensive experiments are conducted on four benchmarks, which demonstrate our trackers with light structure yet achieves the state-of-the-art performance, especially in scale variation, background cluster, deformations and similar distractors challenges.

heading: A APPENDIX
text: A.1 EXPERIMENT RESULTS ON OTB100
OTB100 contains 100 challenging video sequences, and the evaluation metrics are same with UAV123. In Figure 5, our method achieves the best performance in precision scores, i.e., 0.913, and the second best success scores 0.684. As for the 9 categories of the challenge, SiamCAN ranks 1st or 2nd in Deformation, Background Clutters, Scale Varation and Out-of-Plane Rotation. The results demonstrate that our trackers can handle the deformation, background clutters, scale variation and out-of-plane rotation, in Figure 6.
A.2 MORE EXPERIMENT RESULTS ON UAV123

Review summaries:
The architecture of the tracker is standard siamese. The novelty is at a technical level, modules of the "cross-guided" type have been proposed. It does bring an improvement, but not to the state-of-the-art level.  There is no significant insight, training, updating novelty or theoretical. Recent short-term trackers output segmentation, the proposed tracker outputs a bounding box.