SiamCAN:Simple yet Effective Method to enhance Siamese Short-Term Tracking

1 INTRODUCTION . Visual object tracking is the fundamental task of computer vision , aiming at tracking unknown object of which the information is given by the first frame . Although great progress has been achieved in recent years , a robust tracker is still in desperate demand due to tricky challenge such as scale variation , appearance deformation and similar object with complex background which can deteriorate tracking performance ( Wu et al . ( 2013 ) ; Zhang et al . ( 2014 ) ) . Recently , Siamese Network based trackers have taken a vital place in SOT field due to its accuracy and speed . Since ( Tao et al . ( 2016 ) ) and ( Bertinetto et al . ( 2016 ) ) introduced Siamese networks in visual tracking , Siamese structure has been adopted as baseline for researchers to design efficient trackers ( Li et al . ( 2018 ) ; Zhu et al . ( 2018a ) ; Zhang & Peng ( 2019 ) ; Li et al . ( 2019 ) ; Xu et al . ( 2020 ) ; Chen et al . ( 2020 ) ) . After siamRPN ( Li et al . ( 2018 ) ) being proposed to gain more accurate anchor boxes , region proposal network has become an essential part of tracker . However , the anchor scales are manual-set which go against the fact that the tracking target is unknown . Besides , the performance of the Siamese based trackers depends greatly on offline training by using massive frame pairs . Therefore , it highly increases the risk of tracking drift when facing significant deformation , similar object distractors , or complex background , due to the undiscriminating feature learned from the target when the category of the target is excluded from the training dataset . In these years , the attention mechanism has become the spotlight in computer vision which inspires the relative works not only in detection task but also in visual tracking ( He et al . ( 2018 ) ; Abdelpakey et al . ( 2018 ) ; Wang et al . ( 2018 ) ; Zhu et al . ( 2018b ) ) . The attention mechanism includes channel attention and spatial attention , the former tends to generate a set of channel-weights for modeling interdependencies between channels while the latter focuses on finding the informative part by utilizing the inter-spatial relationship of features . Considering these benefits , Siamese based trackers try to introduce attention module to distinguish target from complex background . Nevertheless , the performance of these trackers is not satisfactory for exploiting the expressive power of the attention mechanism inappropriately . Based on the limitations discussed above , we design a simple Cross-attention Guided Siamese network ( SiamCAN ) based tracker with anchor-free strategy which performs better than the state-ofthe-art trackers when facing the similar object challenge . SiamCAN takes template channel attention to guide the feature extraction of search image by which can strengthen the ability of tracker to overcome distractors and complex backgrounds , performing better than most of Siamese-based trackers , as shown in Figure 1 . The main contributions of this work are : • We formulate a cross-attention guided Siamese framework ( SiamCAN ) including crosschannel attention and self-spatial attention . The cross-channel attention builds an interactive bridge between the target template and search frame to share the identical channel weights . The self-spatial attention focuses on the discriminative part of the correlated feature map , which is complementary to the cross-channel attention . • The proposed tracker is adaptive box regression , without numerous hyper-parameters setting . In order to get more accurate bounding box , we adopt the proper strategy to utilize the merits of anchor-free at the stage of training . • SiamCAN achieves state-of-the-art results on four large tracking benchmarks , including OTB100 ( Wu et al . ( 2013 ) ) , UAV123 ( Mueller et al . ( 2016 ) ) , VOT2018 ( Kristan et al . ( 2018 ) ) and VOT2019 ( Kristan et al . ( 2019 ) ) . The speed of tracker can also achieve 35 FPS . 2 RELATED WORK . In this section , we briefly review the recent Siamese based trackers , the anchor-free approaches and attention mechanism in both tracking and detection filed . 2.1 SIAMESE NETWORK BASED TRACKER . The pioneering works , SINT ( Tao et al . ( 2016 ) ) and SiamFC ( Bertinetto et al . ( 2016 ) ) , first introduce the siamese network in tracking filed . Due to its fast speed with light structure , Siamese network draws great attention from the visual tracking community . SiamFC tries to use siamese network to learn the feature of both target template and search frame , and compare the similarity of them to find the most confident candidates . Although tracks fast , it can not handle the scale variation problem by applying several scales of feature map . Inspired by Faster-RCNN ( Ren et al . ( 2015 ) ) from object detection , SiamRPN ( Li et al . ( 2018 ) ) draws on the region proposal network to get more various scale ratio bounding boxes . Since then , the RPN module has become an essential part of the tracker ( Zhu et al . ( 2018a ) ; Zhang & Peng ( 2019 ) ; Li et al . ( 2019 ) ; Dong & Shen ( 2018 ) ; Fan & Ling ( 2019 ) ) . However , the complexity of anchor design makes the performance of trackers depend greatly on the effect of anchor training . 2.2 ANCHOR-FREE APPROACHES . In recent time , Anchor-free approaches have developed fast . The achor-free work can be divided into two categories . The first one ( Kong et al . ( 2019 ) ; Law & Deng ( 2018 ) ) aims to estimate the keypoints of the objects , while , the other ( Redmon et al . ( 2016 ) ; Tian et al . ( 2019 ) ) tends to predict the bounding box for each pixel which can avoid presetting the scale ratio of anchors . Not only is anchor-free approach popular in detection field , but it is suitable for target estimation in tracking field due to its high efficiency . SiamFC++ takes example by FCOS ( Tian et al . ( 2019 ) ) to design regression subnetwork and add centerness branch to eliminate the low quality samples . SiamCAR ( Guo et al . ( 2020 ) ) changes the basic network structure additionally , merging the multi-layers features before correlation . Different from SiamCAR , SiamBAN ( Chen et al . ( 2020 ) ) puts emphasis on the label assignment which improves the tracking performance . Our method differs from the above trackers in details ( Section4.3 ) . 2.3 ATTENTION MECHANISM . Attention mechanism has been the focus of the detection filed , on account to its powerful ability of enhancing deep CNNs . SE-Net ( Hu et al . ( 2018 ) ) firstly puts forward the mechanism to generate channel weights in return to direct the learning of channel attention . After that , CBAM ( Woo et al . ( 2018 ) ) utilizes both max-pooling and average-pooling to generate the merged attention , includes channel and spatial attention . Recently , ECA-Net ( Wang et al . ( 2020b ) ) finds that avoiding dimensionality is of great importance for channel attention learning , and propose a cross-channel interaction strategy which performs better than SE-Net . In the tracking field , the recent trackers began to equip with the attention mechanism to get better performance . SA Siam ( He et al . ( 2018 ) ) simply combines the SE-Net and SiamFC to get both discriminative and general features which boost the tracking performance . RASNet ( Wang et al . ( 2018 ) ) designs residual attention , general attention and channel attention to learn target feature better . SATIN ( Gao et al . ( 2020 ) ) uses hourglass network as backbone and designs a cross-attention module for exemplar branch to combine the channel and spatial attention from shallow and deep layers . However , these trackers only calculate alongside each branch and neglect the information flow between them , as a result , the ability of attention mechanism can not be fully utilized . 3 OUR APPROACH As shown in Figure 2 , the proposed framework mainly consists of three components , the feature extracted Siamese network , the cross-attention module and anchor-free bounding box regression subnetwork with foreground classification subnetwork . 3.1 FEATURE EXTRACTED SIAMESE NETWORK . Like the most Siamese based tracker , we adopt the fully convolution network without padding , which guarantees the accurate location calculation . Feature extracted network composes of two parts , template branch and search branch . Both of them share the same layer parameters of backbone , by this mean , CNNs can learn the relative feature for them to calculate similarity in the subsequent operation . The template branch intends to encode the exemplar feature in the first frame while the other branch aims to encode the candidates feature which may involve target in the follow-up frames . Set the input in template branch as It , the subsequent frames in search branch as Is . We feed the It and Is into backbone , and get the feature φl ( It ) and φl ( Is ) from different l-th backbone layers . Next , the given features are sent to the corresponding branch after having a convolution with a neck layer to reduce feature channel size to 256 and get the template feature ψt ( It ) with the search feature ψs ( Is ) . At last , crop the 7×7 patch from the center of template feature . 3.2 CROSS-ATTENTION NETWORK . Attention mechanism is created for the purpose that enforce CNNs to focus on the special parts which is of great importance , i.e. , channels information and spatial information . The channel attention is designed to explore the interdependencies between channels while the spatial attention tends to make CNNs pay more attention to the most critical areas of the feature . Different from ( He et al . ( 2018 ) ; Wang et al . ( 2018 ) ) , the channel attention is used between two branches not being applied as self-attention . Moreover , SATIN ( Gao et al . ( 2020 ) ) designs a module also called cross-attention , but the ’ cross ’ means the combination of different layers which is different from our method . In this paper , the target branch feature ψt ( It ) is sent to global average pooling to get aggregated feature Yt , i.e. , Yt = 1 WH ∑W , H i=0 ψt ( It ) ( 1 ) Given the aggregated feature , the channel weight is obtained by performing a 1D convolution of size k , i.e. , Vi = σ ( ∑k j=1 ω jyji ) , y j i ∈ Ω k i ( 2 ) Where σ is a Sigmoid function , ω indicates the parameters of 1D convolution and Ωki indicates the set of k adjacent channels of yi . To let search branch learns the information from target template , we multiply the channel weights with the search feature , i.e. , ψ̃s ( Is ) = ψs ( Is ) ∗ V ( 3 ) 3.3 CLASSIFICATION AND ANCHOR-FREE REGRESSION SUBNETWORK . As shown in Figure 2 , the correlation feature map is calculated by the depth-wise correlation operation between ψ̃s ( Is ) and ψt ( It ) , i.e. , F clsw×h×c = ψ̃s ( Is ) ? ψt ( It ) ( 4 ) F regw×h×c = ψ̃s ( Is ) ? ψt ( It ) ( 5 ) where ? denotes depth-wise convolution operation . Then , we apply self-spatial attention to the feature map in order to focus on discriminative part automatically , i.e. , F̃ clsw×h×c = σ ( f ( [ AvgP ( F cls w×h×c ) ; MaxP ( F cls w×h×c ) ] ) ) ( 6 ) F̃ regw×h×c = σ ( f ( [ AvgP ( F reg w×h×c ) ; MaxP ( F reg w×h×c ) ] ) ) ( 7 ) After that , we use two convolution layers with kernel size 1×1 to reduce the number of channel from 256 to 2 and 4 respectively for each branch and concatenate the feature maps from different layers of backbone by the trainable weights α , i.e. , P clsw×h×2 = ∑N l=1 αl ∗ F̃ clsl : w×h×2 ( 8 ) P regw×h×4 = ∑N l=1 αl ∗ F̃ regl : w×h×4 ( 9 ) where N denotes the total number of the backbone layers we use . The classification feature map has two channels , the one represents the foreground and the points ( i , j ) on Pclsw×h×2 ( 0 , i , j ) refer to the probability scores of target , the other represents the background and the points ( i , j ) on Pclsw×h×2 ( 1 , i , j ) refer to the probability scores of background . The regression feature map has four channels , each of them represents the four direction distances from the points location in search branch input to the four sides of the bounding box respectively , that is to say , each point ( i , j ) in Pregw×h×2 ( : , i , j ) is a vector which can denoted as ( l , r , t , b ) . Classification label and regression label . For anchor based methods , the positive sample and the negative one are classified by the value of Intersection over Union between anchor and groundtruth . In this paper , We use ellipse and circle figure region to design label for points ( i , j ) in feature map , which is inspired by ( Chen et al . ( 2020 ) ) . The ellipse E1 center and axes length are set by groundtruth center ( gxc , gyc ) of groundtruth size ( gw 2 , gh 2 ) , We also get the circle C2 with 0.5gw ∗ 0.5gh ( ( gw2 ) 2 + ( gh2 ) 2 ) 1 2 as radius , i.e. , ( B ( pi ) − gxc ) 2 ( gw2 ) 2 + ( B ( pj ) − gyc ) 2 ( gh2 ) 2 = 1 ( 10 ) B ( pi ) 2 +B ( pj ) 2 = r2 ( 11 ) where B denotes the calculation for the location of points ( i , j ) in feature map P clsw×h×2 back to the search frame . If the point B ( pi , pj ) falls within the C2 region , it will be given a positive label , and if it falls outside the E1 area , it will be given a negative label , i.e. , label = 1 , ifC2 ( p ( i , j ) ) < r 2 −1 , ifE1 ( p ( i , j ) ) > 1 0 , otherwise ( 12 ) For regression branch , the regression targets can be defined by : dl ( i , j ) = pi − gx0 , d t ( i , j ) = pj − gy0 ( 13 ) dr ( i , j ) = gx1 − pi , d b ( i , j ) = gy1 − pj ( 14 ) where ( gx0 , gy0 ) , ( gx1 , gy1 ) denote the left-top and right-bottom coordinates location of the groundtruth . Loss function . We employ cross entropy loss to train the classification network . To predict more accurate bounding box , we adopt the DIoU loss ( Zheng et al . ( 2020 ) ) to train the regression network , i.e. , Lreg = 1− IoU + ρ2 ( p , pgt ) c ( 15 ) where ρ ( . ) is the Euclidean distance , p and pgt denote the central points of predicted box and groundtruth and c is the diagonal length of the smallest enclosing box covering the two boxes . For regression branch training , DIoU loss can optimize the bounding faster than GIoU loss ( Rezatofighi et al . ( 2019 ) ) . The overall loss function is : L = λ1Lcls + λ2Lreg ( 16 ) where constants λ1 and λ2 weight the classification loss and regression loss . During model training , we simply set λ1 = 1 , λ2 = 1 without hyper-parameters searching .

Summary: The architecture of the tracker is standard siamese. The novelty is at a technical level, modules of the "cross-guided" type have been proposed. It does bring an improvement, but not to the state-of-the-art level.  There is no significant insight, training, updating novelty or theoretical. Recent short-term trackers output segmentation, the proposed tracker outputs a bounding box.