Inducing Stronger Object Representations in Deep Visual Trackers

1 INTRODUCTION . Single object visual tracking is a well studied , classical problem in computer vision . It is defined as follows : given a video sequence and an initial annotation ( commonly an axis aligned bounding box ) that localizes some target object of interest , produce the object annotations corresponding to the same object for the remaining frames of the sequence . Because tracking is such a generic visual capability , it is of interest in both theoretical and practical computer vision . Object tracking is found in a wide range of computer vision applications including autonomous vehicles , surveillance , and robotics . In theoretical computer vision , object tracking has been used to develop several theories of invariant object representations ( Lim et al. , 2005 ) . Tracking is also a fundamental capability found in many natural visual systems . Prior to the availability of large annotated video datasets most video-based feature learning models were trained using unsupervised learning objectives , many of them inspired by Slow Feature Analysis ( SFA ) ( Wiskott & Sejnowski , 2002 ) . With the availability of large annotated video datasets such as ILSVRC 2015 Object Detection from Video Challenge ( Russakovsky et al. , 2015 ) and YouTube-Bounding Boxes ( Real et al. , 2017 ) , it has become popular to learn features directly by optimizing tracking objectives . Deep learning approaches have been successfully applied to attain state-of-the-art results on several tracking benchmarks such as VOT2018 ( Kristan et al. , 2017 ) . Though seemingly similar to detection , in that trackers receive images as input and are expected to output bounding boxes , neural trackers are tasked to solve a subtly different problem from detectors . Some key differences include : • Siamese trackers are conditioned on the target image , most detectors are not . One notable exception is the recent work on “ target driven instance detection ” that uses a Siamese architecture similar to those used in tracking to detect particular instances of objects ( Ammirato et al. , 2018 ) . Whereas detectors represent all classes of interest in theirs weights , trackers are expected to form a representation of the target object using a template image . • Siamese trackers often incorporate a small displacement prior by restricting the search area to be near the previous bounding box . The latter constraint often results in center-cropped images of the target input to the “ search ” branch of the Siamese network . Furthermore , the action of cropping restricts the presence of background objects in the search image . This means that the network can often achieve a low tracking loss by detecting the most salient object in the center of the search image . Currently the prevalent approach adopted by many neural trackers is that of “ tracking by detection ” ( Breitenstein et al. , 2009 ) .Recently , deep correlation Siamese networks have been central components in state-of-the-art tracking approaches ( Li et al. , 2019 ; 2018 ; Bertinetto et al. , 2016 ) . We refer to these object tracking networks as “ Siamese trackers ” . These networks functionally resemble object detection networks ( such as Fast-RCNN ( Ren et al. , 2015 ) and YOLO ( Redmon et al. , 2016 ) ) in that they localize semantic objects by producing bounding box outputs . Unlike object detection networks , which are trained to detect object categories , Siamese trackers are trained to detect the object using the provided template in a video sequence . These networks are trained end-to-end on video datasets on a per-frame detection loss . The network architectures used by Siamese trackers emphasize tracking by detection via correlation of the target and search images in a learned feature space ( Bertinetto et al. , 2016 ; Held et al. , 2016 ) . However , because the training objective is minimized endto-end , networks need not necessarily optimize the tracking objective by feature matching . The aim of this paper is to study these trackers ’ reliance on the target representation and show that Siamese trackers often default to tracking by center saliency detection ( Mahadevan & Vasconcelos , 2012 ) – without necessarily forming a discriminative target representation . In other words , a simple strategy that neural trackers can use to get good performance is to track the object that is most salient in the search region instead of attempting to rely on matching features with the target template ( see Figure 4 ) . On the basis of these insights , we introduce an auxiliary instance driven detection objective Ammirato et al . ( 2018 ) that improves tracking performance over our baseline . An example of our detection output is shown in Figure 1 . 2 PRIOR WORK . Like many fields in computer vision , single object tracking has become dominated by deep learning approaches . However , unlike image classification ( Krizhevsky et al. , 2012 ) and object detection ( Sermanet et al. , 2013 ) , which were arguably the first successes of deep learning on large scale natural image datasets , object tracking had been relatively untouched by deep learning until 2016 with the publications of GOTURN ( Held et al. , 2016 ) and MDNet ( Nam & Han , 2016 ) . These seminal works have spawned successors , many of which still dominate tracking performance in the most popular benchmarks ( Kristan et al. , 2017 ; Wu et al. , 2013 ; Valmadre et al. , 2018 ) . This shift was perhaps triggered by the release of ImageNet Object Detection from Video in 2015 ( Russakovsky et al. , 2015 ) , the first per-frame annotated large scale video dataset . GOTURN is an example of the so called “ tracking by detection ” approach , where tracking is performed by recursive detection on video frames . GOTURN and it ’ s derivatives implement Siamese-networks to directly regress the bounding box coordinates of the target object . These networks typically consists of four main elements : the target network , the search network , the join layer , and the output network . The target and search networks compute feature maps of the corresponding to the target object and the search region , respectively . These feature maps are combined in the join layer ; most recent trackers use convolution/correlation to combine the feature maps ( Bertinetto et al. , 2016 ) . Finally , the combined feature maps are fed to the output layer which consists of a trainable network that outputs the final bounding box or bounding box proposals ( Li et al. , 2018 ; 2019 ; Held et al. , 2016 ) . Most recently SiamRPN ( Li et al. , 2018 ) , SiamRPN++ ( Li et al. , 2019 ) , and their derivatives have been the dominant state-of-the-art approaches to single object tracking . These works combine region proposals popular in the detection literature with Siamese tracking . Later we will introduce detection as an auxiliary task for tracking . Training a model to jointly track and detect has recently been proposed in ( Feichtenhofer et al. , 2017 ) . However , this work is quite orthogonal to ours as it does not consider object instance representation and tracks by temporally linking detected instances . 3 BASIC TRACKING MODEL . Our basic tracking model takes inspiration from recent works starting with GOTURN ( Held et al. , 2016 ) and culminating in SiamRPN++ ( Li et al. , 2019 ) . Our model uses a lightweight backbone network ( MobileNetV2 ) , and is somewhat simpler than recent state-of-the-art models . Nevertheless , our model contains key elements that are common to most recent deep trackers and therefore we believe our conclusions also apply to these models . Although the model doesn ’ t outperform stateof-the-art , it attains competitive performance . This section details the architecture of the tracking network , the targets and loss it is trained with , the training data sampling method , data augmentation , and post-processing of the model outputs . 3.1 TRACKING NETWORK . The overall architecture of our tracking network is shown in Figure 2 . The tracker takes two images as input : the target image ( xt ) is the centered crop of the object instance to be tracked and the search image ( xs ) is a crop of the image in which the target is to be localized . The frame corresponding to the target image precedes the frame corresponding to the search image . After re-sizing , the target and search image have shapes 127×127×3 and 255×255×3 , respectively . We use the same aspect preserving cropping mechanism detailed in ( Bertinetto et al. , 2016 ) . Feature maps corresponding to both images are computed using a pre-trained 0.75-MobileNet-224 backbone network ( Howard et al. , 2017 ) . Because the weights are shared between the target and search branches the two images are encoded using the same function ( i.e . the backbone network ) denoted by fθ . Feature maps corresponding to the target and search images have shapes 32×32×384 and 64×64×384 , respectively . We denote these feature maps as zit = fθ ( xt ) and z i s = fθ ( xs ) , where i is the feature plane index . Next we project these feature maps using a trainable 1 × 1 convolution to 256 dimensions . The resulting target and search features are combined by convolving each search feature map with the corresponding target feature map with zero-padding ( “ same ” convolution ) , to obtain a combined activation map zi = zit ∗ zis that has shape 64 × 64 × 256 . We refer to this operation as crossconvolution . A similar operator for combining the target and search features was first introduced in ( Bertinetto et al. , 2016 ) and has since been employed in many recent tracking architectures . Finally the resulting feature map is projected to six channels using a trainable 1 × 1 convolution layer , resulting in a final output of size 64×64×6 . It is worth noting , we have found that inserting additional trainable layers after the cross-convolution stage significantly reduces performance . The tracker outputs n-bounding box proposals and their respective confidence ( where n is a hyperparameter we set to 5 ) . Our bounding box proposal parameterization is inspired by the fully convolutional heatmap and offset parameterization introduced for points in ( Papandreou et al. , 2017 ) , which we adapt to bounding boxes . The first two of the six output channels represent the center location of the bounding box . These two channels are used to represent the positive and negative classes of the binary heatmap . These channels are trained with a binary cross-entropy loss with targets that are constructed by placing a fixed radius ( 10 pixels ) circle of ones centered at the location of the ground truth bounding box , the heatmap targets are set to zero elsewhere . The remaining four feature maps are used to represent the spatial offsets to the top-left and bottom-right bounding box corners relative to their location in the feature map . The offset channels are trained with an L1 regression loss . To produce the final bounding box output we enumerate the top n modes in the heatmap by using non-maximal suppression with a fixed square window ( 6 × 6 pixels ) centered on each mode . The mean offset is computed within each window to give the offset corresponding to each bounding box corner . In contrast with ( Papandreou et al. , 2017 ) , we do not use Hough voting to aggregate the offsets . The model is trained on short sub-sequences taken from the ILSVCR 2015 Detection from Video ( Russakovsky et al. , 2015 ) and the GOT-10k ( Huang et al. , 2018 ) video datasets . To encourage the network to learn invariant representations of the target across long time scales , the target and search images are sampled such that they are unlikely to be temporally proximal frames . Suppose a given dataset sequence consists of T frames and we sample a sub-sequence of length τ starting at index t such that t + τ < T . The target image is selected uniformly from the interval [ 0 , t − 1 ] , to respect the fact that the target template must appear before the search image . Networks were trained with several target image selection mechanisms including always using the first image from the sequence and sampling random causal targets . The latter lead to better performance , likely due to increased diversity . During evaluation , the target image is cropped from the first frame using the ground truth bounding box and held it fixed for the rest of the sequence . We apply the following data augmentation to both target and search images during training : random translation and scaling , as well as random perturbations to the contrast , hue , and saturation . The model was trained using the Adam optimizer ( Kingma & Ba , 2014 ) for 1e7 steps with an initial learning rate of 10−4 which was reduced by a factor of 10 at 9.5e6 steps . The heatmap and regression loss are combined using a weighted sum . The weight corresponding to the heatmap loss is set to 1.0 and regression weight is set to 0.3 .

Summary: This paper proposes to learn a visual tracking network for an object detection loss as well as the ordinary tracking objective for enhancing the reliability of the tracking network. The main motivation is that, current state-of-the-art models based on the Siamese architecture often blindly predict the center of a search window as target location due to the bias in datasets and training strategies. This issue is alleviated in this paper by introducing an auxiliary task, target detection in the entire image space. The auxiliary task is conducted by another branch on top of the visual feature shared with the tracking branch. By learning to detect object in the entire image space, the shared feature extractor will be trained to capture discriminative and unique appearance features of target.