Inducing Stronger Object Representations in Deep Visual Trackers
Authors: Ross Goroshin
Jonathan Tompson
Debidatta Dwibedi
Venue:  (2020)
Keywords: Object Tracking
Computer Vision
Deep Learning

Abstract: Fully convolutional deep correlation networks are integral components of state-of-
the-art approaches to single object visual tracking. It is commonly assumed that
these networks perform tracking by detection by matching features of the object
instance with features of the entire frame. Strong architectural priors and conditioning
on the object representation is thought to encourage this tracking strategy.
Despite these strong priors, we show that deep trackers often default to “tracking-
by-saliency” detection – without relying on the object instance representation. Our
analysis shows that despite being a useful prior, salience detection can prevent the
emergence of more robust tracking strategies in deep networks. This leads us to
introduce an auxiliary detection task that encourages more discriminative object
representations that improve tracking performance.

Under review as a conference paper at ICLR 2020
INDUCING STRONGER OВJECT REPRESENTATIONS IN DEEP VISUAL TRACKERS
Anonymous authors Paper under double-blind review
ABSTRACT
Fully convolutional deep correlation networks are integral components of state-of- the-art approaches to single object visual tracking. It is commonly assumed that these networks perform tracking by detection by matching features of the object instance with features of the entire frame. Strong architectural priors and condi- tioning on the object representation thought to encourage this tracking strategy. Despite these strong priors, show that deep trackers often default to "tracking by saliency" detection - without relying on the object instance representation. Our analysis shows that despite being useful prior, salience detection can prevent the emergence of more robust tracking strategies in deep networks. This leads us to introduce auxiliary detection task that encourages more discriminative object representations that improve tracking performance.
INTRODUCTION
Single object visual tracking is a well studied, classical problem in computer vision. It is defined follows: given video sequence and initial annotation (commonly an axis aligned bounding box) that localizes some target object of interest, produce the object annotations corresponding to the same object for the remaining frames of the sequence. Because tracking is such generic vi- sual capability, it is of interest in both theoretical and practical computer vision. Object tracking is found in wide range of computer vision applications including autonomous vehicles. surveillance, and robotics. In theoretical computer vision, object tracking has been used to develop several theo- ries of invariant object representations (Lim et al., 2005). Tracking also a fundamental capability found in many natural visual systems. Prior to the availability of large annotated video datasets most video-based feature learning models were trained using unsupervised learning objectives, many of them inspired by Slow Feature Analysis (SFA) (Wiskott & Sejnowski, 2002). With the availability of large annotated video datasets such as ILSVRC 2015 Object Detection from Video Challenge (Russakovsky al.) (2155) and YouTube-BBi Buuuu--Bnnniiggggg Boxes (Real et al.) (017), it has become pop- ular to learn features directly by optimizing tracking objectives. Deep learning approaches have been successfully applied to attain state-of-the-art results several tracking benchmarks such as VOT2018 (Kristan et al. 2017).
Though seemingly similar detection, in that trackers receive images input and are expected to output bounding boxes, neural trackers are tasked to solve a subtly different problem from detectors. Some key differences include:
Siamese trackers are conditioned on the target image, most detectors are not. One notable exception is the recent work on "target driven instance detection" that uses a Siamese archi- tecture similar to those used in tracking to detect particular instances of objects (Ammirato et al., 2018). Whereas detectors represent all classes of interest in theirs weights, trackers are expected to form a representation of the target object using a template image.
Siamese trackers often incorporate a small displacement prior by restricting the search area to be near the previous bounding box.
The latter constraint often results center-cropped images of the target input to the "search" branch of the Siamese network. Furthermore, the action of cropping restricts the presence of background objects in the search image. This means that the network can often achieve low tracking loss by detecting the most salient object in the center of the search image.

---

Under review as a conference paper at ICLR 2020
The output of our conditional detector. The two examples show: Left- the detector input. Middle - the heatmap output of our detector depicting the likelihood of the target object centered on each pixel. Right the "target" template image specifying the object instance detect.
Figure 1:
Currently the prevalent approach adopted by many neural trackers is that of "tracking by detection" (Breitenstein al. 2009)..ecently, deep correlation Siamese networks have been central compo- nents in state-of-the-- art tracking approaches (Li et al. 2019 2018 Bertinetto et al. 2016). We refer to these object tracking networks as "Siamese trackers" These networks functionally resemble ob- ject detection networks (such as Fast-RCNN (Ren et al. 2015) and YOLO (Redmon et al. (016)) in that they localize semantic objects by producing bounding box outputs. Unlike object detection net- works, which are trained to detect object categories, Siamese trackers are trained to detect the object using the provided template in a video sequence. These networks are trained end-to-end on video datasets on per-frame detection loss. The network architectures used by Siamese trackers empha- size tracking by detection via correlation of the target and search images in learned feature space (Bertinetto et al.) [2166] Held et al. (016). However, because the training objective is minimized end- to-end, networks need not necessarily optimize the tracking objective by feature matching. 'he aim of this paper is to study these trackers' reliance the target representation and show that Siamese trackers often default to tracking by center saliency detection (Mahadevan & Vasconcelos) (012) without necessarily forming a discriminative target representation. In other words, simple strategy that neural trackers can use to get good performance is to track the object that is most salient in the search region instead of attempting rely on matching features with the target template (see Fig- ure 4.. On the basis of these insights, we introduce an auxiliary instance driven detection objective Ammirato et al. (2018) that improves tracking performance over our baseline. An example of our detection output is shown in Figure
2
PRIOR WORK
Like many fields in computer vision, single object tracking has become dominated by deep learning approaches. However, unlike image classification (Krizhevsky al. 2012) and object detection (Sermanet et al. 2013). which were arguably the first successes of deep learning on large scale nat- ural image datasets, object tracking had been relatively untouched by deep learning until 2016 with the publications of GOTURN (Held et al.) 2016) and MDNet (Nam & Han] 2016). These seminal works have spawned successors, many of which still dominate tracking performance in the most popular benchmarks (Kristan et al. 2017] Wu et al.) 2013] Valmadre et al., 2018). This shift was perhaps triggered by the release of ImageNet Object Detection from Video in 2015 (Russakovsky et 2015), the first per-frame annotated large scale video dataset. GOTURN is an example of the so called "tracking by detection" " approach, where tracking is performed by recursive detection on video frames. GOTURN and it's derivatives implement Siamese-networks to directly regress the bounding box coordinates of the target object. These networks typically consists of four main elements: the target network, the search network, the join layer, and the output network. The target and search networks compute feature maps of the corresponding to the target object and the search region, respectively. These feature maps are combined in the join layer: most recent trackers use convolution/correattio to combine the feature maps (Bertinctoo et al.) 2016). Finally, the combined feature maps are fed to the output layer which consists of trainable network that outputs the final bounding box or bounding box proposals (Li et al. 2018 2019 Held et al. 2016). Most recently SiamRPN (Li et al. 2018), SiamRPN++ (Li et al. 2019), and their derivatives have been the dom- inant state-of-the-art approaches to single object tracking. These works combine region proposals popular in the detection literature with Siamese tracking. Later we will introduce detection as an auxiliary task for tracking. Training model to jointly track and detect has recently been proposed in (Feichtenhofer et al.) 2017). However, this work is quite orthogonal to ours as does not consider object instance representation and tracks by temporally linking detected instances.
2

---

Under review as a conference paper at ICLR 2020
Layer conoeeeeee Peneeeee
64x64x2
32*32=384
MobileNet
Conv 1x1
Healmap
.2.x256
64x64x258
Conv 1x1
84*64*384
64x64x4
...X266
Offsets
MobileNet
Conv 1x1
Figure 2:
The Siamese tracker takes the target and search images input and outputs the bounding box proposal heatmap and offsets.
BASIC TRACKING MODEL
Our basic tracking model takes inspiration from recent works starting with GOTURN (Held et al. 2016) and culminating in SiamRPN++ (Li et al. 2019). Our model uses lightweight backbone network (MobileNetV2)), and somewhat simpler than recent state-of-the-ate models. Nevertheless, model contains key elements that common most recent deep trackers and therefore believe our conclusions also apply to these models. Although the model doesn't outperform state- of of-the-art, it attains competitive performance. This section details the architecture of the tracking network, the targets and loss it is trained with, the training data sampling method, data augmentation, and post-processing of the model outputs.
3.1 TRACKING NETWORK
The overall architecture of our tracking network is shown in Figure 2] The tracker takes two images input: the larget image (z;) is the centered crop of the object instance be tracked and the search image (x,) is a crop of the image in which the target is to be localized. The frame corresponding to the target image precedes the frame corresponding to the search image. After re-sizing, the target and search image have shapes 127 127 x and 255 x 255 x 3. respectively. We use the same aspect preserving cropping mechanism detailed in (Bertinetto et al. 2016). Feature maps corresponding backbone network (Howard
to both images are computed using a pre-trained MobileeobieNet-222 al. 2017). Because the weights are shared between the target and search branches the two im- ages are encoded using the same function (i.e. the backbone network) denoted by fe. Feature maps corresponding to the target and search images have shapes 32 32 x 384 and 64 64 x 384, respec- tively. We denote these feature maps 2 fo())) and 2 fo(xz). where is the feature plane index. Next we project these feature maps using a trainable 1 x 1 convolution to 256 dimensions. The resulting target and search features the corresponding target feature map with zero-padding ("same" convolution). to obtain a combined activation map z' 2 that has shape 64 64 X 256. We refer this operation as cross- convolution. A similar operator for combining the target and search features was first introduced in (Bertinetto al. 2016) and has since been employed in many recent tracking architectures. Finally the resulting feature map is projected to six channels using trainable x 1 convolution layer, re- 6. is worth noting, we have found that inserting additional trainable layers after the cross-convolution stage significantly reduces performance.
combined by convolving each search feature map with
sulting in final output of size 64
The tracker outputs n-bounding box proposals and their respective confidence (where n is hyper- parameter we set to 5). Our bounding box proposal parameterization is inspired by the fully con- volutional heatmap and offset parameterization introduced for points (Papandreou et al. (017). which we adapt bounding boxes. The first two of the six output channels represent the center location of the bounding box. These two channels are used to represent the positive and negative classes of the binary heatmap. These channels are trained with binary cross-entropy loss with tar- gets that are constructed by placing a fixed radius (10 pixels) circle ones centered at the location of the ground truth bounding box, the heatmap targets
elsewhere. The remaining
3

---

Under review as a conference paper at ICLR 2020
Detector Output
MobileNet
Join Layer
MobileNet
Join Layer
MobileNet
Tracker Output
Figure 3: Joint tracking and detection model.
four feature maps are used to represent the spatial offsets the top-left and bottom-right bounding box corners relative to their location in the feature map. The offset channels are trained with an L1 regression loss. To produce the final bounding box output we enumerate the top n modes in the heatmap by using non-maximal suppression with fixed square window (6 x 6 pixels) centered on each mode. The mean offset is computed within each window to give the offset corresponding to each bounding box corner. In contrast with (Papandreou al. 2017). do not use Hough voting to aggregate the offsets.
The model is trained on short sub-sequences taken from the ILSVCR 2015 Detection from Video (Russakovsky et al., 2015) and the GOT-10k (Huang et al. 2018) video datasets. To encourage the network to learn invariant representations of the target across long time scales, the target and search images are sampled such that they are unlikely to be temporally proximal frames. Suppose given dataset sequence consists of T frames and sample sub-sequence of length starting at index such that t + < T. The target image is selected uniformly from the interval [0, - 1]. respect the fact that the target template must appear before the search image. Networks were trained with several target image selection mechanisms including always using the first image from the sequence and sampling random causal targets. The latter lead to better performance, likely due to increased diversity. During evaluation, the target image is cropped from the first frame using the ground truth bounding box and held it fixed for the rest of the sequence. We apply the following data augmentation to both target and search images during training: random translation and scaling, as well as random perturbations to the contrast, hue, and saturation.
The model was trained using the Adam optimizer (Kingma & Ba 2014) for 1e7 steps with initial learning rate 10-4 which was reduced by a factor of 10 at 9.5e6 steps. The heatmap and regression loss are combined using weighted sum. The weight corresponding to the heatmap loss is set to 1.0 and regression weight set to 0.3.
3.2 PROPOSAL SELECTION
Proposal selection is performed using the same heuristics as those presented in (Li et al. 2018). Namely, a penalty for abrupt changes between adjacent frames in bounding box aspect ratio and center location is imposed. The penalty is applied to the scores output by the heatmap channel of the network before outputting the final bounding box prediction corresponding to the maximal score of the penalized heatmap.
INDUCING DISCRIMINATIVE TARGET REPRESENTATIONS
Siamese trackers may learn a variety strategies that minimize the per-frame tracking loss. We will show in Section 4 that center saliency detection in the neighborhood of the previously tracked

---

Under review as a conference paper at ICLR 2020
Full
Detector Heatmap
Target Crop
Search Crop
Tracker Heatmap
Offset
Bottom Right Offset
Figure 4: Joint tracking and detection model output. Top: ground truth target crop if input to the target branch. Bottom: random patch from the first frame is input to the target branch.
bounding box attains good performance the sequence consists of a smoothly moving un-occluded object in uncluttered background (Mahadevan & Vasconcelos, 2012). In fact, such tracking strategy is more robust to changes in appearance of the object than an appearance matching strategy. The center saliency strategy does not require a discriminative representation of the target object instance, leaving the target feature branch of the network virtually unused. Unfortunately. the vast majority of sebbeeeuseces sampled during training contain smoothly moving objects with little or no occlusion. Several works (Li et al., 2019; Held et al. (2016) apply data augmentation to the search crop, which reduces center bias learned by the network. Nevertheless manual data augmentation alone limits the types of appearance variations that the tracker can learn. Furthermore, despite discouraging discriminative target representations, centering the target in the search window is an important prior that ultimately leads to better tracking performance.
As mentioned in Li et al. (2019), random translation augmentations during training are crucial to prevent the network from overly relying on simple priors, such center bias and saliency detection, while at the same time improving generalization of search and target representations. Neverthe- less, priors that rely temporal coherence are indeed useful, and so seek a mechanism that encourages discriminative target representations, while allowing the tracker to utilize temporal co- herence. Our proposed solution is add instance driven detection as an auxiliary task. The joint tracker/detector network can be implemented by adding another branch and join layer to the tracking network as shown Figure 3 The detector branch is similar to the search branch except for the type of data it is trained on. The detector is trained on temporally decoherent, un-cropped frames. Unlike the tracker, cannot rely on center saliency detection, and must rely on more discriminative target representation to minimize the detection loss. Because the target representation is shared between the tracker and detector, increasing the weight on the detection task encourages a more discrimina- tive representation of the target (we crucial for avoiding failures in rare but critical portions of the video where the tracker cannot rely on center saliency alone (e.g. occlusion, presence of other salient objects, etc). The detector only outputs a heatmap which is trained to respond at the center of the target object's bounding box in the search image. We omit the offset channels and regression loss in the detector branch.
weight of 1.0 in experiments). Such representation
Figure 4 depicts a representative output of the jointly trained model (tracker-detector) for a frame of the basketball sequence in VOT2018. The Full Frame is the original frame from the dataset. Proposed bounding boxes, output by the tracker are overlaid on the Full Frame image in blue. The final bounding box output is depicted in red. After re-sizing standard resolution of 255 255 the Full Frame is input to the detector, which outputs the Detector Heatmap. In order to visualize two-channel heatmaps, the soft-max computed across the two channels. The Target Crop shows the image that is input to the target branch and held fixed for the entire sequence. The remaining three images visualize the output of the tracker. The Detector Heatmap, defined over the full frame, responds strongly to the non-target players. However, the Tracker Heatmap suppresses it's response to the non non-centered players. Because the detector sees only full frames, it must rely on appearance alone to detect the target, thereby encouraging a more discriminative feature representation of the target. To further confirm this effect we applied the same model to the basketball sequence again, however now we input randomly cropped patch (not containing the image of the target player)
5

---

Under review as a conference paper at ICLR 2020
to the target branch. The Search Crop was still initialized using the ground truth bounding box corresponding to the first frame, ensuring that the target is at the center of the search window. Despite having imput the incorrect Target Crop, the tracker still outputs large logit scores at locations corresponding to the tracked player, located at the center of the Search Crop. Also note that the final bounding box output by the tracker is approximately the same (however the proposals less diverse). In contrast, the logit scores of the detector are much smaller at the player locations, implying that the detector is more sensitive the target representation. To confirm that the tracker is capable of tracking by saliency detection alone, we evaluated the tracker while feeding random patches extracted from the first frame into the target branch. Although the performance significantly decreased (last row in the Table of Figure 5). the tracker still tracks the target in the vast majority of frames and even maintains no failures in some sequences.
40
38
without detector
Model
Robustness
Accuracy
SiamRPN++
14.87
0.5915
36
Robustness
SiamMas
15.40
0,5889
34
SiamRPN
18.39
0.5639
Ours
17.68
0.5562
32
Ours (no detector)
26.61
0,5361
30
Ours (random target)
38.50
0.4740
28
Training steps
Figure 5: Left: Mean and standard error plot of VOT2018 robustness versus training iteration. The error bars were obtained using six random seeds. The dashed line represents the training iteration when the learning rate was reduced. Right: Table comparing final VOT2018 performance to recent state-of-the-art Siamese trackers.
The effect on the number of tracking failures of training with the detection auxiliary task is shown in the plot in Figure 5 The y-axis depicts the supervised robustness of VOT2018. The robustness is defined as the mean number of failures weighted by sequence length in "supervised evaluation" on the entire dataset (60 videos). A failure is triggered when the output bounding box has zero loU with the ground truth. The tracker is reset after failure by the evaluation toolkit (Kristan al., 2017 Huang et al., 2018). The training curves were obtained by averaging the smoothed training curves corresponding to six random initializations for models trained with and without the detector. The standard error curves are shown in light colors. Each point on the curve corresponds to full supervised evaluation on the VOT2018 dataset. This corresponds to a total of 563 full VOT2018 evaluations for each random seed (3378 total evaluations) as opposed to single evaluation typically reported in the tracking literature. In the early stages of training. the detection sub-task actually hin- ders performance, however it helps reduce the number of failures later in training. Our hypothesis is that in the early phase of training, the tracker learns to track by saliency, the detection task actually interferes with learning this strategy. In the later phase of training, the tracker begins to rely more heavily on appearance matching in features space. This is when a discriminative target represen- tation actually proves to be useful. In the following section we performing additional analysis to support these conclusions. For completeness, the table in Figure3.coppare compares our best model with an official implementation of S on the VOT2018 dataset.
PERTURBATION ANALYSIS
The improved robustness achieved by adding the detector can be attributed to several factors. Detec- tion may induce a more robust target representation, on saliency detection, or both. In order to measure these potential effects, we evaluate the sensi- tivity of our architecture to variations in the input conditioning of our network; namely the target location, search location, and target time step. The first experiment is shown in Figure 6 whereby we demonstrate the sensitivity of our tracker to perturbations the search window location using
it may force the tracker to rely less heavily
http:///ithuuh.om/////////

---

Under review as a conference paper at ICLR 2020
the following procedure. First, sample pair of adjacent frames from the VOT dataset and crop a target representation on frame using the ground target location. Next, we crop a search window on the next frame t+] using the target location from frame t, and we infer the predicted box location and calculate loU to the ground truth box at frame /. + 1. We then repeatedly crop additional search windows with displacements in the horizontal and vertical directions, where displacements are scaled by a box size factor (box width, + box height,) /2. We then calculate a heatmap of the IoU change function of this displacement, normalized by IoU the center displacement (0, 0). We then repeat this procedure for all adjacent frame pairs for all sequences in VOT and average the heatmap responses. The resultant heatmap is rendered in Figure
dy
888
83
40.0
44000
866
65
1%
6.4
6.
122.0
130.0
5.0
t target
1+1 search
320.0
-0.00
40.0
120.0
120.0
-2.0
40.0
120.0
de 1% bux skoel
da 1% box stowl
Figure 6: Search window location sensitivity analysis our network. Left: sampling procedure, Middle: fraction of IoU (normalized by the IoU at displacement (0, 0)) for our network without detector, Right: fraction IoU for our network with detector.
As shown in Figure 6 our network is able to infer the next-frame bounding box location despite large displacements the search window. This is largely a result of mitigation of center bias via translation augmentation of the search window during training. Note that 100% shift in bounding box location (relative to the bounding box size) an extremely large spatial displacement at 24fps. At search window displacements larger than approximately 150%, the target is no longer visible in the search window, and so as expected the loU drops to zero. Additionally, there is no significant difference between these response maps of our architecture trained with and without the detector. This suggests that the improved performance obtained by adding the detector cannot be attributed to reduced center bias in the tracker.
We also analyze the sensitivity of the network to target crop location. The procedure is similar to the search window experiment above, except we instead keep the search window constant and vary the position of the target crop. The result of this experiment is shown in Figure and and demonstrates interesting tracking behavior. First, the target moves by small displacements the tracker shifts it's output bounding box by the same fraction, causing an expected sharp roll-off in IoU (since we calculate loU to unmodified target location). This indicates that tracker is indeed paying attention to the target crop in order to infer the object extent. However, if the target crop is moved far enough in either direction (approximately 50% of the box size), the network begins to ignore the target and
3280
120
0.99
dy
0.96
.00.0
6.87
dx
2%
46.0
0.78
123.0
120.0
target
t+1 search
-120
-4.0
40.0
120.0
-120.
40.0
40.0
120.0
(%6 size)
ox 1% box size!
Figure 7: Target crop location sensitivity analysis of our network. Left: sampling procedure, Mid- dle: fraction of IoU for our network without detector, Right: fraction of loU for our network with detector.

---

Under review as a conference paper at ICLR 2020
0.99
with detuctor
0.9
0.97
0.98
034
0.9
dt
t-dt target
t+1
search
Temporal displacement (trames)
Figure 8: Target crop temporal sensitivity analysis of our network. Left: sampling procedure, Right: mean fraction of IoU for our network without and without detector (normalized by the iou at dt 0.)
switches to saliency detection. In this case, the IoU actually increases from the central low to some background constant.
During frame transitions with poor target quality, saliency detection often a desired characteristic it allows the network continue tracking until the target resembles it's template. However, a drop of approximately 15% (difference in background intensity in Figure 7 of bascline IoU highlights the accuracy lost when relying on saliency alone; since the network does not have precise represen- tation of the tracked target, it tends to include other salient features from the background. Note that our jointly trained tracker/detector model outperforms the baseline tracker, where the background IoU lost due to saliency is slightly less.
Lastly, we examine our tracker's 's IoU sensitivity to the temporal displacement between the target frame and the search frame. Similar to the target displacement experiment above, we use constant search window and instead choose a ground truth target bounding box some di frames the past. The further back in time the target cropped, the more "stale" it is. We expect that as the temporal displacement between the current frame and the template grows, the visual appearance of the tem- plate becomes poorer representation of the specific object track, and correspondingly expect some drop in IoU as a result. Here, we normalize the loU by the loU measured when a target at dt 1.
The effect of target staleness on average IoU is shown shown in Figure g Firstly, both of our network variants show asymptotic drop in performance. In the large displacement regime, when direct template matching impossible, the network must use more abstract representation of the target object. Bounding box accuracy suffers in this regime as the network can no longer use the template to infer the true object extent. Instead, it must rely on salient features in the search image. However, some of this performance decrease is mitigated by our network variant that includes the detection branch.
6 CONCLUSION
In this work we presented thorough study of the representations learned by Siamese trackers. We showed that these trackers often rely on saliency detection despite being designed track by template matching in feature space. We proposed auxiliary detection task that induces stronger target representations and improves performance. These findings were empirically validated by performing perturbation experiments.
REFERENCES
Phil Ammirato, Cheng- Yang Fu, Mykhailo Shvets, Jana Kosecka, and Alexander C Berg. Target driven instance detection. arXiv preprint arXiv:1803.044 2018.
Luca Bertinetto, Jack Valmadre. Joao Henriques, Andrea Vedaldi, and Philip HS Tom. Fully- convolutional siamese networks for object tracking. In European conference computer vision,
8

---

Under review as a conference paper at ICLR 2020
pp. 850-866 350-865. Springer, 2016.
Michael D Breitenstein, Fabian Reichlin, Bastian Leibe, Esther Koller-Meier, and Luc Van Gool. Robust berkkinggb--detettion using detector confidence particle filter. In 2009 IEEE 12th Inter- national Conference on Computer Vision, pp. 1522. IEEE, 2009.
Christoph Feichtenhofer. Axel Pinz, and Andrew Zisserman. Detect track and track to detect. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5038-3046 2017.
David Held, Sebastian Thrun, and Silvio Savarese. Learning track at 100 fps with deep regression networks. In European Conference on Computer Vision, pp. 749-765. Springer, 2016.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: /004:0/77004466// 2017.
Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high- sigh-divessity benchmark for generic object tracking in the wild. arXiv preprint arXiv: 1810.1 1981, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv::442.6699 2014.
Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, Luka Ce- hovin Zaje, Tomas Vojir, Gustav Hager, Alan Lukezic, Abdelrahman Eldesokey, al. The visual object tracking vot2017 challenge results. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1949-1972, 2017.
Alex Krizhevsky, lya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High performance visual tracking with siamese region proposal network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8971-8980, 2018.
Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4282-4291, 2019.
Jongwoo Lim, David A Ross, Ruei-Sung Lin, and Ming-Hsure Yang. Incremental learning for visual tracking. In Advances in neural information processing systems, pp. 793-800, 2005.
Vijay Mahadevan and Nuno Vasconcelos. On the connections between saliency and tracking. In Advances in Neural Information Processing Systems, pp. 1664-1672, 2012
Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for vi- sual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4293-4302, 2016.
George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bre- gler, and Kevin Murphy. Towards accurate multi-person pose estimation the wild. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4903-4911, 2017.
Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtube- boundingboxes: A large high-precision human- annotated data set for object detection in video. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5296- 5305, 2017.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.

---

Under review as a conference paper at ICLR 2020
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-253 2015.
Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint 131.xi::1112.2222 2013.
Jack Valmadre, Luca Bertinetto, Joao F Henriques, Ran Tao, Andrea Vedaldi, Arnold WM Smeul- ders, Philip HS Torr, and Efstratios Gavves. Long-term tracking in the wild: A benchmark. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 670-685, 2018.
Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invari- ances. Neural computation, 14(4):715-770, 2002.
Yi Wu, Jongwoo Lim, and Ming- Hsuan Yang. Online object tracking: benchmark. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2411-2418, 2013.
10


Reviews:
[{"review_id": "rJlGv0Watr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper examines the performance of Siamese single-object trackers. The authors claim that state-of-the-art Siamese trackers mostly rely on saliency detection in the center of the search window while ignoring the target instance representation and propose an additional object detection branch during training to mitigate this effect.\n\nStrengths:\n+ Analysis how perturbation target and search image influence tracking performance\n+ Demonstrates that including detection objective during training improves performance\n\nWeaknesses:\n- Makes claims about Siamese trackers in general but only experiments with authors’ architecture\n- Analyses in Figs. 6–8 not very helpful/conclusive\n- No test case where capturing appearance is important for tracking and authors’ approach helps\n- No comparison of tracker trained without detection objective on real vs. random targets\n\nThe paper is well motivated and has a clear hypothesis. However, its execution unfortunately leaves a lot of room for improvement and getting it into acceptable shape would require a major revision. Some details on my main criticisms:\n\nClaim is way too general. The authors state that Siamese trackers in general suffer from the center bias problem. However, the authors do not analyze any other trackers than their own one. I understand that training a state-of-the-art tracker with the additional object detection branch could possibly require resources beyond what’s available to the authors. However, it’s not clear to me why the authors do not use the pre-trained version of at least one or two existing trackers to demonstrate their shortcomings (like Fig. 4)\n\nAnalyses in Fig. 6–8 are not very helpful. The only real effect the detector shows in Figs. 6–8 is a tiny improvement in IoU in Fig. 8 – otherwise the analyses neither support the claim nor do they reveal why detection as an additional objective actually helps. To establish that there is a center bias and trackers ignore the appearance term, the following two analyses could be done:\n\n- Identify a set of test cases where capturing appearance is important (e.g. temporary occlusion) and demonstrate that your approach improves performance on these cases.\n\n- For the table in Fig. 5, also show how a conventional tracker (i.e without object detection branch) performs on random targets. If the hypothesis is correct that the object detection objective during training reduces center bias and increases reliance on appearance, then a conventional tracker should show a smaller reduction due to random targets than your improved tracker does. The same quantitative analysis could also be done for existing state-of-the-art trackers, as it does not require training. \n \nMinor Comments:\n- Caption of Fig. 4: is instead of if?\n- Figure 5\n\t- y-axis label: lower is better despite “robustness” suggesting the opposite\n- Why do the numbers in the figure not match those in the table?\n- What is the center bias baseline, i.e. what is the performance if center of search image is predicted without any network?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJe0cj3z9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n= Summary\nThis paper proposes to learn a visual tracking network for an object detection loss as well as the ordinary tracking objective for enhancing the reliability of the tracking network. The main motivation is that, current state-of-the-art models based on the Siamese architecture often blindly predict the center of a search window as target location due to the bias in datasets and training strategies. This issue is alleviated in this paper by introducing an auxiliary task, target detection in the entire image space. The auxiliary task is conducted by another branch on top of the visual feature shared with the tracking branch. By learning to detect object in the entire image space, the shared feature extractor will be trained to capture discriminative and unique appearance features of target.\n\n\n= Decision\nAlthough the main motivation is convincing and the manuscript is well written, I would recommend to reject this submission mainly due to its limited contribution and weakness in experimental analysis. \n\n(1) In the experiments, the practical benefit of adding the auxiliary detection task is demonstrated, but the final scores of the proposed model are clearly below those of current state of the art in terms of both reliability and accuracy. Further, it is not explained why the proposed model is worse than the other models in performance and what can be claimed as an advantage of the proposed method even in this situation. Also, I do not understand why the proposed model is not based on the current state of the art like SiamRPN++ but is built upon a manually designed/low-performance model. \n\n(2) The experiments in Section 5 do not demonstrate the advantage of the proposed model at all. In Figure 6 and 7, the difference between the proposed model and its reduced version without the auxiliary task looks quite subtle, and it is hard to say which one is better than the others. In Figure 8, adding the auxiliary detection task results in even worse tracking performance. \n\n(3) More qualitatively and quantitatively analysis should be done on the other tracking benchmarks and be compared with other tracking models recently proposed too.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lT_rjdtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates representations learned by Siamese trackers. The paper argues that existing trackers rely on saliency detection despite being designed to track by template matching in feature space. An auxiliary detection task is proposed to induce stronger target representations in order to improve tracking performance. Experiments are performed on VOT2018 tracking dataset.\n\nThe paper investigates an interesting and active research problem of stronger object representations for deep visual object tracking. However, the proposed solution of just integrating an additional detection task branch within the Siamese tracking architecture is naive. The main idea of integrating instance driven detection as an auxiliary task is borrowed from [1].  [1] also utilizes a Siamese architecture that is similar to the ones generally used in visual object tracking to localize particular instances of objects. Therefore, the novelty of the proposed tracking approach is limited.\n\nSome recent works, such as [2, 3] have also investigated a similar problem of richer object representations for deep visual tracking. These approaches are desired to be discussed and empirically compared in order to fully validate the strength of the proposed approach. \n\nThe paper shows some qualitative analysis. However, most of it is limited to just few frames of an image sequence. Tracking datasets, such as VOT and OTB, provide additional analysis tools (i.e., attribute analysis) to thoroughly evaluate visual trackers. Such analysis is missing in the paper. For instance, the main argument of this paper is that current approaches rely on center saliency and likely struggle in the presence of occlusion. How does the proposed approach fare, compared to SOTA, on the subset of VOT image sequences that are labeled with occlusion?\n\nOn page 3, it is stated that \"Our model uses a lightweight backbone network (MobileNetV2), and is somewhat simpler than recent state-of-the-art models  .....................  Although the model doesn’t outperform state-of-the-art, it attains competitive performance.\" The reviewer does not fully agree with this statement. A comprehensive empirical evaluation is crucial to fully access the merits of the contributions. State-of-the-art visual object trackers [2, 3, 4] achieve competitive tracking performance while being computationally efficient and fast. Therefore, a proper state-of-the-art comparison is desired to compare the proposed tracker with SOTA methods. Further, currently experiments are only performed on the VOT2018 dataset. The reviewer recommends to perform additional experiments on other large-scale datasets, such as TrackingNet [5] and Lasot [6] and compare the performance with SOTA methods that are also investigating the problem of richer object representations for tracking. \n\n[1] Phil Ammirato, Cheng-Yang Fu, Mykhailo Shvets, Jana Kosecka, Alexander C. Berg: Target Driven Instance Detection. CoRR abs/1803.04610 (2018).\n[2] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, Michael Felsberg: ATOM: Accurate Tracking by Overlap Maximization. CVPR 2019.\n[3] Goutam Bhat, Martin Danelljan, Luc Van Gool, Radu Timofte: Learning Discriminative Model Prediction for Tracking. CoRR abs/1904.07220 (2019).\n[4] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, Junjie Yan: SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks. CVPR 2019.\n[5] Matthias Müller, Adel Bibi, Silvio Giancola, Salman Al-Subaihi, Bernard Ghanem: TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild. ECCV  2018.\n[6] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, Haibin Ling:\nLaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking. CVPR 2019.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to learn a visual tracking network for an object detection loss as well as the ordinary tracking objective for enhancing the reliability of the tracking network.  The reviewers were unanimous in their opinion that the paper should not be accepted to ICLR in its current form.  A main concern is that the proposed method shows improvement over a relatively weak base system.  Although the author response proposed to include additional analysis, but the reviewers felt that without the additional analysis already included it was not possible to change the overall review score.
Decision: Reject