Do Different Tracking Tasks Require Different Appearance Models?
Authors: Zhongdao Wang
Hengshuang Zhao
Ya-Li Li
Shengjin Wang
Philip Torr
Luca Bertinetto
Venue: NeurIPS 2021 Poster (2021)
Keywords: tracking
self-supervised learning

Abstract:  Tracking objects of interest in a video is one of the most popular and widely applicable problems in computer vision. However, with the years, a Cambrian explosion of use cases and benchmarks has fragmented the problem in a multitude of different experimental setups. As a consequence, the literature has fragmented too, and now novel approaches proposed by the community are usually specialised to fit only one specific setup. To understand to what extent this specialisation is necessary, in this work we present UniTrack, a solution to address five different tasks within the same framework. UniTrack consists of a single and task-agnostic appearance model, which can be learned in a supervised or self-supervised fashion, and multiple ``heads'' that address individual tasks and do not require training. We show how most tracking tasks can be solved within this framework, and that the same appearance model can be successfully used to obtain results that are competitive against specialised methods for most of the tasks considered. The framework also allows us to analyse appearance models obtained with the most recent self-supervised methods, thus extending their evaluation and comparison to a larger variety of important problems. 

Do Different Tracking Tasks Require Different Appearance Models?
Zhongdao Wang!
Hengshuang Zhao 3.4
Ya-Li
Shengjin Wang 1,2+
Philip H.S. Torr
Luca Bertinetto
Beijing National Research Center for Information Science and Technology (BNRist) *Department of Electronic Engincering, Tsinghua University Torr Vision Group, University of Oxford "The University of Hong Kong Fiie AI
https://zhongdao gi github..r So/ononrneeeeeeeeeeeeeeeeeeeeeee
Abstract
Tracking objects of interest in video is one of the most popular and widely applicable problems in computer vision. However, with the years, a Cambrian explosion use cases and benchmarks has fragmented the problem in a multitude of different experimental setups. As a consequence, the literature has fragmented too, and now novel approaches proposed by the community are usually specialised to fit only one specific setup. To understand to what extent this specialisation is necessary, this work we present UniTrack, a solution address five different tasks within the same framework. UniTrack consists of single and task-agnostic appearance model, which can be learned in a supervised or self-supervised fashion, and multiple "heads" that address individual tasks and do not require training. We show how most tracking tasks can be solved within this framework, and that the same appearance model be successfully used to obtain results that competitive against specialised methods for most of the tasks considered. The framework also allows to analyse appearance models obtained with the most recent sulf-super methods, thus extending their evaluation and comparison to larger variety of important problems.
Introduction
Unlike popular image-based computer vision tasks such as classification and object detection, which (for the most part) unambiguous and clearly defined, the problem of object tracking has been considered under different setups and scenarios, each motivating the design of separate set of benchmarks and methods. For instance, for the Single Object Tracking (SOT) and Video Object Segmentation (VOS) communities [70, 29, 48], tracking means estimating the location of an arbitrary user-annotated target object throughout video, where the location of the object is represented by bounding box in SOT and by a pixel-wise mask in VOS. Instead, in multiple object tracking of (often given) detections across video frames to address the problem of identity association and forming trajectories.
settings (MOT [41], MOTS [57] and PoscTrack [2]), tracking means connecting
Despite these tasks only differing in the number of objects per frame to consider and observation format (bounding boxes, keypoints or masks), the best practices developed by the methods tackling them vary significantly.
"Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Though the proliferation of setups, benchmarks and methods is positive in that it allows specific use cases to be thoroughly studied, we argue it makes inereasingly harder effectively study one of the fundamental problems that all these tasks have in common, i.e. what constitutes good representation to track objects throughout a video? Recent advancements in large-scal sige-scale models for language [15, 6] and vision [24, 10] have suggested that strong representation can help addressing multiple down- stream tasks. Similarly, we speculate that a good representation is likely to benefit many different tracking tasks, regardless of their specific setup. In order to validate our speculation, in this paper we present a framework that allows to adopt the same appearance model to address five different tracking tasks (Figure 2). In our taxonomy (Figure 4), consider existing tracking tasks problems that have either propagation or association at their core. When the core problem is propagation (as in SOT and VOS), one has to localise target object in the current frame given its location in the previous one. Instead, in association problems (MOT, MOTS, and Pose' (oseeTrack), target states in both previous and current frames are given, and the goal is determine the correspondence between the two sets of observations. We show how most tracking tasks currently considered by the community can be simply expressed starting from the primitives of propagation association. For propagation lasks, we employ existing box and mask propagation algorithms [5, 61, 58]. For association tasks, we propose a novel cconstruction--bsed metric that leverages fine-grained correspondence measure similarities between observations. In the proposed framework, each individual task is assigned to a dedicated "head" that allows to represent the object(s) in the appropriate format to compare against prior arts on the relevant benchmarks.
Note that, in framework, only the appearance model contains parameters that can be learned via fack-propagation and that we do not experiment with appearance models that have been trained on specific tracking tasks. Instead, we adopt models trained via recent self-supervised learning (SSL) techniques and that have already demonstrated their effectiveness on a variety of image-basee tasks. Our motivation twofold. First, SSL models are particularly interesting for our use-case, as they are explicitly conceived to be of general purpose. As a byproduct, our work also serves the purpose of evaluating and comparing appearance models obtained from self-supervised learning approaches (see Figure 1). Second, we hope to facilitate the tracking community in directly benefiting from the rapid advancements of the self-supervised learning literature.
To summarise, the contributions of our work are as follows:
We propose UniTrack, a framework that supports five tracking tasks: SOT [70], VOS [48]. MOT MOTS [57], and TrackT [2]; and that can be easily extended to new ones.
We show how UniTrack can leverage many existing general-purpon purpose appearance models to achieve a performance that is competitive with the state-of-the-aas on several tracking tasks.
We propose a novel reconstruction -asod similarity metric for association that preserves fine-grained visual features and supports multiple observation formats (box, mask and pose).
We perform extensive evaluation of self-supervised models, significantly extending the empirical analysis prior literature to video-based tasks.
2
The UniTrack Framework
2.1
Overview
Inspecting existing tracking tasks and benchmarks, we noticed that their differences can be roughly categorised across four axes, illustrated in Figure 2 and detailed below.
1. Whether the requirement is to track a single object (SOT [70, 29], VOS [48]), or multiple objects (MOT [48], MOTS [57], PoseTrack [2]).
2. Whether the targets are specified by a user the first frame only (SOT, VOS), or instead are given in every frame, e.g. by pre-trained detector (MOT, MOTS, PoseTTrack)
3. Whether the target objects are represented by bounding-boxe (SOT, MOT), pixel-wise masks (VOS, MOTS) or pose annotations (roseTrack).
4. Whether the task class-agnostic, iee. the target objects can of any class (SOT, VOS); or if instead they are from predefined set of classes (MOT, MOTS, PoscTrack).

---

InsDis
McCo.v1
PCL-v1
PCL-V2
SimCLR-V1
McCo-v2
SimCLR
SeLaiv2
InfoMin
BYOL
DeepCluster
SWAV
Barlow Twins
DetCo
VFS
Figure High-level overview the performance sixteen self- supervised learning models on five tracking tasks: SOT, VOS, MOT, Posc'rackkng and MOTS. A higher rank (better performance) corresponds to a vertex nearer to the outer circle. A larger area of the pentagon signifies better overall performance of its respective appearance model. Results of vanilla maageNet-superviseee model indicated with gray dashed line reference. Notice how the best model VFS [74] dominates on four the five tasks considered.
Single object, user-specified
Multiple objects, detector-specified
Box
Level-1
Appearance
Model
Task Single Object Tracking (S0T)
Task: Muti Object Trocking (MOT) Datasetsc MOT challenge, RITII
Datisets: (8-200, VOI-16/13
Clor-aghertic
Gooss-peeett
Mask
Level-2
Propagation
Association
lnn: Video Ubject Segnertatioon (VLD))
Dalisets: UNV J016/ 2017
Carn-ggennt
Gons-opergtt
Level-3
Pose
SOT
vos
MOTS
MOT
PaseTrack
Clasi-spedffe
Gbos-upeeifi
Figure 2: Existing tracking problems and their respective benchmarks differ from each other under several aspects: the assumption could that there is single or multiple objects to track: targets can be specified by the user in the first frame only, assumed to be given at every frame (e.gr be known (class-
Figure 3: Overview of UniTrack. The framework can be divided three levels. Level-I: a trainable appearance model. Level-2: the fundamental primitives of propa- gation and association. Level-3; task task-specific heads.
provided by detector); the classes of the targets specific) or unknown (class-agnosse agaassagnnst;;);; the representation of the targets can be bounding boxes, pixel-wise masks, or pose annotations.
Typically, in single-object tasks the target is specified by the user in the first frame, and it can be of any class. Instead, for multi-object tasks detections are generally considered given for every frame, and the main challenge is to solve identity association for the several objects. Moreover, in multi-object tasks the set of classes to address is generally known (e.g. pedestrians cars).
Figure 3 depicts a schematic overview of the proposed UniTrack framework, which can be understood as conceptually divided in three "levels". The first level is represented by the appearance model, responsible for extracting high-resolution feature maps from the input frame (Section 2.2). The second level consists of the algorithmic primitives addressing propagation (Section 2.3) and association (Section 2.4). Finally, the last level comprises multiple sasc-specific algorithms that make direct use of the primitives of the second level. In this work, illustrate how UniTrack be used to obtain competitive performance on all of the five tracking tasks of level-3 from Figure Moreover, new tracking tasks can be easily integrated.
Importantly, note that the appearance model is the only component containing trainable parameters. The reason opted for shared and non task saek-specific representation is twofold. Firstly, the large amount different setups motivated us investigate whether having separately-trained models for each setup is necessary. Since training specific datasets can bias the representation towards limited visual concepts (e.g. animals or vehicles) and limit its applicability to "open-world" settings, we wanted to understand how far shared representation go. Second, wanted to

---

Inpu
Prediction
Predictonss
Pranc A
Forter
Franze
Frmeet
Propagation
(b) Association
Figure 4: Propagation v.s. Association. In the propagation problem, the goal is to estimate the target state at the current frame given the observation in the previous one. This is typically addressed for one object at the time. In the association prohlem, observations in hoth previous and current frames are given, and the goal is to determine correspondences hetween the two sets.
provide the community with multiple baselines that can be used to better assess newly proposed contributions, and that can be immediately used on new datasets and tasks without the need of retraining.
2.2 Base appearance model
The base appearance model takes as input 2D image I and outputs a feature map X = Rd x W xc, Since ideally appearance model used for object propagation and association should be able to leverage fine-grained semantic correspondences between images, we choose a network with a small stride of 8, so that its output in feature space can have relatively large resolution.
We refer the vector (along the channel dimension) of single point in the feature map as point vector. We expect a point vector of E RRC from the feature map X1 to have a high similarity with its "true match" point vector zi in X2, while being far apart from all the other point vectors zz in X2; Le. we expect 6 sx)j, > s(xj, xg), Vj A i, where s(,) represents a similarity function.
In order to learn fine- mne-grained correspondences, synthetic datasets (e.g. Flying Chairs for optical flow [16]). With real-world data, it intractable to label pixel-Ievel correspondences and train models in a fully-supervised fashion. To overcome this obstacle, in this paper we adopt representations obtained with self-supervision. We experiment both with models trained with approaches that leverage pixel-wise pretext tasks [27, 58) and, inspired by prior works that have pointed out how fine- fine-graine greinga correspondences emerge middle-level features [39, 74], with models obtained from image-level tasks (e.g. MoCo [24], SimCLR [10]).
sully-supervieed methods are only amenable for
2.3 Propagation
Figure 4a schematically illustrates the problem of propagation, which we use as a primitive to address SOT and VOS tasks. Considering the single-object case, given video frames {1))_ and an initial ground truth observation for each time-step t. In this work we consider three formats to represent objects: bounding boxes, segmentation masks and pose skeletons.
Problem definition.
input, the goal is to predict object states {
Mask propagation. In order to propagate masks, we rely on the approach popularised by recent video self-supervised methods [27, 58, 35, 31]. Consider the feature maps of a pair of consecutive frames X,-1 and X,, both E R' and the label mask 2t-1 [0. 1] of the previous frame 2, where = H x W indicates its spatial resolution. We compute the matrix of transitions K;_, [lx as the affinity matrix between X,-1 and X,. Each element ki.s is defined as
(xp((x)-1:<1) /T)
ki. Softmax(X-- :7)g
(I)
exp((xt_11x+)) /T)
where (.--) indicates inner product, and is a temperature hyperparameter. As [27], we only keep the top K values for each row and set other values to zero. Then, the mask for the current frame at time i predicted by propagating the previous prediction: 2, = Ki_ -- Mask propagation
proceeds in recurrent fashion: the output mask of the current frame is used as input for the next one.
Note this corresponds to the ground-truth initialisation when t 1, and the latest prediction otherwise.

---

Pose propagation. In order to represent pose keypoints, we use the widely adopted Gaussian belief maps [66]. For a keypoint p. we obtain a belief map 2" E [0, 1]a by using a Gaussian with mean equal to the keypoint's location and variance proportional to the subject's body size. In order to propagate a pose, we can then individually propagate each belief map in the same manner as mask propagation, again as of = -
Box propagation. The position of an object can also be more simply expressed with four- dimensional vector z = (u, v. w, h), where (u, v) are the coordinates of the bounding-box center, and (w. h) are its width and height. While one could reuse the strategy adopted above by simply converting the bounding- boox to a pixel-wise mask, we observed that using this strategy leads to inacourate predictions. Instead, we use the approach of SiamFC [5], which consists in performing cross-corred correlaorol (XCORR) between the target template 2;-1 and the frame X, to find the new location of the target in frame representation can be resized accordingly. We also provide Correlation Filter-based alternative (DCF) [54, 61] (see Appendix B.
correlatrtl is performed at different scales, so that the bounding-bow -box
2.4 Association
Problem definition. Figure 4b schematically illustrates the association problem, which we use as primitive to address the tasks of MOT, MOTS and PoseTrack. In this case, observations for object states |Tm are given for all the frames {1,)7-, typically via the output of a pre-trained detector. The goal here is to form trajectories by connecting observations across adjacent frames according to their identity.
Association algorithm. We adopt the association algorithm proposed in JDE [65] for MOT, MOTS and PoseTrrack tasks, of which detailed description can be found in Appendix C.I. In summary, we compute an N x M distance matrix between N alrcady-existing tracklets and M "new" detections from the last processed frame. We then use the Hungarian algorithm [30] to determine pairs of matches between tracklets and detections, using the distance matrix as input. To obtain the matrix of distances used by the algorithm, we compute the linear combination of two terms accounting for motion and appearance cues. For the former, we compute a matrix indicating how likely a detection corresponds to the object state predicted by a Kalman Filter [28]. Instead, the appearance component is directly computed by using feature-map representations obtained by processing individual frames with the appearance model (Section 2.2). While object-level features for box and mask observations can be directly obtained by cropping frame-level feature maps, when an object is represented via pose it first needs to converted a mask (via a procedure described Appendix C.2).
A key issue of this scenario is how to measure similarities between object-level features. We find existing methods limited. First, objects are often compared by computing the cosine similarity of owerage-pooled object-level feature maps [84, 51]. However, the operation of average inherently discards local information, which is important for fine-grained recognition. Approaches [18, 52] that instead to some extent do preserve fine-graine grained information, such as those computing the cosine representation
similarity of (flattened) feature maps, do not support objects with differently (situation that occurs for instance with pixel-level masks). To cope with the above limitations, we propose a reconstruction-be based similarity metric that able to deal with different observation formats, while still preserving finennrained information.
Reconstruction Similarity Metric (RSM). Let [ denote the object-level features of N existing tracklets, f; E R xC and St, indicates the spatial size of the object, i.e. the area of the box or the mask representing it. Similarly, {dz} 1d
د denotes the object-level features of M new detections. With the goal of computing similarities obtain an N x M affinity matrix feed to the Hungarian algorithm, we propose a novel reconstruction-bassee similarity metric (RSM) between pairs (i,j), which is obtained as
RSM(i.)) 1 (cos(tt, f(-s)) cos(d,, d)-1)),
(2)
where Eee-y represents t; reconstructed from d, and d represents d, reconstructed from f.. In multi-object tracking scenarios, observations are often incomplete due to frequent occlusions. As such, directly comparing features between incomplete and complete observations often fails because of misalignment between local features. Suppose dj is a detection feature representing a severely occluded pedestrian, while t; a tracklet feature representing the same person, but unoccluded. Likely,
5

---

Cumret descslans
6.
Tha suckle funan
dg. They j-bh denee bun
Fltten me
e 2,2/51y
Ru. #fratur vecturs ofs,
d: Foulam dinscaion
Tronicso
Re./ Roccommaction namt
Reconstruct,, (10mid4
[ - Rp.:4;
Excchnng and moonstrantd, from St.
yielding d...
Recoustrusson Smillrity Mat::
R
Rus
RSME
T BZinged
Affinily matrix A = softmax(TO") (row-wise sottmax norallized)
Figure 5: Reconstruction Similarity Metric (RSM): First, object-level features of existing tracklets and current detections are flattened and concatenated. Then, an affinity matrix between the two feature sets is computed. For a pair of tracklet &, and detection dz. we "extract" the corresponding sub matrix from the entire aftinity matrix as linear weights and reconstruct & from d, using these linear weights. The similarity between the original object-level feature and its reconstructed version is finally taken the RSM. We want the metric to be symmetric, we perform reconstruction both forward (f, < d,) and backward (f, + dj)-
directly computing the cosine similarity between the two will not be very telling. RSM addresses this issue by introducing a step of reconstruction after which the co-oceurring parts of point features will be better aligned, thus making the final similarity more likely to be meaningful.
The reconstructed object-level feature map & is a simple linear transformation of dj. i.e. be- = Re-jdjj. where Ri-jj E R² K*ay is a transformation matrix obtained as follows. We first flatten and concatenate all object-level features belonging to tracklet (i.e. the set of ob- servations corresponding
object) into single feature matrix T E RRC, syxxC Simi- larly, we obtain all the object-level feature maps of a new set of detections D E RR )x0 Then, we compute the affinity matrix A = Softmax(TD") and "extract" individual Ra-j map- pings as aat-matriees of A with respect the appropriate (i.j) tracklet-detection pair: R =
A E Sy E Sy' E Sy E 8y For a schematic representation of the procedure just described, see Figure 5.
RSM can be interpreted from an attention [55] perspective. The feature map of tracklet 4, being reconstructed can be seen as a set of queries, and the "source" detection feature dj can be interpreted both keys and values. The goal is reconstruct the queries by linear combination of the values. The linear combination (attention) weights are computed using the affinity between queries and keys. Specifically, we first compute a global affinity matrix between t; and all the dy for j' L.... M, and then extract the corresponding sub-matrix for t, and dy as the attention weights. Our formulation leads to a desired property: the attention weights approach zero, the corresponding reconstructed point vectors will approach zero and so the RSM between f; and dj.
Measuring similarity by reconstruction is popular in problems such few-shot learning [67, 80], self-supervised learning [38], and person re-identification [26]. However, reconstruction is typically framed as a ridge regression optimal transport problem. With Onn²) complexity, RSM is more efficient than ridge regression and it has a similar computation cost to calculating the Earth Moving Distance for the optimal transport problem. Appendix D shows series of ablation studies illustrating the importance of the proposed RSM for the effectiveness of UniTrack on association- typp tasks.
3 Experiments
Since UniTrack does not require task-specific training, we were able to experiment with many alternative appearance models (see Figure 3) with little computational cost. In Section 3.1 we perform an extensive evaluation to benchmark a wide variety off-the-shelf, modern self-supervised models, showing their strengths and weaknesses on all five tasks considered. In this section we also conduct a correlation study with the so-called "linear probe" strategy [81], which became a popular
'Here we use numpy- matrix slicing notation to represent a submatrix, Lee. Ali j. k indicates submatrix of A with row indices ranging from to j and column indices ranging from to l.

---

way to evaluate representations obtained with self-supervised learning. Then, in Section 3.2 we compare UniTrack (equipped with supervised or unsupervised appearance models) against recent and task-specific tracking methods.
Implementation details. We use ResNet-18 [25] or ResNet-50 as the default architecture. With ImageNet-suuprrised appearance model, we refer to the ImageNet pre-trained weights made available in PyTorch's "Model Zoo". To prevent excessive downsampling, we modify the spatial stride of 1ayer3 and layer4 to 1, achieving a total stride of = 8. We extract features from both layer3 and layer4. We report results with layer3 features when comparing against aask-specific methods (Section 3.2), and with both layer3 and layer4 when evaluating multiple different representations
(Section 3.1). Further implementation details are deferred to Appendix and C.
Datasets and evaluation metrics. For fair comparison with existing methods, we report results on standard benchmarks with conventional metrics for each task. Please refer to Appendix A for details.
3.1 UniTrack as evaluation platform of previously-learmed representations
The process of evaluating representations obtained via self-supervised learning (SSL) often vnvolves additional training [17, 24, 10], for instance via the use of linear probes [81], which require fix the pre-trained model and train an additional linear classifier on top of it. In contrast, using UniTrack as evaluation platform (1) does not require any additional training and (2) enables the evaluation
battery of important video tasks, which have generally been neglected in self-supervised suparvised-leariing papers in favour of more established image-Ievel tasks such as classification.
In this section, we evaluate three types of SSL representations: (a) Image-level representations learned from images, e.g. MoCo [24] and BYOL [21]: (b) Pixel-level representations learned from images (such as DetCo [72] and PixPro [73]) and (c) videos (such as UVC [35] and CRW [27]). For all methods considered, we use the pre-trained weights provided by the authors.
Results are shown in Table and 2, where we report the results obtained by using features from either layer3 or layer4 of the pre-trained ResNet backbone. We report both results and separate them by a Yf' in the table. Note that, for this analysis only, for association-tyy type tasks motion cues are discarded to better highlight distinctions between different representations and avoid potential confounding factors. Figure provides high-level summary of the results by focusing on the ranking obtained by different SSL methods the five tasks considered (each represented by a vertex in the radar-style plot). Several observations can be made:
(1) There is no significant correlation between "linear probe accuracy" on ImageNet and overall trucking performance. The linear probe approach [81] has become a standard way to compare SSL representations. In Figure 6, we plot tracking performance on five tasks (y-axes) against ImageNet top-1 accuracy of different models (x-axes), and report Pearson and Spearman (rank) correlation coefficients. We observe that the correlation between ImageNet accuracy and tracking performance is small, i.e. the Pearson's 's r ranges from -0.38 to +0.20,, and Spearman's 's p ranges from -0.36 to +0.26. For most tasks, there is almost correlation, while for VOS the two measures are mildly inversely correlated. The result suggests that evaluating SSL models on five extra tasks with UniTrack could constitute useful complement to ImageNet linear probe evaluation, and encourage the SSL community to pursue the design of even more general purpose representations.
(2) A vanilla ImageNet-truinedd supervised representution is surprisingly effective across the board. On most tasks, it reports a performance competitive with the best representation for that task. This is particularly evident from Figure 1, where its performance is outlined as gray dashed line. This result suggests that results obtained with vanilla ImageNet features should be reported when investigating new tracking methods.
(3) The best self-supervised representation ranks first on most tasks. Recently, it has been shown how SSL-trained representations can match or surpass their supervised counterparts on ImageNet classification (e.g. [21]) and many downstream tasks [17, 72]. Within UniTrack, although no individual SSL representation is able to beat the vanilla ImageNet-trained representation on every single task, we observe that the recently proposed VFS [74] ranks first on every task, except for single-object tracking. This suggests that advancements of the sulf-suue learning literature can directly benefit the tracking community: it is reasonable to expect that pewly-proposed representations will further improve performance across the board.
7

---

Representation
SOT (70)
VOS (48]
MOT
MSTS|57]
PoseTrack [2]
AUXCatoex 1
AUCber 1
3 J-meant
HOTA?
I0FIT
HOIAT
IDP1
1Ds
Rondow frit.
10.3/9.
28.0/200
/339
84/89
84/85
208/2311
25.0/28.7
40.2/38.5
88792/ 99193
InngeNettmmp map.
58.67:99.5
62.0/539
62.3/57.99
75.6/732
63376L8
68.47 69.4
70.277L8
73/7/73.3
6969/7101
TnsDis [7]]
47.6747.3
618/51.1
62.6760.1
66.7/739
57.96619
68.4 68.0
60.6/70.3
72.4/739
7106/ 2015
69.2/74.1
69,47619
70.67 89.3
71.6770.9
72.8/739
6872/7092
PCEV1 [34]
56.8731.5
35.3/350
60.1 38.8
74.8 68.8
(2.1159.1
67.6/ 65.2
69.7 167.3
73.3/711
10694
PIRI [42]
43.8/51.
61.2/53.4
65.7
62.01 73.
54 64./66.9
66.07 67.4
66.7/69.9
/
7235/7173 7173
PCLV2 [D]]
54,9/50.3
62.5/516
61./ 52.5
74.97229
62.7 16.8
68.3/ 66.6
76.5
73.5/73.4
6859 88489
SimCLRVI [10]
51.3
56.5
$7.77 63.2
72.3/73.
/
M
53.7:
53.3
61.1/54.0
72.01 74.9
61.2/62.8
67.5/ 67.3
169.6
73.0/733
6932/ 7702
SimCLRV2 [1]]
50,0154.7
61.7/ 86.8
68.4/5844
67.6775.7
SS.1/63.3
69.1
7
72.5/
7228/ 0228/7855 78556
[e]aV2[]]
59.0/9.6
63.17 14.2
60.31 40.2
68.8/ 68.9
$9.07
66.8 66.
68.7 66.7
72.9/ 72.3
6963/7815
Infomin [53]
51.
66.7/734
1661
68.8
72.5 74.0
7066/
BusLow [79]
65.5/55.5
60.51 60.1
61.7157.8
63.7/74.5
55.41
68.7 1
69.57
12.31 74.3
7131 76556
BYOL
45.3/55.5
58,9 $6.8
65.3/749 74.9
S6.8 $6.8/62.9
70.1 66.8
70.8/69.3.
7
7213 8032
DespClester11) [7]
51,57529
61.2/6122 61.2
59,3153.4
66.9/
51,8663,5
67.7
66988
72.7/73.7
7018 7283
SwAV [S]
69.2:52.4
65.6/
/69.5
72.7
/777
VTS [74]
51.1
60.3/43.
62.8 15688
74.17 77.0
62.67 63.9
71.07 68.0
72.17 22:1770/4
74.3
6731 709)
PixPro [73]
40.5:449.2
47.4
52.4
61.7/66.7
58.6
64.2/ 66.1
167.6
72.4/73.1
7163/ 6953
DelCe [?2]
55.0/47.11
53.2
62.3/5561
75.3/72.9
61.8/61.6
67.8/ 66.8
70.0/69
/3.9/73.3
7357 8009
TimeCycle [64]
24.2
57.5/48.7
51.87 48.9
68.7/28.
25,5
69.9/471
71.3749.3
72.0/62.3
7837 27684
Table 1: Tracking performance of pre- pre-trained image- based SSL models. All methods employ a ResNet-50.
Representation
VOS1-51
MDI HI]
MO1S|77
Poselrack 121
AUC
J-meant
IDF1
HOTAT
IDFT
HOTAT
IDF1
ID+L
Randon Iniz.
16./18.2
36.1/ 32.1
33.0/ /6.7
18.4/ 14.6
20.2/123
34,5/
39.9/ 37.6
$2.8/503 50.5
65317/ 66020
ImageNet-nap rup.
55.07
61.87552.
58.4/ 46.7
24.8770
62.3
61.6/ 68.6
09.87 70.5
6808/ 7024
Color. [58]+mem.
41.6/455
56.7/ 58.7
53.67 59.7
56.8 55.5
68.8 66.1
69.4/ 66.3
72.472.6
UVC[ES]
/ 38.7
59.9
55.9
66.9/64.5
57.7/
69.9768.7
69.6 69.4
72.6/72.8
6843 69772
CRW [27]
40.37 49.1
54.9/5499
63.2 660.7
67.87 73.0
58.4/111
69.07 71.3
09.27 71.9
72/7/73.0
67997 6761
Table 2: Tracking performance of pre-trained video-bused SSL models. All methods employ ResNet-18. In the above two tables, we report results with [layer3/ / 1ayer4] features in each cell, and the best performance between the two is bolded. We use the bolded values to rank the models in each column, and visualise (column- wise) better performance with darker cell colors. Best results each column are underlined.
SOT
Vos
MOT
MOTS
ToseTrack
rxpeer ovt
IraD Macovt
Sulaii
Inloiin
PCL.vf
RVCL
PRRL
Daescholern2
1:78
0.1
StrCLBst
McCovv
DatDo
0.0 Imagehiat Toa-1 Acc.
112
Iragahal Top-1 Acc
IragelesTop- Acc.
Imagetiet Top- 10-11Ac Acc
Imagelie Top-1 Acc
Figure 6: Tracking performance is poorly correlated with ImageNet accuracy. On the x-axes plot ImageNet linear probe top-1 accuracy and on the y-axes the tracking performance five tracking datasets. Correlation coefficients (Spearman's p and Pearson's 's r) are shown in the left bottom of each plot.
(4) Pixel-level SSL representations do not seem to have a consistent advantage in pixel-level tasks. In Table 2 and at the bottom of Table we compare recent SSL representations trained with pixel- level proxy tasks: PixPro [73], DetCo [72], TimeCycle [64], Colorization [58], UVC [35] and Contrastive Random Walk (CRW) [27]. Considering that pixel-level models leverage more fine- grained information during training, one may expect them to outperform image- image-based models in the tracking tasks where this important. is not straightforward to compare pixel-level SSL models with lmage-level ones, as the two types employ different default backbone networks. However, note how good image-based models (MoCo-v1, SimCLR-v2) are on par with their supervised counterpart in all tasks, while good pixel-level models (DetCo, CRW) still have gaps with respect to their supervised counterparts in tasks like SOT and MOT. Moreover, from Table 1, one can notice how the last three rows, despite representing methods leveraging pixel-level information during training, actually outperformed by image-level representations on the pixel-level tasks of VOS, MOTS and PoseTrack.
(5) Video data can benefit representation learning for video tasks. The top-ranking VFS is similar to MoCo, SimCLR and BYOL in terms of learning scheme: they all perform contrastive learning on image level features. The most important distinction is the training data. Previous SSL methods mostly train on still-image based datasets (typically ImageNet), while VFS employs large-scale video dataset Kinetics [9]. Clearly, this is not very surprising, as training on video data can help closing the domain gap with the bided- downstream tasks considered in this paper.
8

---

Methods
IDFIT
[Ds
MOTAT
HOTA?
Melhods
IDFIT
IDs]
<MOTAT
[65]
55.8
1344
64.4
CTTacker [47]
57.2
1897
67.6
48.8
TackRCNN [57]
42.4
567
40.6
TubelK
1236
57.3
MAT [23]
928
PountTrack [76]
42.9
TraDes [19]
64.7
1144
70.1
53.2
GMPHD [S0]
65.6
366
60.0
CSTrack
71.8
1071
70.7
59.8
COSTA
Fair [82]
UniTrack ImageNer?
67.2
622
63.9
UniTrack ImagoNert
71.8
683
74.7
99.
Unilinck_ VPS
68.2
69.7
UniTrack_ VPS
70.3
829
(b) MOTS MOTS split.
(a) MOT MOT 16 [41] text split, private detection.
Methods
J-mocart
Mathods
AUCC
Sipprvised:
Methods
OpenSVAI [4]]
MDPN [22]
DPFt
Ws.
MOLA!
62.4
50.6
Sipervized::
STM [15]
SaamMask [62]
[56]
54.3
79.2
SammPN [53]
SiamFC[S]
SiamRPN-- [32]
58.2
Miracle [78]
KeyTrack [19]
64.0
Unsupervized:
Unsupervized:
TWVA [19]
64.7
Colorization [58]
34.6
UDT [59]
59.4
LightTrack!
52.2
64.8
TymeCylee [61]
[VC
40.1
LUDI (60]
UDT+ 159]
63.2
JalltackkImmgeNet UniTrcc_ VTS
73.2
CRW [27]
64.8
LUDT+ [S0]
(c) PoseTTrnnke@o//TTooo [2] split.
742
7091
63.
UniTrck ImageNer
UniTack VFS
62.8
Unitrack_ ImageNeLDDFF
Unittack, ImageNet XCors
UniTrack VTS_
55.5
60.3
61.8
(d) VOS@ [48].
(e) SOTTOTB-2015 [70].
Table 3: Comparison with task-tailored unsupervised and supervised methods five typical tracking tasks. f
indicates methods using identical observations.
3.2 Comparison with task-specific tracking methods
Unsupervised methods. We observe that UniTrack performs competitively against unsupervised state-of-the-art methods in both the propagation-type tasks considered (Table 3d and 3e). For SOT, UniTrack with DCF head [61] outperforms UDT [59] (a strong recent method) by 2.4 AUC points, while is surpassed by LUDT+ [60] by 2.1 points. Considering that LUDT+ adopts an additional online template update mechanism [13] while ours does not, we believe the gap could be closed. In VOS, existing unsupervised methods are usually trained on video datasets [35, 27], and some of the most recent outperform UniTrack (with an ImageNet-trainee representation). Nonetheless, when we use VFSSttrinee representation, this performance difference reduced 2%. Finally, note that for association-tyy tasks we are not aware of any existing unsupervised learning method, and thus in this case we limit the comparison to supervised methods.
Comparison with supervised methods. In general, UniTrrack with a ResNet- 18 appearance model already performs on par with several existing task-speeific supervised methods, and in several tasks even shows superior accuracy, especially for identity-related metries. (1) For SOT, UniTrack with DCF head outperforms SiamFC [5] by 3.6 AUC points. This is significant margin considering that SiamFC is trained with large amount of crops from video datasets with annotated bounding boxes. (2) For VOS, UniTrack surpasses SiamMask [62] by 4.1 T -mean points, despite this being trained on the joint set of three large-scale scale video datasets [37, 14, 75]. (3) For MOT, we employ the same detections used by the state-of-the-an art tracker FairMOT [82]. The appearance embedding FairMOT is trained with 270K bounding boxes of 8.7K labeled identities, from MOT-specific dataset. In contrast, despite our appearance model not being trained with any MOT-specific data, our IDFI score is quite competitive (71.8 8 v.s. 72.8 of FairMOT), and the ID switches are considerably reduced by 36.4%. from 1074 to 683. (4) For MOTS, we start from the same segmentation masks used by the COSTA [1] tracker, and observe degradation in terms of ID switches (622 the 421 of the state of the art), and also a gap in IDFI and sMOTA. (5) Finally, for pose tracking. we employ the same pose estimator used by LightTrack [43]. Compared with LightTrack, the MOTA of Trickack degrades of 1.3 points because of an increased amount of ID switches. However, the IDF-1 score improved by significant margin (+21.0 points). This shows UniTrack preserves identity more accurately for long tracklets: even if ID switches occur more frequently, after a short period UniTrack is able to correct the wrong association, leading higher IDF-1.
Notice how, overall, UniTrack obtains more competitive performance on tasks that have association at their core, i.e. MOT, MOTS and PoscTrack. Upon inspection, we observed that most failure cases

---

in propagation-yye type tasks regard the "drift" occurring when the scale of the object is improperly estimated. In future work, this could be addressed for instance by a bounding- regression module to refine predictions, or by carefully designing a motion model. For association-type tasks, the consequences of any type of inaccuracy are isolated to individual pairs of frames, and thus much less catastrophic by nature.
4 Related Work
To the best of our knowledge, sharing the appearance model across multiple tracking tasks has not been extensively studied in the computer vision literature, and especially not in the context of SSL representations. Some existing methods do share a common backbone architecture across tasks. For instance, STEm-Seg [4] addresses VIS [77] and MOTS; while TraDeS [69] addresses MOT, MOTS and VIS. However, both methods need to be trained separately and on different datasets for every task. Conversely, we reuse the same representation across five tasks. A promising direction for future work would be to use UniTrack to train a shared representation in multi-task fashion. Only few relevant works do adopt multi-task approach [62, 83, 40], and they usually consider SOT and VOS tasks only. In general, despite the multi-task direction being surely interesting, it requires the availability of large-scale datasets with annotations in multiple formats, and costly training. These are two of the main reasons for which we believe that having a framework that allows to achieve competitive performance on multiple tasks with previously-traie trained models is a worthwhile endeavour.
Self-supervised model evaluation. Given the difference between the pretext tasks used to train self-supervised models and the downstream tasks used to evaluate them, the comparison between self-supervised approaches has always been a delicate matter. Existing evaluation strategies typically require additional training once seneral-purpose representation has been obtained. One strategy keeps the representation fixed, and then trains additional task-specific heads with very limited capacity (e.g. linear classifier [20, 10, 24] regression head for object detection [20]). A second strategy, instead, leverages SSL to obtain particularly effective initializations, and then proceeds to fine-tune such initialized models the downstream task of interest. A wider range of tasks can be tested using this setup, such as semantic segmentation [17, 20] and surface normal estimation [20, 63). In contrast, UniTraak provides a simpler way to evaluate SSL. models, one that does not require additional training or fine-tuning. Also, this work is the first to extend SSL evaluation to set of diverse video tasks. We believe this contribution will allow the study of self-supervised learning methods with a broader scope of applicability. Our work is also related to a line of self-supervised learning methods [27, 58, 35, 31] that learn their representations in task-agnostic fashion, and then test it on propagation tasks (SOT and VOS). The design of UniTrack is inspired by their task-agnostic philosophy, while significantly extending their scope a new set of tasks.
5
Conclusion
Do different tracking tasks require different appearance models? In order to address this question, the proposed UniTrack framework has been instrumental, as it has allowed to easily experiment with alternative representations on wide variety of downstream problems. Although the answer is not a resounding "no", as only sometimes a single shared appearance model can outperform dedicated methods, we argue that unified framework is an appealing alternative to task-specific methods. The main reason is that it allows us to make the most of the progress made in the representation learning literature
extra cost. With the rapid development of self-supervised learning, and the large amount of compurational resources dedicated to it, we believe it is reasonable to expect that, in the future, general- purppse representation will be able outperform task-specific methods across the board. Until then, UniTrack could still serve as useful evaluation tool for novel representations, especially considering the lack of correlation with the standard linear- finear-probe approach. We believe this will encourage the community to develop self- representations that are of "general purpose" in a broader sense.
Broader impact. Upon reflection, we believe that progress in tracking applications and self- supervised learning beneficial for society, as can significantly impact (for instance) the de- velopment of autonomous vehicles, which we consider net positive for society. We also recognise that the same technologies could constitute a threat if deployed for surveillance by entities hostile to civil liberties.
10

---

6 Funding Transparency Statement
This work was supported by the National Natural Science Foundation of China under Grant No. 61771288, Cross-Medin Intelligent Technology Project of Beijing National Research Center for Information Science and Technology (BNRist) under Grant No. BNR2019TD01022 and the research fund under Grant No. 2019GQG0001 from the Institute for Guo Qiang, Tsinghua University.
This work was also supported by the EPSRC grant: Turing AI Fellowship: EP/W002981/1, EP- SRC/MURI grant EP/N019474/J. We would also like to thank the Royal Academy of Engineering and FiveAI.
References
[1] Costa st tracker. httpeer//ooccaallleeeeeeeeeeaeee
[2] Mykhaylo Andriluka, Umar Tqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Posetrack: A benchmark for human pose estimation and tracking. In CVPR, 2018.
[3] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-lahelling via simultaneous clustering and representation learning. In ICI.R, 2020.
[4] Ali Athar, Sabarinath Mahadevan, Aljoša Ošep. Laura Leal-Taixé, and Bastian Leibe. Stem-seg: Spatio- temporal embeddings for instance segmentation in videos. ECCV, 2020.
[5] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. convltconlllutionall siamese networks for object tracking. In ECCV workshops, 2016.
[6] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leameer. 145,, 2020.
[7] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
[8] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In NewrIPS, 2018.
[9] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset.
In CVPR, 2017.
[10] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. 2020.
[11] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv:2006. 10029, 2020.
[12] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved baselines with momentum contrastive learning. arXivv2003.00 042997, 2020.
[13] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. Eco: Efficient convolution operators for tracking. In CVPR, 2017.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: large-scale hicrarchical image database. In CVPR, 2009.
[15] Jacob Devlin. Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. 2018
[16] Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg. Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt. Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In ICCV, 2015.
[17] Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In CVPR, 2021.
[18] Masoud Faraki, Xiang Yu, Yi-Hsuan Tsai, Yumin Suh, and Manmohan Chandraker. Cross-domain similarity learning for face recognition in unseen domains. vrXiv:2103.07503 2021.
[19] Vang Fu, Sifei Liu, Umar Iqhal, Shalini De Mello, Humphrey Shi, and Jan Kautz, Learning to track instances without video annotations. In CVPR, 2021.
[20] Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self-supervised visual representation learning 1:Xxvy/905.00
2019.
[21] Jean-Bastien Grill, Florian Strub, Florent A Altché, Corentin Tallec, Pierre H Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap
your own latent: A new approach to self-supe learning.
(tt1:2006007733. 2020.
[22] Hengkai Guo, Tang Tang, Guozhong Luo, Riwei Chen, Yongchen Lu, and Linfu Wen. Multi-domain pose
network for multi-person pose estimation and tracking. In ECCV, workshop, 2018
[23] Shoudong Han, Piao Huang. Hongwei Wang, En Yu, Donghaisheng Liu, Xiaofeng Pan, and Jun Zhao. Mat: Motion- aware multi-object tracking. wwXiv:2009.04777 2020.
11

---

[24] Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[25] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR. 2016.
[26]
Lingxiao He, Jian Liang. Haiqing Li, and Zhenan Sun. Deep spatial feature reconstruction for partial person re-identification: Alignment-Iree approach. In CVPR, 2018.
[27]
Allan Jabri, Andrew Owens, and Alexei A Efros. Space-time correspondence a contrastive random walk.
In NeurlPS, 2020.
[28]
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.
[29]
Matej Kristan, Jiri Matas, Aleš Leonardis, Tomák Vojíf, Roman Pflugfelder, Gustavo Fernandez, Georg Nebehay, Fatih Porikli, and Luka Čehovin. A novel performance evaluation methodology for single-target
trackers. IEFE TPAMI, 2016.
[30] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[31] Zihang Lai, Erika Lu, and Weidi Xie. Mast: A memory-augmented self-supervised tracker. CVPR, 2020.
[32] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. In CVPR, 2019.
[33] Bo Li. Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High performance visual tracking with siamese region proposal network. In CVPR, 2018.
[34] Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR, 2021.
[35] Xueting Li, Sifci Liu, Shalini De Mello, Xiaolong Wang. Jan Kautz, and Ming- Hinng Yang. Joint-task self-supervised learning for temporal correspondence. In NeurlPS, 2019.
[36] Chao Liang, Zhipeng Zhang, Yi Lu, Xue Zhou, Bing Li, Xiyong Ye. and Jianxiao Zou. Rethinking the competition between detection and reid multi-object tracking. arXiv:2010.12
2020.
[37] Tsung- Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[38] Songtao Liu, Zeming Li, and Jian Sun. Self-emd: Self-supervised object detection without imagenet. arXiv:2011. 133667 2020.
[39] Jonathan Long, Ning Zhang, and Trevor Darrell. Do convnets learn correspondence? In NeurlPS, 2014, [40] Alan Lukezie, Jiri Matas, and Matej Kristan. D3s-a discriminative single shot segmentation tracker. In CVPR, 2020.
[41] Anton Milan, Laura Lcal-Taixé, Ian Reid, Stefan Roth, and Konrad Schindler. Mot16: A benchmark for multi-object tracking. orXiv:/603.000 2016. Ishan Misra and Laurens van der Maaten. Self-supervieed learning of pretext- representations. In CVPR, 2020.
[42]
[43] Guanghan Ning and Heng Huang. Lighttrack: generic framework for online top-down human pose tracking. In CVPR, workshop, 2020.
[44] Guanghan Ning, Ping Liu, Xiaochuan Fan, and Chi Zhang. A top-down approach articulated human pose estimation and tracking. In ECCV, workshop, 2018
[45] Seoung Wug Oh, Joon- Young Lee. Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In ICCV, 2019.
[46] Bo Pang. Yizhuo Li, Yifan Zhang. Muchen Li, and Cewu Lu. Tubetk: Adopting tubes to track multi-object in one-step training model. In CVPR, 2020.
[47] Jinlong Peng, Changan Wang, Fangbin Wan. Yang Wu, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu. Chained-tracker: Chaining paired attentive regression results for end-to-end joint multiple-ohject detection and tracking. In ECCV, 2020.
[48] F. Perazzi, Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine- Sorkine-Homung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016.
[49] Michael Snower, Asim Kadav, Farley Lai, and Hans Peter Graf. 15 keypoints is all you need. In CVPR. 2020.
[50] Young- mung--min&o Song and Moongu Jeon, Online multi-object tracking and segmentation with gmphd filter and simple affinity fusion. arXiv:2009.00110 00000000000000000000000000000000 2020.
[51] Yifan Sun. Qin Xu, Yali Li, Chi Zhang, Yikang Li, Shengjin Wang, and Jian Sun. Perceive where to focus: learnnng visibility-aware plrt-level features for partial person re-identification. In CVPR, 2019.
[52] Yifan Sun, Liang Zheng. Yi Yang, Qi Tian, and Shengjin Wang. Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). In ECCV, 2018
[53] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. In NeurlPS, 2020.
[54] Jack Valmadre, Luca Bertinetto, Joao Henriques. Andrea Vedaldi, and Philip HS Torr. End-to-end representation learning for correlation filter based tracking. In CVPR, 2017.
12

---

[55]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
[56]
Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, and Liang-Chiee -hhchhhhhhhhh Chen. Feelvos: Fast end-to-end embedding learning for video object segmentation. In CVPR, 2019.
[57] Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmentation. In CVPR, 2019.
[58] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In ECCV, 2018.
[59] Ning Wang, Yibing Song, Chao Ma, Wengang Zhou, Wei Liu, and Houqiang Li, Unsupervised deep tracking. In CVPR. 2019.
[60] Ning Wang, Wengang Zhou, Yibing Song, Chao Ma, Wei Liu, and Houqiang Li. Unsupervised deep representation learning for real-time tracking. IJCV, 2020.
[61] Qiang Wang, Jin Gao, Junliang Xing, Mengdan 7hang, and Weiming Hu. Defnet: Discriminant correlation filters network for visual tracking. arXiv: 170400000 2017.
[62] Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr. Fast online object tracking and segmentation: A unifying approach. In CVPR, 2019.
[63] Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015.
[64] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In CVPR, 2019.
[65]
Zhongdao Wang, Liang Zheng, Yixuan Liu, and Shengjin Wang. Towards real-time multi-object tracking. In ECCV, 2020.
[66] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In CVPR, 2016.
[67] Davis Wertheimer, Luming Tang, and Bharath Hariharan. Fine-grained few-shot classification with feature map reconstruction networks. CVPR, 2021.
[68] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In ICIP, 2017.
[69] Jialian Wu, Jiale Cao, Liangchen Song. Yu Wang, Ming Yang. and Junsong Yuan. Track to detect and segment: An online multi-object tracker. In CVPR, 2021.
[70] Yi Wu, Jongwoo Lim, and Ming- Hsuan Yang. Online object tracking: A benehmark. CVPR, 2013.
[71] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric
instance discrimination. In CVPR, 2018.
[72] Enze Xie, Jian Ding, Wenhai Wang. Xiaohang Zhan, Hang Xu, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. arXiv:2102. 10Xiv:2102.04803, 04200,880, 2021.
[73] Zhenda Xie, Yutong Lin, Zheng Zhang. Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring
pixel-level consistency for unsupervised visual representation learning. In CVPR, 2021.
[74] Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity perspective. arXiv:2103. 17266,33,, 2021.
[75] Ning Xu. Linjie Yang, Yuchen Fan. Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: Sequence-to-sequenee video object segmentation. In ECCV, 2018.
[76] Zhenbo Xu, Wei Zhang. Xiao Tan, Wei Yang, Huan Huang, Shilei Wen, Errui Ding, and Liusheng Huang. Segment as points for efficient online multi-object tracking and segmentation. In ECCV, 2020.
[77]
Linjie Yang. Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019.
[78]
Dongdong Yu, Kai Su, Jia Sun, and Changhu Wang. Multi-person pose estimation for pose tracking with enhanced cascaded pyramid network. ECCV, workshop. 2018
[79] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv:2103.0 0333330 2021.
[80] Chi Zhang. Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In CVPR, 2020.
[81] Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In CVPR, 2017.
[82] Yifu Zhang, Chunyu Wang. Xinggang Wang, Wenjun Zeng, and Wenyu Liu. Fairmot: On the faimess of detection and re-identification multiple object tracking. arXiv:2004.01880 2020.
[83] Zhipeng Zhang, Bing Li, Weiming Hu, and Houweng Peng. Towards accurate pixel-wise object tracking
by attention retrieval. arXiv:2008.0277 02745, 2020.
[84] Kuan Zhu, Haiyun Guo, Zhiwei Liu, Ming Tang, and Jinqiao Wang. Identity-guided human semantic parsing for person re-identification. In ECCV, 2020.
13


Reviews:
[{"review_id": "j1bH97sDwPN", "reviewer": "Reviewer_igQk", "summary": "This paper proposes a unified visual tracking method using a single and task-agnostic appearance model, in which this model can be learned via n a supervised or self-supervised strategy. For this purpose, they examine appearance model performance on five tracking tasks and describe it using radar charts in fig1. Experimental results show that  most tracking tasks can be solved within the proposed framework.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper considered a unified framework for the appearance model. However it is unclear why the method can contribute improve the visual tracking performance. While visual tracking environments considerably vary over time, visual tracking method needs to be specialized to a current visual tracking environment, rather than using a unified appearance model. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "All compared methods were not pre-traineded using ImageNet, thus comparisons can be unfair. \nThe experimental evaluations are insufficient to demonstrate the effectiveness of the proposed unified appearance model under various visual tracking environments. The method needs to be evaluated using single target tracking datasets including OTB, VOT, LaSOT, OxUvA, TLP, TrackingNet, UAV123, TC, and GOT-10k. The compared methods are not recent state-of-the-art methods. More recent methods need be compared. It needs to be proven theoretically that the proposed appearance modeling strategies can cover most changes in target appearance and adapt to change the appearance under a current visual tracking environment.  It would be better to discuss and compare the proposed method against the fusion method of multiple appearance models. The paper also needs to explain why box, mask, and pose representations are considered. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "ifahbkK8hZT", "reviewer": "Reviewer_KcZZ", "summary": "This paper categorizes tracking object tasks into five lines and divides tracking algorithms into two classes: propagation and association. It indicates no necessity to fragment the problem of tracking in so many different specifications and separate trained models for each one. A unified tracking solution is proposed to address five different tasks using the same appearance model, which is task-agnostic and can be learned in a supervised or self-supervised way. The proposed model contains multiple heads to address individual tasks without training, and a reconstruction-based similarity metric is designed for association that preserves fine-grained visual features and supports multiple observation formats (box, mask and pose). I feel this work is interesting and has strong contributions to the tracking community, but experimental results are somewhat weak.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n-- The method is novel. It should be the first simple and universal framework that simultaneously supports five tracking tasks (SOT, VOS, MOT, MOTS, PoseTrack), and can also be easily extended to new ones. \n\n-- Compared with traditional cosine similarity methods, which do not support varying spatial sizes, this work proposes a reconstruction-based similarity metric that is able to effectively deal with different observation formats while preserving fine-grained information.\n\n-- The framework has achieved competitive performance against dedicated methods on the five tasks of SOT, VOS, MOT, MOTS and PoseTrack. Moreover, unlike existing self-supervised learning methods that require additional training, the method proposed in this paper does not require additional training for different tasks.\n\nWeaknesses\n\n-- This framework does not train an appearance model on a single target task, nor does it apply multi-task learning on multiple tasks. The complementary benefits of different tracking tasks are not explored and exploited, since these tracking tasks have some common properties that can be utilized for the enhancement to each other. It would be interesting if authors perform shared representation learning across different tasks and fine-tune task heads to consider their specialty, although domain difference and optimization object conflict might exist.\n\n--UniTrack performs comparable with SOTA methods on MOT, MOTS and poseTrack, but greatly worse on SOT and VOS. Moreover, some settings make comparison unconvincing. For example, in the task of SOT, comparison is only made on a small-scale and somewhat out-of-date dataset and recent SOTA methods are missing. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Five hours"}, "extra_text": {}}, {"review_id": "TPyHEUkF4sd", "reviewer": "Reviewer_CNPB", "summary": "This paper proposes using a unified self-supervised framework for various video-related tracking tasks including single-object bounding box (SOT) or mask propagation (VOS) or multiple object bounding box (MOT), mask (MOTS) or keypoints (PoseTrack) association of detections across video frames. The idea proposed in this paper is inspired by the finding that mid-level representations of networks trained either with full-supervision (e.g., for object classification) or via self-supervised learning for a pre-text task are believed to contain information for establishing dense correspondence between objects and instances. The authors propose to use a single backbone pre-trained network to extract fc3/fc4 features for these various tasks and further employ methods based on dense affinity/ discriminative correlation filters/Hungarian matrix, etc to perform the various propagation or association tasks. The main finding that the authors share is that a common task-agnostic mid-level representation learned using ImageNet is competitive with many supervised and unsupervised task-specific algorithms designed for individual tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper is original in several aspects: -- (a) in investigating a large number (5) of different but closely related tracking tasks together in a unified manner, (b) in highlighting the competitiveness of a single representation learned using ImageNet for many video-related tasks and proposing ways to use them effectively for the different tasks, (c) in analyzing for the first time as to how well self-supervised representations work at these tasks and what their shortcomings are. In summary, this paper proposes a new task, a new research paradigm and a new algorithm.\n\nQuality: Overall the method is technically sound, well-motivated and the experiments are thorough. However, I note some concerns below.\n\n1. With regards to unsupervised learning approaches for association-type tasks there exists the following recently published work, where the authors train a unified network for MOTS and PoseTrack: \"Fu et al., Learning to Track Instances without Video Annotations, CVPR 2021\". It is not cited. It would be good to cite this work in the reference section and compare against it in Table 1(e) of the main paper and Table 5 of the supplementary material.\n\n2. How critical is the modification of the stride of the original ImageNet18 to the performance of UniTrack? Did the authors also apply this modification to all the networks of the competing methods that they compared against? If not, for a fairer comparison, could the authors also provide the equivalent performance numbers for UniTrack (in Table 1), but without this stride modification?\n\nClarity: The paper is quite clear and easy to understand. The supplementary document is extensive and provides detailed implementation steps. A few typos in the main document are noted below:\n\n- ln 26: \"form\" should be \"forming\"\n- ln 27: \"differ\" should be \"differing\"\n- ln 207: \"calculate\" should be \"calculating\"\n- ln 263: \"/\" should be \".\"\n\nSignificance: This work is likely to have a broad impact across the computer vision research community and is likely to be of interest to a wide audience. The fact that it proposes to unify networks for many different, but related video tracking tasks is also quite nice and influential from the point of view of practitioners of computer vision technology as well. It encourages simplification of network design and hence reduction in inference compute cycles, which is an important consideration in real-world applications.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitations of their work in the broader societal context.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "0Lby00Jy5WM", "reviewer": "Reviewer_7ktv", "summary": "The paper investigates the possibility of using a single appearance model for all tracking-related tasks/subtasks (the paper proposes a precise classification of these tasks). The authors propose an elegant unified framework with 3 levels: basic appearance model (level 1), propagation and association blocks (level 2), and task-specific heads (level 3). The only level that needs a training phase is level 1, while the other levels are training-free. Another contribution is the introduction of the RSM metric, which is used to measure the similarity between features within the proposed association algorithm. Following a rich experimental section, the authors conclude by giving a negative answer to the question contained in the title of the paper itself.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-**Originality**: The authors propose a method with some innovative elements and an interesting combination of well-known techniques to tackle the problem of object tracking in videos. The main innovative contribution, in my opinion, is the RSM metric, which is used to measure the similarity between features within the proposed association algorithm. The presented framework, as a whole, is interesting, although it takes many techniques and ideas from pre-existing works (which are correctly cited by the authors). Overall, I believe that the originality level of the paper is not high, but still acceptable.  \n-**Quality**: Technically, the paper is of good quality. The claims of the paper are well supported by the theoretical treatment, but they lose some points due to the experimental results that, in my opinion, are not always amazing. However, the authors highlight the weaknesses of their approach and explain the reasons behind the most underwhelming results. The paper therefore shows how the use of a different appearance model for each tracking subtask is not strictly necessary to have competitive results; but in my opinion it also highlights how a specific approach may in some cases be necessary to obtain SoTA results.  \n-**Clarity**: The paper is written in a clear and well-organized way. From reading the paper, you have all the information you need to re-implement the method. Furthermore, the supplementary material further enriches the discussion by making the paper even clearer. My only note in this sense is related to the figures: some of them, in particular Figure 3, would in my opinion have benefited from more space and a greater amount of detail; relegating the overall scheme of the framework to a small figure with few details can be a bit limiting (the details are however contained in the paper body).  \n-**Significance**: Some elements that the paper proposes could be reused by the community, such as the RSM metric and the categorization of the various tracking-related subtasks. The framework as a whole can also have some relevance when used as a baseline for comparison with other methods, particularly for methods that use a specific approach to a given subtask. However, it does not seem to me that the conclusions of the paper are particularly significant, because the experimental results (that are not always excellent) make the authors' main claim a little bit weak.  \n-**Overall Opinion**: The paper is well done and contains some interesting elements. Not particularly groundbreaking, but acceptable. Given the experimental results, I would propose the framework just as a baseline for task-specific methods, because I don't think the answer to the question that gives the paper its title is a definitive “no”.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable, as the paper does not have a significant social impact.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initially the paper received mixed positive review: three accept (7) and 1 reject (3).  The reviewers appreciated the originality (unified framework for 5 tracking tasks, new similarity measurements and association algorithm, evaluation of supervised and self-supervised representations on 5 tracking tasks).  The reviewers' main concerns were: 
1. additional results with new unsupervised methods.
2. some writing/clarity issues
3. didn't use multi-task learning to train the framework.
4. comparison on SOT uses out-of-date dataset and older methods. need further evaluation on more recent datasets.
5. for tracking model, appearance model needs to be specialized for the current environment, rather than a unified appearance model.
6. missing theoretical analysis that appearance models can cover most changes in target appearance.
7. possible unfair comparisons since some methods not pre-trained on imagenet.

In the response, authors provided more experiment results (point 4: SOT results on 5 larger datasets; point 1: another SSL method) and further clarifications and explanations (points 2, 3, 5, 6, 7). The 3 positive reviewers were satisfied with the response, while the negative reviewer was only partially satisfied and raised their rating to "4".  The remaining concern of the negative reviewer was mainly about the theoretical analysis (point 6).  The AC tends to discount this concern, since theoretical analysis seems outside of the scope of this paper, given the large amount of empirical results. The AC agrees with the 3 positive reviewers that the paper provides an interesting new way to unify tracking tasks, and to evaluate representation learning methods on tracking tasks. Thus the AC recommends accept. The authors should update the paper according to the review comments and discussion.
Decision: Accept (Poster)