ReMOTS: Self-Supervised Refining Multi-Object Tracking and Segmentation(1st place solution for MOTSChalelnge 2020 Track 1)
Fan Yang∗1,2 , Xin Chang1 , Chenyu Dang1 , Ziqiang Zheng3 , Sakriani Sakti1,2 , Satoshi Nakamura1,2 , andYang Wu4

arXiv:2007.03200v2 [cs.CV] 8 Jul 2020

12Nara Institute of Science and Technology, JapanRIKEN, Center for Advanced Intelligence Project, Japan
3
UISEE Technology (Beijing) Co. Ltd., China
4
Kyoto University, Japan(May-30th-2020) and won the 1st place on Challenge 1. In
this paper, we represent our solution.

Abstract
We aim to improve the performance of Multiple Object
Tracking and Segmentation (MOTS) by refinement. However, it remains challenging for refining MOTS results,
which could be attributed to that appearance features are
not adapted to target videos and it is also difficult to findproper thresholds to discriminate them. To tackle this issue, we propose a self-supervised refining MOTS (i.e., ReMOTS) framework. ReMOTS mainly takes four steps to refine MOTS results from the data association perspective.
(1) Training the appearance encoder using predicted masks.
(2) Associating observations across adjacent frames to form
short-term tracklets. (3) Training the appearance encoder
using short-term tracklets as reliable pseudo labels. (4)Merging short-term tracklets to long-term tracklets utilizing adopted appearance features and thresholds that areautomatically obtained from statistical information. Using
ReMOTS, we reached the 1st place on CVPR 2020 MOTS
Challenge 1 [4], with a sMOTSA score of 69.9.

2. Method Details
Overall, we apply the tracking-by-detection strategy togenerate MOTS results. Since our ReMOTS is an offline approach, we refine the data association by retraining the appearance feature encoder. In each step of ReMOTS, we givea practical guidance to quantitatively select hyperparameters. Our approach is illustrated in Figure 1. After obtaining object-instance masks, we perform: (1) encoder training with intra-frame data, (2) associate masks to short-term
tracklets by a short-term tracker, (3) inter-short-tracklet encoder retraining, and (4) merging short-term tracklets.

2.1. Generate Object-Instance Masks
Referring to how the public detection is generated, weobtain object-instance masks using the Mask R-CNN X152of Detectron 2 [5] and X-101-64x4d-FPN of MMDetection [2]. We fuse their segmentation results by a modified Non-maximum Suppression (NMS). Unlike the traditional NMS, where the IoU (Intersection over Union) is applied, we propose a new metric named IoM (Intersectionover Minimum) for it since heavily overlapped masks may
also have low IoU values. The python code of IoM is asfollows.

1. Introduction
Multiple Object Tracking (MOT), which depends on information from the bounding box, faces a great challenge,
since different objects may stay in the same bounding box
and increase the ambiguity to distinguish them. Recently,
some researchers in this filed have moved their eyes to Multiple Object Tracking and Segmentation (MOTS) and hope
to take advantage of object-instance masks. Under such abackground, the first MOTS challenge is organized to explore solutions for MOTS. We participated in this challenge∗ Corresponding email: yang.fan.xv6@is.naist.jp

def pixel_iom(target,prediction):"""3
Inputs:4
target: binary mask, array([H,W])
5prediction: binary mask, array([H,W])
6
Outputs:7
iom_score: float
8"""12

Hypotheses:
92
Raw id

95 id
93 id
96id

t4
Short-term
Tracker

t5

t5

t6

t6

t7

t7

Short-term
Tracklets

Object-instanceSegmentation𝑓).
𝑓)/𝑓)0𝑓)1

if same split tracklet ID:
set inf
elif temporal overlapping:
set infelse:
set cosine distance

t8t8𝑓)*𝑓)+𝑓),𝑓)-t1
t2

Distancet3

Appearance
Encoder

Appearance
Encodert3
t4

ID2 ID3Inter-short-term-tracklet Trainingt2

t2
Ground Truth:

Hierarchical Clustering

t1Cosine
Similarity

Intra-frame Training

t1

Cutting threshold=𝟏 − 𝜽𝒂𝒑𝒑
𝒎𝒆𝒓𝒈𝒆

inf

inf

0.1

0.4

inf

inf

0.5

0.2

0.1

0.5

inf

inf

Intra-frame
Cosine Affinity

0.4

0.2

inf

inf

Intra-short-tracklet
Cosine Affinity

t3
t4
t5

clusters
𝜽 𝒂𝒑𝒑
𝒍𝒐𝒏𝒈

t6
t7
t8

Distance Matrix Wlong

Long-term
Tracklets

Figure 1: The illustration of ReMOTS Framework.
Estimated Bounding Boxes
in Test Set

Form a Mini Batch
Input by the Ratio 1:1

Ground-truth Tracklets
in Training Set

Ground Truth:
ID7ID8

Hypotheses:#2
id#3idtn

Intra-frame
Observations

Intra-frame
sampling

t1

P

Augmentation

NP

Temporaloverlapped &t3 non-overlappedt2

N Inter-tracklet

t4

sampling

t5

tracklets

Figure 2: Constructing training samples for intra-frame training. P and N represent positive and negative samples, respectively.

9101112

intersection = np.logical_and(target,prediction)
min_area = min(np.sum(target),np.sum(prediction))
iom_score = np.sum(intersection) / min_area
return iom_score

After performing our modified NMS, the remaining
masks may still have overlapped areas. Therefore, we only
keep the mask with the top confident score for each overlapping area.

2.2. Encoder Training with Intra-frame Data
We take an off-the-shelf appearance encoder and its
training scheme from an object re-identification work [3].
SeResNeXt50 is used as the backbone and its globalaverage-pooling output, which is a 2048-dimension vector,
is used as the appearance features. The triplet loss [1] isapplied to train the appearance encoder. To adapt the appearance feature learning to the target videos, we incorporate intra-frame observations of target videos into a novel
offline training process.
As Figure 2 shows, we can sample triplets from the training set only referring to the ground-truth tracklets. In test
set, since Non-maximum Suppression (NMS) is performed,
we assume that predicted object masks are exclusive within
the same frame, and therefore it is easy to form negative
pairs with intra-frame observations. Before tracking, we
create a positive sample by augmenting an anchor sample. The augmentation process can dramatically change thepixel content of the anchor sample without altering identity.
Finally, we take triplets from the training set and target set
to form a mini-batch input by the ratio of 1 : 1. Using suchnew training samples, we retrain the appearance encoder toobtain more discriminative appearance features.

2.3. Short-term Tracker
After intra-frame training, we apply the appearance encoder to generate appearance features for data association.
Since the tracker part is not our main focus, we build a simple tracker that only associates two-frame observations atonce. Using the dense optical flow function of OpenCV,
we generate optical flow between two adjacent frames, andthen warp the mask from previous frame to current frame tocalculate IoU of cross-frame masks. The distance matrix isformulated as follows:
short
Wprev,curr=(inf, if IoU (maskprev , maskcurr ) = 0
fprev fcurr
otherwise
1 − kfprev
kkfcurr k ,(1)where maskprev and maskcurr respectively denote themask of the previous frame and the mask of the current
shortframe; Wprev,curr
is their edge weight (i.e., distance); fprev
and fcurr are their appearance features.
Besides constraining data association with low IoU values, we also hope to constrain data association with low appearance similarity. However, it is tricky to heuristically determine a threshold for constraining. We tackle this issue byanalyzing the intra-frame distribution. Specifically, the histogram of appearance cosine similarity between intra-framemasks can be approximated by a normal distribution, andwithin three standard deviations is 99.7% of the observation
pairs (see Figure 4). We set an appearance affinity thresholdEstimated Short-termtracklets in Test Set

Ground Truth:
ID8

Hypotheses:
#6
#2 id
#5 id
#3 idid

Ground-truth Tracklets
in Training Set

t1

t1

PP

t2
ID7

Form a Mini Batch
Input by the Ratio 1:1

t3

t3

Inter-short-termtracklet samplingNt4

t2

N Inter-tracklet
sampling

t5

t4
t5

Temporal overlapped &non-overlapped tracklets

Temporal overlapped
short-term tracklets

Figure 3: Constructing training samples for inter-short-tracklet training. P and N represent positive and negative samples, respectively.

appat three standard deviations, as value θshort.

𝜽 𝒂𝒑𝒑
𝒔𝒉𝒐𝒓𝒕

Figure 4: The appearance threshold value for short-term tracking.
appConsequently, using automatically obtained θshort, we
shortfurther process Wprev,curr by
short
Wprev,curr
=



inf,
short,Wprev,curr

app
short
if Wprev,curr> 1 − θshort.
otherwise(2)short
to determine
We apply linear assignment on Wprev,curr
the association of masks between the previous frame andthe current frame. Due to misdetection and occlusion, sucha process can only generate short-term tracklets. However,short-term tracklets reduce the risk of mixing different identities, which is an important condition in next-step process.

2.4. Inter-short-tracklet Encoder Retraining
As we assume each short-term tracklet may only containa unique identity, they can be used as dependable pseudolabels to train the feature encoder. However, different shortterm tracklets, which have no overlap in the temporal domain, may still hold the same identity. Therefore, we dointer-short-tracklet retraining under the constraint that sampled short-term tracklets must be temporally overlapping
within the same video.
We illustrate the process of training data sampling forinter-short-tracklet training as Figure 3 shows. Within avideo, we first sample two identities that appear in a randomly chosen frame, and then randomly choose anotherframe for one of the selected identities, thus constructing atriplet. Other settings of inter-short-tracklet training are thesame as intra-frame retraining. We update the appearance
features after inter-short-tracklet retraining and use them in
the next step.

2.5. Merging Short-term Tracklets
With better appearance features and more robust spatiotemporal information of short-term tracklets, we are able tomerge short-term tracklets into long-term ones. The merging process is summarized in Figure 1. Short-term tracklets
association is formulated as a hierarchical clustering problem on a weighted graph, in which each node represents a
tracklet and the graph edges are represented in a distancematrix W long , defined aslong
Wk1,k2=




inf,
if k1 = k2
if Distance(Πk1 , Πk2 ) > θt,
inf,
if Πk1 ∩ Πk2 6= ∅PP
fik1 fjk2
 1i∈Πk1j∈Πk2 1 − kf k1 kkf k2 k , otherwise
Nk1 Nk2
ijinf,(3)longwhere for tracklets Tk1 and Tk2 , Wk1,k2
is their edge weight(i.e., distance); Πk1 and Πk2 are their temporal ranges; fik1
and fjk2 are their appearance features at frame i and j, andNk1 and Nk2 are the number of observations within thetracklets, respectively.
Whenever the matching condition between two shortterm tracklets violates any of the following three principles:(1) different short-term track ID, (2) the temporal gap between two short-term tracklets are within θt frames (we use
θt = 15 ), and (3) no temporal overlap between two shortterm tracklets, we set their distance value to be infinite. To
hold these constraints in the whole process of hierarchicalclustering, we apply the centroid linkage criteria to determine the distance between clusters.
The main challenge of applying hierarchical clustering is
on how to set a proper cutting threshold. We do not give aheuristic value, and we let the data speak for themselves instead. We suppose that intra-frame and inter-short-tracklet
appcosine similarity histograms can be separated at θmerge(seeFigure 5) after inter-short-tracket retraining, though small
overlapping might exist. Without accessing to the groundtruth, this could be a reasonable boundary to distinguishobjects based on appearance features. Therefore, we setapp
1 − θmerge
as the cutting threshold in hierarchical clustering.

Rank

MethodsMOTSA↑IDF1↑

MOTSA↑

MOTSP↑MODSA↑MT↑ML↓

1st place
2nd place
3rd place

ReMOTS (ours)
PTPMPT

69.968.868.4

75.0
68.5
64.9

83.9
82.6
82.6

84.0
84.1
83.9

85.1
83.7
84.4

248
244248121910

Table 1: The performance on CVPR 2020 MOTS Challenge test set (up to submission deadline at May-30th-2020).
Sequence

MethodsMOTSA↑IDF1↑

MOTSA↑

MOTSP↑MODSA↑

MT↑

ML↓

MOTS20-01
MOTS20-06
MOTS20-07
MOTS20-12

ReMOTS
ReMOTS
ReMOTS
ReMOTS

68.5
74.9
65.4
71.8

81.9
76.8
68.3
82.0

83.9
88.8
80.6
82.9

82.5
85.0
81.9
87.2

84.8
90.6
81.6
83.8

8156
36480444

Table 2: The performance of ReMOTS on each sequence of CVPR 2020 MOTS Challenge test set (up to submission deadline at May30th-2020).

fine-grained adaption, has not been considered yet.

t1
t2

Temporaloverlapped
Short-term
Tracklets

t3
t4

Intra-frame
Cosine Affinity

𝜽𝒂𝒑𝒑𝒍𝒐𝒏𝒈

Intra-shorttracklet
CosineAffinity

Figure 5: The appearance threshold value for merging short-term tracklets.

4. Conclusion
We present our solution which wins the CVPR 2020
MOTS Challenge 1. In our proposed ReMOTS framework,
intra-frame training and inter-short-tracklet training are introduced for learning better appearance features for more
effective data association, which are our main contributions.
Besides, we quantitatively demonstrate how to select proper
thresholds by analyzing the statistical information of tracklets, which could be useful for other multiple object tracking works. The main limitation of ReMOTS is that it cannot
be used in real-time scenarios, but it may bring insights todesign better online MOTS method with feature adaptation.

2.6. Experimental Setup
The only neural network model - appearance encoder [3], used in this work, is not our contribution andour ReMOTS can do the same refining when other appearance models are used. Therefore, we do little change tothe default setting of [3], except for forming novel training
samples in our intra-frame training and inter-short-tracklet
training. Here, we omit the other details described in [3].

3. Results
We report the performance of our ReMOTS on theMOTChallenge evaluation system, with metrics introduced
in [4]. In Table 1, we list the performance of top-3 methods
up to the submission deadline. Our method mainly outperform the other two methods in terms of IDF1 score, andtherefore leads to state-of-the-art performance in this challenge. The detailed performance on each test sequence islisted in Table 2. Though the same method is applied, it canbe observed that the performance of each sequence varies alot. This may be attributed to the diversity between videos,
which calls for more exploration in automatically adapting
MOTS models to target videos. Our ReMOTS analyzes thestatistical information at the entire video level, but the temporal local statistical information, which might be useful for

ACKNOWLEDGEMENTS
This work was supported by JSPS KAKENHI Grant
Numbers JP17H06101.

References[1] Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio.
Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11(Mar):1109–1135, 2010.
[2] et al. Chen, Kai. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.
[3] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei
Jiang. Bag of tricks and a strong baseline for deep person
re-identification. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition Workshops, pages
0–0, 2019.
[4] Paul Voigtlaender, Michael Krause, Aljos̆a Os̆ep, Jonathon
Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, andBastian Leibe. MOTS: Multi-object tracking and segmentation. In CVPR, 2019.
[5] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-YenLo, and Ross Girshick. Detectron2.