0ReMOTS: Refining Multi-Object Tracking andSegmentation(1st Place Solution for MOTS 2020 Challenge 1)
Fan Yang1,2, Xin Chang1, Chenyu Dang1, Ziqiang Zheng3, Sakriani Sakti1,2,
Satoshi Nakamura1,2, Yang Wu4
1Nara Institute of Science and Technology, Japan
2RIKEN Center for Advanced Intelligence Project, Japan
3UISEE Technology (Beijing) Co. Ltd., China
4Kyoto University, Japan

Background of Multi-Object Tracking and Segmentation (MOTS)• Problem: detect, segment, and track multiple objects in videos.
• Input: a video sequence contain that multiple RGB images.
• Output: 2D mask and corresponding track ID at each frame.
• Application: action recognition, automatic driving, and others.
Input Video Dataframe kMOTS

Instance Segmentation
frame kframe ktrack_id 1
track_id 2

frame k+1

frame k+1

frame k+1
track_id 1
track_id 2

detect+segment

detect+segment+track1

Our solution for MOTS

Use off-the-shelf models

Step 1
Input Video Data

Instance Segmentation

frame kframe kframe k+1

frame k+1

detect+segment

2Instance Segmentation
We take off-the-shelf models:
X-101-64x4d-FPN of MMDetection + Mask R-CNN X152 of Detectron 2,
which refers to the public detection and segmentation methods.
But, how to fuse instance masks from different models?
Fusing boxes – using NMS
Fusing masks – may also using NMS – but change IoU to IoM (Intersection over Minimum).
22
Instance masks:

1

Pixel_IoU = 1/3 = 0.33

10.1
0.1

Pixel_IoM = 1/2 = 0.5

Acceptable for bounding box,
But not for mask.
Pixel_IoU = 0.01/2 = 0.005

Pixel_IoM = 0.01/0.01 = 13

Our solution for MOTS
We proposed an offline method, as ReMOTS (Refining Multi-Object Tracking andSegmentation ).

Step 1
Input Video Dataframe kOur main contributions:
1. Refine appearance features
2. Automatically decide threshold
Step 2
MOTSInstance Segmentation
frame kframe ktrack_id 1
track_id 2

frame k+1

frame k+1

frame k+1
track_id 1
track_id 2

detect+segment

detect+segment+track4

Intra-frame Training and Short-term Tracking

Hypotheses:

t4
Short-term
Tracker

t5

t5
t6

t6
95 id
93 id
96
92 id
Raw idAppearance
Encoder

t3

t3

t8

t8

Short-term
Tracklets

Object-instanceSegmentationframe k().
()/()0()1

if same split tracklet ID:
set inf
elif temporal overlapping:
set infelse:
set cosine distance

t7

t7

()*
()+(),()-

Estimated Bounding Boxes
in Test Set

t1
t2

Distance

Appearance
Encodert4
ID2 ID3Inter-short-term-tracklet Trainingt2

t2
Ground Truth:

Hierarchical Clustering

t1

CosineSimilarity

Intra-frame Training

t1

t3

Cutting threshold=2 − !&''
456%5

inf

inf

0.1

0.4

inf

inf

0.5

0.2

0.1

0.5

inf

inf

Intra-frameCosine Affinity

0.4

0.2

infinfIntra-short-tracklet
Cosine Affinity

t4
t5

clusters

t6! &''
"#$%

t7
t8

Distance Matrix Wlong

Form a Mini Batch
Input by the Ratio 1:1

Long-term
Tracklets
Ground-truth Tracklets
in Training Set

Ground Truth:
ID7

ID8

Hypotheses:#2
idframe k+1

For masks of frame k,consider all of IoU > 0 masksof frame k+1 for matching

#3
idtn

Intra-frame
Observations

t1

P

Augmentation

N

Intra-frame
sampling

P

Temporaloverlapped &t3 non-overlapped! '(("#$%&tracklets
t4t2

N Inter-tracklet
samplingt5Intra-frame Training
Appearance Encoder

BoT-Reid (TMM 2020)

CosineDistance

EmbeddingAppearance
Features0.2

0.4

infinfinf0.3

LinearAssignmentAppearance Distance Matrix

5Inter-short-term-tracklet Training

t5

t3

ID2 IDAppearance
3

Hypotheses:
Encodert6

92
Raw id

95 id
93 id
96
id

t4
Short-term
Tracker

t4

t7

t5
t6
t7

t8
Short-term
t5
Tracker
Object-instanceSegmentation(Inter-short-term-tracklet
Training(t8

Short-term
Tracklets

t6().
()/()0()1

if same split tracklet ID:
set inf().
()/elif temporal overlapping:()0set inf()1else:set cosine distancet8
Short-term
Tracklets
Estimated Short-termtracklets in Test Set

Ground Truth:
ID8

Hypotheses:
#2id

#6
#5 id
#3 id
id

t2

inf

0.1

0.4

inf

inf

0.5

0.2

0.1

0.5

inf

inf

0.4

0.2

inf

inf

inf

inf

Cutting threshold=2 − !&''
456%5clusters
Intra-framet3&''! "#$%
Affinity
0.1Cosine 0.4

t3
t4

t4
t5
t6

Cutting thresh=2 − !&''
456%5t7

Intra-short-tracklet
Cosine Affinityt8

Distance Matrix Wlong

inf

inf

0.5

0.2

0.1

0.5

inf

inf

Intra-frame
Cosine Affinity

0.4

0.2

infinfIntra-short-trackle
Cosine Affinity

Long-term
Tracklets
clusters

Distance Matrix Wlong

Form a Mini Batch
Input by the Ratio 1:1

Ground-truth Tracklets
in Training Set

t1

t1

Pt2
ID7

inf

if same split tracklet ID:
set inf
elif temporal overlapping:
set infelse:
set cosine distancet7e()*()+(),()-)*)+(),()-Distance

t4

t2

t3

t1
HierarchicalCluDistancet3

Appearance
Encoder

CosineCosine Similarity
Similarity

Ground Truth:

t1

t2

Inter-short-term-tracklet Training

Appearance
Encoder

Intra-framet2Training

Hierarchical Clustering

t1

Appearance
Encoder

Intra-frame Training

t1

Inter-short-termtracklet samplingNt5
Temporal overlapped
short-term tracklets

P

t2
t3

N Inter-tracklet
sampling

t4
t5

Temporal overlapped &non-overlapped tracklets

6What Happened in Each Step of Appearance Training

J (H1, H2) represents Jaccard Index of two normalized histograms H1 and H2.
(2) After Intra-frame training ontest set without labels

t4 t4
IDShort-term
2 ID3Tracker
Hypotheses:

t5 t5
t6 t6().
(Short-term ()/)0
Tracker ()1t2
t3
t4
t5t6

if same split tracklet ID:()*()+(),()-

infinf (). 0.1

0.4

inf

inf

0.2

()/
()0
()1

0.5

CosineDistance
Similarity

t3 t3

()*
Appearance (()+),
Encoder ()-Hier
Inter-short-term-tracklet Training

Appearance
Encoder

Appearance

GroundTruth:
Encodert1
Inter-short-term-tracklet
Training
Intra-frame
Training

Appearance
Encoder

Intra-frame Trainingt1 t1
t2 t2(3) After Inter-short-tracklet trainingon test set with pseudo labels

Intra-short-tracklet instance mask

Intra-frame instance masks

CosineSimilarity(1) After Trained on the train
set only

7

same inf
split tracklet
ID:
0.1 if 0.5
infMerging Short-term Tracklets

Hypotheses:

95 id
93 id
96
92 id
Raw idt7().
t8 ()/()0Object-instance
()1t3
t4

Short-term
Tracker

Segmentation

if same split tracklet ID:
set inf
elif temporal overlapping:
set infelse:
set cosine distance

t5
t6
t7
t8

inf

inf

Short-term
infTracklets

inf0.10.4()*()+(),()().
()/()0()1

if same split tracklet ID:
set inf
elif temporal overlapping:
set infelse:
set cosine distance

0.1

0.4Distancet4 ()*()+t5 (),
(t6 )-

t2

Distance

Appearance
Encoder

ID2 ID3

t3

CosineSimilarity

Ground Truth:

Appearance
Encoder

Hierarchical Clustering

Inter-short-term-tracklet Training

CosineSimilarity

t2
Inter-short-term-tracklet
Training

Hierarchical Clustering

t1

Appearance
Encoder

Intra-frame Training

t1

inf

inf

0.1

0.4

infinf0.5

0.2

Cutting threshold=2 − inf!&''
456%5
0.5
inf

0.1
0.4

0.2

inft1
t2

0.2

0.5

inf

inf

Intra-frame
Cosine Affinity

0.2

inf

inf

Intra-short-tracklet
Cosine Affinity

t5t4

t7

t6

! &''
"#$%

t4

t6

t5

clusters

t3

t3
clusters
Intra-frame
Cosine Affinity

Distance Matrix Wlong

0.5

t2

Cutting threshold=2 − !&''
456%5

Intra-short-tracklet
Cosine Affinity

inf

t1

! &''
"#$%

t8
Long-term
Tracklets

t7
t8

Distance Matrix Wlong

Long-term
Tracklets
t1
t2

Temporaloverlapped
Short-term
Tracklets

t3
t4Intra-frame
Cosine Affinity!&''"#$%

Intra-shorttracklet
CosineAffinity8Comparison with others on MOTSChallenge 1

May benefit from refinement

May benefit from mask fusion

Since our strategy can be easily adapted to others, will other methods get better
performance by applying our appearance encoder and merging?

9

Limitations of ReMOTS
1.

An offline approach.
- It worth to explore how to bring it to online approach.

2.

It is challenging for ReMOTs to handle objects with similar appearance.
e.g., good for persons (wear different clothes) but not very useful forvehicles (similar textures)

3.

Trajectory is not considered in our short-term tracker. Failed to associate fast
moving objects.

Slowly moving person with diverse clothes?

Fast moving car with similar appearance

10Conclusion• Unlabeled target videos can be used for learning better appearance
features, but should take care of the potential of introducing noises.
• The suitable hyper parameters for data association may varies from caseto case, and the statistical information of tracklets can be used to adjustthem.
• It is preferred to accommodate some insights of ReMOTS to online
MOTS.

11

Thanks for your listening