MOTS: Multi-Object Tracking and Segmentation
Andreas Geiger
Autonomous Vision Group
University of Tübingen / MPI for Intelligent Systems

University of Tübingen
MPI for Intelligent Systems
Autonomous Vision GroupMOTS: Multi-Object Tracking and Segmentation[Voigtlaender, Krause, Osep, Luiten, Sekar, Geiger & Leibe, CVPR 2019]Motivation
I Datasets for multi-object tracking
I MOTChallenges
I MOT15 [Leal-Taixe et al., 2015]
I MOT16, MOT17 [Milan et al., 2016]
I CVPR19 [Dendorfer et al., 2019]

I KITTI Tracking [Geiger et al., 2012]
I VisDrone2018 [Zhu et al., 2018]
I DukeMTMC [Ristani et al., 2016]
I UA-DETRAC [Wen et al., 2015]
I ...

I Led to great progress in the community
I But annotations are only on the bounding box level
3

Are bounding boxes enough?

Object Tracking vs. Segmentation

I In difﬁcult cases, bounding boxes are a very coarse approximation
I Most pixels of the bounding box belong to other objects

5Two Communities

Object Tracking

Semantic Segmentation / Instance Segmentation
6

Can we unite the two?

MOTS: Multi-Object Tracking and Segmentation
I Dense pixel-wise annotations are tedious, hard work .. but we did it!

KITTI MOTS

8

MOTS: Multi-Object Tracking and Segmentation
I Dense pixel-wise annotations are tedious, hard work .. but we did it!

MOTSChallenge
8MOTS: Multi-Object Tracking and Segmentation
I How? 4 student assistants & semi-automatic annotation procedure
KITTI MOTS
train

val

MOTSChallenge
train# Sequences

12

9

4

# Frames

5,027

2,981

2,862

# Tracks Pedestrian

99

68

228

# Masks Pedestrian (total)

8,073

3,347

26,894# Masks Pedestrian (annot.)

1,312

647

3,930

# Tracks Car431

151-# Masks Car (total)

18,831

8,068-# Masks Car (annot.)

1,509

593

9

Data Annotation

Data Annotation
I Starting point: existing box level tracking annotations
I Fully convolutional network converts bounding boxes to segmentation masks

11Data Annotation
I Starting point: existing box level tracking annotations
I Fully convolutional network converts bounding boxes to segmentation masks
I First, 2 instances per track are manually annotated
I However, the trained segmentation model will not be perfect
I Repeat until annotations are good:
1. Annotators ﬁx worst errors with polygon annotations
2. Add new annotations to training set of FCN
3. Re-train FCN (pre-train on all, ﬁne-tune per object)⇒ Allows for adaptation to appearance and context of each object
4. Re-generate masks using FCN

11

Data Annotation
I Manual corrections ensure consistency and high quality
I Large savings in annotation timeI KITTI MOTS: only 13% of car boxes / 17% of pedestrian boxes manually annotated
I MOTSChallenge: 15% of pedestrian boxes manually annotated12

Evaluation Metrics

Evaluation Metrics
I We consider mask-based variants of the CLEAR MOT metrics[Bernardin and Stiefelhagen, 2008]
I Need to associate predictions to ground truth instances
I Box-based tracking: boxes might overlap
I Requires bi-partite matching
I Mask-based tracking: masks are disjoint
I Establishing correspondences is greatly simpliﬁed
I Hypothesized and ground truth masks are matched iff mask IoU > 0.5

14Evaluation Metrics(Soft) Multi-Object Tracking and Segmentation Accuracy / Precision:
MOTSA = 1 −MOTSP =gT
P|T P ||T P | − |F P | − |IDS||F N | + |F P | + |IDS|
=|M |
|M |

sMOTSA =gT
P − |F P | − |IDS|
|M |

f=TPX

IoU(h, c(h))

h∈T PI c: mapping from hypotheses to ground truth
f soft number of true positives
I TP: true positives, TP:I FN: false negatives, FP: false positives, IDS: ID switches
I M: set of ground truth segmentation masks15

TrackR-CNN Baseline...

TrackR-CNN

FeatureExtractiont-1

DuringTraining

Image
FeaturesBounding Box
Regression...
Temporally
Enhanced
Image
FeaturesSharedweights
FeatureExtraction

t2x
3D ConvSharedweights
FeatureExtraction

...

t+1

Loss

Classification+ Scoring

Region
Proposal
Network

CAR:0.99
0.99
CAR:
0.99
CAR:
CAR:
0.99
CAR:
0.99

Image InstanceSegmentation
Ground Truth

Loss Video Tracking
Ground Truth
DuringOnline Track Evaluation
AssociationMask
Generation

Previously
TrackedObjects...
AssociationEmbedding
128-D AssociationVectors

Key Idea:
I Detection, segmentation, and data association with a single ConvNet
I Extend Mask R-CNN by 3D convolutions and association head17

TrackR-CNN

Association Head:
I Predict association vector
for each detection
I Detections of same instance should
be close in embedding space
I Detections of distinct instances
should be distant from each other18

TrackR-CNN
Training:
I Learned using batch-hard triplet loss [Hermans et al., 2017]:
1 X
max|D|d∈Dmax kae − ad k2 − min kae − ad k2 + α, 0e∈D:

ide =idd

e∈D:

ide 6=idd

I Mini-batch: 8 consecutive frames
I Mine furthest detection of same instance and closest detection of other instance
I Require separation by not more than margin α
Inference:
I Associate detections over time based on
Euclidean distance in embedding space and bi-partite graph matching
19

Experimental Evaluation

Results of TrackR-CNN on MOTSChallenge

I Crowded scenes can lead to missing detections and id switches
21

Results of TrackR-CNN on MOTSChallenge

I Crowded scenes can lead to missing detections and id switches
21

Results of TrackR-CNN on MOTSChallenge

I Crowded scenes can lead to missing detections and id switches
21

Results of TrackR-CNN on MOTSChallenge

I Crowded scenes can lead to missing detections and id switches
21

Results of TrackR-CNN on MOTSChallengeI Crowded scenes can lead to missing detections and id switches
21

Results of TrackR-CNN on MOTSChallenge

I Crowded scenes can lead to missing detections and id switches
21

Results of TrackR-CNN on MOTSChallenge

I Crowded scenes can lead to missing detections and id switches
21

Results of TrackR-CNN on MOTSChallenge

I Crowded scenes can lead to missing detections and id switches
21

Results of TrackR-CNN on KITTI MOTS

I Most objects distinguished well but some erroneous detections remain (red)22Results of TrackR-CNN on KITTI MOTS

I Most objects distinguished well but some erroneous detections remain (red)22Results of TrackR-CNN on KITTI MOTS

I Most objects distinguished well but some erroneous detections remain (red)22Results of TrackR-CNN on KITTI MOTS

I Most objects distinguished well but some erroneous detections remain (red)22Results of TrackR-CNN on KITTI MOTS

I Continuation of track with same ID after missing detection (red)23Results of TrackR-CNN on KITTI MOTS

I Continuation of track with same ID after missing detection (red)23Results of TrackR-CNN on KITTI MOTS

I Continuation of track with same ID after missing detection (red)23Comparison to Box Detection + Mask Prediction

Top: TrackR-CNN

Bottom: TrackR-CNN (box) + Mask R-CNN

I Training with masks avoids confusion between similar nearby objects

24Comparison to Box Detection + Mask Prediction

Top: TrackR-CNN

Bottom: TrackR-CNN (box) + Mask R-CNN

I Training with masks avoids confusion between similar nearby objects

24Quantitative Results on KITTI MOTSsMOTSAMOTSA

MOTSP

CarPedCarPedCar

Ped

TrackR-CNN (mask)

76.2

46.8

87.8

65.1

87.2

75.7

Mask R-CNN + Optic Flow Propagation

75.1

45.0

86.6

63.5

87.1

75.6

TrackR-CNN (box) + Mask R-CNN

75.0

41.2

87.0

57.9

86.8

76.3

GT Boxes (orig) + Mask R-CNN

77.3

36.5

90.4

55.7

86.3

75.3

GT Boxes (tight) + Mask R-CNN

82.5

50.0

95.3

71.1

86.9

75.4

I TrackR-CNN improves over training on single instances and box tracks
I Compared to the ﬂow propagation baseline, our method runs in real-time25Quantitative Results on MOTSChallengesMOTSA

MOTSA

MOTSP

jCC [Keuper et al., 2018] + Mask R-CNN

52.7
48.0
49.3
47.8
48.3

66.9
62.7
64.0
61.1
63.0

80.2
79.8
79.7
80.0
79.9

GT Boxes (tight) + Mask R-CNN

55.8

74.5

78.6

TrackR-CNN (mask)
MHT-DAM [Kim et al., 2015] + Mask R-CNN
FWT [Henschel et al., 2018] + Mask R-CNN
MOTDT [Long et al., 2018] + Mask R-CNN

I MOTS is challenging – even with perfect ground truth bounding boxes
I Segmenting pedestrians in crowded scenes is difﬁcult
26

Ablation Study: Temporal Model on KITTI MOTS

Temporal componentsMOTSA

MOTSA

MOTSP

CarPedCarPedCar

Ped

1xConv3D

76.1

46.3

87.8

64.5

87.1

75.7

2xConv3D

76.2

46.8

87.8

65.1

87.2

75.7

1xConvLSTM

75.7

45.0

87.3

63.4

87.2

75.6

2xConvLSTM

76.1

44.8

87.9

63.3

87.0

75.2

None

76.4

44.8

87.9

63.2

87.3

75.5

I Conv3D improves for pedestrians, but ConvLSTM does notI But overall effect is limited → Better ways to incorporate temporal context?
27

Ablation Study: Association Mechanism on KITTI MOTS

Association Mechanism

sMOTSA

MOTSA

MOTSP

Car

PedCarPedCarPed

Association head

76.2

46.8

87.8

65.1

87.2

75.7

Mask IoU

75.5

46.1

87.1

64.4

87.2

75.7

Bbox IoU

75.4

45.9

87.0

64.3

87.2

75.7

Bbox Center

74.3

43.3

86.0

61.7

87.2

75.7

I Mask IoU: associate based on IoU of mask warped using optic ﬂow (PWC-Net)
I Bbox IoU: associate based on bounding box warped using median optic ﬂow
I Bbox Center: associate based on unwarped box center distance28More Results

29SummaryI MOTS: new task, annotations, metrics, baselines
I Training beneﬁts from time-consistent instance segmentations compared toI Single image instance segmentations
I Box-based tracking data

I Be the ﬁrst to beat our baseline!
I Annotations and code: https://www.vision.rwth-aachen.de/page/mots

30

KITTI MOTS Challenge

Coming soon: http://www.cvlibs.net/datasets/kitti/eval mots.php
31

Thank you!
http://autonomousvision.github.io