Supplementary Material for
MOTS: Multi-Object Tracking and Segmentation
Paul Voigtlaender1 Michael Krause1 Aljos̆a Os̆ep1 Jonathon Luiten1
Berin Balachandar Gnana Sekar1 Andreas Geiger2 Bastian Leibe1
1
RWTH Aachen University 2 MPI for Intelligent Systems and University of Tübingen{voigtlaender,osep,luiten,leibe}@vision.rwth-aachen.de{michael.krause,berin.gnana}@rwth-aachen.de andreas.geiger@tue.mpg.de

1. Losses for the Association Headand the contrastive loss [2]TrackR-CNN uses association scores based on vectors
predicted by an association head to identify the same objectacross time. In our baseline model, we train this head using a batch hard triplet loss proposed by Hermans et al. [3],which we state again here: Let D denote the set of detections for a video. Each detection d ∈ D has a corresponding
association vector ad and is assigned a ground truth track
id idd determined by its overlap with the ground truth objects (we only consider detections which sufficiently overlap with a ground truth object here). For a video sequenceof T time steps, the association loss in the batch-hard formulation with margin α is then given by

Lcontrastive =Lbatch hard =1 Xmax
|D|d∈Dmax kae − ad k−d∈Dide =idd

X Xd∈Dmax(α − kae − ad k, 0)2 .

e∈D

ide 6=idd(3)
Table 1 compares the performance of these different variants of the loss function on the KITTI MOTS validation set.
It can be seen that the batch hard triplet loss performs better than just considering all pairs of detections (Batch AllTriplet), or using the conventional contrastive loss (Contrastive). Especially for pedestrians performance using thecontrastive loss is low.

e∈D:

ide =idd

min kae − ad k + α, 0 .

(1)

e∈D:
ide 6=idd

Intuitively, each detection d is selected as an anchor andthen the most dissimilar detection with the same id is selected as a hard positive example and the most similar detection with a different id is selected as a hard negative example for this anchor. The margin α and maximum operation
ensure that the distance of the anchor to the hard positive issmaller than its distance to the hard negative example by atleast α.
In order to justify our choice of the batch-hard loss, wealso report results using two alternative loss formulations,
namely the batch all loss [3] which considers all pairs ofdetections, i.e.
Lbatch all =1 X X
kae − ad k2 +|D|2e∈D

1 XX
max kae − ad k−|D|2d∈D e∈Dkae − ad k + α, 0(2)Association LosssMOTSA
Car
Ped

Batch Hard Triplet
Batch All Triplet
Contrastive

76.2
75.5
76.4

46.8
45.3
43.2

MOTSA
Car
Ped
87.8
87.1
88.7

65.1
63.8
61.5

MOTSPCar
Ped
87.2
87.1
86.7

75.7
75.6
75.2

Table 1: Different Association Losses for TrackR-CNN.
Comparison of results on the KITTI MOTS validation set.

2. Details of the Annotation Procedure
We noticed that wrong segmentation results often stemfrom imprecise or wrong bounding box annotations of theoriginal MOT datasets. For example, the annotated bounding boxes for the KITTI tracking dataset [1] are amodal,i.e., they extend to the ground even if only the upper bodyof a person is visible. In these cases, our annotators corrected these bounding boxes instead of adding additional
polygon annotations. We also corrected the bounding box

sMOTSA

MOTSA

MOTSP

CarPedCarPedCarPed

GT Boxes (orig) + Filling
GT Boxes (orig) + Ellipse
GT Boxes (orig) + MG

33.7
52.3
77.3

-66.1
-31.9
36.5

55.5
74.0
90.4

-57.7
-14.5
55.7

71.8
74.9
86.3

54.6
57.4
75.3

GT Boxes (tight) + Filling
GT Boxes (tight) + Ellipse
GT Boxes (tight) + MG

61.3
70.9
82.5

-1.7
17.2
50.0

83.9
91.8
95.3

22.0
42.4
71.1

75.4
78.1
86.9

60.5
64.2
75.4

Table 2: Ground Truth Results on KITTI MOTS. +MGdenotes mask generation with a KITTI MOTS fine-tuned
Mask R-CNN..
level tracking annotations in cases where they containederrors or missed objects. Finally, we retained ignore regions that were labeled in the source datasets, i.e., image
regions that contain unlabeled objects from nearby classes(like vans and buses) or target objects that were to small tobe labeled. Hypothesized masks that are mapped to ignore
regions are neither counted as true nor as false positives inour evaluation procedure.

3. Ground Truth Experiments
We performed additional experiments to demonstrate thedifficulty of generating accurate segmentation masks evenwhen the ground truth bounding boxes are given (see Table 2). As in the main paper, we consider two variants ofthe ground truth: the original bounding boxes from KITTI(orig), which are amodal, i.e. if only the upper body of aperson is visible, the box will still extend to the ground,and tight bounding boxes (tight) derived from our segmentation masks. We created masks for the boxes by simply
filling the full box (+Filling), by inserting an ellipse (+Ellipse), and by generating masks using the KITTI MOTS
fine-tuned Mask R-CNN (+MG). In each case, instance ids
are retained from the corresponding boxes.
Our results show that rectangles and ellipses are notsufficient to accurately localize objects when mask-basedmatching is used, even with perfect track hypotheses. The
problem is amplified when using amodal boxes, which often contain large regions that do not show the object. Thisfurther validates our claim that MOT tasks can benefit frompixel-wise evaluation. The relatively low scores for pedestrians also imply a limit to post-hoc masks generation using
the KITTI fine-tuned Mask R-CNN.

4. Visualization of Association Vectors
We present a visualization of the association vectors produced by our TrackR-CNN model on a sequence of theKITTI MOTS validation set in Figure 1. Here, all association vectors for detections produced by TrackR-CNN on se-quence 18 were used for principal component analysis andthen projected onto the two components explaining most of
their variance. The resulting two dimensional vectors wereused to arrange the crops for the corresponding detectionsin 2D. The visualization was created using the TensorBoard
embedding projector. It can be seen that crops belonging
to the same car are in most cases close to each other in theembedding space.

5. Qualitative results
We present further qualitative results of our baseline
TrackR-CNN model on the KITTI MOTS and MOTSChallenge validation sets including some illustrative failurecases. See Figures 2, 3, 4 and 5 on the following pages.

References[1] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the KITTI vision benchmark suite. InCVPR, 2012. 1[2] R. Hadsell, S. Chopra, and Y. Lecun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006. 1[3] A. Hermans, L. Beyer, and B. Leibe.
In defense ofthe triplet loss for person re-identification. arXiv preprint
arXiv:1703.07737, 2017. 1

Figure 1: Visualization using PCA on the association vectors of detections generated by TrackR-CNN on sequence 18of KITTI MOTS. Detections with similar appearance are grouped together by minimizing the association loss.

Figure 2: Qualitative Results on MOTSChallenge. While complex scenes with many occluding objects often work well,there can still be missing detections and id switches during difficult occlusions, as in this example (highlighted by redellipses).

Figure 3: Qualitative Results on KITTI MOTS. In simpler scenes, the model is able to continue a track with the same ID
after a missing detection (highlighted by red ellipses).

Figure 4: Qualitative Results on KITTI MOTS. In a rare failure case, pylons are confused for pedestrians (highlighted byred ellipses). In most cases, detections correspond to real instances of the class.

Figure 5: Qualitative Results on KITTI MOTS. In less crowded scenes, distinguishing objects works well but some erroneous detections (highlighted by red ellipses) might still happen.