STEP: Segmenting and Tracking Every Pixel
Andreas Geiger
Autonomous Vision Group
University of Tübingen and MPI for Intelligent Systems

Covered Paper

STEP: Segmenting and Tracking Every Pixel
Mark Weber, Jun Xie, Maxwell Collins, Yukun Zhu,Paul Voigtlaender, Hartwig Adam, Bradley Green, Andreas Geiger, Bastian Leibe, Daniel Cremers, Aljosa
Osep, Laura Leal-Taixe and Liang-Chieh Chen
NeurIPS Track on Datasets and Benchmarks 2021

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

2Overview

I Task: Video Panoptic SegmentationI Goal: Assign semantic classes and track identities to all pixels in a video
I Contribution: New benchmarks (KITTI-STEP, MOTChallenge-STEP) & new metric
M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

3

Why Segmentation matters

I Estimating drivable area
I Semantic understanding of surroundings
I Pixel-precise instance understanding
M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

4

Why Tracking matters

I Anticipate the temporal evolution of objects
I Obstacle avoidance
I Motion planning
M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

5

Segmenting and Tracking every Pixel

MOTS: No dense segmentation

Panoptic Segm.: No tracking

I Existing datasets and benchmarks either lack dense segmentation or tracking
I Our goal: Segmenting and tracking every pixel (STEP) for long time periods
I Video Panoptic Segmentation [Kim et al. 2020], but new benchmarks and metrics
M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

6

Evolution of Visual Scene Understanding

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

7

From Tracking to STEP

New: Spatially and temporally dense annotated KITTI-STEP and MOTChallenge-STEP

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

8

From Tracking to STEP

New: Spatially and temporally dense annotated KITTI-STEP and MOTChallenge-STEP

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

8

From Tracking to STEP

New: Spatially and temporally dense annotated KITTI-STEP and MOTChallenge-STEP

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

8

From Tracking to STEP

New: Spatially and temporally dense annotated KITTI-STEP and MOTChallenge-STEP

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

8

Contributions

I KITTI-STEP and MOTChallenge-STEPI A novel pixel-centric metric STQ

1

ST Q = (SQ × AQ) 2

I Baselines tackling bothsegmentation and tracking

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

9KITTI-STEP and MOTChallenge-STEPExisting Datasets
I KITTI-MOTS andMOTSChallenge

Missing segmentation
labels.

I Cityscapes-VPS

Every clip has 6 annotated frames (every 5thframe) and spans only
1.8 seconds.

I Synthetic datasets

Issues with insufﬁcientphoto-realism and thus
domain shift.

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

11

Annotation Process
I KITTI-MOTS andMOTSChallenge as basis
I Annotate every framesemi-automatically withsemantic segmentation
I Merge tracks andsemantic segmentation

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

12

Dataset Comparison (Training Set)
Dataset statistics

Cityscapes-VPS

KITTI-STEP

MOTChallenge-STEP# Sequences (trainval/test)# Frames (trainval/test)# Tracking classes# Semantic classes

450 / 502,700 / 30081921 / 29
8,008 / 10,1732
192/2
1,125 / 950
17# Annotated Masks†
Every frame annotated
Annotated frame rate (FPS)

72,171
73.4

126,529310

17,232
330

Annotated frames per seq.†
Track length (frames)†

Max/Mean/Min
6/6/6
6/3/1

Max/Mean/Min
1,059 / 381 / 78
643 / 51 / 1

Max/Mean/Min
600 / 562 / 525
569 / 187 / 1M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

13

Tracking the most salient classes

KITTI-STEP

MOTChallenge-STEPM. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

14

Track length distribution

KITTI-STEP

MOTChallenge-STEP

I In real-world sequences, tracks last much longer than a few frames
I STEP enables evaluation of long-term tracking

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

15

STQ: Segmentation and Tracking Quality

Why a new metric?
The focus of benchmark papers is usually on the dataset and on baselines.
Few papers do a thorough analysis on the metric aspect.
I What is a bad metric?
I What is a good metric?
I What properties do we want?
I Can the metric be tricked?
Existing metrics such as Video Panoptic Quality [Kim et al. 2020] andPanoptic Tracking Quality [Hurtado et al. 2020] build upon metrics for panoptic
segmentation and multi-object tracking, thereby inheriting their drawbacks.
M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

17

Metric design
Metric Properties

STQPTQ

VPQ

P1: Analyze full videos at pixel level (not segment level)
P2: Avoid thresholding (e.g., for TP vs. FP classiﬁcation)
P3: No penalty for ID recovery (correcting mistakes)
P4: Consider precision and recall for association
P5: Decouple segmentation and tracking errors

3
3
3
3
3

7
7
7
77(3)
77(3)
7

I Panoptic Tracking Quality (PTQ): Penalizes error recovery, negative scores
I Video Panoptic Quality (VPQ): Designed for sparse annotations & short clips

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

18

Metric design†Nr.

STQ (↑)PTQ (↑)

VPQ† (↑)#10.71

1.00.0VPQ computed on whole sequence.

I STQ is the only metric that properly penalizes ID transfer

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

19

Metric design†Nr.

STQ (↑)PTQ (↑)

VPQ† (↑)#2#30.72
0.82

0.8
0.8

0.40.53

VPQ computed on whole sequence.

I STQ and VPQ encourage long-term track consistency

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

19

Metric design†Nr.

STQ (↑)PTQ (↑)

VPQ† (↑)

#4
#5

0.79
0.65

0.75
0.86

0.5
0.75

VPQ computed on whole sequence.

I Only STQ reduces score when removing semantically correct segments

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

19Formal Deﬁnition

The task of Segmenting and Tracking Every Pixel (STEP) requires a function
f (x, y, t) 7→ (c, id)
which maps every pixel (x, y, t) to a semantic class c and a track ID id.
I We denote the ground-truth as gt(x, y, t) and the prediction as pr(x, y, t)
I STQ measures Association Quality (AQ) and Segmentation Quality (SQ)

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

20

Association Quality (AQ)
We deﬁne the prediction and ground-truth for a particular id as follows:
prid (id) = {(x, y, t)|pr(x, y, t) = (∗, id)}
gtid (id) = {(x, y, t)|gt(x, y, t) = (∗, id)}
I The proposed AQ is designed to work at a pixel-level of a full video (P1)
I All associations have an inﬂuence on the score, no IoU threshold (P2)
I Semantic segmentation errors are not penalized in AQ (P5), only in SQ

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

21

Association Quality (AQ)
We deﬁne the true positive associations (TPA) of a speciﬁc ID as follows:
T P A(p, g) = |prid (p) ∩ gtid (g)|Similarly, false negative associations (FNA) and false positive associations (FPA) canbe deﬁned to compute precision Pid and recall Rid . To account for the effect of bothprecision and recall (P4), we deﬁne the basic building block IoUid for AQ as follows:
IoUid (p, g) =Pid (p, g) × Rid (p, g)
Pid (p, g) + Rid (p, g) − Pid (p, g) × Rid (p, g)

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

22

Association Quality (AQ)

Following our goal of long-term track consistency, we encourage ID recovery (P3) byweighting the score of each predicted tube by its TPA. Association Quality (AQ):AQ(g) =1|gtid (g)|XT P A(p, g) × IoUid (p, g),p,|p∩g|6=∅

1AQ =|gt tracks|X

AQ(g).

g∈gt tracks

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

23

Segmentation Quality (SQ)
We use Intersection-over-Union (IoU) to measure segmentation quality.
Formally, given pr(x, y, t), gt(x, y, t) and class c we deﬁne:prsem (c) = {(x, y, t)|pr(x, y, t) = (c, ∗)}gtsem (c) = {(x, y, t)|gt(x, y, t) = (c, ∗)}We then deﬁne the Segmentation Quality (SQ) as the mean IoU score:
IoU (c) =|prsem (c) ∩ gtsem (c)||prsem (c) ∪ gtsem (c)|SQ = mIoU =1 X
IoU (c)|C|c∈C

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

24

Segmentation and Tracking Quality (STQ)

We combine both scores into Segmentation and Tracking Quality (STQ) via thegeometric mean:
1

ST Q = (AQ × SQ) 2

I STQ hence requires methods to perform well in both segmentation and tracking

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

25

Baselines

Panoptic-DeepLab

Panoptic-DeepLab: [Cheng et al., 2017]
I State-of-the-art per-frame panoptic segmentation network
I 3 branches: semantic segmentation, center heatmap, pixel-to-center offsets.
M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

27

Extensions to Tracking
Single-frame baselines:
I B1: IoU Association. The predicted thing segments of two consecutive frames
are matched by Hungarian Matching with a mask IoU threshold δ = 0.3. To
account for occluded objects, unmatched predictions are kept for 10 frames.
I B2: SORT [Bewley et al., 2016]. Bi-partite matching between sets of Kalman ﬁlter
track predictions and object detections based on the bounding box overlap.
I B3: Mask Propagation. Uses RAFT optical ﬂow [Teed et al., ECCV] to warp eachpredicted mask at frame t − 1 into frame t, followed by the IoU matching (B1).
Multi-frame baseline:
I B4: Center Motion. Add prediction head to the base network in order to regress
every pixel to its instance center in the previous frame. Inspired by CenterTrack.
M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

28

Results

Results on KITTI-STEP
KITTI-STEP

OF

STQ

AQ

SQ

VPQ

PTQ

B1: IoU Association
B2: SORT
B3: Mask Propagation

773

0.58
0.59
0.67

0.47
0.50
0.63

0.71
0.71
0.71

0.440.42
0.44

0.48
0.48
0.49

B4: Center Motion
VPSNet (Kim et al.)

73

0.58
0.56

0.51
0.52

0.67
0.61

0.40
0.43

0.45
0.49

I Single-frame methods (separate segmentation and tracking) perform best
I Combining SotA segmentation and tracking yields best results (B3)
I More work needed to exploit full potential of multi-frame methods
M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

30

VideoM. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

31

Resources
I KITTI-STEP: http://www.cvlibs.net/datasets/kitti/eval step.php
I MOTChallenge-STEP: https://motchallenge.net/data/STEP-ICCV21/
I DeepLab2: https://github.com/google-research/deeplab2
I ICCV 2021 Workshop:

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

32

Summary

I We present a new perspective on the task of video panoptic segmentation
I We provide a new benchmark (STEP) focusing on measuring algorithm
performance at the most detailed level possible, taking each pixel into account
I Our benchmark and metric are designed for evaluating algorithms in real-world
scenarios where understanding long-term tracking performance is important
I We believe that this work provides an important STEP towards a dense,
pixel-precise video understanding

M. Weber et al.: STEP: Segmenting and Tracking Every Pixel. NeurIPS Track on Datasets and Benchmarks, 2021.

33

Thank you!
http://autonomousvision.github.io