CVPR 2020 Workshop on Autonomous Driving
1st Place Solution for Challenge 2:
BDD 100K Tracking
Fan Yang1,2, Xin Chang1, Sakriani Sakti1,2, Satoshi Nakamura1,2, Yang Wu3
1Nara Institute of Science and Technology, Japan
2RIKEN Center for Advanced Intelligence Project, Japan
3Kyoto University, Japan0

Background of BDD 100K Tracking Challenge• Problem: detect and track multiple objects in videos, there are 8 categories of objects, as
Pedestrian, Rider, Car, Bus, Truck, Train, Motorcycle, Bicycle.
• Input: a video sequence contain that multiple RGB images.
• Output: 2D bounding boxes and corresponding track ID at each frame.

1Difficulties in BDD 100K Tracking
2. Weak Illuminations at night scenes

1. Classification ambiguity

pedestrianorrider?

2

Object Detection (off-the-shelf method)
Detector:
Faster_rcnn_X_101_32x8d_FPN of Detectron 2(https://github.com/facebookresearch/detectron2)

Training Dataset:
BDD 100K object detection training set, labeled key frame images extracted from the videos at 10th
second.

Inference Setting:Confidence threshold: 0.6
NMS IoU threshold: 0.75
Crop patch size: resize to be 128 x 128 pixels

3Difficulties in BDD 100K Tracking
2. Weak Illuminations at night scenes

1. Classification ambiguity

pedestrianorrider?

3. Fast motion
Our focus, solving thisissue by proposed arobust trajectory
initialization method.

4

The Trajectory Initialization for Tracking (our main focus)
To model the trajectory by
Kalman Filter or others
Without correct trajectory,
cannot predict the correct
future position.

chickenand egg
problemWithout the correctly predicted
future position, cannot correctly
initialize the trajectory.

Exiting solutions:
Initialization by copy-paste boxes to calculate center distance or IoU (e.g., DeepSORT [1])Defects: Fast moving objects do not have overlapped positions across frames.
Initialization trajectory by appearance features (e.g., MOTbyReid [2])Defects: Object may have the similar appearance features, especially for cars at night scenes.
Our solution:
Initialization trajectory by hard fusing appearance features and IoU of expanded boxes.
[1] Wojke at el. “Simple Online and Realtime Tracking with a Deep Association Metric” (ICIP17)[2] Tang at el. “Multiple People Tracking by Lifted Multicut and Person Re-identification” (CVPR17)

5Our Initialization (our main contribution)
Initializing trajectory by hard fusing appearance features and IoU of expanded boxes.
frame kThe ground truth isframe k+1(,)

But between
andcenter distance isshort, IoU is larger.
It is the first time to observe
We expand

to beat frame k, we cannot predict its position at frame k+1.

and check its IoU with observations at frame k+1.

is excluded due to the IoU (,)=0.

is excluded due to both of the height and width has reduced 2 times (approx. 100 m).
We compare the appearance similarity fromby linear assignment.

toandonly to decide the association

6Our Initialization (our main contribution)
It is challenging to estimate the accurate motion.
High IoU does not equal the high correlation
frame 1frame 2

no estimated motion
for the initialization" 𝟐𝐚𝐎

𝐎𝟏𝐚𝐎𝟐𝒂! 𝟐𝐚 , 𝐎𝟐𝐛 ) > 0
IoU (𝐎! 𝟐𝐚 , 𝐎𝟐𝒂 ) = 0
IoU (𝐎! 𝟐𝐚 , 𝐎𝟐𝐛 ) < C2CD (𝐎! 𝟐𝐚 , 𝐎𝟐𝒂 )C2CD (𝐎𝐎𝟐𝒂! 𝟐𝐚 ), 𝐎𝟐𝐛 ) > 0
IoU (Exp( 𝐎! 𝟐𝐚 ), 𝐎𝟐𝒂 ) > 0IoU (Exp( 𝐎! 𝟐𝐚 , 𝐎𝟐𝐛 ) < C2CD (𝐎! 𝟐𝐚 , 𝐎𝟐𝒂 )C2CD (𝐎𝐎𝟐𝐛frame 1frame 2

no estimated motion
for the initialization

𝐎𝟏𝐚𝐎𝟐𝐛" 𝟐𝐚𝐎

7

Our Initialization (our main contribution)
It is challenging to estimate the accurate motion.
High IoU does not equal the high correlation
frame k𝐎𝐤𝐚

frame k+1
real motion

" 𝐤'𝟏
𝐎𝐚

𝐎𝐤'𝟏
𝒂

estimated motion

𝐎𝐤'𝟏𝐛
frame k

𝐎𝐤𝐚

𝐤&𝟏
! 𝐤&𝟏
! 𝐤&𝟏 𝐤&𝟏
IoU (𝐎𝐚 , 𝐎𝐛 ) < IoU (𝐎𝐚 , 𝐎𝒂 )
𝐤&𝟏
! 𝐤&𝟏
! 𝐤&𝟏 𝐤&𝟏C2CD (𝐎𝐚 , 𝐎𝐛 ) < C2CD (𝐎𝐚 , 𝐎𝒂 )
𝐤&𝟏
! 𝐤&𝟏
! 𝐤&𝟏 𝐤&𝟏
SoR (𝐎𝐚 , 𝐎𝐛 ) < SoR(𝐎𝐚 , 𝐎𝒂 )frame k+1
real motion
estimated motion" 𝐤'𝟏
𝐎𝐚𝐎𝐤'𝟏
𝒂𝐎𝐤'𝟏
𝐛

𝐤&𝟏
! 𝐤&𝟏
! 𝐤&𝟏 𝐤&𝟏
IoU (𝐎𝐚 , 𝐎𝐛 ) > IoU (𝐎𝐚 , 𝐎𝒂 )
𝐤&𝟏
! 𝐤&𝟏
! 𝐤&𝟏 𝐤&𝟏
C2CD (𝐎𝐚 , 𝐎𝐛 ) > C2CD (𝐎𝐚 , 𝐎𝒂 )
𝐤&𝟏
! 𝐤&𝟏
! 𝐤&𝟏 𝐤&𝟏
SoR (𝐎𝐚 , 𝐎𝐛 ) < SoR(𝐎𝐚 , 𝐎𝒂 )8

Our Initialization (our main contribution)
Cascade Matching
App + IoU
Matching
MatchedTracks

Unmatched
Detections

Unmatched
TracksCascadeMatching

Observations

IoU
Matching
MatchedTracks

Unmatched
Detections

Unmatched
Tracks9

Our Initialization (our main contribution)

10

Our Initialization (our main contribution)

ID = 1

ID = 1

ID = 2

11

MOTS20-01 (static camera)

MOTS20-06 (stroller-mounted camera)

MOTS20-12 (stroller-mounted camera)

KITTI-MOTS-0003 (car-mounted camera, turning scene)

KITTI-MOTS-0018 (car-mounted camera, pedestrian-car scene)Our Initialization (our main contribution)

13

Our Initialization (our main contribution)

123

Observations

MotionFeaturePositionAffinity

Assignment
Optimization
Matching

Observations

Appearance
FeatureAppearance
Affinity

Assignment
Optimization
Matching

Observations

MotionFeatureMotionAffinity

Appearance
FeatureAppearance
Affinity

Motion Feature

4

Observations
Appearance Feature

IoU = 0Assignment
Optimization
Matching

Neural Network
Matching

Inf Distance

Detections
IoU > 0

Fused
Affinity

Appearance
Distance

FilteredDistanceMatching

14

Our Initialization (our main contribution)

12

34

Single-instanceObject Tracking

Single-classMulti-instanceObject Tracking

Single-classMulti-instance Object
Tracking and Segmentation

Multi-classMulti-instanceObject Tracking

Our Focus

Detection

Online DataAssociation

Offline DataAssociation

Section 3

Section 4

Model-agnostic
Data Association
Refinement
Section 515

Our Tracking
Our strategy to fuse appearance feature and trajectory:
Always decide the association by appearance, but constrained by box trajectory.

frame k+1
Initialized Speed

Initialized Speedframe k+2

Appearance Encoder

BoT-Reid [3]For predicted position offrame k, consider all of IoU> 0 observations of frame
k+1 for matching

CosineDistance

EmbeddingAppearanceFeatures[3] Luo at el. “Bag of Tricks and a Strong Baseline for Deep Person Re-Identification” (TMM20)

0.2

0.4

infinfinf

0.3

LinearAssignmentAppearance Distance Matrix

16Evaluation of Our Results

Challenges in correctly detect objects

17

Conclusion• Good trajectory initialization are important for MOT when objects aremoving fast.
• We proposed an robust trajectory initialization approach by hard fusing
appearance features and IoU of expanded boxes.

18Thanks for your listening